Summary of Reddit Comments on Video Cards/GPUs for LLM Models

NVIDIA GPUs

• NVIDIA RTX 3090 and RTX 3080 are popular choices for LLM tasks due to their high VRAM.
  • Pros: Great computational power, useful for parallel processing, availability of applications optimized for multi-GPU setups.
  • Cons: Complex setup, increased power consumption, heat generation, and cost.
  • Dual GPUs offer redundancy in case of failure.
• RTX 2080 Ti with 22GB VRAM is a cost-effective option and can provide high performance per dollar.
• Tesla V100 SXM3 offers high performance but requires a specific server setup, and an adapter may be needed for PCIe.

System Setup Considerations

• Consider a system with ample PCIe lanes and a powerful PSU to accommodate multiple GPUs.
• A system with DDR5 memory or a strong CPU like AMD Ryzen 9 5900X/5950X is recommended.
• Linux is often favored over Windows for LLM tasks due to better support.

Recommendations for Budget

• Aim for 2x RTX 3090 or RTX 3090 Ti and build the rest of the PC around it for a comprehensive system.
• Consider used GPUs like RTX 3090, 2x RTX 4060 16GB, or modded RTX 2080 Ti for budget-friendly options.

Connectivity and Interface

• Choose reliable connectors like Oculink 4i for GPUs, as Thunderbolt may be unstable.
• Invest in AI accelerators like Coral's accelerator for specialized AI tasks.

Alternative Solutions

• Consider AI accelerators as a potential alternative to traditional GPUs for specialized tasks.
• General opinion leans towards GPUs like RTX 3090 for cost-effective high performance in LLM models.

By combining these recommendations, users can create an efficient and powerful system tailored to their LLM requirements while considering factors like cost, performance, and compatibility.