What is GPU?

GPUs are the hardware backbone of the AI revolution. When researchers discovered in the early 2010s that GPUs could accelerate deep learning training by orders of magnitude compared to CPUs, it triggered a cascade of breakthroughs that continues today. The reason GPUs are so effective for AI is architectural: while a CPU has a small number of powerful cores optimized for sequential tasks, a GPU has thousands of smaller cores designed to perform many simple calculations simultaneously. Matrix multiplication, the fundamental operation in neural networks, maps perfectly onto this parallel architecture.

Training large AI models requires vast amounts of GPU compute. Training a frontier large language model today requires thousands of high-end GPUs running for weeks or months, consuming megawatts of power and costing tens to hundreds of millions of dollars. This concentration of required compute is one reason why only a handful of organizations can train frontier models from scratch. The democratization of AI applications is only possible because trained models can be served via APIs and cloud AI platforms without each user needing their own GPU cluster.

For inference, GPU requirements are substantially lower than for training, though still significant at scale. Techniques like quantization, which reduces the numerical precision of model weights, and batching, which processes multiple requests together, improve GPU utilization efficiency. Small language models are partly attractive because they can perform inference on consumer-grade GPUs or even without GPUs entirely, enabling edge AI deployments on laptops and mobile devices.

The GPU supply chain has become a geopolitical issue as demand for AI compute has outstripped supply. NVIDIA dominates the AI GPU market, with its H100 and successor chips becoming the essential infrastructure of AI development. Alternative approaches including TPUs, custom AI accelerators from major cloud providers, and novel chip architectures are all competing to reduce dependence on a single supplier and improve the economics of AI compute at scale.