Two years into the generative-AI demand surge, the infrastructure story is no longer “Nvidia has the GPUs.” It’s “every major hyperscaler is designing its own silicon, the supply chain is being rearchitected around AI workloads, and enterprise buyers are quietly making procurement decisions that will shape their options for the next decade.” The chip conversation has moved from deep-tech curiosity to a first-order strategic concern.

As 2024 closes, the picture is more interesting than the narrow Nvidia-vs-AMD framing suggests.

The Nvidia Position

Nvidia’s dominance in training and, to a lesser degree, inference remains real. H100 and H200 GPUs are still the de facto default for new large-model training runs. The CUDA software ecosystem is a genuine moat — a decade of libraries, compilers, and developer familiarity that alternatives have struggled to dislodge. The Blackwell architecture ramping through late 2024 promises meaningful gains for large-scale training and inference alike.

But three things are changing the picture around this position, rather than unseating it outright: hyperscaler custom silicon, AMD’s credible challenge with MI300, and a flourishing inference-specialist segment.

Hyperscaler Silicon Is Serious Now

Every major hyperscaler has now shipped real custom AI silicon into production:

Google’s TPU line is the most mature, with v5e and v5p offerings widely used for both internal and customer workloads. Gemini’s training and serving at scale is on TPUs, which is the most concrete testament to the program’s maturity.

AWS Trainium and Inferentia have moved from “interesting option” to “competitive default for many workloads” over the course of 2024. Trainium2 is ramping into general availability; Inferentia2 has become the cost-optimized path for many production inference workloads on AWS.

Microsoft’s Maia 100 launched in 2024 as the first in-house AI accelerator, reflecting Microsoft’s desire to reduce dependence on Nvidia for the massive OpenAI-driven capacity it’s building out.

Meta’s MTIA is now on its second generation and running significant production inference workloads.

The pattern is consistent: each hyperscaler is using custom silicon to own more of the economics of their own AI workloads, particularly inference, where the margins matter most and the optimization opportunity is largest.

AMD’s Credible Challenge

AMD’s MI300X has found real traction through 2024. It’s not displacing Nvidia at the top end of training, but it’s become a credible option for large-memory inference workloads and for customers who want a second source. Microsoft, Meta, and Oracle are all running MI300X at meaningful scale. The ROCm software story is still rougher than CUDA, but the gap is narrower than a year ago.

For enterprise buyers, the significance isn’t that AMD will “win”; it’s that a two-supplier world is emerging at the high end for the first time in this cycle, which changes pricing dynamics and supply risk.

The Inference Specialists

A quieter but important development is the maturation of inference-specialist silicon. Groq’s LPU has demonstrated throughput numbers that are attracting serious attention for latency-sensitive workloads. Cerebras and SambaNova have continued to find niches. Startups like d-Matrix, Tenstorrent, and Rain AI are each pursuing distinct architectural bets that could matter as inference costs become the dominant AI line item for most enterprises.

The broader point: training and inference are splitting into distinct silicon markets with different winners.

What This Means for Enterprise Buyers

For most enterprises, the practical implications of the silicon landscape show up in two places: cloud costs and vendor optionality.

On cost, the choice of underlying accelerator is increasingly visible in pricing. Running on TPUs in GCP, Trainium on AWS, or Maia on Azure can reduce costs materially for equivalent workloads — but often requires porting work, framework adjustments, and the willingness to accept capability constraints.

On optionality, the advice most infrastructure leaders are hearing from their architecture teams: don’t lock in at the framework level to a single accelerator family if you can reasonably avoid it. PyTorch compiled for multiple backends, higher-level abstractions like JAX or Triton, and model formats that port across accelerators are worth the investment for anything that will run at scale.

Supply Chain and Capacity

The supply side remains tight. HBM (the high-bandwidth memory that gates AI accelerator capacity) is the real bottleneck through 2024; SK Hynix, Samsung, and Micron are all building aggressively, but demand continues to outpace. TSMC’s advanced packaging capacity is the other constraint.

For enterprise buyers, this manifests as uneven availability — spot GPU capacity is tight, and reserved capacity requires meaningful commitments. Hyperscalers’ ability to offer large customers dedicated capacity has become a competitive differentiator.

What to Watch in 2025

A few threads worth tracking: Nvidia’s response to hyperscaler internal silicon — whether it leans further into system-level differentiation (networking, memory, software) to defend share; the maturation of open-compiler stacks like Triton and MLIR that would lower the switching cost between accelerators; regulatory moves in both the US and China that continue to shape the export-controlled segment of the market; and the increasing visibility of inference-specific chip economics as GenAI workloads scale into production.

Conclusion

The silicon layer has quietly become one of the most consequential strategic surfaces in enterprise AI. For 2025 planning, the right frame isn’t “which chip is best” but “which silicon decisions am I making, knowingly or not, that will constrain my options in two years?” The answer for most enterprises is to optimize for portability at the software layer, negotiate at the capacity layer, and keep watching a market that is shifting faster than most infrastructure markets do.

References

Nvidia (2024). Blackwell Architecture Technical Brief.

Google Cloud (2024). TPU v5p and v5e: Performance Whitepaper.

AWS (2024). Trainium2 and Inferentia2 Technical Overview.

Microsoft (2024). Maia 100: Custom AI Accelerator Technical Overview.

Patel, D. & Nishball, D. (2024). The AI Hardware Landscape. SemiAnalysis.