Training a frontier AI model is a one-time cost. Serving it to a billion users, one token at a time, forever, is not — and that shift is why 2026 has turned into the most crowded chip race in computing history. Here's how Nvidia's Blackwell and Vera Rubin stack up against every serious inference challenger: hyperscaler custom silicon, dedicated startups, and the custom ASICs quietly being built by Broadcom and Marvell behind the scenes.
Why "Inference Chips" Are Suddenly Their Own Category
Training a model happens once. Inference — actually running that model for users — happens billions of times a day, forever, and now accounts for roughly two-thirds of total AI compute spend. That's why nearly every major tech company is now designing chips specifically optimized for the decode phase of inference (generating output tokens quickly and cheaply) rather than just buying general-purpose GPUs built for training.
Nvidia: Blackwell Today, Vera Rubin Next
Blackwell (GB200/GB300) is Nvidia's current-generation flagship, still the default choice for most large-scale AI training and inference. But the real headline of 2026 is Vera Rubin, unveiled at CES and now in full production.
Vera Rubin isn't one chip — it's a seven-chip platform: the Rubin GPU (336 billion transistors, TSMC 3nm, up to 288GB of HBM4 memory, 50 PFLOPS of NVFP4 inference compute), the Vera CPU (88 custom Arm "Olympus" cores), NVLink 6 switches, ConnectX-9 networking, BlueField-4 DPUs, and Spectrum-6 Ethernet. A full Vera Rubin NVL72 rack packs 72 GPUs and 36 CPUs, delivering 3.6 exaflops of NVFP4 inference and — according to Nvidia — up to 10x lower cost per inference token and 4x fewer GPUs needed to train mixture-of-experts models, compared with Blackwell.
Watch: What Vera Rubin Actually Changes
The Twist: Groq Is Now Part of Nvidia's Platform
This is the detail that changes how you should think about "Nvidia vs. Groq" as a rivalry: on Christmas Eve 2025, Nvidia signed a $20 billion non-exclusive licensing deal for Groq's LPU (Language Processing Unit) inference technology, and hired Groq's founder Jonathan Ross, president Sunny Madra, and much of the engineering team. Groq technically remains an independent company running GroqCloud under a new CEO, but the core LPU technology and its creators are now working inside Nvidia.
The first product of that deal, the Groq 3 LPU, was unveiled at GTC 2026 as the seventh chip in the Vera Rubin platform — an SRAM-based decode-phase accelerator (512MB SRAM per die, 150 TB/s bandwidth) that works alongside Rubin GPUs: Rubin handles the compute-heavy prefill stage, while Groq 3 LPUs handle fast, latency-sensitive token generation. A full LPX rack of 256 LPUs paired with a Vera Rubin NVL72 claims up to 35x higher inference throughput per megawatt than Blackwell alone, for trillion-parameter models. Regulators, including a Senate inquiry, are still scrutinizing whether the "licensing" structure functions as a de facto acquisition.
Hyperscaler Custom Silicon: Trainium, Ironwood, Maia, MTIA
Every major cloud provider now builds its own chips — mainly to cut inference costs for their own internal workloads, not to sell competing hardware to the public.
Amazon: Trainium3 (and Trainium4 coming)
AWS's Trainium3 is its first 3nm chip: 2.52 PFLOPS of FP8 compute, 144GB of HBM3e, and 4.9 TB/s of bandwidth per chip. A Trn3 UltraServer links up to 144 chips for 362 PFLOPS of FP8 compute and 706 TB/s of aggregate bandwidth, and UltraClusters can scale to hundreds of thousands of chips. Anthropic, OpenAI, and Uber are among the named customers. Trainium4 is already announced for late 2026/2027, promising 3x the FP8 throughput and — notably — support for Nvidia's NVLink Fusion interconnect, letting Trainium and Nvidia GPUs work in the same rack.
Google: Ironwood (TPU v7), with TPU v8 on the way
Ironwood is Google's seventh-generation TPU, the first designed specifically for inference rather than training: 4,614 FP8 TFLOPS per chip, 192GB of HBM3e, and pods that scale to 9,216 chips for 42.5 exaflops of combined compute. Anthropic has committed to using up to one million Ironwood TPUs. Google has already previewed an eighth generation that splits the job in two: TPU 8t for training and TPU 8i for inference, both targeting TSMC's 2nm process later in 2026.
Microsoft: Maia 200
Maia 200 is Microsoft's inference-first accelerator: TSMC 3nm, over 140 billion transistors, 216GB of HBM3e at 7 TB/s, plus 272MB of on-chip SRAM, all within a 750W envelope. Microsoft claims it beats Trainium3 on FP4 performance and Ironwood on FP8, with 30% better performance-per-dollar than its existing fleet. It currently powers OpenAI's GPT-5.2 models and Microsoft 365 Copilot — but it is not rentable by outside customers; it's purely internal infrastructure.
Meta: MTIA
Meta's Training and Inference Accelerator (MTIA) line has moved from a 7nm first generation through a 5nm second generation to a 300-series now in production, with a 400-series delivering 6 PFLOPS of FP8 and 18 PFLOPS of MX4 compute with 288GB of HBM. Like Maia, MTIA is entirely internal — it runs Meta's own recommendation and ranking systems, with no external cloud offering.
OpenAI's Own Chip: Jalapeño
In June 2026, OpenAI and Broadcom unveiled Jalapeño, OpenAI's first custom "Intelligence Processor" — an inference-only ASIC designed around OpenAI's own understanding of LLM workloads: memory movement, kernels, and serving patterns. It reportedly went from initial design to manufacturing tape-out in just nine months, partly using OpenAI's own models to help accelerate chip design. Initial deployment is targeted for the end of 2026, expanding through gigawatt-scale data centers with Microsoft and other partners in subsequent years. OpenAI has been explicit that this is an inference chip, not a training replacement — heavy training work will keep relying on Nvidia and AMD hardware for now.
AMD's Answer: Instinct MI400 Series
AMD's Instinct MI450 and flagship MI455X (CDNA 5 architecture, TSMC 2nm) pack 320 billion transistors, up to 40 PFLOPS of FP4 compute, and 432GB of HBM4 memory at 19.6 TB/s bandwidth — memory capacity AMD points to as an advantage over Nvidia's Rubin generation. The "Helios" rack-scale system bundles 72 MI455X accelerators for roughly 2.9 exaflops of FP4 performance. The headline deal: OpenAI committed to 6 gigawatts of AMD Instinct capacity, starting with a 1GW MI450 deployment in the second half of 2026 — a contract potentially worth $90 billion to AMD, sweetened with warrants letting OpenAI acquire up to roughly 10% of AMD's shares if milestones are hit.
Cerebras: The Wafer-Sized Chip
Cerebras' WSE-3 takes a completely different approach — instead of cutting a silicon wafer into hundreds of small chips, it uses almost the entire 300mm wafer as one processor: 4 trillion transistors, 900,000 AI cores, and 44GB of on-chip SRAM delivering roughly 21 petabytes per second of internal memory bandwidth (thousands of times faster than a GPU's HBM). Because there's no need to shuttle data between separate chips, Cerebras claims dramatic inference speed advantages — over 2,500 tokens per second per user on a 400-billion-parameter Llama 4 Maverick model, more than double Nvidia's DGX B200. Cerebras is targeting an IPO in 2026, though roughly 80% of its revenue reportedly comes from a single customer, UAE-based G42.
Watch: How the Wafer-Scale Chip Actually Works
Other Notable Chip Makers
- Qualcomm (AI200 / AI250): Data-center inference cards built around Qualcomm's Hexagon NPU. The AI200 (2026) offers a huge 768GB of LPDDR memory per card; the AI250 (2027) introduces near-memory computing for a claimed 10x effective bandwidth gain. Saudi Arabia's HUMAIN has committed to 200 megawatts of deployment starting in 2026.
- Intel (Crescent Island): Intel's first GPU built on its Xe3P architecture, explicitly engineered for data-center inference with an emphasis on low power and high throughput, targeted for 2026 launch — part of Intel's broader attempt to re-enter the AI accelerator conversation after its Gaudi line struggled to gain share.
- SambaNova: A reconfigurable dataflow architecture (RDU) competing on throughput and memory capacity for large-model serving; reportedly in acquisition talks with Intel at a valuation far below its earlier private funding rounds.
- Tenstorrent: RISC-V-based inference processors led by veteran chip architect Jim Keller, positioning against proprietary instruction sets like Nvidia's CUDA ecosystem.
The Company Behind Most of These Chips: Broadcom
A recurring name across this entire list is Broadcom, which doesn't sell a competing chip brand of its own — it's the design and manufacturing partner that turns a hyperscaler's architecture into real, working silicon at TSMC. Broadcom is the confirmed design partner behind Google's TPU line, Meta's MTIA, and OpenAI's new Jalapeño chip, and reports over $73 billion in AI-related backlog, with a public target of $100 billion in annual AI chip revenue by 2027. Marvell is the main rival in this "ASIC design partner" role, with confirmed work on Amazon's Trainium program. In effect, the AI chip war isn't just Nvidia versus everyone else — it's also a quieter competition between Broadcom and Marvell to be the silicon partner of choice for the next hyperscaler that wants to build its own chip.
Watch: The Custom-Silicon Race in Context
Side-by-Side Comparison
| Chip / Platform | Maker | Approach | Public Availability |
|---|---|---|---|
| Blackwell / Vera Rubin | Nvidia | General-purpose GPU + rack-scale co-design | Yes — every major cloud |
| Groq 3 LPU | Nvidia (ex-Groq team) | SRAM-heavy, deterministic decode accelerator | Via Vera Rubin racks + GroqCloud |
| Trainium3 / 4 | Amazon (AWS) | Custom ASIC, training + inference | Yes — AWS EC2 |
| Ironwood (TPU v7) | Google (w/ Broadcom) | Custom ASIC, inference-optimized | Yes — Google Cloud only |
| Maia 200 | Microsoft | Custom ASIC, inference-only | No — internal only |
| MTIA | Meta (w/ Broadcom) | Custom ASIC, ranking/recommendation + inference | No — internal only |
| Jalapeño | OpenAI (w/ Broadcom) | Custom ASIC, inference-only | No — OpenAI infrastructure |
| Instinct MI450 / MI455X | AMD | General-purpose GPU | Yes — multiple clouds |
| WSE-3 / CS-3 | Cerebras | Wafer-scale single chip | Yes — Cerebras Cloud, AWS |
| AI200 / AI250 | Qualcomm | Custom NPU-based inference card | Yes — launching 2026/2027 |
Frequently Asked Questions
Is Groq still a competitor to Nvidia, or part of Nvidia now?
Both, in a sense. Groq licensed its core LPU technology and much of its leadership team to Nvidia for $20 billion, and the resulting Groq 3 LPU now ships as part of Nvidia's Vera Rubin platform. Groq itself continues operating independently as GroqCloud under new leadership, but the relationship is far closer than a typical rivalry.
Can I actually rent Google's TPU, Microsoft's Maia, or Meta's MTIA?
TPUs are rentable through Google Cloud. Maia and MTIA are not — both remain purely internal infrastructure for Microsoft's and Meta's own products.
Which chip is fastest for inference?
It depends on the metric. Cerebras and Groq-class SRAM-heavy chips post the highest raw tokens-per-second numbers for latency-sensitive, single-user workloads. Nvidia's Vera Rubin and the major hyperscaler ASICs are built more for throughput and cost efficiency at massive concurrent scale. There is no single "fastest" chip across every workload.
Why does Broadcom keep showing up in this story?
Broadcom doesn't sell a competing chip brand — it's the engineering and manufacturing partner that helps companies like Google, Meta, and OpenAI turn their own chip designs into real silicon at TSMC. It's effectively the "arms dealer" behind much of the custom-ASIC side of this race.
Bottom Line
Nvidia still supplies the large majority of AI compute, and Vera Rubin — now folding in Groq's inference technology as its seventh chip — is designed to extend that lead into the inference era specifically. But nearly every major AI company that once solely depended on Nvidia is now also building or renting its own inference silicon: Amazon, Google, Microsoft, Meta, and now OpenAI itself, alongside independent challengers like AMD, Cerebras, and Qualcomm. None of this displaces Nvidia in training anytime soon — but the inference layer, where the actual day-to-day cost of AI lives, is genuinely becoming a multi-vendor market for the first time.
This is one of the fastest-moving corners of the tech industry — specs, deal terms, and shipping timelines described here reflect public announcements as of mid-2026 and are subject to change as these companies ship real hardware.
No comments:
Post a Comment