AI Inference

What it is

Running Trained Models for Users

Inference is running a trained model to answer a request: a chat reply, a search result, a line of code, a car's next steering command. It is now most of the AI compute in use.

• Share: Deloitte estimates inference was about a third of AI compute in 2023 and half in 2025, and forecasts about two-thirds in 2026.
• Volume: Google processed 3.2 quadrillion tokens in May 2026, up from 9.7 trillion two years earlier; ByteDance's Doubao handles more than 120 trillion tokens a day.
• Price and demand: the price of a given level of model quality falls between 9 and 900 times a year, depending on the benchmark (Epoch AI), but reasoning models and agents use far more tokens per task. Agents produced about 71% of tokens on OpenRouter by August 2026.

Where it runs

Most Inference Runs in Big Data Centers

• Hyperscale and AI factories: nearly all inference in 2026 runs in large data centers, often on the same clusters that train models. See Hyperscaler Data Centers.
• Company sites: Deloitte puts on-premises enterprise AI at about $50 billion in 2026. See Enterprise Data Centers.
• Edge: Akamai is putting Nvidia GPUs across 4,400 edge locations for low-latency inference. See Edge Data Centers.
• Devices: cars, robots and phones run models on their own chips, a market Deloitte puts under $5 billion. See Vehicles and Devices.

Latency

Four Kinds of Inference

• Interactive chat and voice: the first word must arrive in under about a second. Groq, Cerebras and Google's Ironwood TPU are pitched here.
• Agents and coding: many steps and huge token counts per task, the fastest-growing class.
• Batch: offline jobs run for the lowest cost per token.
• Ranking and recommendation: feeds and ads served in milliseconds, the work of Meta's MTIA chips.

Flagship

Cerebras in Oklahoma City: A Site Built for Inference

• Site: more than 300 Cerebras CS-3 systems, opened in mid-2025, one of a planned network with 85% of capacity in the US.
• Contract: in January 2026 OpenAI contracted up to 750 MW of Cerebras low-latency inference capacity through 2028, a deal Reuters valued above $10 billion.
• Chip: Cerebras builds each processor from a whole silicon wafer, keeping the model in on-chip memory to answer faster than GPUs.

Chips

Custom Chips Are Taking Inference From Nvidia

• Custom silicon: Google's TPUs, Amazon's Trainium, Meta's MTIA and Microsoft's Maia handle a growing share of their owners' inference; Anthropic has access to up to 1 million Google TPUs.
• Nvidia: licensed Groq's low-latency chip design for about $20 billion in December 2025. The first US-made Blackwell wafer came out of TSMC Arizona in October 2025. No US fab is confirmed for TPU, Trainium or MTIA chips.
• China: Huawei plans about 600,000 Ascend 910C chips in 2026, made at SMIC, and holds about half of China's AI chip market (Bernstein); Alibaba has shipped 470,000 of its own chips.

Platforms

Inference Platforms and Fleets

Related coverage

Related Coverage

• AI Training
• Vehicles and Devices
• Chip Layer
• AI Factories