AI Training

What it is

Thousands of Chips Working as One

Training builds an AI model by running data through it trillions of times and adjusting its weights. Every GPU in the job computes, then pauses to exchange results with every other GPU, then computes again, in lockstep.

• Power swings: because the chips move together, the whole building's load rises and falls together. Meta reported swings of tens of megawatts during its Llama 3 405B run.
• Utilization: that run used 38% to 43% of the GPUs' peak math throughput; the rest was lost to communication, memory and waiting.
• Training mix: after pre-training on text, models are refined with reinforcement learning. Epoch AI estimates reinforcement learning took 10% to 200% of pre-training compute for GPT-5, up from 1% to 10% in early 2025.

Grid

Grid Operators Now Regulate Training Loads

• Trips: ERCOT recorded 26 trips of large data center loads from 2023 to 2025; in one, load at about 60 points across 25 to 30 substations dropped at once and did not come back right away.
• NERC: its May 2026 Level 3 alert, the most serious kind, followed a 1,500 MW data center load drop during a 230 kV line fault. It proposes registering every computational load of 20 MW or more.
• Texas rule: since August 1, 2026, large computational loads in ERCOT must ride through voltage dips and frequency swings and return to 90% of demand within two seconds.

See Scaling and Battery Storage.

Flagship

Colossus 1: Batteries Behind a Training Cluster

• Batteries: SpaceXSI's Colossus 1 in Memphis installed 168 Tesla Megapacks, about 655 MWh, which the company says handle outages and demand surges from training.
• Next door: Colossus 2 reached about 1 GW in 2026 and has trained seven models at once, up to 10 trillion parameters, on about 555,000 Blackwell GPUs. See SpaceX Colossus.
• Chip-level fix: Nvidia's GB300 racks add capacitors, ramp limits and a "power burner" that holds draw steady when GPUs go idle, cutting peak grid demand by 30% on a training job, by Nvidia's measurement.

Failures

At a Million GPUs, Something Breaks Every Few Minutes

• Interruptions: Meta's 16,384-GPU Llama 3 run had 419 unexpected interruptions in 54 days, 59% of them GPU faults, and still kept more than 90% effective training time.
• Scale: Epoch AI estimates one failure every 3 hours at 16,000 GPUs, every 30 minutes at 100,000 and every 3 minutes at 1 million.
• Checkpoints: jobs save their full state every few minutes so they can restart. Epoch puts a Llama 405B checkpoint at about 4.9 TB; Google cut save time from 30 minutes to under 5 with tiered storage. See Storage and Archival.

Facts

Training Runs and Load Events

Related coverage

Related Coverage

• Frontier Training Runs
• AI Factories
• AI Inference
• Scaling