DatacentersX > Stack > Cooling & Thermal Management

Cooling & Thermal Management

Cooling is the constraint that determines how much compute a data center can physically host. Every watt of electrical power delivered to a server must leave the building as heat. The engineering discipline of cooling and thermal management designs the end-to-end path by which heat moves from silicon die through package, board, chassis, rack, row, and mechanical plant, and finally rejects that heat to atmosphere or to a reuse stream.

The dominant driver reshaping this discipline is accelerator power density. Through the 2010s, a typical hyperscale rack drew 8 to 15 kilowatts and air handling was sufficient. The arrival of dense accelerator systems raised rack power into the 30 to 50 kilowatt range, at which point air cooling becomes uneconomic and increasingly infeasible. Current AI training racks exceed 120 kilowatts, and next-generation Rubin-class and post-Rubin reference designs anticipate 250 to 600 kilowatts per rack. At those densities liquid is no longer an option, it is a requirement.

The content below organizes cooling by heat-transport mechanism, from air at the low-density end through liquid loops in the middle to dielectric immersion at the high-density frontier, followed by the facility-side systems that ultimately reject heat from the building and manage the water that makes the loops work.

The thermal chain

Before surveying cooling modalities, it helps to trace the physical path heat takes from generation to rejection. A compute workload dissipates energy at the transistor level. That heat conducts through the die, through the integrated heat spreader, through thermal interface material, into a primary heat extractor (a heat sink for air cooling, a cold plate for direct-to-chip, or the dielectric fluid itself for immersion). The primary extractor transfers heat to a transport fluid, usually air or water. That transport fluid flows through the chassis and rack, then into a facility-level loop.

The facility loop carries heat to a heat exchanger, where it passes to a second loop (typically a chilled-water or condenser-water loop) that routes to the mechanical plant. At the plant, heat is rejected to atmosphere through a cooling tower, dry cooler, or hybrid system, or in an increasing minority of sites, to a thermal reuse pathway such as district heating.

Each link in this chain has a temperature approach (the delta between the source and the sink at that stage) and an associated energy cost. The sum of approaches and losses sets the minimum supply temperature at the chip, and the efficiency of the chain sets PUE.

Modalities by rack density

Modality	Typical Density Range	Heat Transport Fluid	Primary Use Today
HVAC and Air Handling	Up to ~30 kW per rack	Air	Enterprise, colocation general-purpose, ancillary loads in AI facilities
Liquid Cooling	30 to 100+ kW per rack	Water or water-glycol, closed loop	AI training clusters, HPC, dense hyperscale
Direct-to-Chip Cooling	50 to 250+ kW per rack	Water through cold plates on CPU and accelerator packages	Mainstream approach for current-generation AI accelerators
Immersion Cooling	Effectively unbounded at rack scale	Dielectric fluid, single-phase or two-phase	Crypto at scale, select HPC deployments, emerging AI pilots
Cooling Tower and Heat Rejection	Facility-level, all densities	Water to air, air to air, or hybrid	Terminal heat sink for every facility that does not reuse heat
UPW and Cooling Water Systems	Facility-level, all densities	Makeup, treatment, chemistry, blowdown for every water-based loop	Foundational water infrastructure supporting liquid, DTC, and tower systems

HVAC and air handling

Air-based cooling uses CRAC and CRAH units, in-row coolers, containment systems, and raised-floor or overhead supply to move conditioned air through server intakes and extract hot exhaust. The fundamental limit of air cooling is heat capacity: air carries roughly one three-thousandth the heat of water by volume, which forces large volumetric flow rates to remove high thermal loads. Beyond about 30 kilowatts per rack, the fan power required and the floor space given over to air-handling equipment make air cooling uneconomic.

Air has not disappeared. Enterprise data centers, the cloud and SaaS floors of most colocation sites, the lower-density racks in hyperscale facilities, and ancillary equipment inside AI facilities (networking gear, storage, management servers) remain predominantly air-cooled. Air handling is the default mechanical system that liquid technologies supplement rather than fully replace.

Detailed HVAC and air handling coverage

Liquid cooling

Liquid cooling is the category that subsumes all closed-loop water-based thermal extraction at the row and rack level. The defining component is the Coolant Distribution Unit, a facility-to-rack heat exchanger that isolates the primary facility water loop from the technology cooling loop that carries water directly into server hardware. CDUs come in in-row, in-rack, and sidecar form factors, with capacities ranging from tens of kilowatts to multiple megawatts per unit.

Adjacent to the CDU sit manifolds, quick-disconnect fittings, leak detection, and the rear-door heat exchangers or row-level heat exchangers that handle racks still running air-cooled components. Liquid cooling is the umbrella that contains both direct-to-chip and, by some taxonomies, rear-door air-to-liquid approaches.

Detailed liquid cooling coverage

Direct-to-chip cooling

Direct-to-chip, sometimes called direct liquid cooling, places a water-cooled cold plate in direct thermal contact with the integrated heat spreader of each CPU and accelerator package. Water flows through microchannels machined into the cold plate, picks up heat from the die, and returns to the CDU. Because water has vastly higher heat capacity than air and because the thermal path from silicon to coolant is short, DTC handles accelerator TDPs well beyond what any air solution can manage.

Direct-to-chip is the mainstream approach for current-generation AI accelerators. The NVIDIA GB200 NVL72 reference design is direct-to-chip. AMD MI300X and MI325X deployments are direct-to-chip. Intel Gaudi deployments at hyperscale are direct-to-chip. The entire frontier training stack has standardized on this approach as the equilibrium between thermal capacity, serviceability, and integration complexity.

A residual question with DTC is what to do about the remaining 15 to 25 percent of rack heat that cold plates do not capture (memory, power delivery, network interfaces, drives). The two dominant answers are rear-door heat exchangers that handle the air-cooled portion and hybrid designs that cold-plate additional components.

Detailed direct-to-chip cooling coverage

Immersion cooling

Immersion cooling submerges server hardware in a dielectric fluid. In single-phase immersion, the fluid remains liquid and circulates between tank and external heat exchanger by pump. In two-phase immersion, the fluid boils at the die surface (working fluids are engineered to boil in the 50 to 60 degree C range at atmospheric pressure), rises as vapor, condenses on a cold coil, and falls back as liquid, with no pump required in the immersion bath itself.

Immersion offers the highest theoretical thermal capacity of any mainstream approach and handles essentially unbounded rack density because the entire board is in contact with the cooling fluid. The tradeoffs are fluid cost, material compatibility with existing server components, serviceability (wet hardware, fluid management), and in the two-phase case regulatory attention to the perfluorinated compounds historically used as working fluids.

Immersion is mature in cryptocurrency mining, where the economics justify the capex and the hardware fleet is homogeneous and purpose-built. In AI and HPC it remains at pilot and select deployment scale. The open question is whether immersion displaces direct-to-chip at some future density threshold, or whether DTC variants (cold plate on every power-dissipating component) absorb the density envelope first.

Detailed immersion cooling coverage

Cooling tower and heat rejection

Every cooling modality above transfers heat into a facility water or refrigerant loop. That loop terminates at the mechanical plant, where heat is finally rejected to atmosphere (or, in a small but growing minority of sites, diverted to reuse). The three dominant terminal rejection technologies are open-circuit cooling towers, closed-circuit fluid coolers, and dry coolers, with hybrid systems combining evaporative and dry stages.

Open cooling towers reject heat through evaporation, achieving the lowest approach temperatures but consuming water continuously through evaporation, drift, and blowdown. Dry coolers reject heat purely through air-side convection, consuming no water but operating at higher approach temperatures that limit achievable chilled-water supply and raise compressor load. Hybrid systems run dry in cooler ambient conditions and switch to wet operation in peak heat.

Site selection, local water rights, and ambient climate drive the choice. Facilities in water-stressed regions increasingly specify dry or hybrid rejection even at the cost of higher PUE. Facilities in cooler climates pursue outside-air economization and free cooling to minimize mechanical refrigeration hours.

Detailed cooling tower and heat rejection coverage

UPW and cooling water systems

The water loops that make liquid and direct-to-chip cooling possible are themselves engineered systems requiring ongoing chemistry management, filtration, and makeup. The technology cooling loop feeding cold plates runs ultrapure water or water-glycol at tightly controlled conductivity, pH, and particulate levels, because scaling or biofouling in microchannel cold plates would raise thermal resistance and eventually block flow.

The facility water loop running between CDUs and cooling towers tolerates higher mineral content but still requires biocide, scale inhibitor, and corrosion inhibitor dosing, along with blowdown cycles to prevent concentration buildup. Makeup water enters from municipal supply, well, or recycled source and passes through softening, reverse osmosis, or deionization depending on loop requirements.

Water usage effectiveness, WUE, measures liters of water consumed per kilowatt-hour of IT load. AI data centers at scale consume tens of millions of gallons per year and face growing scrutiny from local jurisdictions over withdrawal permits and wastewater discharge. The design space for water systems is widening: dry-cooled facilities sacrifice PUE for zero water consumption, while evaporatively cooled facilities accept water consumption as the price of thermodynamic efficiency.

Detailed UPW and cooling water systems coverage

Engineering tradeoffs that cut across modalities

Several axes of tradeoff appear in every cooling decision regardless of modality:

Supply temperature and mechanical load. Raising the water supply temperature to the rack reduces or eliminates the need for mechanical chilling, which collapses compressor energy and PUE. Direct-to-chip systems can accept supply water at 30 to 45 degrees C, enabling free cooling for most of the year in most climates. Air-cooled systems with 18 to 24 degrees C supply air cannot.

Water consumption versus energy efficiency. Evaporative heat rejection is thermodynamically efficient but consumes water. Dry rejection consumes no water but raises compressor load. The choice is site-specific and increasingly regulatory.

Density versus serviceability. Higher-density modalities (DTC, immersion) deliver more compute per square foot but complicate field service. Hot-aisle access in air-cooled rows is trivial; cold-plate replacement on a live GB200 tray is not. Operations models have to evolve in parallel with thermal capacity.

Capital versus operating cost. Liquid infrastructure costs more to install than air but consumes dramatically less energy for the same IT load. The crossover depends on density, utilization, and local energy cost, but at AI training densities liquid wins on total cost of ownership within two to three years.

Related coverage