What it is
Seeing Every GPU, Every Second
Observability collects metrics, logs and traces from every GPU, switch and server; AIOps uses software, increasingly AI agents, to find and fix problems from that data.
- Telemetry: Nvidia's DCGM reports each GPU's utilization, power, temperature, memory errors and link events, typically every second, into Prometheus and Grafana.
- Stragglers: in a synchronous job, one slow machine sets the pace for all. ByteDance found about 0.5% of machines were stragglers.
- Automatic recovery: Nvidia's Mission Control and in-house systems at Meta, ByteDance and Alibaba detect failures, swap nodes and restart jobs from checkpoints without people.
- Market: the agentic AIOps market is estimated at about $780 million in 2025, rising to about $8.7 billion by 2032 (MarketsandMarkets). Datadog launched GPU monitoring in April 2026.
Flagship
ByteDance MegaScale: Training That Repairs Itself
- Scale: a training system for more than 10,000 GPUs, described by ByteDance at NSDI 2024.
- Monitoring: heartbeats from every node, sampled network traffic each step and a GPU event monitor that draws heat maps of slow machines.
- Results: more than 100 automatic recoveries over several weeks, with more than 90% of faults found and fixed automatically, diagnosis in under 10 minutes and the job back to its prior progress within 15.
- Efficiency: 55.2% of peak math throughput on 12,288 GPUs, 1.34 times Nvidia's Megatron-LM. See Compute Operations.
Tools
Observability Tools and Systems
| Tool or system | What it does | Status |
|---|---|---|
| ByteDance MegaScale (China) | 100+ automatic recoveries on 10,000+ GPUs; 90%+ of faults fixed automatically | Measured, 2024 |
| Nvidia Mission Control | Health checks, leak monitoring, automatic hardware and job recovery; Prometheus and Loki | Shipping |
| Nvidia DCGM and dcgm-exporter | Per-GPU utilization, power, temperature, memory errors and link events, sampled every second | Shipping |
| Prometheus, Grafana, Loki | Open-source metrics, dashboards and logs | Open source |
| Datadog GPU Monitoring | Ties GPU telemetry to jobs; finds idle GPUs; GPUs are 14% of customers' compute spend | April 2026 |
| CoreWeave health checks | Passive and active checks, fabric-error correlation; SemiAnalysis's benchmark for neoclouds | Reported |
| Meta Fleetscanner and Ripple | Fleet-wide scans for silent data corruption every 45-60 days and in the background | Operating |
| Alibaba Unicron (China) | Error detection in 0.3-5.6 seconds; up to 1.9x training efficiency | Measured, 2024 |
Record
Dataset Record
- Dataset
- GPU observability, straggler detection and automatic recovery
- Sources
- ByteDance (arXiv 2402.15627), Alibaba (arXiv 2401.00134), Nvidia documentation, Datadog, SemiAnalysis, Meta engineering, MarketsandMarkets
- Last verified
- October 2026