Observability and AIOps

Telemetry, straggler detection and automatic recovery · Flagship: ByteDance MegaScale

In a synchronous training job, one slow or failed machine stalls thousands of GPUs, so detection speed is money. ByteDance's MegaScale fixes more than 90% of faults automatically on 10,000+ GPUs, and Alibaba's Unicron detects errors in seconds instead of a 30-minute timeout.

How fast a failed training job is detected, on a log scale: a 30-minute default timeout, under 10 minutes with ByteDance MegaScale's monitoring, and 0.3 to 5.6 seconds with Alibaba Unicron; and ByteDance's finding that stragglers, about 0.5% of machines, set the speed of the whole job TIME TO DETECT A FAILED NODE (LOG SCALE) 1 s 1 min 30 min Default timeoutNCCL library default ByteDance MegaScaleHeartbeats, RDMA sampling Alibaba UnicronIn-band error detection ONE SLOW MACHINE SLOWS THE WHOLE JOB (BYTEDANCE) Stragglers were about 0.5% of machines but set the pace of 10,000+ GPUs. Removing hosts about 10% slow raised throughput about 0.7%. 90%+ of faults found and fixed automatically; restart within 15 minutes.
Faults auto-fixed
90%+ (ByteDance)
Stragglers
~0.5% of machines
Unicron detection
0.3-5.6 s
AIOps market, 2032
~$8.7B (fcst)
Hub
Operations

What it is

Seeing Every GPU, Every Second

Observability collects metrics, logs and traces from every GPU, switch and server; AIOps uses software, increasingly AI agents, to find and fix problems from that data.

  • Telemetry: Nvidia's DCGM reports each GPU's utilization, power, temperature, memory errors and link events, typically every second, into Prometheus and Grafana.
  • Stragglers: in a synchronous job, one slow machine sets the pace for all. ByteDance found about 0.5% of machines were stragglers.
  • Automatic recovery: Nvidia's Mission Control and in-house systems at Meta, ByteDance and Alibaba detect failures, swap nodes and restart jobs from checkpoints without people.
  • Market: the agentic AIOps market is estimated at about $780 million in 2025, rising to about $8.7 billion by 2032 (MarketsandMarkets). Datadog launched GPU monitoring in April 2026.

Flagship

ByteDance MegaScale: Training That Repairs Itself

  • Scale: a training system for more than 10,000 GPUs, described by ByteDance at NSDI 2024.
  • Monitoring: heartbeats from every node, sampled network traffic each step and a GPU event monitor that draws heat maps of slow machines.
  • Results: more than 100 automatic recoveries over several weeks, with more than 90% of faults found and fixed automatically, diagnosis in under 10 minutes and the job back to its prior progress within 15.
  • Efficiency: 55.2% of peak math throughput on 12,288 GPUs, 1.34 times Nvidia's Megatron-LM. See Compute Operations.

Tools

Observability Tools and Systems

GPU observability and AIOps tools
Tool or systemWhat it doesStatus
ByteDance MegaScale (China)100+ automatic recoveries on 10,000+ GPUs; 90%+ of faults fixed automaticallyMeasured, 2024
Nvidia Mission ControlHealth checks, leak monitoring, automatic hardware and job recovery; Prometheus and LokiShipping
Nvidia DCGM and dcgm-exporterPer-GPU utilization, power, temperature, memory errors and link events, sampled every secondShipping
Prometheus, Grafana, LokiOpen-source metrics, dashboards and logsOpen source
Datadog GPU MonitoringTies GPU telemetry to jobs; finds idle GPUs; GPUs are 14% of customers' compute spendApril 2026
CoreWeave health checksPassive and active checks, fabric-error correlation; SemiAnalysis's benchmark for neocloudsReported
Meta Fleetscanner and RippleFleet-wide scans for silent data corruption every 45-60 days and in the backgroundOperating
Alibaba Unicron (China)Error detection in 0.3-5.6 seconds; up to 1.9x training efficiencyMeasured, 2024

Record

Dataset Record

Dataset
GPU observability, straggler detection and automatic recovery
Sources
ByteDance (arXiv 2402.15627), Alibaba (arXiv 2401.00134), Nvidia documentation, Datadog, SemiAnalysis, Meta engineering, MarketsandMarkets
Last verified
October 2026