NuAxiSemiconductor trade intelligence

Tens of thousands of Nvidia GPUs sit idle in Meta's Prometheus supercluster, just waiting for a failure

What happened

Meta revealed this week that more than 11 percent of the GPUs at its Prometheus supercluster in New Albany, Ohio, are either dead or sitting idle, waiting to step in when a GPU fails. Prometheus is among Meta's largest compute deployments to date, powered by Nvidia's Grace Blackwell and (soon) Vera Rubin rack systems, and Meta says the supercluster will eventually scale to a gigawatt of capacity spanning about a million accelerators. The cluster is currently powered by a combination of GB200 and GB300 systems, each with 72 GPUs per NVLink domain, but only 64 are ever in use at any given moment, with the other eight essentially hot spares; Meta did this because a single failed GPU would take out an entire rack and interrupt the training run. Meta says that when it began the first phase of the Prometheus deployment it was experiencing roughly 1,000 interruptions per day for every 8,000 GPUs, and in the year since it has cut interruptions by 99 percent, to about 11 per day per 8,000 GPUs. Meta also says it achieved more than 85 percent goodput for pretraining and more than 90 percent for reinforcement learning on its latest model Muse Wonder, and as it works toward a million GPUs it hopes to reduce the interruption rate to just two per day for every 8,000 GPUs, or about 100,000 GPU-hours mean time between interrupts. The source does not state any implication for GPU market supply or demand.