Meta Prometheus 超级集群超11%的GPU闲置待命
Tens of thousands of Nvidia GPUs sit idle in Meta's Prometheus supercluster, just waiting for a failure
The Register
以下为原文逐段中文翻译，保留图表与说明。AI 翻译并经 AI 对照复核，仍可能存在错误；核对数据和引用时请同时查看原文。译文发布：2026年10月21日 07:50（北京时间）。
（广告）
（系统）
以及其他有趣的事实：Zuckercorp（扎克伯格的公司）如何设法让一百万块 GPU 持续运转足够久，以训练出一个模型。
扎克伯格的公司（Zuckercorp）本周披露，Meta 位于俄亥俄州新奥尔巴尼（New Albany）的 Prometheus 超级集群中，超过 11% 的 GPU 要么已经损坏，要么处于闲置状态，等待在下一块 GPU 掉线时顶替上阵。
这一庞大的冗余 GPU 储备，只是这家超大规模云服务商（hyperscaler）为使其 AI 训练任务不脱轨而采取的若干措施之一。
Prometheus 是 Meta 迄今规模最大的计算部署之一。Meta 表示，该超级集群由 Nvidia 的 Grace Blackwell 和（即将推出的）Vera Rubin 机架系统驱动，最终将扩展至 1 吉瓦的算力规模，覆盖约一百万个加速器。
（广告）
但任何在大规模计算集群上工作过的人都明白，无论是用于大语言模型（LLM）训练还是传统高性能计算（HPC）工作负载，芯片越多，问题必然越多。
（广告）
在一个 8 块 GPU 的节点中，出现 GPU 故障的概率相对较低。但在一个 10 万块 GPU 的集群中，故障在所难免。Meta 正在构建的规模还要大 10 倍，并且需要跨多个数据中心进行分布式训练。让整个系统持续平稳运转足够久、直到抵达下一个检查点（checkpoint），成了一大难题。
在本周发布的一篇博客文章中，Meta 罕见地披露了其为防止故障打乱 Muse Spark 等模型、以及其最新发布的 Muse Wonder 模型的开发所做的工作。
闲置 GPU 是持续运转的代价
Meta 表示，当其在新奥尔巴尼启动 Prometheus 部署的第一阶段时，每 8,000 块 GPU 每天大约会发生 1,000 次中断。
中断并不意味着 GPU 故障。它们只是意味着某个环节崩溃、失败或超时，迫使训练任务停止并回滚到最后一个已知的良好检查点（checkpoint）。
Meta 称，自那以来的一年间，它已能将中断次数减少 99%，降至每 8,000 块 GPU 每天约 11 次。
其实现方式之一，是将 Prometheus 的部分 GPU 保留为储备。该集群目前由 GB200 和 GB300 系统组合驱动，每个 NVLink 域包含 72 块 GPU。然而，任意时刻实际在用的只有 64 块，其余 8 块本质上是热备件。
Meta 这样做，是因为单块 GPU 故障就会拖垮整个机架并中断训练任务。如今，当某块 GPU 故障时，一块备件可以插入并顶替其位置。虽然这节省了时间——Meta 不必将整个机架从集群中剔除，也不必等人更换故障部件才能重新开始训练——但 GPU 故障仍会导致中断，尽管时间更短。
（广告）
并行弹性
为缓解这一问题，Meta 利用了这样一个事实：大规模训练任务通常使用数据并行（data parallelism），即模型的多个副本驻留在集群中。
“当故障可以被隔离到某个数据并行模型副本时，我们允许单个故障副本下线，从而降低硬件故障的影响，而集群其余部分重复该步骤并继续有效训练，”Meta 解释道，“受损的分片只需在可用的备用主机上重启并重新同步，从而将一个潜在的系统全面停摆，转化为整体吞吐量一次轻微、暂时的下降。”
换言之，只有当故障超出数据并行域时，Meta 的训练任务才会被中断。一旦发生这种情况，该公司便实施了一套全面的检查点（checkpointing）机制，你可以将其大致想象为自动保存或修订历史。这样，即使真有故障发生，Meta 也无需从头重新开始。
“我们的分布式检查点（checkpointing）架构会异步地将模型状态——包括梯度和优化器数据——保存到多级存储中，”该公司写道。
NVLink 的成长阵痛
Meta 表示，最棘手的挑战之一，是让用于连接 72 块 GPU 的 NVLink 互连 fabric 表现得像一个巨大的加速器。早期的 H100 和 H200 系统每个 NVLink 域仅有 8 块 GPU，这意味着链路更短、数量也更少。但在 Nvidia 的 Blackwell 代际中，这一能力扩展到了原来的九倍。
“在超过 60 万条 NVLink 连接下，即便只是微小的信号衰减，也会触发频繁的 NVLink 恢复周期，”该公司写道。
（广告）
为克服这一问题，Meta 不得不部署一套持续的遥测系统，并在系统投产前开展 L11 诊断老化测试（diagnostic burns），以隔离薄弱环节。
供电是另一个挑战。为一次训练任务对数万块 GPU 进行升温和降温，会引入被称为瞬态（transients）的巨大功耗尖峰，可能压垮供电设备。
据 Meta 称，通过专注于其集群的稳定性和可靠性，它在预训练上实现了超过 85% 的“有效吞吐”（goodput，即有用吞吐量的衡量指标），在其最新模型 Muse Wonder 的强化学习上则超过 90%。
该公司的雄心不止于此。在向一百万块 GPU 迈进的过程中，Meta 希望将其中断率降至每 8,000 块 GPU 每天仅两次，或者说平均故障间隔时间约为 10 万 GPU·小时。®
展开英文原文 · 与中文逐段对应
Tens of thousands of Nvidia GPUs sit idle in Meta's Prometheus supercluster, just waiting for a failure
REG AD
systems
And other fun facts about how Zuckercorp aims to keep a million GPUs running long enough to train a model
More than 11 percent of the GPUs at Meta’s Prometheus supercluster in New Albany, Ohio, are either dead or sitting idle waiting to step in when the next drops out, Zuckercorp revealed this week.
The massive pool of redundant GPUs is just one of several measures the hyperscaler has taken to keep its AI training runs on track.
Prometheus is among Meta's largest compute deployments to date. Powered by Nvidia's Grace Blackwell and (soon) Vera Rubin rack systems, Meta says the supercluster will eventually scale to a gigawatt of capacity spanning about a million accelerators.
REG AD
But as anyone who has worked on a large scale compute cluster understands, whether for LLM training or traditional HPC workloads, more chips inevitably means more problems.
REG AD
In an eight-GPU node, the chance of having a GPU fail is relatively low. In a 100,000-GPU cluster, failures are unavoidable. Meta is building something that's 10x larger and requires distributed training runs across multiple datacenters. Keeping the entire thing humming long enough to reach the next checkpoint becomes a major headache.
In a blog post published this week, Meta offered a unique look at the work it's done to keep failures from derailing the development of models like Muse Spark or its most recent release Muse Wonder.
Idle GPUs are the cost of continuity
When Meta began the first phase of its Prometheus deployment in New Albany, the company says, it was experiencing roughly 1,000 interruptions per day for every 8,000 GPUs.
Interruptions don't mean GPU failure. They simply mean something crashed, failed, or timed out, forcing the training run to stop and roll back to the last known good checkpoint.
Meta claims in the year since it's been able to cut down the number of interruptions by 99 percent, to about 11 per day per 8,000 GPUs.
One of the ways it's done this is by keeping some of Prometheus' GPUs in reserve. The cluster is currently powered by a combination of GB200 and GB300 systems, each with 72 GPUs per NVLink domain. However, only 64 of them are ever in use at any given moment. The other eight are essentially hot spares.
Meta did this because a single failed GPU would take out an entire rack and interrupt the training run. Now, when a GPU fails, one of the spares can slot in and take its place. While this saves time — Meta doesn't have to drop the entire rack from the cluster or wait for someone to swap the faulty component to start training again — a failed GPU would still result in an interruption, albeit a shorter one.
REG AD
Parallel resiliency
To mitigate this, Meta took advantage of the fact that large scale training jobs usually use data parallelism, where multiple copies of the model are resident within the cluster.
"When a fault can be isolated to a data-parallel model replica, we reduce the impact of a hardware failure by allowing a single faulty replica to be taken offline, while the remaining cluster repeats that step and continues productive training," Meta explained. "The compromised slice simply restarts on available spare hosts and resynchronizes, converting a potential total system halt into a minor, temporary reduction in aggregate throughput."
In other words, Meta's training runs only get interrupted if the fault exceeds the data parallel domain. When this happens, the company has implemented a comprehensive checkpointing scheme, which you can think of a bit like auto-save or a revision history. That way if something does fail, Meta doesn't have to start over from scratch.
"Our distributed checkpointing architecture asynchronously saves the model state — including gradients and optimizer data — to multi-tier storage," the company wrote.
NVLink growing pains
Meta says one of the most difficult challenges was making the NVLink fabrics used to connect 72 GPUs behave like one great big accelerator. Earlier H100 and H200-based systems featured just eight GPUs per NVLink domain, which meant the links were far shorter and less numerous. But this functionality was expanded nine-fold with Nvidia's Blackwell generation.
"With over 600,000 NVLink connections, even marginal signal degradation triggers frequent NVLink recovery cycles," the company wrote.
REG AD
To overcome this, Meta had to implement a continuous telemetry system and conduct L11 diagnostic burns to isolate weak links before the systems were put into production.
Power was another challenge. Ramping up and down tens of thousands of GPUs for a training run can introduce massive spikes in power consumption called transients, which can overwhelm power supply equipment.
According to Meta, by focusing on the stability and reliability of its cluster, it was able to achieve more than 85 percent "goodput" — a measure of useful throughput — for pretraining and more than 90 percent for reinforcement learning on its latest model Muse Wonder.
The company's ambitions don't stop here. As it works its way toward a million GPUs, Meta hopes to get its interruption rate down to just two per day for every 8,000 GPUs, or about 100,000 GPU-hours mean time between interrupts. ®
查看本期独立摘要
中文独立摘要 · AI 整理
这件事
Meta披露其Prometheus超算集群中逾11%的GPU处于损坏或空置待命状态，一旦有芯片掉线即顶上。该集群最终将扩展至约百万加速器、容量达一吉瓦。为保障连续性，每72-GPU的NVLink域仅用64个、留8个作热备；Meta还将中断率从每天千次降至约11次，预训练有效吞吐超85%。
本页概括所列来源的重点，属于独立摘要；完整报道请阅读原站。整理时间：2026年10月21日 07:43（北京时间）。
信息从哪里来
据The Register报道，引述Meta技术博客。
- The Register 报道正文