Trillion-Parameter MoE Post-Training Runs on Huawei Ascend

Featured

01 Post-Training a Trillion-Parameter MoE Off the GPU

Someone ran full-parameter post-training of a trillion-parameter mixture-of-experts (MoE) model end-to-end on non-NVIDIA hardware: a Huawei Ascend NPU cluster. This kind of training is a systems problem on its own. Memory won't fit, communication won't overlap with compute, and the kernels can't saturate the chip. Mature toolchains assume you're on NVIDIA GPUs with CUDA.

SLAI T-Rex answers with a layered optimization playbook, from parallelism strategy through compute-communication scheduling down to the low-level kernels. It reaches 34.22% model FLOPs utilization (MFU), a 2.93x gain over the open-source baseline recipe. The absolute number isn't striking. Landing it on non-GPU hardware while keeping training stable is what's worth recording.

The report's back half reads as corroboration. Its team built an operations-research data pipeline on DeepSeek-V4-Flash, using solver-verified synthetic optimization documents for continued pretraining and fine-tuning. The resulting specialist hits 71.81% zero-shot Pass@1, nearly four points above GPT-5.4-Mini. Its role is to prove the non-CUDA path can ship a usable model end-to-end. The headline stays the systems engineering.

For teams worried about NVIDIA lock-in, or forced to build a training stack on constrained compute, the playbook beats any single metric. Read the 34% and 2.93x not as a performance record but as an ecosystem signal: frontier-scale training off the GPU is moving from "it runs" to "it runs in production." This is a technical report. Its payload is engineering practice, not a new capability.

Key takeaways: - The real deliverable is a non-CUDA playbook, not another trillion-parameter model: how to fit memory, hide communication, and patch kernels when post-training a trillion MoE on Ascend. Archive it if you run constrained compute or want to avoid NVIDIA lock-in. - Read the 34% MFU and 2.93x gain as an ecosystem signal — frontier training off the GPU going from "runs" to "production-ready" — not as a performance record. - The operations-research specialist (71.81%, past GPT-5.4-Mini) is proof the whole path works end-to-end, not the main point.

Source: SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD

02 Turn the KV Cache Into Trainable Memory

The current mainline for autoregressive video is Self Forcing: keep training the model on its own generated history instead of real video. That narrows the gap between seeing ground truth in training and generating from its own output at inference. It leaves one loose end. Once early frames are written into the KV cache, they're frozen as read-only, and the training signal from later frames can't teach the earlier ones how that memory should have been written.

SGF closes that gap with two training passes. The first runs the model inference-style and records intermediate states. A second pass reconnects gradients in parallel, letting the loss from later frames optimize how early content gets encoded into the cache. It deliberately avoids backpropagating through the whole serial generation, which is nearly untrainable. The point isn't a new architecture. It's promoting the read-only cache to a first-class trainable signal.

A 5-second training window extrapolates to minutes. Subject consistency, background layout, and temporal stability all hold better than Self Forcing — and those are exactly where long video breaks first.

Key takeaways: - The real increment is turning the KV cache into a trainable "memory write" path, not a new architecture. In principle it stacks onto an existing Self Forcing pipeline. - Extrapolating from a 5-second window to minutes saves long-video training cost, which is more concrete than the quality numbers alone. - The dimensions that drift first, subject consistency and temporal stability, improve — but the only long-range results so far are the authors' own. Verify on your own data once the model ships.

Source: Self Gradient Forcing: Native Long Video Extrapolation

03 nDCG Ranks Documents, RAG Runs on the Set

Retrieval evaluation has a long-standing blind spot: nDCG scores each document on its own and sums the weights, never asking how documents relate to each other. When the reader of retrieval results shifts from a human to an LLM or agent, what caps generation quality is the whole set the model receives. Redundancy, conflict, complementarity — nDCG sees none of it.

The paper builds a set-level benchmark, SetwiseEvalKit: three tiers, nine dimensions, about 28,000 rubrics. Tested on 12 mainstream rerankers, the best set-level coverage still lands under 45%, and cross-document coordination is broadly weak. The companion method, Rubric4Setwise, is training-free. It converts the rubric scoring criteria directly into a set-selection signal, reaching better downstream generation with fewer documents and retrieval rounds.

For teams building RAG, the call is blunt: your retriever's nDCG can look great while the set you hand the model is bad.

Key takeaways: - nDCG only measures single-document relevance; the ceiling is set quality. A good retriever score doesn't mean a good set reaches the model. - Twelve mainstream rerankers top out below 45% set-level coverage, with cross-document coordination the shared weak spot. Don't pick a reranker on single-document ranking metrics alone. - Rubric4Setwise is training-free with an open implementation, converting eval rubrics straight into a selection signal. Worth trying if you build RAG retrieval.

Source: Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking

04 A Scaling Law for Baking Facts Into Weights

Injecting facts into a model now has a predictable scaling law. The paper trains a hypernetwork (a network that generates another network's weights). It produces a fixed LoRA adapter that injects a batch of facts into a target model. Because injection capacity is separated from the model's general ability, the authors can cleanly measure how injection scales.

The finding: injection quality moves smoothly along a power law on three axes — hypernetwork depth, hypernetwork width, and target-model size. On out-of-distribution QA, where the phrasing wasn't seen directly in training, its scaling exponent is steeper than both LoRA and full fine-tuning.

For practitioners, the useful part isn't any single score. It's the curve. That curve lets you estimate ahead of time, on your own data and model size, whether baking knowledge into weights pays off. These are abstract-level conclusions; the curve's exact shape and extrapolation limits still need the full paper.

Key takeaways: - A hypernetwork generating a fixed LoRA decouples fact injection from general ability. That's both what sets it apart from direct fine-tuning and what makes scaling measurable. - All three scale axes follow a predictable power law, so small-scale experiments can extrapolate whether large-scale investment is worth it. Less blind trial. - On OOD generalization it scales steeper than LoRA and full fine-tuning — the bigger you go, the more it wins. Whether it pays off still needs a test on your own corpus.

Source: Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models