Today's Overview
- NVIDIA Crams Language, Image, Video, Audio, and Action Into One Set of Weights. Cosmos 3 bets a single mixture-of-transformers can do every modality, and third parties rated it best open model in text-to-image, image-to-video, and robot policy.
- The Same KV Quantization Looks Fine in Prefill and Falls Apart in Long Decoding. KVarN shows the error compounds across timesteps, uses variance normalization to tame outlier token-scales, and takes 2-bit KV quantization to a new SOTA — calibration-free, with a vLLM implementation.
- Writing What You Learned In Context Back Into the Weights. "Language models need sleep" drops the metaphor: the mechanism is distillation plus self-rehearsal on synthetic data. But the abstract dodges the two hard questions — what to write back, and how to avoid forgetting.
- Sampling Budget Goes From Hand-Tuned Threshold to Learned Policy. Framing "how many samples to draw" as an MDP, an RL-trained controller small enough to run on CPU beats strong baselines on the "fewer samples, no accuracy drop" tradeoff.
Featured
01 NVIDIA Bets on One Model for Every Modality
Physical AI has run on two tracks. Either you assemble specialized models — a vision-language model for understanding, a video generator for simulation, a world-action model for output — or you train one unified backbone that handles every modality. Cosmos 3 commits to the second. A single mixture-of-transformers (a multi-expert Transformer variant) packs language, image, video, audio, and action into the same weights, and the abstract says outright that it aims to subsume those separate systems.
The tradeoff is visible from the abstract. Action sequences get treated as the same kind of generable sequence as video and images. That makes sense for embodied agents: perception and action share one world representation and can, in theory, transfer to each other. Unification has a cost the abstract leaves unexplored — which tasks genuinely benefit from the shared representation, and which just got forced into one frame. That split needs the full paper's ablations to judge.
The third-party endorsements are worth recording. Artificial Analysis rated the post-trained version best open text-to-image and image-to-video model; RoboArena rated it best policy model. At minimum, "do everything" didn't visibly lose any single category — the exact place unified architectures usually break. Code, weights, and the synthetic dataset ship under the OpenMDW open license, giving teams chasing Physical AI a base they can build on directly.
Key takeaways: - Physical AI's architecture is splitting into two camps; Cosmos 3 is the "one model, all modalities" side, and that's the bet to weigh against assembling specialists. - Third parties rated it best open model in text-to-image, image-to-video, and robot policy — so unification didn't obviously compromise any single track this time. - The abstract gives conclusions, not tradeoffs; to see which tasks the shared representation actually helps, you need the full ablations.
Source: Cosmos 3: Omnimodal World Models for Physical AI
02 KV Quantization That Passes Prefill and Fails Decoding
Test-time scaling — spending more compute at inference for better answers — is a settled win. The cost is that long-horizon decoding grows the KV-cache, and memory becomes the new bottleneck. KV quantization is the obvious fix, but almost every existing method evaluates in a prefill-style setup: compress one known input once, with static error.
KVarN names the real problem. In autoregressive decoding, quantization error compounds across timesteps, each step feeding its mistake into the next, and the root cause is mis-estimated scale on a few individual tokens. The method runs one Hadamard rotation, then a variance-normalized bidirectional scaling along both axes of the K and V matrices, targeting exactly those outlier token-scale errors. That cuts the accumulation sharply.
On generative benchmarks — MATH500, AIME24, HumanEval — 2-bit precision sets a new SOTA for KV quantization. It needs no calibration and ships with a vLLM implementation.
Key takeaways: - The eval setup can hide the real problem: KV quantization that passes in prefill may compound error in long decoding, so re-test against a real decode scenario before deploying. - The root cause is mis-estimated token-scale, not general precision loss — targeting it beats blanket precision cuts. - Teams running long reasoning or agent deployments and stuck on KV-cache memory should try it: 2-bit, calibration-free, vLLM-ready, low cost to adopt.
Source: KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks
03 Writing In-Context Knowledge Back Into the Weights
Close the conversation window and the temporary knowledge is gone. However well a model learns in context, it can't fix that into long-term parameters and keep accumulating. That's a real problem.
Strip away the human-memory-consolidation metaphor and the mechanism is two existing ideas bolted together. First, Knowledge Seeding distills a "smaller self" into a larger network to trade for capacity — on-policy distillation plus RL imitation learning. Second, RL generates a synthetic-data curriculum for self-rehearsal.
The abstract avoids the two hardest points. How does it decide which in-context knowledge is worth writing back to the weights, and what keeps old capabilities from collapsing after the edit? The abstract mentions replay, but this is a proof of concept with no comparison numbers. Whether it solves a step beyond existing knowledge-editing and memory-adaptation methods is not something the abstract can settle.
Key takeaways: - The real pain in continual learning is fixing in-context knowledge into parameters; the direction is worth tracking, but track the mechanism, not the metaphor. - The core is distillation plus self-rehearsal on synthetic data — confirming the novelty needs the full paper and comparison experiments. - The abstract doesn't answer "what to write" or "how to avoid forgetting," so teams in memory or continual learning should wait for full evaluations.
Source: Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
04 Should a Learned Controller Own Your Sampling Budget?
Deciding when to stop sampling used to mean hand-set thresholds or assumptions about the answer distribution. Both are brittle; swap the model or task and you retune. This paper frames "how many samples to draw" as a Markov decision process — modeling each "stop or continue" round as a stateful decision — and trains a lightweight RL controller that jointly trades off accuracy, latency, and compute.
The controller reads only statistics of the final answer, so it trains and deploys on CPU. Against strong baselines like ASC and ESC, it gets a better tradeoff on "fewer samples without losing accuracy." The open question is transfer: whether the same small controller works after switching models or task domains needs the full paper to confirm.
Key takeaways: - Turning the sampling budget from a hand-tuned threshold into a learned policy is a real way to cut test-time scaling costs. - The controller depends only on answer statistics and trains and deploys on CPU, so integration cost is low. - Verify cross-model and cross-task transfer before shipping; retraining per scenario would erode the value.
Source: Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling
Also Worth Noting
Today's Observation
Three papers landed on the same dial today: whether to use test-time scaling is no longer the question — that it's too expensive is. KVarN quantizes the KV-cache and Value-Aware eviction evicts it, cutting from the memory side two different ways: one compresses the cache to 2-bit, the other throws out the unimportant KV. The RL adaptive-sampling paper cuts from the compute side, teaching the model to draw fewer samples. In a single day, the cost of long reasoning got cut from two completely different layers — memory and sampling — by groups that didn't know the others were cutting. That's the tell that test-time scaling has passed the "prove it works" stage and entered the "get it cheap enough to ship" stage. Note how scattered the incisions are: no unified framework coordinates them; memory, eviction, and sampling each go it alone. This cost-cutting is still in early, multi-point probing, and no one has landed the decisive cut.
For action: if you already run long reasoning or reasoning models in production, stop evaluating KV quantization, cache eviction, and sampling budget as three separate topics. Measure your own cost structure first — how much is memory, how much is sampling — then decide which of today's three cuts to make first, instead of chasing whichever paper has the prettiest benchmark.