Production Traffic, Video Pretraining, and Smarter Distillation

Featured

01 Training: Retire Old Models Before Chasing New Ones

Self-hosted LLM fleets often suffer from one-way expansion. New base models arrive, but older models remain tied to specific workloads. Limited GPU capacity fragments into ever-smaller service pools.

This team started with failures from over 200 internal applications. It grouped production requests by task and separately evaluated instruction following, function calling, and internal enterprise work. Deterministic verifiers and calibrated model judges tracked quality instead of a single leaderboard detached from real traffic.

The team trained separate GRPO experts for three capability axes, then merged them through two-stage spherical linear interpolation. Each expert exposed different problems, including semantic collapse, excessive tool use, and verbosity-based reward hacking.

The merged model scored 69.6 without reasoning mode, beating an internal baseline at 65.8 despite having about one-seventh the total parameters. Instruction following rose from 0.83 to 0.85, while function calling increased from 0.77 to 0.79.

Those gains were enough to move 50% of platform traffic onto one model, totaling 116 million monthly requests. The bigger payoff is fleet consolidation: retiring old services and pooling their GPUs again. Evaluation bias and merge stability across future iterations still need confirmation from the full paper.

Key takeaways:

• Segment evaluation by production traffic before deciding which capabilities to train.
• Small gains in instruction following or function calling can determine whether workloads migrate and old models retire.
• Check for cross-task reward interference, semantic collapse, excessive tool use, and verbosity hacking.

Source: From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix

02 Robotics: Action Labels Aren't the Only Bottleneck

Robots may not need vastly more expensive action trajectories to generalize across environments. ZimaBlue first learns visual change from over 120,000 hours of human and robot egocentric video. The footage covers contact, tool use, and long-horizon behavior.

A unified action representation then connects this unlabeled experience to control data from different robots. Against training only on target-robot data, zero-shot success on real robots rose from 36.1% to 77.8%. Unseen tasks gained the most.

ZimaBlue also uses an asynchronous Slow-Fast system for real-time control. Its lightweight branch predicts actions at 30 Hz on an RTX 4090. Action-free video cannot directly teach movement, but it can supply transferable physical experience that makes scarce action labels more valuable.

Key takeaways:

• Expand egocentric video before spending the entire data budget on action collection.
• Action-free pretraining primarily improves cross-environment generalization; it does not replace control data.
• Evaluate both unseen-task performance and real-time control speed.

Source: ZimaBlue: Scalable Video Pre-Training for Generalizable World Action Models

03 Multimodal: Pixel Text Needs Four Ingredients

Four choices drive pixel-based text generalization: resolution and font scale, real image-text pairs, layout-aware rendering, and multilingual training. Controlled ablations isolate each variable instead of hiding them behind one model score.

Adaptive resolution helps with high-resolution documents. Real images ground the representation in visual content. Preserving layout prevents pixel shortcuts, while two-stage training improves alignment across languages.

Pixel Linguist II applies this recipe to 280 million samples. It reports new best results across English, cross-lingual, and multilingual visual-semantic tasks. Performance also stays steady after compressing visual tokens by 80%.

Document intelligence teams can turn these four variables into a training checklist. Before adopting the compression claim, verify task-level quality loss and the encoder's added compute cost.

Key takeaways:

• Cover resolution, real imagery, layout, and multilingual data together; fixing one alone will not solve generalization.
• Real image-text data determines whether the model understands visual text or memorizes rendering patterns.
• An 80% reduction in visual tokens is promising, but validate quality loss and encoding cost for each task.

Source: On the Design Fundamentals of Pixel Text Representation Learning

04 Training: Distillation Can Trade Recall for Reasoning

Forward-KL distillation transfers different abilities depending on when it begins. During pretraining, reasoning and factual memory improve together. During mid-training, reasoning keeps advancing while factual acquisition slows.

The authors attribute this split to uneven teacher confidence. Teachers guide procedural reasoning well, while students already learn high-confidence facts early. Switch Distillation uses predictive entropy to decide whether the teacher is confident enough to supervise each target.

Confident predictions trigger distillation. Other cases fall back to standard cross-entropy training. Against ordinary next-token prediction, the method reaches 1.61–1.71× the reasoning performance while preserving 96.7%–96.8% of factual memory. The gains persist after post-training.

Distillation timing and scope act as controls over what the student inherits. They can favor problem-solving procedures or stored knowledge.

Key takeaways:

• Do not assume pretraining-stage distillation gains will carry into mid-training.
• Apply mid-training distillation selectively based on teacher confidence, rather than across every token.
• Track reasoning, knowledge, and factual recall separately; aggregate scores can hide capability splits.

Source: Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

05 Agent: GUI Products Need Three Infrastructure Layers

UI-Venus-2 brings phones, browsers, and desktops into one closed loop. It expands three areas together: environment coverage, task construction, and reward verification.

A deep-research workflow generates tasks based on real product functions. Visual keypoints, multi-model voting, and trajectory-level evaluation make reinforcement-learning feedback more trustworthy.

The system also adds safety controls for consequential actions. Its reliability still needs validation through the full report and real deployments. GUI agent teams should copy the infrastructure strategy, not just compare another model score.

Key takeaways:

• Evaluate GUI agents by cross-device coverage and closed-loop stability, not isolated success rates.
• Task generation and reward verification are becoming barriers equal to model capability.
• Product evaluations must test safety boundaries around high-risk actions.

Source: UI-Venus-2 Technical Report