Trillion-Parameter Multimodal, 4B Agents Match 671B, PPO Exposed

Featured

01 Multimodal One Training Run, a Family of Models

"Unified multimodal" has been a buzzword for years. But a trillion-parameter model that trains text, image, video, and audio from scratch under a single autoregressive objective — with a public technical report — is a first.

ERNIE 5.0 makes two design choices worth watching. Ultra-sparse MoE with modality-agnostic expert routing: regardless of whether the input is text or video frames, the same routing logic applies, letting the model learn which experts handle what. Then there's the elastic training paradigm: during a single pre-training run, the model simultaneously learns sub-models of varying depths, expert capacities, and sparsity levels. At deployment time, you pick the configuration that fits your hardware.

This is fundamentally an engineering answer to: you've trained a trillion-parameter model, now how do teams with different compute budgets actually use it? The report includes detailed expert routing visualizations and elastic training analysis.

Key takeaways: - Modality-agnostic routing avoids the complexity of separate pathways per modality — the key design decision for unified multimodal - Elastic training turns "one model" into "a model family," lowering the deployment barrier for large models - First public technical report of its kind at trillion-parameter scale

Source: ERNIE 5.0 Technical Report

02 Agent Model Too Small? Send More of Them

Deep learning's dominant narrative has been depth scaling — making a single model deeper and stronger. But when a task is inherently "wide" — say, gathering information across multiple dimensions — the bottleneck isn't individual capability. It's organizational efficiency.

WideSeek-R1 proposes a complementary axis: width scaling. A lead agent decomposes the task, multiple sub-agents execute information retrieval in parallel, and the entire system is jointly optimized through multi-agent reinforcement learning (MARL). The result: WideSeek-R1-4B hits 40.0% item F1 on the WideSearch benchmark, matching standalone DeepSeek-R1-671B — a 167x parameter gap.

Performance keeps improving as you add more parallel sub-agents, suggesting width scaling has its own scaling law.

Key takeaways: - Width scaling naturally complements depth scaling, especially for inherently parallelizable tasks like information retrieval - 4B x N matching 671B x 1 validates "small model swarms" as a viable cost-reduction path - MARL joint training is the critical ingredient — not just stitching agents together, but training them to collaborate

Source: WideSeek-R1: Exploring Width Scaling for Broad Information Seeking via Multi-Agent Reinforcement Learning

03 Training PPO's Trust Region Is Wrong for LLMs

PPO is the de facto standard for LLM reinforcement learning, but this paper identifies a structural flaw: ratio clipping is pathological in large-vocabulary settings. The probability ratio is a per-token sample estimate, not a true policy divergence measure. Low-probability tokens have wildly volatile ratios and get over-clipped — they can't learn. High-probability tokens barely move the ratio and evade clipping entirely, risking catastrophic drift.

DPPO takes the direct approach: drop the heuristic clipping and estimate actual policy divergence (Total Variation or KL). Binary and Top-K approximations keep memory overhead manageable. Experiments show improved stability and efficiency over existing methods.

If you're running RLHF or GRPO pipelines, this is a PPO alternative worth investigating.

Key takeaways: - Ratio clipping under large vocabularies systematically misallocates constraint strength — this explains common instabilities in LLM RL training - DPPO's divergence constraint is theoretically cleaner and remains engineering-feasible through approximation - A credible alternative to PPO for RLHF and GRPO pipelines

Source: Rethinking the Trust Region in LLM Reinforcement Learning

04 Training 90% of Your PRM Data Is Wasted

Training multimodal process reward models (PRMs) typically requires large-scale Monte Carlo annotation — expensive to produce. But this paper delivers good news: most of that data is redundant. Random subsampling saturates quickly, meaning the useful information is concentrated in a small fraction of samples.

What makes a data point informative? Two factors: the mix ratio of positive and negative labels, and label reliability (average MC score of positive samples). Based on these signals, the authors propose BIS (Balanced-Information Score) filtering.

Validated on InternVL2.5-8B and Qwen2.5-VL-7B backbones, BIS matches or exceeds full-data performance using just 10% of the training set.

Key takeaways: - Smart filtering beats brute-force data scaling for PRM training - BIS requires no extra annotation — it works directly from existing MC signals - For teams training visual reasoning PRMs, this means an order-of-magnitude reduction in training cost

Source: Training Data Efficiency in Multimodal Process Reward Models