A 3B-Active MoE Matches Last Year's 32B

Featured

01 A 3B-Active MoE Matches Last Year's 32B

Tencent released Hy-Embodied-VLM-1.0, an embodied foundation model for the physical world, pitched on efficiency. The headline number: an MoE architecture activates only 3B parameters yet approaches a previous model that activated 32B. It gains 8.4% on average over the same-generation Hy-Embodied-0.5 and takes best-in-class at its size on 19 of 38 embodied benchmarks, beating Qwen3.6-A3B and Cosmos 3.

The method matters more than the scores. Tencent splits capability into a three-tier, action-oriented taxonomy — state understanding, action-transition reasoning, sequential and adaptive reasoning — and builds its pre-training and post-training data pipeline around that structure. For practitioners, the real question is what the "latency-sensitive deployment" section actually reports. The abstract offers benchmark results and a qualitative claim of "high inference efficiency," with no measured latency, throughput, or on-device numbers.

Efficient embodied agents tend to leave a gap between a good demo and something that ships. Fewer active parameters does not mean lower end-to-end latency — the data pipeline, the Hy-ViT2 vision encoder, and scheduling all eat into the budget. To judge whether it deploys, read the full report and see which trade-offs it spells out.

Key takeaways: - MoE matching last year's 32B with 3B active is a real signal for compute-constrained embodied deployment, but confirm the small activation actually converts to low end-to-end latency. - The abstract gives benchmark scores and no measured latency or throughput. Judging whether it ships means reading the deployment section in full. - Teams working on robotics or embodied agents should look at the action-oriented capability taxonomy and data pipeline. The methodology is more reusable than the leaderboard rank.

Source: Hy-Embodied-VLM-1.0: Efficient Physical-World Agents

02 Attention Compression Learns Where to Spend Its Codebook

The O(N²) cost of attention has always been the transformer bottleneck. Vector-quantized attention approximates all keys with M codewords, dropping the cost to O(MN) — but it hands the whole key space one uniform codebook budget. Attention quality is far from uniform. The regions that actually draw attention get a coarse approximation, while regions nobody looks at hold the same expressive capacity.

AVQ allocates by importance instead. It starts from a small set of codewords, identifies the most critical codes during the forward pass, then refines them with pre-trained sub-codewords. Fine quantization happens only where it should. A custom Triton kernel folds the whole adaptive refinement into Flash Attention's block computation, holding O(MN) with little overhead.

Whether that overhead is truly negligible is the question the full paper's measurements have to answer. The engineering completeness of this step decides whether it ships.

Key takeaways: - Attention compression is past "can we cut complexity" and into "where should the codebook budget go" — spatial adaptivity. - Uniform codebook allocation is waste. Importance-driven allocation is the more sensible resource model. - The complexity formula is not the point; whether the Triton kernel makes refinement overhead negligible is. Teams optimizing long-sequence inference should check its measurements.

Source: AVQ-Attention: Adaptive Vector-Quantized Attention

03 SAM2 Fits on a Phone — But at What Accuracy?

Video foundation models like SAM2 can segment anything, but phones and laptops can't run them. Shrinking to the edge is nearly unavoidable. MobileSAM2 uses a distillation method called HyperKD: it builds the teacher's temporal and multi-granularity knowledge into a hypergraph — a structure that captures higher-order relations — then aligns that to a lightweight student. The idea is nice.

The number that decides everything is missing. The abstract claims "promising generalization" but never states how much accuracy it loses or how much speed it buys. Teams working on edge perception or embodied AI should note the method, but adoption waits on the quantitative comparison in the full paper — especially on video, where temporal tasks are exactly where lightweight models break. ECCV acceptance says the work is solid, and solid is not the same as shippable.

Key takeaways: - Foundation models moving to the edge is a clear trend, and SAM2 has joined the queue. - HyperKD builds knowledge into a hypergraph before distilling — a fresh idea, but the abstract gives no accuracy-for-speed ratio. - Teams in edge perception or embodied AI should watch it, but usability waits on the full paper's numbers.

Source: MobileSAM2: Lightweight Segment Anything for Spatial Intelligence

04 Your Accuracy Number Might Be Bent by Answer Length

Multiple-choice benchmarks lean on a move that looks neutral: rank each candidate by its conditional log-prob — the sum of per-token log probabilities — and pick the highest. But log-prob accumulates per token, so more tokens means a heavier penalty, and long answers lose by default. The standard fix is length normalization, and this ICML paper shows with experiments that it often overcorrects. After normalization the bias flips toward long answers — same bias, opposite direction.

The authors propose Bayesian accuracy, which adds an explicit prior over answer length and computes a posterior that cancels the linear length effect. It's drop-in and needs no extra forward passes. Across several benchmarks and few-shot settings, they measure lower length bias than either prior method. How much this helps depends on how spread out answer lengths are in your task — the narrower the spread, the smaller the effect.

Key takeaways: - If you run evals that rank multiple-choice answers by log-prob, the accuracy number may reflect answer length rather than model ability. - Length normalization is not a cure-all. It can flip the bias from favoring short answers to favoring long ones. - Bayesian accuracy is a zero-cost alternative worth trying on evals where answer lengths vary widely.

Source: Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative