Training Bigger Models Without Paying the Full Price

Featured

01 Fine-Tune Agents Without Backpropagation

Agentic ESOpt fits full-parameter fine-tuning of a 27B model into inference-grade memory. It perturbs model parameters, runs complete tasks, then weights parameter updates by each trajectory's reward. This avoids both step-level credit assignment and the memory cost of gradients and optimizer states.

The same black-box feedback can co-optimize parameters, prompts, skills, and test-time compute. Full-parameter optimization raises Qwen-3.5-27B performance by 6.69% over the no-skill baseline on WebArena-Lite. Automatically designed test-time heuristics beat their baselines in 28 of 36 settings.

Inference-grade memory does not mean low total compute. Every parameter perturbation may require a full trajectory, making sampling expensive. Teams should check sample counts, wall-clock time, and compute-matched gains before adding gradient-free parameter search to their training stack.

Key takeaways:

• Whole-trajectory rewards can evaluate parameter perturbations when step-level credit assignment breaks down.
• Inference-grade memory makes full-parameter optimization more feasible, but says little about total cost.
• Judge the method by sampling volume, wall-clock time, and task gains under equal compute.

Source: Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements

02 Spend Limited Budgets on Data

Image-generation teams now have a more specific planning rule than “bigger is better.” Abra runs controlled flow-matching Transformer experiments across compute budgets from (10^19) to (10^22) FLOPs. Training loss, generation quality, and optimal guidance settings all follow predictable curves.

The compute-optimal point sits near 200 image tokens per parameter, ten times the Chinchilla ratio for language models. Diffusion models also tolerate overtraining well. Under a fixed budget, more data may therefore beat a blindly larger model.

These curves could help estimate model size, data volume, and training time before an expensive run. The findings come from one controlled model family. Teams still need to test whether they transfer across architectures, data quality levels, and high-resolution tasks.

Key takeaways:

• Validate data volume before defaulting to a larger parameter count.
• Small controlled experiments can project training curves and reduce costly configuration mistakes.
• Treat 200 image tokens per parameter as a starting point, not a universal recipe.

Source: Abra: Scaling Diffusion Image Training

03 Scale Vision Without Activating Everything

Vision encoders can gain parameter capacity without activating every parameter for every input. MoE-ViE adds sparse experts to a CLIP-style encoder. Fine-grained experts, auxiliary-loss-free load balancing, and dedicated kernels aim to prevent uneven expert usage and higher latency.

The paper also uses frame-level distillation and freezing to add video capability while preserving image knowledge. Its abstract does not describe the full deployment path.

Sparse scaling may improve the capacity-to-compute curve on the vision side. Real gains still depend on throughput, routing stability, kernels, and hardware. Those factors need full-paper review and deployment tests.

Key takeaways:

• Evaluate vision MoE systems by total capacity, active parameters, and end-to-end latency.
• Distillation may preserve image capability while extending an encoder to video.
• Dedicated kernels may determine whether sparse experts deliver real deployment gains.

Source: MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

04 Benchmark Scores Do Not Guarantee Delivery

Most agent benchmarks begin with researchers deciding which capabilities matter. StartupBench instead derives tasks from adopted AI products and their workflows. It evaluates professional scenarios with end-to-end deliverables, then applies detailed scoring criteria to capture complex requirements.

Complex instruction following and domain knowledge emerge as the main barriers. Under one testing framework, even the strongest model completes only about 30% of tasks. Many failures show substantial partial progress, but doing most steps is not the same as delivering reliably.

Agent teams should track completion rates on tasks derived from real demand, not only isolated benchmark scores. They should also avoid treating market provenance as proof of commercial success or production readiness. The full paper must confirm the distribution of failure causes.

Key takeaways:

• Tasks derived from existing user demand test capability transfer better than researcher-selected exercises.
• Separate partial progress from complete delivery; the roughly 30% completion rate exposes a large gap.
• Add end-to-end delivery tests, but do not treat market origin as proof of production reliability.

Source: StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows