Agents Face Long Horizons and Better Training Data

Featured

01 Training | Complete Tasks Can Still Fail

Synthetic agent tasks need instructions, an initial environment, a reference solution, and a verifier. Having every component is not enough. Conflicting assumptions can make tasks unsolvable or cause verifiers to reject valid solutions.

FACET preserves goals, dependencies, state transitions, and operational constraints from the source. It builds and repairs the runtime before generating the final instructions, solution, and verifier. When execution tests find a failure, FACET repairs only the faulty component.

The paper reports that successful trajectories provide data-efficient supervision. Fine-tuning improves Terminal-Bench 2.1 performance across model sizes. Tests with alternative generation methods also support the importance of executable environments and aligned verifiers. Adaptive curricula would require further design and experiments.

Key takeaways:

• Check whether instructions, environments, solutions, and verifiers share the same execution state.
• Finish implementing and repairing the environment before generating final task components.
• Use execution tests and targeted repairs to preserve components that already work.

Source: FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis

02 Evaluation | Decisions Compound Over 20 Seasons

FM-Bench puts agents in charge of football clubs for 20 seasons. Across roughly 340 to 400 decisions, consequences accumulate each year. Rivals also respond to trades and resource allocation.

A deterministic engine calculates the final score without LLM judges or human grading. The test is not whether a model can call 26 tools. It measures how well agents balance cash, renewals, facilities, and squad planning over time.

All 15 frontier models completed the full cycle, but rankings stabilized only late. The arena champion rotated among ten models. Top performers reduced slow-return investments near the end, avoided idle cash, and renewed contracts early. No model inferred hidden market prices from hundreds of failed bids. Agent-managed memory either became an ever-growing archive or a plan rewritten from scratch each year.

Key takeaways:

• Long-horizon evaluations should let early decisions continue shaping later states.
• Competitive environments expose strategic consistency better than static tasks.
• Memory systems need refinement and updating, not endless accumulation.

Source: FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents

03 Code Intelligence | Let Agents Study the Repository First

General coding agents often struggle with project conventions and hidden repository knowledge. Programming ability alone does not close that gap.

SkillForge moves repository learning earlier. Before real issues arrive, it recreates core functions covered by existing tests to generate synthetic issues. The agent then extracts reusable skills from repair traces and links them to relevant code entities.

The paper reports consistent gains across open and closed models. It does not provide exact improvements or preparation costs in the abstract. The economics depend on cross-defect transfer. If every new issue requires fresh exploration, SkillForge only moves the cost into preparation.

Key takeaways:

• Repository adaptation can shift from reactive learning to advance skill distillation.
• Synthetic issues offer a training path when historical repair data is scarce.
• Measure skill transfer, update costs, and reductions in per-issue exploration.

Source: SkillForge: Self-Distilling Agents for Project-Specific Issue Resolution

04 AI for Science | Retrosynthesis Needs Multiple Correct Answers

Retrosynthesis naturally has multiple valid answers. One target molecule can support several workable routes. Training against a single reference mislabels reasonable, unrecorded predictions as errors.

This work applies Top-K generation throughout training and inference. The model learns to produce several candidates instead of merely sampling more at deployment. C3LM trains on about 45.6 million verified reactions. ChemCensor constraints and a novelty reward expand candidate coverage while filtering implausible chemistry.

The paper reports leading results on the out-of-distribution URSA-expert-2026 benchmark. Language models and traditional systems also explore complementary reaction spaces, supporting combined approaches. More candidates do not automatically mean more useful routes. Check chemical validity, Top-K accuracy, and genuine candidate diversity before adoption.

Key takeaways:

• Do not treat one reference route as the complete ground truth.
• Top-K works best as a training objective, not just a larger sampling budget.
• Evaluate chemical plausibility and candidate diversity together.

Source: Training Chemical Plausibility-Aware Large Language Models for Single-Step Retrosynthesis