Agents Score 58.1%, but Only 2.8% Pass Every Test

Featured

01 Agents Can Code but Still Cannot Deliver

Real software work loops through interface exploration, implementation, debugging, and visual inspection. RecreationWorld puts that entire loop inside one task. An agent receives a running reference app but no prescribed workflow. It must discover the app's behavior, rebuild it, and inspect the interface.

The reference app also serves as an answer engine. Hidden tests can score actual runtime behavior, not just code or screenshots. Unified native-control and coding tools support Ubuntu, macOS, Windows, Android, and the web.

Training on these trajectories improved performance across five out-of-distribution coding and hybrid-operation benchmarks. Agents also inspected rendered output more often. Yet RecreationBench exposes the remaining gap across 250 tasks. GPT-6 Astra scored 58.1% overall, but only 2.8% of tasks passed every programmatic test.

Key takeaways:

• Hybrid agents should switch autonomously between interface control, coding, and validation.
• Running reference apps can reward both visual similarity and correct behavior.
• Agents reproduce static interfaces better than complex interactions and computation.

Source: RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

02 Code Tasks Without Issues or Commits

For coding agents, code is plentiful. Tasks with reliable answers and automatic grading are not. CodeMidas treats existing functionality as the answer, then asks agents to infer specifications by exploring its behavior.

It runs the original implementation to generate tests and filters unreliable samples through execution checks and repeated solving attempts. This process produced 5,545 training tasks from 3,185 open-source repositories across 23 languages.

Reinforcement learning on these tasks improved all five evaluated benchmarks. Whole-program construction rose 17%, code repair improved 11.7%, and terminal tasks gained 8.5%. Teams with large codebases could turn internal functionality into a scalable source of verifiable training tasks.

Key takeaways:

• Existing functionality can generate training tasks without issues or commits.
• Execution results from original code can ground tests and verifiers.
• Large code repositories can become scalable sources of reinforcement learning tasks.

Source: CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

03 Longer Reasoning Does Not Guarantee Better Translation

Adding a reasoning trace to machine translation does not reliably improve quality. Results depend on the model, the reasoning language, and the task domain. Reasoning length also has a non-monotonic relationship with translation quality.

A short trace may be insufficient, but extending it can stop helping or make results worse. Using hierarchical meta-summarization, the authors identify common stages: comprehension and planning, translation and drafting, then refinement and verification. Patterns also differ across domains.

Long reasoning should not be a global default for translation products. Test each model, language, and domain combination. Enable reasoning only when the quality gain justifies its added latency and compute.

Key takeaways:

• Reasoning length is not a quality control that teams should simply maximize.
• Test each model, language, and domain combination separately.
• Enable reasoning only when its gains exceed the added latency and compute costs.

Source: When Does Reasoning Help in Machine Translation? A Hierarchical Analysis of LRM Reasoning Traces

04 Does “I Don’t Know” Mean the Model Forgot?

Machine unlearning evaluations can miss protected facts or dangerous ideas that appear only in intermediate reasoning. Checking the final answer is not enough.

GUARD turns information that would have leaked into a coherent safe-exit process. It generates non-leaking reasoning first, then produces a consistent refusal. The method distills that behavior into model parameters. Its NFRS metric also checks structural stability, fluency, and unsupported replacements for forgotten facts.

The abstract reports lower leakage across privacy and harmful-intent tests on two distilled reasoning models, while preserving reasoning ability. Safer output proves stronger behavioral suppression, not true removal. Parameter-level forgetting still requires attack tests and mechanistic evidence.

Key takeaways:

• Audit both intermediate reasoning and final answers when evaluating unlearning.
• Safe refusals also need checks for repetition and fabricated replacements.
• GUARD improves reasoning-level safety, but does not prove complete knowledge removal.

Source: GUARD: Natural Forgetting in Large Reasoning Models via Guided Answer-Reasoning Distillation

05 Longer Skills Can Make Agents Worse

Many skill-optimization methods keep polishing natural-language instructions. Wording is not the only problem. Branching tasks with step dependencies make linear instructions long and ambiguous about what comes next.

GraphSkillEvo rewrites skills as flowcharts. Nodes encode actions, while edges specify context-dependent transitions. The workflow becomes something teams can inspect and revise. It then mutates and combines several candidates to preserve useful steps and broaden the search.

Across five agent benchmarks, GraphSkillEvo beat SkillOpt by 4.01% on GPT-5.4-nano and 1.76% on GPT-5.4. The smaller model gained more. Per-task results are still needed before claiming graph structure consistently closes capability gaps.

Key takeaways:

• Represent dependencies explicitly in branching tasks instead of extending prompts.
• Graphs make redundant steps, missing actions, and incorrect transitions easier to find.
• Smaller models may benefit more from explicit workflows, but gains need task-level validation.

Source: GraphSkillEvo: Evolutionary Optimization of Graph-Structured Agent Skills