Welcome to the eighth issue of Agents in Practice!
📬 Agents in Practice is a weekly newsletter on agentic AI research and applications. Subscribe here →
ICML 2026 Agent Reproductions Challenge
With the improvement in agents, running experiments and writing a paper has gotten so much easier than before. However, that means the number of paper submissions to conferences has increased too. ICML 2025 reported 12,107 submissions, and that number roughly doubled to 23,918 this year. This has put a significantly increased burden on reviewers.
To look at the paper review process at a different angle, Hugging Face ran a hackathon where participants attempted to reproduce the main results of the paper. They indexed all 6,341 accepted papers from ICML 2026 and extracted the main claim for each one. The participants could then use their coding agents to try to attempt to reproduce the paper. Each attempt was judged by an automated GLM 5.2 judge that decided whether a claim was verified, falsified, toy, or inconclusive.
Out of 6,341 papers, 2,226 papers were attempted. Hugging Face reports that 51% (1,103) of examined papers had at least one claim verified, but also that 23% of examined papers had at least one claim falsified or contested.
Personal Thoughts
I participated in the ML Reproducibility Challenge years ago and presented my work at NeurIPS. Reproducing a paper was a great learning experience, helping me learn more about the paper’s claims and understand the reasoning behind the implementation. Maybe coding agents can also “learn” from reproducing the papers?
Read more
SkillTV-Bench
With agents’ traces getting longer and longer, evaluating the agent just by its final output has become a noisy, lossy signal. To understand if the agent efficiently solved the task, or to verify that the agent did not break any of the task requirements, agent traces must also be verified. Agent traces do not have a single “golden truth” that we can evaluate against, so LLM judges are a common way to evaluate the traces. The authors introduce SkillTV-Bench, a benchmark for LLM judges that focuses on evaluating judges for skill-augmented agents. In the benchmark, judges can “inspect the skill utilization and orchestration in the agent execution.” With the benchmark, the authors also propose that signals from SkillTV-Bench can be used to improve judge behavior. SkillTV-Evolve analyzes false accepts and rejects and converts recurring failures into judge skill revisions.
Personal Thoughts
Even with the best observability UI, agent traces are becoming difficult to read and comprehend due to their sheer length. It is even worse when the reader is not a domain expert in the task the agent is given, which is often the case as agents grow in power and capability. Good judge agents could minimize expensive human reviews and evaluations, although they will not fully replace human judgment.
Read more
One-liners
- Meta’s new coding agent Muse Code has a two-tier pricing model, with the “contributor tier” that lets Meta use session data for training being massively discounted.
- Benchmarks specific to languages and frameworks are continuing to appear. Rails Foundation shared the Agents on Rails initiative. Kotlin now has its own Kotlin Benchmark based off of Multi-SWE-bench. Next.js has an AI Agent Evaluations page that tests success rates with AGENTS.md bundled with Next.js documentation.
- Boundary-Bench finds that coding agents’ performance degrades when security controls are enforced.
Subscribe to Agents in Practice
Get new issues of Agents in Practice — a newsletter summarizing exciting new research and applications in agentic AI — delivered to your inbox.