Welcome to the fourth issue of Agents in Practice!
Coding agents plan far ahead in latent space
Transformer language models are known to develop internal representations of their tasks in latent space. The authors of this paper investigate the internal representations of coding agents working on SWE-Bench-Verified and SWE-Bench-Pro and show that those representations contain useful signals throughout the trajectory. These signals from the model’s residual stream can be decoded with a linear probe to extract useful information such as whether the current program parses, passes its test suite, reduces the number of failing tests, or introduces regressions. The authors also show that probes trained to predict the outcome of a future edit succeed up to roughly 25 steps in advance.
Personal Thoughts
The authors report AUC up to 0.83 for correctness-related properties across two models and two benchmarks. This is still a probing result, not a ready routing mechanism, but if it could be generalized to other agent tasks and improved in performance, it could be useful in deciding when to route to a bigger agent or to a human.
Read more
Test-writing benchmark from open-source repositories
TestEvo-Bench is a new benchmark that focuses on coding agents’ ability to generate and update tests after a codebase has undergone changes. The authors set up a three-phase pipeline for mining new tasks from open-source repositories, allowing the benchmark to stay live and support contamination-aware evaluation. 1. Recent adjacent commits from Java and Maven repositories are mined, forming candidate commit pairs that build cleanly and pass tests. 2. The candidate pairs are reviewed so that old tests pass on old code and new or updated tests pass on new code, while old tests fail on new code for update tasks. 3. The candidate pairs are again filtered so that typo updates or coverage updates of existing tests are removed, as well as cases where new tests pass on old code.
The current snapshot contains 746 test-generation tasks and 509 test-update tasks from 59,950 candidate co-evolution records across 152 open-source Java projects. The authors report that the strongest evaluated agents reach up to 77.5% success on test generation and 74.6% on test update, with lower performance on the most recent tasks and under tighter per-task cost limits.
Personal Thoughts
Good tests can help agents find their own mistakes and automatically fix bugs. Overly permissive tests can give agents false confidence. False positives in tests can confuse the agent, leading to generated code that does not follow intended behavior, or wasting tokens as the agent tries to reason through the discrepancy between the failing test and the intent. I saw Claude once attempt to undo an entire repo-wide refactor because it was failing tests written before the refactor, only to align itself midway by reading documentation and realizing that the tests were outdated. Since then, I set up a process so that code changes are always accompanied by updated documentation and tests.
Read more
Personal Anecdote: AI Scientist Collapse
AI Scientist is a system by Sakana AI for using LLMs to perform research independently. I have been running my own version of AI Scientist for a while, and had the chance to review its research notes this week. Overall, the system independently conducted valid experiments and produced interesting results. However, I found that it would too often dismiss a certain idea as not worth pursuing, so I investigated the reasoning behind its decision. I found out that it would first write clear, scoped dismissals like “no detected improvement in this experiment”. However, as the agent iterated and referenced old research logs, it became more and more defeatist, writing vaguer dismissals like “no improvement,” then going even further and saying “verified negative” or “dead end”. It went all the way to saying that the approach was “unwinnable”.
This reminded me of model collapse, where models trained on recursively generated data lose diversity in their outputs. It resembles the same kind of self-reinforcing narrowing: once the agent’s own logs started framing an idea as hopeless, later passes treated that framing as stronger evidence than it really was. After I pointed this out, the agent did correct its own mistakes and started exploring more diverse paths again, but it was a good reminder that even strong frontier agents can become myopic if an independent second eye does not correct their course.