Welcome to the sixth issue of Agents in Practice! This week I do not have anything substantial to share as a personal anecdote, so I decided to replace it with a One-liners section summarizing some other things that have been happening. Tell me what you think!
📬 Agents in Practice is a weekly newsletter on agentic AI research and applications. Subscribe here →
SentinelBench: Does your agent wait and monitor?
Although agents are becoming increasingly capable, some things are beyond an agent’s control. Sometimes, meaningful state changes can only come from the environment, so the agent must wait and monitor until they occur. Microsoft released SentinelBench, a benchmark that tests how well agents can wait and monitor for state changes. It contains 100 tasks across 10 synthetic web environments that mimic real-world products (GitHub -> MicroHub, Robinhood -> MicroHood, etc.). The authors also provide a baseline using three models (GPT-5.4 at low reasoning, GPT-4o, and Qwen 3.5:9B) with two waiting strategies (sleep and wait_for).
- The
sleepstrategy simply blocks the agent for a prespecified time, resemblingsleep(time). - The
wait_forstrategy takes a baseline textual snapshot of the page, then checks once per second for textual changes. For each new, previously unevaluated change, it invokes the LLM to determine whether the condition has been met.
In their baseline experiments, they show that both model choice and waiting strategy affect benchmark performance.
Personal Thoughts
I often ask my coding agent to run small-scale fine-tuning jobs or evaluations, which can take minutes to hours. Depending on the agent harness, the waiting strategy can vary considerably. Some agents wait for a few minutes, check whether the task has finished, then wait a few more minutes. It would be interesting to see how such harnesses impact SentinelBench scores.
Read more
Databricks’ coding-agent benchmark
Databricks shared how they created an internal coding benchmark to evaluate models and harnesses on actual tasks in their codebase. They find that harnesses have a major impact on context efficiency. When Databricks ran a given model at the same reasoning effort through its native harness and Pi, quality remained similar while costs differed by more than 2× in some cases. Pi also sent about one-third as much context per turn. They also find that open models like GLM 5.2 are now in the top capability tier.
Personal Thoughts
Nowadays, every new model that comes out with better benchmark scores raises suspicions about whether the scores are real or whether the model has been “benchmaxxed” (optimized specifically to perform well on public benchmarks). This is increasingly difficult to determine ahead of time because many benchmarks consist of data mined from the internet. Such internal benchmarks may provide a useful signal that public benchmarks cannot.
Read more
One-liners
- SEED collects agent trajectories, converts them into training-time hindsight skills, and distills the skills’ behavioral effects back into the policy model.
- MM-IssueLoc is a benchmark for isolating the effect of visual evidence on repository-level issue localization, explicitly testing whether images help, hurt, or are ignored.
- NOOA is NVIDIA’s new Python framework that represents an agent as a native Python object, in which a method with
...becomes an agentic loop.
Subscribe to Agents in Practice
Get new issues of Agents in Practice — a newsletter summarizing exciting new research and applications in agentic AI — delivered to your inbox.