Agents in Practice #5: Proactive Memory Agent, Unreliable Benchmarks, and the Rise of Local LLMs

Welcome to the fifth issue of Agents in Practice!

📬 Agents in Practice is a weekly newsletter on agentic AI research and applications. Subscribe here →

Proactive memory agent to help long-horizon agents

Figure 1 from the paper illustrating system integration

In long trajectories, useful signals for the next decision are often sparsely scattered across the trajectory. Meta AI proposes having a separate “memory agent” that keeps a structured memory bank. This memory agent is invoked at a fixed interval and updates its bank based on the task description, recent trajectory, and current memory bank. The memory bank contains three components: status (perceived task status), knowledge (stable facts about the task and environment), and procedural memory (what the agent tried and what happened). The memory agent then chooses whether to “intervene,” in which case it adds a transient memory context into the action-agent call. The memory agent is trained via distillation followed by reinforcement learning for calibration.

Personal Thoughts

For my projects, I already have a “memory bank” for the agent consisting of JSON artifacts and Markdown documentation. However, the agent often fails to check the bank when it should, resulting in ungrounded behavior. Having a trained intervention policy could be a smart way to ensure the bank is accessed reliably.

Read more

• arXiv preprint
• GitHub Repo

Life, Liberty, and the Pursuit of reliable benchmarks

Chart showing types of issues in SWE-bench Pro

In February, OpenAI found that SWE-bench Verified is contaminated by showing that models (GPT-5.2, Claude Opus 4.5, Gemini 3 Flash) can output patch-specific details (even the exact gold patch) from minimal cues like an incomplete snippet of the task description. As a result, OpenAI said they would report results from the public split of SWE-bench Pro as it is not perfect but empirically seems to suffer less from contamination issues. Five months later, OpenAI has now retracted their recommendation to adopt SWE-bench Pro. They say that after auditing tasks with both investigator agents and human software engineers, 4 categories of problems were found:

1. Overly strict tests that enforce specific implementation details
2. Underspecified prompts that do not fully describe requirements that the hidden tests enforce
3. Low-coverage tests that allow incomplete fixes to pass
4. Misleading prompts that contradict what is tested or point the agent to a wrong path

Personal Thoughts

It is easy to create large benchmarks by mining examples from repositories, as most steps could be automated. However, making sure all examples are “complete” is a difficult task that requires rigorous human review. OpenAI’s earlier post on SWE-bench Verified being contaminated showed why we need new benchmarks fast, matching the pace of agent capability. This post on SWE-bench Pro shows the other side: the benchmark needs to be slowly and thoroughly reviewed before it can be used to accurately portray the model’s capability.

Read more

• OpenAI Post
• ScaleAI Post about SWE-bench Pro
• SWE-bench Pro arXiv preprint

Personal Anecdote: Exploring local LLM solutions

Three blue brain forms compressing into an orange module above a compact local computer

Over the past few days, many compact open-weight models were released claiming performance competitive with DeepSeek-V4-Flash/Pro-Max.

• Motif’s Motif-3-Beta came out with 314B total parameters and 13B active per token.
• Poolside’s Laguna-S-2.1 came out with 118B total and 8B active.
• Upstage’s Solar-Open2-250B came out with 250B total and 15B active.

These models seem to be targeting users that have a decent local setup: Poolside directly mentions that Laguna S 2.1 is a small enough size to run on a single NVIDIA DGX Spark. Of course, these models are more “specialized,” in the sense that they do not offer all the features that frontier models have: Laguna S 2.1 is designed just for coding and long-horizon work, Upstage only supports three languages, and none of the three support multi-modal input like vision. Still, it makes me excited about how much performance could come out of local models when both new hardware and new models come out, and at what point they will become largely indistinguishable from frontier models for day-to-day users.

Subscribe to Agents in Practice

Get new issues of Agents in Practice — a newsletter summarizing exciting new research and applications in agentic AI — delivered to your inbox.