Welcome to the 15th issue of Agents in Practice.
📬 Agents in Practice is a weekly newsletter on agentic AI research and applications. Subscribe here →
LLMs building Codebases from Scratch
Researchers from Meta introduce End2End-SWE (E2E-SWE), a benchmark for evaluating how well coding agents build working codebases from scratch. The agent receives a natural-language specification and an empty workspace to write code. Once the code is generated, the project is installed in a fresh environment and evaluated with hidden tests. A task is considered solved if every test passes. Each E2E-SWE task is generated by an LLM and a software engineer, where the LLM generates test suites and drafts the specification, while the engineer reviews the generated content, makes judgment calls, and steers the LLM.
Experiments with 13 models show success rates ranging from 11.7% to 67.7%, with Claude Opus 5 performing best. The authors also find that most models concentrate their longest reasoning near the beginning of the task, which they suggest reflects planning the codebase before implementing it. The authors do note that some models do exhibit signals of memorization of the repositories that were used to create this data.
Personal Thoughts
For now, I am a bit skeptical of using coding agents to build large codebases from scratch without human oversight, unless it is for prototyping or demoing ideas. A fun benchmark may be evaluating agents on building codebases from scratch explicitly for prototyping purposes, since prototypes don’t require the same level of reliability as production codebases and the benchmark instead rewards quickly generating working code.
Read more
Measuring Agents in Production
Researchers from UC Berkeley, Intesa Sanpaolo, UIUC, Stanford, and IBM study how teams build and evaluate agents deployed in production and find that the primary bottleneck of deploying agents is reliability.
The case studies also find that teams often favor simple model solutions, mostly using proprietary frontier models (85%) and not using any post-training (70%). They find that practitioners build agents primarily to increase productivity through automation, and 93% of deployed agents serve human users. The study also finds that 66% of the agents allow response times of minutes or longer with 17% having no explicit limit.
Humans are also still involved with the process, with system prompts being written completely by humans in 34% of surveyed deployed systems and by humans with LLM assistance in 45%. On evaluation also, human evaluation was the most popular method with 74% of respondents using it.
(Note that the study reflects deployments from April to November 2025, so the results may not reflect the current state of agent deployments.)
Personal Thoughts
These types of surveys are always interesting as they give a different perspective from researchers on how agents are being used in practice. It seems natural to see that with reliability being the bottleneck, humans are still heavily involved in the process, and that simple model solutions are preferred. It will be interesting to see what new techniques will be developed to improve reliability and reduce human involvement in the future.
Read more
Personal Anecdote: Trying Strata for local LLM
Strata is a GitHub repo that allows people to run Qwen3.8-Flash-Next on consumer hardware by using quantized models and offloading unused experts from GPU to RAM and putting the n-gram table on an SSD. I had a 3090 machine with 64 GB RAM so I installed and tried it with the IQ3_XXS quantized model using OpenCode as the harness.
Although definitely not the quality of a Frontier-lab model (even GPT-6 Luna or Claude Haiku 5.5), it was fast and capable for a local model! I will be using it for personal log management, as it is personal information that I do not want to send to the cloud, and it doesn’t require the quality of a Frontier-lab model or the concurrency of a cloud model. I will be writing more about my experience with Strata in the future.
One-liners
- Following the decision models covered last week, Microsoft released Microsoft-Decision-1 (Oct 9) based on Qwen3.5-9B.
- Meta released Muse Wonder and Muse Wonder Full (Oct 12), models focused on personal and professional agent tasks.
- Google opened Gemini 4 Argon to developers (Oct 13) through the paid Gemini API, with up to 1M output tokens in long-decode mode, sharing the context budget with input.
- Reflection released Beam (Oct 16), its first model for coding, reasoning, and long-running agent tasks.
- Embedding models also got updates. Cohere released Embed 5 (Sept 30), with Pro and Fast variants sharing an embedding space. Google released EmbeddingGemma 2 (Oct 6) that maps text, images, audio, and video into a shared space.
- OpenAI announced textGrain (Oct 5), a text watermark rolling out to eligible ChatGPT and Codex output in the EU.
Subscribe to Agents in Practice
Get new issues of Agents in Practice — a newsletter summarizing exciting new research and applications in agentic AI — delivered to your inbox.