Reality check on self-distillation and Redesigning source control

Welcome to the 16th issue of Agents in Practice.

Reality check on self-distillation

Self-distillation trains a model using guidance from a teacher version of itself. In agent training, the teacher can receive extra hints, such as task-solving strategies or later observations from a training attempt, while the student learns to act without those hints.

Researchers from Zhejiang University and Ant Group compare 10 self-distillation methods for training LLM agents against GRPO, a reinforcement learning (RL) baseline that learns from task rewards. The authors note that previous papers often reuse published baseline scores under different experimental conditions, making it difficult to tell whether the gains come from the method or the setup.

They evaluate the methods on ALFWorld, ScienceWorld, Search-QA, and WebShop using Qwen3.5 models of 2B, 4B, and 9B parameters and found that only 2 of the 10 methods outperformed GRPO when averaged over the 12 settings. These methods also tended to require more training time than GRPO while performing worse in many settings.

Personal Thoughts

Submissions to ML conferences have increased steadily, with an estimated 60,000 abstract registrations for ICLR 2027. Now, verifying that a method works as claimed seems just as important as proposing a new method.

arXiv Preprint

Redesigning source control

In their blog post, Anthropic introduced Scion, an internal version control system built to handle the volume of code produced by agents. The company reports that its daily merged code changes grew from around 2,500 in May to over 30,000 in September, and that maintaining hundreds of Git mirrors became difficult as the merge rate increased.

Since most clients are agents and jobs that are already online, Scion keeps operations such as commits, merges, and history queries on the server. Clients fetch the files they need, and some jobs can use the API without checking out code at all. Anthropic reports that CI and research jobs now check out code in seconds rather than minutes. However, they do stress that Scion is designed for enormous monorepos and is not a general replacement for Git.

Personal Thoughts

Both implementing and reviewing code are getting automated rapidly, accelerating the speed of development. The surrounding infrastructure around coding needs to be modified to match this new pace. We may see more infrastructure being introduced that optimizes CI to remove bottlenecks in the development cycle.

Anthropic Blog Post

One-liners

On the model side, MiniMax released M3.1 Flash (Oct 19) through its API, and Recursive released Flint (Oct 21), an open-weight model for local coding. Recursive says its automated research system helped develop Flint.
Perplexity’s Computer now runs fully locally on Apple silicon Macs (Oct 20), keeping both the model and task execution on the device.
Cognition’s Devin Oncall (Oct 19) is now generally available for investigating production incidents.
Google Cloud announced Agent Compute Engine (Oct 22), a service for running agents in isolated environments, with preview access planned in the coming months.

Reality check on self-distillation and Redesigning source control