Subscribe
Transcript
Generated: 2026-04-29 04:49 UTC
---
MayaBefore we jump in, here's a quick setup for this episode on topic_03_anthropic_multi_agent_research_system podcast. You'll hear Maya and Maya work through the topic together.
MayaImagine you are trying to fix a bug in a huge codebase. One person is searching the repo. Another person is reading the test failures. A third person is checking the framework docs. A fourth person is reviewing the patch for security problems.
MayaNow imagine all of them are A I agents.
LeoThat sounds powerful. It also sounds like a recipe for chaos.
MayaExactly. And today's episode is about that tension.
MayaWe're looking at Anthropic's engineering write-up, How we built our multi-agent research system. It is about research agents, not coding agents directly. But it is one of the clearest production write-ups on what happens when you move from a single tool-using agent to a coordinated team of agents.
LeoSo the question for us is not, "Should every coding agent become multi-agent?"
MayaRight. The better question is:what can coding-agent teams learn from a production multi-agent system?
LeoLet's start with the basics. When people say "multi-agent system," what do they actually mean?
MayaIn plain language, it means more than one model-driven worker is involved in solving the task. Anthropic describes agents as language models autonomously using tools in a loop. Their Research feature uses a lead agent that plans a research process, then creates subagents that search for information in parallel.
MayaSo instead of one agent doing everything step by step, you get a structure like this:
MayaUser asks a complex question. Lead agent makes a plan. Lead agent creates subagents. Each subagent investigates a slice of the question. Subagents return findings. Lead agent synthesizes the answer.
LeoSo it's like a research manager with a small team of analysts.
MayaThat's a great analogy. And the important detail is that the subagents have their own context windows. They can each go deep on one part of the question without crowding everything into the lead agent's context.
LeoSo the benefit is not just parallelism. It is also context separation.
MayaExactly. Anthropic describes subagents as a kind of compression mechanism. Each subagent searches through a lot of information, then returns the important pieces to the lead agent.
MayaFor coding, that is an important idea. A subagent might inspect a part of a monorepo, read documentation, or analyze test failures, then return a short, structured finding.
LeoBut the Anthropic post also says something interesting: most coding tasks are not as naturally parallelizable as research tasks.
MayaYes, and that caveat is crucial.
MayaResearch can often be breadth-first. Suppose the user asks, "Find all major companies working on A I agents in healthcare." One subagent can search startups. Another can search public companies. Another can search academic spinouts. Another can search F D A-related examples. Those branches are somewhat independent.
MayaCoding is different. If two agents edit the same file at the same time, they can conflict. If one agent changes the A P I and another writes tests against the old A P I, you now have coordination problems.
LeoSo a multi-agent coding setup should not be five agents randomly writing patches.
MayaExactly. The safer pattern is: parallelize investigation, then centralize final patch ownership.
MayaLet's use our recurring course example. The bug is: users can submit an empty shipping address.
MayaA lead coding agent could create three subagents.
MayaSubagent one: "Find where shipping address validation happens in the frontend and backend. Return file paths and relevant functions. Do not edit files."
MayaSubagent two: "Reproduce the failing behavior and identify the smallest test that should fail. Do not edit files."
MayaSubagent three: "Check whether there is shared validation logic or existing patterns for empty string handling. Do not edit files."
MayaThen the lead agent reads those findings and makes one coherent patch.
LeoThat sounds much less chaotic. The subagents investigate. The lead agent integrates.
MayaRight. And that is the first lesson for coding-agent training data: don't only collect final patches. Collect delegation behavior.
LeoLet's unpack that. What makes a good delegation prompt?
MayaAnthropic emphasizes that the lead agent has to teach subagents what to do. Each subagent needs an objective, an output format, guidance on tools and sources, and clear boundaries.
MayaIn the early versions they describe, vague delegation caused problems. For example, if a lead agent simply says "research the semiconductor shortage," multiple subagents might duplicate the same work or chase different time periods.
MayaFor coding, vague delegation could look like: "look into the validation bug."
LeoThat's not enough. The subagent might inspect random files, propose a patch, or duplicate another agent's work.
MayaExactly. A better delegation is:
Maya"Search the repository for shipping address validation logic. Focus on frontend form validation and backend request validation. Return the top three relevant files, the function names, and a one-sentence hypothesis about where the bug lives. Do not edit files."
MayaNotice the pieces:
MayaObjective: find validation logic. Scope: frontend and backend. Output format: files, functions, hypothesis. Boundary: do not edit files.
LeoThat is much more useful.
MayaAnd for training teams, the delegation prompt becomes a data artifact. We can label whether it was clear, whether it caused duplicate work, whether it gave the right boundary, and whether the lead agent used the result.
MayaA single-agent trajectory tells us what one agent did. A multi-agent episode tells us something richer: how work was divided, how information moved, and how the final decision was made.
LeoOne thing I liked in the Anthropic post is the idea of scaling effort to query complexity.
MayaYes. They found that agents needed explicit rules for how much effort to spend. Simple fact-finding might need one agent and a handful of tool calls. More complex research might need several subagents. Very complex research might use more than ten.
MayaThe coding version is very practical. You don't want a swarm of agents for a one-line formatting change.
LeoBecause then the overhead is bigger than the work.
MayaRight. Here's a coding-friendly effort ladder.
MayaFor a single-file style fix: one agent, quick edit, maybe one targeted test.
MayaFor a small bug with a clear error message: one lead agent, targeted search, targeted test.
MayaFor an unknown bug in a large repo: lead agent plus localization and reproduction subagents.
MayaFor a dependency migration: lead agent plus documentation, compatibility, and test-impact subagents.
MayaFor a major feature: lead agent plus planning, implementation, test-design, review, and risk-analysis agents.
LeoSo one data label we can collect is whether the agent under-invested, over-invested, or matched the task complexity.
MayaExactly. That is valuable for training. A good coding agent should not just know how to solve tasks. It should know how much process the task deserves.
LeoAnthropic also spends a lot of time on tools.
MayaThey do, and this connects directly to our earlier course topic on the Agent-Computer Interface. Tool descriptions matter because agents choose tools based on what they understand the tool can do.
MayaIf the answer is in a workspace document but the agent searches the web, the agent is doomed from the start. If a coding agent has a semantic repo-search tool but only uses raw grep, it may miss relevant code. If a patch tool is poorly described, the agent may corrupt files.
LeoSo tool descriptions are part of the harness.
MayaExactly. Anthropic even describes using an agent to test flawed tool descriptions and rewrite them. They report that improving tool ergonomics reduced task completion time for future agents.
MayaFor coding-agent data collection, that means we should store tool-selection mistakes.
MayaFor example:
MayaThe agent used broad web search instead of internal docs. The agent ran the entire test suite instead of the focused test. The agent used a brittle text replacement instead of a structured patch tool. The agent ignored the code-search tool and opened random files.
MayaThose are not just errors. They are training signals.
LeoAnother principle from the Anthropic post is "start wide, then narrow down."
MayaYes. Agents often start with overly specific searches. In research, that means they might search a long complicated phrase and get no useful results.
MayaIn coding, the same thing happens. A model might search for the exact error sentence from a user report, fail to find it, and conclude there is no relevant code.
MayaA better strategy is broad to narrow.
MayaFor the shipping-address bug:
MayaStart broad: search for "address", "shipping", "validate", "required". Then narrow: inspect checkout form, request schema, shared validators. Then target: test empty string and whitespace-only input.
LeoSo a good trajectory does not just show successful search. It shows search strategy.
MayaExactly. Training teams should log search queries, file reads, and what the agent did after each result. Did it narrow intelligently, or did it keep firing random queries?
LeoMulti-agent systems complicate evaluation because different agents can take different valid paths.
MayaRight. Anthropic points out that traditional evaluations often assume the system should follow a prescribed path. But in a multi-agent system, one run might use three sources and another might use ten. Both can be valid if the final answer is correct and the process is reasonable.
MayaFor coding agents, this is important. Two agents might fix the same bug differently. One patches a shared helper. Another patches the A P I schema. A third patches both frontend and backend. The question is not whether they followed one exact script. The question is whether the final state is correct, maintainable, and safe.
LeoSo what should we evaluate?
MayaAt minimum:
MayaDid the final patch solve the task? Did tests pass, including hidden or reviewer-added tests? Was the patch minimal and maintainable? Did the agent use tools reasonably? Did subagents duplicate work or leave gaps? Did the system spend an appropriate amount of tokens and time? Did it avoid unsafe actions?
MayaAnthropic describes using Large Language Model judges with rubrics for research outputs, including factual accuracy, citation accuracy, completeness, source quality, and tool efficiency. For coding, a parallel rubric might include functional correctness, test coverage, code quality, scope control, tool efficiency, and safety.
LeoAnd human evaluation still matters.
MayaAbsolutely. Anthropic says human testers caught issues automation missed, like agents choosing low-quality sources. In coding, human reviewers catch maintainability issues, architecture violations, and subtle product mismatches that tests may miss.
LeoThe production section is sobering.
MayaIt is. Anthropic explains that long-running agents are stateful. They maintain context across many tool calls, and small errors can compound. If a tool fails early, the agent might follow a completely different path.
MayaCoding agents have the same problem. Imagine a test command fails because dependencies were not installed. The agent might misread that as a code failure and start editing the wrong file.
LeoSo production systems need checkpoints, resumability, retry logic, and tracing.
MayaExactly. And those mechanisms also generate data.
MayaA checkpoint tells us where the agent was. A retry record tells us which failures are common. A trace tells us why the agent made a decision. A deployment record tells us which harness version was running.
MayaThat is why observability is a training-data problem, not just an operations problem.
LeoLet's translate this into the course's central data question. What should model-training teams collect?
MayaFor a single coding agent, we collect a trajectory: prompt, files read, commands run, edits, tests, failures, retries, final patch.
MayaFor a multi-agent coding system, we need something larger: a delegation graph.
MayaThat graph includes:
MayaThe lead agent's plan. Each subagent's assigned task. The tools each subagent used. The artifacts each subagent created. The findings returned to the lead. Which findings the lead used or ignored. The final patch. The verifier results. The human review outcome.
LeoSo the unit of data becomes a team episode.
MayaExactly. A multi-agent episode is not just many trajectories. It is trajectories plus coordination.
MayaAnd coordination is where many failures happen:
MayaSubagents duplicate work. The lead gives vague instructions. One subagent returns too much detail. Another returns a conclusion without evidence. The lead ignores the one useful finding. The final patch combines incompatible suggestions.
MayaEach of those failures teaches something.
LeoLet's wrap this into practical takeaways.
MayaTakeaway one: multi-agent systems are best for tasks with separable branches. In coding, parallelize investigation before patching.
MayaTakeaway two: delegation is a skill. A good subagent task includes objective, scope, tools, output format, and boundaries.
MayaTakeaway three: effort scaling matters. The agent should match the amount of work to the task complexity.
MayaTakeaway four: tool descriptions are part of the harness. Bad tool descriptions create bad behavior.
MayaTakeaway five: evaluation should judge final outcomes and reasonable process, not force one exact path.
MayaTakeaway six: for training data, collect delegation graphs, not just final patches.
LeoAnd the caveat?
MayaMore agents do not automatically mean better coding. More agents mean more coordination. The value comes when the work can be divided cleanly and the system can observe, evaluate, and learn from the division.
LeoSo the clean one-sentence lesson is:
MayaMulti-agent architecture turns agentic coding from one worker's trajectory into a team's coordination problem.
LeoAnd for data teams, that means the logs need to capture the team, not just the final answer.
MayaExactly.
MayaBefore you move on, take one coding task you have seen recently and ask:
MayaCould this task benefit from subagents? If yes, which parts should be parallelized? Which agent should own the final patch? What would you log so a training team could learn from the run?
LeoThat is the bridge from production architecture to model-training data.
MayaAnd that is why this Anthropic article belongs in the course.
CreditsThanks for listening. The producer is William Liu. Join us for the next episode.