Why the unit that matters is no longer the code file but the record of the work that produced it
The series overview lays out the shared mental models and expert disagreements that recur throughout the podcast.
The core loop — inspect, act, observe, revise — and the artifacts it leaves behind
Topic 1 gives the foundation: agentic coding, repository-level tasks, tools, feedback, and trajectories.
Completion, generation, editing, agentic work — why fixing a repository is not writing a snippet
Search, file views, edits, execution — the same model looks careful or confused depending on its tools
This episode explains the interface layer between the model and the computer: search, file views, edits, execution, and feedback.
An agent run is only reusable if it was recorded — what a trajectory has to capture
Real GitHub issues, pinned environments, and fail-to-pass tests — how repository-level evaluation got honest
Linter-gated edits and scoped file views — designing the interface an agent actually reads
Edit formats, unit-test feedback, and why the second attempt is the one that tells you something
A reference-material episode explaining Aider’s code-editing benchmark and polyglot benchmark.
What a cloud agent leaves behind when the work is done — an artifact trail for audit, evaluation, and training
A reference-material episode on GitHub Copilot cloud agent as an observable production workflow.
A map of how experts judge coding agents — and why review is its own skill
A topic-overview map of how experts judge coding agents: the benchmark families and what each is shaped to see, the verification signals from hidden tests to human review and their blind spots, code review as a first-class capability, and why reviewer trust — not raw bug-catch count — decides whether a review agent is used. Built around one running example: a data-library fix that must pass hidden tests and survive code review.
How do experts evaluate coding agents without fooling themselves? Maya and Leo map four working areas — benchmark families, verification signals, code review as its own skill, and reviewer trust — through one running example: a data-library patch that must pass hidden tests AND survive human review. Includes the real expert splits: single leaderboard vs. fresh private evals, tests-as-oracle vs. human/verifier judgment, and bug recall vs. precision in review.
The working loop behind a coding-agent score — beyond one leaderboard number
The practical on-ramp to Topic 2: what you actually do when a coding agent lands a real change and someone needs a verdict. Maya and Leo walk the evaluation loop in four moves — pick the instrument that fits the work (bug-fix vs. feature vs. terminal), record what signal judged it (visible tests, hidden tests, oracle, human review), read the trajectory when the path carries signal the diff erased, and judge both the change and the engineering process. Built around one running example: a data-library fix that passes every hidden test but silently breaks refunds for one currency.
A coding-agent score is a receipt, not the meal. Maya and Leo build the practical evaluation loop you run when an agent lands an actual change: pick the instrument for the work (SWE-bench-style hidden tests vs. Aider's edit-reliability split), notice that pass/fail is a property of the judge not the patch, read the trajectory when the path holds signal the final diff erased, and judge both the change AND the review of it. One running example carries it — a data-library patch that passes every hidden test but disables refunds for one currency — plus the honest limit: even a careful workflow has a shelf life when benchmarks get contaminated.
How to read coding-agent leaderboards without being fooled by a single number
A coding-agent leaderboard reports one number, but that number is a bundle — task shape, the room the agent worked in, the judge that scored it, language coverage, contamination risk, and the failure modes a ranking never shows. Maya and Leo pull the bundle apart across the real benchmark families (issue-fixing, edit-and-repair, terminal, feature-development, open-ended, and mobile) so you can read the next benchmark headline critically.
Two coding agents post the identical score — one fixed clean Python bugs in a single attempt, the other took fifty tries with a reranker and quietly dropped flaky tasks. This episode unpacks why a single leaderboard number is incomplete: task shape, harness, scoring method, languages, contamination, and the cost and failure modes the ranking hides. Grounded in SWE-bench Verified, SWE-bench-Live, Aider's polyglot leaderboard, and the terminal, feature, and open-ended benchmark families.
Why reviewing a patch is a separate capability from writing one — and how to measure it
Writing a patch and reviewing one are two different jobs — and almost nobody scores the second directly. Maya and Leo treat code review as its own agentic capability: reading the pull request's intent, understanding the diff against the surrounding repository, localizing to the exact line, judging severity, and earning a real human reaction (accepted comments, author responses, applied fixes, post-merge outcomes). Built around one running example — a data-library fix that passes every test yet silently disables refunds for one currency, a bug only a reviewer reading intent against context ever catches.
Review is the other half of agentic coding. Maya and Leo argue that reviewing someone else's patch is a separate skill from writing your own — generative versus forensic — and walk a reviewer's desk: reading PR intent, understanding the diff against untouched code, exact-line localization, severity, and the acceptance loop (accepted/rejected comments, author responses, applied fixes, post-merge signals). One running example threads through: a data-library patch that passes every visible test yet quietly kills refunds for one currency. Includes the honest limitation — human acceptance is the best ground truth we have and it's still biased.
Why a reviewer that catches more bugs can still be the worse reviewer
Review quality is not bug recall. This concept deep-dive builds the review-quality scorecard row by row — recall and precision, the signal-to-noise line, specificity and actionability, the special danger of hallucinated issues, and the trust row they all feed into — using one running example: two agents reviewing the same data-library patch, where the quieter reviewer that catches fewer bugs is the one the developer keeps reading. Maya and Leo show why noise sits on the cost side of the ledger for review the way it never did for patch generation, and why trust can only be measured by what a human does next.
A map of the code-review benchmark family — and why review is graded differently from patches
A map episode of the code-review benchmark family. Maya and Leo walk eight benchmarks — CodeReviewer, CodeReviewQA, CodeFuse-CR-Bench, CR-Bench, SWE-PRBench, CodeReviewBench, Code Review Bench by Martian, and MetaMateCR — grouped into five rising bars: writing the review comment, understanding the diff, being useful despite noise, whole-pull-request realism, and production trust. The throughline: unlike a patch benchmark, a review benchmark has no green-checkmark oracle, so each one invents its own ground truth — and that choice is what makes them different instruments.
Hand the same pull request to eight code-review benchmarks and they won't agree on what 'good' means. Maya and Leo map the code-review benchmark family — CodeReviewer, CodeReviewQA, CodeFuse-CR-Bench, CR-Bench, SWE-PRBench, CodeReviewBench, Martian's Code Review Bench, and MetaMateCR — as five rising bars from comment generation to production trust, and explain why review is graded so differently from patch generation: there's no executable oracle, so every benchmark invents its own ground truth, and false positives finally count against you. A map before the per-benchmark deep dives.
The human-cleaned answer key that became the field's default coding-agent score
SWE-bench Verified is the human-cleaned subset of SWE-bench that became the default coding-agent score. This episode explains why the original benchmark's auto-harvested tasks were noisy — underspecified issues, over-strict hidden tests, broken environments — how human engineers (with OpenAI) validated 500 tasks to fix that, how the hidden fix-flip and guard tests decide "percent resolved," and what a clean number still cannot tell you about other languages, feature work, review quality, or contamination.
Beating contamination with a calendar, not a scrub
SWE-bench-Live is the continuously-refreshed answer to a benchmark's slow expiry. Maya and Leo unpack why it exists — a fixed set of public GitHub issues gets contaminated and goes stale — and how Live sidesteps both by automatically curating fresh, post-training issues each month, scoring them with the same hidden catch/guard tests inside auto-built containers. They contrast it with Verified's hand-cleaned frozen set, explain how frozen lite/verified splits keep ranking fair, and walk the real costs: environment flakiness, test-validity decay, perpetual maintenance, and noisy small snapshots.
How do you keep a code-fixing benchmark honest when models keep memorizing its answer key? Maya and Leo dig into SWE-bench-Live — the live, contamination-resistant variant of SWE-bench that automatically curates fresh GitHub issues every month so the tasks are newer than the model. Same resolve-the-issue, hidden-test scoring as the original, but the calendar does the auditing that contamination-checking can't. They cover the automated curation pipeline, the agentic environment setup that makes monthly refresh feasible, the frozen splits that keep head-to-head ranking fair, and the honest limits: containers don't guarantee isolation, tests decay over time, upkeep never ends, and each month's snapshot is small and noisy. A paper deep-dive in the Evaluation and Code Review topic.
When being right isn't enough — measuring whether a model can hand over an edit the tool can actually apply
A deep dive on the Aider polyglot benchmark and why it scores something most leaderboards hide: not just whether a coding model finds the fix, but whether it can deliver that fix as an edit the tool can mechanically apply. Built on 225 challenging Exercism exercises across six languages (C++, Go, Java, JavaScript, Python, Rust), the benchmark reports two numbers — a pass rate and a percent-well-formed — so a fumbled handoff stops looking identical to a genuine reasoning failure. Maya and Leo unpack edit formats (whole-file vs. diff), the counterintuitive finding that harder formats make models write worse code, and the benchmark's honest limits: self-contained puzzles rather than real repos, likely training-data contamination, and run-to-run variance.
The benchmark that makes the messy terminal itself the exam
Most coding-agent benchmarks hand the agent a clean room — repo installed, tests wired up. Terminal-Bench 2.0 refuses: it drops the agent into a raw, sandboxed command line and makes operating the environment end-to-end — install, build, debug, reach the goal — the actual exam. Grading checks the final state of the world rather than the commands typed, each hard task ships human-authored verification, and frontier agents are still well short of solving it. Maya and Leo unpack why the terminal, not the patch, is where agents break — using the running example of a data-library fix that first has to build and run before it can pass hidden tests or survive review.
Terminal-Bench 2.0 makes the messy terminal itself the test: a coding agent is dropped into a raw sandboxed shell and must do the whole job — install, build, debug, reach the goal — graded on the final state of the environment, not the path it took. Maya and Leo cover why it's curated-and-brutal rather than huge-and-shallow, why frontier agents still fall well short (the failures are state-tracking and unchecked-command failures, not coding failures), and the honest limits: environment drift, small samples, and reward hacking of outcome checks. Tied back to the running data-library example, where getting the project to build is half the battle before any test or reviewer sees the patch.
When the task isn't a bug fix but a whole feature — and the score collapses
A deep dive on FeatureBench, a benchmark that grades coding agents on building whole features — not fixing single bugs. Maya and Leo unpack its core trick: deriving feature-sized tasks by tracing unit tests along a project's dependency graph (the test suite as a stencil), checking that each feature is cleanly removable, and grading whole capabilities with hidden execution tests. The episode's headline is the score drop — a frontier model strong on bug-fixing benchmarks falls off a cliff on feature tasks — and what that gap reveals about the difference between the narrow patch skill and the wide engineering skill. Built on the running data-library example, stretched from a bug fix to a streaming-support feature ask.
When the test is a live opponent: goal-oriented, competitive software engineering
CodeClash evaluates coding agents on open-ended, competitive goals instead of fixed bugs. Two codebases are improved over many tournament rounds, then dropped into a shared arena where a winner falls out of the competition itself — no spec, no hidden test grading the diff, and an opponent that can't be memorized. Maya and Leo unpack what this measures, how it differs from issue-fixing benchmarks like SWE-bench, and the honest limits: results are noisy, the arena is synthetic, and the headline that top models lose every round to human experts measures one narrow skill — open-ended strategic development — not the well-scoped fixes models are already good at.
What happens when a coding benchmark replaces the hidden-test answer key with a live opponent? Maya and Leo dig into CodeClash, which scores agents on goal-oriented software engineering: two codebases improve over many tournament rounds, then compete head-to-head in an arena where the winner is decided by score, resources, or survival. They contrast it with one-shot, fixed-target benchmarks like SWE-bench, explain why a live adversary is contamination-resistant, and stay honest about the limits — noisy outcomes, a synthetic arena, and the finding that models are tactical but not strategic and let their codebases rot. The result that gets overread: top models lose every round to expert humans in this setting — which is one narrow lens, not a verdict on bounded coding.
When the judge of an agent's work is another model — decomposed, repeated, and calibrated
When executable tests run out, can a language model judge a coding agent's whole trajectory well enough to pick the best work? This episode unpacks LLM-as-a-Verifier — a framework that scores agent runs by decomposing the judgment into separate criteria, repeating the verification to beat sampling noise, and reading the model's own logprobs so its confidence stays inside the score, then ranking candidates in a round-robin tournament. Maya and Leo show where it helps that tests can't reach, and where it inherits four honest failure modes: verifier hallucination, calibration drift, self-preference bias, and cost.
Tests pass or fail — but a lot of engineering judgment never lands in a test. LLM-as-a-Verifier asks whether a language model can be a disciplined judge of a coding agent's whole trajectory. Maya and Leo walk three moves that turn 'rate this run' into a measurement — break the question into criteria, ask more than once, and read the model's confidence through its logprobs — then use a tournament to pick the best of several candidates. They land it on the topic's running argument (tests-as-oracle vs. model judgment) and name the four limits every model-as-judge inherits: hallucination, calibration drift, self-preference bias, and compute cost.
The paper that turned code review into a learnable, three-part task
The foundational code-review paper. Maya and Leo unpack CodeReviewer — a model pre-trained specifically on review situations (diffs paired with the human comments left on them, across nine languages) rather than on plain code. It splits reviewing into three scoreable jobs: judge the diff (the Verdict), write the comment (the Note), and fix the code (the Patch). The honest confession: the Verdict and Patch have answer keys, but the Note — the part that decides whether a reviewer is trusted — is scored against one human comment among many valid ones, so a high comment score means plausible, not useful. Plus the limitations that drove the rest of the field: dataset noise and a single-hunk view that can't see whole-PR, cross-file bugs.
Can you pre-train a model just for code review? CodeReviewer says yes — feed it diffs paired with real human review comments across nine languages, then split reviewing into three trainable jobs: a Verdict (is this diff okay?), a Note (write the comment), and a Patch (fix the code). Maya and Leo walk the three tasks, the review-shaped pre-training behind them, and the honest limitations: the comment score rewards sounding like one specific reviewer, the training comments are gloriously noisy ('lgtm', 'nit', 'rebase'), and a single-hunk keyhole can't see the whole-PR, cross-file bug — the exact gap later review benchmarks were built to close. Grounded in the running example: a data-library fix that passes the tests but silently disables refunds for one currency.
Grading whether a model understood the change before it ever writes the comment
CodeReviewQA grades the part of code review that comment-generation benchmarks can't: did the model actually understand the change? Maya and Leo unpack how the paper turns review comprehension into multiple-choice question-answering over a diff — recognizing the change type, localizing it to the exact line, and identifying the right fix — so a single fuzzy 'bad comment' becomes a readable per-step diagnosis. They run the topic's data-library patch through the three gates, then weigh the catch: multiple choice tests recognition, not the open-ended review you do in the wild.
Closing the reality gap — grading a whole code review, not a clean snippet
CodeFuse-CR-Bench attacks the 'reality gap' in code-review evaluation: instead of grading a clean snippet question, it drops a reviewer model into real pull requests with full repository context — the linked issue, the PR, the repo state — and grades the whole review end to end. Maya and Leo unpack the design: a fused rule-based checker (location, syntax) paired with a model-based quality judge, why no single model turns out to be well-rounded across review's dimensions, how redundant context becomes its own test of focus, and the benchmark's own limits (single language ecosystem, a model-as-judge with its own blind spots, and the gap that always remains between a frozen benchmark and a live reviewer).
Why catching every defect can still make a review agent useless
A code review agent that finds every bug can still be useless if it buries the real ones in noise. This episode unpacks CR-Bench, which measures the real-world utility of AI code review agents — how it scores a comment on defect coverage, exact-line localization, severity, and actionability; why it counts false positives as first-class; how it differs from patch-generation benchmarks; and the hard frontier it surfaces between catching issues and keeping a developer's trust.
Grading review agents on whole, real pull requests — against what human reviewers actually flagged
SWE-PRBench moves coding-agent evaluation from writing patches to reviewing them. Maya and Leo unpack a benchmark that grades agents on whole, real pull requests — multi-file diffs as humans submitted them — against the comments real reviewers actually left before merge. The humbling result: frontier models recover only a minority of human-flagged issues on the diff alone, and counterintuitively often do worse when handed more file context, because attention dilutes. A clear-eyed look at why review is a separate skill from generation, plus honest limitations: the oracle is model-judged, the human gold standard is itself noisy, and the slice is open-source review culture.
A deterministic, auditable code-review benchmark — known bugs, two model judges, every trace in the open
CodeReviewBench scores AI code reviewers against bugs it already knows the location of, then has two independent language models grade whether each reviewer caught them — a deterministic, fully-published alternative to human-annotated review benchmarks. Maya and Leo walk through how it builds planted-bug test cases across five languages, why it splits local-logic from cross-file defects, why coverage and validity are one dial you can't game separately, and why publishing every score, trace, and judge's reasoning is the point — alongside the honest limits: a finite known-bug answer key and graders that are themselves language models.
Can a code-review benchmark be rigorous AND auditable by anyone? CodeReviewBench plants known bugs, has two independent model judges score reviewer comments, and publishes every trace so you can re-check the leaderboard yourself. Maya and Leo cover the planted-bug bench across TypeScript, Python, React, Ruby and Java; the coverage-versus-validity dial you can't game one side of; the local-logic versus cross-file split that exposes reviewers who can't see around corners; and the two real limits — a finite answer key and model graders that can be fooled. Builds on the prior SWE-PRBench episode's human-annotated ground truth as the complementary opposite.
A code-review benchmark that grades itself against real developer behavior
Martian's Code Review Bench (v0) is a code-review benchmark that doesn't fully trust itself. It pairs a curated offline gold set of marked bugs with a live behavioral signal — which review comments developers in a large pool of real open-source pull requests actually act on — and treats disagreement between the two as evidence the benchmark itself is stale. Maya and Leo unpack the 'gold room and the street' design, why acted-upon is a usefulness proxy for review (and where it breaks), and the v0 honesty: the two halves don't agree yet, no tool clears a modest recall bar, and the gold set likely undercounts real issues.
A code-review benchmark that grades itself. Maya and Leo dig into Code Review Bench v0 from Martian: a curated offline gold set paired with a live signal from real open-source pull requests tracking which review comments developers actually acted on — with divergence between the two read as proof the benchmark is wrong, not just the tool. They cover the self-correcting, monthly-refreshed design as a defense against benchmark overfitting, why 'acted-upon' is a behavioral proxy for review usefulness, and the v0 limits: offline and online still diverge, no tool clears a modest recall bar, the gold set likely undercounts issues, and the behavioral street is open-source norms, not every codebase.
When a developer applies the AI's fix in production, evaluation finally touches ground truth
MetaMateCR is Meta's production study of AI-assisted fixes to code-review comments at scale: instead of just flagging a defect, the system drafts the actual fix and the real signal is whether a developer applies it. Maya and Leo unpack why the applied-fix rate is the strongest real-world evidence in the whole topic — revealed preference in production, impossible to game like a leaderboard — including a sharp developer-in-the-loop lesson (showing the draft to reviewers slowed reviews down; showing it only to authors fixed that) and the honest limitation: the strongest signal is also the least portable, bolted to one company's scale and proprietary comment-fix data.
Why a 90%-on-needle-function score tells you the model can locate code — and almost nothing about whether it should touch it
RepoQA's Searching Needle Function task looks impressive — 500 tests across 50 repositories and five languages, with top models scoring around 90%. But what is that score a score of? Maya and Leo take opposite sides on whether a controlled find-the-function probe still earns its keep in 2026: Maya defends it as the necessary bottom rung every harder repo task depends on, Leo presses the context-length collapses, memorization inflation, and structural blind spots that show the probe doesn't track the job. They converge on a layered evaluation ladder and one blunt synthesis — RepoQA is a unit test for 'find the code,' not an integration test for 'understand the codebase,' and definitely not a safety check for 'make the right change.'
Why a coding agent is a system, not a model — and why the crew, not the driver, wins races
A topic-overview map of the coding agent as a system, not a model. Maya and Leo walk four pillars — product-style async workflows that turn a task into a reviewable pull request, single-vs-multi-agent orchestration, harness engineering (context, tools, memory, validators), and observability as the flywheel that feeds debugging, evaluation, training, and safety. Built around one running example: a pit crew servicing a single race car — a payments service silently dropping one in a thousand transactions — where the model is the driver and the harness, orchestration, and telemetry are the crew that actually wins the race.
How an autonomous agent turns one assigned issue into a draft pull request you review like a teammate's
GitHub's cloud coding agent, opened up. Maya and Leo follow one assigned task through four rooms — Intake (the many doors a task enters through), the Workshop (a throwaway GitHub Actions environment where the agent plans, edits, and runs the real tests), the Hand-off (a draft pull request, never an auto-merge), and the Back-and-forth (where a PR review comment becomes the agent's next prompt). Along the way they show the harness made concrete in a settings page — default GitHub and Playwright MCP servers, custom-instruction files, a memory preview — and the guardrails that encode intended task size: no auto-merge, one repo per task, a hard 59-minute ceiling, and branch protections that block the agent until it's explicitly trusted. Running example: an API field rename across forty files, reviewable the whole way down.
How a cloud coding agent turns a 'definition of done' into parallel sealed tasks that hand back evidence, not promises
OpenAI's Codex, made concrete. Maya and Leo follow what it takes to hand a cloud coding agent a job and trust what comes back, organized around three things you pack before sending a task off — the Brief (an explicit 'definition of done' plus a repo-root AGENTS.md the agent reads before any work), the Sandbox (a fresh, isolated cloud environment preloaded with your repo, with no internet access by default though you can enable it per environment, where containment is the safety feature and parallelism falls right out of isolation), and the Receipt (cited terminal logs and test outputs so claims arrive with proof, plus a proposed pull request the human still has to approve). They trace the agent-first loop — plan locally where being wrong is expensive, outsource the long implementation to parallel cloud tasks — across the IDE, CLI, and cloud surfaces, including code review with @codex review. Running tension: Leo's honest objection that parallelism is a throughput lie if every branch funnels back through one human reviewer, and Maya's steel-manned answer — fan out the work where a test can be the judge, keep close the work where you have to be.
When a team of agents beats one — and the exact seam where fanning out stops helping
Anthropic's engineering writeup on the multi-agent system behind their Research feature, turned into a conversation. Maya and Leo unpack the orchestrator-worker shape — a lead 'Planner' agent that spawns parallel 'Scout' subagents, each with its own context window, plus a citation agent that checks the footnotes — and the ~90% win it posted over a single strong agent on research. Then they stage the real split: when does fanning out help, and when does it corrupt the work? Built around the topic's running example — a pit crew on one race car, a payments service dropping one transaction in a thousand — they find the seam: fan out the wide, separable investigation; keep one hand on the deep, entangled fix.
How a chain of forgetful agents finishes a project too big for any one of them — the baton, not the brain
Long-running coding agents work in shifts, and every shift starts with amnesia — so the project survives only in what the last one wrote down. Maya and Leo unpack Anthropic's harness write-up around three durable artifacts left on the workbench — a Punch List (a locked-down JSON feature list that doubles as a test sheet), a Shift Log (the handover notes), and a Save Point (git commits you can roll back to) — plus a forced ritual: read everything, build one feature, verify it in a real browser, write it down, commit. They ground it in the topic's payments-bug car, now a multi-day rebuild no single session could hold, and stay honest about the limits: the agent can't see alert modals, and whether one general agent or a crew of specialists is the right design is left open.
Imagine a relay where every runner forgets the race the instant they hand off the baton — that's a long-running coding agent. Maya and Leo dig into Anthropic's harness engineering for agents that build across dozens of memoryless sessions: the two opposite failure modes (one-shotting the app vs. declaring a half-built app done), the split between a one-time setup agent and the coding shifts that follow, and the three artifacts that carry the work — a structured feature list locked into JSON so the model can't quietly rewrite its own scorecard, a progress log, and git save points. They reuse the topic's payments-service bug as a multi-day fix, then mark the seams the source admits: blindness to browser alert modals, web-app-only tuning, and the open question of single agent vs. specialist crew.
Why an agent can't grade its own homework — and how a skeptical second agent turns a six-hour build into something that actually works
Hand a coding agent a whole application to build over hours and it will happily finish, look at its own broken work, and call it great. Maya and Leo dig into Anthropic's write-up on harness design for long-running application development, where the fix is architectural: borrow the adversarial shape of a GAN and split the job into a Maker that builds, a Critic that tries to tear the work apart, and a Contract that pins down what 'done' means before any code is written. They unpack why separating the roles buys tractable skepticism rather than free skepticism, why the Critic must click through the live app like a user instead of grading a screenshot, how the exact wording of the grading criteria quietly steers the whole product, and why a strong harness is scaffolding you tear down as the models get better — costs, blind spots (the agent can't hear its own music app), and all.
A million lines, fifteen hundred pull requests, zero human-written code — and the four jobs the environment had to do
What were the humans doing if agents wrote every line? Maya and Leo dig into OpenAI's harness-engineering report — a five-month run that shipped roughly a million lines of code across about fifteen hundred pull requests with no code written by human hands. The episode names the four jobs the environment had to do: the Map (a ~100-line agent file as a table of contents, because context is scarce), the Rails (mechanical enforcement via layered architecture and linters whose errors carry the fix), the Mirror (verification loops that give the agent senses and hard thresholds), and the Janitor (scheduled agents that clean up AI slop). Leo steel-mans the cost — the heavy harness ran a lot more expensive than a solo agent — and they settle on what's transferable (the discipline) versus local (the economics), landing on the season's turn: in an agent-first world, you build the garage, not the car.
When a search process out-designs the best hand-built harness — by refusing to throw its own history away
Every episode of this topic, the harness was hand-built by smart people. This paper asks: who builds the harness, and why is that still a human? Maya and Leo unpack Meta-Harness, an outer loop that searches over harness source code. An agentic proposer reads a filesystem holding every prior candidate's code, scores, and full execution traces — refusing the aggressive feedback compression that hobbles older prompt optimizers — and writes the next candidate. The payoff: a discovered harness beats a state-of-the-art context manager by ~7.7 points at a quarter of the context budget, a single found math harness lifts five held-out models by ~4.7 points, and discovered harnesses out-search the best hand-engineered baselines on terminal coding tasks. The running pit-crew example goes up a floor — a second race engineer who redesigns the crew after reviewing every past race in full telemetry — and Leo presses the honest soft spots: the search cost the abstract never prices, and whether the coding win used held-out tasks or just held-out runs.
Why the edit format — not the model — quietly sets your coding agent's score
Same model weights, double-digit swings in success: this episode digs into Can Bölük's 'The Harness Problem,' which benchmarks 16 models and shows the edit format — how the agent tells a file what to change — can matter as much as a model upgrade. Maya and Leo walk through three editing dialects (patch, search-and-replace, and the content-hash 'hashline' that swaps copying text for pointing at it), why the weakest models gain the most, and the live fight over whether harness research should happen openly or stay locked inside each vendor's walls.
Guides steer, sensors catch, and the loop between them is the engineering — Böckeler's case for naming the field
The closing frame of Topic 3: Birgitta Böckeler's case that wrapping a coding agent isn't folklore — it's a software-engineering discipline. Maya and Leo unpack guides (feedforward controls that steer before the agent acts) and sensors (feedback controls that catch after), the two sensor speeds — computational (fast, reliable) versus inferential (slow, probabilistic) — and the three regulation harnesses: maintainability, architecture-fitness, and the admitted wall, behaviour. What earns the word 'discipline': lifecycle-wide placement and a steering loop where every repeated failure improves the net instead of the catch. Carried on the topic's running example — the pit crew that stops servicing the car and writes the inspection manual every garage must follow.
How to read a coding-agent improvement study without crediting the wrong thing
A topic-overview map for reading coding-agent improvement studies without crediting the wrong thing. Maya and Leo introduce the five-link Causal Chain — lever, setup, result, artifact, implication — and four 'gyms' one bug-fixing agent gets sent to: the World-Model Room (learn what code does when it runs), the Task Forge (manufacture training tasks), the Reward Range (RL and reward design), and the Terminal Yard (data-engineered environments). Built around one running example — a single team trying to make its agent better at fixing one real repository bug — and a staged debate on synthetic vs. real training tasks.
Teaching a model what code does when it runs, not just how it looks
The first deep dive of Topic 4 opens the World-Model Room. Maya and Leo unpack CWM — Meta's open-weights Code World Models LLM — whose lever is mid-training a 32B model on observation-action traces from a Python interpreter and agentic Docker environments, so it learns what code does when it runs, not just how it looks. They show the payoff on the topic's running bug — a cache that returns stale data on the second call, which a read-the-code model misses and a world model catches by simulating two calls — and stage the real attribution debate: did the trace footage win the SWE-bench number, or did the verifiable-environment RL on top? The throughline: the artifact worth copying is the recipe for turning program runs into world-model training data, and the open checkpoints are an invitation to settle the credit.
Where a model's in-head simulation of code quietly breaks — and why
Topic 4's second deep dive takes the code world model from the last episode and puts it on the workbench. Maya and Leo walk through Debugging Code World Models — a diagnosis, not an improvement study — that asks where a model's in-head simulation of a running program quietly breaks. They name two close-up failures: honest traces that write the full state after every step get token-heavy enough to exhaust the budget on long programs, and errors that pile up in string-valued variables because subword tokenization lets the model see strings only in misaligned pieces. Then they stage the real split on the long run: is the step-thirty drift a fundamental Transformer limit on state tracking, or something more fixable? The paper's controlled permutation-tracking benchmark separates generating the next action from propagating state — and when correct actions are handed in, state tracking holds, so the drift was wrong-action generation, not broken memory. The throughline: the artifact here is a failure map, and a simulator you can't trust everywhere is still useful if you know exactly where not to trust it.
A cage of runnable bug tasks, and the two levers it lets you pull at once
Topic 4's third deep dive steps all the way out of the agent's head and into the gym. Maya and Leo break down SWE-Gym — the first environment for training real-world software-engineering agents — built from around 2,438 executable Python tasks, each a runnable codebase with stood-up dependencies, wired-in unit tests, and a plain-English problem. Because the tasks actually run, every agent attempt (search, commands, failed test, retry, final diff) gets auto-stamped resolved-or-not by a real check — a mountain of trajectories labeled with ground truth, for free. The episode's spine is that SWE-Gym pulls two levers at once and the hosts refuse to blur them: the successful trajectories fine-tune the agent (a stronger single swing, up to ~19 points absolute), and all trajectories — wins and losses — train a separate outcome verifier that picks the best of many sampled attempts at test time (best-of-sixteen, reaching state-of-the-art for open-weight agents at ~32% on SWE-bench Verified and ~26% on Lite). They stage the real split — better agent vs. better picking — and resolve it into a relay that compounds: the agent must land a correct patch in the batch for the verifier to find it, but the selection lever scales on a second axis (test-time rollouts) and neither curve has plateaued. The throughline: the reusable artifact was never the athlete — it's the cage and the record of every graded swing taken in it.
Stop catching bugs, start manufacturing them — four ways to break working code, verified by a real test
Topic 4's fourth deep dive turns scarcity into manufacturing. The whole public world holds only a few thousand real, runnable bug-fixing tasks — so Maya and Leo unpack SWE-smith, a pipeline from the SWE-bench team that stops catching bugs and starts making them: point it at a Python repository, stand up its environment, and automatically forge verified bug tasks by breaking working code. They walk the four bug-forging stations — an LM rewriting a function to be subtly wrong, blunt procedural edits to the syntax tree, combinations that make compound fractures, and PR mirroring that reverts a real human pull request — landing about fifty thousand tasks from roughly a hundred and twenty-eight repos, an order of magnitude past prior work. The load-bearing distinction: the bug's ORIGIN can be synthetic, but its FAILURE never is — every surviving task must make a real test go red. The episode stages the genuine intra-paper split over bug provenance (cheap-but-invented vs. authentic-but-scarce reverted PRs) and resolves it into a portfolio: the realistic source sets the quality ceiling, the infinite sources fill the volume. The proof it transfers — an open agent trained on the manufactured flood resolves about forty percent of the real SWE-bench Verified exam, state of the art among open models — and the durable artifact is the bug-manufacturing pipeline, not the model.
Reinforcement learning that never runs the code — grading patches against the fix a human actually merged
Topic 4's fifth deep dive throws out the stopwatch. Every prior episode graded a coding model by RUNNING the code — build the gym, run the tests, pass or fail is the truth. SWE-RL refuses to run anything: Maya and Leo unpack a method from Meta that does reinforcement learning on raw software-evolution data — the public history of issues, pull requests, and merges — and grades each generated patch by how closely it resembles the fix a human actually merged. The reward is a text-overlap score (a sequence-matching ratio, the kind behind a diff view), a smooth dial from zero to one rather than a single pass/fail bit, with a negative penalty for any output that won't even apply. That dense signal scales to essentially all of GitHub with no environment to build. The episode stages the real split it provokes — does grading resemblance teach correct repair, or just mimicry of human style, since a genuinely better-but-different patch scores low? It resolves into a division of labor: the soft reward buys REACH (every merged patch becomes a training signal), execution buys CERTAINTY (a passing test is ground truth). The proof it transfers: the resemblance-trained 70B Llama resolves about forty percent of the executable SWE-bench Verified exam — best at the time for any model under a hundred billion parameters — and, trained only on software fixes, it got BETTER at math, reasoning, and language it never trained on, while a supervised-copy baseline got worse. The durable artifact is the similarity-reward recipe, not the model.
Last episode SWE-smith manufactured executable tasks where verified failure — run the test, watch it go red — was the whole point. This one pulls the floor out: SWE-RL does reinforcement learning and never runs a line of the code it produces. Maya and Leo dig into a Meta paper that trains on raw software-evolution data (the public record of issues, pull requests, and merges) and grades each generated patch by its TEXT SIMILARITY to the patch a human actually merged — a smooth overlap score instead of pass/fail, with a negative penalty for outputs that won't even apply, computed across millions of pull requests with no environment to stand up. They stage the real fight it starts: a resemblance reward might teach style rather than skill, and it under-credits a correct-but-different fix that diverges from the human's — but a passing-test signal is something you can only compute a thousand times, while resemblance you can compute a billion. It resolves into a division of labor — soft reward for reach, execution for certainty — and the punchline lands twice: the resemblance-trained 70B Llama resolves about two in five real, executable SWE-bench Verified issues (best at the time under a hundred billion parameters), AND it generalized outward, getting better at math, code reasoning, and language it never trained on while a supervised baseline degraded. The takeaway: execution tells you WHETHER it worked, resemblance tells you WHICH WAY is warmer — and a warm signal you can compute a billion times beats a perfect one you can run only a thousand. The artifact you keep is the reward recipe, not the checkpoint.
How coaching a failing agent turns a too-sparse reward into a teachable one
Reinforcement learning from verifiable rewards is honest but unforgiving: on real software-engineering tasks a base model almost never passes, so the reward almost never fires and there is nothing to learn from. Agent-RLVR keeps the strict unit-test reward and adds guidance — strategy, error feedback, and interaction notes fed back after a failed attempt so the agent can re-attempt and win on its own. Those manufactured successes tripled resolve rate on a 72B model, and the same logged attempts trained a reward model that picks the best patch at test time. Maya and Leo stage the real split — is the test the teacher, or is the guidance? — and resolve it into a division of labor: guidance for reach, the verifiable reward for truth.
Mining commit history for executable tasks, and picking the winning patch with two graders that fail differently
Executable software tasks are scarce because the grader is the expensive part. R2E-Gym mines them straight from commit history — back-translating verified human fixes into tasks and keeping only the generated tests that actually separate broken from fixed. Then it tackles a second problem nobody had nailed: when you can't run the hidden test, which of the agent's candidate patches do you ship? An execution verifier and an LLM judge each cap in the low forties on SWE-bench Verified — but they fail on different patches, so combining them lifts an open 32B model to fifty-one percent. Maya and Leo stage the execution-versus-judgment fight in full and resolve it into a rule worth keeping: a perfect grader is a fantasy, two graders that fail in opposite directions are a strategy.
From hand-making one training task at a time to running the data factory: four stations from environment to feature creation, on one line
Every other paper in this topic perfected one kind of training task — forge a bug, mine a commit, grade with two judges. SWE-Hub stops making dishes and builds the line. It operationalizes a 'data factory': four integrated stations on one floor — an Env Agent that turns any repo into a runnable multi-language container, a high-throughput engine that presses out cluster-validated localized bug-fixes, a Bug Agent that builds realistic system-level repairs with human-like issue reports, and SWE-Architect that goes past repair into building whole features from a spec. Maya and Leo stage the real fight in full: is covering creation, not just verified repair, a leap in capability or a crack in the verified foundation? They resolve it into a rule worth keeping — breadth across the lifecycle is the right bet, but trust is not uniform across the floor, so an honest factory labels which station each task came off and never lets a soft build-grade pass as a verified fail-to-pass.
How a synthetic task pipeline taught a 32B model to drive a terminal — and matched models twice its size
A coding model can write a flawless shell script and still freeze the moment it has to drive a live terminal. Nemotron-Terminal treats that gap as a data-engineering problem, not a model-size one: it manufactures terminal-training tasks entirely synthetically — seed-based generation for fidelity, skill-based construction for coverage — then does the real work in selection, with hard quality filtering, a difficulty curriculum, and long-context training. Maya and Leo walk the two 'task kitchens' and the three engineering 'dials', stage the seed-vs-skill design debate, and unpack how a 32B model jumps from roughly 3% to 27% on Terminal-Bench and matches much larger systems — while staying honest about the synthetic-dialect risk and how far a single benchmark generalizes.
The topic built ten gyms; this is the sealed final exam — a real production iOS app where the best agent scores twelve percent
Topic 4 spent nine episodes building gyms — places to train coding agents. SWE-Bench Mobile builds the sealed final exam, and it comes back brutal. Drawn from a real production iOS app, each task hands the agent a written product spec, a Figma design, and a quarter-million-line mixed Swift/Objective-C codebase, then grades it on held-out tests that only pass if the project first builds in a simulator — run as a hosted challenge so the questions stay locked. The best of twenty-two agent-model configurations scores about twelve percent. Maya and Leo unpack why mobile is harder than the SWE-bench everyone quotes, stage the model-capability-vs-agent-design debate (the same model swings up to six-fold across agents; simpler prompts beat elaborate ones), and land on the capstone reading: the model sets the ceiling, the agent design decides how much of it you reach — and a low score on the real job is the first honest measurement, not a verdict that agents are useless.
Why the deliverable for a model-training team is a replayable work record, not a pile of code
A topic-overview map for the unglamorous question every model-training team eventually faces: when a coding agent finishes a run, what do you actually write down? Maya and Leo argue that the deliverable is not code but a replayable work record, and walk a six-drawer filing cabinet — Task, Environment, Trajectory, Verification, Failure-and-Safety, Governance — built around one running example: a single bug-fix run treated as a hospital chart. They show how the same run becomes three different products (trainable, eval-only, safety-sensitive) and stage the real expert split — public benchmarks vs. fresh private evals — as a first-person debate resolved by watching the contamination gap.
The valuable output of a coding agent isn't the patch — it's the replayable record of how the work happened. Maya and Leo open the six-drawer filing cabinet of an agentic-coding data record (task, environment, trajectory, verification, failure-and-safety, governance) through one running example: a single bug-fix run as a hospital chart. They show why failed steps are worth more than clean wins, why a passing test isn't the same claim as 'a senior would ship this,' and how the same run sorts into trainable, eval-only, or safety-sensitive products. Includes the staged debate — shared public benchmarks vs. fresh private evals — resolved by watching the gap between the two scores. The throughline: collect work traces, not just code.
Why the commit pin and the runnable environment are the two load-bearing fields in a replayable work record
A field-by-field read of the agentic-coding data record. Maya and Leo open the top three drawers of one bug-fix run — task, environment, trajectory — and argue that the two least glamorous fields, the commit pin and the runnable environment, are the load-bearing ones: freeze the world the agent saw, or every downstream replay, re-grade, and failure reproduction falls apart. Then they stage the real expert split on the verification drawer — executable tests vs. human/LLM judgment — and resolve it by recording all three signals separately so the place they disagree becomes the most valuable label in the record.
What actually has to be in a coding-agent's record so someone else can replay it six months later on a clean machine? Maya and Leo open the top three drawers of one bug-fix run — task, environment, trajectory — and make the case that the two boring fields nobody gets promoted for, the commit pin and the runnable environment, are the load-bearing ones: a patch is directions with no map unless the repo is frozen at one commit, and a written description of an environment trusts a hostile future to still install. They show the three drawers are a dependency chain, then stage the genuine expert split on verification — executable tests (deterministic, trainable) vs. human and LLM judgment (maintainability, intent) — and land on recording all three signals separately, because a 'tests-green but human-rejected' disagreement is the most valuable label in the whole record.
Why failed and recovered agent runs — labeled step by step and typed by action — are the richest training and diagnostic data in the record
What's actually inside a coding-agent trajectory, and should you keep the runs that went wrong? Maya and Leo live inside the trajectory drawer of one bug-fix run — the agent edits the wrong file, the test fails, it reads the error and recovers — and argue that the recovery is the most teachable second in the run, even though most teams archive only 'solved in two tries.' They stage the real expert split over training on failures (imitation learning imitates the flailing) versus keeping them (clean wins never show recovery), and resolve it: it's labeled failures vs. raw failures, where step-level labels and an action taxonomy turn a junkyard of broken runs into a parts catalog you can both train on and study.
The final diff is the destination; the trajectory is the drive — and you learn very different things from each. Maya and Leo open the trajectory drawer of one bug-fix run, where the agent edits the wrong file, the test fails, and it reads the error and recovers on the second try — then watch the team archive only 'solved in two tries' and throw away the single most teachable second. They take opposite sides of a genuine split: train only on clean successes (every token is a token the model might imitate, so failures teach the flailing) versus preserve failed and recovered runs (clean first-try wins show error recovery exactly zero times). The argument collapses to labeled failures vs. raw failures: keep the negatives, but only once step-level labels and an action taxonomy mark the wrong move and the recovery and type each step — search, edit, test, retry — so the recurring failure shapes (can't-find-the-file, invented API, misread test, over-editing, the same-move loop) separate into a map you can train on and query. One honest limit: a trajectory shows the plays you ran, never the better play you never called.
Why a passing test is a fact and not a verdict — and how the rich labels above the test line (code quality, review acceptance, severity, acted-on comments, post-merge regressions) are the ones training teams actually need
Once a coding agent's patch passes every test, what's left to label? Maya and Leo open the verification drawer of one bug-fix run — public tests green, hidden tests green, verifier score high — and then the senior engineer rejects it in ninety seconds: 'this works, but nobody will be able to maintain it.' That gap between the green machines and the red human is the episode. They climb the verification ladder (public tests, hidden tests, verifier score, human review), show that a review is a transcript of comments and severities rather than a yes/no, and stage the real expert split over code review: optimize for recall (catch every bug, tolerate noise) versus trust (a noisy reviewer gets muted even when it's sometimes right). The resolution: stop scoring reviewers on theoretical bug recall and score them on whether a developer actually acted on the comment — the quietest label in the drawer and the one that settles the fight.
A patch can be green on every machine and still be wrong to ship. Maya and Leo open the verification drawer of one bug-fix run — public tests, hidden tests, and the verifier model all say yes, then the senior engineer rejects it with one line about maintainability — and use that green-machine / red-human gap to unpack why training teams need labels beyond pass/fail. Green is a fact; good is a judgment. They climb the verification ladder from the weak green of public tests the agent could see, to the stronger green of hidden tests it couldn't, to the verifier score, to human review — and show that review acceptance isn't a yes/no but a transcript: comments, severities, author responses, and whether the fix got applied. Then they take opposite sides of a genuine split on what makes a reviewer good: maximize defect recall and flag everything (a missed bug is the expensive one) versus maximize precision and trust (high-volume noisy comments get the whole reviewer muted, so the real bug at comment thirty-seven goes unread). Both extremes fail, and the resolution is to change what you measure — score a reviewer on whether developers acted on its comments, the referee that settles recall versus trust and the highest-value field in the drawer. One sobering label sits above the rest: the post-merge regression, the only verdict reality gives you after every earlier signal said yes. The honest limit: the richest labels are human judgments, and human judgments are noisy — so record who said it and what happened next, never just 'good: yes.'
How the exhaust of a live coding-agent product — branches, reviews, merges, post-merge outcomes — becomes the most valuable training data you have, but only after it survives a gauntlet of privacy, secret-scan, license, and contamination gates
The best training data you could want is the run that already happened in production — a real ticket, an agent's pull request, a reviewer's two changes, a clean merge, three weeks live with no rollback — and at most companies it gets deleted at midnight. Maya and Leo follow that record out of a live coding-agent product and ask how its exhaust becomes usable training data. Telemetry hands you four labels reality wrote for free (did it merge, did the developer rewrite it, did the comments get applied, did it survive post-merge), but it is guilty until proven clean: it has to run a gauntlet of four gates — privacy, secret-scan, license, contamination — and fail any one and it is not data. Then they take opposite sides of the real split: manufacture clean synthetic tasks you fully control, or mine the radioactive-but-real production stream. The resolution is a division of labor — synthetic for volume and coverage, telemetry for distribution-truth and reality's labels — and the quiet punchline that the teams who get to learn from production are the ones who built the governance plumbing first.
Imagine the perfect training example: a real engineer's ticket, a coding agent's pull request, a human reviewer's two requested changes, a clean merge, and three weeks in production with no rollback — every label you would pay for, written by reality for free. At most companies that record is deleted at midnight because it ran in a customer's private repo. Maya and Leo use that to unpack module 5.4 — turning production telemetry into training data. Telemetry is the exhaust of a live product (branches, reviews, CI runs, merges, post-merge outcomes), and its appeal is simple: a synthetic task is something you describe, a telemetry record is something that actually happened, carrying four reality-labels nothing else can give you. But it is radioactive, so before a single record becomes data it runs a gauntlet of four gates: privacy (whose information is in it), secret-scan (a leaked credential poisons the weights and can never be un-leaked, so the whole record is quarantined), license (are we allowed to train on this code), and contamination (does it overlap the eval set and teach the model its own exam). Then they stage the genuine split — synthetic data, clean and scalable by construction, versus production telemetry, the only real distribution — and land on a division of labor: synthetic for volume and controllable coverage, telemetry for distribution-truth and the labels reality writes for free, with the quiet punchline that whether you get to use telemetry at all is decided by whether you built the privacy, secret-scan, license, and contamination plumbing before you needed it. Governance is the permission slip, not the tax. The honest limit: even cleaned telemetry is a mirror of today's product, blind to the tasks users never tried, and its free labels are still tired-human judgments — a merge can mean 'this is good' or 'it's Friday, ship it' — so never confuse 'this happened in production' with 'this was good engineering.'
Why the last drawer of the data cabinet — license, secret-scan, and private-info clearance plus the train/eval split — decides whether every benchmark number above it is trustworthy or a memorization score in a capability costume
A team's benchmark climbs past eighty, the champagne comes out — and then a skeptic finds an eval task sitting in the training data. Same repo, same bug, same fix. The model didn't learn to debug; it learned to recite. Maya and Leo open the last drawer of the data cabinet — governance — and find it doing two jobs in one coat: clearance (is this code licensed, is there private information or a live secret buried in it, each a standing flag the record carries as proof) and placement (is this a study card or a test card). Then they fight about the hard half: a random ninety-ten split is the bug, because cards from one repository share DNA and old-versus-new snapshots leak a codebase's future — but holding whole repos out for eval can starve a small team of its best training data. They land on a real resolution: keep your hardest repos in training, seal a separate small private eval up front, keep it fresh by harvesting new held-out tasks, and plant canaries to catch the leaks you missed. The punchline that stings: the honest team posts a lower number than the sloppy one and loses the demo — until production, where a memorization score evaporates and a real one holds. A benchmark is only worth as much as the wall between training and test.
A team trains an agent on its full data lake, the benchmark climbs past eighty, and then someone pulls one eval task and finds it sitting in the training data — same repo, same bug, same fix. The score wasn't capability. It was memorization in a costume. Maya and Leo use that to open the sixth and last drawer of the agentic-coding record card — governance — and module 5.5 of the curriculum: governance and splits. The drawer does two jobs. Clearance asks whether a record may exist in the corpus at all: is the code licensed for training, is a real person's data in it, is a live credential buried in a log line (and the brutal rule — a secret fails the whole record, not the line, because a model has no delete key), each check stamped as a standing flag so provenance travels with the data. Placement asks which pile a cleared record belongs to: train, eval-only, or safety. Then they stage the genuine fight over the train/eval split. A random shuffle is a self-inflicted lie, because the unit isn't independent — near-duplicate cards from one repository and old-versus-new code snapshots let the model recite answers it studied; you cannot half-leak a repository. But a strict structural holdout can yank a small team's hardest, richest repos out of training to protect a benchmark, a purity tax not everyone can pay. The resolution is real: keep the hard repos in training, designate a separate small private eval up front that nothing in training has touched, keep it honest with freshness (continuously harvest new held-out tasks created after the latest training cut), and audit the wall with canaries — uniquely planted markers that trip when the model emits something it could only have learned from the sealed eval. The honest limits land hard: perfect separation is impossible because open source is one shared corpus, so you bound contamination rather than close it; and governance costs the wrong people — the careful team posts a lower number than the sloppy one and loses the demo, until production settles it. A benchmark number is only worth as much as the wall between your training and your test.
Why a data product is a contract, not a folder
Six episodes opened the drawers; this finale asks the question that makes them worth anything — what shape must the record take? Maya and Leo separate a schema from a list: required typed fields enforced at the door, join keys that wire the six layers so cross-layer questions become one query, and one shared record that training, eval, safety, and product each read as a different view. Then they fight the real schema-design tension — rigid up-front normalization versus flexible append-as-you-learn — and land on a typed spine plus a versioned extension area. The episode closes on the limit a schema can't fix: it guarantees shape, never truth.
