T5E0 · Data Products for Agentic Coding Training

Transcript

MayaPicture a coding agent that just spent forty minutes fixing one bug. It read nine files, ran the tests four times, made a wrong guess, backed it out, and finally landed a patch that passes. The run ends. Now — somebody walks over and asks for that work. Not the patch. The work. And almost every team, when you ask that, hands you the patch and a green checkmark and shrugs.

LeoSo they throw away the other thirty-nine minutes.

MayaThey throw away the part worth keeping. The patch is the answer at the back of the textbook. The forty minutes is the worked solution — and a worked solution is what you can teach from.

LeoThat's the thing I want to nail down today, because the curriculum behind this topic is almost stubborn about it: the deliverable for a model-training team is not a collection of code files.

MayaNot even close. The line I keep coming back to from the source — collect work traces, not just code.

LeoRight.

MayaEverything in this topic is one argument: the high-value unit isn't the file you produced, it's the replayable record of how the work happened.

LeoThen orient people. Across this series we've climbed a ladder — what agentic coding is, how you evaluate it, how the harness shapes it, and last stretch, how training signals move the score. Where does today sit?

MayaToday is the receiving end. Every earlier topic produced runs — agents passing or failing, leaving a mess of logs behind. Topic five asks the question nobody upstream wanted to own: when all that finishes, what exactly do you write down, and in what shape, so a training team, an eval team, and a safety team can each use it without tripping over each other.

LeoSo less "make the agent better" and more "capture what happened well enough that someone can make it better later."

MayaThat's the turn. The earlier topics are about doing the work. This one is about filing it.

Leo{chuckle} You say "filing" like it's exciting.

MayaThe least exciting word in machine learning, and the one that decides whether a million dollars of agent runs are an asset or a landfill. Let me give you the image I want to use all episode.

LeoGo.

MayaOne finished run on one bug-fix ticket. Picture it as a single record card — an old hospital chart, except the patient is a piece of engineering work. You don't just write "recovered." You write what they came in with, what you tried, what the labs said, what went wrong, and what you're not allowed to share. Same ticket, all episode — every deep-dive zooms into a different part of that card.

LeoA chart for a bug fix. Okay, I can hold that.

MayaAnd the card has drawers — six of them, one filing cabinet. That's our map for the whole topic. Want me to walk the cabinet?

LeoWalk it. But slowly, because I already suspect the order matters.

MayaIt does, good instinct. Top drawer: the Task. What was even asked? The prompt, the task type — bug fix versus feature build — acceptance criteria, which repository, pinned to an exact commit, the language and framework. The thing people skip is the commit pin. Don't freeze the repo at the state the agent saw, and the whole record rots — you can never replay it.

LeoBecause the repo keeps moving underneath you.

MayaRight. A record you can't replay is an anecdote. Slide that one shut, open the Environment drawer. The Docker image or setup script, the locked dependency versions, the public tests the agent can see, the hidden tests it can't, and the exact command that decides pass or fail.

LeoHold on — public tests and hidden tests both live in the record?

MayaBoth. And keeping them apart inside the record is its own discipline we'll get to. Now the Trajectory drawer — and this is the one the topic is really about. Every message, every tool call, every file the agent read, every command it ran, the outputs it got back, every edit, every retry — with timestamps, in order.

LeoThat's the thirty-nine minutes.

MayaThat's the thirty-nine minutes. The other drawers describe the work; this one is the work.

LeoLet me push, though. That's a firehose. Most of a trajectory is the agent flailing — opening the wrong file, running a command that errors. Why keep the flailing?

MayaBecause the flailing is the lesson. A trajectory that goes straight to the answer teaches you almost nothing. A trajectory where the agent misreads a stack trace, edits the wrong function, watches a test stay red, and recovers — that's a recovery you can train on. Throw out the failed steps and you've kept the highlight reel and burned the game film.

LeoHm. Fair.

MayaThen the Verification drawer. Not just "passed." The public-test result, the hidden-test result, a verifier model's score if you have one, the human review, and the final outcome — did this get merged, rejected, or edited before merge.

LeoThat's more than a checkmark.

MayaA checkmark is one bit. This drawer is the difference between "tests went green" and "a senior engineer would actually ship this." Those are not the same claim, and a training team that treats them as the same trains an agent to chase green, not to chase good.

LeoSay that one again, slower — chase green versus chase good.

MayaA test passing tells you the code does a thing. Human review tells you it does the right thing, in a way someone will maintain. If your only label is the test, your agent learns to satisfy tests — which it can do in some deeply cursed ways. That's its own later episode.

LeoThat's the reward-hacking ghost. Noted for later.

MayaNext to it, the Failure-and-Safety drawer. A taxonomy, not a feeling. Did it fix the wrong place? Hallucinate an API that doesn't exist? Write an over-broad patch that touches forty files to fix one? Game the test? Run an unsafe command? Each is a named failure mode, tagged on the record.

LeoSo "it failed" becomes "it failed this specific way."

MayaWhich is the only kind of failure you can fix. "It failed" is a shrug. "It hallucinated a method on a library it never imported" is a curriculum.

Leo{chuckle} The failures are the syllabus.

MayaAnd the bottom drawer — the one everybody discovers exists only after it bites them: Governance. The license on that code. A scan for personal information. A scan for secrets — keys, passwords, tokens the agent might've seen. The train-versus-eval split decision. A contamination flag. Any restriction on how the data can be used.

LeoThe legal-and-trust drawer.

MayaThe drawer that decides whether the other five are usable at all. A gorgeous trajectory sitting on top of a leaked customer secret is a radioactive record. Six drawers — Task, Environment, Trajectory, Verification, Failure-and-Safety, Governance. That's the cabinet. That's Topic five.

LeoLet me say the cabinet back in one breath, because I want the listener holding it. What was asked, where it ran, what happened step by step, whether it was actually good, how it broke if it broke, and whether you're even allowed to use it.

MayaThat's it. That's the whole record card.

LeoNow here's where I get suspicious. This sounds like the schema designer's dream — every field, every run, forever. Nobody can afford that. So which drawers are non-negotiable and which are nice-to-have?

MayaGood — and the honest answer is it depends on what the record is for, and that's actually the deepest idea in the topic. The same run can become three completely different products depending on which bin you sort it into.

LeoTell me the bins.

MayaThe Trainable bin: records you'll learn weights from — fine-tuning, reward models, reinforcement learning. The Eval-Only bin: records you use to measure the agent and must never, ever train on. And the Safety bin — secrets, private code, exploit trajectories that need locked handling.

LeoAnd the same bug-fix run could land in any of them.

MayaThe same run, sorted three ways, becomes three products. And the cardinal sin of this whole field is mixing the Trainable and Eval-Only bins.

LeoRight, because if your eval data leaks into your training data —

Maya— then your benchmark score is a lie. You're testing the agent on questions it already saw the answers to. It looks brilliant and it's memorized the exam.

LeoYeah.

MayaContamination. It's the quiet killer in this space, and it lives entirely in that governance drawer.

LeoOkay. This is the part I actually wanted to fight about — a real disagreement among serious people. And it's not the synthetic-versus-real-tasks fight; we did that last topic. It's downstream of it.

MayaStage it. Pick your side.

LeoThe split is: where should a team's evaluation signal come from? One camp says lean on shared public benchmarks. The other says you have to build fresh, private evals you keep sealed. I'll take public benchmarks. You take private.

MayaDeal. Make your case. Sharpest form.

LeoPublic benchmarks are how the field can even talk to itself. SWE-bench, the verified set — when everyone reports on the same tasks, I can compare your agent to mine to last year's. The second you go private, every team's "we hit sixty percent" means nothing, because nobody can check it. A number nobody else can reproduce is marketing, not science.

MayaAnd the moment a benchmark is public and popular, it's contaminated. The tasks are on GitHub; they're in the next training crawl. Teams — sometimes without meaning to — tune toward the public set, because that's the number on the leaderboard. So your shared yardstick is being sawed shorter every quarter, and everyone's "sixty percent" is partly memorization you can't see.

LeoThat's a real hit. But — fine, the latency of contamination is real, except you're describing a reason to refresh public benchmarks, not abandon them. Build SWE-bench-Live, keep pulling fresh issues, canary the old ones.

MayaI'll give you that — refreshing helps, and the canary idea is exactly right. But here's where my side doesn't move: for the decisions that matter to your business, you need an eval on your code, your stack, your failure modes — and that one has to be private and fresh, because it's worth gaming and nobody else can be trusted to keep it clean.

LeoHm.

MayaThe public benchmark tells you how you stack up against the field. The private eval tells you whether the thing actually works on the software you ship. Those answer different questions.

LeoOkay, I'll concede the shape of it. Public for comparability, private-and-fresh for the decisions that bite — contamination governance the referee between them. And the thing that settles the rest is measurable: track the gap between an agent's public-benchmark score and its score on a freshly-pulled private set. If that gap keeps widening, the public number is rotting.

MayaWatch the gap. That's the instrument. And notice — that whole fight lives in two drawers: the split decision and the contamination flag down in governance, plus the freshness of the task up top. The debate isn't abstract. It's a data-modeling choice.

LeoWhich is the thing I keep underestimating. It feels like plumbing and it's where the science is kept honest or broken.

MayaLet me make it concrete one more way, on our one card. Take that bug-fix run. As trainable data, you keep the trajectory and the recovery and strip the hidden tests so the agent can't memorize them. As eval-only data, you freeze the whole thing, hide the trajectory, and never let a single byte near a training run. As safety data, you pull it entirely if the secret scanner found a live key in one of those command outputs.

LeoOne run. Three completely different handling rules.

MayaOne run, three products, three sets of rules. That's the craft of this topic — not collecting more, but sorting right.

LeoLet's preview where the deep-dives go, quickly, so people know the trip.

MayaWe open the cabinet drawer by drawer. One episode on the full record — what it contains and why the commit pin and the executable environment are load-bearing. Then the trajectory drawer — why the failed runs, the retries, the unsafe moves, are worth more than the clean wins.

LeoThe game film over the highlight reel.

MayaThen the verification drawer — the labels past pass-fail: quality, maintainability, which review comments got accepted, post-merge regressions. Then the bridge from live product telemetry into training data, and the privacy and licensing gauntlet it has to survive.

LeoAnd governance gets its own episode, I'm guessing.

MayaIt earns one — train-eval separation, secret and PII scanning, canaries. And we close on the schema itself, the six layers as one thing you could hand a team.

LeoGood map.

MayaBefore we go, let me leave the listener a small dictionary, because a few of these words do real work and people fake knowing them. Quick and clean.

LeoFire them at me, I'll keep the answers honest.

MayaA trajectory is the full ordered record of one attempt — every message, tool call, command, output, edit, and retry — not just the final patch.

LeoA hidden test is a check the agent can't see while it works, so it can't tune its patch to pass it — which is what makes the verification trustworthy.

MayaA verifier is whatever decides if the work is good — an executable test, or a model trained to judge — handing back a signal you can learn from or grade with.

LeoContamination is when evaluation data leaks into training data, so the agent has effectively seen the exam — and the benchmark score stops meaning anything.

MayaA train-eval split is the up-front decision about which records you're allowed to learn from and which you've sealed off purely to measure with.

LeoA canary is a marked task you plant so you can later detect whether it leaked into someone's training set.

MayaThat's the kit. And most of this topic is just using those words carefully.

LeoSo here's what I'll sit with. The whole series, we treated the agent as the thing you improve. Today flips it — the record of the agent's work is the asset, and the agent is almost a byproduct of how well you filed it.

MayaSo here's the question to carry into the rest of the topic. Think about the last time one of your systems did something genuinely useful and you didn't write down how it happened. If you had to learn from that moment now — which drawer of the record did you forget to keep open?