T5E1 · The Agentic Coding Data Record

Transcript

MayaA team hands me a finished record of an agent fixing a bug. Clean prompt, the patch, a green pass. I try to re-run it — and the build won't even start. The library it imported moved on. The repo it edited had thirty commits land since. The test that went green? Gone, renamed. So I'm holding a photo of a meal I can't cook.

LeoA record you can't replay.

MayaWhich the last conversation already told us is just an anecdote. Today I want to make that brutal — what specifically has to be in the record so it survives contact with a second person, six months later, on a different machine.

LeoLast time we opened the whole filing cabinet — six drawers, the high-altitude tour. Task, environment, trajectory, verification, failure-and-safety, governance.

MayaThe map.

LeoRight, the map. So today's the opposite move — we stop pointing at drawers and we open the top three and actually read what's filed inside. The fields. The ones that look like boring metadata and quietly decide whether any of it works.

MayaThat's the episode. And I'll warn you up front — the two least glamorous fields in the whole record are the two that carry it. I want to earn that claim.

LeoThen start where the agent starts. Same bug-fix run we've been carrying — one ticket, one card. What's the very first thing written down?

MayaThe task layer. And it's tempting to think that's just "the prompt" — what the human asked. It's more. It's the prompt, yes, but also the task type — is this a bug fix or a feature build, because you grade those differently. The acceptance criteria — what counts as done. Which repository. And then the field everyone treats as a footnote.

LeoThe commit.

MayaThe commit. The exact snapshot of the repo, pinned to one specific point in its history. Not "the repo." The repo as it was the instant the agent looked at it.

LeoOkay, push on that for me, because to a listener it sounds pedantic. Repo's the repo. Why does the exact snapshot earn top billing over, I don't know, the actual patch?

MayaBecause the patch only means anything relative to the snapshot. Picture it — the agent changes line two hundred of a file. Six months later that file's been refactored, line two hundred is a comment now. Apply the same patch and you either get garbage or a merge conflict. The diff is an instruction that only makes sense against the exact starting state it was written for.

LeoSo without the pin, the patch is a set of directions with no map.

MayaThat's exactly it. Turn left at the gas station — when the gas station's been gone for a year. The commit pin is what freezes the gas station in place.

LeoHuh. Okay.

MayaAnd here's the part that flips it from pedantic to load-bearing. The pin isn't for the patch. It's for everything downstream of the record. You can't re-run the trajectory without it. You can't re-grade the verification without it. You can't reproduce a failure to study it without it. One missing field upstream and the entire rest of the card becomes un-replayable. It's a load-bearing wall.

LeoFirst of my two boring heroes. What's the second?

MayaThe second is the whole second drawer. The environment. And this is the one teams get most wrong, because they write down the names of things instead of the things.

LeoMeaning what — they note "Python, version three-eleven" and call it environment?

MayaThat's the trap exactly. They record a description of the environment. What you actually need is a runnable environment — the Docker image or the setup script that rebuilds the exact box the agent worked in, the locked dependency versions down to the patch number, and the precise command that decides pass or fail.

LeoLocked versions — meaning not "we used the testing library," but the exact build of it.

MayaDown to the digit. Because "we used version two-point-something" is how you get a record that passed on Tuesday and fails on Thursday for reasons that have nothing to do with the code. A description tells you about the environment. A runnable environment is the environment.

LeoLet me steel-man the lazy version, though, because somebody's thinking it. Disk is cheap, but rebuilding a full container for every single run is real engineering cost. Why not just snapshot the package list and reconstruct when you actually need it?

MayaFair, and people do exactly that and get burned. Because "reconstruct later" assumes every dependency you listed still exists, still installs, still resolves the same way. Half of "later" is a package that got yanked, a registry that changed defaults, a transitive dependency that bumped a version. The whole point of the record is that later is hostile. A description trusts the future to cooperate. A runnable environment doesn't have to.

LeoSo the commit pin and the runnable environment — your two boring heroes — they're really the same idea twice.

MayaSay it.

LeoFreeze the world the agent saw. The code at one instant, the machine at one instant. Everything else in the record is a story about that frozen world, and if the world isn't frozen the story doesn't parse.

MayaThat's better than how I said it. Yeah. Replayability isn't a nice property you add at the end. It's two fields you either captured at the start or you didn't.

LeoOkay. We've got the world frozen. Now the agent actually does the work. Third drawer.

MayaThe trajectory. Every message, every tool call, every file it read, every command it ran, the output it got back, every edit, every retry — timestamped, in order. We spent real time on this last episode, so I won't relitigate why you keep the failures —

Leo— the game film, not the highlight reel. Keep the flailing because the recovery is the lesson. I've got it.

MayaRight. But there's one thing about the trajectory I want to add today that the overview skipped, because it lives in the record's structure, not its philosophy. The trajectory has to be aligned to the environment. Every command output in the trajectory was produced by a specific environment state. If you store the trajectory but lose which environment produced it, you've got a transcript of a conversation where you can't hear one of the speakers.

LeoOof.

MayaSo these three drawers aren't independent files. The task pins the world, the environment makes the world runnable, the trajectory is the recording of what happened in that world. Pull any one and the other two lose meaning. That's why the order matters — it's not filing tidiness, it's a dependency chain.

LeoLet me say the chain back, slow, because I think it's the spine of the whole record. The task says what and where — pinned. The environment says the exact machine — runnable. The trajectory says what happened, step by step — aligned to that machine. Three drawers, one replayable unit.

MayaOne replayable unit. And now we hit the drawer where serious people actually fight. Verification.

LeoThis is the one I've been waiting for, because I don't think it's settled at all.

MayaIt isn't. And it's worth saying where this whole record shape comes from — the benchmark work behind today's episode, the SWE-bench line of it, is exactly this: pin a real repo to one commit, ship it with hidden tests against a runnable environment, and see if an agent can resolve the issue. That's the card we've been describing, drawn from a real source.

LeoSo the drawers aren't somebody's whiteboard taxonomy. They're how the actual evaluations are built.

MayaRight. And it's a different fight than last time — last episode we argued about where your eval data comes from, public versus private. This is upstream of that. This is: once you have the run, what decides whether it was good? What signal do you even write into the verification drawer?

LeoAnd there are two camps, and they don't fully get along. Pick sides — I'll take the one I actually believe.

MayaGo. Strongest form.

LeoExecutable tests. Full stop, where you can. A test is a fact. It runs, it's deterministic, it passes or it fails, and it'll give you the same answer at three in the morning a year from now. You can train on it, you can grade on it, you can put it in a reward loop and trust the loop.

MayaThat's the strong case, yeah.

LeoThe whole reason agentic coding is even trainable is that code can check itself. The second you replace a test with a judgment, you've smuggled in something you can't reproduce.

MayaAnd a test only checks what someone thought to test. I'll give you deterministic — but deterministic isn't the same as complete. A patch can pass every test and be a maintenance disaster. It hardcodes a value that happens to satisfy the assertion. It deletes the failing test and writes a trivial one. The tests go green and a senior engineer would reject it on sight — and your record, if tests are the only signal, files that run as a success.

LeoThen write more tests. The failure mode you're describing is a weak test suite, not a reason to abandon tests for a model that hallucinates whether code is good.

MayaYou can't test your way to "is this maintainable." You can't write an assertion for "did this match the intent of the ticket." Some of what the verification drawer needs to hold is genuinely judgment — code quality, review-readiness, whether a human would actually merge it. That's not a gap in your test suite. It's a different kind of question, and pretending a test answers it is how you train an agent to game the test.

LeoHm.

MayaBut — let me concede the real thing. You're right that a model verifier you can't reproduce is dangerous. A judge that scores the same patch differently on Tuesday and Thursday is worse than no judge.

LeoAnd I'll give you mine. There's a class of questions a test genuinely can't reach — intent, maintainability, "would a reviewer trust this." I can't write `assert maintainable` and mean it. So it's not tests or judgment.

MayaIt's tests for what tests are good at, judgment for what only judgment can reach — and the record holds both signals, kept separate and labeled by source. The hidden-test result is one field. The verifier's score is a different field. The human reviewer's decision is a third. You never collapse them into one "good: yes."

LeoBecause the moment you collapse them, you lose the ability to ask which signal was wrong when the agent ships a bad patch that passed.

MayaThat's the resolution. Don't pick a verifier — record all three, never blend them. And here's the thing that settles which one to trust on a given run: watch where they disagree. When the tests pass but the human rejected, that gap is the most valuable label in the whole record. That's a test that's lying to you, caught.

LeoThe disagreement is the signal.

MayaThe disagreement is the signal. A run where tests, verifier, and human all agree teaches you a little. A run where they split teaches you exactly where your cheapest signal is failing.

LeoOkay, that actually reframes the whole drawer for me. The verification layer isn't "the grade." It's three graders' answers stapled together so you can audit them against each other.

MayaAnd that's the difference between a checkmark and a record. A checkmark is one number that's already thrown away the disagreement. A record keeps the disagreement on purpose.

LeoSo pull it together. We've opened — what — three and a half drawers, really deeply. Where does that leave the listener?

MayaWith the load-bearing parts named. The commit pin and the runnable environment freeze the world, so everything's replayable. The trajectory records what happened in that world, aligned so it's still readable later. And the verification drawer holds multiple signals side by side instead of one blended verdict, so you can tell which signal lied.

LeoAnd the two drawers we haven't opened —

Maya— failure-and-safety, and governance — each earn their own episode later. Today was the engine of the record. The fields that, if you skip them, mean none of the rest can be trusted.

LeoI keep landing on the same uncomfortable thing. The most valuable fields are the ones with the lowest status. Nobody gets promoted for pinning a commit.

MayaNobody's ever been thanked for a Dockerfile. [chuckle] And yet.

LeoAnd yet it's the whole record.

MayaSo here's what I'd leave the listener sitting with. Picture the last debugging session you'd genuinely want to learn from — a real one, a hard one. If you handed someone only what you wrote down, could they re-run it from scratch on a clean machine?