T5E2 · Trajectory Data for Training and Analysis

Transcript

MayaHere's a thing that happened on a real run. The agent's first move is wrong — it edits the wrong file. Test fails. So it reads the error, backs out the change, searches again, finds the actual file, and the second edit passes. Now the team archives that run. And the way they archive it — they keep the final diff and a note that says "solved in two tries."

LeoTwo tries. So they kept the score and threw away the play-by-play.

MayaThey kept the outcome. What they threw away is the single most teachable second in the whole run — the moment the agent looked at a failing test and figured out it was in the wrong place. That recovery is the lesson. And "solved in two tries" can't teach it to anyone.

LeoLast episode we opened the top three drawers — task, environment, trajectory — and you made the case that the commit pin and the runnable environment are what make the whole thing replayable.

MayaThe boring heroes. Right.

LeoSo today we live inside one of those drawers. The trajectory. Not "is it recorded" — we settled that — but what's actually in a trajectory that makes it worth training on, and what most teams quietly lose.

MayaThat's the episode. And the twist I want to land is going to sound backwards at first. The failures aren't the noise around the signal. For a lot of training, the failures are the signal.

LeoOkay, but ground me first. What is a trajectory, concretely, for someone half-listening on a walk?

MayaPicture the agent narrating its own work as it goes. "I'm reading these three files. I'm running this command. Here's what came back. Okay, that's an error — let me try this instead." Every message, every tool call, every file it opened, every command, the output it got, every edit, every retry — in order, with timestamps. The trajectory is the agent thinking out loud, written down.

LeoSo the diff is the destination. The trajectory is the drive.

MayaAnd you learn very different things from each. The diff tells you where it ended up.

LeoMm-hm.

MayaThe trajectory tells you how it drove — did it take the highway or wander every side street, did it read the map before turning, did it notice when it was lost.

LeoAnd "did it notice when it was lost" is the one nobody captures.

MayaThat's the one teams underrate — because it only shows up in the runs that went wrong.

LeoLet's go straight at that, then, because it's the counterintuitive part. The instinct is obvious — you want to train on your best work. Keep the clean runs, the ones that nailed it first try, and throw out the messy ones. Why is that instinct wrong?

MayaBecause the clean runs only teach the agent what success looks like when nothing goes wrong. And in real engineering, something always goes wrong. The test fails. The API you assumed doesn't exist. The fix breaks two other things. If every example the model ever sees is a frictionless first-try win, you've trained an agent that has never once watched anyone recover — and the entire job is recovery.

LeoHold on, let me push, because I think there's a real counter here and I half-believe it. If I feed the model a thousand runs full of wrong turns and dead ends, aren't I also teaching it the wrong turns? I'm showing it bad moves. Some of that has to stick.

MayaThat is exactly the live disagreement. And it's not a strawman — serious people sit on both sides of it. So let's actually have it. Pick your side.

LeoFine. I'll take it. Train on clean successes. My case is simple — every token you put in front of the model is a token it might imitate. A failed trajectory is a transcript of an agent doing the wrong thing for nine steps before it stumbles into the right one. Reward the whole thing and you've just told the model that nine steps of flailing is an acceptable way to work. Imitation learning imitates. I don't want it learning the flail.

MayaBut you're imagining we keep the failure as if it succeeded. Nobody serious does that. The point isn't to reward the flailing — it's to label it. The failed run isn't one undifferentiated blob you bless or burn. It's a sequence of steps, and you can score them step by step.

LeoScore them how, though?

MayaThe wrong-file edit gets marked as the wrong move. The moment it read the error and self-corrected gets marked as the good move. You're not training the agent to flail. You're training it to do what the recovery step did.

LeoThat assumes you actually have step-level labels. Most teams don't. They have a pass-fail at the end and nothing in the middle. So in practice — in the messy real world where nobody labeled step seven — feeding in raw failures is feeding in noise.

MayaHm. Okay.

MayaThat's a fair hit, and I'll give it to you. Without step-level structure, a pile of failed trajectories is closer to noise than signal. You're right about the default case.

LeoAnd I'll give you mine — because I don't actually believe in throwing the failures away. A failed run where the agent recovered is, if you can label the turn, the most concentrated lesson you'll ever get. A clean first-try win shows error recovery exactly zero times. I just want the label before I want the data.

MayaSo we're closer than it looked. It's not failures versus successes. It's labeled failures versus raw failures. The negative trajectory is gold — but only once you've marked where it turned.

LeoAnd the way you settle which camp is right on a given dataset —

Maya— is whether anyone did the labeling. That's the whole thing. The disagreement collapses the moment you ask "are these step-labeled?" If yes, keep the failures, they're the richest data you've got. If no, you're shipping noise and calling it diversity.

LeoOkay. That reframes it for me. So step labels are the hinge. What do they actually look like? Because "good move, bad move" sounds too simple to be a real schema.

MayaIt is too simple, and that's where the action taxonomy comes in. Instead of a thumbs up or down, you tag what kind of move each step was. Was this a search? A file read? An edit? A test run? A retry after a failure? You build a small vocabulary of action types, and every step in every trajectory gets stamped with one.

LeoGive me why that's worth the effort. Tagging every step sounds like a lot of clipboard work.

MayaBecause once the steps are typed, the failures stop being anecdotes and become a map of how agents fail. You can suddenly ask questions across thousands of runs. The curriculum actually names the recurring failure shapes, and they're worth hearing because each one is a different disease.

LeoName them. But not as a list — I'll lose them.

MayaThen let me give you the rogues' gallery. There's the agent that can't find the right file — it edits confidently in the wrong place. There's the one that invents an API — calls a function that simply doesn't exist, because it sounded plausible. There's the one that misreads a failing test — the test is telling it one thing and it acts on another.

LeoThose are three distinct diseases already. Keep going.

MayaAnd there's the over-editor — asked to fix one line, it rewrites half the module. And there's the one that just loops — same failed move, again, again, again, never updating.

LeoOof. I've watched that last one live.

MayaEveryone has. And here's why the taxonomy earns its keep — those are five completely different problems with five different fixes. Can't-find-the-file is a search-and-navigation weakness. Invents-an-API is a grounding problem. The loop is a stuck-state problem. If all you have is "it failed," they're indistinguishable. If your steps are typed, they separate cleanly, and now your failure data is a diagnosis instead of a complaint.

LeoSo the action taxonomy is what turns a junkyard of broken runs into a parts catalog.

MayaThat's the line. A junkyard you walk past. A parts catalog you shop from. Same broken cars. The difference is whether anyone sorted them.

LeoLet me make sure I've got the spine, because this is where I'd lose a listener. The trajectory is the recording of the drive, not just the destination. Failed drives are worth keeping — but only after you've labeled them step by step, and typed each step with an action taxonomy. Do that, and the failures become a map of how agents break, which is exactly what you need to make them break less.

MayaThat's the whole drawer. And it loops right back to where we started — "solved in two tries" threw all of that away. Two tries is the score. The trajectory is the game film. And you don't get better by watching the score.

LeoThere's a second use I want to make sure we name, though, because it's not all training. You keep saying analysis.

MayaRight — and this is the title's other half. Trajectory data isn't only for teaching the model. It's for understanding it. The exact same step-labeled, action-typed trajectory that you train on is also the thing a researcher reads to answer "why did our agents regress this month?" or "where in the run do they actually waste time?"

LeoOne artifact, two jobs. Train on it, and study it.

MayaAnd that's why the structure matters so much. A trajectory you can replay and a trajectory you can query — those are the same trajectory, just well-labeled.

LeoHuh.

MayaSkip the labels and you can do neither well.

LeoOkay, one honest limitation before we close, because we've been selling the failures hard. Where does this break?

MayaIt breaks on cost and on honesty. Step-level labeling is genuinely expensive — somebody, or some trusted automatic process, has to look at every turn and judge it, and that doesn't scale for free. And there's a subtler trap. A trajectory shows you the steps the agent took. It doesn't show you the better step it never considered. So you can study failures all day and still be blind to the move that was never on the table.

LeoThe film shows every play you ran. It can't show the play you didn't call.

MayaThat's the honest edge of it. Trajectory data makes the agent's actual process legible — its searches, its errors, its recoveries, its loops. It does not, on its own, tell you what a better agent would have done instead. That's a different drawer, on a different day.

LeoSo here's what I'd hand the listener to chew on.

MayaGo.

LeoThink about the last time you fixed something hard — code, a config, anything. You worked through the dead ends and recovered. So if the recovery is the lesson — why do we keep throwing the recordings away?