T5E4 · From Production Telemetry to Training Data

Transcript

MayaPicture the single best training example you could possibly want. A real engineer opened a real ticket. A coding agent took it, worked the repo, opened a pull request. A human reviewer read it, asked for two changes, the agent made them, the reviewer approved, it merged — and three weeks later it's still running in production, no rollback, no regression. That is a complete, reality-graded record of good engineering work.

LeoThe dream record, right there. Every label you'd pay for, and reality wrote them for free.

MayaFor free. And here's the part that should sting. At most companies, that exact record gets deleted at midnight. Because it ran inside a customer's private repo, and the session logs are on a retention timer, and nobody ever decided to keep it.

LeoSo the best data you'll ever have is the data you're already generating and throwing away.

MayaAnd there's the whole episode. The gold isn't in a dataset you go buy. It's in your own product's exhaust — and the question is whether you can catch it without catching everything that makes it radioactive.

LeoLast time we climbed the verification drawer — public tests, hidden tests, verifier score, human review — and you kept hammering that the richest label was the quietest one, whether a developer acted on a comment.

MayaThe labels above the test line. Judgments, not facts.

LeoRight. And every one of those labels we discussed, we sort of assumed someone created it on purpose, in a lab, for the data. Today's flip is — what if the people creating those labels aren't doing it for you at all? They're just... working.

MayaExactly the turn. The curriculum module behind today is production telemetry to training data. And telemetry is a fancy word for one plain idea — the trail of what actually happened when real people used the real product. Not a benchmark someone built. The exhaust of normal work.

LeoDefine it cleanly for me, because "telemetry" gets used to mean everything from a crash log to a click counter.

MayaFair. In our world, telemetry is the byproduct stream from a live coding-agent product. A developer files a task. The agent spins up a branch, reads files, runs commands, writes a patch, opens a pull request. People comment on it. Tests run in CI.

LeoSo all the events the product emits just by operating.

MayaExactly. Someone merges it or closes it. Weeks later it either holds up or it breaks. Every one of those is an event the product already records to do its job — and stacked together, they're a trajectory and a verification record that nobody had to hand-build.

LeoSo it's the same record card we've been carrying all topic — task, environment, trajectory, verification — except—

Maya—except you didn't author it. The product authored it by running.

LeoHuh.

MayaAnd that's the appeal in one line. A synthetic task is something you describe. A telemetry record is something that actually happened.

LeoOkay, but I want to push on "happened," because production is messy. A benchmark task is clean — one problem, known answer, isolated environment. A production session is a human getting interrupted, abandoning the branch, coming back, pulling in a colleague, force-pushing over their own history. Is that even usable, or is it just noise with timestamps?

MayaIt's both, and separating them is most of the work. But before we get to the cleanup — let's be precise about what's so valuable that it's worth the mess. Because there's a specific category of label here you cannot get any other way.

LeoGo.

MayaReality's labels. Think about what production hands you that a lab can't. Did the pull request actually merge — a real maintainer decided. Did the developer accept the agent's patch as-is, or rewrite half of it — that rewrite is a free correction signal. Did the reviewer's comments get applied. And the big one — post-merge, did it survive, or did it page someone. Four labels, and a human or the world assigned every one of them as a side effect of just doing the job.

LeoSo you're not even paying labelers. The labels are exhaust too.

MayaHere's the magic and the trap in the same sentence. The labels are exhaust — which means they're attached to real people's real private work. And that's where this stops being a data problem and becomes a governance problem.

LeoThis is the part I actually wanted to get to, because I think it's where teams either do it right or quietly create a disaster. Walk me through it.

MayaSo picture telemetry as raw water coming off the floor of the product. Before a single drop becomes training data, it has to pass through a gauntlet — and if it fails any one stage, it doesn't get to be data. Let me give you the gates.

LeoWalk me through them. I'll hold each one.

MayaStart with privacy, because it's the first thing that should stop you. That session ran inside someone's repository. There's customer code in it, maybe customer data in the test fixtures, maybe a developer's name and email in a commit. None of that can flow into a model that other customers will use. So the privacy gate asks — whose information is in this record, and do we have any right to learn from it.

LeoRight.

MayaThen there's secrets — and this one's sharper than privacy, because it's not about consent, it's about poison. A real session can contain an API key someone pasted into a config, a database password in an environment variable, a token in a log line. If that flows into training, you've built a model that has, somewhere in its weights, a memory of a live credential. The secret-scan gate exists to catch those and quarantine the whole record.

LeoAnd "quarantine the whole record," not "delete the one line" — because if the model ever regurgitates it, you've leaked a customer's production key through your weights.

MayaYou've leaked it and you can't un-leak it. A model doesn't have a delete key. That's why secrets fail the record, not just redact a field.

LeoOkay. Privacy, secrets. Keep going.

MayaLicensing's next, and it's the unglamorous one. The code in that repo came with terms. Some licenses are fine to learn from, some are explicitly not, some are ambiguous enough that your lawyers will want a say. The license gate asks — are we even allowed to train on this material. It's not glamorous, but it's the gate that keeps the whole dataset legally usable instead of a liability someone discovers in two years.

LeoThe boring gate that saves the company. Fine. What's the last one?

MayaThe subtle one — the one people forget until it's burned them. Contamination. Some of the tasks flowing through your production product are going to overlap with your evaluation benchmarks — the same repositories, the same issues, sometimes the literal same problems that are sitting in your test set. If you let those into training, your model has now seen the answers to its own exam.

LeoOh — so your eval scores go up and you've learned nothing. You just taught to the test through the back door.

MayaThrough the back door, by accident, because the telemetry didn't know it was contaminated. So the contamination gate has to check every incoming record against the eval sets and pull anything that overlaps. Miss it, and every benchmark number you report afterward is a lie you're telling yourself.

LeoLet me say the gauntlet back, because this is the spine. Raw telemetry comes off the product, and before it's data it runs four gates — privacy, who's in it; secrets, what poison is in it; license, are we allowed; contamination, does it leak the exam. Fail any one, it's not data.

MayaAnd the order matters less than the principle — telemetry is guilty until proven clean. The default for a production record is "do not train on this." It earns its way in.

LeoThat reframes the whole thing. I came in thinking the hard part was collecting the data. You're saying collection is easy — the product's already doing it — and the hard part is the permission to use what you collected.

MayaYou've put your finger on the inversion at the heart of the module. And it sets up the real fight, because once governance is hard, a serious camp says — why fight it at all?

LeoYeah, let me take that side, because I think it's stronger than you're going to want it to be. If telemetry is this radioactive — private code, secrets, license traps, contamination — then synthetic data wins on the merits.

MayaOn what merits, specifically?

LeoYou generate tasks from open repos you fully control. You construct the environments. You write the tests. No customer's information is ever in it, so privacy is a non-issue, secrets can't leak because there are none, licensing is clean by construction, and contamination is controllable because you know exactly what you made.

MayaAnd the cost of all that control?

LeoThere isn't one — that's the point. Plus you can scale it to a million tasks and dial the difficulty. Why would I take on a legal and security nightmare when I can manufacture clean data instead?

MayaBecause clean data is also fake data, and the gap shows up exactly where it costs you. Synthetic tasks are generated from your assumptions about what engineering work looks like. Production telemetry is what engineering work actually is — including the parts you'd never think to synthesize.

LeoLike what, concretely?

MayaThe weird half-specified ticket. The bug that only reproduces with three services running. The reviewer who rejects a technically-correct patch for a reason no rule would generate. You can't manufacture the real distribution. You can only sample it.

LeoBut a lot of that "real distribution" is noise. You said it yourself — abandoned branches, interruptions, force-pushes. I can generate a clean signal at volume. You're handing me a messy signal I have to spend a fortune cleaning, and it might still be contaminated.

MayaAnd your clean signal is confidently teaching the model a world that doesn't exist. Here's the tell — synthetic data is great at the tasks you knew to generate, and blind to the ones you didn't. It can't surprise you. Production telemetry surprises you constantly, and the surprises are the whole point, because that's where your model is actually failing in the wild.

LeoOkay. The surprises are the part I can't manufacture. I'll give you that one fully — synthetic can't show me a failure mode I didn't already imagine.

MayaAnd I'll give you yours, because it's real. For raw volume, for coverage of cases that are rare in production, for anything where the privacy and license cost isn't worth it — synthetic wins, easily. I'm not arguing you stop generating. I'm arguing you can't only generate.

LeoSo it's not human-or-synthetic. It's synthetic for the volume and the coverage you can control, telemetry for the distribution-truth and the labels reality writes for free.

MayaYou just said the resolution, and there's a clean division of labor in it. Synthetic answers "can the model do this kind of task at all" — breadth, at scale, safely. Telemetry answers "is the model actually working for real people on real repos" — and it carries the four reality-labels nothing else can give you. One is your training volume. The other is your ground truth.

LeoAnd the thing that decides whether telemetry is even on the table isn't the model team at all. It's whether the governance gauntlet exists.

MayaAnd there's the quiet punchline. The teams that get to learn from production aren't the ones with the best models. They're the ones who built the privacy, secret-scan, license, and contamination plumbing before they needed it. Governance isn't the tax on using telemetry. Governance is the permission slip.

LeoSay more on that, because it sounds like a slogan and I want it to be a mechanism.

MayaIt's a mechanism. If you haven't built secret scanning, you literally cannot touch production logs without risking a credential in your weights — so you don't touch them, and that whole gold stream is off-limits to you. If you haven't built contamination checking, every model you train on telemetry has uncertain eval numbers, so you can't trust your own results. The plumbing isn't downstream of the data decision. It's upstream. No gauntlet, no telemetry, full stop.

LeoSo the honest limitation, then — because we've made telemetry sound like the prize. Where does it actually let you down?

MayaTwo places, and they're real. First, even after the gauntlet, telemetry is biased toward your current product's users and your current agent's behavior. It tells you how today's agent fails — it can't tell you about the tasks your users never even tried because the agent's bad at them. It's a mirror of the present, not a map of the possible.

LeoSo it makes the agent better at what it already sort of does, and stays blind to what it never attempts.

MayaRight — the trap of learning only from your own exhaust, you optimize the world you already have. And second — every reality-label we got excited about is also a judgment made by a tired human in a hurry. A merge can mean "this is good" or it can mean "it's Friday and the tests pass, ship it." The labels are free, but free doesn't mean clean. You still have to treat a production merge as a noisy signal, not gospel.

LeoSo the data that reality labeled for free still needs you to ask what reality actually meant.

MayaEvery time. The exhaust is the most valuable stream you have and the most contaminated — by secrets, by license, by your own benchmarks, and by the messy reasons humans really click "approve." Catch it, run the gauntlet, and never confuse "this happened in production" with "this was good engineering."

LeoHere's what I'd leave the listener sitting with.

MayaGo ahead.

LeoThink about the system you work on right now. It's already generating a record of real work — branches, reviews, merges, the things that broke later. If you had to start keeping that as training data tomorrow, which gate would stop you first — privacy, secrets, licensing, or the fact that you've never once checked whether it overlaps your own tests?