T2E23 · RepoQA: A Unit Test for Find-the-Code, Not Understand-the-Codebase

Transcript

LeoFive hundred tests. Fifty repositories, five languages, ten functions each. Back in 2024 the strongest proprietary models scored around ninety percent on it — and here's the part that stuck with me: that ninety percent tells you almost nothing about whether the model could get real work done in any one of those fifty repos.

MayaThat's RepoQA. And you just put your finger on the whole tension — the test it aces and the job we actually care about are not the same test.

LeoRight. And this lands right in the middle of our evaluation thread — the one where we keep saying every lab grades its own homework, so the only question that matters is: a score of *what*, exactly?

MayaSo let's name the thing precisely. RepoQA's core task has a slightly clinical label — Searching Needle Function. The setup, though, is easy to hold. You hand the model a big slab of real code, repository-sized. Then a written description of what one single function does. The model's whole job is to hand back that one function.

LeoFind the needle.

MayaFind the needle. Not fix it. Not explain the repo. Not reason about what calls what. Locate the one function that matches the description, and return it.

LeoAnd how does it get graded? Because "did you find the function" sounds binary, but code is fiddly — whitespace, a renamed variable.

MayaGood instinct. It isn't exact-match. The grader scans everything in the context, picks the function most similar to what the model returned, and then checks whether that similarity clears a bar — a BLEU-style overlap score, set by default around zero point eight.

LeoBLEU's the old machine-translation metric — how many chunks of text you share with the reference answer.

MayaWord and phrase overlap, essentially. Which for this task is honestly fine, because the target is one concrete block of source code. Return the right function and you'll overlap it heavily. Return the wrong one and you won't.

LeoHere's what nags me about that grader, though. Overlap works when the answer is a chunk of code you can line up against a reference. But the moment the answer is a real explanation — half prose, half code — overlap stops meaning "correct."

MayaAnd that's a known crack. There's follow-on work arguing that a metric like BLEU is just inadequate for repository question-answering, where the reply mixes English and code. RepoQA gets away with it only because it pinned the answer down to one function. Narrow the task hard enough, and a cheap metric suddenly works.

LeoAnd it's not even uniform across the five languages, right?

MayaNo — performance swings by language. Which is itself telling. If "find the function" were one clean, general skill, it wouldn't wobble depending on whether it's Python or Rust in front of you.

LeoSo then what *is* it good for? Give me the honest one-liner.

MayaHere's how I hold the whole benchmark in my head. RepoQA is the eye chart at the optometrist.

LeoGo on.

MayaYou read the letters, top row to bottom row, and it proves one narrow thing — your vision resolves fine detail at distance. That's real. It's measurable. It's necessary. But nobody hands you car keys because you nailed the eye chart. Reading the chart isn't driving.

LeoAnd yet — if you flunk the eye chart, they are definitely not letting you drive.

MayaThat's exactly the move. Necessary, not sufficient. RepoQA is the eye chart for coding agents. If a model can't even locate the function you described, sitting right there in the context, don't expect it to safely rewrite it. But passing means you passed the eye chart — not the road test.

LeoAnd the authors basically say that themselves, don't they? I remember their FAQ being unusually humble for a benchmark paper.

MayaThey call it elementary — a starting point. The line is roughly: if a model fails this, don't expect it to pass the harder repo tasks. They were never claiming it ranks who's best at software engineering.

LeoOne more strange thing from the original work — they found that stripping the code comments out sometimes made retrieval *better*.

MayaWhich is the tell, right? Comments exist to help a human understand. If deleting them helps the model find the function, the model isn't understanding anything — it's pattern-matching the signature. The eye chart, again.

LeoOkay, but here's where I want to push. It's 2026. Does an eye-chart test still earn its keep? Because the field sprinted right past "find the function." Look at what the successor benchmarks actually ask. One of them — SWE-QA — is hundreds of human-validated questions about real repositories: intention, cross-file reasoning, multi-hop dependency chains. That's the job. "Find this one function" isn't even on the list.

MayaFair.

LeoReal repository understanding is messy developer questions, tool use, chasing a dependency across six files —

Maya— sure, and I'd argue it earns its keep *precisely because* it's narrow. Leo, the reason it's useful is that it moves one variable. Long context, one described target, one correct answer. When the score changes, you know what changed.

LeoOne variable. Okay.

MayaAnd there's a real dependency argument underneath it. Every one of those messy tasks you just listed — cross-file reasoning, multi-hop dependencies — silently assumes the model can locate the relevant code in the first place. You can't reason across files you failed to find. So a clean probe for "can it even find the described function" is testing the thing the harder tasks all quietly stand on.

LeoThat's the strong version.

MayaThat's the strong version. Fail this, and everything above it is guesswork.

LeoExcept the score changes in ways that should scare you. Grow the context and watch. There's a long-code benchmark where one frontier model fell from twenty-nine percent down to three. Three. Same model — just more code piled around the needle.

MayaThat's brutal.

LeoAnother one slid from around seventy down to forty. So "it can find the function" isn't even a stable property. It's a function of how much hay surrounds the needle, not a fixed skill you either have or don't.

MayaOkay. I'll give you that one, cleanly. The primitive is context-length-fragile, and a single ninety-percent headline hides that completely. But notice *why* we can even see the collapse — the task is narrow enough that context length is the only thing changing. The clean probe is what exposes the fragility. A messier benchmark would've buried it.

LeoFine. But narrow probes have a second failure mode — they get passed for the wrong reason. There's work showing high scores coming from memorized public answers. Stack Overflow the model already swallowed in training, not reasoning over the source in front of it. Move to repositories published after the training cutoff, and the scores just fall off.

MayaThe memorization tax. Yeah — and that's the sneakiest one, because it doesn't look like cheating. The model genuinely "knows" the answer. It just knows it from training, not from the code you handed it. On a popular open repo you can't tell the two apart. Swap in something published after the cutoff and the gap between remembering and reading finally shows up.

LeoAnd flat retrieval — just shoveling files into the window — loses to graph-based retrieval that models inheritance and cross-file calls. Because code dependencies aren't a line you read top to bottom. They're a graph.

MayaAnd that one I take seriously, because it's the real indictment. The eye chart tests linear acuity, and code is structural. There's lovely work on AST-aware read and edit operations — treating the code as its actual tree instead of a flat wall of text — that lifts pass rates *and* burns fewer tokens.

LeoThere's even a long-tail-repo benchmark where switching from direct answering to a tool-using agent buys you about a thirteen-point jump. That gap is the entire argument that you have to *explore*, not just retrieve.

MayaExplore, not retrieve.

LeoSo where does that leave RepoQA? Retire it?

MayaNo. Demote it. Put it on a rung — the same evaluation ladder we've climbed before, find-the-code sitting at the bottom. It should never pretend to be a higher one.

LeoSo which rung is the one that actually bites when you're about to edit?

MayaThe edit-readiness gate — a couple rungs up, past localization-with-evidence and grounded cited QA on fresh repos. Before you touch anything, state which modules you'd affect, the risks, the invariants, the tests you'd expect to move.

LeoBefore you touch anything.

MayaBefore a single line. And that edit-readiness rung is the one I care about most, honestly — because it's the whole difference between a model that pauses when it hasn't localized enough evidence, and one that confidently rewrites the wrong file. The eye chart can't tell those two apart. Both of them might read the letters just fine.

LeoThat's the safety gap in one sentence.

MayaAnd at the very top of the ladder, execution in a sealed room — run the tests somewhere the model can't see or tamper with the gold answer.

LeoAnd that top rung matters because harnesses get gamed. Shared agent-and-evaluator environments, exposed gold answers, weak string matching. The exploit list is real, and it's embarrassing.

MayaThe gaming-the-grader problem we've hit before. If the agent can reach the answer key, some model, somewhere, eventually will.

LeoSo the one-line version I'm taking home — RepoQA is a unit test for "find the code."

MayaNot an integration test for "understand the codebase." And absolutely not a safety check for "make the right change." Three different questions. The eye chart only answers the first one.

LeoWhich is completely fine — as long as nobody reads a ninety-percent eye-chart score and hands the model the keys to the whole repository.

MayaHere's what I keep circling back to. We agreed the eye chart is necessary before the road test — you'd never skip it. So for your own agents: what's the smallest "did it even find the right code" gate you'd put in front of an edit, before you let it change a single line?