T1E2 · When a language model becomes an agent

Transcript

MayaThe enterprise research agent has already searched the web, opened an incident report, run a small analysis, and stored a suspect vendor name. Now it reaches for the document tool to brief the security team.

LeoAnd the permission gate flashes amber. Send, or stop?

MayaThe model weights haven't changed since lunch. The system's agency just became visible.

LeoWe drew the boundary between alignment and broader safety in our foundation; today the boundary moves from one model reply to the whole deployed loop.

MayaAn ordinary language model takes some text and predicts a useful continuation. Give that model a goal, ways to act, a place to remember, rules about authority, and signals from what happens next, and you have something different: a system that can pursue an outcome across steps.

LeoThat's the word “agent” doing real work, then.

MayaYes—but agency isn't a jewel hidden inside the weights. It's the behavior produced by the model plus the job it was given, the tools it can reach, the memory it carries, the permissions it holds, the feedback it receives, and the environment that answers back.

LeoSo the unit we inspect is not “Which model?” It's “Which model inside what arrangement?”

MayaThat's the central move. Think of the model as a navigator. A navigator with no destination, vehicle, map, fuel, or keys can suggest a route. Add those pieces, and the surrounding system can actually travel.

LeoMaybe into a ditch.

MayaPotentially. Or safely to the right place. The point is that the ditch and the destination both depend on more than the navigator.

LeoLet's test that against our research agent. The goal is to investigate a safety incident. The web and code tools are its hands. The document tool is its publishing arm.

MayaMemory is its notebook. Permissions are its badge. Feedback is what tells it that a search failed, a script crashed, a reviewer approved, or a report was rejected. The environment is the changing workspace—web pages, files, people, and systems that react to its actions.

LeoAnd the model?

MayaThe model proposes the next move and often interprets the result. Crucially, the surrounding software decides what counts as a valid move, what gets executed, what is recorded, and when a human must intervene.

Leo“Often” matters there.

MayaIt does. Some designs let the model control the loop. Others use fixed software to choose which model outputs can become actions. Agency comes in degrees because autonomy, task complexity, time horizon, and supervision come in degrees.

LeoThat matches *AI Agents That Matter*. It treats agenticness as a spectrum shaped by goals and environment, supervision, and system design—not a binary sticker on a model.

MayaAnd then the paper makes a wonderfully inconvenient point: evaluating an agent is not the same as evaluating a model. A single answer score can hide retries, tool costs, brittle shortcuts, and the scaffold doing most of the work.

LeoMm-hm. If two systems use the same underlying model but one retries twenty times, consults five tools, and spends far more, “same model” tells us almost nothing about the deployed product.

MayaOr reverse it. Two different models inside the same careful scaffold may behave more similarly than their model leaderboards suggest.

LeoWait—the wrapper can dominate the comparison?

MayaSometimes. Not always. The paper found that simple baselines could rival more elaborate agent designs on a coding benchmark while costing much less. Its broader warning is the durable part: measure what matters in the use case, include cost, keep genuine holdouts, and make the setup reproducible.

LeoBecause a benchmark can accidentally reward the choreography instead of the capability.

MayaOr reward sheer repetition. If success rises because the system keeps rolling the dice, calling that architectural intelligence would be misleading.

Leo[chuckle] A very expensive way to look clever.

MayaFor our enterprise agent, accuracy alone is especially thin. A correct incident summary that took uncontrolled actions, exposed a confidential record, or quietly exhausted the tool budget is not a good outcome.

LeoNor is a safe-looking report if the agent skipped the one document that would overturn its conclusion.

MayaGood. Now the evaluation has several things to observe: task quality, resource use, policy compliance, the action trace, and whether the result survives a changed environment. That's a system evaluation.

LeoWhich brings us to *ToolEmu*. It asks how to find dangerous tool-use failures without wiring every test to a real bank account, inbox, or production database.

MayaIts answer is an emulated sandbox. Another language model imitates what tools and environments would return, so evaluators can stage many high-stakes scenarios without performing the dangerous action in the real world.

LeoThe clever part is consequence without consequence.

MayaNicely put. The agent still has to interpret a situation, choose a tool, fill arguments, and react to results. Researchers can inspect risky trajectories—private data disclosure, financial harm, actions taken under ambiguity—while the assets are fictional.

LeoBut now a model is testing a model, and another model may be judging the failure. My evidence alarm is making a noise.

MayaIt should. The paper checked its emulator and evaluator against human judgments, which strengthens the case that the sandbox finds plausible failures. It does not turn simulated failure rates into the prevalence we should expect in a particular deployment.

LeoLet's argue that, because teams routinely blur those claims. I'll take the skeptical seat: I would not approve a deployment decision from an emulated benchmark alone. The fake environment may omit awkward details, and the automated judge may share blind spots with the agent.

MayaI'll defend the sandbox. I would use it early because rare, severe failures are expensive and sometimes irresponsible to hunt in live systems. Fast synthetic scenarios can expose classes of hazard before anyone builds the full integration.

LeoFine, hazard discovery survives—but if you call it a real-world risk estimate—

Maya—I've outrun the evidence. Agreed. The resolution is layered evidence: use emulation to discover and stress-test, then validate important findings with human review and increasingly realistic environments.

LeoAnd preserve held-out scenarios, or the agent builders learn the exam.

MayaThat's where *AI Agents That Matter* tightens the argument. A sandbox can be valuable and still be benchmark-fragile. Reproducibility, cost controls, and holdouts are part of safety, not clerical cleanup.

LeoNice convergence. ToolEmu expands what we can test; the evaluation paper limits what we may claim.

MayaNow add *Governing AI Agents*. It borrows a familiar institutional problem: a principal delegates work to an agent, but the agent has information, discretion, and opportunities the principal cannot fully observe.

LeoHere “agent” carries legal and economic baggage, not just software architecture.

MayaRight, and the analogy is useful because our research agent acts on behalf of a company. Who set its objective? Whose documents may it open? Who is accountable if it sends an accusation? Which actions require fresh consent?

LeoThe model card won't answer those.

MayaNor will a benchmark score. Governance lives in the authorization structure, the logs, the approval gates, and the people or institutions responsible for the deployment.

LeoThe paper also warns that familiar controls can strain when decisions are hard to interpret and actions happen at machine speed and scale.

MayaSo it argues for technical and legal infrastructure around visibility, inclusion, and liability. I read that as a design prompt: make affected interests visible, make consequential actions inspectable, and make responsibility land somewhere real.

LeoI want one caution. Agency law is an analogy and a body of doctrine built for human relationships. It doesn't magically solve machine behavior.

MayaAgreed. It sharpens questions of delegation and accountability; it does not prove that a language model has human motives, legal personhood, or a stable inner objective.

LeoGood. Anthropomorphism sneaks in fast here. We say “the agent decided,” and suddenly people imagine a tiny executive living in the neural network.

MayaBetter to say what happened. The deployed loop selected an action after reading its goal, context, memory, and tool results. That description is less romantic and more auditable.

LeoAlso more useful during an incident. “The AI went rogue” gives me nowhere to look.

MayaWhile “the document tool accepted a send action without confirming the recipient” names a control surface.

LeoLet's walk the control surfaces without turning this into a checklist recital.

MayaUse the authority envelope. At its center is the goal: what outcome may the system pursue? Around that sits reach: which tools and data may it touch? Then persistence: what can it remember, for how long, and who can correct it?

LeoThe outer edge is consequence. What can happen before a human gets another chance to say no?

MayaYes. Feedback connects the whole envelope. If the agent is rewarded only for finishing quickly, it may learn—no, the deployed loop may favor—shortcuts that a quality or safety signal would have discouraged.

LeoGood correction. We often can't tell whether a repeated failure lives in learned behavior, the prompt, the tool interface, or the reward signal without intervention tests.

MayaSwap one component while holding the others steady. Remove memory. Narrow permissions. Change the feedback. Replace the model. Move from simulation to a realistic test environment. Watch which behavior moves.

LeoThat's evidence I can use. It turns “agentic” from a vibe into a causal investigation.

MayaAnd it reveals why model-only safety claims are incomplete. A well-behaved model can be deployed inside a reckless loop. A fallible model can be surrounded by confirmations, least privilege, monitoring, and reversible actions that contain mistakes.

LeoContain, not erase.

MayaPrecisely. System controls reduce opportunity and impact; they do not certify inner alignment. Model improvements matter too. The mistake is asking either layer to carry the whole burden.

LeoOur enterprise example makes that concrete. The same model drafting a private note has modest reach. Connect it to web search, code execution, durable memory, internal documents, and outbound messaging, and the operational risk changes sharply.

MayaThen change the human review rule and it changes again. Approval before every search may make the system useless. Approval only after an external message may arrive too late. The design problem is where judgment belongs.

LeoAnd whether the reviewer can understand the evidence before the clock runs out.

MayaThat's an unresolved piece. We don't yet have universal measures for agency, ecological validity, or safe autonomy. Different domains tolerate different error, delay, and oversight costs.

LeoSo a banking agent, a lab assistant, and a calendar helper shouldn't inherit one generic “agent safety” score.

MayaNor should a research demo be treated as a deployment guarantee. We can make bounded claims: this model, with this scaffold, these permissions, this budget, these test environments, under this supervision, showed this behavior.

LeoThat's less catchy than “the model is an agent.”

Maya[chuckle] It is. But it's the sentence that helps a builder decide what to change.

LeoThen the takeaway is operational: draw the whole loop, test the whole loop, and govern the whole loop.

MayaAnd keep the weights on the map—just don't mistake them for the territory.

LeoAgency is the deployed relationship among capability, delegated purpose, action channels, persistence, authority, feedback, and a world that pushes back.

MayaWhich part of your enterprise research agent’s deployed loop would you change first to reduce real-world agency risk?