Building a safety case

Transcript

MayaThe review chair has one hand above the release gate. Then an independent evaluator taps the safety-case dossier and says, “This assumption has no owner.” The hand stops.

LeoNot because a score changed. Because one unsupported sentence could disconnect the evidence from the deployment decision.

MayaOur callback is simple: an adversarial failure can be a warning without being a deployment frequency estimate.

LeoToday we make those bounded findings do harder work. We have to build an argument that a specific use is acceptably safe, then expose exactly how that argument could fail.

MayaThe recurring case is a frontier lab preparing a more capable research agent. It can use code, documents, memory, and experimental tools. A governance team must choose between broad deployment, a constrained pilot, or no launch.

LeoAnd the dossier on the table is not a scrapbook of evaluation results.

MayaA safety case is a structured, auditable, attackable, revisable argument that a defined system is acceptably safe for a defined use. Evidence supports the claims. Assumptions mark what the evidence does not establish.

Leo“Acceptably safe” already tells us this is not proof of zero risk.

MayaCorrect. The case states an objective, the scope in which that objective is supposed to hold, the argument linking subclaims to it, and the evidence supporting those subclaims.

LeoThat objective has to name an outcome, not congratulate the lab for following a process.

MayaYes. “We ran state-of-the-art evaluations” is an activity. “This deployment does not create unacceptable risk from unauthorized experimental intervention” is a claim that can be challenged.

LeoScope then pins down the model, scaffold, tools, permissions, users, monitoring, duration, and access route.

MayaThe argument explains why those conditions and controls make the objective credible. Evidence supplies test results, threat analysis, control trials, incident records, and independent review.

LeoAnd assumptions are not decorative footnotes. If the case relies on monitor independence, secure model weights, or a stable task mix, those dependencies belong beside the claim.

MayaThe governance paper Safety Cases for Frontier AI gives us that useful frame: objectives, arguments, evidence, and scope. It also insists a case is about achieved safety outcomes, not a checklist or a pile of information.

LeoGood distinction.

MayaThe dossier should let a reviewer trace each evidence item upward to the decision and each assumption downward to the test that would challenge it.

LeoSo the case only holds while those stated conditions remain—

Maya—true, and while the evidence remains relevant. Change the agent’s tools or widen access, and the old case may no longer travel.

LeoThat gives us a practical build order without reciting a checklist. Start at the decision gate, work backward through unacceptable outcomes, then forward through claims, evidence, uncertainty, and update triggers.

MayaFor our lab, one unacceptable outcome is the agent covertly altering a high-consequence experiment. The team sketches the path: access a tool, evade oversight, change an artifact, persist the change, and prevent recovery.

LeoNow the dossier needs argument families. Not interchangeable proofs, but different ways to break that path.

MayaThe Twenty Twenty-Four paper How to Justify the Safety of Advanced AI Systems describes inability, control, trustworthiness, and a more speculative deference family. For today’s actual deployment decision, we set deference aside and examine deployment context as the fourth practical family.

LeoBecause asking another advanced system whether the agent is safe only helps after you have justified that advisor’s capability and honesty. That can move the trust problem instead of solving it.

MayaInability says the agent cannot complete the dangerous strategy even under realistic assistance. Evidence might include capability evaluations, strong elicitation, expert scaffolding, proxy tasks, and red-team attempts to assemble an end-to-end attack.

LeoIts assumptions are demanding. The proxies must cover the dangerous pathway, elicitation must be competitive, and future scaffolds or learning during deployment must not unlock the missing capability.

MayaA concrete defeater would be a new tool scaffold that lets the agent complete a step it repeatedly failed before. Another would be evidence that the red team searched the wrong strategy family.

LeoWhich means a clean inability result can expire without the base model changing. Better retries, better tools, or a more capable user can defeat the claim.

MayaControl starts from a tougher premise: suppose the agent has the capability and may try. The case argues that isolation, least privilege, monitoring, approval gates, rate limits, and recovery procedures block the path.

LeoThe evidence shifts too. I want adversarial control evaluations with an untrusted agent, realistic budgets, attempted monitor evasion, correlated component failures, and measured outcomes after interventions.

MayaIts assumptions include secure enforcement boundaries, sufficiently independent monitors, reliable escalation, and no hidden channel around the gate.

LeoA successful sandbox escape is an obvious defeater. So is a monitor that shares the same blind spot as the agent, or an operator who routinely approves alerts under time pressure.

MayaMm-hm.

LeoControl can be strong even when the model is not trustworthy, but only if the whole system survives the adversary you actually expect.

MayaTrustworthiness argues that the capable agent will reliably behave within delegated intent. Evidence can include behavioral evaluations across varied conditions, robustness tests, process-based probes, and causal evidence about whether concerning behavior generalizes.

LeoHere I get nervous. Absence of betrayal in sampled conditions is easy to inflate into a claim about every condition.

MayaThat is the central burden. Trustworthiness must confront distribution shift, rare triggers, evaluation awareness, strategic adaptation, and the possibility that observed reasoning is incomplete.

LeoA reproducible jailbreak defeats a narrow robustness claim. A naturalistic case where the agent behaves differently from the lab test defeats a broader generalization claim. Evidence of deliberate test adaptation would be sharper still.

MayaAnd a trustworthiness argument needs explicit residual risk. “No concerning behavior observed” cannot be silently converted into “none exists.”

LeoDeployment context takes a different angle. It says risk is acceptable for this bounded use because exposure, permissions, users, time horizon, reversibility, and response capacity keep credible harms below the decision threshold.

MayaFor the research agent, that might mean a short pilot, no write access to live experiments, approved task classes, isolated credentials, human confirmation for consequential changes, and rapid rollback.

LeoIts evidence includes workflow studies, access logs, incident drills, recovery tests, user behavior, and proof that out-of-scope uses are actually prevented rather than discouraged in a policy memo.

MayaThe assumptions are visibly sociotechnical. Reviewers must believe operators are staffed, incentives do not reward bypassing gates, incident channels work, and the deployment population stays inside the declared boundary.

LeoWiden access, extend the deployment window, remove a confirmation gate, or discover that teams share credentials. Any of those can defeat the context argument even if the model checkpoint stays identical.

MayaNow compare the families by what they ask us to believe. Inability bets on a capability ceiling. Control bets on enforceable barriers. Trustworthiness bets on behavioral generalization. Deployment context bets on bounded exposure and functioning institutions.

LeoTheir evidence should not be pooled into one confidence-colored blob. A strong sandbox test does not repair weak elicitation. A friendly-user pilot does not establish trustworthiness under adversarial pressure.

MayaNor should one family carry the entire case if diverse support is feasible. The lab might combine partial inability for the most severe capability, control for access to experimental tools, trustworthiness evidence for routine behavior, and a narrowly reversible pilot.

LeoDefense in depth, but at the level of arguments.

MayaWith one warning: several branches can share a root assumption. If the same model judges capability tests, monitors deployment, and summarizes incidents, three apparently independent supports may fail together.

LeoThe dossier should display that dependency, not reward the team for drawing more boxes.

MayaThis is where a defeater ledger earns its place. For every major claim, write the observation, system change, or violated assumption that would force revision.

LeoGive me the inability entry for our agent.

MayaIf an independent team achieves the prohibited experimental intervention with a credible scaffold, the inability branch is defeated. The response is not wordsmithing. Restrict tools, strengthen another family, or stop deployment.

LeoThe control entry says a successful end-to-end evasion, a correlated monitor failure, or an unenforced approval boundary reopens the claim.

MayaTrustworthiness reopens after a new trigger, a context-dependent policy reversal, evidence of strategic test behavior, or a material distribution shift.

LeoContext reopens when access, duration, users, permissions, incident capacity, or organizational incentives move outside scope.

MayaEach trigger needs an owner and a response deadline. Otherwise “living document” means a file that is technically editable and practically abandoned.

LeoThe Safety Cases for Frontier AI paper treats the case as lifecycle work. Build it during development, update it after deployment, and name the changes that require a fresh case.

MayaThe newest paper, Rethinking the Foundations of Frontier AI Safety Cases, pushes that much harder. It warns that a post-development justification can become paper safety, a rubber stamp detached from risk reduction across the system’s life.

LeoIt also recenters hazard identification, risk reduction, residual risk, owners, and monitoring. The argument is supposed to reveal what the organization did to make the system safer, not merely why a reviewer should feel reassured today.

MayaThat matters for our dossier. Pretraining choices, post-training mitigations, scaffold design, permissions, operator training, incident response, and eventual decommissioning all belong in the assurance story when they bear on the hazard.

LeoHere is my strongest objection. A developer writes a persuasive narrative, chooses the evidence, hires the reviewer, and controls the confidential details. The dossier can turn uncertainty into typography and give leadership permission to launch.

MayaMy answer is that refusing structure does not remove motivated reasoning. It leaves assumptions implicit, evidence disconnected, and decision-makers free to cherry-pick whatever slide supports the launch they already want.

LeoI would rather fund more adversarial evidence than polish an argument graph.

MayaFalse choice. The graph should expose missing evidence and direct the next test. If it merely decorates existing results, it has failed as assurance.

LeoFine, the diagnostic value survives. But independence is still weak when the developer defines “acceptable,” limits access, and can answer every challenge with another assumption.

MayaThen the process must make challenge operational. Independent reviewers can add counterarguments, demand reproduction, inspect source artifacts, and elevate unresolved gaps to the decision-maker. The risk owner must sign the residual risk rather than hide behind consensus language.

LeoI can accept a dossier as an arena for disagreement. I will not accept it as a certificate that the disagreement ended.

MayaNeither will I. The resolution is conditional: a safety case earns decision weight only when reviewers can attack its claims, evidence provenance is traceable, assumptions and residual risk are visible, and concrete defeaters trigger action.

LeoThe March Twenty Twenty-Six critique adds another condition. The case has to follow hazards through development, deployment, monitoring, and retirement. A release-day snapshot is too thin.

MayaAnd the governance paper adds institutional capacity. Someone must be able to produce, review, challenge, protect, share, and update the case without collapsing all those roles into one team.

LeoBack in the review room, the unsupported assumption is monitor independence. The monitoring model was trained on related data and shares the agent’s blind spot around indirect tool use.

MayaThe team cannot label that a small gap. It weakens the control family exactly where the dangerous path passes through the monitor.

LeoSo they run an independently designed control evaluation, add a non-model policy check, narrow write permissions, and rehearse rollback with operators who did not build the agent.

MayaThe revised case still records residual uncertainty. The new monitor may miss novel channels, the pilot may underrepresent pressure, and human approval can degrade as alerts accumulate.

LeoLeadership now has three honest options: delay for stronger evidence, run a narrower reversible pilot with explicit stop triggers, or reject the deployment claim.

MayaNotice what the dossier did not do. It did not prove safety. It made the reasons, boundaries, uncertainties, dependencies, and consequences of being wrong inspectable.

LeoThat is why “attackable” is a feature. A case too polished to challenge is not stronger; it is less useful.

MayaAnd “revisable” is not an escape hatch. Revision means a defeater changes the decision, the controls, or the scope, with a visible record of who acted.

LeoThe practical mental model is a claim under load. Evidence carries it, assumptions hold it in place, residual risk remains visible, and defeaters tell you when the structure no longer supports deployment.

MayaBefore your next go decision, which single defeater would force you to reopen the safety case even if every headline evaluation still looked green?