What we check before an AI agent touches a real system
Answer
Before an agent we build gets access to a real system, it passes a fixed preflight: bounded scope, a human approval gate, an eval set it must clear, a kill switch, an audit trail, a cost ceiling, and a tested rollback path. Any check that fails blocks go-live, and any action that cannot pass one stays behind a person. The list is short because it has to be enforced, not admired.
TL;DR
- An agent that can write to a real system is a production system; it is held to that standard before go-live, not after the first incident.
- The preflight: bounded scope, a human approval gate, an eval set, a kill switch, an audit trail, a cost ceiling, and a rollback path. All seven, or it does not ship.
- Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, or inadequate risk controls. Each of those is checkable up front.
- Full autonomy is rare in practice: 7% of companies run fully autonomous agents in production, and approval-first is the dominant operating model. Autonomy is earned by measured history.
- The checklist is copyable, and copying it is encouraged; the discipline of refusing to ship without it is the work.
An agent with write access is a production system
An agent is software that plans and executes steps against your systems, not a chat window. The difference matters the moment it can write. A wrong chat answer wastes a reader's minute; a wrong write posts a payment, closes a ticket that was never resolved, or sends a customer something untrue. From that moment the agent is a production system, and it is held to the standard every other production system meets, before go-live rather than after the first incident.
The industry numbers describe what happens otherwise. Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, or inadequate risk controls.01 Full autonomy is also far rarer than the demos suggest: in Bain's Automation and AI Pathfinder Survey 2026 of 951 companies, 7% run fully autonomous agents in production,02 while the most common operating model, cited by 38% of respondents, is human approval required, with another 32% running guardrails with exceptions.02 Cost, value, control: each stated reason for cancellation is checkable before an agent ships. That is the preflight's job. We build bounded agents for defined workflows, and none of them goes live without it.
The preflight
Seven checks, in the order we run them. Each has a pass condition you can state in one sentence; if you cannot state it, the check has not passed. An agent that fails any of the seven does not touch a real system. One distinction before the list, because the words collide: this preflight is not the Seven Gates, which is the methodology every workflow build clears before go-live. The preflight is the agent-shaped checklist that runs inside that methodology on the day a workflow includes an agent that can write.
1. Bounded scope
The agent gets a written scope: the systems it may touch, the actions it may take, the data it may read, with everything else denied by default. The scope fits on a page, and the owner of the workflow signs it. If the scope cannot be written down, what exists is a demo with credentials, and the fix is to narrow the job until the page can be written.
2. A human approval gate
Every write to a system of record passes a named person before it executes: in the loop, not in a report afterward. The gate is a real work surface: the approver sees what the agent intends and why, approves or corrects in one action, and the corrections are kept. Approval-first is the dominant operating model for production agents today,02 and the gate narrows action by action as the measured record justifies it.
3. An eval set it must clear
Before the agent ships, and before every change after that, it runs against graded cases that state what correct means for this workflow, the ugly ones included: the ambiguous request, the malformed document, the case it should refuse. The pass condition is mechanical: clear the agreed bar or the release does not go out. A team that cannot produce its eval set does not know whether its agent works; it knows nobody has complained yet.
4. A kill switch
One action, available to named people, stops the agent immediately. It is tested before go-live: someone presses it and everyone watches the agent stop. If stopping the agent takes a deploy, a ticket, or finding the one engineer who knows how, there is no kill switch, and the first bad afternoon will prove it.
5. An audit trail
Every action is recorded with what the agent saw, what it decided, what it did, and who approved it, stored outside the agent so the record survives restarts, upgrades, and the agent's own retirement. The pass condition: a reviewer can reconstruct any action later without asking an engineer. After an incident, the first question is what happened; the trail is the difference between an answer and an argument.
6. A cost ceiling
The agent gets a hard budget per run and per period, enforced in code, with an alert before the line and a stop at it. An agent that can retry, branch, and call tools can spend without bound, and escalating cost is the first reason Gartner names for canceled agentic projects.01 The ceiling is tested the way the kill switch is: watch it fire, and watch the in-flight work park cleanly instead of vanish.
7. A rollback path
For every write the agent can make, there is a known, rehearsed way to undo it, exercised before go-live on records that do not matter yet. Writes that cannot be undone, a payment sent, a message delivered, a record purged, stay behind the approval gate permanently. Reversibility is what separates an action an agent may take from an action it may only propose.
The checklist, in copyable form
The same seven, laid out to be lifted into your own runbook. The TL;DR promises a copyable checklist; this is it.
| Check | Pass condition, in one sentence | How it is tested | If it fails |
|---|---|---|---|
| Bounded scope | The systems, actions, and data fit on one written page, signed by the workflow owner | Read the page; try to name an action the agent could take that is not on it | Narrow the job until the page can be written |
| Human approval gate | Every write to a system of record passes a named person before it executes | Trace one write end to end and find the person in the path | Writes become propose-only until the gate exists |
| Eval set | The agent clears the agreed bar on graded cases, ugly ones included, before every release | Run the suite; a release without a green run does not go out | Scope narrows to the cases it clears; the rest routes to a person |
| Kill switch | One action, available to named people, stops the agent immediately | Press it before go-live and watch the agent stop | The agent runs only in supervised windows until the switch works |
| Audit trail | A reviewer can reconstruct any action later without asking an engineer | Pick a past action and reconstruct it from the record alone | Instrumentation lands first; nothing else proceeds without the trail |
| Cost ceiling | A hard budget per run and per period is enforced in code, with an alert before the line | Trip the ceiling on purpose and watch in-flight work park cleanly | The agent runs with a manual budget watch and a lower cap until enforced |
| Rollback path | Every permitted write has a rehearsed undo, exercised on records that do not matter yet | Roll one back before go-live | The action moves behind the approval gate permanently |
Seven checks, each with the pass condition stated and the test that proves it. Copy it; the discipline of refusing to ship without it is the work.
What failing a check means
A failed check does not end the project; it shrinks it, and the table's last column is the shrinking. The scope narrows, the irreversible actions move behind the gate, and the agent ships small, with the eval history and the trail that let it earn more. After go-live, the same checks become the standing discipline of agent monitoring and fleet governance: monitoring against named thresholds, controls to pause, roll back, or retire an agent, and evals run before every change.
When the seven are too many, and when they are not enough
Both cases exist, and a checklist that cannot say so is a ritual. The full seven are overkill for an agent that only reads, drafts for a person, or acts on records a single operator can trivially restore: there the lighter half applies, bounded scope, the eval set, and the cost ceiling, and the rest switches on the day the agent can write. The seven are not enough when the agent's actions reach beyond your own systems: anything that touches a customer, money, or a decision in the growing statutory category of consequential decisions needs the review posture those decisions demand, which is its own subject: which documents should skip human review maps that boundary, and what an MCP server does to your guardrails is the test procedure for the controls on a system that predates the agent. The stage-by-stage methodology, with the checkpoint each stage must pass, is on the Approach page.
Sources
Every load-bearing claim above, with its source and the date we checked it.
| Source | Reference | Accessed |
|---|---|---|
| 01 Gartner press release, 25 June 2025, basis verified | https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027 (opens in a new tab) | |
| 02 Bain, Your AI Budget Is Growing. Your Returns Aren't. Here's Why. (1 June 2026), basis verified | https://www.bain.com/insights/your-ai-budget-is-growing-your-returns-arent-heres-why/ (opens in a new tab) |
Frequently asked questions
- Does the approval gate defeat the point of an agent?
- No. The agent still does the work: reading, planning, drafting, routing, preparing the write. The person makes the irreversible call. The gate starts wide because trust starts at zero, and it narrows action by action as eval history and the audit trail justify it.
- When can an agent act without approval?
- When three things hold: measured performance at the agreed bar, a clean audit trail, and reversible actions. An action that cannot be rolled back keeps the gate no matter how good the history looks.
- Is the full list needed for a read-only agent?
- No. Read-only work needs the lighter half: bounded scope, the eval set, and the cost ceiling. The full seven switch on the day the agent can write, because the trigger is write access, not team size or traffic.
- Our agent is already live without most of this. What now?
- Retrofit in the same order, instrumentation first. Turn on the trail, build the eval set around observed behavior, then gate the writes and set the ceilings; none of it usually requires rebuilding the agent.