What we check before an AI agent touches a real system

An agent with write access is a production system

An agent is software that plans and executes steps against your systems, not a chat window. The difference matters the moment it can write. A wrong chat answer wastes a reader's minute; a wrong write posts a payment, closes a ticket that was never resolved, or sends a customer something untrue. From that moment the agent is a production system, and it is held to the standard every other production system meets, before go-live rather than after the first incident.

The industry numbers describe what happens otherwise. Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, or inadequate risk controls.⁰¹ Full autonomy is also far rarer than the demos suggest: in Bain's Automation and AI Pathfinder Survey 2026 of 951 companies, 7% run fully autonomous agents in production,⁰² while the most common operating model, cited by 38% of respondents, is human approval required, with another 32% running guardrails with exceptions.⁰² Cost, value, control: each stated reason for cancellation is checkable before an agent ships. That is the preflight's job. We build bounded agents for defined workflows, and none of them goes live without it.

The preflight

Seven checks, in the order we run them. Each has a pass condition you can state in one sentence; if you cannot state it, the check has not passed. An agent that fails any of the seven does not touch a real system.

1. Bounded scope

The agent gets a written scope: the systems it may touch, the actions it may take, the data it may read, with everything else denied by default. The scope fits on a page, and the owner of the workflow signs it. If the scope cannot be written down, what exists is a demo with credentials, and the fix is to narrow the job until the page can be written.

2. A human approval gate

Every write to a system of record passes a named person before it executes: in the loop, not in a report afterward. The gate is a real work surface: the approver sees what the agent intends and why, approves or corrects in one action, and the corrections are kept. Approval-first is the dominant operating model for production agents today,⁰² and the gate narrows action by action as the measured record justifies it.

3. An eval set it must clear

Before the agent ships, and before every change after that, it runs against graded cases that state what correct means for this workflow, the ugly ones included: the ambiguous request, the malformed document, the case it should refuse. The pass condition is mechanical: clear the agreed bar or the release does not go out. A team that cannot produce its eval set does not know whether its agent works; it knows nobody has complained yet.

4. A kill switch

One action, available to named people, stops the agent immediately. It is tested before go-live: someone presses it and everyone watches the agent stop. If stopping the agent takes a deploy, a ticket, or finding the one engineer who knows how, there is no kill switch, and the first bad afternoon will prove it.

5. An audit trail

Every action is recorded with what the agent saw, what it decided, what it did, and who approved it, stored outside the agent so the record survives restarts, upgrades, and the agent's own retirement. The pass condition: a reviewer can reconstruct any action later without asking an engineer. After an incident, the first question is what happened; the trail is the difference between an answer and an argument.

6. A cost ceiling

The agent gets a hard budget per run and per period, enforced in code, with an alert before the line and a stop at it. An agent that can retry, branch, and call tools can spend without bound, and escalating cost is the first reason Gartner names for canceled agentic projects.⁰¹ The ceiling is tested the way the kill switch is: watch it fire, and watch the in-flight work park cleanly instead of vanish.

7. A rollback path

For every write the agent can make, there is a known, rehearsed way to undo it, exercised before go-live on records that do not matter yet. Writes that cannot be undone, a payment sent, a message delivered, a record purged, stay behind the approval gate permanently. Reversibility is what separates an action an agent may take from an action it may only propose.

What failing a check means

A failed check does not end the project; it shrinks it. The scope narrows, the irreversible actions move behind the gate, and the agent ships small, with the eval history and the trail that let it earn more. After go-live, the same checks become the standing discipline of agent monitoring and fleet governance: monitoring against named thresholds, controls to pause, roll back, or retire an agent, and evals run before every change. The stage-by-stage methodology, with the gate each stage must pass, is on the Approach page.

Frequently asked questions

Does the approval gate defeat the point of an agent?

No. The agent still does the work: reading, planning, drafting, routing, preparing the write. The person makes the irreversible call. The gate starts wide because trust starts at zero, and it narrows action by action as eval history and the audit trail justify it.

When can an agent act without approval?

When three things hold: measured performance at the agreed bar, a clean audit trail, and reversible actions. An action that cannot be rolled back keeps the gate no matter how good the history looks.

Is the full list needed for a read-only agent?

No. Read-only work needs the lighter half: bounded scope, the eval set, and the cost ceiling. The full seven switch on the day the agent can write, because the trigger is write access, not team size or traffic.

Our agent is already live without most of this. What now?

Retrofit in the same order, instrumentation first. Turn on the trail, build the eval set around observed behavior, then gate the writes and set the ceilings; none of it usually requires rebuilding the agent.