Which documents should skip human review?

What fraction of documents can safely skip human review?

The fraction that safely skips review is not a percentage you can borrow from anyone; it is whatever is left after you remove every field that is both consequential and unverifiable. A document skips review when every consequential field on it can be checked against a second source the business already holds and every action it triggers can be reversed. That is a property of the document type and your systems, not a dial on the model.

The objection to any smaller answer was put cleanly in a December 2025 Ask HN thread on production document processing: "Human in the loop doesn't help because the human would just have to read the document themselves to ensure accuracy, defeating the point of the automation." That holds only if review is all-or-nothing. The commenter who quoted it answered it in the same breath, from an auto insurance claims project: the system submits the report number to pull the details of the people involved, and "if the names, vehicle type, etc don't match, that automatically flags that something's off," with the verdict that "semi-auto beats manual readily."⁰¹ That is verifiability doing the routing, field by field, and nobody re-read the whole document.

Why is a confidence threshold the wrong test?

A confidence threshold is the wrong test because confidence measures the model's ambiguity about its own output. What an error would cost the business is a separate quantity, and the threshold never sees it. A confidence-only queue assumes a correlation between the model's uncertainty and the buyer's consequence that it never verifies, because the two live on different axes and nothing in the pipeline ties them together. A model can be certain and wrong on the one field that posts a payment, and unsure on a field nobody downstream reads.

The benchmark evidence is that the signal has moved. On WorkBench, a sandbox of five business databases and 26 read-and-write tools across 690 tasks, the best office agent completed 43 percent of tasks in March 2024, and the 2026 revisit finds the best agent completing 98 percent.⁰² Unintended harmful actions such as emailing the wrong person fell from 26 percent of tasks to 1.9 percent over the same span, and the authors report that capability and safety improve together on this benchmark rather than trade off.⁰² When completion sits that high, "the system could not finish" has stopped being the signal that finds errors, so a queue built on the model's own doubt is now looking in the wrong place; the residual is concentrated in mistakes the authors describe as occasionally resulting in irreversible harm.

Read that number at its real scope. The authors cap their own trend: any model trained after 2024 may have seen the benchmark, so the over-time results should be read as an upper bound, and the 1.9 percent is a sandbox measurement across 690 scripted tasks against five mock databases, never a production incident rate.⁰²

What is an escape rate, and why can a confidence queue never report one?

An escape rate is the proportion of documents that skipped review and carried an error that should have stopped them. A confidence-only queue can report its exception rate honestly and its escape rate not at all, because the cases it never routed are the cases nobody ever looked at, so the errors among them stay unmeasured by construction. The exception rate tells you how much work went to a person. The escape rate tells you how much bad work went to your system of record, and it is the number that matters.

The failure it names is familiar to anyone running agents in production. A system reports success while the record is left in the wrong state: a subscription changed, a message sent twice, a job left half finished, and nobody notices until much later. Those are escapes. No confidence score contains them, because the model was not in doubt.

How do you measure your escape rate starting next week?

You measure your escape rate by re-reviewing a random sample of the documents your system already passed. Draw N auto-approved items at random from a recent period. N is set by the precision you need, since the margin of error on a proportion narrows with the square root of N. Have the domain owner re-check each drawn item against the same second sources the system should have used. The escape rate is the count of errors that should have stopped a document, divided by N. Repeat the draw each month so the figure tracks drift. It runs on data you already have, and it survives a vendor swap because you defined it against your own outcomes.

Which two axes actually decide routing?

Routing is decided by two axes, consequence and verifiability, applied to each field rather than to the whole document. Consequence is what an undetected error costs downstream and whether the action it triggers can be undone. Verifiability is whether the value can be checked against a second source the business already holds: a plate number against the vehicle record, a vendor bank detail against the supplier master, an invoice total against the purchase order, a policy number against the policy administration system. Anything verifiable is checked in code, not by a person; anything unverifiable and irreversible routes regardless of what the model reports about itself.

The consequence axis is about to acquire a statutory definition. Colorado's SB 26-189, signed 14 May 2026 and applying to consequential decisions made on or after 1 January 2027, will define consequential decisions as those relating to an individual's access to, eligibility for, or compensation related to education, employment, housing, financial or lending services, insurance, health-care services, or essential government services and public benefits.⁰³ That list is the high-consequence end of the axis written into law, and any field feeding one of those decisions sits at the top of it. What counts as an error on such a field, and the record you owe the consumer afterward, is set by the accuracy bar you sign before the build.

The model's confidence stays in this design. Confidence sets the floor; consequence and verifiability set the rule. A case below the confidence threshold still routes, and a case above it routes anyway when consequence and verifiability say so. That floor is what the shipped mechanism in why pilots stall routes on, and consequence and verifiability sit above it.

How does routing look field by field on a real document?

Routing renders one field at a time, and the decision for each field falls out of its second source and its reversibility rather than out of a global setting. The same two documents most mid-market operations already process, an auto claim police report and an accounts payable invoice, classify cleanly this way.

Read the table and your own fields sort themselves. The rows that route are the rows with no second source and no undo, and they are the reason a person is on the workflow at all.

In what order do the routing rules run, and why that order?

The routing rules run in a fixed order, because a later rule must never be able to release what an earlier rule stopped. First, any field that is both irreversible and unverifiable routes to a person. Second, a consequential change to a verifiable field, such as a vendor bank detail that differs from the supplier master, routes even when its format is valid. Third, a verifiable field whose second source disagrees routes or auto-rejects. Fourth, a verifiable field whose second source agrees and whose action is reversible auto-passes and is logged. The confidence floor sits beneath all four: below the floor, a field routes no matter which rule would otherwise have cleared it.

Reversibility comes first because a reversible error is recoverable and an irreversible one is not, so the scarce hour of human attention goes to the actions that cannot be taken back. An action that is free and reversible runs on its own and is logged. An action that is irreversible, or visible to another human, emits a structured request for a person to approve. The rule the order enforces is that a guardrail lives outside the agent loop and leaves a record the agent cannot rewrite.

Why does escalating on judge disagreement route the wrong cases too?

Escalating on judge disagreement, where a second model grades the first and you route the cases they disagree on, routes the wrong cases for the same reason confidence does. It surfaces text that two models read differently, which is a measure of linguistic ambiguity. Consequence is a different axis, and disagreement does not touch it. Two models can agree, confidently and wrongly, on a bank detail they both misread, and it sails through; they can split on a harmless free-text note and it gets a person. Both of the obvious rules, low confidence and judge disagreement, select on the model's own internal signals and miss the consequence axis entirely, which is why neither can tell you what an escaped error will cost.

When does confidence routing actually work?

Confidence routing works in one situation: a genuinely calibrated classifier on a stable, narrow document type, where confidence has been measured against real outcomes and the low-confidence cases really are the error-prone ones. Where you have checked calibration on held-out real data and the document type does not drift, confidence is a fine floor and a decent triage order for the queue. It stops working the moment the field mix shifts or a new document variant arrives, because calibration is a property of a distribution and not of a model, and the confidence that was honest in January is guessing by March.

What has to be on the review screen, and what happens to every correction?

The review screen has to put the document, the extracted fields, and the reason each field routed on one surface, with correction in place, or the gate is slower than doing the work by hand. A gate slower than the manual task gets bypassed, and a bypassed gate is worse than none, because everyone upstream still believes it is holding while nothing is being caught.

Every correction a reviewer makes lands in the eval set, because a correction is a labelled example of exactly the case the system got wrong, and a gate that discards its corrections pays the review cost twice and learns nothing. The cross-check and the write-back use the same integration into your systems, so the routing is only ever as good as the integration budget behind it; a rule that names a second source nobody connected is a comment, not a control.

What is a normal exception rate, and which workflows should never have a queue?

There is no normal exception rate to borrow, and the honest figure for your document type is one you derive in an afternoon on documents you already have. Take a batch you already processed, apply the field-level rules, and count how many documents carry at least one field that is both consequential and unverifiable; that proportion is your floor exception rate before a line of code is written. A borrowed rate is worse than no rate, because it sets an expectation your real document mix will not meet, and the gap surfaces in production as a queue larger than the business planned to staff. The rate you measure here is also the input to whether the gate pays for itself.

Two kinds of workflow should never have a queue, at opposite ends of the same test. Some skip it entirely: every consequential field is verifiable and every action reversible, so you route none and check all in code. Others should never carry one because nothing on them is verifiable and everything is consequential. The queue is the whole job. Automation there adds a step instead of removing one, and the honest answer is do not automate this. If applying the rules routes nearly every field, the workflow is not a candidate, and no confidence slider changes that.

If the only reason an item is in your queue is that the model said it was unsure, you do not know your escape rate. Classifying a field by consequence takes someone who knows what a wrong value costs three systems downstream, who is not the engineer who built the pipeline, and getting that classification onto paper against a real document set is the work. That is the shape of a Ship engagement, and it starts with the readiness check: your fields sorted on the two axes, with the workflow owner in the room. The next step is a measurement, not a demo.

Frequently asked questions

What is an escape rate?

An escape rate is the proportion of documents that skipped review and carried an error that should have stopped them. A confidence-only queue can report its exception rate but never its escape rate, because the cases it auto-approved are the ones nobody re-checked. You measure it by re-reviewing a random sample of what the system already passed.

Does human review defeat the point of automation?

No, if review is routed field by field rather than applied to whole documents. A field checkable against a second source is verified in code, and only fields that are both consequential and unverifiable reach a person. Reviewing every document by hand does defeat the point, which is why the routing rule, not a blanket gate, is the mechanism.

Is model confidence useless for routing?

No, but confidence cannot decide routing on its own. A case below the confidence threshold still routes, and a case above it still routes when consequence and verifiability call for it. Confidence measures the model's ambiguity, which is neither consequence nor verifiability, so it serves as a triage signal only on a calibrated classifier over a stable document type.