Every agent we build spends weeks answering questions that nobody sees. It reads the same ticket the human reads, forms the same answer, and then the answer goes into a log instead of to a customer. We call it shadow mode, and it is the least negotiable part of how we work.
The reason is unglamorous. A demo proves an agent can be right. Shadow mode tells you how it is wrong.
What the logs actually show
In the first week of shadow mode on a support agent, the interesting column is not accuracy. It is disagreement: the cases where the agent's answer and the human's answer differ. There are usually four kinds, and only one of them is the agent being bad at its job.
The agent is wrong. It misread the policy, missed a condition, hallucinated a detail. This is the category everyone expects and it is usually the smallest.
The human was wrong. Not often, but it happens, and when it does it tends to be systematic — one shift applying a policy that changed six months ago. We have handed clients a list of these and watched the room go quiet.
Both are defensible. The policy genuinely does not say. These are the most valuable rows in the log, because they are not an AI problem at all. They are an unwritten-rule problem that has been costing the team time forever, and now it is visible.
The question was not answerable. Missing data, broken integration, a system that was down. Infrastructure, wearing a costume.
Only the first category is a prompt or retrieval problem. The other three are findings about the business, and clients consistently tell us afterwards that they were worth more than the agent.
Why we do not skip it for "simple" agents
The argument against shadow mode is always the same: this one is easy, the scope is narrow, let us just turn it on with a low traffic percentage and watch.
Staged rollout and shadow mode are not the same thing, and the difference matters. In a staged rollout, being wrong reaches a customer. In shadow mode, being wrong reaches a log. Early on, when the disagreement rate is highest, you want the log.
There is also a quieter benefit. The team whose work the agent is about to take over spends three weeks watching it get things right. By go-live they have opinions about it, they have corrected it, and they trust it in a way that no launch presentation produces. We have never had a support team resist an agent they had already argued with.
What ends shadow mode
Not a date. A number, agreed before the engagement started, plus a flat rule: no open disagreement in the "agent is wrong" category that we cannot explain.
Explain, not fix. Some failures are acceptable if the escalation path catches them. What is not acceptable is a failure nobody can account for, because that means the model is doing something we do not understand, and the next thing we do not understand will not be caught by the same guardrail.
The uncomfortable part
Sometimes shadow mode says do not ship.
We have had two engagements where the disagreement log made it clear that the function was not well-defined enough to automate. The humans doing it were not following a process; they were exercising judgement that nobody had written down and nobody could articulate on request.
That is a real finding. It is also a difficult call to make three weeks into a paid build. We make it anyway, because the alternative is shipping something that will be quietly switched off in four months, and everybody will have wasted more than the build cost.