The agent that nobody could stop
A distribution company ran an agent that automatically rerouted freight when weather events hit regional hubs. It worked well in testing and it worked well in the first three months of production. Then a data feed started returning stale timestamps, the agent interpreted a resolved storm as an ongoing one, and it spent eleven hours rerouting trucks that were already at their correct destinations. Nobody stopped it quickly because nobody was sure who was allowed to stop it, or how.
That is not an AI problem. That is a contract problem. The agent had never been told what it owed the people running it.
A production agent is not a prototype that survived
Most agents I see in enterprise environments started as prototypes that performed well enough that someone said ship it. The prototype mindset carries over. The team treats the agent as a clever tool rather than as a system with obligations, and that gap eventually costs them.
A serious production agent has a contract with its operators. Not a legal document. A set of behaviours the system commits to so that the humans around it can do their jobs. That contract has four clauses and I have never seen a stable long-running agent that violated any of them.
Recoverability means the agent can be unwound not just restarted
Restarting an agent after a failure is table stakes. Recoverability means something harder: the actions the agent took while it was misbehaving can be identified, assessed, and reversed or corrected without heroic manual effort.
In a manufacturing context I worked with, an agent was responsible for generating purchase orders against a dynamic bill of materials. When an upstream pricing model fed it bad data, it issued about forty orders at incorrect quantities before anyone caught it. The agent itself restarted fine. The problem was that nobody had built a reconciliation log that tied each order back to the specific model output that drove it. The operations team spent two days manually tracing what had happened. Recoverability was not about the agent, it was about the audit trail the agent was supposed to leave behind it.
Every action an agent takes in production should carry a provenance record: what input triggered it, what reasoning step produced it, what the agent believed at the moment it acted. Not for compliance theatre. For the person who has to clean up at two in the morning.
Observability means a human can read the agent's current state without asking it
Language model based agents are not transparent by default. They produce outputs but they do not naturally expose the intermediate state that tells an operator whether the system is behaving normally or is about to do something expensive.
I distinguish between logging and observability. Logging is recording what happened. Observability is being able to answer, right now, what is this agent doing, what does it currently believe, and is that belief grounded in fresh data. Those are different questions and they require different instrumentation.
In a field operations deployment, we built a simple status surface that showed, for each active agent session, the last verified data timestamp, the current task the agent had selected, and a confidence band on that selection. It was not sophisticated. It was a dashboard any dispatcher could read in ten seconds. That surface caught three cases in the first month where the agent had latched onto a stale work order and was about to dispatch a technician to a job that had already been closed. The dispatcher saw the stale timestamp, intervened, and the agent was corrected before it acted.
If your operators cannot read the agent's state without interrogating it through a chat interface, you do not have observability. You have a black box with a personality.
Clear ownership means one person is accountable when the agent acts
This is the clause teams resist most because it feels political. It is not political. It is operational.
When an agent acts autonomously in a production environment, something will eventually go wrong in a way that requires a human decision: stop the agent, let it continue, escalate, roll back. That decision needs to happen in minutes, not after a meeting about who owns the AI programme. The only way it happens in minutes is if one person has been designated, in advance, as the accountable party for that agent in that operational domain.
In a financial services context, an agent was handling a subset of client data reconciliation tasks overnight. When a connectivity issue caused it to skip a validation step and post partial records, three people each assumed one of the others was handling it. The agent ran for four hours in a degraded state because the ownership question had never been answered cleanly.
Ownership does not mean that person built the agent or understands the model. It means they have the authority to act, they know the escalation path, and they are reachable. One person, one agent, one phone number. Everything else is a committee.
The off switch has to work and everyone has to know where it is
This sounds obvious until you are in the room when someone needs to use it.
I have seen off switches that required a deployment pipeline to trigger. I have seen off switches that stopped the agent process but left downstream jobs it had queued still running. I have seen off switches that existed in documentation but had not been tested since the system was deployed six months earlier.
A working off switch has three properties. It is fast, meaning it stops the agent's ability to take new actions within seconds not minutes. It is complete, meaning it also drains or cancels queued actions the agent had already committed. And it is tested on a schedule, not just assumed to work because it worked once.
The distribution company from the opening story had an off switch. It stopped the agent process. It did not cancel the rerouting instructions the agent had already written to the transport management system queue. The trucks kept moving on bad instructions for another two hours after the agent was technically off. A complete off switch would have caught that.
The practical takeaway for teams shipping agents now
Before you promote an agent from pilot to production, run it through four questions.
- Can every action this agent takes be traced back to the specific input and reasoning that produced it, and can that action be reversed or corrected without manual archaeology?
- Can an operator read the agent's current state and data freshness from a surface they can check in under thirty seconds?
- Is there one named person who has the authority and the obligation to act when this agent behaves unexpectedly, and do they know that?
- Has the off switch been tested in the last thirty days, and does stopping the agent also stop the downstream effects of what it had already queued?
If any of those answers is no or we think so, the agent is not ready for production. It is ready for more work.
The teams I have seen run agents stably over months are not the ones with the most sophisticated models. They are the ones who treated the agent as a system with obligations and held it to them.