The Number That Surprises Every Operator
The first time a finance operations team I worked with saw their monthly AI spend broken out by category, they expected token costs to dominate the report. They were running a document review agent across thousands of trade confirmations a day. Tokens were eleven percent of the total cost. The rest was everything else they had not planned for.
This is not unusual. It is the norm. When you read about AI agents in vendor materials, you see pricing in dollars per million tokens. That number is real but it is almost irrelevant to your actual budget conversation. If you are building a business case or trying to explain a cost overrun to a CFO, you need to talk about the five other buckets that nobody puts in the brochure.
Evaluation Is a Full Time Engineering Problem
Before you run an agent in production you have to know whether it works. That sounds obvious. What is not obvious is that building a proper evaluation harness for a production agent is weeks of engineering work, and then it never stops.
In manufacturing I have seen quality inspection agents where the eval set had to be maintained by a domain expert, a process engineer who understood what a defect actually meant. That person's time is not free. Every time the model was updated, every time the prompt changed, every time a new product line was added, the eval set had to be reviewed and extended. We estimated roughly forty hours of combined engineering and domain expert time per major eval cycle. That is a recurring cost, not a one-time setup.
For a distribution client running a routing and exception-handling agent, we built automated regression tests but still needed human sign-off on edge cases before any change went live. The tooling to support that workflow, the storage for labeled examples, the compute for running evals at scale, added up to more than the token bill in months where we did several releases.
If you skip serious evaluation you will find out your agent is wrong in production, which costs more than evaluation ever would.
Monitoring in Production Is Not Optional and Not Cheap
A model that worked last Tuesday can behave differently this Tuesday. Upstream model updates, prompt drift, changes in the data it receives, shifts in user behavior, all of these move the ground under you. You need observability into what the agent is actually doing, not just whether it returned a 200 status code.
For a field operations client, we built logging that captured the full reasoning trace on every agent action, not just the final output. That logging volume was significant. Storing it, indexing it so you can query it when something goes wrong, building dashboards that a non-engineer can read, that is infrastructure work. It takes time to build and money to run.
Beyond storage, someone has to look at the dashboards. In every production deployment I have run, we have had at least one alert fire in the first ninety days that required a human to investigate and make a judgment call. In a finance context, that human needs to understand both the business logic and the agent behavior. That combination is not common and it is not cheap to staff.
The practical rule I use is this. Budget monitoring at roughly the same order of magnitude as your token spend. If you are spending a thousand dollars a month on tokens, plan to spend something close to that on the infrastructure and labor to know whether those tokens are producing correct outputs.
Integration Is Where Projects Go to Die Slowly
An agent that cannot reliably read from and write to your actual systems is a demo, not a product. Integration work is almost always underestimated because it looks like plumbing and people assume plumbing is fast.
In practice, enterprise systems have authentication flows that change without notice, APIs that return inconsistent schemas depending on which version of the backend is running, and data that is formatted for human readers rather than machine consumers. I spent three weeks on a single integration for a manufacturing client because their ERP exported dates in four different formats depending on which module generated the record. The agent had to handle all four or it would fail silently on a meaningful percentage of inputs.
Field operations is worse. You often have connectivity issues, legacy systems that were not designed to be called programmatically, and data that lives in spreadsheets managed by people who have been doing it that way for fifteen years. Getting the agent to the data is sometimes the majority of the project.
Integration also has ongoing maintenance costs. Systems change. Vendors push updates. A downstream API changes its response structure and your agent starts returning garbage until someone catches it. Plan for integration maintenance as a recurring line item, not a one-time project cost.
The Human in the Loop Is a Feature That Costs Real Money
Every serious production agent I have shipped has a human review step somewhere. This is correct. Models make mistakes. The question is not whether to have a human in the loop but where to put them and how to make that work efficient.
In a finance context, we had an agent processing loan modification requests. It would draft a recommendation and flag its confidence level. Anything below a threshold went to a human reviewer. That reviewer was not doing less work than before, they were doing different work. Instead of reading raw documents they were reviewing structured agent outputs and either approving or correcting them. The speed gain was real and significant. But the human cost did not disappear.
What often happens is that organizations budget for the agent and forget to budget for the review workflow. They assume the humans will just fit it into their existing day. Then the agent ships, volume picks up, and the review queue becomes a bottleneck that limits the value the agent can deliver.
Design the human review workflow before you build the agent. Know who is reviewing, how long it takes, what their escalation path is, and what happens when they disagree with the agent. That workflow has a cost. Put it in the budget.
The Hidden Costs That Show Up After Month Three
There are costs that do not appear in your initial estimate because they only emerge once the system has been running long enough to accumulate history.
Prompt maintenance is one. Prompts that worked at launch start to drift as the use cases expand, as users find edge cases, as the underlying model changes. Someone has to own the prompt, test changes, and deploy updates carefully. In every deployment past the six-month mark, I have seen prompt maintenance become a meaningful ongoing engineering task.
Data labeling is another. As you find cases where the agent was wrong, you need to label those cases, understand the pattern, and either fix the prompt or add them to your eval set. Labeling is slow and requires domain knowledge. In manufacturing quality control, a labeled example of a defect the agent missed might take thirty minutes of an expert's time to properly document and categorize.
Finally, there is the cost of incidents. Every production system has incidents. When an AI agent is involved in an incident, the investigation is harder than a traditional software bug because you are often trying to reconstruct why a model made a particular decision. Good logging reduces this cost. Bad logging makes it very expensive. I have seen incident investigations in financial operations take two days of senior engineering time because the logging was insufficient to explain what the agent had done.
A Practical Way to Budget for This
When I scope a production agent deployment now, I use a rough multiplier. Whatever the token cost estimate is for steady-state operation, I multiply it by six to get a starting point for total operating cost. That multiplier covers evaluation infrastructure and labor, monitoring tooling and review time, integration maintenance, human review workflow, and incident response capacity.
That number is not precise. It will be higher in heavily regulated sectors like finance where audit requirements add overhead. It will be lower in more contained use cases with clean data and simple integrations. But it is a better starting point than token cost alone, and it forces the right conversations early.
The practical takeaway is this. Before you approve a production agent deployment, ask for a cost breakdown that includes evaluation, monitoring, integration maintenance, and human review labor. If those line items are missing from the estimate, the estimate is wrong. The token bill is the easy part. Everything else is where the real budget goes, and where the real work of making an agent actually useful happens.