The Model Is Not Your Problem
Every order entry automation project I have seen fail in the last three years failed at the same place. Not the model. Not the infrastructure. Not the change management. It failed because nobody could tell the agent what a valid order line actually looked like, and nobody had mapped the customer's part numbers to the internal catalogue before the first invoice ran through production.
I have watched a distribution business spend four months selecting an AI vendor and six weeks on integration, then discover during UAT that their top forty customers each used a different naming convention for the same SKU family. One customer called it a 'hex bolt M8x25 zinc'. Another called it 'fastener 8mm 25mm ZP'. The internal catalogue had a third string entirely. The agent matched on partial text and got it wrong at a rate that would have caused real financial exposure if anyone had let it run unsupervised. The project paused for another three months while the data team built a synonym library that should have existed on day one.
That is not an AI failure. That is a scoping failure. And it is the most common one I see.
What Catalogue Mapping Actually Means in Practice
Catalogue mapping is the process of building a durable, auditable link between every way a customer might describe a product and the single canonical record in your system. It sounds straightforward. It is not.
In a mid-size industrial manufacturer I worked with, the product catalogue had accumulated over twelve years of mergers. The same physical component existed under three legacy part numbers depending on which acquired business originally stocked it. Customer purchase orders referenced whichever number their procurement team had saved in their ERP years ago. The automation agent had to resolve incoming line items to a single master record before it could do anything useful, and that resolution required a mapping table that nobody had built, maintained or even fully documented.
The practical work here is not glamorous. It involves pulling every distinct product description string from the last two to three years of inbound orders, clustering them by similarity, and then having a domain expert validate each cluster against the catalogue. You are building ground truth. You are teaching the system what counts as a correct match before you ask it to match anything autonomously. In that manufacturer project, this exercise surfaced over two hundred part number conflicts that the business had been resolving manually and invisibly for years. The automation project forced them into the open. That is actually a benefit, not a complication, but you have to budget for it.
Line Matching Is Where Accuracy Lives or Dies
Once you have a catalogue you trust, line matching is the next hard problem. A purchase order line is rarely clean. It carries a quantity, a unit of measure, a product description of variable quality, sometimes a customer part number, sometimes a price, and sometimes a note that contradicts all of the above.
In a food distribution context I worked in, inbound orders from smaller customers frequently had unit of measure mismatches. A customer would order in cases when the catalogue priced in eaches, or order in kilograms when the system expected units. The agent could extract the line correctly and match the product correctly and still produce a wrong order if the UOM conversion was not handled explicitly. We built a UOM normalisation layer as a pre-processing step before any model inference ran. That layer alone reduced line-level errors by more than any prompt tuning we did on the extraction model.
The lesson is that line matching accuracy is a function of data preparation, not model capability. You can improve a model's extraction of ambiguous text by a few percentage points with better prompting. You can improve its matching accuracy by twenty or thirty points by cleaning the catalogue and building explicit normalisation rules. The leverage is in the data layer.
How to Scope It So Accuracy Is Provable
Provable accuracy is not a nice-to-have. It is the thing that gets an operations leader to sign off on autonomous processing and stops the project from living in pilot purgatory forever. Here is how I scope it.
First, define your accuracy metric at the line level, not the order level. An order with fifty lines where one line is wrong is not a successful order. Finance and operations teams know this. The agent needs to be evaluated the same way. Agree on a line-level match rate threshold before you build anything.
Second, build a labelled evaluation set from historical orders before you touch a model. Pull three to six months of orders that were processed correctly by your team, extract the raw input documents, and treat the correct processed output as your ground truth. This set is your benchmark. Every version of the system gets scored against it. If you cannot build this set, you cannot prove accuracy, and you should not go live.
Third, segment your evaluation set by customer and by order complexity. A system that is ninety-four percent accurate overall but sixty percent accurate on your ten largest customers is not ready. Segmented accuracy tells you where the data problems are concentrated so you can fix them before they become production incidents.
- Define line-level accuracy thresholds with the business owner before scoping the build
- Build a labelled historical evaluation set as the first technical deliverable
- Segment accuracy by customer tier and order complexity, not just overall rate
- Treat catalogue mapping and UOM normalisation as engineering work, not configuration
- Set a confidence threshold below which the agent routes to human review, and measure the volume hitting that threshold weekly
The Confidence Threshold Is Your Safety Net
No model gets every line right. Anyone who tells you otherwise is selling you something. The practical answer to model uncertainty is a well-designed confidence threshold that routes low-confidence lines to a human reviewer rather than letting them fail silently or process incorrectly.
In a field operations business I worked with that processed equipment rental orders, we set the initial autonomous processing threshold conservatively. About thirty percent of lines went to human review in the first two weeks. That sounds high, but it was intentional. We used that review queue as a live data collection mechanism. Reviewers corrected the agent's output, those corrections fed back into the catalogue mapping and the synonym library, and within six weeks the human review volume had dropped to under eight percent. The threshold approach let us go live quickly and improve continuously without exposing the business to a wave of errors on day one.
The key is that the threshold is not a static setting. It should be reviewed weekly in the first quarter of operation and adjusted as the underlying data quality improves. Treat it as an operational metric, not a configuration parameter you set once and forget.
What Finance and Manufacturing Get Wrong About This Differently
Finance operations teams tend to underestimate the catalogue problem because they think of order entry as a document processing challenge. Extract the data from the PDF, push it to the ERP, done. They invest heavily in the extraction layer and are then surprised when matched records are wrong even though the extraction was perfect. The data was extracted correctly. It just could not be matched to anything real because the mapping did not exist.
Manufacturing operations teams tend to underestimate the UOM and specification problem. A part number match that ignores finish, tolerance or revision level is not a match. In precision manufacturing, an agent that matches on part number but misses a revision suffix can create a quality incident, not just an order error. The domain knowledge about what constitutes a valid match has to be encoded explicitly. The model will not infer it from the part number string alone.
Both sectors share the same root cause. They treat the data infrastructure as something that will sort itself out during the build. It will not. It needs to be the first workstream, not a dependency that gets resolved under pressure during testing.
The Practical Takeaway
Before you write a single line of agent code or sign a contract with a vendor, spend two weeks doing this. Pull every distinct product description string from your last two years of inbound orders. Pull every part number, every UOM, every customer alias. Map them to your catalogue. Find the gaps, the conflicts and the ambiguities. Count them.
That count is your real project scope. If you have three hundred unmapped variants, your build is three months. If you have three thousand, it is longer and you need a dedicated data owner, not just an engineer. The model is the easy part. The data is the work.
Order entry automation that actually runs in production and earns trust from the operations team is built on a catalogue you can defend and a matching logic you can explain. Everything else is a demo.