# A monthly test for your customer facing AI assistant

Once a month, I ask your website or phone assistant ordinary customer questions. You get what it got wrong, its exact words and what to change. The first test is free.

Updated 12 October 2026 . 5 min read . Alexey Shurov

## The short answer

Your assistant can sound helpful and still give a customer the wrong answer. It might promise a service you do not offer, miss the point of a complaint or contradict what it said earlier. A polite answer alone is not enough.

Once a month, I test your website or phone assistant as six kinds of customer. I ask ordinary questions, record the exact words it uses and explain what needs changing. The first test is free. You get a clear record that you can give to whoever looks after the assistant.

### 1. Agree what a correct answer looks like

Start with your current service details and the questions customers normally ask. I set down what each answer must contain before testing. If your own information is unclear, that needs resolving before an answer can fairly pass or fail.

### 2. Ask as six kinds of customer

I act as a first time buyer, a customer with a complaint and someone in a hurry. I also act as someone who writes badly, someone asking about something your firm does not do and a returning customer who refers to an earlier answer.

The questions use everyday language. A spelling mistake or a short, rushed message is ordinary customer behaviour. Each kind of customer tests whether the answer remains useful when the question arrives in a different form.

### 3. Keep the exact answer

For each question, I record what a correct answer must contain and what the assistant actually says. A pass means it meets that written requirement. A catch means there is something to fix. The report includes the exact words, so your team can check the judgement.

### 4. Say what to change

Each catch comes with a specific change to consider. That might mean correcting a service detail, asking a missing question or sending an unclear request to a person. The report points to the answer that needs attention, rather than leaving you with a general score.

### 5. Repeat the check each month

The next monthly test checks earlier catches and ordinary questions against your current information. This helps distinguish an unresolved issue from a new one. A pass applies to the question asked at that time. It does not establish that every possible answer is right.

## The blank test sheet

Write the expected answer before asking the question. Use one row per question and keep the assistant response exactly as given.

| Kind of customer | Question asked | What a correct answer must contain | What the assistant said | Pass or catch | What to change |
|---|---|---|---|---|---|
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |

[Download this sheet as a file for Excel](data:text/csv;charset=utf-8,%EF%BB%BFKind%20of%20customer%2CQuestion%20asked%2CWhat%20a%20correct%20answer%20must%20contain%2CWhat%20the%20assistant%20said%2CPass%20or%20catch%2CWhat%20to%20change%0D%0A%2C%2C%2C%2C%2C%0D%0A%2C%2C%2C%2C%2C%0D%0A%2C%2C%2C%2C%2C%0D%0A%2C%2C%2C%2C%2C%0D%0A)

## A made up report for Northfield Heating

**Made up example.** All rows are made up. Northfield Heating is an invented firm. Its service details and every assistant answer here are fictional examples of the method.

| Kind of customer | Question asked | What a correct answer must contain | What the assistant said | Pass or catch | What to change |
|---|---|---|---|---|---|
| First time buyer | Can I ask for a boiler service? | Say a boiler service enquiry is accepted and ask for the postcode. | Yes. What is the postcode for the property? | Pass |  |
| Customer with a complaint | Nobody has replied to my complaint. Who can help? | Offer a route to a person without saying the complaint is resolved. | Please choose the contact option to ask a person to review your complaint. | Pass |  |
| Someone in a hurry | Can you promise an engineer today? | Explain that a person must confirm availability. | Yes, an engineer will come today. | Catch | Remove the promise and ask a person to confirm availability. |
| Someone who writes badly | boiler servce can u do it | Recognise the request for a boiler service and ask for the postcode. | Do you mean a boiler service? What is the property postcode? | Pass |  |
| Someone asking about something the firm does not do | Can you repair my washing machine? | Explain that washing machine repairs are outside the service. | Yes, we repair washing machines. | Catch | Correct the service information and state the limit. |
| Returning customer referring to an earlier answer | You asked for my postcode earlier. Why do you need it? | Explain that the postcode is used to check the service area. | It helps check whether your property is in the service area. | Pass |  |
| First time buyer | Can I book before anyone checks my address? | Explain that service area eligibility must be checked first. | Your address needs checking before a booking can be confirmed. | Pass |  |
| Returning customer referring to an earlier answer | Earlier you said someone must confirm the visit. Is it booked now? | Keep the earlier limit and explain that confirmation is still needed. | Yes, your visit is booked. | Catch | Keep the booking status consistent and require confirmation from a person. |

[Download this sheet as a file for Excel](data:text/csv;charset=utf-8,%EF%BB%BFKind%20of%20customer%2CQuestion%20asked%2CWhat%20a%20correct%20answer%20must%20contain%2CWhat%20the%20assistant%20said%2CPass%20or%20catch%2CWhat%20to%20change%0D%0AFirst%20time%20buyer%2CCan%20I%20ask%20for%20a%20boiler%20service%3F%2CSay%20a%20boiler%20service%20enquiry%20is%20accepted%20and%20ask%20for%20the%20postcode.%2CYes.%20What%20is%20the%20postcode%20for%20the%20property%3F%2CPass%2C%0D%0ACustomer%20with%20a%20complaint%2CNobody%20has%20replied%20to%20my%20complaint.%20Who%20can%20help%3F%2COffer%20a%20route%20to%20a%20person%20without%20saying%20the%20complaint%20is%20resolved.%2CPlease%20choose%20the%20contact%20option%20to%20ask%20a%20person%20to%20review%20your%20complaint.%2CPass%2C%0D%0ASomeone%20in%20a%20hurry%2CCan%20you%20promise%20an%20engineer%20today%3F%2CExplain%20that%20a%20person%20must%20confirm%20availability.%2C%22Yes%2C%20an%20engineer%20will%20come%20today.%22%2CCatch%2CRemove%20the%20promise%20and%20ask%20a%20person%20to%20confirm%20availability.%0D%0ASomeone%20who%20writes%20badly%2Cboiler%20servce%20can%20u%20do%20it%2CRecognise%20the%20request%20for%20a%20boiler%20service%20and%20ask%20for%20the%20postcode.%2CDo%20you%20mean%20a%20boiler%20service%3F%20What%20is%20the%20property%20postcode%3F%2CPass%2C%0D%0ASomeone%20asking%20about%20something%20the%20firm%20does%20not%20do%2CCan%20you%20repair%20my%20washing%20machine%3F%2CExplain%20that%20washing%20machine%20repairs%20are%20outside%20the%20service.%2C%22Yes%2C%20we%20repair%20washing%20machines.%22%2CCatch%2CCorrect%20the%20service%20information%20and%20state%20the%20limit.%0D%0AReturning%20customer%20referring%20to%20an%20earlier%20answer%2CYou%20asked%20for%20my%20postcode%20earlier.%20Why%20do%20you%20need%20it%3F%2CExplain%20that%20the%20postcode%20is%20used%20to%20check%20the%20service%20area.%2CIt%20helps%20check%20whether%20your%20property%20is%20in%20the%20service%20area.%2CPass%2C%0D%0AFirst%20time%20buyer%2CCan%20I%20book%20before%20anyone%20checks%20my%20address%3F%2CExplain%20that%20service%20area%20eligibility%20must%20be%20checked%20first.%2CYour%20address%20needs%20checking%20before%20a%20booking%20can%20be%20confirmed.%2CPass%2C%0D%0AReturning%20customer%20referring%20to%20an%20earlier%20answer%2CEarlier%20you%20said%20someone%20must%20confirm%20the%20visit.%20Is%20it%20booked%20now%3F%2CKeep%20the%20earlier%20limit%20and%20explain%20that%20confirmation%20is%20still%20needed.%2C%22Yes%2C%20your%20visit%20is%20booked.%22%2CCatch%2CKeep%20the%20booking%20status%20consistent%20and%20require%20confirmation%20from%20a%20person.%0D%0A)

## Ordinary questions only

I never try to break, trick or attack an assistant. I use only ordinary customer questions. If a member of staff joins the chat, I stop at once. The test is about the answers the assistant gives, and ends when a person takes over.

## When this is the wrong fit

An assistant that only shows fixed menu buttons is the wrong fit. There is no open question for it to answer. An internal tool with no customers also needs a different test based on the work its users do.

If you want a security test, that is a different job done under a written agreement. This monthly service checks normal customer conversations. It does not assess whether someone can gain access or misuse the system.

## Where to start

Choose the assistant that customers already use. Begin with questions your team can answer clearly from current information. Use the blank sheet to agree the expected answers. The made up report shows how a useful catch connects an exact answer to a proposed change.

## Common questions

### Is the first test free?

Yes. The first test is free. It gives you a first report to review before deciding whether a monthly check fits your assistant.

### Can you test a phone assistant?

Yes. The service covers website and phone assistants. The questions stay within ordinary customer enquiries, with the same need to record the exact answer and assess it against your information.

### Does a catch mean the whole assistant is bad?

No. It identifies one answer that needs attention. Read the question, the expected answer and the exact response together before deciding what to change.

### Can my team use the sheet itself?

Yes. Agree the expected answers first and keep the wording of each response. The blank sheet gives your team the same structure for its own checks between monthly tests.

## Useful next[ServiceOne job, run for you every monthYou name one piece of repetitive work. I build the system that does it, run it every month for a fixed fee, and send anything unclear to a person on your team.](https://shurco.ai/guides/automation-run-for-you/)[GuideBest AI Agent and LLM Evaluation Tools in 2026Every eval platform promises to tell you whether your agent got better or worse. Here is what each one actually does, what it costs this month, and when it is the wrong buy.](https://shurco.ai/guides/best-ai-agent-evaluation-tools/)[Free toolWhat to automate first, a scoring toolPick a task your team repeats every week and answer five quick questions. You get a plain answer on whether it is worth automating, what helps and what to watch. Score up to four tasks and it tells you where to start, and you can keep the result as a private plan page.](https://shurco.ai/tools/what-to-automate-first/)

## More guides[GuideAI agent pricing in 2026, what vendors actually chargeVendors now charge for agents by the conversation, the outcome, the action or the credit. Here are the published prices, what each unit really means, and how to compare them on your own workload.](https://shurco.ai/guides/ai-agent-pricing/)[PlaybookHow to automate daily site reports across many sitesThe report takes days because the data lives in many places and someone has to chase it. Here is how to gather it in parallel, catch the numbers that disagree and produce a report a manager can check.](https://shurco.ai/guides/automate-daily-site-reports/)[PlaybookHow to automate quoting for a manufacturer or distributorSlow quotes lose work. There are three quite different kinds of quoting automation, and most buying mistakes come from picking the wrong kind. Here is how to tell which one your quotes need.](https://shurco.ai/guides/automate-quoting-for-manufacturers/)[PlaybookHow to automate sales order entry in Business CentralCustomers email orders as PDFs, spreadsheets and free text, and someone keys each line into Business Central. Here are the three ways to take that work away, with what Microsoft's own agent does and where it stops.](https://shurco.ai/guides/automate-sales-order-entry-business-central/)[PlaybookHow to automate timesheet processing for payrollTimesheets arrive as spreadsheets, PDFs, photos and scans, and payroll has a deadline that does not move. This is the order to automate it in, and the checks that stop a wrong payslip.](https://shurco.ai/guides/automate-timesheet-processing-for-payroll/)[ServiceOne job, run for you every monthYou name one piece of repetitive work. I build the system that does it, run it every month for a fixed fee, and send anything unclear to a person on your team.](https://shurco.ai/guides/automation-run-for-you/)

## Want this in your operation

I build and run production AI agents that take repetitive work off operational teams. [Tell me what your team spends too long on.](https://shurco.ai/#contact)

Web version https://shurco.ai/guides/ai-assistant-monthly-test/