The short answer
Your assistant can sound helpful and still give a customer the wrong answer. It might promise a service you do not offer, miss the point of a complaint or contradict what it said earlier. A polite answer alone is not enough.
Once a month, I test your website or phone assistant as six kinds of customer. I ask ordinary questions, record the exact words it uses and explain what needs changing. The first test is free. You get a clear record that you can give to whoever looks after the assistant.
1. Agree what a correct answer looks like
Start with your current service details and the questions customers normally ask. I set down what each answer must contain before testing. If your own information is unclear, that needs resolving before an answer can fairly pass or fail.
2. Ask as six kinds of customer
I act as a first time buyer, a customer with a complaint and someone in a hurry. I also act as someone who writes badly, someone asking about something your firm does not do and a returning customer who refers to an earlier answer.
The questions use everyday language. A spelling mistake or a short, rushed message is ordinary customer behaviour. Each kind of customer tests whether the answer remains useful when the question arrives in a different form.
3. Keep the exact answer
For each question, I record what a correct answer must contain and what the assistant actually says. A pass means it meets that written requirement. A catch means there is something to fix. The report includes the exact words, so your team can check the judgement.
4. Say what to change
Each catch comes with a specific change to consider. That might mean correcting a service detail, asking a missing question or sending an unclear request to a person. The report points to the answer that needs attention, rather than leaving you with a general score.
5. Repeat the check each month
The next monthly test checks earlier catches and ordinary questions against your current information. This helps distinguish an unresolved issue from a new one. A pass applies to the question asked at that time. It does not establish that every possible answer is right.
The blank test sheet
Write the expected answer before asking the question. Use one row per question and keep the assistant response exactly as given.
| Kind of customer | Question asked | What a correct answer must contain | What the assistant said | Pass or catch | What to change |
|---|---|---|---|---|---|
Download this sheet as a file for Excel
A made up report for Northfield Heating
Made up example. All rows are made up. Northfield Heating is an invented firm. Its service details and every assistant answer here are fictional examples of the method.
| Kind of customer | Question asked | What a correct answer must contain | What the assistant said | Pass or catch | What to change |
|---|---|---|---|---|---|
| First time buyer | Can I ask for a boiler service? | Say a boiler service enquiry is accepted and ask for the postcode. | Yes. What is the postcode for the property? | Pass | |
| Customer with a complaint | Nobody has replied to my complaint. Who can help? | Offer a route to a person without saying the complaint is resolved. | Please choose the contact option to ask a person to review your complaint. | Pass | |
| Someone in a hurry | Can you promise an engineer today? | Explain that a person must confirm availability. | Yes, an engineer will come today. | Catch | Remove the promise and ask a person to confirm availability. |
| Someone who writes badly | boiler servce can u do it | Recognise the request for a boiler service and ask for the postcode. | Do you mean a boiler service? What is the property postcode? | Pass | |
| Someone asking about something the firm does not do | Can you repair my washing machine? | Explain that washing machine repairs are outside the service. | Yes, we repair washing machines. | Catch | Correct the service information and state the limit. |
| Returning customer referring to an earlier answer | You asked for my postcode earlier. Why do you need it? | Explain that the postcode is used to check the service area. | It helps check whether your property is in the service area. | Pass | |
| First time buyer | Can I book before anyone checks my address? | Explain that service area eligibility must be checked first. | Your address needs checking before a booking can be confirmed. | Pass | |
| Returning customer referring to an earlier answer | Earlier you said someone must confirm the visit. Is it booked now? | Keep the earlier limit and explain that confirmation is still needed. | Yes, your visit is booked. | Catch | Keep the booking status consistent and require confirmation from a person. |
Download this sheet as a file for Excel
Ordinary questions only
I never try to break, trick or attack an assistant. I use only ordinary customer questions. If a member of staff joins the chat, I stop at once. The test is about the answers the assistant gives, and ends when a person takes over.
When this is the wrong fit
An assistant that only shows fixed menu buttons is the wrong fit. There is no open question for it to answer. An internal tool with no customers also needs a different test based on the work its users do.
If you want a security test, that is a different job done under a written agreement. This monthly service checks normal customer conversations. It does not assess whether someone can gain access or misuse the system.
Where to start
Choose the assistant that customers already use. Begin with questions your team can answer clearly from current information. Use the blank sheet to agree the expected answers. The made up report shows how a useful catch connects an exact answer to a proposed change.