You ship a prompt fix on Friday. Monday brings a complaint about the same call type, and you cannot show the old failure now passes. To turn failed customer calls into test cases for your AI voice agent, save each as a replayable test.

Flag failed calls with clear rules

Write down what a failed call is before you look for one. A short list of signal types works, and each one can be checked by code:

  • No customer speech, or silence after the greeting.
  • The caller asks for a human.
  • The agent loops or repeats itself.
  • The agent gives a wrong or invented answer.
  • The agent skips a required step.
  • A handover goes to a queue nobody picked up.
  • The caller hangs up early.

"Asked for a human" earns its place. Gartner's 2026 survey of 3,566 customers found that 87% say a human option is essential when a company uses GenAI. A caller who asks for a person has told you the agent lost them.

Next, add hard-fail logic. One critical error should fail the whole call, even when the checklist score is high. Skipping identity checks or inventing a balance are examples.

Then score every call, not a sample. Put threshold alerts on low scores and keyword alerts on phrases like "speak to someone" or "this is wrong". A sample misses the rare failures, and those are the ones you need as tests.

Expect a blind spot. A checklist can pass a call that still failed the customer. A telehealth QA team saw the AI score an assessment decently for covering every topic, but a human auditor marked it down because the assessor never followed up on a key answer. So add two more triggers: human spot checks and complaint tickets, each linked to its call ID.

The output is a flagged list. Each row holds a call ID, the signal type and a severity.

Extract the scenario from the transcript

Break each flagged call into five parts. A test needs all of them:

  1. Goal. What the agent was calling about, such as a reminder before a loan payment is due.
  2. Caller. Who they are and how they behave: hurried, angry, quiet, or quick to interrupt.
  3. Facts given. The details the caller stated and the account data the agent held.
  4. Failure point. The exact turn where the call went wrong.
  5. Expected behaviour. What a correct agent does at that turn.

Cut the call at the failure turn. Replay only the lead-up, then let the agent answer live. Everything after the failure was shaped by the bad reply.

Check the speaker labels first. If diarization swaps caller and agent turns, you will replay the agent's own words as the caller's. Read the failure turn by ear before you trust the label.

Redact before you store anything. Replace names, phone numbers, account numbers and amounts with placeholders such as {CALLER_NAME} and {AMOUNT_DUE}. Pattern matching can cut too much or too little around proper nouns, so have a person review a sample of the output.

Keep the caller's exact failure words. Add one or two paraphrases too, because real callers never repeat themselves word for word.

Here is an illustrative example. Say an agent calls to remind a borrower of a due payment.

  • Caller: disputes the amount and asks for a person.
  • Failure point: the agent restates the amount instead of acknowledging the dispute.
  • Caller words: "That figure is wrong. Put me through to someone."
  • Variant: "I don't owe that much. Can I talk to a real person?"
  • Expected: the agent does not repeat the disputed figure, offers a human, and hands over with the dispute noted.

Buyers often ask how a disputed amount is handled and whether anyone chases an unanswered handover. Write both into the expected behaviour.

Store each failure as a versioned test

Give every test the same record, so anyone can read it, run it and trace it back. Keep it as a file in version control, next to the agent's prompt.

Field What it holds
ID, owner, date added Who maintains the test, and since when
Source call ID Link to the real call, for audit
Agent version Prompt, model, voice and knowledge base it was written against
Persona, opening context Caller type, mood and the account data loaded
Caller turns The scripted lead-up, ending at the failure point
Pass criteria Checks a judge can answer yes or no
Severity, tags Hard or soft, plus the failure type

Write each pass criterion as a check, not a feeling. "Must offer a human." "Must not state an amount missing from the account data." "Must say it is AI in the first turn." A vague line like "handles the dispute well" cannot be graded the same way twice.

Split the checks into hard and soft rules. Hard rules cover compliance, disclosure and wrong facts, and only they block a release. Soft rules cover tone and pacing, and they report a score.

Tag every test by failure type, such as "human request" or "invented figure". Then you can run one slice after a change.

Make disclosure a standing check on every test. In the EU, Article 50 of the AI Act says people must be told at the latest at the first interaction that they are talking to AI. It has applied since 2 August 2026. In the US, the FCC's 2024 ruling puts AI-voice calls under the TCPA, which means consent is required.

Rerun the test and judge the result

Replay the lead-up turns, let the agent answer live, and score only that reply against the pass criteria. Feed the stored caller turns in order. Stop at the failure point and grade what the agent says next.

LLM agents answer differently on every run, so one green run proves little. As a starting point, run each test five to ten times, and tune that to your risk. For hard rules, require every run to pass. For soft rules, set a pass rate, such as 80% to begin with, and watch it over time.

Build the judge in two layers:

  • Rule checks for hard rules. Code can test whether the reply offers a human, contains an amount missing from the account data, or says "AI" in turn one.
  • A model grader for soft rules like tone. Give it the criterion and the reply, and ask for a yes or no with a reason.

Calibrate the grader before you trust it. Have a human auditor score a batch of replies, perhaps 30 to 50 as a starting size, then compare. Automated scores must match the human auditor. If they disagree, tighten the criterion wording.

Rerun on every prompt, model, voice or knowledge change, and nightly too. Nightly runs catch drift when a vendor updates a model under you. Make a hard-rule failure fail the build.

Test by voice early. Talk to the agent in a browser session before you use a phone number. Then place a test call to your own phone to check the telephony path.

Measurement is where many AI service projects fall down. Gartner's analysis of 432 use cases found that 42% of them could not be measured. A suite with pass rates gives you a number.

Close the loop on every release

Make this a weekly habit. Review the flagged list, add a handful of new tests (5 to 10 is a reasonable start, so adjust to your volume), and retire duplicates. Then track two numbers: the suite's pass rate and the repeat-failure rate. That second number counts how often a failure you already fixed comes back. If it falls, the suite is working.

Insight7 covers the call side of this loop. It can score every call against a weighted scorecard your team owns, not a sample. It can also alert on keywords and low scores by email or Slack. For voice campaigns, now in beta, a dashboard lets a reviewer read each transcript when the call ends. You can also talk to a campaign's agent in the browser.

Related reading: Containment Rate vs Resolution Rate vs CSAT for AI Agents Explained, Human-in-the-Loop Improvement Workflow for AI Agents With Approval Steps for Script Changes and How to Run Quality Assurance on AI Voice Agents Using the Same Scorecard as Human Agents.

Questions people also ask

How to test voice AI agents?

Replay stored scenarios. Feed the caller's turns, let the agent answer live, and score the reply against yes/no checks. Start by talking to the agent in a browser, then place a phone test.

What counts as a failed call for an AI voice agent?

A failed call hits any signal on your written list, or has one critical error, whatever the checklist score says. A call where only the AI spoke should not count as reached either.

How many test cases does a voice agent need before launch?

No standard number exists. Cover every hard rule and every failure type on your list with a few tests each. Then add new tests every week from real failures.

How do you test a voice agent that gives different answers each time?

Judge the reply against criteria, never exact wording. Run each test several times and set a pass rate. Hard rules need every run to pass. Soft rules, such as tone, can use a lower rate.

Should I use real customer calls as test data?

Yes. Real calls carry the phrasing and interruptions that invented scripts miss. Redact personal data first, then have a person check a sample, because automated redaction can cut too much or too little. Also check your consent and retention terms.

How often should I rerun voice agent tests?

Rerun after every prompt, model, voice or knowledge change, and nightly. Run the hard-rule tests before each campaign goes live. A vendor can update a model without telling you, and the nightly run catches that.

How do I know if my automated call score matches a human reviewer?

Have a human auditor and the automated scorer grade the same batch of calls. Compare criterion by criterion, not only totals. Automated scores must match the human auditor. Rewrite any criterion where they split.

Turn the next failure into a test

A failed call you have not turned into a test will come back. This week, pull the ten worst calls from the last seven days. Write each as a test with a hard pass rule, then rerun them after your next change.

Book a demo of Insight7