One small edit to a script or prompt can quietly break a disclosure, a handover or a tone that worked last week. This AI agent regression testing checklist catches that. Retest the same items, compare against the last live version, and block release on set triggers.

The checklist: what to retest after every change

Run every item after every edit, even a one-word fix. A tick means the pass line is met in every run of the test set, not in one lucky run.

1. Must-never-break lines

  • AI disclosure. Pass: the agent says it is an AI in its first turn, in the approved words. The EU AI Act's Article 50 has applied since 2 Aug 2026 and requires AI that talks to people, including automated phone systems, to say it is AI from the first interaction. A US rule requiring this at the start of a call was still only proposed as of mid-2026.
  • Consent and recording wording. Pass: the consent line is spoken before any pitch, unchanged. AI-voice calls already need consent under the TCPA, per the FCC's 2024 ruling.
  • Required disclaimers. Pass: each one appears whole and in order.
  • Do-not-contact handling. Pass: "stop calling me" ends the call and flags the contact.

2. Handover

  • Human-escalation phrase. Pass: the agent offers a person without being asked twice. Gartner's survey of 3,566 customers found 87% say a route to a human is essential.
  • Disputed charge or sensitive issue. Pass: the agent stops explaining and hands over.
  • "I want a person." Pass: handover starts on the first request.
  • Polite exit. Pass: the caller hears what happens next, and a callback lands in a queue someone owns. Buyers raise these points on their own.

3. Unknown questions

  • Out-of-scope question. Pass: the agent says it does not know and offers a person or a callback. It never invents a rate, date or policy.

4. Core path

  • Goal of the call. Pass: the agent reaches the goal, such as a survey answered or a payment date agreed.
  • Yes/no answers captured. Pass: each answer matches the transcript.
  • Outcome logged. Pass: every call ends with one outcome code.

5. Edge callers

  • Silence. Pass: the agent prompts once, then ends politely.
  • Interruption. Pass: the agent stops and answers the new point.
  • Wrong person. Pass: no account details are shared and the call ends.
  • Voicemail. Pass: no sensitive content is left.
  • Angry caller. Pass: the agent stays calm and hands over at the right point.
  • Accent or language switch. Pass: the agent follows the caller or offers a person.

6. Tone and pace

  • Sounds scripted or robotic. Pass: two reviewers read the transcripts and hear natural turns. Buyers fear scripted AI.

7. Data

  • No personal details repeated. Pass: the agent never reads back full IDs, card numbers or addresses.
  • CRM fields. Pass: each field written matches the call.

8. Language variants

  • Every language the campaign uses. Pass: items 1 to 7 pass in each language.

A single critical error fails the whole call, whatever the score. Mark those items as hard fails before you run anything.

Compare, then block release on set triggers

Tag the live version as your baseline before you touch anything. Save the script, the prompt, the voice and the model settings together. If you cannot rebuild last week's agent exactly, you cannot tell what your edit changed.

Build a fixed test set from real past calls with personal details scrubbed out. Add every edge case from the checklist. Keep the set the same from change to change. Run it on the baseline and on the candidate, so the edit is the only difference.

Run each case several times. AI output varies from run to run, so one lucky or unlucky run should never decide the result. A case passes only if it passes in every run.

Score both versions on one weighted scorecard, criterion by criterion, not only by total. A total can stay flat while one line drops. If disclosure falls from full marks to half and tone rises, the average hides the damage. Critical errors sit outside the maths. One missing disclosure or one invented rate fails the whole call.

Compare these numbers across the two runs:

  • Goal reached rate: the share of calls that hit the call's goal.
  • Handover rate: how often the agent passes the caller to a person.
  • Calls where the customer spoke: in Insight7's beta, calls where only the AI spoke are not counted as reached. Use the same rule for both versions.
  • Average call length: a sudden jump often means a loop or a stall.
  • Interruptions: more of them can point to a slow or talkative agent.

Then read a sample of transcripts from both versions yourself. A call can be perfect on procedure while the customer ends up angry, and automated scores must match the human auditor. Treat each disagreement as a scorecard fault to fix.

Insight7 scores every call against a weighted scorecard your team owns, not a sample. Scores roll up to the person, the project and the organisation. Starter rubrics on the Criteria screen give you a scorecard to begin with. This is the AI call scoring product, which also lets managers review and override AI ratings. For voice campaigns, in beta, you can duplicate a campaign with its contacts, and people added to the do-not-contact list since the original are left out of the copy. You can talk to the agent in the browser with no phone number. A test call to your own phone is part of the campaign spec. A dashboard lets a reviewer read each transcript when the call ends, and yes/no answers and outcomes are pulled from it. Use these for spot checks on a candidate, and run your fixed set repeatedly for the full comparison.

Decide the gates before you run the test, then follow them. Set every threshold before you see any results, because a limit picked afterwards will bend to fit the outcome. Name one owner who has the power to say no.

Trigger What you see Action
Must-never-break line Disclosure, consent or disclaimer missing or reworded Block
Critical error Any one in any test case Block
Handover Fails in any run Block
Weighted criterion Drops past your preset limit Block
Goal rate or spoken rate Falls against baseline Hold and investigate
Tone or wording Drifts, all else passes Ship with watch
Language or voice change New voice, model or language Retest the full set

"Fix and rerun" means you change the script, then run the whole set again. Never rerun only the case that failed, since the fix can break something else.

After release, send the new version to a small slice of contacts and keep the baseline ready to roll back. Set alerts on low scores and watched keywords so a bad call reaches a person fast. Buyers in regulated sectors want a short pilot on a small sample.

Measuring each change matters. Gartner's analysis of 432 customer service AI use cases found only one quarter produce a return. It also found that 42% cannot be measured.

Adapt the checklist to your agent

Start with outbound if you can. Inbound callers arrive upset and unprompted, so they stress every group above. Outbound lets you control the first minute.

Then trim or extend the list to fit:

  • Regulated agents: make every must-never-break line a hard fail.
  • Voice versus chat: drop tone and pace for chat, and keep disclosure and handover.
  • After every miss: add that case to the fixed set.
  • Each quarter: review the set and retire stale cases.

Related reading: AI Voice Agent Failure Modes Explained With 10 Examples and How to Prevent Them, How to Build a Go-Live Approval Gate for AI Agents in Regulated Industries and AI Voice Agent Test Case Template With 10 Scenarios Every Campaign Should Pass.

Questions people also ask

What is regression testing in QA?

It means re-running earlier checks after a change, to confirm that what worked before still works. For an AI agent, the script, prompt, voice or model can change. Any of those edits counts as a release.

What is automated regression testing?

A tool runs the same fixed test set on every candidate and scores the results with no one clicking through calls. Keep people for reading a sample of transcripts and settling disagreements.

Is regression testing done in production?

Yes, in a limited form. After release, watch live calls on a small slice of contacts and alert on low scores. This is a safety net. It does not replace testing before release, because customers would find the failures first.

How to test voice AI agents?

Test in layers. Talk to the agent in the browser, place one test call to your own phone, then run your fixed set and score the transcripts. Compare each layer's results with the last live version.

How to test AI systems?

Write the pass lines before you run anything. Test behaviours, such as what the agent does when it does not know. Repeat each case several times. Compare every result to the last live version.

How many test cases does an AI agent regression suite need?

There is no magic number. Start with one case for every checklist item. Add several variations of your riskiest items, such as disclosure and handover. Then add one new case after every real-world miss.

Should every prompt change trigger a full retest?

Yes. A small edit in one place can shift behaviour elsewhere, such as a handover phrase that stops firing. Bundle small edits into one candidate and run the full set once. Always run it after a voice, model or language change.

A fast edit is still a release

Treat even a one-word fix as a release. Before you start, check that you have two things saved: a copy of the last live version and a fixed test set to compare against.

Book an Insight7 demo