An A/B test of an AI voice agent runs two versions of one call element, such as a script, voice or offer. Both run at once on randomly split contacts from one segment. Change one variable and judge it by one metric.
Write one hypothesis and one metric
Start with a sentence in this shape: "If we change X, metric Y moves by Z, because…" For example: "If the agent opens with a 10-second greeting instead of a 25-second one, promise-to-pay rate rises by 2 points, because fewer people hang up before the ask." The "because" tells you what to look for in transcripts later.
Pick a primary metric that counts a real result. Good ones are a promise to pay, KYC completed, or an offer accepted. "Call answered" is a poor choice. A person can pick up and say nothing, or a carrier can connect a silent line. Insight7's beta campaigns don't count calls where only the AI spoke as reached. Use the same rule in your test, or both arms will include empty calls.
Tie the metric to the campaign goal. When you run a voice arm next to SMS, management will look at conversion, so conversion is the proof you need.
Then add guardrails, which are metrics that must not get worse:
- Opt-outs and do-not-call requests
- Complaints
- Handover requests to a human
- Call length
A variant that lifts conversion but doubles opt-outs is not a win. Insight7's beta dashboard pulls yes/no answers and call outcomes from the transcript, which gives you the primary metric and some guardrails in one place.
Change one thing per variant
Change one variable between your two arms, because otherwise you cannot say what caused the lift. If arm B has a new opening line and a new voice, a better result could come from either. Pick one test type per run:
- Script: the opening line, the order of questions, or the wording of the disclosure.
- Voice: the voice itself, the language, or the accent.
- Offer: the price, the deadline, or the incentive.
Local language is a strong voice variant. Insight7's beta voice campaigns can run in Yoruba and Igbo, and the manager picks the language before the voice. A lender could test an English voice against an Igbo voice on the same segment and read the difference in promise-to-pay rate.
Keep everything else fixed: calling hours, retry rules, the contact list and the agent persona. Insight7's beta campaigns have calling hours and retry rules. A campaign can also be duplicated with its contacts. After you duplicate, check in the app that the settings carried over, so both arms start the same.
Check each variant before you spend contacts on it. In the beta, you can talk to a campaign's agent in the browser with no phone number needed. A broken variant then fails in your own browser, not on a real customer.
Treat consent and AI disclosure as fixed in both arms. In the EU, any AI that talks to people, including automated phone systems, must say it is AI from the first interaction. In the US, AI-voice calls already need consent under the TCPA. You can test the disclosure wording, but both arms must meet the rule.
Split one segment at random
Assign contacts to arms by chance, and run both arms at the same time. If you run version A this week and version B next week, you test the calendar as well as the script. Pay days, weather and a bank holiday can all move answer rates.
Pick your segment first, such as lapsed borrowers with a loan balance under a set size. Then give every contact a random number and sort by it. The first half goes to A and the second half to B. Don't split by name, file order or sign-up date, because those carry patterns.
Build arm B by duplicating arm A. In Insight7's beta, a campaign can be copied with its contacts, and anyone added to the do-not-contact list since the original is left out of the copy. A carrier's temporary refusal does not use up a retry attempt, so a flaky network won't shorten one arm's attempts. Plan three to four attempts per contact in both arms, because many customers answer only on a second or third try.
Keep other channels out of the test. If SMS also reaches these people, you can't tell which channel caused a payment. One digital lender carved out a sublist of about a thousand people reachable only by voice AI, so any conversion from that group counted cleanly.
Finally, check the split worked. Compare the arms on prior balance, region and past answer rate. If they differ a lot, reshuffle before you dial.
Size the sample before you dial
Work out how many contacts each arm needs before you launch. For a yes/no result such as promise to pay, you need three inputs: your baseline rate, the smallest lift worth detecting, and the error rates you accept. The usual choices are a 5% significance level and 80% power. Evan Miller's sample size calculator does the maths, and the standard two-proportion formula gives a close answer.
A worked example: your lender's current script gets a promise to pay from 10% of contacts. You want to detect a rise to 12%. The standard two-proportion formula at 5% significance and 80% power gives roughly 3,800 contacts per arm, or about 7,700 in total. Treat these figures as illustrations and confirm them in the calculator with your own rates.
The number moves a lot with the size of the lift. By the same formula, and again as approximate figures:
- A lift from 10% to 14% needs about 1,000 per arm.
- A lift from 10% to 11% needs about 15,000 per arm.
So a small segment can only detect big changes. If your list is short, test bolder variants, such as a new offer rather than a reworded greeting. You can also run the test longer, or pool several waves of the same segment.
Count by contacts assigned to each arm, not calls answered. Your voice or opening line affects answering, so dropping unanswered contacts can skew the comparison.
Then fix your stop rule. Write down the sample size and the end date, and do not stop early because a dashboard looks good. Evan Miller explains that significance levels assume the sample size was fixed in advance. Stopping at the first significant reading inflates false wins.
Read results on every call, then decide
Compare the primary metric between arms with a two-proportion z-test or a confidence interval for the difference. Call it a win only if the interval sits fully above zero and every guardrail held. If B beat A by 2 points but the interval runs from minus 0.3 to 4.3, you have no result yet.
Then find out why it won. A rate tells you which arm did better. Transcripts tell you the reason, and your hypothesis said what to look for: tone, how the disclosure landed, how the agent handled objections. Reviewing every call, not a roughly 2% manual sample, shows what explains the gap. After that, read the best and worst calls of both arms.
Insight7's beta campaign dashboard shows calls in progress and lets a reviewer read each transcript when a call ends, with yes/no answers and outcomes pulled from the transcript. That lets you compare arms on disclosure wording or objection handling across the calls, not a few hand-picked ones.
Before you decide, check three things:
- Time and novelty: Did one arm get more evening calls, or a burst of early answers that faded?
- Guardrails: Opt-outs, handover requests, complaints and call length.
- Segments: Does the lift hold across regions and balance bands?
Then pick one outcome: ship, iterate or drop. Log the hypothesis, sample size and result. Run the winner on a fresh segment as a confirmation test.
Gartner found that of 432 AI customer-service use cases, only a quarter show positive ROI and 42% cannot be measured. A logged, tested result puts you in the group that can show it.
Related reading: Average Outbound Calls Per Hour in a Call Center Compared With AI Voice Agent Capacity, How to Read AI Call Campaign Results and Pick Customers for a Follow-Up Campaign and AI Voice Calls vs SMS vs Email for Customer Reactivation Campaigns Compared by Response and Conversion.
Questions people also ask
How long should an A/B test of an AI voice agent run?
Run it until you reach the sample size you set in advance, and for at least one full week so weekday and pay-day patterns hit both arms. Set the end date before launch.
How many calls do I need to A/B test a voice agent?
It depends on your baseline rate and the smallest lift you care about. Smaller lifts need far more contacts. Enter your baseline and target in a sample size calculator first.
Can I test two AI voices on the same customer list?
Yes. Keep the script, offer, calling hours and retries identical, and change only the voice. Insight7's beta lets the manager pick the language before the voice, so you can test languages the same way. Assign contacts at random.
What should I measure when testing voice AI scripts?
Pick one primary result tied to the campaign goal, such as a promise to pay or a completed KYC. Add guardrails for opt-outs, complaints and handovers. Don't count calls where only the AI spoke as reached.
Do I need to tell customers the AI voice agent is AI in both test arms?
Yes. Disclosure is a fixed rule, not a variable. In the EU, automated phone systems must say they are AI from the first interaction. In the US, AI-voice calls need consent under the TCPA. Test the wording only.
Can I A/B test voice AI against SMS?
Yes, but contacts who get both channels muddy the result. Give each channel its own random group, or keep a holdout list only voice can reach. Compare conversion per contact assigned, not per call answered.
How do I know if my A/B test result is statistically significant?
Run a two-proportion z-test, or build a confidence interval for the difference, once you reach the planned sample. If the interval sits fully above zero and guardrails held, the lift is likely real. Checking early inflates false wins.
Write the win down first
Insight7 can score every call against a weighted scorecard your team owns, so you can compare both arms on the same criteria. Before you dial, check that your win number, sample size and stop date are written down.

