To measure whether an AI agent change helped, keep a random slice of customers on the old version. That holdout group is your baseline. Pick one metric before launch and run to the planned sample size. Judge the lift by its confidence interval.
Split traffic randomly and keep it stable
Randomise by customer ID, not by call. A customer who rings three times should meet the same agent version all three times. If you split by call, one person sees both versions, their results blur together, and the two groups stop being independent.
Assign with a salted hash. Hash the customer ID plus a test-specific salt and take the result modulo 100. Buckets 0 to 9 form the holdout. Everyone else gets the new version. The same ID always lands in the same bucket, so assignment stays sticky. Use a new salt for each test so groups do not carry over.
Pick the holdout size on purpose. Five to 20 percent is typical. A small holdout protects more customers from a weak change, but it fills slowly. By simple arithmetic, a 10 percent holdout takes five times longer to reach a given sample than a 50 percent one. Size it from your power calculation in the "Size the test, then run it untouched" section.
Check for sample ratio mismatch (SRM). If you planned 90/10 and the counts drift away from that, the split is broken. Run a chi-square test on the arm counts every day. A tiny p-value means stop and debug. Common causes are lost logs, bots and assignment bugs.
Keep everything else equal.
- No other prompt edits, campaigns or knowledge base updates touch either arm.
- Do-not-contact lists and consent rules apply to both arms the same way.
Choose the holdout type. A per-change holdout lasts for one test. A long-lived holdout stays on an old baseline for months. The "Decide, roll out, and keep a small holdout" section covers the long-lived kind.
Define one primary metric before you start
Write down one primary metric and the rule for acting on it before launch. Gartner analysed 432 AI customer service use cases. Only a quarter showed positive ROI, and 42% could not be measured at all. Fixing your measurement first keeps you out of that group.
Pick the outcome the change should move: issue resolved, payment made, funded account or booked meeting. Write the decision rule beside it, such as "ship if the lift is at least 2 points". Pull the outcome from a system of record like the CRM or billing table, not the agent's own call summary. Set an attribution window, for example 7 days after the contact, and use it in both arms.
Then add guardrail metrics. These catch a change that wins the primary metric by causing harm elsewhere:
- Escalation-to-human rate. Gartner found that 87% of 3,566 customers say a company using GenAI for service must offer a way to reach a human. A change that hides the exit can lift your primary metric and still lose trust.
- Complaint rate, opt-out rate and critical-error rate.
Next, add a quality metric by scoring every call against one weighted scorecard. Manual review samples about 2% of calls, which is too thin to compare two arms. A call can follow every procedure step and still leave the customer angry, so a plain checklist score can hide what matters. Let one critical error fail the whole call, and check before the test that automated scores match a human auditor.
Use the same scorer and rubric for both arms. Hide the arm label from human reviewers where you can.
Size the test, then run it untouched
Work out the sample per arm before launch. For a rate metric, use this formula: n per arm = (1.96 + 0.84)² × [p1(1−p1) + p2(1−p2)] ÷ (p2 − p1)². The 1.96 is for a 5% false-positive rate, and the 0.84 is for 80% power.
Here is a worked example. The figures below are calculated from that formula, not taken from a source. Say 20% of contacts resolve today and the smallest lift worth shipping is 2 points, so p1 = 0.20 and p2 = 0.22. The formula gives about 6,500 customers per arm. At a 90/10 split, the holdout needs those 6,500, so the whole test needs about 65,000 customers. A 50/50 split needs only about 13,000.
Fix the duration in advance. Divide the sample by your weekly volume, then round up to whole weeks so every weekday is covered twice or more. Do not stop early because the result looks good. Checking after every observation and stopping at the first significant result pushes the real false-positive rate well above the 5% you planned. This is the "peeking" problem, described in Evan Miller's essay on how not to run an A/B test.
Freeze everything but the change. Lock the prompt, model and knowledge base in both arms. Log the agent version on every call.
Watch only guardrails during the run. Check the SRM test and the harm metrics from the "Define one primary metric before you start" section. Agree a kill switch first, such as "stop if critical errors double".
Count reach the same way in both arms. Calls where only the AI spoke are not counted as reached. A carrier's temporary refusal does not use up a retry.
Insight7 can score every call against a weighted scorecard, not a sample, so both arms get the same scorer. In beta, a campaign dashboard shows calls in progress and lets a reviewer read each transcript. Any campaign can also be duplicated with its contacts, and anyone added to the do-not-contact list since the original is left out of the copy.
Read significance with the interval, not only p
Pick the test that fits the metric. For rates such as resolution or payment made, use a two-proportion z-test, or a chi-square test, which gives the same answer for two arms. For continuous metrics such as handle time or QA score, use a Welch t-test. If the data is skewed, use a bootstrap.
A p-value answers one question. If the change did nothing, how often would chance alone produce a gap this big? A p of 0.03 does not mean there is a 97% chance the change works. It also says nothing about how big the gain is.
So report the lift with a 95% confidence interval. For two proportions, take the difference and add or subtract 1.96 times its standard error. Take the earlier example, with 20% against 22% and 6,500 customers per arm. The standard error is about 0.7 points, so the lift is 2 points with an interval of roughly 0.6 to 3.4. The p-value is about 0.005, so the result is statistically significant. But the interval reaches below your 2-point bar, so the gain might not be worth shipping.
That gap is statistical versus practical significance. Decide using the interval and your pre-set minimum worthwhile effect. Ship when the whole interval clears it.
Handle the rest of the readout with care:
- Many metrics or segments: divide alpha by the number of tests (Bonferroni). Label any subgroup cut as exploratory.
- Guardrails: a primary win with a worse escalation or critical-error rate is not a ship.
- Small samples: the interval will be wide. Extend the test or call it inconclusive.
Decide, roll out, and keep a small holdout
Apply the rule you wrote before launch. Ship if the whole interval clears your minimum effect and no guardrail got worse. Stop if the primary metric fell or a guardrail broke. Iterate if the result is inconclusive, changing one thing at a time.
Roll out in stages, for example 25%, then 50%, then 100%. Watch the guardrails at each step.
Keep 1 to 5 percent of customers on the old version for months. This long-term holdout catches drift and novelty effects, where a gain fades once the newness wears off.
Log the result, the interval and the agent version, so the next change starts from a known baseline.
Related reading: How to Monitor AI Voice Agent Latency, Dead Air and Interruptions in Production, 8 Best AI Agent Monitoring and QA Tools for Contact Centers and How to Turn Failed Customer Calls Into Test Cases for Your AI Voice Agent.
Questions people also ask
What is a holdout group?
A holdout group is a random set of customers who keep getting the old agent while everyone else gets the change. It is also called a control group.
How big should a holdout group be?
Let the power calculation set it. Ten percent works when volume is high. If weekly volume is low, a 10% holdout may take months to fill, so move toward a 50/50 split.
How do you evaluate AI agent performance?
Compare the agent against a holdout on three things. Use one primary outcome from your system of record, a quality score on every call, and guardrails such as escalations and complaints. Never judge it on call summaries alone.
How long should a holdout test run?
Run until the smaller arm reaches its planned sample, rounded up to whole weeks. Then add your attribution window. If you count payments within 7 days of contact, the last customers need a week to mature.
What is a good p-value for an AI agent test?
Pick the threshold before launch. 0.05 is common. Use a stricter one for costly or hard-to-reverse changes, or when you test many metrics. A small p says the gap is unlikely to be chance, not that the gain is big.
What if my holdout and test groups end up different sizes?
Unequal sizes are fine if planned, such as 90/10. Check observed counts against the planned ratio with a chi-square test. A mismatch means a broken split. Fix it and restart.
Can I use a before-and-after comparison instead of a holdout group?
Use it only when you cannot split traffic, and treat it as weak evidence. Seasons, campaigns and customer mix shift alongside your change. A holdout lives through the same weeks, so those shifts cancel out.
A change has only worked if the group that got it beat a random holdout on the metric you named before launch. Insight7 scores every call the same way in both arms. Pick the metric first and split by customer.

