AI voice agent pilot success criteria are the numbers, agreed before launch, that decide if the agent is rolled out, fixed or dropped. One change matters most. Write the baseline, target, sample size and pass/fail line down before the first call.

Measure your baseline before any AI call

Pull a full quarter of current results for the same call type and the same audience. Take them from your CRM or dialer. A collections lead at a 200-agent lender would pull last quarter's pre-due-date reminder calls, not all outbound calls.

Capture five numbers:

  • Answer rate
  • Completion or conversion rate
  • Cost per completed outcome
  • Handle time
  • Complaint or opt-out rate

Define "reached" the way the pilot will. A call where only the AI spoke should not count as reached. If your history counts every connected call, your baseline will look better than the pilot can show.

History has a weakness. Seasonality and list quality shift results from month to month. A control group fixes this. Keep a slice of contacts on your human callers or your SMS flow, and compare the AI against that slice running at the same time. Use history as a sanity check, not the main yardstick.

Write the baseline down before the first AI call. After launch, the pre-AI number cannot be rebuilt. The risk is real. In a Gartner review of 432 AI customer service use cases, 42% had unclear ROI, with leaders saying they did not know the value produced.

Last, check that you can read outcomes at the call level. Each transcript should be open to a reviewer, with yes/no answers and outcomes taken from it. That makes your outcome definition checkable.

Size the sample so the result means something

Sample size depends on four inputs: your baseline rate, the smallest lift you care about, confidence and power. Common settings are 95% confidence (a 5% significance level) and 80% power. A university statistics guide says power should not fall below 80%. Rarer outcomes and smaller lifts both need more calls.

The standard two-proportion formula gives the reached contacts needed in each group:

  • n = (1.96 + 0.84)² × [p1(1 − p1) + p2(1 − p2)] ÷ (p2 − p1)²
  • p1 is your baseline rate and p2 is your target rate.
  • 1.96 and 0.84 are the standard values for 95% confidence and 80% power.

Look at the bottom of the formula. The lift is squared there. Halve the lift you want to prove and you need about four times the calls. A small pilot cannot prove a small lift. Aim at a lift big enough to see, or accept a wider range of possible results and plan to extend.

Use these rules when you run it:

  • Count reached contacts, not dials. Use the same "reached" definition as your baseline.
  • Split at random. Assign contacts to the AI arm or the control arm by chance, so list quality cannot tilt the result.
  • Keep both arms alike. Use the same time of day, list segment and retry rules.
  • Run in rounds. Read each batch to fix problems early. Take the final read only on the full sample.

This also fits how buyers think. Banks want a short, low-risk pilot on a small sample, with payment only after a measured lift. That is our advice too. Pick a segment where a big lift is plausible, such as early-stage reminders, so a small sample can still give a clear answer.

Write the pass, fail and extend thresholds

Set three zones in writing before launch, and give each one a decision.

  • Pass: The primary metric beats the baseline by at least the minimum lift. The lower end of the 95% range sits above zero. Every guardrail is green.
  • Extend: The lift is positive, but the range is too wide to trust. Add calls from the same split.
  • Fail: There is no lift, or any guardrail is breached.

Put it in one table so nobody argues later. The numbers below are an example.

Metric Baseline Target Guardrail limit Decision rule
Payments after reminder (primary) 10% 15% none Pass if lift is at least 5 points and the range's lower end is above 0
Human handover, polite set from control set before launch set before launch Any breach fails
Complaints and opt-outs set from control no rise set before launch Any breach fails
Consent and AI disclosure 100% 100% 100% Any miss fails

Add hard stops. A QA lead knows a checklist score can hide what matters, and one critical error should fail the whole call. Treat a compliance breach the same way, whatever the average says.

Name who signs off and the date of the read-out. Write down what a pass triggers: scale, price or scope. A champion then has a case to carry upstairs, since buyers set the vendor's cost against their own cost to acquire or serve a customer.

Leaders will want that proof. Gartner found customer service AI spending up 38%, while overall service budgets rose 2%.

Lock the rules and score every call

Freeze the script, calling hours, retry rules and consent marking before the first call. If you change one mid-pilot, restart the count. Otherwise you are comparing two different pilots.

Insight7's voice campaigns are in beta and switched on per organisation. A manager sets a campaign type, consent marking, calling hours and retry rules. You can test the agent by talking to it in the browser, with no phone number needed. Copying a campaign keeps its contacts. Anyone added to the do-not-contact list since the original is left out of the copy.

Two rules protect your counts. Calls where only the AI spoke are not counted as reached, and a carrier's temporary refusal does not use up a retry. A dashboard shows calls in progress, and a reviewer can read each transcript when the call ends. Each campaign call is held as a real conversation and scored when it ends. Every call can be scored against a weighted scorecard your team owns, not a sample. Insight7's AI call scoring evaluates 100% of conversations.

Related reading: Outbound Call Campaign Benchmarks for Answer Rate, Completion Rate and Conversion, 12 AI Voice Agent KPIs Growth and Customer Success Teams Should Track and How to Read AI Call Campaign Results and Pick Customers for a Follow-Up Campaign.

Questions people also ask

What are good success criteria for an AI voice agent pilot?

Good criteria name one primary metric tied to the campaign's business outcome, such as payments made, KYC completed or surveys finished. Add two or three guardrails, a sample size and written pass, extend and fail zones, all set before launch.

How many calls do you need for a voice AI pilot?

It depends on your baseline and the smallest lift you want to prove. Put both into the two-proportion formula above. Because the lift is squared, halving the lift you want to prove needs about four times the reached contacts.

How long should an AI voice agent pilot run?

Run it until you reach the planned sample of reached contacts. Cover at least one full weekly cycle. Check your own data for weekday patterns in answer rates, then make sure both arms see the same days.

How do you set a baseline for a voice AI pilot?

Run a live control group beside the AI arm. Have human callers or your SMS flow work a random slice of the same list, with the same hours and retry rules.

What should a pilot pass/fail threshold look like?

It should state a minimum lift, a confidence range whose lower end sits above zero, and a limit for every guardrail. Add one hard stop. A single compliance breach fails the pilot, whatever the average says.

How do you test a voice AI agent before going live?

Talk to the agent yourself first, in a browser or through a test call to your own phone. Listen for scripted-sounding replies and for how the agent hands over to a human.

What consent rules apply to voice AI outbound calls?

Rules vary by region, so check each market you call. In the US, AI-voice calls already need consent under the TCPA, following the FCC's 2024 ruling. In the EU, Article 50 requires AI that talks to people to say it is AI, and it has applied since 2 August 2026.

Put it in writing first

A pilot is proof only when the baseline, target, sample and threshold are written first. Insight7 scores every call against a scorecard your team owns, so the read-out rests on all calls. If the pilot hit your target tomorrow, what would you do on Monday?

Book a demo of Insight7