Launch is Monday, and a contact centre lead has a voice agent that sounds fine in the demo. Should she run 500 fake calls or try 200 real customers? For simulated caller testing vs live pilot testing for AI voice agents, use both, simulation first.
Simulated vs live pilot: side-by-side table
In simulated caller testing, one AI plays the caller and another AI runs your voice agent. You script hundreds of calls, run them in minutes, and read the results before any customer is involved. In a live pilot, real customers get real calls from the agent, but only a small sample of them.
| Simulated caller testing | Live pilot testing | |
|---|---|---|
| Cost | Low. Build personas, score, re-run | Higher. Call minutes, human review |
| Risk | No customer exposed | Real customers hear mistakes |
| Coverage | Hundreds of edge cases, repeatable | Only what callers happen to say |
| Realism | Scripted, polite, on topic | Real noise, moods and surprises |
| Speed | Hours | Days to weeks |
| What it finds | Logic gaps, bad wording, unsafe replies | Answer rates, drop-off, how it sounds |
Cost: simulations are cheap, pilots cost trust
Simulation costs mostly people time, and a pilot costs money plus the risk of a bad call. Run in text or in a browser, a simulation needs almost no telephony spend.
Proof matters because good intentions do not guarantee returns. Gartner looked at 432 AI customer-service use cases. Only a quarter showed positive ROI, a quarter were negative, and 42% could not be measured. Whichever method you pick, decide what you will measure before you start.
Simulation cost drivers
- Building personas and scenarios.
- Scoring each run.
- Re-running the whole set after every prompt change.
Pilot cost drivers
- Per-minute call charges.
- Human review time for every transcript.
- The cost of one bad call reaching a customer, which is hard to price and easy to remember.
Set both against your own numbers. Buyers who need a business case set a vendor's cost against what it costs them to acquire or serve a customer. Ask for a price range early, in writing, so you can compare vendors and brief your team. Banks go further. They will not pay to find out whether it works, so they want a short pilot on a small sample, with payment only after a measured lift.
A worked example. Say a collections lead plans 200 contacts at 2 to 4 minutes a call. That is about 400 to 800 call minutes at most. Now multiply 200 by your own answer rate to see how many real conversations you will get. As an illustration only, an answer rate of 40% would give about 80. Check whether that is enough to see each key case a few times. If not, add contacts, or fill the gap with simulated runs first.
Risk and coverage: what each method misses
Simulation misses real callers, and a pilot misses rare cases. Each one covers the other's blind spot.
Risk. A simulation puts no customer in harm's way. That makes it the right place for dangerous cases: an angry caller, a disputed charge, a vulnerable customer, a prompt injection attempt. A pilot exposes real people to every mistake the agent makes. Regulated buyers name an AI agent that goes rogue or makes a public mistake as their main blocker. They want proof from real deployments, not a better demo.
Coverage. Simulation can run hundreds of edge cases, then run the same case again after every fix. A pilot covers only what callers happen to say. A rare case, such as a caller in distress, may never turn up in 200 calls.
Rules for live calls. Real customers bring legal duties. Here are three examples from different regions:
- EU: AI Act Article 50 has applied since 2 August 2026. Any AI that talks to people must say it is AI from the first interaction.
- US (federal): The FCC's 2024 ruling confirms that TCPA rules apply to calls using AI-generated voices. That means getting consent before you call.
- US (Colorado): The Chatbot Safety Act, effective 1 January 2027, requires public-facing conversational AI to disclose that it is AI.
Add these to your pilot checklist. Rehearse the disclosure wording in simulation first.
Handover. 87% of customers say a company using GenAI for service must offer a way to reach a human, according to Gartner. Simulation checks the trigger and the words. A pilot shows whether a human really picks up. Buyers ask: is the handover polite, what happens to a disputed amount, and does anyone chase a handover nobody answered? Put those three questions in your test plan.
Realism: only real callers surprise you
A simulated caller is built from a script or a persona, so it is polite, fluent and on topic. Real callers are not. Simulation tends to miss background noise, strong accents, people talking over the agent, long silences, dropped lines, and the caller who asks how you got their number.
A pilot shows what simulation cannot. It gives you answer rates, the point where people hang up, the best time of day to call, and whether customers think the agent sounds fake. Buyers fear AI that sounds scripted, and they keep asking what happens when the customer goes off script. Local accents and languages also change how much people trust what they hear.
You can close part of the gap. Build your simulated personas from real call transcripts, then score them on the same QA criteria you use for live calls. A failure in simulation then means what a failure in production would mean.
Insight7's voice campaigns are in beta. You can test a campaign's agent by talking to it in the browser, with no phone number needed. A test call to your own phone is also in the beta spec. While a campaign runs, a dashboard shows calls in progress, and a reviewer can read each transcript when the call ends. Each call can then be scored against a weighted scorecard your team owns.
When to use each, and in what order
Run simulation first, then a small pilot, then a staged ramp. Each step answers a question the one before it cannot.
Pick simulation first when you:
- change a script or prompt
- add an edge case
- rewrite compliance wording
- set handover rules
- are about to ship any release
Pick a pilot first when you:
- need real answer rates, tone and conversion numbers
- must win sign-off from a risk team that wants proof from live calls
The sequence
- Internal test calls by your own staff.
- Simulated stress tests, including the dangerous cases.
- A small pilot with a human watching.
- A staged ramp, in steps.
Set exit rules and pass criteria before the pilot starts. Name one reviewer. Add a manual kill switch. Pick three measures: handover rate, complaint count and QA scorecard score. Take thresholds from your human baseline, write down the pause points, such as a run of failed handovers, and ask your risk team to sign them.
Start with outbound, and pilot one use case. An outbound pilot has a controlled audience, and we advise starting there before inbound. Keep the sample small.
A pilot fits when:
- you need real-world numbers
- the script has passed simulation
- a human can watch every call
Related reading: AI Voice Agents vs AI Chatbots Compared for Customer Service Teams and Build vs Buy AI Voice Agents Decision Framework for Engineering and CX Leaders.
Questions people also ask
How do you test AI voice agents before launch?
Test in layers. Start with internal calls by your own staff, then run simulated callers on edge cases and unsafe requests. Finish with a small pilot that a human watches. Set pass criteria before each step.
What is simulated caller testing for voice AI?
One AI plays a customer and follows a persona or script, while another AI runs your agent. Each run is scored and can be repeated. It suits rare, risky and repeat-after-fix cases.
How long should a voice AI pilot run?
Run it until each key case has appeared several times, not for a fixed number of days. Cover a full calling cycle, such as weekdays and weekends. Use exit rules to stop early if results turn bad.
How many calls do you need in a voice agent pilot?
Work backwards from your rarest important case. Say it appears in 1 of 20 conversations. Then about 100 conversations would show it roughly five times. Divide by your answer rate to get contacts.
Can simulated calls replace a live pilot?
No. Simulation cannot show real answer rates, drop-off, background noise or whether the voice sounds fake. Risk teams also want proof from live calls.
How do you test AI to human handover on a voice agent?
Simulate the triggers first: a caller asks for a person, a disputed charge, a distressed customer. Then, in the pilot, have staff make test calls. Check that a human picks up, sees the context, and is polite.
Do you need consent to test a voice AI agent on real customers?
Often yes. A pilot on real customers is live calling. In the US, AI-generated voices fall under TCPA consent rules. In the EU, the agent must say it is AI. Ask your legal team to confirm.
Break it, prove it, then ramp
Insight7 can score every call against a weighted scorecard your team owns, so pilot results are measured, not guessed. Break it in simulation, prove it on a small pilot, and never let a customer find the first mistake.
