Do you build a new program for quality assurance on AI voice agents, or reuse the scorecard your human agents already face? It depends on which criteria measure the customer's outcome and which only measure human effort.

Start with the human scorecard you have

Use one scorecard for both, because it gives you a fair like-for-like comparison. If the AI and your humans face different rules, a score gap tells you nothing.

Sort every criterion into keep, rewrite or drop.

  • Keep the criteria that measure the customer's outcome. These are resolution, compliance disclosures and required-step capture, such as confirming a policy number or a payment date. A customer does not care who or what made the call.
  • Rewrite empathy and tone. A human "sounds warm" by choice. An AI "sounds warm" because of its prompt and voice. Score the same thing, but write the evidence in terms of what the customer heard.
  • Drop criteria that measure human effort. Typing speed, hold etiquette and fatigue-driven drift do not apply to an AI.

A human scorecard also never checks things an AI can get wrong, such as inventing a fee. The next section covers those.

You do not need a blank page. Insight7 has starter rubrics on the Criteria screen that give teams a ready-made scorecard. Teams can also import existing rubrics or build custom scorecards against company playbooks. Scoring then runs against a weighted scorecard the team owns, not a sample.

If you have call recordings but no rubric, the free Call QA Scorecard Builder builds one from real call data.

Add criteria only an AI agent needs

Add a short block of criteria that a human scorecard never checks, because an AI fails in ways people do not.

  • Says it is AI at the start. Score the first interaction, not the end of the call.
  • Never invents facts. A made-up fee or policy term is the worst failure. The agent should say it does not know and route the question.
  • Handles interruptions and silence. Check that it recovers when the customer talks over it or goes quiet.
  • Offers a human, politely. Buyers ask whether the handover is polite and whether anyone chases one nobody picked up.
  • Escalates sensitive or disputed amounts.
  • Sounds local and natural. Buyers fear AI that sounds scripted or robotic, so score pronunciation, accent and pacing.

Add a critical-error rule. A checklist score can hide what matters, so one critical error should fail the whole call. An insurer we spoke with scores a missed policy number as a zero for the whole evaluation. For an AI, make a compliance miss, an invented fee and a refused handover the same kind of zero.

Then move weight toward outcome and risk, and away from script adherence. An AI follows the script every time, so that criterion no longer separates good calls from bad.

Disclosure is a scored criterion, and the rules differ by region:

  • EU: AI Act Article 50 has applied since 2 Aug 2026. Any AI that talks to people, including phone systems, must say it is AI from the first interaction.
  • US: AI-voice calls need consent under the TCPA (FCC 2024 ruling). The proposed "say it is AI" rule was not final as of mid-2026.
  • Colorado: The Chatbot Safety Act takes effect 1 Jan 2027 and requires public-facing conversational AI to disclose it.

Gartner's 2026 survey of 3,566 customers found that 87% say a company using GenAI must offer a way to reach a human. Give the handover criterion real weight.

Sample calls: score all of them, review some

Score every AI call automatically, then put human listening on top. An AI agent makes each call the same way, so one bad prompt repeats across thousands of calls. Insight7 evaluates 100% of conversations instead of a 2% manual sample.

Choose which calls a person hears:

  • The lowest scores and every critical-error flag.
  • Every handover to a human, and every dispute.
  • Calls where only the AI spoke or the customer stayed silent. In Insight7's voice campaigns (beta), these are not counted as reached.
  • A random slice of average calls, to check the scorer itself.

Keep the scores clean. Voicemail, one-sided, hold, internal and too-short calls can be excluded, and calls can come in from an upload, Google Drive or Zoom.

Then calibrate. Start with 20 to 30 calls: have a human auditor score them and compare the results with the automated scores. Fix the criterion wording wherever the two disagree. One telehealth team found its medical-history criterion kept scoring zero when the patient's answer came later or in different words. Write criteria so a topic covered later still counts.

Add a human-agent baseline, with the same period and the same call type. Without it, you cannot tell whether an AI score of 82 is good.

Gartner's analysis of 432 AI service use cases, as reported by CX Dive, found that only a quarter produce a return. Another 42% cannot be measured. Scoring every call gives you that proof.

Score AI and humans side by side

Run the same weighted scorecard on both groups, then compare by criterion before you look at the total. Two totals of 82 can hide very different calls. The AI may win on disclosure and required steps, where it never forgets. It may lose on empathy and on questions it was not scripted for.

Keep the comparison fair:

  • Match call types. Compare payment reminders with payment reminders, and complaints with complaints. A mixed pool punishes whoever gets the harder calls.
  • Roll scores up three ways. Look at each agent (human or AI), each campaign and each call type.
  • Track the trend after every change. Each prompt or script edit should show up as a before and after on the criteria it touched.

Pick the behaviours you compare with care. Our analysis of 6,209 sales conversations found that the top 6.9% of performers ask 37% more questions per call and score 68% higher on empathy and rapport. AI agents are most likely to trail a strong human on those criteria, so give them their own lines on the scorecard.

Insight7 scores calls against a weighted scorecard your team owns, and scores roll up to the person, the project and the organisation. Its call scoring has built-in calibration, so managers can review, override and fine-tune ratings. Compliance and QA teams also get an audit trail for scored calls.

When the auto score and your human auditor disagree, let the auditor override it. Then review the pattern in calibration and fix the criterion wording, not just the one score.

Act on results every week

Set alerts first, so bad calls find you. Insight7 alerts can watch keywords and low scores, send a weekly digest of the weakest calls, and notify by email or Slack. Pick your own threshold from your baseline.

Then fix the AI one cause at a time. Map each low criterion to a likely cause:

  • Prompt: the agent skips or garbles a required step.
  • Knowledge gap: it invents or dodges an answer.
  • Voice: the accent or pacing sounds off.
  • Handover rule: it fails to offer a human.
  • Telephony: silence or dropped audio.

Change one thing, then re-score the next batch of calls. If the criterion moves, keep the change. If not, you guessed the wrong cause.

Set a stop rule before launch. If the critical-error rate passes your limit, pause the campaign and roll back the last change. Name one owner.

Because the scorecard is shared, the findings also feed human coaching. A weak score on a real call can become a practice session, hidden from the learner until a manager approves it. Buyers want proof that coaching changed behaviour, so aim practice at the gap your QA data shows.

Last, track the business metric, such as resolution, conversion or complaints. Buyers want a measured lift, not a pretty QA score.

Related reading: Human-in-the-Loop Improvement Workflow for AI Agents With Approval Steps for Script Changes, AI Voice Agent QA Scorecard Template With Criteria for Accuracy, Compliance and Customer Experience and How to Improve AI Voice Agent Performance From Real Customer Conversations.

Questions people also ask

Can you use the same QA scorecard for AI voice agents and human agents?

Yes, for the criteria that measure the customer's outcome. Keep those weights identical so scores stay comparable. Add a separate AI block and a critical-error flag, and version the scorecard so every change is traceable.

How do you QA an AI voice agent?

Score every call automatically. Have a person listen to flagged calls and a random slice. Calibrate against a human auditor, then re-score after each fix. Before launch, you can talk to a campaign agent in the browser with no phone number (beta).

How many AI voice agent calls should you review?

Score all of them, and have a person listen to every flagged call. Add a random slice of average calls that one auditor can finish in a week. Widen it after any prompt change, then shrink it as scores stabilise.

What should a QA scorecard for AI voice agents include?

Include the shared core: resolution, compliance disclosures and required steps. Add the AI block and critical-error flags. Write each criterion as a clear pass or fail with transcript evidence, so the auto scorer and the auditor agree.

How do you measure AI voice agent performance?

Track criterion scores, critical-error rate, resolution and handover rate. Read handover rate both ways. Too low can mean the agent traps callers, and too high can mean it resolves too little.

Do AI voice agents have to say they are AI?

Increasingly, yes, and rules differ by region. Put the disclosure in the agent's first sentence, such as "This is an automated assistant." Then one scorecard rule works everywhere. Ask your legal team which markets apply.

How often should you review AI voice agent calls?

Review flagged calls weekly, and daily after launch or any prompt change. Run a calibration against your human auditor monthly, or sooner when the two start to disagree.

One scorecard, a few new rules

One scorecard keeps AI and humans honest once you add AI-specific criteria and a critical-error rule. This week, have a human score 20 recent AI calls. Compare those scores with your automated ones.

Book a demo of Insight7