TL;DR

  • This AI voice agent QA scorecard grades each call on Accuracy, Compliance and Customer Experience, with weights that add up to 100.
  • One critical error fails the whole call, whatever the score.
  • Score every call, not a sample. Check automated scores against a human auditor before you trust them.

The scorecard at a glance

Copy this 14-row table and score every call against it. Weights add up to 100. Every criterion names the evidence a reviewer can point to, so two people can check the same call and get the same result.

Group Criterion Weight What it checks Evidence
Accuracy (35) Right answer from approved knowledge 10 Answer matches the knowledge base or script Transcript vs source
No invented facts 9 No made-up price, policy or promise Transcript
Correct data lookup or capture 8 Names, dates and yes/no answers recorded correctly Transcript, CRM record
Intent understood, task done 8 Agent grasped the request and finished the job Transcript
Compliance (35) AI disclosure at the start 8 Says it is AI in the first moments First 15 seconds of audio
Consent and calling hours 8 Consent marked, call inside allowed hours, number not on do-not-call list Campaign log, timestamp
Identity check before sensitive data 7 Caller verified before any account detail Transcript order
No restricted claims 6 No guarantees or advice the business bans Transcript
Data handling and PII 6 Personal data not repeated or stored wrongly Transcript, audio
Customer Experience (30) Pacing, no talking over caller 6 Natural speed, handles interruptions Audio overlap
Silence and latency 6 Short gaps, no dead air Audio timestamps
Clear human handoff 7 Offers a human when asked, passes context Transcript, queue log
Tone 5 Polite, not scripted or robotic Audio
Call outcome 6 Caller got what they came for, left calm Outcome tag, sentiment

A few notes on using it:

  • Weights are a starting point. Calibrate them against your own risk and results. A later section shows how to shift them by campaign type.
  • Reached means a human spoke. Calls where only the AI spoke should not count as reached.
  • Campaign settings feed the evidence. Consent marking, calling hours and a do-not-call list sit in the campaign setup, so reviewers can check the log against the call.
  • Score every call. Sampling misses the rare failures, and full coverage replaces the roughly 2% manual sample most QA teams rely on.

Fail the call on critical errors

Set a short list of auto-fail items, and score any call that hits one as zero. A checklist score can hide what matters. A call can look perfect on procedure and still leave the customer angry or break a rule with legal risk.

  • No AI disclosure where the law or your policy requires it
  • Calling a number on the do-not-contact list
  • Sharing account data before identity is verified
  • A made-up policy, price or promise
  • Refusing or blocking a request for a human
  • Abusive or off-script content

Keep scoring the weighted rows on a failed call, because coaches still need them. Tag the call "critical error" and name the item, so your trend report shows which rule breaks most often.

A mid-sized insurer's QA team already works this way. In its spreadsheet scoring, missing the customer's policy number zeroes the whole evaluation, because the industry is heavily regulated. Pick the few steps your business cannot miss, and make each one a gate.

Use a 0 to 3 scoring guide

Score each row from 0 to 3, then turn the scores into points with one formula.

Score Meaning
0 Fail
1 Partial
2 Meets the standard
3 Strong

Anchor each level with a real example, so two reviewers read it the same way.

Criterion 0 1 2 3
Right answer from approved knowledge Wrong answer Right, but misses a key condition Right and complete Right, complete, in plain words
AI disclosure at the start Never said Said after the pitch began Said in the first moments First sentence, and the caller can ask for a human
Silence and latency Dead air the caller fills with "hello?" Several noticeable pauses Short, natural gaps Smooth throughout

The formula: criterion score ÷ 3 × weight, summed across all rows, gives a total out of 100.

Worked example, four rows:

  • Right answer, weight 10, scores 3. That earns 10.
  • Identity check, weight 7, scores 2. That earns 4.67.
  • Human handoff, weight 7, scores 1. That earns 2.33.
  • Silence and latency, weight 6, scores 2. That earns 4.

The four rows earn 21 of 30 possible points, or 70.

Not applicable rows: drop the row's weight and rescale. If the handoff never came up, remove its 7 points. The other rows earn 18.67 of 23, which is 81.

Suggested starting bands: 85 and above passes, 70 to 84 goes to coaching, and below 70 goes to review. Adjust them after calibration.

Score accuracy against approved sources

Judge every answer against the knowledge base or script, never against the reviewer's memory. Put each wrong answer into one of two buckets.

  • Wrong: the agent used real source material but got it wrong, such as a fee quoted from an old price list. Score the "right answer" row down.
  • Made up: the agent said something no source contains, such as an invented refund rule. Score the "no invented facts" row down. Treat it as a critical error if it is a policy, price or promise.

The split matters because the fixes differ. Wrong answers point to stale content. Made-up answers point to weak guardrails in the agent's prompt.

Next, check data capture. Compare each yes/no tag and call outcome with what the caller said, and spot-check a few tags per batch. Skip accuracy scoring on calls where only the AI spoke, because no caller answer exists to check.

Check compliance on every call

Tie each compliance row to the rule behind it, then score every call. A missed disclosure or a call to a blocked number may happen on only a few calls. A sample will skip most of them, and one miss can be a legal problem.

These rules are general information, not legal advice. Check them with your own counsel. Rules differ by region, and these are two examples:

  • EU: Article 50 of the AI Act says people must be told they are dealing with an AI system, at the latest at the first interaction. It has applied since 2 August 2026. This feeds the "AI disclosure at the start" row.
  • US: The FCC's 2024 ruling confirms that AI-generated voices fall under the TCPA. Callers need prior consent, must identify themselves and must offer an opt-out. This feeds the "consent and calling hours" row.

The "identity check" and "data handling" rows come from your own data policy.

Measure customer experience beyond politeness

Score what the caller felt, using signals you can see in the audio and the log. Tone alone is too soft to score. Four signals work well:

  • Latency and gaps: time from the caller's last word to the agent's reply.
  • Interruptions: how often the agent talks over the caller.
  • Repeated questions: the agent asks again for something the caller already said.
  • Handoff: the caller asked for a human and got one.

Handle the handoff row with care. Gartner's 2026 survey of 3,566 customers found that 87% say a company using GenAI for service must offer a way to reach a human. So score a blocked request as 0 and treat it as a critical error. Also score how the handoff feels. Buyers ask whether it is polite, what happens to a disputed amount, and whether anyone chases an unanswered handover.

Buyers also fear AI that sounds scripted. The real test is what happens when a customer goes off script. So add a call outcome row with a sentiment tag.

Calibrate scores against a human auditor

Trust automated scores only after they match what a human auditor would say. Run this check before launch.

  1. Have two reviewers score the same 20 to 30 calls, without seeing each other's marks.
  2. Compare the results criterion by criterion, then add the automated score as a third column.
  3. Where scores differ by more than one point, rewrite the anchor wording until both reviewers read it the same way.
  4. Re-run the check every month, because scripts, products and agent prompts change.

Watch for rows that score zero when a required topic comes up later or in different words. A telehealth provider found its medical-history criterion did this, scoring zero even when the patient's later answer satisfied it. Fix it in the criterion. Say the topic counts if it is covered anywhere in the call, in any wording, and list two or three example phrasings.

Adapt the weights to your use case

Shift weight toward the risk your campaign carries. Voice campaigns usually have a type, such as customer survey, lead follow-up, cold outreach or win-back. Start from these splits, then calibrate.

Campaign type Accuracy Compliance Customer Experience
Cold outreach, collections, reminders 25 45 30
Lead follow-up, win-back 30 30 40
Customer survey 25 25 50
Inbound support 45 30 25

Score call stages differently too. A welcome call, a sales call and a closing call need different criteria. Leave out stages that are not performance moments.

Run it on every call with Insight7

Insight7 scores every call against a weighted scorecard your team owns, not a sample, and your QA manager can enter this table as that scorecard. Starter rubrics on the Criteria screen give you a ready-made scorecard to adapt. When you import calls, you can exclude voicemail, one-sided, hold, internal and too-short calls. Alerts can watch for low scores and keywords, then notify you by email or Slack. In voice campaigns (in beta), a dashboard pulls yes/no answers and outcomes from each transcript, and you can test the agent in the browser with no phone number. To draft your rubric from real calls, try the free Call QA Scorecard Builder.

Related reading: Human-in-the-Loop Improvement Workflow for AI Agents With Approval Steps for Script Changes, How to Run Quality Assurance on AI Voice Agents Using the Same Scorecard as Human Agents and How to Improve AI Voice Agent Performance From Real Customer Conversations.

Questions people also ask

What should an AI voice agent QA scorecard include?

Include three weighted groups: Accuracy, Compliance and Customer Experience, each row naming its evidence. Add a critical-error gate that zeroes the call, a 0 to 3 scale with anchored examples, and pass bands. Fourteen rows cover answers, disclosure, consent, handoff, latency and outcome.

How do you score an AI voice agent call?

Check for critical errors first. If there are none, score each row 0 to 3, divide by 3, multiply by the row's weight, and add the rows to reach 100. Drop not-applicable rows and rescale. Then apply your bands: pass, coach or review.

How many calls should you QA for an AI voice agent?

Score every call where you can. Automated scoring replaces the roughly 2% manual sample. If you must start smaller, review every call from a new campaign or after a prompt change, plus every low-score alert.

How do you test AI voice agents before going live?

Talk to the agent yourself, then score test calls with the same card. Try interruptions, off-script questions and requests for a human. Use a browser test or a test call to your own phone, so no real customer is involved. Fix every critical error before launch.

What is a critical error in AI voice agent QA?

A critical error is one failure serious enough to fail the whole call, whatever the weighted score. Examples are missing AI disclosure, calling a blocked number, leaking account data, inventing a price, or blocking a human. Choose by legal risk and customer harm, and keep the list short.

What metrics matter most for AI voice agent performance?

Track critical-error rate first, then task completion, answer accuracy against source, handoff success, latency, and the share of calls where a human spoke. Trend them by campaign and criterion, so you fix the rule that breaks most.

How is QA for an AI voice agent different from QA for human agents?

An AI agent repeats the same behaviour on every call, so one flaw spreads at scale. You score invented facts, disclosure, silence and latency, which rarely apply to people. Fixes go to the prompt, knowledge base and settings, not individual coaching.

Check this before you start

A scorecard is only as good as its match with a human judge. If automated scores drift from what a trained auditor would say, coaches fix the wrong things and the critical-error gate misses real risk. Before you launch, have two people score the same 20 calls and see whether their results agree.

Book a demo of Insight7