TL;DR

  • To measure speech recognition accuracy, pull 30 to 60 real clips per accent and language.
  • Have people write reference transcripts, run every vendor on the same clips, and compute word error rate (WER) per group.
  • Judge vendors on your worst group, not the average.

Collect sample clips from real customer calls

Pull clips from your own recordings, not from a public test set. Public sets are mostly clean read speech. Your callers are on 8 kHz phone lines, in traffic, switching languages mid-sentence.

Build the sample in layers. Each layer is a column in your sampling sheet:

  • Language and accent: Indian English, Scottish English, Latin American Spanish, Gulf Arabic, and Nigerian English with Pidgin, Yoruba, Igbo or Hausa. Pick the ones your customers speak.
  • Channel: 8 kHz phone, app audio, browser audio.
  • Noise: quiet room, street, call centre floor.
  • Speaker: gender and age band.
  • Code-switching: tag clips where the caller mixes languages.

Cut clips of 10 to 30 seconds. Shorter clips lack context. Longer ones are slow to transcribe by hand. Get consent first, and strip names, card numbers and addresses from the audio before anyone outside your team hears it.

How many clips? Count words, not clips. This is general statistical method, not a vendor rule. The margin of error is roughly 1.96 × √(p(1−p)/n), where p is the WER and n is the number of reference words. At 15% WER, 1,000 words give about plus or minus 2 points. About 3,000 words narrow it to roughly 1.3 points. Errors cluster inside calls, so the true margin is wider. Two vendors 1 point apart on a small group are probably tied.

Start with 30 to 60 clips per group as your floor. Grow each group toward several thousand words before you buy.

In our calls, buyers ask for Nigerian accents, Pidgin and native languages, and staff tune out voices that sound foreign. Treat each one as its own group, never as noise in an "English" average.

If you use automated scoring on these calls, test whether it penalises any accent group. Compare score gaps between your groups.

Write human reference transcripts first

Have people write the answer key before any machine output exists. If a transcriber sees an ASR draft, they copy its guesses, and your WER favours that vendor. Give the clips to humans cold.

Use this process:

  • Two transcribers per clip. Pick people who speak the accent or language natively.
  • One style guide. Fix how to write numbers, fillers like "um", code-switched words and false starts. For Pidgin, Yoruba and Igbo, pick one spelling for each common variant and list it.
  • Adjudication. A third person settles every disagreement. Count how often the first two differ, because that rate is the noise floor of your test.

Then run each vendor on the same audio with the same settings:

  • the same sample rate, with no re-encoding for one vendor and not another
  • language hint or auto-detect, set the same way for every vendor
  • diarization on or off, the same for all
  • custom vocabulary on or off, tested both ways

Last, normalise text on both sides before scoring. Lowercase everything and strip punctuation. Convert numerals to one form, and decide whether tone marks and diacritics count. WER penalises every difference from the reference, including harmless style differences. Apply one normaliser to every vendor so none gains or loses on formatting.

Calculate word error rate per accent group

Compute WER as errors divided by reference words, then pool the errors inside each group. The formula is (substitutions + deletions + insertions) ÷ words in the human reference.

Here is a short example. The reference is "send two hundred naira to Ade", which is 6 words. The machine wrote "send to hundred naira Ade now". That is one substitution (two became to), one deletion (the word "to" is gone) and one insertion ("now"). Three errors over 6 words gives 50% WER. The substitution changed the amount, which matters most.

Pool, never average. Add up all errors in a group and divide by all reference words in that group. Say clip A has 10 words and 1 error. Clip B has 100 words and 20 errors. Averaging the two clip rates gives 15%. Pooling gives 21 ÷ 110, or about 19%. Pooling is honest, because a 10-word clip should not count as much as a 100-word clip.

Add a confidence interval. Use bootstrap resampling:

  1. Resample whole clips with replacement.
  2. Recompute the pooled WER.
  3. Repeat 1,000 times.
  4. Take the 2.5th and 97.5th percentiles as your range.

Resample clips, not words, because errors cluster inside a call.

Know the limits of WER.

  • It can pass 100% when a model inserts many words.
  • It treats "the" and a loan amount as equal.
  • It punishes valid spelling variants.
  • It is harsh on tonal and agglutinative languages, where one long word packs in several parts.

So also report character error rate (CER). The open-source Python package jiwer computes both.

Score what the call needs. Add error rates for names, numbers, amounts and key terms. Measure diarization too, meaning how often words land on the wrong speaker. A call score built on the wrong speaker's words is wrong, even when WER looks fine.

Gaps between groups are real. A 2020 PNAS study by Koenecke and colleagues found that major vendors made far more errors for Black speakers than for white speakers, among US English speakers. Your per-group table catches the same pattern in your own customers.

Compare vendors on your worst group

Rank vendors by their worst accent group and by the gap between groups. An average hides the callers who get the worst service. A vendor at 12% on one group and 31% on another loses to a vendor at 16% on both.

Fill one row per vendor:

Vendor WER per group CER Worst-group WER Names and numbers Latency Price per hour Languages Data hosting
Vendor A your run your run your run your run your run from its pricing page from its language list from its terms
Vendor B your run your run your run your run your run from its pricing page from its language list from its terms

Take the price, language and hosting cells from each vendor's own pages. Write "not published" in any cell the vendor's page leaves empty.

Trust your numbers over their benchmarks. Vendor charts usually come from clean, read speech. Your callers are on noisy phone lines. If a vendor's claim and your result disagree, your result wins.

Weigh build against buy. Try the cheap fixes first. Add custom vocabulary for product names. Then switch vendors if one handles your worst group better. Fine-tune an open model such as Whisper only when no vendor closes the gap and you have hours of transcribed audio from that group. Rerun the test every quarter and after any model update, because vendors change models without much notice.

Ask the data questions early. Buyers ask where recordings live, which models touch them and who owns them. Hosting rules differ by region. Examples are EU data residency, US state privacy laws, and Nigeria's central bank rule on payment data. Put each answer in the table next to the WER.

Check accuracy inside your call scoring

A transcript error becomes a scoring error, so test the two together. Insight7 scores every call against a weighted scorecard your team owns, not a sample. Test its transcription on your own clips like any other vendor. When a call scores low, read its transcript to see whether misheard words drove the score. QA leaders say automated scores must match the human auditor, so have an auditor rate a sample of calls and compare. Yoruba and Igbo campaigns on Insight7 Voice are in beta. Nigerian Pidgin is also in beta. Run the same clip test on any language before you rely on it.

Related reading: How AI Voice Agents Handle Code-Switching Between Languages Like Spanglish, Hinglish and Pidgin, How to Build an AI Voice Agent In-House With the Full Technical Stack Explained and How to Connect an AI Voice Agent to Your Own SIP Trunk or Carrier.

Questions people also ask

How do you measure speech recognition accuracy?

Collect real clips, have humans write reference transcripts, and run each vendor on the same audio. Normalise the text, then compute pooled word error rate per accent group. Add character error rate and checks on names and numbers.

What is a good word error rate for speech recognition?

No single number is good everywhere. Judge it against the human noise floor, which is how often your two transcribers disagree. Then set a limit for each task. A loan amount needs a tighter limit than small talk.

How many audio samples do I need to test speech to text?

Start with 30 to 60 clips per group, and count words rather than clips. When two vendors land within a point or two, add clips until their bootstrap intervals stop overlapping.

Why does speech recognition perform worse on some accents?

Accents, dialects and phone channels that are thin in a model's training data get more errors. Ask each vendor whether it has trained on your callers' speech, then check with your own clips.

Is word error rate enough to judge a vendor?

No. WER weighs every word equally and punishes valid spelling variants. Pair it with character error rate, name and number accuracy, diarization errors, latency and data hosting. Rank vendors on the worst group.

How do you test speech recognition for low-resource languages?

Use the same method, but expect fewer vendors and noisier references. Hire native transcribers, write a spelling guide for each variant, and report CER next to WER. Treat any vendor's claimed support as unproven until your clips confirm it.

Should I fine-tune a speech model or switch vendors?

Try custom vocabulary first, then another vendor. Fine-tune an open model such as Whisper only when no vendor closes the gap and you hold hours of transcribed audio for that group. Retest on clips the model never saw.

Start with your hardest accent

Average accuracy hides the groups your customers are in. This week, pull 30 real clips for your hardest accent, have two people transcribe them cold, and run two vendors on the same audio. Compare worst-group WER.

Book a demo of Insight7