Code-switching is when a speaker mixes two or more languages in one sentence, as in Spanglish, Hinglish or Nigerian Pidgin. For AI voice agents, this means the agent can no longer pick one language per call. It has to follow the caller mid-sentence.
What code-switching is, in plain words
Code-switching is when bilingual speakers shift from one language or dialect to another as they talk. People switch because of the topic, their mood, or who is listening. Research links it to changes in listener, setting or topic.
Researchers split it into two kinds. Switching at sentence boundaries is one kind. Switching inside a sentence, also called code-mixing, is the other. The second kind is the hard case for machines. A borrowed word, like "email" inside Hindi, is usually treated as a loan. A switch brings in a fresh word or phrase.
Here are three examples we wrote:
- Spanglish: "Necesito cambiar mi password antes del lunes."
- Hinglish: "Mera payment kal tak ho jayega, don't worry."
- Pidgin with English: "Abeg, I wan update my account details before end of month."
Nigerian Pidgin is an English-based contact language. It draws its syntax and vocabulary from English, Portuguese and Nigeria's indigenous languages, and adds words of its own. It has its own grammar. Speakers still switch between Pidgin and English.
Why mixed speech breaks recognition
Mixed speech breaks recognition because every stage of a voice agent assumes one language at a time. A mid-sentence switch confuses the first stage, and the errors then pile up through the next ones.
The pipeline runs in order: speech recognition turns sounds into words, language ID picks the language, an understanding step finds the intent, the agent writes a reply, and a text-to-speech voice says it.
Five things go wrong at the recognition end:
- Language ID runs too coarsely. Many systems label a whole utterance or a whole call. A switch in the middle is missed, or the line is forced into the wrong language.
- The language model has rarely seen the word order. It has seen plain Spanish and plain English, but rarely "cambiar mi password". It tends to pick a likelier, similar-sounding word, which is often wrong.
- Accents clash with training sounds. A Hindi speaker's English, or an English word said with Yoruba tones, may not match what the model learned. Phone audio makes this worse.
- Spelling has no standard. Hinglish appears in Roman script and in Devanagari. "Kal", "kal" and "कल" can all be right, so transcripts and intent matching drift.
- Training data is thin. Labelled mixed speech is scarce, and Pidgin and many African languages have far less.
The costly part comes after the transcript. A wrong transcript becomes a wrong intent, a wrong answer or a missed consent. Names, amounts, dates and phone numbers do the most harm. Mishearing "fifteen" as "fifty" in a repayment call is a compliance error, not a typo.
The reply side has its own problem. The voice must say mixed words naturally, and the agent must pick which language to answer in. If it picks wrong, the caller hears a stranger.
Here is an illustrative example, not a real case. A collections lead at a 200-agent lender has a caller who says, "I'll pay kal, after salary." The transcript lands as "I'll pay call after salary", and the agent hears no promise at all. A plain transcript score cannot tell you whether the agent is safe.
How current systems try to cope
Vendors use four main approaches. Multilingual end-to-end models train on mixed data, so one network hears both languages. Some systems run language ID on short frames instead of whole calls. Others run several recognisers in parallel and keep the best result. Many also tune voices to a locale.
Each approach has a cost. Parallel recognisers add compute and delay, and a phone call has little room for delay. Switching too eagerly also causes errors, because a short frame gives language ID little to go on. Microsoft's documentation shows the limit. Its continuous language ID does not detect a switch inside a sentence, such as English words dropped into Spanish.
Phone audio is usually compressed and noisy, which can blur the detail that separates similar sounds. Every method above tends to get weaker on a real call. Custom vocabulary or prompt hints for product names help. So does a plain rule for which language the agent answers in.
Where the limits still are
Switches inside a single word or phrase, rare language pairs, Pidgin and other low-resource varieties, noisy lines, fast speech and city slang all cause errors. Names and numbers fail first.
A vendor's language count tells you little. Each language is usually benchmarked alone, and a model that scores well on clean Spanish and clean English can still fail on "cambiar mi password". Ask for results on your pair, on phone audio.
For a regulated buyer, design for misheard consent or amounts:
- Have the agent read back amounts, dates and yes/no answers.
- Hand over to a human when confidence is low.
Two rules and one survey finding shape that design:
- EU rule: Article 50 of the AI Act has applied since 2 August 2026. Automated phone systems must say they are AI from the first interaction.
- US rule: AI-voice calls need consent under the TCPA, per the FCC's 2024 ruling.
- Survey finding: 87% of 3,566 surveyed customers say a company using GenAI for service must offer a way to reach a human.
Training a mixed-language model needs labelled mixed speech that you probably do not own. Buy the engine and own the test set.
How to test code-switching before launch
Run this test in one week, with an ops lead and one engineer. Each step has one job.
- Build a test set from real calls. Start small per language pair and scale the size to your risk. Balance the clips by switch type (between sentences, inside a sentence, borrowed words), accent, age, line quality and device. Get consent and redact personal data first.
- Write reference transcripts with bilingual people. Give them a style guide for spelling and script, so "kal" and "कल" are treated the same way.
- Measure three scores. Word error rate counts substitutions, insertions and deletions against the number of reference words. It treats "the" and "fifty" as equal errors, so it hides the costly mistakes. Score entity accuracy for names, amounts, dates and phone numbers on its own. Score intent accuracy too.
- Test the switch itself. Does the agent follow the caller when they change language? Does it reply in that language?
- Test live behaviour on phone-quality audio. Check latency, interruptions, silence and handover when confidence is low.
- Score each call with a pass or fail rule. One critical error on consent or an amount fails the whole call. QA leaders say a checklist score can hide what matters.
- Keep testing after go-live. Sample real calls every week and re-run your test set after each vendor model update.
Insight7 Voice campaigns can run in Yoruba and Igbo, and the manager picks the language before the voice. This is in beta. Nigerian Pidgin is a beta campaign language through Spitch or Intron voices. Measure it on your own callers. In beta, you can talk to a campaign's agent in the browser with no phone number. A dashboard also lets a reviewer read each transcript when the call ends. After launch, Insight7 can score every call against a weighted scorecard your team owns, not a sample.
Put three questions to any vendor. Which language pairs did you test? On what audio? With what entity accuracy?
Related reading: How to Measure Speech Recognition Accuracy for Your Customers' Accents and Languages, How to Build an AI Voice Agent In-House With the Full Technical Stack Explained and AI Voice Agent Architecture Explained From Telephony to Speech Recognition, LLM and Text-to-Speech.
Questions people also ask
What is code-switching in speech recognition?
It is when a recognizer hears speech that moves between languages and must pick words from both. A system built for one language per call usually fails at the switch point, so the transcript gets worse there.
Can AI understand Hinglish?
Partly. Short, common phrases on clear audio often work. Results drop with fast speech, phone noise, and names or numbers. Score entity accuracy on your own callers before you trust it.
Can AI voice agents understand Nigerian Pidgin?
Support is early. Insight7 lists Nigerian Pidgin as a beta campaign language through Spitch or Intron voices. Pidgin varies by region, so test it on callers from your own market.
Why is code-switching so hard for speech recognition?
Models learn word order mostly from single-language text, so mixed sentences can mislead them. Scarce labelled data and inconsistent spelling make errors harder to measure and fix.
How do you test a voice agent in more than one language?
Build one test set per language pair from real calls. Score word error, entity and intent accuracy separately. Re-run the set after every vendor model update, because an update can quietly make mixed speech worse.
Does a voice agent need to reply in the language the caller used?
Usually yes. Mirror the caller's main language and follow their switches. Read consent statements in the language the caller agreed to proceed in. Set this as a written rule and let callers change it.
Is Spanglish a language, and does that matter for voice AI?
Spanglish is often described as a way bilinguals talk, with no single standard form. It matters because there is no reference spelling. You need your own style guide so transcripts are scored the same way each time.
Code-switching breaks the one-language-per-call assumption that most speech systems make. Test on your own callers' real mixed speech. Score names, numbers and consent separately, and keep a human route open. Trust the agent only on what you have heard it get right.

