AI voice agent architecture is a chain of four layers: telephony, speech recognition, a language model and text-to-speech. The change that matters is that each layer adds its own delay and can fail alone. The caller hears only the total.

Four layers sit between caller and answer

An AI voice agent is software that holds a phone or web conversation in real time. It listens to the caller and decides what to say. It looks up facts in systems such as a CRM when it needs them. Then it speaks in a synthetic voice and can pass the call to a person.

Here is the full loop:

  • Caller → phone network → SIP trunk → media stream
  • Media stream → voice activity detection and endpointing → speech to text
  • Speech to text → LLM, with tools and CRM lookups → text to speech
  • Text to speech → audio back to the caller
  • Barge-in arrow: caller speech during playback → stop text to speech and cancel the LLM → back to listening
  • Handoff branch: LLM or rules → transfer to a human agent

Each layer has one job:

  • Telephony carries the call and delivers audio frames, small slices of sound sent many times a second.
  • Speech recognition turns frames into partial transcripts, which it revises, then a final transcript.
  • The language model reads the transcript and state, then streams tokens (word pieces) as it writes.
  • Text to speech turns tokens into audio chunks and sends them back down the call.

Streaming at every hop matters more than any single model. A batch design waits for each layer to finish, so delays add up. A streaming design starts the next layer on the first partial output. A fast model behind a batch hop still feels slow to a caller.

Telephony and speech recognition set the floor

A call reaches your agent through the phone network. The caller's carrier hands it across the PSTN to your carrier, which delivers it on a SIP trunk. SIP (RFC 3261) starts and ends the call. The audio travels separately as RTP packets. RFC 3550 defines RTP and leaves quality of service to the network, so you must handle jitter and packet loss yourself. Browser calls skip the carrier and use WebRTC.

Phone audio is narrowband. The common phone codec, G.711, was built for voice frequencies on telephone lines, not for rich sound. In practice, narrowband audio loses the high frequencies that help separate sounds like "f" and "s". Models trained on wide-band audio can lose accuracy here, especially on digits, names, accents and local languages. Test recognition on real phone recordings, not studio clips.

Speech recognition should stream. It sends partial transcripts that change as the caller speaks, then a final one. Voice activity detection spots speech. Endpointing is the silence threshold that decides the caller has finished. Too short, and the agent cuts people off. Too long, and you add dead air. Diarization (who spoke) is easy when the caller has their own channel. It gets weak when two people share one line.

Carriers cause quiet failures:

  • Voicemail: answering machine detection must tell a recording from a person before the agent talks.
  • Refusals: a carrier's temporary refusal is not a real no-answer. In Insight7's beta campaigns, it does not use up a retry, and calls where only the AI spoke do not count as reached.
  • Silence: some calls connect but deliver no audio from the caller. An automatic campaign pause for this case is in development.

LLM and voice set how natural it sounds

The language model reads the system prompt, the conversation so far and the latest transcript. It then streams its reply token by token. When it needs a fact, it calls a tool, such as a CRM lookup, and waits for the result. Time to first token matters more than total generation time. The voice layer can start speaking as soon as the first sentence is ready.

Voice replies must be short and spoken in style. A caller cannot skim, so one idea per turn works best. Tool calls add dead air, so have the agent say a short filler line before the lookup starts.

Text to speech should also stream. Watch time to first audio byte. Split the reply at sentence boundaries so the first chunk plays while the rest is still being made. Buyers fear AI that sounds scripted or robotic, and they ask for local accents and languages.

Barge-in is the hard part. When the caller interrupts, stop playback, cancel the LLM and return to listening. Miss one step and the agent talks over the caller.

Two failures matter most. The model can invent facts, so give it guardrails and clear escalation rules. Handover also needs design: buyers ask whether it is polite, how sensitive issues are treated, and whether anyone picks up.

Insight7's voice campaigns are in beta and must be switched on per organisation. Managers set a goal, an audience, the AI caller and the rules, and each call is scored when it ends. The beta spec describes consent marking, calling hours and retry rules, so check the detail in the app. You can talk to the agent in the browser first. Insight7 Voice is a lower-cost engine that uses ElevenLabs voices, and it supports campaigns in local languages such as Yoruba and Igbo.

Spend a 1-second latency budget like money

Aim for about one second from the end of the caller's speech to the first audio of the reply. People set a high bar. Stivers' study of 10 languages found that speakers across cultures aim to keep the gap before the next turn as short as they can. No voice agent matches human timing end to end, so you spend a budget instead.

The table is an illustrative budget, not measured Insight7 data and not a published standard. Replace each line with your own p95 measurements.

Line Illustrative range
Network and telephony, caller to media server 40 to 100 ms
Endpointing wait 200 to 500 ms
Speech recognition, final transcript 50 to 200 ms
LLM, first token 200 to 400 ms
TTS, first audio 100 to 250 ms
Return network 40 to 100 ms

Worked example. Take 80 + 300 + 100 + 300 + 150 + 80. That is 1,010 ms, inside an 800 to 1,200 ms working target. Cut the endpointing wait first. It is the biggest line and a setting changes it. Use 150 ms after a yes/no question and 400 ms after an open one.

What moves each line:

  • Network: put the media server in the same region as the models.
  • LLM: a smaller model and a shorter prompt cut first-token time. Each tool call adds a full round trip.
  • TTS: sentence chunking lets the first sentence play early.
  • All hops: reuse open connections, so you skip handshakes.

Track p95, not the average. The average hides the slow calls, like a cold connection or a slow CRM lookup. Callers remember the worst pause.

Related reading: How to Connect an AI Voice Agent to Your Own SIP Trunk or Carrier, Build vs Buy AI Voice Agents Cost Comparison With a 12-Month Total Cost of Ownership Model and AI Voice Agent Cost Per Call Broken Down by Telephony, Speech Recognition, LLM and Voice.

Questions people also ask

What is an AI voice agent?

It is software that takes part in a live phone or web conversation. It hears free speech, decides what to say, checks systems for facts and answers aloud. A phone menu only matches keypresses.

What is the architecture of an AI voice agent?

Four streaming layers (telephony, speech recognition, language model, text to speech) sit under an orchestrator. The orchestrator runs turn-taking, barge-in, tool calls, human handoff and call logging.

How much latency is acceptable in an AI voice agent?

Aim for 800 to 1,200 ms from the caller's last word to the first reply audio, measured at p95. Set a tighter limit for yes/no questions. Callers forgive a pause after a hard question sooner than after a simple one.

What causes latency in AI voice agents?

The usual causes are a long endpointing wait, a large model with a long prompt, and CRM calls made one after another. A distant media server and cold connections add more. Find the slowest calls, then trace each layer.

How do AI voice agents handle interruptions?

Voice activity detection hears the caller during playback. The orchestrator stops the audio, cancels generation and trims the saved history to the words the caller actually heard. Echo cancellation stops the agent hearing itself.

Is it better to build or buy a voice AI agent?

Buy when you need voice soon or voice is not your core product. Companies that built their own chatbots still call voice AI too hard to build in-house. Build only if voice is the product and you can run carriers, monitoring and compliance.

Do AI voice agents have to say they are AI?

Often, yes, and rules differ by region. In the EU, Article 50 has applied since 2 August 2026. In the US, a federal disclosure rule is only proposed, and Colorado's state rule starts 1 January 2027. The FCC's 2024 ruling already applies TCPA consent rules to AI voice calls.

Judge the weakest layer first

A voice agent is only as good as its weakest layer, so budget latency and plan failures for each one. Insight7's beta voice campaigns let you test an agent in the browser before any call goes out. Ask in your next meeting: which layer fails first on our worst call, and who gets alerted?

Book a demo of Insight7