You can build an AI voice agent in-house or buy one. The choice depends on how core voice is to your product, how fast you need it live, and whether you can staff a real-time engineering team. This guide covers the stack, steps, team and timeline.

Define one call before you write code

Pick one outbound call and write down four things: the goal, the audience, one success metric, and one handover rule. For example, a reminder call to customers with a payment due in three days. The metric is "paid before the due date". The handover rule is "transfer to a person if the customer disputes the amount". A narrow scope keeps the first build small enough to finish and test.

Start with outbound, not inbound. Inbound volume often exists because companies never send proactive updates, so outbound calls can remove some of it.

Then apply a build-vs-buy rule. Build only when all three are true:

  • Voice is core to your product.
  • You have time.
  • You already employ real-time engineers.

Our advice is to buy when you need it fast, because building is the wrong choice when you need it "yesterday". One commercial bank built its own apps and chatbots but plans to partner for voice, because it lacks the in-house skills.

Assemble the six layers of the stack

Build a voice agent as six layers, and decide for each one whether to build or rent it.

Layer Job Build or buy Main risk
Telephony SIP trunk or CPaaS, numbers, carrier reach Buy Poor answer rates on unfamiliar numbers
Speech to text Streaming transcript, speaker split, end-of-speech signal Buy Errors on narrow-band phone audio
LLM dialog Reply logic and tool calls into CRM, dialer and knowledge base Buy the model, build the prompts and tools Improvised answers
Text to speech Voice and accent Buy Robotic tone, wrong accent
Orchestration Turn detection, barge-in, call state Build, on a framework Awkward gaps and talking over people
Observability Transcripts, traces, call scoring Build or buy Calls you cannot review

Pick the pipeline shape first. A cascaded pipeline chains speech to text, an LLM and text to speech. You can swap each part, read each transcript and debug each hop. A speech-to-speech realtime API takes audio in and gives audio out from one model. It cuts hops and can lower latency, but you see less of what happened inside. A QA manager reviewing a bad call wants the text at every step, so choose cascaded when auditability matters.

Use a framework for orchestration. Open-source projects such as LiveKit Agents and Pipecat manage audio streams, interruption handling and swappable speech and model plugins. That saves weeks of work on the hardest part. Your engineers then spend their time on prompts, tools and testing.

Treat language as a stack choice. Buyers ask for local accents and native languages. Pick a text-to-speech voice for each language you will dial. Test any vendor's language claim on your own call audio.

Wire the pipeline to answer fast

Set a latency budget for every hop and measure each one. Aim for a short gap between the customer finishing and the agent replying, and judge it against your own recorded calls. A long gap sounds like a bot, and people hang up or talk over it. Split the wait into four parts: network transit, the final speech-to-text result, the LLM's first token, and the text-to-speech first audio byte. Log each part on every call to see which hop is slow.

Stream at every stage. Send audio to speech to text in small chunks, pass partial transcripts to the LLM, and start speaking as soon as the first sentence of the reply is ready. Use WebSockets, not request and response calls.

Handle turn detection and barge-in. Voice activity detection only hears silence, so it cuts off a customer who pauses mid-sentence. Add a semantic end-of-turn check that asks whether the thought is finished. When the customer interrupts, stop the audio at once and drop the unsent reply. Otherwise the agent can repeat the amount due after they said "I already paid".

Plan for phone audio. Phone audio is narrow-band, so pick a speech-to-text model tuned for telephony and test it on real recordings.

Design for failure. Count a call as reached only when the customer actually spoke. A temporary carrier refusal should not use up one of the contact's retry attempts. Use numbers local to the customer's country, since they tend to be answered more often.

Add guardrails, handover and consent

Scope the agent so it cannot wander. Regulated buyers, banks first, say their main fear is an agent that goes rogue or makes a public mistake. Four controls address that fear:

  • Write a prompt that names the one job.
  • Give each tool the least permission it needs. A reminder agent can read a due date and cannot issue a refund.
  • Add a no-answer fallback. When the agent lacks the fact, it says so and offers a person.
  • Log every decision: prompt version, tool call and reply.

Design the handover before launch. Buyers ask whether the transfer is polite, what happens with a disputed amount, and whether anyone chases an unanswered handover. Build three parts:

  • A trigger list: dispute, distress, a request for a human, repeated misunderstanding.
  • A warm transfer that passes a transcript summary, so customers never repeat themselves.
  • A queue with a named owner and an alert for long waits.

Gartner's survey of 3,566 customers found that 87% want a route to a human.

Check consent and disclosure by region. Confirm rules with counsel:

Document your data path. Buyers ask where recordings live, which AI models touch them, and who owns them. Write those answers down before your first pilot call.

Staff the team and ship in phases

Staff a small, cross-skilled team, and plan on months to production, not weeks. A typical plan looks like this, based on general engineering judgement. A browser prototype can run in days. A pilot-ready agent with telephony, guardrails and integrations tends to take about a quarter or more. These are planning estimates, not benchmarks.

Role Owns Headcount
Real-time backend engineer Orchestration, streaming, barge-in 1 to 2
LLM engineer Prompts, tools, evals 1
Telephony or DevOps engineer SIP, numbers, scaling 1
Conversation designer Scripts, handover wording 1
QA owner Test calls, scoring 1
Compliance partner and product owner Consent, scope, metric Part-time
Phase Work
Weeks 1 to 2 Scope one call, prototype in the browser
Weeks 3 to 8 Telephony and the full pipeline
Weeks 9 to 12 Guardrails, handover, integrations
After week 12 Small pilot, then monitoring

Test in three rounds. Use simulated callers first, then a human test call, then a small live sample. Banks want a short, low-risk pilot on a small sample. Score every pilot call so you can see what failed. Gartner found only a quarter of 432 AI service use cases show positive ROI, so measure from day one.

Buy when you lack the headcount

If this headcount is out of reach, Insight7 is the buy option, and its voice campaigns are in beta. Managers set a goal, an audience, the AI caller and the rules. They can test the agent in the browser, mark consent, and set calling hours and retries. Campaigns can run in Yoruba and Igbo, and Nigerian Pidgin is in beta too, so test each language on your own audio. Calls can go out on local Nigerian numbers through a carrier gateway. A dashboard shows calls in progress and each transcript when the call ends. Each campaign call is scored when it ends. Insight7's released call scoring can also grade every call against a scorecard your team owns.

Related reading: How to Measure Speech Recognition Accuracy for Your Customers' Accents and Languages, How AI Voice Agents Handle Code-Switching Between Languages Like Spanglish, Hinglish and Pidgin and 10 Best Speech-to-Text APIs for AI Voice Agents Handling Accents and Noisy Phone Lines.

Questions people also ask

How do I build an AI voice agent using the OpenAI Realtime API?

Connect a speech-to-speech session from your telephony layer. Check the current OpenAI Realtime documentation for supported transports, such as WebRTC or WebSockets. Set instructions and tools, then stream call audio in and out. Log transcripts yourself, because the model hides its inner steps.

How do I set up an AI voice agent for outbound calls?

Get a verified number, load only contacts who gave consent, and set calling hours by time zone. Script a voicemail rule and a retry limit. Test once on your own phone.

What is the tech stack for an AI voice agent?

You need telephony, streaming speech to text, an LLM, text to speech, orchestration and observability, plus CRM and dialer integrations. Rent most layers. Build the prompts, tools and handover logic.

How long does it take to build an AI voice agent?

Expect days for a browser demo and about a quarter for a pilot-ready agent. Most of that time goes to telephony tuning, handover design and compliance review.

What team do I need to build an AI voice agent?

A narrow pilot can run on two engineers, one strong in real-time systems, plus a part-time conversation designer. Name one metric owner.

How do you test voice AI agents before launch?

Test against recorded audio with accents, noise and interruptions. Check each latency hop and every handover trigger. Then place real calls to your own team.

What are the consent requirements for AI voice outbound calling?

Get consent before the first call and store proof of when, how and in what wording. Honour do-not-call requests and calling-hour limits.

Build the edge, buy the rest

Building a voice agent is a staffing and reliability commitment, and the model is the smallest part. Build the layer that sets you apart, buy the rest, and never ship a call you cannot score.

Book a demo