A support lead replays a call. The agent quoted a refund window that was right two years ago, because the old article was still indexed. Retrieval-augmented generation (RAG) for AI voice agents fixes this: the agent looks up approved help center passages before speaking.
RAG gives a voice agent a lookup step
Retrieval-augmented generation for voice is a loop: listen, transcribe, retrieve, generate, speak. The agent turns the caller's speech into text and searches your approved content for matching passages. The language model then speaks an answer built from them.
The idea comes from a 2020 paper by Lewis and colleagues. They paired a language model with an outside, searchable memory. They aimed to show where an answer came from and to update knowledge.
Here is the pipeline, step by step:
- Speech to text produces a transcript of the caller's turn.
- Query rewrite uses call context. "What about sale items?" becomes "refund window for sale items."
- Embedding turns that query into a vector, a list of numbers that captures meaning.
- Hybrid search runs vector search for meaning and keyword search for exact terms, such as plan names or error codes.
- Reranking re-sorts the candidates with a slower, more careful model.
- Top-k passages go into the prompt. The number k is kept small.
- Generation follows an instruction to answer only from those passages.
A model alone guesses from its training data. With RAG it reads your current text first. That lowers invented answers, but does not remove them. Retrieval can still return the wrong chunk, and the model can still misread a right one.
Insight7's voice and chat agents are grounded in the company knowledge base. Its separate AI Knowledge Base product ingests internal docs, product guides and playbooks as a centralized source of truth for reps. Managers can test a campaign agent by speaking to it in the browser, with no phone number needed. This is in beta. Every call can also be scored against a weighted scorecard your team owns. Add your own criterion for source use, so calls where the agent answered without a source get flagged.
Buyers often ask how to stop an agent inventing answers outside its knowledge base. Write the fallback first. If no passage supports the answer, the agent says so and hands off to a person. Buyers raise handoff design on their own: is it polite, and does anyone chase a missed handover?
Chunk your help center for spoken answers
A chunk is one passage that the index stores and returns as a unit. Its size sets the trade. Big chunks carry more context but blur the match, so the right passage ranks lower. Small chunks match sharply but may drop the exception that changes the answer.
Microsoft's Azure AI Search guidance gives two starting points for fixed-size chunks: about 200 words with 10 to 15% overlap, or 512 tokens with 25% overlap. It treats both as values to test. It also describes semantic chunks, which break content into meaningful units that keep context across sentences.
| Method | Best for | Risk |
|---|---|---|
| Fixed size with overlap | Plain prose | Cuts through steps and tables |
| By H2 or H3 section | Help center articles | Long sections stay too big |
| Semantic break | Messy, mixed pages | Slower to build, odd splits |
| One question and answer pair | FAQ pages | Answer loses its context |
For a help center, split by heading first. Keep the article title and heading path as metadata, so a chunk never loses its parent topic. Keep step lists and tables whole. Attach the last-updated date and the article URL to every chunk.
Voice adds its own rules. Write each chunk so it can be read aloud in two or three sentences. Drop phrases like "click here". Spell out numbers and units, so "30d" becomes "thirty days".
For example, a "Returns and refunds" article becomes three chunks: refund window, start a return, and items we cannot refund. Each carries its heading path, the same URL and the date. A caller asking "what about sale items?" pulls the refund window chunk, and the agent can answer in one breath.
Latency budget: every retrieval step costs time
Treat retrieval as a spend against a very small time budget. A PNAS study of ten languages found that responses cluster within about 200 milliseconds of the end of the question. A caller will not wait for a voice agent the way they wait for a search page. A long pause feels like a dropped line.
Every step of the RAG loop adds to the wait. Latency depends on your stack, so time each step with your own test calls.
| Step | Why it costs time | Lever |
|---|---|---|
| Speech recognition | Waits for the end of the turn | Stream partial transcripts |
| Query rewrite | An extra model call | Skip it on single-turn questions |
| Vector and keyword search | Index size, network hop | Smaller index, nearby region |
| Reranker | A slower, careful model | Rerank fewer candidates, or drop it |
| Model first token | Prompt length grows with k | Top-3 instead of top-10 |
| Text-to-speech first audio | Waits for the first sentence | Stream audio as text arrives |
Three tactics save the most time:
- Start early. Begin retrieval on the partial transcript, before the caller finishes.
- Cache. Store answers for your top repeated questions.
- Run in parallel. Retrieve while the agent says a short filler such as "let me check that."
Cutting has a cost. Fewer chunks and no reranker lower recall, so the agent answers wrongly or hands off more often. A caller hears a repeated filler phrase right away. Rotate several, and use them only when retrieval runs long.
To decide what to cut, measure recall at k and p95 latency together. Remove a step, rerun the same questions, and keep the cut only if recall holds.
Cite sources the agent can say and show
Name the source in plain words, and never read a URL aloud. The agent can say "our refund policy page, updated in March." A caller can remember a page name and a month, but not a web address.
The citation lives in the pipeline, not in the model's memory:
- Every chunk carries an ID, a title, a URL and an updated date.
- The prompt makes the model return the IDs of the chunks it used, alongside its answer.
- A check step rejects any answer with no valid ID. Nothing reaches text-to-speech until it passes.
- The IDs are logged with the transcript, so a reviewer can open the exact passage.
When the channel allows, send the article link by SMS or chat after the call.
Set a fallback rule. If the top retrieval score falls below your threshold, or no chunk supports the claim, the agent says it cannot confirm the answer and hands off to a person. Gartner's 2026 survey of 3,566 customers found that 87% say a human option is essential when a company uses GenAI. A handoff path is therefore a requirement for most callers.
Treat freshness as part of citing. Show the version date in the spoken answer. Re-index whenever an article changes, and delete the old chunks, or the stale refund window comes back.
Disclosure belongs here too. Rules such as EU AI Act Article 50, which has applied since 2 August 2026, require AI that talks to people, including phone systems, to say it is AI from the first interaction. Colorado's chatbot law, effective 1 January 2027, asks the same of public-facing conversational AI.
Test retrieval before it talks to customers
Build a test set from real questions, and score retrieval before any caller hears the agent. Pull a few dozen questions from past calls and tickets. Mark the help-center passage that should answer each one. Include questions your help center cannot answer, so you can check that the agent hands off instead of guessing.
Track four numbers:
- Recall at k: the right chunk appears in the top results.
- Share of answers with a valid source ID.
- p95 latency.
- Handoff rate.
Rerun the set after every change to chunk size, reranker or prompt. A drop in recall tells you which change caused it.
Then talk to the agent yourself. Keep the live pilot small. Banks want a short pilot on a small sample, paying only after a measured lift.
Related reading: How to Find Knowledge Base Gaps From Questions Your AI Agent Couldn’t Answer, How to Give Customers a Case Reference Number and SLA From an AI Voice Call and How AI Agents Recognize Returning Callers and Pick Up Open Cases.
Questions people also ask
How does RAG work in simple terms?
The agent searches your help center before it speaks. It turns the caller's question into a search, pulls the closest passages, and writes its reply from them. Think of an open-book exam.
What is the best chunk size for RAG?
No size wins everywhere. Treat chunk size and overlap as settings. Try two or three sizes on your own question set. Keep the one with the best recall at k.
How does RAG work with a vector database?
The database stores each chunk as a vector next to its text and metadata. At query time it finds the vectors nearest to the question's vector. Metadata filters, such as product or language, narrow the results first.
How much latency does RAG add to a voice agent?
It depends on your stack, so measure it. Retrieval adds an index lookup, an optional rewrite and an optional rerank. Track p95 for each step, because callers notice slow outliers.
What is the difference between RAG and fine-tuning for a voice agent?
Fine-tuning changes behaviour, such as tone and format. RAG supplies current facts at answer time. For help-center content that changes often, use RAG, because editing an article updates the agent after a re-index.
How do you stop a voice agent from making up answers?
You can lower the risk but not remove it. Tell the model to answer only from retrieved passages. Reject answers with no valid chunk ID. Hand off to a person when retrieval confidence is low. Test with questions your help center cannot answer.
How should a voice agent cite its sources out loud?
It should name the page and its date, such as "our shipping policy, updated in June." Skip the URL and offer to text the link. Take the spoken name from the chunk's title and date metadata, not the model's memory.
Voice agents are only as good as the help-center passages they can find. This week, run 20 real customer questions through your retrieval step. Fix the articles that return no usable chunk first.
