Your weekly handoff report shows the AI agent passed the same refund-timing question to humans again and again. The help content lacks the answer. This guide shows how to find knowledge base gaps from unanswered questions: log, group, write, verify.
Log every question your agent couldn't answer
Start by defining "couldn't answer" in four signals, then record each one. A question counts as a miss when any of these happens:
- The agent gives a fallback reply, such as "I'm not sure about that."
- The agent hands the conversation to a human.
- The customer asks again or rephrases the same question.
- The customer gives a thumbs-down or abandons the chat.
Give every miss one row in a shared sheet. Keep the fields short:
- Verbatim question, in the customer's own words
- Channel (voice, chat, email)
- Matched article, if any
- Agent reply
- Outcome (answered, fallback, handoff, abandoned)
- Date
- Customer type
The "matched article" field matters most later. A blank means no content was found. A filled cell with a miss means the content exists but failed.
Do not trust handoffs alone. A confident wrong answer never shows up as a handoff, so the customer leaves with bad information and your report stays clean. Manual sampling can cover as little as 2% of calls, which is the gap Insight7's call scoring is built to close. So score every conversation against a scorecard your team owns. Add a criterion such as "answer matches approved content." Read transcripts of finished calls to confirm what the agent said.
Log the handoff as carefully as the miss. Gartner found that 87% of 3,566 customers say a company using GenAI for service must offer a way to reach a human. Your handoff log is evidence of trust, as well as a source of gaps.
Group the misses by topic and cause
Group in two passes. First cluster by what the customer wants, then tag each cluster with why the agent failed. "When will my refund land?" and "Where is my money?" are one intent, even though the words differ.
Pass one: cluster by intent. Tag a first batch by hand, such as last week's misses, to learn what your real clusters look like. After that, turn each question into an embedding and group by similarity, so paraphrases land together. Pick a similarity cutoff for automatic merging. Send borderline pairs to a person, and tune the cutoff on your own log.
Pass two: tag the cause. Use five tags:
- Content missing
- Content outdated
- Content present but badly worded
- Content present but not retrieved
- Out of scope, so a human must answer
The fixes differ by tag. Researchers who studied real retrieval-based systems found that a question can have no answer in the documents, or the answer can exist but rank too low to be returned. One needs a new article. The other needs a rewrite or a retrieval fix. Use your "matched article" field to tell them apart. A blank means missing. A filled cell on a miss means one of the other tags.
Rank by volume times cost. Count the questions in each cluster, then multiply by what each one costs you, such as handoff minutes or compliance risk. Say a cluster of 30 refund questions at six minutes each: it outranks 50 questions a human answers in seconds. A support lead can review the top five clusters in one sitting.
Write the missing answer, then publish it
Fix what exists before you write anything new. Rewrite the "outdated" and "badly worded" clusters first, because those edits are fast. They also stop wrong answers.
For each "content missing" cluster, write one article with this pattern:
- Title: the customer's own wording, such as "When will my refund arrive?"
- First sentence: the answer, in plain words.
- Conditions and exceptions: card refunds versus bank transfers, public holidays, partial refunds.
- Owner and review date: one named person and a date.
Write one question per article. A single-topic article is easier for retrieval to match cleanly. Then paste the real phrasings from your log into the article as alternate wordings. "Where is my money?" belongs in the refund article, because customers say it that way.
Draw the facts from your source of truth: internal docs, product guides and playbooks, the same material an AI knowledge base ingests. A subject-matter owner approves every article. Compliance also reviews anything on a regulated topic. If you mine past calls for wording, copy the phrasing and never copy personal details.
Next, decide which gaps should stay open. Questions with fixed answers suit the agent. Cases that need judgment and care belong with people, so set hard rules that send them to a human. Disputed amounts and sensitive issues should always hand off. Buyers ask three things. Is the handover polite? What happens to a disputed amount? Does anyone chase a handover nobody picked up?
Tag these clusters "out of scope" and write a handoff rule instead of an article.
Verify the fix before customers see it
Test every new article against real questions before it goes live, then check it again on live traffic. Measure each cluster on its own. Gartner analysed 432 AI customer-service use cases. Only a quarter showed positive ROI, and 42% could not be measured at all. Per-cluster numbers keep your fixes out of that unmeasured group.
Build a test set from your log. For each gap, take the original question and add 3 to 5 paraphrases. Add a few out-of-scope questions, such as a disputed charge, that must still hand off.
Run it before and after publishing. Check three things:
- The right article was retrieved. Ragas, an open evaluation framework, defines context recall as how many relevant pieces of information were successfully retrieved. Low recall means your article is not being found.
- The answer matches the article and nothing is invented.
- Handoff still fires when it should.
Insight7's AI Voice and Chat Agents are grounded in the company knowledge base for accuracy and compliance. Every call can be scored against your own scorecard, which helps you re-check live answers. In beta, you can talk to a campaign agent in the browser to try questions before launch.
Re-score live traffic after one to two weeks. Track fallback rate, handoff rate and wrong-answer rate for each cluster. A cluster is fixed only if all three move the right way.
Read a small sample of the conversations that now pass. The agent can be confidently wrong, and a dashboard pass will not show it. Size the sample to the time your team has each week.
Run the loop every week
Hold a short review every week. A support operations lead at a 200-agent lender can run it with the open clusters on screen. Give each cluster one owner and one publish date, so nothing sits in a shared inbox.
Retire a cluster only after verification passes twice in a row. One good week can hide a retrieval problem that shows up later.
Track two numbers over time: the count of open gaps, and the share of questions answered without a handoff. Falling gaps with a rising answered share is proof the loop works.
Related reading: How to Design Tool Calls That Let AI Voice Agents Look Up Accounts, Create Cases and Book Callbacks, How to Prepare Your Knowledge Base for an AI Voice Agent to Answer Customer Questions and How Retrieval-Augmented Generation Works for AI Voice Agents Answering From Your Help Center.
Questions people also ask
What is a knowledge base gap?
It is a topic customers ask about that your content misses, gets wrong, or states in a way the agent cannot find. Gaps are missing, outdated or unfindable. Each needs a different fix.
How do I find knowledge base gaps in my AI chatbot?
Pull every fallback, handoff, rephrased question and thumbs-down from the last seven days. Cluster them by intent and count each cluster. Then score all conversations, not a sample, to catch confident wrong answers that never trigger a handoff.
Why does my AI agent say it doesn't know the answer?
Retrieval usually found nothing relevant. The article may not exist, or it may use different words than customers do. A rule can also make the agent decline some topics on purpose.
How often should I update my AI agent's knowledge base?
Update an article the moment a price, policy or process changes, and review the rest on its review date. Give every article an owner, so stale content gets caught early.
How do I stop my AI agent from making up answers?
Ground the agent in approved content, so it answers from your documents. Then check each answer against the article it used. Read a small sample of passing conversations weekly, because wrong answers can sound confident.
How many questions should I review each week?
Start with every miss from one week and tag them by hand. After that, review the top clusters plus a small sample of conversations that passed. Size both to the time your team can protect.
When should an AI agent hand a question to a human?
Hand off when the customer asks for a person, or the agent lacks the context to answer. Add hard rules for disputed amounts, sensitive issues and judgment calls.
Turn every miss into a to-do item
Every unanswered question is a free to-do item for your knowledge base. This week, tag the top ten fallbacks and handoffs from the last seven days and write one answer. Test it on three paraphrases first.
