AI agents recognize returning callers by matching the incoming phone number to a contact record in your CRM or helpdesk. The agent then finds the open case and confirms identity before sharing anything.
Caller history starts with a phone number match
The agent finds a caller's history in three steps. The phone network passes the caller's number, called ANI or caller ID, to the agent. The agent then queries your CRM or helpdesk with that number. If a contact matches, the agent also pulls the cases linked to it.
A good record gives the agent enough to pick up where the last call stopped:
- Past calls and their outcomes
- Case status (open, pending, resolved)
- The last agent who handled it
- Promised callbacks and their due dates
- Transcripts or summaries of earlier calls
The agent reads this record before it speaks. A caller with an open refund case hears about the refund. A new caller gets a fresh start.
A number match is a strong hint. It is weaker than proof. A family can share one phone. A customer can swap SIM cards. Scammers can fake caller ID, which is why regulators require carriers to use STIR/SHAKEN call authentication. Call authentication confirms the number is genuine. Identity needs its own check.
Matching goes wrong in four predictable ways
A phone number is a weak key, so design for four failures.
- Duplicate records. One customer has two contact entries, and the open case sits on the other one.
- Shared numbers. A family or a small business uses one phone, so one number maps to several people.
- New numbers. The caller switches SIM or phone, and the number matches no one.
- Stale case status. The CRM says "open" for a case that was closed yesterday, or the reverse.
The fixes are the same everywhere. Match on more than one field, such as number plus email or account ID. Rank the candidate records, for example by most recent activity. Then ask a confirming question before using any record. If nothing fits, treat the caller as new and let a human link the records later.
Some callers have several open cases. The agent can pick the most recently updated one and say so: "I see your refund request from Tuesday. Is that why you're calling?" If two cases are close in date, it should ask which one the caller means.
Our advice is to copy good human agents. A bank-affiliated lending arm told us its agents confirm the caller is on the registered number and ask security questions before they share loan details. An AI agent can follow the same rule: look up the record first, then share only after the check. Checking identity before releasing sensitive account information is also how banks approach it.
What the agent says when it knows you
The agent should say it is an AI in its first sentence, then use what it knows carefully. Three openings cover most calls:
- First-time caller: "Hi, this is the virtual assistant for Northline Energy. I'm an AI. How can I help?"
- Returning caller with an open case: "Hi, I'm Northline's AI assistant. I see a recent open request on this number. Is that why you're calling?"
- Uncertain match: "Hi, I'm Northline's AI assistant. Can I check who I'm speaking with before we start?"
The returning-caller line confirms that a case exists and shares nothing about it. Details such as amounts, dates and notes wait until the caller passes a check.
Disclosure rules differ by region, and the list is growing:
- EU: Article 50 of the AI Act has applied since 2 Aug 2026. Any AI that talks to people, including automated phone systems, must say it is AI from the first interaction.
- Colorado, US: The Chatbot Safety Act takes effect on 1 Jan 2027 and requires public-facing conversational AI to disclose that it is AI.
- US federal: AI-voice calls already need consent under the TCPA after the FCC's 2024 ruling. The FCC's proposed "say it is AI" rule was still not final in mid-2026.
Plan the exit before launch. Gartner's 2026 survey of 3,566 customers found that 87% call an option to reach a human essential. Offer the route out early, and have the agent pass a short summary to the person who picks up.
Buyers raise three handover questions on their own. Is the transfer polite? What happens to a sensitive issue or a disputed amount? Does anyone chase a handover that nobody picked up?
Verify before you share any case details
The caller ID opens the file, and a second proof releases anything sensitive.
Use two layers:
- Knowledge question. Ask for something only the customer knows, such as the last four digits of an account or a recent transaction amount. Pick facts that sit in your CRM, not facts a stranger could find online.
- One-time code. Send a short code by SMS to the phone number on file. A caller who holds only a spoofed number never sees it.
Then split what the agent may say into tiers. Before any check, it can say that a case exists, whether it is open, and when the team will next act. After a passed check, it can give balances, payment details, health information and notes. Your compliance lead should write this list down once.
Be careful with voice-based checks. Voice cloning software can imitate a person's voice, so "say your passphrase" is a weak test. Prefer a code sent to a registered device.
If a check fails, allow two or three attempts, then stop and pass the call to a human with a note. Never say which answer was wrong. "I can't verify you yet" gives a fraudster nothing to adjust.
Privacy limits decide what history you keep
Keep the least history that still lets the agent do its job. GDPR's data minimisation principle limits personal data to what the purpose needs. Its storage limitation principle says to keep it for the shortest time possible. In practice:
- Store a case summary and status, not raw account numbers.
- Set a retention period per data type.
- Scope access by role.
- Never paste personal details into prompts or scripts.
People in the EU and UK can ask to see their data or have it removed. The same Commission page lists the right to access and the right to removal. Recording consent rules differ by region, so have your legal team set the opening notice.
Buyers ask where recordings live, which AI models touch them and who owns them. Some want hosting inside their own country, so settle this early.
Insight7 connects with telephony systems such as Dialpad, Aircall, RingCentral, Talkdesk and Five9, and writes recordings, summaries and transcripts back to Salesforce and HubSpot. Its beta inbound voice line is a number people ring, answered by an AI agent that can look the caller up. Callback requests go to a queue. Your team can also score every call against a scorecard it owns, and you can add an identity check as one criterion on any scorecard. A reviewer can read each transcript after the call.
Ask every vendor how long data is kept, how it is deleted, and whether you can review every call.
Related reading: How AI Voice Agents Verify Caller Identity Before Sharing Account Information, Caller Authentication Methods for AI Phone Agents Compared by Security and Customer Effort and 15 Inbound Call Types AI Voice Agents Can Resolve Without a Human.
Questions people also ask
How does an AI agent know who is calling?
It matches the caller's number to a CRM contact. A logged-in app session, a typed account number or a reference code identifies a caller better than a phone number does.
Can an AI voice agent remember my previous calls?
Not on its own. It reads whatever your systems stored: summaries, case notes and transcripts. If nothing was saved, nothing is recalled. Your retention rules set what it "remembers".
How does an AI agent find my open case or ticket?
It needs cases linked to your contact record by an ID, and a status field it can filter on. A case reference number gives it a direct route when the phone number matches nothing.
Does the AI have to tell me it is an AI when I call back?
We advise disclosing at the start of every call, since each call is a fresh interaction. The EU rule applies from the first interaction.
Is it safe for an AI agent to share account details over the phone?
Only after a second check passes, such as a one-time code or a knowledge question. Even then, the agent should read back masked figures, like the last four digits.
What happens if the AI recognizes the wrong caller?
A good agent drops the record when the caller says "that's not me". It shares nothing, treats the call as new, and logs the mismatch so your team can fix the entry.
How long do AI voice agents keep call history, and can I ask for it to be deleted?
Each company's retention policy sets that, so ask for the period in writing. In the EU and UK, people have a right to removal, though some records must be kept by law.
Start with clean records
Recognition is only as good as the records behind it. Check that your CRM holds one clean, current record per customer, with open-case status that updates in real time. Then see the lookup run on your data.
