TL;DR

  • A human-in-the-loop improvement workflow lets the agent surface problems while people approve every script change.
  • Roles, risk-tiered approval rules and a change log keep the loop safe.
  • Test each change before release, and log who approved what.

Map the loop and name the roles

Draw the loop on one page, then put a named person on every step. The agent surfaces problems, and people decide what changes.

  1. Call: the AI agent handles the conversation.
  2. Score: every call is scored against the team's scorecard.
  3. Flag: a low score, critical error or failed handover raises a flag.
  4. Propose: the QA analyst writes a change request.
  5. Review: the script owner and a risk reviewer check it.
  6. Approve: the approver for that risk tier signs off.
  7. Test: the change runs in a test call.
  8. Release: a small share of calls gets it first.
  9. Monitor: scores are watched, then the loop returns to step 1.
Role May do May not do
QA analyst Flag issues, attach evidence, propose changes Edit the live script
Script owner Write and edit wording, run tests Approve their own change
Compliance or risk approver Approve or reject high-risk changes Author the change they approve
Operations lead Schedule releases, order a rollback Skip a required approval
Auditor Read the log, sample decisions each quarter Change tickets or scripts

No one approves their own work. That one rule makes the table a control.

People stay in the loop because customers expect it. In a Gartner survey, 87% of 3,566 customers said a company using GenAI for service must offer a way to reach a human. Regulated buyers, banks first, name an AI that "goes rogue" as their main worry. They also ask who chases a handover nobody picked up.

Collect evidence from every call

Base every script change on all calls. A 2% manual sample can miss a wording problem that shows up in only a few calls, and it carries rater bias. Scoring every call lets a change request cite a pattern with call links, using a weighted scorecard the team owns.

Five signals should open a change request:

  • A low score on a script-driven criterion.
  • A critical-error fail.
  • A keyword alert, such as a customer asking "Am I talking to a robot?"
  • A failed handover.
  • Customer drop-off at the same point in the script.

Alerts can watch keywords and low scores, send a weekly digest of the weakest calls, and notify by email or Slack.

Let one critical error fail the whole call. QA leaders warn that a checklist score can hide what matters, because a call can be perfect on procedure and still leave the customer angry.

Calibrate before you trust the numbers. Have human reviewers score a sample of the same calls and compare. Built-in calibration lets managers review, override and fine-tune AI ratings. Fix scorecard criteria first, then script wording.

An insurer told us its agents kept disputing QA outcomes. It wanted a structured, supervisor-led way to contest a score. Contested scores are useful evidence too, because they show where criteria are unclear.

Write each script change as a ticket

Put every script change in a ticket, and make the ticket the first record in your audit trail. A ticket needs these fields:

  • Ticket ID and requester: who asked, and when.
  • Evidence: links to the calls that triggered it.
  • Old wording and new wording: the exact text, side by side.
  • Reason: which failure the change should fix.
  • Baseline and expected shift: for example, handover failures today and the drop you expect.
  • Risk tier: low, medium or high. This decides who approves.

One ticket carries one change. If a script owner edits the greeting and the consent line together, nobody can tell which edit moved the score. Small tickets also get approved faster, and a bad one is easy to roll back.

The baseline makes the ticket useful later. Gartner analysed 432 AI customer-service use cases. Only a quarter showed positive ROI, and 42% could not be measured. Without a baseline, your change joins that 42%.

Aim each change at a gap your QA data shows, not at a hunch. Training buyers want proof that behaviour changed, not that a session happened. Buyers also need numbers they can carry upstairs.

Set approval rules by risk tier

Sort every ticket into a risk tier, and let the tier set who signs and what testing is needed.

Tier Change types Approvers Test before release Rollback rule
Low Tone, greeting, pacing Script owner's peer, 1 approver Browser test Operations lead reverts if scores drop for two review periods
Medium Offers, pricing, qualification questions Script owner's manager plus operations lead, 2 approvers Browser test, then test call Revert if the target metric worsens against baseline
High Disclosures, consent, AI-identity statements, escalation and handover rules, sensitive data, collections language Compliance or risk approver plus operations lead, 2 approvers Test call, then a small-sample release Revert at the first compliance miss or failed handover

Three rules sit on top of the table:

  • Separation of duties. The author cannot approve. A high-risk change always needs compliance sign-off.
  • Emergency pause. Any approver or the operations lead can pause a script at once. A named reviewer must examine the pause within a set time, such as one working day.
  • Tier disputes. When in doubt, pick the higher tier.

Disclosure and consent wording belongs in the high tier because the law is moving. Examples by region:

A compliance lead at a bank will ask who approved an edit to the AI line, so name that person on the ticket. Handover rules are high risk too, because buyers ask how sensitive issues and disputed amounts are handled. A two-signature rule is the answer you can show buyers who fear a rogue agent.

Test, release and log every decision

Test every approved change before a customer hears it. Run low-risk edits in a browser test, and run medium and high-risk edits as a test call to a reviewer's own phone. Then release to a small share of calls, such as one in ten, and set a watch period with a rollback trigger written on the ticket.

Log each step as it happens. Each ticket should hold:

  • Ticket ID and evidence links
  • Script version before and after
  • Requester, reviewer and approver
  • Timestamps for each step
  • Test result and release date
  • Outcome review: did the metric move as expected?

Make the log read-only. Nobody edits an entry after it is written, and corrections are added as new entries. Keep it as long as your regulator or contracts require. The auditor samples tickets each quarter and checks that every high-risk change has a compliance signature. Keep this change-ticket log in your own ticketing system.

Insight7 supports the scoring side of this loop. It scores every call against a weighted scorecard your team owns. Alerts can watch for low scores and notify by email or Slack. Compliance teams can use calibration workflows to review and override AI scores. The audit trail Insight7 keeps covers call evaluations only, not script-change approvals. Separately, in beta, a Voice campaign's agent can be tested in the browser with no phone number needed. A reviewer can also read each call's transcript when it ends.

Related reading: How to Run Quality Assurance on AI Voice Agents Using the Same Scorecard as Human Agents, AI Voice Agent QA Scorecard Template With Criteria for Accuracy, Compliance and Customer Experience and How to Improve AI Voice Agent Performance From Real Customer Conversations.

Questions people also ask

What is a human-in-the-loop workflow for AI agents?

It is a repeatable process in which the AI agent handles calls and surfaces problems. Named people review, approve and log every change to its behaviour. In this workflow, the agent never rewrites its own script.

Who should approve changes to an AI agent's script?

Approval depends on the risk tier. A peer or manager can sign off low and medium changes. A compliance or risk approver signs off high-risk ones. The author never approves.

How often should you review and update an AI agent's script?

Review scores weekly. A weekly digest of the weakest calls gives you a natural review point. Batch low-risk edits, and run a full script review each quarter.

What should an audit trail for AI agent changes include?

It should let a stranger rebuild any decision a year later. That means who asked, who approved, which exact text changed, and what the metrics did afterwards. Store entries read-only.

Which AI agent script changes need compliance approval?

Any wording a regulator or customer could hold you to. That covers AI-identity statements, consent, disclosures, escalation and handover rules, sensitive data and collections language. Buyers raise handover design on their own, so treat it as high risk.

How do you test an AI agent script change before it goes live?

Talk to the agent in a browser first, since some tools allow this without a phone number. Then replay the triggering scenario in a test call. Finally, release to a small share of calls.

Do customers have to be able to reach a human when an AI agent handles the call?

The disclosure laws above centre on saying it is AI, rather than offering a human. Customers expect a human route anyway: 87% of surveyed customers said companies must offer one.

Let only signed changes reach customers

An AI agent should improve only through changes that a named person approved, tested and logged, with a baseline to prove the result. So take one question into your next meeting: who can approve a disclosure change today, and where is that decision recorded?

Book a demo of Insight7