CX Meets AI: Engineering Call Intelligence That Actually Listens
The Business Challenge: When Support Calls Are a Black Box Before we built our CX dashboard, support calls were essentially invisible to operations and product teams. Companies were sitting on thousands of customer conversations every week — containing critical signals about product issues, service gaps, sales opportunities, and operational failures — but had no systematic way to extract insights. The cost of this blindness was real: Operations couldn’t pinpoint root causes of repeat calls Service leaders had no visibility into whether agents were resolving issues effectively or escalating unnecessarily Sales opportunities mentioned in support calls were completely invisible, leading to missed revenue Without data, process improvements were based on gut feel rather than evidence Why this matters: For companies handling thousands of support calls, even small improvements compound dramatically. A 5% increase in first-call resolution saves agent hours and improves customer satisfaction. Identifying just 10% more sales opportunities in existing calls can represent significant revenue. Catching product issues earlier reduces churn and development waste. The market needed a solution that could actually listen to calls at scale — not just track call volumes, but understand intent, sentiment shifts, product mentions, and operational patterns — and surface this intelligence in a way that operations, product, and sales teams could act on immediately. That’s the problem we set out to solve. Here’s how we built it. The Problem We Faced We were building a customer experience (CX) dashboard to give operators and product teams clear visibility into what happens on support calls. The dashboard had four main sections — Product, Service Quality, Sales Opportunity, and Operations — and needed to surface everything from sentiment trends (initial/mid/final), first-call resolution (FCR) and effective communication KPIs, to operational call-reason breakdowns (warehouse fulfillment failures, wrong-item complaints), and escalation trails. The hard constraint: the dashboard must actually “listen” — not just show volumes — and be dynamic enough to meet wildly different customer needs without rebuilding the whole stack for every account. At project start we had raw call recordings, partial metadata (agent ID, time-on-call), and a set of business questions from stakeholders. We needed: accurate transcription and diarization; robust intent and reason classification; sentiment over call segments; extraction of product mentions and feature requests; sales-opportunity detection; and per-call resolution/escalation tracking — all updated frequently enough to inform operations. Our First Attempt Our initial architecture was straightforward and familiar: stream recordings into S3, run a single ASR (automatic speech recognition) model across everything, push transcripts into a classic NLP pipeline (heuristic regex + keyword lists + a light classifier), and layer metrics into a single monolithic BI dashboard (React + Superset). We used Amazon Transcribe (standard) for ASR, a simple speaker-turn heuristic for diarization, Vader for sentiment, and a logistic-regression classifier trained on 3k labeled call excerpts for reasons (fulfillment, wrong-item, billing, etc.). We shipped a V1 dashboard to a pilot customer within eight weeks. That rollout taught us a lot quickly: Transcription quality varied wildly (WER 18–35%) depending on line noise, accents, and domain phrases — which cascaded into downstream NLP errors. Our single-model approach missed intent nuances: feature requests vs. complaints vs. comparisons were often conflated. Sentiment aggregated for whole calls hid critical dynamics: an angry start with a calm resolution still showed “neutral” overall. The dashboard’s static metric set didn’t align with some customers’ KPIs — one wanted detailed escalation timelines, another wanted product feature-mentions grouped differently. We tracked core metrics: end-to-end processing latency was ~45 minutes per call (batch-only), reason-classifier precision ~0.77, recall ~0.69, and sentiment accuracy ~0.72 against hand-labeled samples. Those numbers were neither stable nor sufficient for operational trust. Why It Failed We learned why the naive stack failed in production: Error Amplification: ASR errors (high WER) directly reduced classification and named-entity extraction accuracy. A single bad transcription could flip a call’s reason tag. One-Size-Fits-All Models: Domain variance (different product names, jargon, call scripts) meant a global model underfit most customers and overfit the pilots. Temporal Blindness: Aggregating sentiment once per call missed transitions (initial frustration → mid-call calming → final satisfaction). KPIs like “effective communication” require segment-level signals. Static Dashboarding: The monolithic dashboard had hardcoded metric definitions and required engineering to add any new view. Customers wanted dynamic breakouts, e.g., seeing “wrong-item” split by warehouse ID or by SKU family — not possible without rebuild. Trust Gap: Operations needed auditable evidence (timestamps, utterance text, escalation points). Our pipeline didn’t carry provenance metadata end-to-end. We could have iterated the original system ad infinitum, but that would chase symptoms. We needed architectural changes that reduced upstream fragility, enabled per-customer specialization, and provided explainability. The Breakthrough We reframed the problem: rather than a single pipeline that outputs “answers,” we would build a modular call-intelligence platform that produces trusted, auditable artifacts (segment-level transcripts, time-aligned sentiment and intent labels, entity records, and embeddings) and a flexible dashboard layer that composes views from these artifacts by configuration. The key design pillars became: Modular audio processing with fallbacks Segment-aware NLP and time-series sentiment Per-customer configuration and on-demand specialization Explainability and provenance at every step Operational SLAs for latency and accuracy Below we describe the architecture, implementation choices, and how we operationalized trust. Implementation Details Models and training: Labeled dataset: 40k calls (>2M utterances) aggregated from pilots (consented), stratified by product line and geography. Holdout test: 5k calls. Intent/reason model: DeBERTa v3-small fine-tuned (Hugging Face Transformers v4.30), initial learning rate 2e-5, batch 32, trained for 4 epochs. Precision/recall on test set: 0.86 / 0.82 (F1 0.84). Sentiment model: BERT-based classifier for 3-way sentiment, but we also used a regression score combined with a rule engine to identify sentiment flips; 3-window accuracy = 0.88. ASR: We tuned acoustic adaptation on common product names and used Lexicon/Custom-Vocabulary features in Amazon Transcribe for customer-specific terms. For persistent low-confidence segments (<0.6), Whisper-large-v2 fallback reduced WER by ~6 percentage points. Per-customer customization: Config model: each customer has a JSON schema describing product taxonomy, escalation tags, radar metrics, and dashboard templates. These configs drive NER dictionaries, custom vocabularies, and dashboard breakouts. Feature toggles:
Every CEO Wants AI-Driven Growth. Most Are Looking in the Wrong Place
I spend a lot of time meeting with CEOs across industries and every single one of them is thinking about AI. Some are already adopting AI tools. Others are deep in evaluations, building business cases, running pilots. A few are still in the exploration phase, trying to separate hype from reality. But they all share the same goal: use AI to drive growth in 2026. What a number of them miss however is that the biggest unlock for AI isn’t the tools themselves but the data and context you feed them. The Expensive Detour The default playbook goes something like this: buy the latest AI tool, implement the newest model, chase what your competitors are doing. New sales AI. New customer service automation. New analytics platform. Another AI layer on top of your existing stack. Each promises transformation. Each costs five, six or seven figures. Each takes months to implement. And most deliver incremental improvements at best. Why? Because you’re trying to build on empty ground. The Answer Is Already in Your Business Most companies are already sitting on the raw material for AI-driven growth. It’s in the thousands of customer conversations happening every week across the business. Sales calls. Support tickets. Implementation check-ins. Success reviews. Onboarding sessions. These conversations contain insights that need to be unlocked: Why customers aren’t buying Where reps excel or struggle What messaging actually resonates What products are in demand Which objections kill momentum What is changing in the market The insights that could transform your revenue trajectory are already there. You’re just not extracting them. The Hidden Cost of Siloed Insights Even when companies analyze conversations, most never capture their full value. The problem isn’t a lack of insight — it’s that insights are trapped, siloed, and disconnected from the people who can act on them. Its a problem we are solving at Insight7.io and the result appears everywhere: Leadership sees fragments, not patterns: Decisions on go-to-market, product development and training rely on anecdotes, not reality. Managers can’t scale coaching: Feedback stays generic because they lack tools and context to develop each rep based on real performance. Reps don’t improve: Delayed, vague feedback disconnected from actual conversations keeps performance flat. When insights are siloed like this, AI tools alone won’t move the needle. Data is only valuable when it flows to the people who can act on it. The Unlock: Conversation Intelligence Across Your Business The companies that will succeed with AI in 2026 won’t be chasing the newest AI model. They will find ways to systematically unlock the intelligence buried in their customer conversations. These companies will: Evaluate 100% of customer interactions — not just a sample Surface patterns across every touchpoint (sales, support, success, implementation) Generate personalized coaching at scale so reps actually improve Flow insights automatically to leadership for strategic decisions This isn’t about replacing human judgment. It’s about giving your people the intelligence layer they need to perform at their best. The Question for 2026 Instead of asking: “How do we use AI to grow?” The better question is: “Are we using the data we already have?” Before you buy another AI tool, ask yourself: What percentage of our customer conversations are we actually learning from? Do insights from those conversations reach the people who can act on them? Can we turn conversation data into systematic coaching and strategic intelligence? If the answer is no, you’re not ready for more AI tools. You’re ready for conversation intelligence. The companies that figure out conversation intelligence won’t just win in 2026 — they’ll build an advantage that’s impossible to copy. If you’re interested in unlocking your customer conversation data, we’re solving this at Insight7.io. Reach out.
A Week, an Idea, and an AI Evaluation System: What I Learned Along the Way
How the Project Started I remember the moment the evaluation request landed in my Slack. The excitement was palpable—a chance to delve into a challenge that was rarely explored. The goal? To create a system that could evaluate the performance of human agents during conversations. It felt like embarking on a treasure hunt, armed with nothing but a week’s worth of time and a wild idea. Little did I know, this project would not only test my technical skills but also push the boundaries of what I thought was possible in AI evaluation. A Rarely Explored Problem Space Conversations are nuanced; they’re filled with emotions, tones, and subtle cues that a machine often struggles to decipher. This project was an opportunity to explore a domain that needed attention—a chance to bridge the gap between human conversation and machine understanding. What Needed to Be Built With the clock ticking, the mission was clear: Create a conversation evaluation framework capable of scoring AI agents based on predefined criteria. Provide evidence of performance to build trust in the evaluation. Ensure that the system could adapt to various conversational styles and tones. What made this mission so thrilling was the challenge of designing a system that could accurately evaluate the intricacies of human dialogue—all within just one week. What Made the Work Hard (and Exciting) This project was both daunting and exhilarating. I was tasked with: Understanding the nuances of human conversation: How do you capture the essence of a chat filled with sarcasm or hesitation? Developing a scoring rubric: A clear, structured approach was essential to avoid ambiguity in evaluations. Iterating quickly: With a week-long deadline, every hour counted, and fast feedback loops became my best friends. Despite the challenges, the thrill of creating something groundbreaking kept me motivated. The feeling of building something new always excites me—it’s unpredictable, and there was always a chance the entire system could fail. Lessons Learned While Building the Evaluation Framework Through the highs and lows of this intense week, I gleaned valuable insights worth sharing: Quality isn’t an afterthought—it’s a system. Reliable evaluation requires clear rubrics, structured scoring, and consistent measurement rules that remove ambiguity. Human nuance is harder than model logic. Real conversations involve tone shifts, emotions, sarcasm, hesitation, filler words, incomplete sentences, and even transcription errors. Teaching AI to interpret this required deeper work than expected. Criteria must be precise or the AI will drift. Vague rubrics lead to inconsistent scoring. Human expectations must be translated into measurable and testable standards. Evidence-based scoring builds trust. It wasn’t enough for the system to assign a score—we had to show why. High-quality evidence extraction became a core pillar. Evaluation is iterative. Early versions seemed “okay” until real conversations exposed blind spots. Each iteration sharpened accuracy and generalization. Edge cases are the real teachers. Background noise, overlapping speakers, low empathy moments, escalations, or long pauses forced the system to become more robust. Time pressure forces clarity. With only a week, prioritization and fast feedback loops became essential. The constraint was ultimately a strength. A good evaluation system becomes a product. What began as a one-week sprint became one of our most popular services because quality, clarity, and trust are universal needs. How the System Works (High-Level Overview) The evaluation system operates on a multi-faceted, evidence-based approach: Data Collection: Conversations are transcribed and analyzed in over 60 languages. Evaluation on Rubrics: The AI evaluates transcripts against structured sub-criteria using our Evaluation Data Model. Scoring Mechanism: Each criterion is scored out of 100, with weighted sub-criteria and supporting evidence. Performance Summary & Breakdown: Overall summary Detailed score breakdown Relevant quotes from the conversation Evidence that supports each evaluation This approach streamlines evaluation and empowers teams to make faster, more informed decisions. Real Impact — How Teams Use It Since launching, teams across product, sales, customer experience, and research have leveraged the evaluation system to enhance their operations. They are now able to: Identify strengths and weaknesses in AI interactions. Provide targeted training to improve agent performance. Foster a culture of continuous, evidence-driven improvement. The real impact lies in transforming conversations into actionable insights—leading to better customer experiences and stronger business outcomes. Conclusion — From One-Week Sprint to Flagship Product What started as a one-week sprint has now evolved into a flagship product that continues to grow and adapt. This journey taught me that the intersection of human conversation and AI evaluation is not just a technical pursuit—it’s about understanding the essence of communication itself. “I build intelligent systems that help humans make sense of data, discover insights, and act smarter.” This project became a living embodiment of that philosophy. By refining the evaluation framework, addressing the nuances of human conversation, and focusing on evidence-based scoring, we created a robust system that not only meets our needs but also sets a new industry standard for AI evaluation.
