The problem – Teams came to us for speed. They had call transcripts and needed a fast way to extract what mattered – a quick TL;DR they could act on. Our summarization service delivered that, and customers relied on it heavily.

But as usage grew, the same request kept coming up: “Can we control the format?”

Instead of a generic summary, customers wanted outputs that matched how they already worked —- an email follow-up ready to send, an executive one-pager for leadership, or a checklist with prioritised action items. They weren’t asking for more text. They were asking for predictable structure.

What they needed were summaries that came back in the exact format they specified, every time.

Why does this matter?

Customers needed to feed summaries into downstream systems like CRMs and ticketing platforms. When field names changed or required sections were missing, those integrations broke. Customers couldn’t build reliable automations on top of unpredictable outputs.

Before we solved this, enterprise customers were manually editing generated summaries to fix formatting issues, wasting time on work that should have been automated. Legal and compliance teams couldn’t rely on summaries when format consistency wasn’t guaranteed.

What’s the benefit of solving it?

After implementing our solution, we achieved 92% structural adherence – summaries now reliably match customer templates. The business impact was significant:

• 75% reduction in manual edits: Enterprise customers stopped spending time reformatting AI outputs
• Reliable automation: Customers could now build downstream automations relying on consistent field names and types
• Faster enterprise adoption: Customers who needed CRM and ticketing system integration adopted the feature quickly
• Increased trust: Legal and compliance teams gained confidence from audit logs and consistent formatting

The difference between 62% and 92% accuracy meant the difference between summaries that required constant human cleanup and summaries that could power business-critical workflows.

Our First Attempt

Our initial implementation was minimal: accept a free-form template string from users, append it as an instruction to the summarization prompt, and call a single large model (OpenAI GPT-4) with the transcript context. The pipeline looked like:

• Transcription (Whisper v1) -> transcript text
• Prompt = “Summarize the call according to this template: [user template]” + transcript
• One-shot model call -> return text to user

This approach worked quickly in demos and solved some cases, but it failed in the real world for several reasons:

• Prompt sensitivity: Outputs varied based on subtle template wording. When a customer used imprecise language (e.g., “Make it sound like an email but not too formal”), the model interpreted that differently each run.
• Structural drift: Headings were renamed, placeholders were dropped, or sections were merged. We saw ~62% structural adherence (heading names + presence of required placeholders) across a 1,000-template test set.
• Malicious / invalid templates: Templates with embedded HTML, code, or attempts to override system instructions could produce unexpected output or security concerns.
• Uncontrolled token usage: Long templates + long transcripts led to high token use and unpredictable costs.
• User error: Many users submitted templates with ambiguous placeholders or filler words, increasing “garbage in, garbage out” failure modes.

We tried several incremental fixes: stricter front-end validation, examples to users, and a longer prompt telling the model to “follow headings exactly”. None of these reliably fixed the core problem. The more we leaned on the single-model approach, the more we saw variable fidelity across template styles and transcripts.

The Solution

We adopted a layered, deterministic pipeline that treats the user template as a first-class artifact: parse → sanitize → canonicalize → plan → generate → validate. The core idea: don’t hand raw user text to the generative model and hope. Instead, turn the template into a machine-checked specification (a schema), use a controlled “meta-prompt” to convert the template into strict generation instructions, and validate output against that schema. We split responsibilities across smaller, specialized components so each step is auditable and testable.

Architecture overview (components and tools)

• Ingress: API (Kubernetes 1.26, FastAPI on Python 3.11)
• Storage: S3 for transcripts, PostgreSQL 15 for metadata
• Workers: Celery 5.2, Redis 7 for task queue and caching
• Models: OpenAI GPT-4 / gpt-4o-mini for generation, GPT-4-Fast for meta-prompting when we needed speed
• Libraries: pydantic v1.10, jsonschema 4.17, spaCy 3.5 for NER, bleach for sanitization
• Monitoring: Prometheus + Grafana, Sentry for errors

Key pipeline stages

• Template Sanitization
  • Strip HTML, disallowed control characters, and executable code with bleach and regex filters.
  • Enforce length limits: template body < 4,096 chars (configurable).
  • Extract explicit placeholders (we support simple placeholder syntax: {{name}}, {{action-items}}, etc.).
• Template Parsing & Schema Generation
  • We convert the cleaned template into a JSON Schema / “blueprint” that captures required sections, headings, and data types (string, list, bullets, optional/required).
  • We validate that the template contains at least one stable anchor (e.g., at least one heading or placeholder). If not, we return a friendly error with suggested fixes.
  • Example conversion rule: a line starting with “###” becomes a required object property; a bullet-list instruction becomes an array type.
• Meta-Prompting (Prompt-of-a-Prompt)
  • We generate a compact, deterministic instruction for the generator model by combining:
    • The normalized schema (short).
    • Example outputs that match the schema (we keep a library of 60 curated examples).
    • Constraints: JSON-only output when requested, strict heading names, maximum token lengths for sections.
  • We use a small, faster model (gpt-4o-mini or an optimized instruction-tuned variant) to turn the user’s natural-language template into the canonical meta-instructions if parsing heuristics cannot deterministically infer the full schema.
• Constrained Generation
  • We ask the model to produce output that either:
    • Emits JSON conforming to the schema, OR
    • Emits text with exact headings and clearly delimited sections.
  • We favor JSON output when downstream systems need to programmatically consume summary fields.
• Validation & Repair
  • We validate the model output against the schema using jsonschema. If it fails, we run a repair pass:
    • Identify missing required fields and call the model with a focused prompt: “You missed X. Fill it using transcript references. Answer only the field X.”
    • We allow up to two repair attempts before falling back to a deterministic extractor (rule-based NER + regex) for simple fields.
• Safety & Audit Logging
  • Every sanitized template, schema, meta-prompt, model outputs, and validation traces are logged to an append-only audit store. We keep hashes for integrity checks.
  • We run adversarial pattern checks (e.g., templates containing “ignore prior instructions”) and reject those with advice for safer phrasing.

Small, concrete examples

We use a JSON Schema to make the generator output precise.


Operational considerations

• Caching: For repeated templates we cache the generated schema and meta-prompt to avoid repeated parsing; cache TTL is 24 hours by default.
• Cost & latency: We sometimes run two model calls (meta + generate). To balance cost, we use meta calls only on first-time templates or when parsing heuristics fail; else we reuse cached meta-prompts.
• Testing: We fuzzed with 5,000 synthetic templates and 10,000 transcripts. We enforced SLOs: 95th percentile latency < 1.5s for generation-only cached runs, < 2.8s for meta+generate flows.
• Developer ergonomics: We provide a template linter and live-preview UI so users iteratively refine templates; that notably reduced malformed submissions.

Results and Tradeoffs

Quantitative outcomes (real metrics)

• Structural adherence: improved from ~62% to ~92% (measured as correct heading names + presence of required placeholders across 5,000 random tests).
• Human edit reduction: post-release, our support-tracked edits of generated summaries dropped by ~75% for enterprise customers who used the configurable mode.
• Repair success: For outputs that initially failed schema validation, our automated repair pass fixed ~70% of cases; the rest required manual review.
• Latency: median end-to-end latency increased from ~420 ms to ~580 ms in cached flows, and from ~860 ms to ~1,240 ms in non-cached meta+generate flows. These are medians measured on a 4-vCPU worker fleet.
• Cost: average API token usage increased by ~28% per completed summary due to schema context and occasional repair calls; net unit cost rose by ~22% considering model choice optimizations.
• Coverage: About 85% of customer templates could be fully auto-parsed into a schema; for the remaining 15%, we used the meta-prompt step. We also provided an “assisted template builder” UI to convert the last 15% to structured templates.

Qualitative outcomes

• Predictability: Customers reported that they could now build downstream automations relying on consistent field names and types.
• Adoption: Enterprise customers who needed downstream ingestion (CRMs, ticketing systems) adopted the JSON-output mode quickly.
• Trust: Audit logs and deterministic validation increased confidence; legal and compliance teams liked the change.

Tradeoffs and limitations

• Complexity: The layered pipeline and schema generation introduced operational complexity. We now maintain more code paths, caches, and monitoring.
• Cost vs. reliability: Guaranteeing structure required extra computation and sometimes multiple model calls. We balanced this with caching and smaller meta-models, but there’s a baseline cost increase.
• Model drift and maintenance: If the base generator model behavior shifts, we may need to update meta-prompting strategies and repair heuristics. Continuous monitoring and automated regression tests are essential.
• User education: Some customers tried to use overly artistic templates. We had to invest in templates best-practice docs, linters, and UI wizards. This helped but is an ongoing support cost.
• Edge-case failure modes: Extremely long transcripts (e.g., >200k chars) still require chunking and merge strategies. Some nuanced judgments (tone, implied sarcasm) remain hard to capture deterministically.

Lessons Learned

• Make the template machine-checkable. The biggest stability wins came when we turned free-form templates into schemas. Once we had machine-checkable contracts, we could validate, repair, and reason about outputs deterministically.
• Meta-prompting is powerful but expensive. Use it sparingly: cache meta-prompts and fall back to rule-based parsing where possible.
• Validate early and loudly. Reject bad templates at ingress with actionable messages. Users appreciate immediate, clear feedback rather than mysterious model failures.
• Provide examples and tooling. A live-preview and linter cut down bad templates by over 40% in our beta cohort.
• Monitor structural adherence, not just human preference. We track schema pass rates, repair rates, and human-edit distances; these metrics are better for engineering than subjective quality scores.
• Balance cost and latency with customer needs. Offer tiers: strict-JSON mode for reliable automation (higher cost), and lightweight TL;DR mode for quick human-only summaries (lower cost).

Final Thoughts

Configurable summaries that are both precise and reliable require more than a good model. They require a system design that treats user intent as a product requirement: sanitize inputs, convert them to explicit contracts, generate deterministically, validate mechanically, and provide clear fallbacks. We improved structural adherence by 30 points, dramatically reduced manual edits, and delivered a system our customers could build automations on.

We still operate within tradeoffs — cost, complexity, and maintenance — but the win is tangible: customers trust the summary format, downstream integrations are simpler, and our support load for formatting issues has dropped. Our next work items are tighter schema discovery UX, incremental streaming generation for very long transcripts, and better provenance linking (aligning generated statements to specific transcript timestamps) so customers can click to verify a generated action item in the original audio.