The Problem We Faced

We needed to replicate our web training model on mobile and ship a streamlined app in one week. The mobile experience had to let users create and take training quickly, Ideally generate a usable training session in under one minute — and the interaction had to feel like a real interview: near real-time, conversational LLM responses. At the same time, we had to prevent the model from going out of scope or hallucinating answers that weren’t related to the training content. Those two constraints: latency and scope control, drove every engineering decision we made.

Concretely, our goals were:

• Training generation end-to-end (input → ready-to-take training) < 60s for first-time users.
• Response time for conversational turns ≈ 200–500 ms perceived latency (streaming preferred).
• High relevance: < 5% of model responses considered “out of scope” in manual QA.
• Maintain battery and network-efficiency acceptable on mid-tier devices (Android API 29+, iOS 14+).

We already had a robust web implementation: server-hosted model, interactive UI, structured templates, and a dataset of role-play scenarios. The mobile challenge was replicating that experience with minimal navigation, predictable runtimes, and safe, on-topic LLM outputs without degrading UX.

Our First Attempt

We started by porting the web flow directly: keep the same server-side inference, same prompts, same template engine, and build a mobile UI (React Native) that calls the same endpoints. That was the fastest way to mirror behavior and keep product parity.

What we implemented first:

• Mobile front-end using existing REST/GraphQL endpoints.
• Server-side LLM calls using our web stack.
• Template-driven training generation: same templates as web, same pipeline for sampling question lists and role instructions.

Why this failed for mobile UX:

• Latency spikes: round-trip times plus server-side model inference caused initial training generation to average 12–18 seconds on good networks — okay — but conversational LLM responses averaged 800–1200 ms per turn, causing the session to feel sluggish on mobile.
• Perceived responsiveness suffered due to a lack of early partial tokens. Our HTTP chunking approach didn’t provide a smooth stream to RN text components; re-rendering and JS bridge overhead made partial tokens batch up.
• Scope drift: the same prompt strategy that worked on web occasionally produced unrelated answers on mobile, especially when users free-typed follow-ups. Without local context checks, the server model hallucinated or went tangential 8–12% of the time — above our target.
• Battery and data: continuous streaming with long-lived connections elevated power usage and data consumption, which we observed in internal testing on mid-tier devices.

We iterated on smaller fixes (retry logic, aggressive caching, smaller payloads), but the fundamental issues were the network/model latency and the prompt/control strategy. These would require architectural changes.

The Breakthrough

We shifted from “replicate web exactly” to “mobile-first hybrid architecture.” Our central insight: combine lightweight on-device inference for an immediate conversational feel with server-side models and vectorized context for accuracy and scope control. This hybrid allowed the user to feel instant responses while the server validated or augmented replies for factual relevance.

Key components we designed and implemented:

• Lightweight on-device LLM for immediate conversational turns.
• Server-side heavy model + RAG (retrieval-augmented generation) for validation and fallback.
• Template-driven training generator optimized for one-minute creation.
• Low-latency streaming and optimistic UI updates.
• Scoped prompts + safety layer to avoid hallucinations.

Latency strategy

• On device, we ran a distilled LLM (Llama 2 7B distilled quantized to 4-bit with GPTQ, via ONNX/ORT Mobile) to generate tentative replies within 80–220 ms on modern mid-tier devices (Snapdragon 7-series class). This gave a near-instant “typing” response.
• Simultaneously, the mobile app sent the same prompt to the server. The server generated a vetted response using full context and RAG. When the server response arrived (typically 300–700 ms on 5G; 700–1500 ms on 4G), we compared it to the on-device response.
• If server and local responses matched at a semantic level (cosine similarity > 0.82 using a compact sentence embedding), we kept the on-device reply. If not, we replaced or merged results with the server response.

Streaming and optimistic UI

• We implemented token-level streaming on-device and from the server. On-device streaming produced immediate token-by-token UI updates. We used a small debounce (25–40 ms) to batch micro-updates and avoid JS bridge thrashing.
• For server streaming, we used HTTP/2 Push + SSE with chunked transfer encoding for fallback. We implemented a merge strategy: show local stream immediately, overlay server tokens as they arrive, and if the server diverged significantly, perform a smooth replace animation.

Template-driven one-minute generation

• We pre-built concise templates for common training types: behavioral interviews, product demos, compliance checks. Templates had placeholders and a metadata header describing difficulty, time, and expected number of turns.
• Generation pipeline: user selects a template → app (locally) populates placeholders with user inputs → lightweight server call to expand into a question set (if needed). On-device templating allowed us to produce a usable training session in under 10–20s for most templates; with server expansion for advanced templates the end-to-end time stayed under 60s.
• For first-launch cold starts we cached template packs in-app (≈400 KB per pack) so the UI workflow required no server round trip for selection.

Scope control and anti-hallucination

• We implemented layered guardrails:
  • Prompt engineering: explicit system instructions with strict role constraints and “do not invent” clauses.
  • Contextual RAG: when a response referenced facts or domain specifics, we retrieved the related training doc snippets from Pinecone (or FAISS on-prem) and included them as grounding context.
  • Post-generation classifier: a lightweight BERT-based classifier (fine-tuned) checked relevance and flagged outputs scoring below threshold 0.65. Flagged responses triggered server-side regeneration with stronger constraints or a canned fallback.
  • Local heuristics: simple regex/topic filters and conversation length limits prevented topic drift.
• We maintained an allowed-topic vector signature per training session: a compact embedding centroid computed from training materials. Each reply embedding was compared to the centroid; replies below similarity 0.6 were rejected.

Model orchestration

• Dual inference flow:
  • Fast path: local LLM = immediate reply (used 90% of the time for UX).
  • Validation path: server LLM = authoritative reply (used to correct or augment).
• We prioritized user-visible latency: always show the local reply first. Users rarely notice subsequent corrections if changes are small and the UI animates replacements gracefully.

Monitoring and metrics

• We added instrumentation at both client and server:
  • Client: per-turn perceived latency, battery delta per session, data bytes per session.
  • Server: inference time, RAG retrieval latency, classifier reject rate.
• Target metrics we aimed for and achieved in pilot:
  • Median perceived first-token time (mobile): 180 ms
  • Median server-validated reply latency: 620 ms on 5G; 980 ms on 4G.
  • Scope-hallucination rate (manual QA): reduced from 8–12% to 2–3%.
  • Training generation time median: 42 sec (first-time); 18 sec (template-only).

Testing and iterative tuning

• A/B tests: local-only vs hybrid validated the UX gains. Users rated hybrid sessions 4.6/5 for “conversationality” vs 3.8/5 for server-only.
• We tuned thresholds (similarity, classifier cutoff) with cross-validation over a labeled dataset of 2,400 past sessions.

Results and Tradeoffs

Results

• Launch readiness: We achieved the one-week launch timeline for a minimum viable product that met the “generate training under 1 minute” goal for the majority of users.
• Latency: perceived near-real-time responses improved significantly. Median first-token time fell into our 200 ms target on modern devices; full server-validated replies averaged under 1s on common mobile networks.
• Relevance: hallucination rate dropped to ~2–3% per our QA sampling. When the classifier flagged an output, the regeneration step produced usable alternatives in >85% of cases.
• UX: users perceived the experience as more conversational; session completion rates increased by ~12% in pilot tests.

Tradeoffs and limitations

• Complexity: hybrid architecture introduced more moving parts (on-device inference, server validation, merge logic). This increased engineering and maintenance cost. We added cross-team complexity (mobile, backend infra, ML ops).
• Device variability: on very low-end devices or older OS versions, on-device LLM latency rose to 500–900 ms, reducing the benefit. We shipped a capability probe at first launch to choose: hybrid vs server-only fallback.
• Model consistency: occasional semantic mismatches required careful replacement UX. We could not fully avoid confusion when the local reply and the server reply diverged significantly and the user had already acted on the local text.
• Storage and app size: shipping a quantized local model added ~20–60 MB depending on the model and quantization. For users with limited storage, we implemented on-demand download with a small stub that used server-only inference until the model finished downloading.
• Privacy vs accuracy: we chose to keep minimal PII on-device. To improve RAG accuracy, some teams wanted more domain docs on-device, but that increased storage and privacy concerns. We opted for server-hosted documents and encrypted transport for retrieval.

Operational costs

• Server-side validation increases API/inference cost. We optimized by:
  • Using the heavy server LLM only when needed (validation failures, long-tail queries).
  • Caching validated responses and reuse across similar sessions.
  • Batch processing RAG retrievals and using compact embeddings to reduce vector DB lookup costs.

Lessons Learned

• Hybrid works well when UX requires immediacy and server validation is needed for correctness. The UX boost from on-device optimistic responses was worth the added complexity.
• Instrument early and often. We instrumented perceived latency at the UI level (first-token time) and correlated it with backend metrics to pinpoint bottlenecks.
• Templates are a force multiplier. Investing in concise, high-coverage templates reduced generation time dramatically and kept sessions focused.
• Grounding is essential. RAG plus a lightweight semantic-similarity filter reduced hallucinations faster than trying to enforce constraints solely through prompt engineering.
• Keep graceful fallbacks. Device detection and capability negotiation (download model vs server-only) prevented poor experiences on older phones.
• UX smoothing matters as much as model quality. Users tolerate small corrections if we avoid jarring replacements; animations and micro-interactions reduce cognitive load.
• Monitor costs and usage. Validation paths improve quality but can double inference costs; selective gating and caching mitigate this.

If we were to continue evolving this system:

• We’d explore smaller, specialized on-device models fine-tuned for role-play/dialogue to further tighten similarity and reduce divergence.
• We’d evaluate 4-bit quantization improvements and ONNX optimization passes to shrink model size and improve on-device latency on low-end devices.
• We’d add stronger personalized safety layers that learn from user corrections to reduce repeated server rounds.

We shipped a mobile experience that felt live, kept users on topic, and met our aggressive launch timeline. The hybrid pattern — optimistic local inference with server-grounded validation — is now our baseline for real-time mobile conversational features.