Stack Sensei — AI Voice Mentor Platform

Technical interviews are spoken conversations under pressure, yet nearly every prep tool makes you type. Stack Sensei 🥋 is a voice-native AI mentor: you talk, it listens, grades, and talks back — asking adaptive technical questions across TypeScript/React, Java/Spring/Hibernate, Bash/Linux, and operations/security, running practice sessions with hints, twenty-question certification exams, and full interview simulations. The product idea is simple; the engineering case study is in what it takes to make voice reliable enough to trust mid-interview — because a voice app that goes silent when one vendor hiccups isn’t a product, it’s a demo.

The problem

Production voice AI has four failure classes, and each one got a deliberate design answer:

• Availability. Hardwiring one TTS vendor means one outage, one rate limit, or one latency spike kills the session at the worst possible moment — mid-answer, mid-exam.
• Expressiveness. One-size-fits-all synthesis sounds identical whether you aced the question or bombed it. No human mentor sounds like that, and learners hear the difference immediately.
• Identity. If failover swaps your mentor’s voice for a random one, the persona breaks — worse than a brief silence, because it feels like the product glitched.
• Economics and lock-in. A single hardwired LLM means one vendor’s pricing, latency, and outages, with no user control over keys or cost — unacceptable for a product whose unit economics are per-conversation.

Design decisions

A three-tier TTS failover chain, each tier chosen for a different property

The chain is not three interchangeable vendors — each tier earns its slot with a distinct property:

1. ElevenLabs (primary) — highest quality, streaming over WebSocket rather than HTTP, so audio starts playing before synthesis completes. Time-to-first-audio is the metric that makes a voice app feel alive.
2. Kokoro (secondary) — an open TTS model I self-host on Fly.io. It runs on the private WireGuard mesh, reachable only at an internal address (shared-tts.internal:8880) with no public IP. It’s tuned for zero cold start: machines never auto-stop, at least one instance always runs, and the model preloads during a ~95-second warmup so failover never waits on model load. Quality drops a grade — but latency and availability become properties I control, not properties I rent.
3. OpenAI TTS (tertiary) — not the best voice, but the most reliable last resort over plain HTTP streaming.

Around the chain: a dedicated health monitor probes the self-hosted tier continuously; a metrics exporter and per-provider usage tracking make cost and failure visible per tier; and a text preprocessor rewrites code identifiers, markup, and symbols into something speakable before any tier sees them — “useEffect” should be spoken as a word, not spelled as line noise.

Emotion mapped into synthesis parameters

The tutoring LLM doesn’t just grade an answer — it yields an emotion context, one of five states, each mapping to concrete ElevenLabs synthesis settings:

Each state adjusts stability, similarity boost, and style. It’s a small mapping table with an outsized effect: the sensei audibly softens when you struggle and fires up when you deliver, which is the difference between a mentor and a screen reader.

Personas that survive failover

Thirteen mentor voices, each defined by structured characteristics — gender, age, energy, style — and each carrying a designated Kokoro fallback voice ID, so when the chain fails over, your mentor still sounds like roughly the same person instead of switching identity mid-sentence. The fallback mapping is honest about quality: each Kokoro substitute carries an internal grade (one persona’s fallback is rated a C+, another an A−), which drove which personas got promoted in the catalog. Voice #13 is “Luca Sensei” — wise, calm, pragmatic systems architect — my own founder persona, the same character as this site’s avatar, selectable in the product with its own portrait.

Bring-your-own-key intelligence with per-user isolation

The LLM layer abstracts eight providers behind one interface — Claude (default: Sonnet), OpenAI (GPT-4o), Gemini, Mistral, DeepSeek, Qwen, Kimi, and local Ollama (llama3:8b) — with per-user API keys, key validation, and availability checks. The isolation detail that matters at scale: circuit breakers are per user, per provider, persisted in the database, so one user’s failing key or exhausted rate limit trips their own breaker without degrading anyone else’s session. Retrieval grounds questions in a 61-document curated knowledge base, and conversation branching is a first-class schema concept (its own migration), letting sessions explore follow-up paths and return — not a prompt trick.

Architecture

  ┌────────────────────────────── BROWSER ──────────────────────────────┐
  │  React 18 SPA · voice recorder · voice-activity detection (VAD)     │
  │  streaming audio playback · Zustand state · certificate viewer      │
  └────────────┬────────────────────────────────────────────▲──────────┘
               │ speech (mic)                     streamed audio
               ▼                                             │
  ┌─────────────────────── KOTLIN / SPRING WEBFLUX ──────────┼──────────┐
  │                                                          │          │
  │  ┌─────────────────┐   ┌──────────────────┐   ┌──────────┴───────┐  │
  │  │ TRANSCRIPTION   │   │ INTELLIGENCE     │   │ TTS CHAIN        │  │
  │  │ ElevenLabs      │──►│ 8-provider BYOK  │──►│ ① ElevenLabs WS  │  │
  │  │ Scribe          │   │ per-user circuit │   │ ② Kokoro (self)  │  │
  │  │ └─ whisper-1    │   │ breakers ·       │   │ ③ OpenAI TTS     │  │
  │  │    fallback     │   │ adaptive         │   │ health monitor · │  │
  │  └─────────────────┘   │ difficulty ·     │   │ usage tracking · │  │
  │                        │ branching ·      │   │ text preprocess  │  │
  │  ┌─────────────────┐   │ 61-doc KB        │   └──────────┬───────┘  │
  │  │ SESSION ENGINE  │◄──┤ emotion context  │              │          │
  │  │ practice · exam │   └──────────────────┘              │          │
  │  │ · interview ·   │                                     │          │
  │  │ certificates    │   PostgreSQL 16 · Flyway · OAuth2   │          │
  │  └─────────────────┘   Stripe · Sentry · Caffeine        │          │
  └─────────────────────────────────────────────────────────┼──────────┘
                                              Fly.io private │ WireGuard mesh
                                                             ▼
                                       ┌──────────────────────────────┐
                                       │ Kokoro TTS (self-hosted)     │
                                       │ shared-tts.internal:8880     │
                                       │ no public IP · zero cold     │
                                       │ start · model preloaded      │
                                       └──────────────────────────────┘

The narration API exposes three modes — generate, stream (octet-stream), and download — with an audio cache and cache-management endpoints behind them, plus a voice-catalog API the frontend reads at runtime.

The voice round-trip

 ① LISTEN                ② THINK                 ③ SPEAK                 ④ ADAPT
 VAD detects end   ──►   chosen LLM grades  ──►  preprocessor makes ──►  practice: hints
 of answer ·             the answer ·            text speakable ·        exam: 20 questions
 Scribe transcribes      picks next question     emotion tunes           → 1-year certificate
 (whisper-1 standby) ·   by difficulty ·         synthesis params ·      (jsPDF render)
 typing always works     emits emotion context   healthy tier streams    interview: no help,
                                                 audio back              full simulation

Each stage has a fallback: text input beside voice, Whisper behind Scribe, two tiers behind ElevenLabs, and a session engine that keeps state even when a provider trips its breaker mid-conversation. The conversation never dies mid-sentence — that single property is where most of the engineering went.

Status and limits

The full voice loop, three session modes, certification with one-year certificates, Stripe billing, and the BYOK layer are live in beta. Retrieval is currently keyword-relevance scoring, not vector search — embeddings over the knowledge base are the next planned step, and I’d rather state that than imply semantic search that isn’t there. There is no voice cloning, and I publish no usage numbers because the honest number is “early.” Onboarding narration ships as pre-generated audio (built by script at deploy time) — a small, practical use of the same TTS pipeline.

What this demonstrates

Stack Sensei is reliability engineering applied to a domain most teams treat as an API call: a failover chain where each tier earns its place with a different property; hybrid infrastructure when a vendor’s latency floor isn’t good enough; model output mapped into perceptual synthesis parameters; personas engineered to survive degradation; and multi-provider abstraction with per-user fault isolation — all in production Kotlin with the observability to prove which tier spoke every sentence. If your product needs a voice that works when vendors don’t, book a consultation.