Voice AI Assistant for Medicare Patients

The Challenge

A healthcare startup needed a voice-first companion that elderly patients could actually use on a bedside tablet, with no menus, typing, or jargon. It had to feel warm and patient, respond fast enough to sustain natural conversation, help with daily routines like medications and reminders, and be operable and observable across a fleet of devices in the field.

The Solution

We built the assistant on the LiveKit Agents framework as a real-time voice pipeline: AssemblyAI streaming speech-to-text, an OpenAI GPT-4.1 reasoning model, and Cartesia speech, with Silero voice-activity detection and a transformer turn detector tuned for slower, gentler speech. A backend-authoritative command protocol keeps voice in sync with a large-type touch interface, so a patient can either speak or tap and both paths behave identically. The assistant handles medication check-ins, spoken reminders, weather, news, trivia and word games, and location-aware web search, and remembers durable facts (family, health, preferences) through a long-term memory layer. We engineered latency deliberately (worker prewarming, parallelized memory loading, and off-critical-path writes) and instrumented everything with Langfuse tracing, Sentry monitoring, and an LLM-judge evaluation suite that grades empathy and clarity, not just tool calls. A device-auth layer, per-device network telemetry, and remote administration make it operable at fleet scale.

Results

• Runs in production on kiosk tablets with end-to-end observability on every call
• True voice-and-touch multimodal interface designed for elderly accessibility
• Six voice skills (medication, reminders, weather, news, trivia, word games) plus web search
• Measured responsiveness gains from targeted latency work (e.g. ~58% shorter game-turn gap)
• 96 backend unit tests plus a 32-scenario conversational eval suite grading empathy and clarity

Technologies Used