Why Clinics Miss 35% of Calls (And Why Chatbots Fail)
In high-volume medical clinics, patient calls peak in intense bursts between 8:00 AM and 11:00 AM. Receptionists are overwhelmed checking in physical patients, verifying insurance, and handling prescription pick-ups. When the telephone rings, it either goes to voicemail or rings out. In healthcare, missed calls represent delayed patient care and lost revenue.
Traditional telephony IVRs ("Press 1 for Cardiology, Press 2 for Appointments") suffer from high abandonment rates exceeding 45%. Patients hang up. Meanwhile, text-based chatbots on WhatsApp or websites do not solve the urgent communication needs of elderly patients or people calling in transit who need an immediate human-like response.
When we architected Ayu Health (AyuApp Pvt Ltd), our mission was clear: create an autonomous voice AI receptionist named Ayu capable of answering clinic phone lines in real-time, holding fluent conversations in five Indian languages (Tamil, Hindi, Telugu, Kannada, and Malayalam), checking real-time doctor calendar availability, and booking appointments directly into our Next.js clinical dashboard.
Status: the agent was built and tested on live phone lines at Ayu Health. It is not in commercial use at the time of writing, because the per-minute cost of a call was too high for clinic budgets. Here is exactly how we built it, optimized latency down to sub-700ms, and eliminated medical hallucinations.
High-Level Architecture Overview
A real-time voice system cannot rely on traditional sequential Request/Response HTTP cycles. An HTTP-based speech-to-text -> LLM query -> text-to-speech chain easily adds 2.5 to 4 seconds of latency, creating an intolerable, robotic pause that causes callers to say "Hello? Are you there?"
To achieve human conversational cadence, we engineered a streaming, event-driven pipeline:
[ Caller Phone Call ]
│
▼ (PSTN Telephony)
[ Plivo Voice API ]
│
▼ (Bi-directional Audio WebSocket - 8kHz μ-law)
[ Fastify Real-Time Gateway (Cloud Run) ]
├── [ Session Manager & Audio Framer ]
├── [ Mem0 Caller Memory Store ]
└── [ Langfuse Observability & Tracing ]
│
▼ (Full-Duplex Streaming WebSocket)
[ Google Gemini Live API ]
│ (Native Speech-to-Speech & Function Calling)
├── [ check_doctor_availability() ]
├── [ book_appointment_slot() ]
└── [ transfer_to_human_desk() ]
│
▼ (WebSocket Event Broadcast)
[ Real-Time Doctor Dashboard (Next.js 15) ]1. Zero-Transcoding Audio Pipeline: Plivo to Gemini Live
The single biggest source of latency in conversational AI is format transcoding (e.g. converting 8kHz telephony audio to 16kHz PCM, sending to Whisper STT, passing text to an LLM, and synthesizing back to audio).
With Gemini Live, Google introduced native multi-modal speech-to-speech streaming. Gemini natively accepts and outputs raw audio streams over WebSockets.
Configuring Plivo XML (PHLO) for Real-Time Media Streams
When an incoming PSTN call hits Plivo, our webhook immediately replies with a element pointing to our authenticated WebSocket endpoint:
<Response>
<Speak>Welcome to Ayu Health Clinic. Connecting you to our receptionist.</Speak>
<Stream bidirectional="true" streamTimeout="86400" keepCallAlive="true">
wss://voice.ayuapp.com/v1/telephony/stream?clinicId=CLINIC_9041&token=AUTH_JWT
</Stream>
</Response>Audio Packet Framing in Fastify
Plivo streams audio packets in base64-encoded μ-law (PCMU) at an 8,000 Hz sample rate in 20ms chunks (160 bytes per packet). In our Fastify backend, we decode and frame the audio chunks into the exact JSON format required by Gemini Live:
import { FastifyPluginAsync } from "fastify";
import WebSocket from "ws";
export const telephonyStreamPlugin: FastifyPluginAsync = async (app) => {
app.get("/v1/telephony/stream", { websocket: true }, (connection, req) => {
const { clinicId } = req.query as { clinicId: string };
// 1. Establish upstream WebSocket connection to Gemini Live API
const geminiWs = new WebSocket(
`wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent?key=${process.env.GEMINI_API_KEY}`
);
geminiWs.on("open", () => {
// Send initial setup frame with system instructions, language constraints & function definitions
geminiWs.send(JSON.stringify({
setup: {
model: "models/gemini-2.0-flash-exp",
generationConfig: {
responseModalities: ["AUDIO"],
speechConfig: {
voiceConfig: {
prebuiltVoiceConfig: { voiceName: "Puck" }
}
}
},
systemInstruction: {
parts: [{ text: CLINIC_PROMPTS[clinicId] }]
},
tools: [{ functionDeclarations: clinicalToolDefinitions }]
}
}));
});
// 2. Stream audio from Caller -> Gemini Live
connection.socket.on("message", (rawMessage) => {
const data = JSON.parse(rawMessage.toString());
if (data.event === "media") {
geminiWs.send(JSON.stringify({
realtimeInput: {
mediaChunks: [{
mimeType: "audio/x-mulaw;rate=8000",
data: data.media.payload
}]
}
}));
}
});
// 3. Stream generated audio from Gemini Live -> Caller
geminiWs.on("message", (geminiResponse) => {
const response = JSON.parse(geminiResponse.toString());
const audioPart = response.serverContent?.modelTurn?.parts?.find(p => p.inlineData);
if (audioPart) {
connection.socket.send(JSON.stringify({
event: "playAudio",
media: {
payload: audioPart.inlineData.data
}
}));
}
});
});
};2. Preventing Hallucinations with Deterministic Function Calling
In healthcare, an AI agent must never guess doctor hours or promise unavailable appointments. A clinical receptionist must be deterministic:
- State Isolation: The agent's knowledge of clinic availability is strictly fetched on-demand via tools.
- Double Confirmation: The caller must explicitly hear the day, date, doctor name, and time slot before the booking is committed.
- Structured Mutation: Appointment records are written directly to MongoDB and broadcasted via WebSockets to the doctor's live Next.js screen.
Here is the tool definition schema provided to the model:
export const clinicalToolDefinitions = [
{
name: "check_doctor_availability",
description: "Check open consultation slots for a specific doctor or medical department.",
parameters: {
type: "OBJECT",
properties: {
department: { type: "STRING", enum: ["General Medicine", "Pediatrics", "Cardiology", "Orthopedics"] },
date: { type: "STRING", description: "YYYY-MM-DD format" }
},
required: ["department", "date"]
}
},
{
name: "book_appointment_slot",
description: "Lock in an appointment after the patient has verbally confirmed the slot.",
parameters: {
type: "OBJECT",
properties: {
slotId: { type: "STRING" },
patientName: { type: "STRING" },
patientPhone: { type: "STRING" },
symptomsBrief: { type: "STRING" }
},
required: ["slotId", "patientName", "patientPhone"]
}
}
];When Gemini triggers checkdoctoravailability, our backend executes a high-speed Redis query against cached slot tables and returns the available slots in under 45ms. The model continues speaking naturally: "Dr. Arunkumar has two slots open this Thursday: 10:30 AM or 4:00 PM. Which one suits you best?"
3. Real-Time Telemetry to the Doctor Dashboard (Next.js 15)
While the AI voice agent is on the phone with the patient, the clinic staff sees the active call state updated live on their dashboard without refreshing the page.
Using Server-Sent Events (SSE) and WebSockets, we dispatch real-time status updates:
- Caller Ringing: Phone number and previous medical history preview via Mem0.
- Live Audio Transcript: Dual-column transcription showing patient speech and AI responses.
- Instant Booking Notification: Doctor's daily calendar UI immediately highlights the newly reserved block in lime green with patient intake notes attached.
Measured Performance in Testing
| Metric | Target | Test result |
|---|---|---|
| End-to-End Voice Latency | < 1,000 ms | 680 ms |
| Call Completion Rate | > 80% | 91.4% |
| Multilingual Recognition | 5 Languages | Tamil, Hindi, Telugu, Kannada, Malayalam |
| Concurrent Active Calls | 50+ per clinic | Benchmarked to 250+ simultaneous streams |
| App Store AI Privacy Review | 100% Pass | Approved on first submission with clear consent disclosures |
Are You Hiring a Freelance App Developer for Voice AI or Healthcare?
Building real-time conversational systems requires full-stack mastery: low-level audio streaming protocols, telephony WebSockets, stateful LLM orchestration, and responsive web/mobile client dashboards.
As a senior freelance developer and tech lead, I help funded startups and healthcare providers build and ship production voice AI applications from scratch.
- Need a custom voice receptionist or AI call workflow?
- Need a complete mobile app (React Native / Flutter) and web dashboard (Next.js)?
- Need Apple App Store & Google Play publishing handled end to end?
Book a 30-minute discovery call to discuss your architecture, or email me directly at senthiltechspot@gmail.com.
Building something like this?
I build React Native apps, their backends and AI voice agents from your Figma designs, and handle the App Store and Google Play release.