AI Development · 2025-03-26 · Michael Ditter

Building an Interactive Voice Assistant with ElevenLabs on Replit

Learn how to create a real-time voice assistant using ElevenLabs for speech-to-text and text-to-speech, FastAPI for WebSockets, and AI models for natural conversation - all deployed on Replit.

Introduction

Creating a real-time voice assistant involves streaming audio from the user to the server (speech-to-text), generating a response (via an AI model), and streaming audio back to the user (text-to-speech). This tutorial walks you through building a complete voice assistant using Python (FastAPI) for the server, WebSockets for bidirectional audio, ElevenLabs for speech recognition and speech synthesis, and OpenAI's GPT for conversation.

Tech Stack and Setup

FastAPI + Uvicorn: Serves a WebSocket endpoint for low-latency, persistent communication and an HTML client interface.
ElevenLabs API: Provides Text-to-Speech (TTS) with lifelike voices and a Speech-to-Text (STT) model (Scribe v1).
OpenAI GPT-3.5 (or any LLM): Handles natural language responses to user queries.
HTML/JavaScript Client: Uses the Web Audio API to capture microphone audio and send it via WebSocket, and plays the assistant's voice responses in real-time.

Server-Side Code (FastAPI WebSocket Backend)

Our server implements a FastAPI app with:

An HTTP GET route to serve the client HTML
A WebSocket route (/ws) to handle the conversation
Logic for turn-taking and interruption handling

Here's the core WebSocket handler from our implementation:

@app.websocket("/ws")
async def websocket_endpoint(ws: WebSocket):
    await ws.accept()
    print("Client connected")
    history = [{"role": "system", "content": "You are a helpful voice assistant."}]
    try:
        while True:
            # Wait for control messages from client ("start" or other)
            data = await ws.receive_text()
            if data == "start":
                # User started speaking: collect audio until "end"
                audio_bytes = bytearray()
                # Read incoming binary frames until we get an "end" text frame
                while True:
                    msg = await ws.receive()
                    if msg["type"] == "websocket.disconnect":
                        raise WebSocketDisconnect
                    if "bytes" in msg and msg["bytes"] is not None:
                        audio_bytes.extend(msg["bytes"])
                        continue
                    if "text" in msg and msg["text"] == "end":
                        break
                        
                # 1. Speech-to-Text (transcribe user audio)
                user_text = await transcribe_audio(audio_bytes)
                await ws.send_text(json.dumps({"type": "transcript", "text": user_text}))
                
                # 2. Generate response with LLM
                assistant_text = await generate_response(user_text, history)
                await ws.send_text(json.dumps({"type": "assistant", "text": assistant_text}))
                
                # 3. Text-to-Speech (synthesize assistant's reply)
                audio_data = await synthesize_speech(assistant_text)
                if audio_data:
                    await ws.send_bytes(audio_data)
    except WebSocketDisconnect:
        print("Client disconnected")
    finally:
        await ws.close()

Client-Side Implementation

The client-side JavaScript handles:

MediaRecorder: We get microphone access via getUserMedia() and feed the stream into a MediaRecorder.
Control messages: We send "start" to signal the server when the user begins speaking, and "end" when recording stops.
Playback: Incoming WebSocket messages are handled based on type (text vs binary audio).

// Client-side JavaScript (excerpt)
const socket = new WebSocket(wsProtocol + window.location.host + "/ws");
socket.binaryType = "arraybuffer";

navigator.mediaDevices.getUserMedia({ audio: true })
  .then(stream => {
    mediaRecorder = new MediaRecorder(stream, { mimeType: "audio/webm" });
    
    // Send audio chunks to the server as they're available
    mediaRecorder.ondataavailable = e => {
      if (e.data.size > 0 && socket.readyState === WebSocket.OPEN) {
        socket.send(e.data);  // send audio chunk as binary
      }
    };
  });

// When the user starts speaking
function startSpeaking() {
  socket.send("start");      // signal start of user speech
  mediaRecorder.start(250);  // start recording in small chunks (250ms)
}

// When the user stops speaking
function stopSpeaking() {
  mediaRecorder.stop();
  socket.send("end");  // signal end of user speech
}

// Handle messages from the server
socket.onmessage = event => {
  if (typeof event.data === "string") {
    // Text message (JSON) - transcript or assistant text
    let msg = JSON.parse(event.data);
    // Display the appropriate message...
  } else {
    // Binary message - audio data of assistant's reply
    let blob = new Blob([event.data], { type: "audio/mpeg" });
    audioPlayer.src = URL.createObjectURL(blob);
    audioPlayer.play();
  }
};

Handling Interruptions

A critical aspect of voice assistants is handling interruptions naturally. In our implementation:

If the assistant is mid-sentence and the user clicks to speak, the client stops current audio playback.
The client then sends "start" to the server to begin transmitting new user audio.
This creates a fluid, human-like conversation experience where users can interrupt the assistant.

Performance Optimization Tips

To create a responsive voice assistant:

Use streaming APIs for STT: Consider streaming speech-to-text APIs like Deepgram or AssemblyAI for faster partial transcripts.
Optimize TTS generation: Use ElevenLabs' optimize_streaming_latency parameter to speed up generation.
Choose efficient audio formats: WebM/Opus for capturing and MP3 for playback offers a good balance of quality and size.
Implement voice activity detection: Auto-stop recording when the user is done speaking, instead of requiring a manual button press.

Deploying on Replit

Deploying this project on Replit is straightforward:

Create a new Python repl
Install dependencies: fastapi, uvicorn, requests, openai
Add your API keys as Replit Secrets (ELEVENLABS_API_KEY, OPENAI_API_KEY)
Create the main.py file with the server code
Run the server with Uvicorn

Conclusion

Building a voice assistant with ElevenLabs on Replit demonstrates how accessible advanced AI voice technology has become. This project combines several technologies (WebSockets, speech recognition, LLMs, and text-to-speech) into a cohesive, interactive experience that feels natural to users.

The techniques shown here can be extended to create voice-enabled chatbots, accessibility tools, interactive tutorials, or even voice-controlled applications. As speech technology continues to improve, the possibilities for creating engaging voice interfaces will only expand.

← All essays · michaelditter.com