← All posts

Self-Hosted Voice AI: The Complete Guide to Running Your Own TTS and STT Stack

Self-Hosted Voice AI: The Complete Guide to Running Your Own TTS and STT Stack

Voice AI has been stuck in a weird place for self-hosters. On one hand, we've had great speech-to-text with Whisper for years. On the other, text-to-speech has been a wasteland of robotic voices, cloud API lock-in, and models that need a GPU the size of a small car.

That changed in 2025-2026. A wave of open-source TTS models landed that are small enough to run on a Raspberry Pi, sound natural enough to fool your family, and are completely free. No API keys. No per-character billing. No "your free tier expired" emails at 2 AM.

This guide covers the four tools that make up a complete self-hosted voice AI stack: Kitten TTS (the tiny newcomer that hit #1 on Hacker News), Piper (the battle-tested workhorse), Kokoro-82M (the quality contender), and voicebox (the server that ties it all together with STT).

Why Self-Host Voice AI?

Before we dive into the tools, let's talk about why you'd want to run your own voice stack instead of using ElevenLabs, OpenAI TTS, or Google Cloud Text-to-Speech.

Cost. Cloud TTS is expensive at scale. ElevenLabs charges $5/month for 30,000 characters — that's roughly 20 minutes of audio. OpenAI's TTS-1 is $15 per million characters. If you're building a voice assistant, generating audiobooks, or running a podcast pipeline, those costs add up fast. Self-hosted is free after the hardware.

Privacy. Your text goes to their servers. For voice assistants, that means every conversation in your home potentially routes through a third party. For audiobook generation, it means uploading entire books to cloud APIs. Self-hosted keeps everything local.

Latency. Round-trip to a cloud API adds 200-500ms minimum. For real-time voice assistants, that's the difference between natural conversation and awkward pauses. Local inference on modern hardware can be sub-100ms.

Customization. Want to fine-tune a voice on your own voice samples? Clone a specific speaking style? Adjust prosody for different contexts? Local models give you control cloud APIs don't.

Offline. Your voice assistant works when the internet doesn't. This matters more than you'd think — ISP outages, travel, or just wanting things to work in the basement.

The Stack at a Glance

Tool Type Size Quality Speed Best For
Kitten TTS TTS 25MB ⭐⭐⭐⭐ ⚡⚡⚡⚡⚡ Edge devices, real-time
Piper TTS ~50MB/voice ⭐⭐⭐ ⚡⚡⚡⚡ Home Assistant, reliability
Kokoro-82M TTS 82M params ⭐⭐⭐⭐⭐ ⚡⚡⚡ Audiobooks, highest quality
voicebox TTS+STT Docker Varies Varies All-in-one server
faster-whisper STT ~1.5GB ⭐⭐⭐⭐⭐ ⚡⚡⚡ Speech recognition

Kitten TTS: The 25MB Powerhouse

Kitten TTS exploded onto the scene and hit 1,003 points on Hacker News with 361 comments — and for good reason. It's a state-of-the-art TTS model that's under 25MB, runs on CPU only, and produces genuinely expressive speech.

What Makes It Special

  • 15 million parameters — smaller than most images on a web page
  • 8 voices — four male, four female, all natural-sounding
  • CPU-only — no GPU required, runs on Raspberry Pi
  • ONNX runtime — portable across platforms
  • Int8 + FP16 quantization — tiny footprint without sacrificing quality
  • Runs anywhere — Raspberry Pi, low-end smartphones, wearables, browsers

The team behind it (KittenML) built it because existing expressive open-source TTS models required big GPUs, and cloud alternatives were too expensive for high-frequency use. Their goal: frontier-quality TTS that runs on edge devices.

Quick Start

# Clone the repo
git clone https://github.com/KittenML/KittenTTS.git
cd KittenTTS

# Install dependencies
pip install -r requirements.txt

# Download the model (25MB)
python download_model.py

# Generate speech
python synthesize.py --text "Hello, I'm running entirely on your CPU." --voice female_1 --output hello.wav

That's it. No Docker, no GPU drivers, no CUDA toolkit. On a Raspberry Pi 4, it generates speech faster than real-time. On a modern laptop, it's nearly instantaneous.

Voices and Quality

The eight included voices cover a good range. They're not ElevenLabs-level hyper-realistic, but they're far beyond the robotic "computer voice" you'd expect from a 25MB model. The expressiveness — pitch variation, natural pauses, appropriate emphasis — is what sets it apart from older lightweight TTS engines.

The current release is trained on less than 10% of their total data, so quality will only improve. The team has hinted at multilingual support and voice cloning in future releases.

Docker Deployment

# Run Kitten TTS as an API server
docker run -d \
  --name kitten-tts \
  -p 8000:8000 \
  ghcr.io/kittenml/kitten-tts:latest

# Generate speech via API
curl -X POST http://localhost:8000/tts \
  -H "Content-Type: application/json" \
  -d '{"text": "Your self-hosted voice AI is ready.", "voice": "female_1"}' \
  --output output.wav

Piper: The Battle-Tested Workhorse

If Kitten TTS is the exciting newcomer, Piper is the reliable veteran. With over 11,000 GitHub stars, it's the most widely deployed open-source TTS engine in the self-hosted world. It's the default TTS backend for Home Assistant's voice pipeline, and for good reason.

What Makes It Special

  • Fast — optimized C++ inference with ONNX runtime
  • Dozens of voices — multiple languages, male/female variants
  • Low resource usage — runs comfortably on a Raspberry Pi 3
  • Home Assistant native — first-class integration via Wyoming protocol
  • Mature — battle-tested across thousands of deployments
  • Streaming output — audio starts playing before generation completes

Quick Start

# Install via pip
pip install piper-tts

# Download a voice model
# Voices available at: https://huggingface.co/rhasspy/piper-voices
wget https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx
wget https://huggingface.co/rhasspy/piper-voices/resolve/main/en/en_US/lessac/medium/en_US-lessac-medium.onnx.json

# Generate speech
echo "Welcome to your self-hosted voice assistant." | \
  piper --model en_US-lessac-medium.onnx --output_file welcome.wav

Home Assistant Integration

This is where Piper really shines. Home Assistant's voice pipeline uses Piper as the default TTS engine:

# Add to Home Assistant via the Wyoming integration
# Settings → Devices & Services → Add Integration → Wyoming Protocol
# Host: localhost (or your Piper server IP)
# Port: 10200

Or run it as a Docker container:

docker run -d \
  --name piper \
  -p 10200:10200 \
  -v /path/to/voices:/voices \
  rhasspy/wyoming-piper \
  --voice en_US-lessac-medium

Once connected, any Home Assistant voice command that generates a spoken response will use Piper. "What's the weather?" → spoken forecast. "Turn off the lights" → spoken confirmation. All local, all free.

Voice Quality

Piper's voices are good but not great. They're clear and intelligible — perfect for assistant-style interactions — but they won't fool anyone into thinking a human is speaking. For audiobooks or content creation, you'll want Kokoro. For quick assistant responses, Piper is ideal.

Kokoro-82M: The Quality Contender

Kokoro-82M sits at the opposite end of the spectrum from Kitten TTS. Where Kitten prioritizes tiny size and universal compatibility, Kokoro prioritizes voice quality. With 82 million parameters, it produces some of the most natural-sounding speech available in open-source TTS.

What Makes It Special

  • 82M parameters — 5x larger than Kitten, but still manageable
  • Near-studio quality — the best open-source TTS voice quality available
  • Voice mixing — blend multiple voices for unique results
  • Auto-stitching — handles long texts by intelligently segmenting and joining
  • Multiple backends — PyTorch, ONNX, and Rust implementations available
  • Active ecosystem — FastAPI wrappers, audiobook generators, MCP servers

Quick Start (via Kokoro-FastAPI)

The easiest way to run Kokoro is through the excellent Kokoro-FastAPI Docker image (5,300+ GitHub stars):

docker run -d \
  --name kokoro \
  -p 8880:8880 \
  ghcr.io/remsky/kokoro-fastapi:latest

# Generate speech
curl -X POST http://localhost:8880/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kokoro",
    "input": "The quality difference is immediately noticeable. This sounds like a real person.",
    "voice": "af_heart"
  }' \
  --output kokoro_output.wav

The API is OpenAI-compatible, so any tool that works with OpenAI's TTS endpoint can be pointed at your local Kokoro instance with a URL change.

Audiobook Generation

Kokoro really shines for long-form content. Tools like abogen (5,600+ stars) and epub2tts-kokoro can turn entire books into audiobooks:

# Generate an audiobook from an EPUB
pip install epub2tts-kokoro
epub2tts my-book.epub --voice af_heart --output my-book-audiobook/

The auto-stitching handles chapter boundaries, paragraph breaks, and long passages intelligently. A full novel takes a few hours on a modern CPU — not real-time, but overnight is perfectly reasonable.

Performance Notes

Kokoro needs more resources than Kitten or Piper. On a Raspberry Pi 5, it runs at roughly 0.5x real-time (30 seconds to generate 60 seconds of audio). On an M-series Mac or modern x86 CPU, it runs at 2-3x real-time. A GPU isn't required but helps significantly for batch processing.

voicebox: The All-in-One Voice Server

Here's where things get interesting. voicebox (agjs/voicebox) is a relatively new project that combines everything into a single, self-hosted, OpenAI-compatible speech server. It's the missing piece that turns individual TTS/STT engines into a unified voice API.

What It Does

  • Speech-to-text via faster-whisper
  • Text-to-speech via Piper and/or Kokoro
  • OpenAI-compatible API — drop-in replacement for OpenAI's /v1/audio/* endpoints
  • Docker deployment — single container, single port
  • Open WebUI integration — voice chat in your local AI chat interface
  • CLI and custom agent support — any tool that speaks OpenAI's audio API

Quick Start

docker run -d \
  --name voicebox \
  -p 8000:8000 \
  -v voicebox-models:/models \
  -e TTS_ENGINE=piper \
  -e STT_ENGINE=faster-whisper \
  ghcr.io/agjs/voicebox:latest

Once running, point any OpenAI-compatible client at http://localhost:8000/v1:

# Transcribe audio
curl http://localhost:8000/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F file="@recording.wav" \
  -F model="whisper-1"

# Generate speech
curl http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tts-1",
    "input": "Your voice server is running.",
    "voice": "alloy"
  }'

Open WebUI Integration

This is the killer use case. Open WebUI supports voice input/output, and voicebox provides the backend:

  1. Start voicebox: docker run -d -p 8000:8000 ghcr.io/agjs/voicebox:latest
  2. In Open WebUI, go to Settings → Audio
  3. Set TTS Engine to OpenAI, endpoint to http://voicebox:8000/v1
  4. Set STT Engine to OpenAI, same endpoint

Now you have a fully local ChatGPT-like experience with voice input and spoken responses. Your browser captures audio, voicebox transcribes it, your local LLM generates a response, and voicebox speaks it back. All local, all private.

Building a Complete Voice Pipeline

Here's how these pieces fit together into a complete self-hosted voice AI stack:

┌─────────────────────────────────────────────────────┐
│                  Your Application                    │
│         (Open WebUI / Home Assistant / CLI)          │
└──────────┬──────────────────────────┬───────────────┘
           │                          │
     ┌─────▼──────┐           ┌──────▼──────┐
     │   voicebox  │           │  Direct API │
     │  (unified)  │           │   Access    │
     └──┬───────┬──┘           └──┬───────┬──┘
        │       │                 │       │
   ┌────▼──┐ ┌──▼────┐      ┌────▼──┐ ┌──▼────┐
   │  STT  │ │  TTS  │      │ Piper │ │Kokoro │
   │Whisper│ │Piper/ │      │       │ │       │
   │       │ │Kokoro │      │       │ │       │
   └───────┘ └───────┘      └───────┘ └───────┘

Option A: The Simple Stack (voicebox)

Best for most people. One Docker container, one API, both STT and TTS handled.

docker run -d --name voicebox -p 8000:8000 \
  -e TTS_ENGINE=piper \
  -e STT_ENGINE=faster-whisper \
  ghcr.io/agjs/voicebox:latest

Option B: The Quality Stack (Kokoro + faster-whisper)

Best for audiobook generation and high-quality voice output.

# STT
docker run -d --name whisper -p 9000:9000 \
  ghcr.io/agjs/voicebox:latest  # STT-only mode

# TTS
docker run -d --name kokoro -p 8880:8880 \
  ghcr.io/remsky/kokoro-fastapi:latest

Option C: The Edge Stack (Kitten TTS + Piper)

Best for Raspberry Pi or low-resource devices.

# Kitten for expressive short responses
docker run -d --name kitten -p 8001:8000 \
  ghcr.io/kittenml/kitten-tts:latest

# Piper for reliable assistant voice
docker run -d --name piper -p 10200:10200 \
  rhasspy/wyoming-piper --voice en_US-lessac-medium

Hardware Requirements

Device Kitten TTS Piper Kokoro faster-whisper
Raspberry Pi 4 (4GB) ✅ Faster than real-time ✅ Real-time ⚠️ 0.3x real-time ⚠️ Slow
Raspberry Pi 5 (8GB) ✅ Instant ✅ Real-time ⚠️ 0.5x real-time ⚠️ Usable
M1 Mac Mini (8GB) ✅ Instant ✅ Instant ✅ 2x real-time ✅ Fast
M4 Mac Mini (24GB) ✅ Instant ✅ Instant ✅ 3x real-time ✅ Fast
Intel N100 (16GB) ✅ Instant ✅ Instant ✅ 1.5x real-time ✅ Fast
Old laptop (8GB) ✅ Instant ✅ Real-time ⚠️ 0.8x real-time ⚠️ Usable

The takeaway: Kitten TTS and Piper run on basically anything. Kokoro wants a somewhat modern CPU. faster-whisper benefits from more RAM and cores but works on a Pi 5.

Real-World Use Cases

1. Local Voice Assistant (Home Assistant)

The most practical use case. Home Assistant's voice pipeline with Piper gives you a fully local Alexa/Google Home replacement:

# configuration.yaml
assist_pipeline:
  stt_engine: whisper
  tts_engine: piper

"Hey Home Assistant, what's the temperature in the living room?" → spoken answer, no cloud involved.

2. AI Chat with Voice (Open WebUI + voicebox)

Run a local LLM (Ollama) with voice input/output:

# Start the stack
docker run -d --name ollama -p 11434:11434 ollama/ollama
docker run -d --name voicebox -p 8000:8000 ghcr.io/agjs/voicebox:latest
docker run -d --name openwebui -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://ollama:11434 \
  -e AUDIO_TTS_ENGINE=openai \
  -e AUDIO_TTS_ENDPOINT=http://voicebox:8000/v1 \
  ghcr.io/open-webui/open-webui:main

Now you're talking to your local AI like it's ChatGPT Advanced Voice Mode — except it's running on your hardware and costs nothing per message.

3. Audiobook Generation

Turn your ebook library into audiobooks:

# Using Kokoro for quality
pip install epub2tts-kokoro
epub2tts ~/books/*.epub --voice af_heart --output ~/audiobooks/

# Batch process a whole directory
for book in ~/books/*.epub; do
  epub2tts "$book" --voice af_heart --output ~/audiobooks/$(basename "$book" .epub)/
done

A typical novel (80,000 words) takes about 8-10 hours on an M-series Mac. Start it before bed, wake up to a finished audiobook.

4. Podcast Production

Generate narration for podcasts without hiring voice talent:

# Generate episode narration
python synthesize.py \
  --text "$(cat episode-script.txt)" \
  --voice male_1 \
  --output episode-42-narration.wav

# Mix with intro/outro music using ffmpeg
ffmpeg -i intro.mp3 -i episode-42-narration.wav -i outro.mp3 \
  -filter_complex "[0][1][2]concat=n=3:v=0:a=1" episode-42-final.mp3

5. Accessibility

Add voice output to any application:

import requests

def speak(text):
    response = requests.post(
        "http://localhost:8000/v1/audio/speech",
        json={"model": "tts-1", "input": text, "voice": "alloy"}
    )
    with open("/tmp/speech.mp3", "wb") as f:
        f.write(response.content)
    # Play with system audio player
    import subprocess
    subprocess.run(["afplay", "/tmp/speech.mp3"])  # macOS
    # subprocess.run(["mpg123", "/tmp/speech.mp3"])  # Linux

Comparison: Self-Hosted vs Cloud TTS

Factor Self-Hosted Cloud (ElevenLabs/OpenAI)
Cost Free (hardware only) $5-500+/month
Privacy Everything stays local Text sent to third party
Latency 50-200ms 200-800ms
Quality Good to excellent Excellent
Voice variety 8-50+ voices 100+ voices
Voice cloning Limited (emerging) Yes (ElevenLabs)
Offline ✅ Yes ❌ No
Rate limits None Per-tier limits
Customization Full control Limited API params
Setup time 15-60 minutes 5 minutes

For most self-hosted use cases — voice assistants, audiobooks, accessibility — the quality gap has narrowed enough that self-hosted is the clear winner on cost and privacy. Cloud still wins for professional voiceover work and when you need dozens of distinct character voices.

Troubleshooting

"Model runs but audio sounds robotic"

  • Piper: Try a different voice model. The "medium" and "high" quality variants sound significantly better than "low."
  • Kokoro: Ensure you're using the full 82M model, not a quantized version. The Q8 version is a good balance.
  • Kitten TTS: The current release is an early checkpoint. Quality will improve with future releases.

"Generation is too slow"

  • Kokoro on Pi: It's going to be slow. Use Piper or Kitten TTS instead.
  • faster-whisper: Use the tiny or base model instead of large-v3. The quality difference is smaller than you'd expect.
  • General: Check CPU usage with htop. Other containers might be competing for resources.

"voicebox can't find my TTS engine"

  • Ensure the TTS engine container is on the same Docker network
  • Check the TTS_ENGINE environment variable matches your setup
  • Verify the engine's API is accessible from the voicebox container: docker exec voicebox curl http://kokoro:8880/health

"Open WebUI voice doesn't work"

  • Check that the audio endpoint is reachable from your browser (not just from within Docker)
  • For localhost setups, use http://localhost:8000/v1 not the Docker container name
  • Some browsers require HTTPS for microphone access — use a reverse proxy with SSL

What's Next for Self-Hosted Voice AI

The pace of improvement in open-source TTS is staggering. A few things to watch:

  • Kitten TTS v2 — The team has hinted at multilingual support, voice cloning, and models trained on their full dataset. If the 10%-data checkpoint is this good, the full model could be transformative.
  • Streaming TTS — Real-time streaming (audio starts playing before the full text is generated) is becoming standard. Kokoro-FastAPI already supports it; expect others to follow.
  • Voice cloning — Open-source voice cloning is catching up to ElevenLabs. Projects like F5-TTS and CosyVoice are already producing impressive results with short reference samples.
  • On-device STT — Whisper alternatives like Moonshine and SenseVoice are pushing STT latency down to near-zero for real-time applications.
  • Multimodal voice assistants — The combination of local LLMs + local TTS + local STT means fully offline voice assistants that understand context, remember conversations, and control your smart home — all without a cloud dependency.

Conclusion

Self-hosted voice AI has crossed the threshold from "technically possible" to "genuinely good." You can now run a complete TTS+STT stack on a Raspberry Pi that sounds natural, responds quickly, and costs nothing beyond the hardware.

Start with voicebox if you want the simplest path — one Docker container gives you both speech-to-text and text-to-speech with an OpenAI-compatible API. Add Kokoro if you want the best voice quality for audiobooks or content creation. Use Kitten TTS if you're running on edge hardware and need something tiny. Piper is your go-to for Home Assistant integration.

The tools are free, the setup is straightforward, and the privacy benefits are real. Your voice data stays yours. Your assistant works offline. And you never get a "usage limit exceeded" email at 2 AM.

That's the self-hosted way.


Have you set up a self-hosted voice AI stack? I'd love to hear about your setup — especially creative use cases I haven't covered. Find me on GitHub or drop a comment below.