Local AI Voice Assistant with Home Assistant and Ollama

Amazon Alexa and Google Assistant send every voice command to corporate servers. A local voice assistant built on Home Assistant, Whisper (speech-to-text), Ollama (LLM), and Piper (text-to-speech) keeps everything on your hardware — faster for local device control, completely private, and no subscription required.

Architecture

Microphone → Wyoming Whisper (STT) → Home Assistant Assist
                                              ↓
                                    Local LLM (Ollama)
                                              ↓
                                    Piper TTS → Speaker

All components run locally. Home Assistant’s Assist pipeline orchestrates them.

Prerequisites

• Home Assistant (HAOS, Supervised, or Docker) — see homeassistant.io
• A machine with 8GB+ RAM for the AI components (can be a Raspberry Pi 5, NUC, or your main PC)
• A microphone (USB or network satellite)
• NVIDIA/AMD GPU optional but speeds up STT and LLM significantly

Step 1: Install Wyoming Whisper (Speech-to-Text)

Wyoming is Home Assistant’s voice protocol. Whisper runs as a Wyoming-compatible service.

Via HA Add-on (easiest if running HAOS):

1. Settings → Add-ons → Add-on Store → search Wyoming Whisper
2. Install → Configure: choose model size:
   • tiny.en: fastest, English only, ~75MB
   • base.en: good balance, English only
   • small: multilingual, ~500MB
   • medium: higher accuracy, ~1.5GB
3. Start the add-on

Via Docker (for non-HAOS):

docker run -d \
  --name wyoming-whisper \
  --restart unless-stopped \
  -p 10300:10300 \
  -v ./whisper-data:/data \
  rhasspy/wyoming-whisper \
  --model base.en \
  --language en

Step 2: Install Piper (Text-to-Speech)

Via HA Add-on:

1. Add-on Store → Wyoming Piper
2. Configure: choose a voice (e.g., en_US-lessac-medium)
3. Start

Via Docker:

docker run -d \
  --name wyoming-piper \
  -p 10200:10200 \
  -v ./piper-data:/data \
  rhasspy/wyoming-piper \
  --voice en_US-lessac-medium

Piper voices are fast and surprisingly natural. For other languages, download voices from huggingface.co/rhasspy/piper-voices.

Step 3: Connect Wyoming Services to Home Assistant

1. Settings → Devices & Services → Add Integration → Wyoming Protocol
2. Add Whisper: host = localhost (or your server IP), port 10300
3. Add Piper: host = localhost, port 10200

Both should appear as configured integrations.

Step 4: Install Ollama for the LLM Brain

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a capable model
ollama pull llama3.1:8b

# For weaker hardware
ollama pull llama3.2:3b

Install the Ollama integration in Home Assistant:

1. Settings → Add-ons → Ollama (if running HAOS) — or point the integration to your Ollama server
2. Settings → Devices & Services → Add Integration → Ollama
3. Enter URL: http://localhost:11434 (or your server IP)
4. Select model: llama3.1:8b

Step 5: Configure the Assist Pipeline

1. Settings → Voice Assistants → Add Assistant
2. Name: “Local AI”
3. Conversation agent: Ollama (llama3.1:8b)
4. Speech-to-text: faster-whisper
5. Text-to-speech: Piper
6. Save

System Prompt for Home Assistant LLM

Home Assistant sends a system prompt with your device list to the LLM. Customize it:

In the Ollama integration settings → Instructions:

You are a smart home assistant. You control home devices using Home Assistant.
Available actions: turn on/off lights, set brightness, check sensor values, run automations.
Be concise. Respond in 1-2 sentences max. When controlling devices, use Home Assistant services.

Setting Up a Voice Satellite

For a dedicated microphone device in a room:

Raspberry Pi Zero 2 W or Pi 4 + USB microphone + speaker:

# Install Wyoming Satellite on the Pi
pip install wyoming-satellite

wyoming-satellite \
  --name "Living Room" \
  --uri tcp://0.0.0.0:10700 \
  --mic-command "arecord -r 16000 -c 1 -f S16_LE -t raw" \
  --snd-command "aplay -r 22050 -c 1 -f S16_LE -t raw"

In Home Assistant: Add Wyoming integration pointing to the Pi’s IP:10700.

This creates a dedicated always-on microphone satellite in any room.

Wake Word Detection

For hands-free “Hey Jarvis” activation:

1. Add-on Store → Wyoming openWakeWord
2. Configure: choose a wake word (hey_mycroft, hey_jarvis, ok_nabu)
3. Add it to your Wyoming Satellite configuration

Testing

1. In Home Assistant → Settings → Voice Assistants → click your assistant → Try
2. Type “Turn on the living room lights” — test before voice
3. If working, test with actual microphone

Voice commands that work natively: “Turn on/off [device]”, “Set [light] to 50%”, “What’s the temperature in [room]?”, “Run [automation]”

The full stack (Whisper + Piper + Llama 3.1 8B + Home Assistant) runs comfortably on an Intel N100 mini-PC or Raspberry Pi 5 with 8GB RAM. Add a GPU for significantly faster response times.