You press an ESP32 voice satellite in your kitchen and say, “It’s way too bright in here, and set the office thermostat to 68.” If your setup routes through cloud endpoints like OpenAI or Anthropic, you are trading network latency, recurring API bills, and smart home privacy for basic convenience. If you have tried replacing that cloud link with a haphazardly configured local model running on a spare mini PC, you likely ran into a frustrating wall: five-second pauses before the lights react, random hallucinations where the model apologizes instead of triggering an action, or severe out-of-memory crashes when multiple speech requests hit your server simultaneously.
Building a local Large Language Model (LLM) voice agent for Home Assistant Assist is no longer an experimental weekend gimmick. Modern open-weights models possess the architectural intelligence needed to parse messy human speech into precise smart home actions. However, your choice of local inference backend determines whether your voice pipeline feels like an instantaneous native appliance or a sluggish, unreliable hobby project. Choosing between Ollama and vLLM isn’t just about container preference—it represents a fundamental architectural decision between resource-flexible simplicity and enterprise-grade, low-latency execution.
Deploy Ollama for single-user homelabs running mixed CPU/GPU hardware where deployment speed matters most. Choose vLLM if you have dedicated NVIDIA VRAM (12GB+) and demand sub-200ms Time-to-First-Token (TTFT) latency, continuous request batching, and zero response stuttering across multi-room Home Assistant voice satellites.
How Does Function Calling Translate Natural Language into Hardware Actions?
To understand why backend performance matters, you first need to understand how modern Home Assistant voice processing works under the hood. Standard text-generation models merely output human-readable conversational prose. For an LLM to control your physical house, it must support Function Calling (also referred to as Tool Use). Function calling bridges raw text interpretation and structured Home Assistant API execution.
When you speak to Home Assistant Assist, your audio is converted to text via a Speech-to-Text (STT) engine like Whisper. The resulting string is wrapped in a structured system prompt generated by Home Assistant integrations (such as the native Assist pipeline or the community-standard Extended OpenAI Conversation component). This prompt exposes your exposed smart home entities along with their current states, available domains, and allowed service calls expressed as JSON schemas.
“name”: “HassTurnOff”,
“description”: “Turns off lights, switches, or fans”,
“parameters”: {
“type”: “object”,
“properties”: {
“domain”: {“type”: “array”, “items”: {“type”: “string”}},
“name”: {“type”: “array”, “items”: {“type”: “string”}}
},
“required”: [“domain”, “name”]
}
}
When the local inference engine processes your query—for example, “Turn off the porch lights and dim the living room to thirty percent”—the LLM does not respond with plain text like “Sure, turning those off for you.” Instead, the model evaluates its exposed tool definitions and generates a strictly formatted JSON payload targeting Home Assistant APIs:
{“name”: “HassTurnOff”, “parameters”: {“domain”: [“light”], “name”: [“Porch Light”]}},
{“name”: “HassLightSet”, “parameters”: {“name”: [“Living Room Light”], “brightness”: 30}}
]
If your local model engine fails to parse tool definitions or experiences severe prompt decoding bottlenecks, speech execution fails. The engine must accept large context windows containing dozens of device schemas, perform precise token pre-fill, and stream formatted output tokens back to Home Assistant without delay.
Ollama vs vLLM: Which Engine Fits Your Homelab Architecture?
Choosing between Ollama and vLLM comes down to balancing system simplicity against raw inference architecture. Both backends provide OpenAI-compatible endpoints that connect directly to Home Assistant, but their underlying memory management and execution pipelines differ dramatically.
Ollama: The Accessible Consumer Workhorse
Ollama is built on top of llama.cpp. Its primary objective is user accessibility and resource flexibility. It abstracts complex hardware management behind a clean CLI and REST API, allowing homelab enthusiasts to pull model weights using simple commands like ollama run qwen2.5:7b.
Architecturally, Ollama relies on GGUF quantization format (e.g., Q4_K_M, Q8_0). Its defining strength is unified memory offloading: if your target model requires 10GB of RAM but your GPU only has 8GB of VRAM, Ollama automatically offloads the remaining weight layers to host system RAM. While offloading layers to system memory introduces performance bottlenecks due to PCIe bandwidth limits, it guarantees that your model runs without abruptly crashing out-of-memory (OOM).
However, Ollama processes requests sequentially by default. When multiple voice satellites send audio queries simultaneously, Ollama queues those requests, causing noticeable response lag across your household setup.
vLLM: High-Performance Enterprise Engine
vLLM was developed at UC Berkeley specifically to eliminate memory waste and low execution speeds in high-concurrency LLM serving environments. Instead of relying on traditional continuous VRAM allocation, vLLM introduces PagedAttention.
In standard transformer architectures, Key-Value (KV) cache memory consumes vast pools of contiguous VRAM, leading to severe fragmentation. PagedAttention breaks down KV cache into fixed-size virtual memory pages, managing them similarly to how an operating system manages virtual memory. This allows vLLM to utilize near-100% of available GPU VRAM without static allocation waste.
Furthermore, vLLM implements Continuous Batching (iteration-level scheduling). Rather than processing a single prompt to completion while blocking incoming requests, vLLM batches incoming execution steps dynamically. If an ESP32 satellite sends a voice command while your server is generating a response for another query, vLLM processes both requests in parallel inside the exact same execution step.
| Architectural Feature | Ollama (llama.cpp Engine) | vLLM Engine | Home Assistant Practical Impact |
|---|---|---|---|
| Memory Architecture | Dynamic layer offloading between system RAM and VRAM. | PagedAttention with dedicated, un-paged GPU memory mapping. | vLLM avoids VRAM fragmentation; Ollama enables running large models on limited hardware. |
| Request Handling | Single-stream default; limited sequential slot management. | Continuous batching (iteration-level scheduling). | vLLM maintains sub-200ms latency even when multiple voice satellites trigger simultaneously. |
| Quantization Support | GGUF (Q4_K_M, Q5_K_M, Q8_0). | AWQ, GPTQ, FP16, BF16, SGLang formats. | Ollama GGUFs offer wider low-bit options; vLLM AWQ/GPTQ maximizes GPU tensor core speed. |
| Setup Complexity | Zero-configuration single binary or Docker container. | Requires explicit Docker run flags, CUDA environment specs, & VRAM ratio tuning. | Ollama deploys in 2 minutes; vLLM requires explicit GPU and CUDA setup. |
| Hardware Compatibility | NVIDIA, AMD (ROCm), Apple Silicon, x86 CPU fallback. | Strictly optimized for NVIDIA CUDA GPUs (AMD ROCm support experimental). | Use Ollama for Mac Minis, NUCs, or AMD GPUs. Use vLLM for dedicated NVIDIA homelab servers. |
Benchmarking Latency and Hardware Requirements for Smart Home Automation
When executing smart home automations via voice, low latency is critical. A delay exceeding 1.5 seconds feels jarring compared to traditional physical light switches. The most important metric in voice-driven local inference is Time-to-First-Token (TTFT)—the delay between Home Assistant terminating speech intake and the LLM emitting its first response token (or tool call structure).
Because Home Assistant system prompts carry heavy context windows—including detailed JSON definitions of every exposed light, switch, sensor, and automation script—the pre-fill phase places high demands on processing power. Below is a real-world benchmark comparing Ollama and vLLM running standard smart home open-weights models across typical homelab hardware setups.
| Model & Parameter Size | Engine & Format | Hardware Target | VRAM / RAM Footprint | Avg TTFT (Context Pre-fill) | Generation Speed |
|---|---|---|---|---|---|
| Qwen 2.5 7B Instruct | Ollama (GGUF Q4_K_M) | RTX 3060 12GB | 5.8 GB VRAM | 480 ms | 42 tok/s |
| Qwen 2.5 7B Instruct | vLLM (AWQ 4-bit) | RTX 3060 12GB | 9.2 GB VRAM (Allocated) | 140 ms | 88 tok/s |
| Qwen 2.5 14B Instruct | Ollama (GGUF Q4_K_M) | RTX 4090 24GB | 10.4 GB VRAM | 310 ms | 65 tok/s |
| Qwen 2.5 14B Instruct | vLLM (AWQ 4-bit) | RTX 4090 24GB | 18.5 GB VRAM (Allocated) | 95 ms | 132 tok/s |
| Llama 3.3 70B Instruct | Ollama (Q4_K_M Offload) | 64GB RAM + 12GB GPU | 11GB VRAM + 34GB System RAM | 4,200 ms | 6.5 tok/s |
Notice the stark difference in context pre-fill performance. Because vLLM optimizes its execution graph using continuous batching and PagedAttention, high context prompts (such as a 3,000-token system prompt containing your Home Assistant entity registry) process significantly faster. vLLM pre-fills systems prompts up to 3x–4x faster than Ollama, making it the superior backend for real-time local voice setups.
The 4-Step Production Setup Protocol for Home Assistant Local Voice
Follow this battle-tested setup workflow to establish a reliable, high-speed local LLM execution pipeline for your smart home setup.
Step 1: Deploy Your Target Engine Container
For dedicated hardware featuring modern NVIDIA GPUs (e.g., RTX 3060, RTX 3090, or RTX 4090), deploy vLLM via Docker Compose to enable high-throughput continuous batching. Ensure the official NVIDIA Container Toolkit is installed on your host system beforehand.
services:
vllm-ha:
image: vllm/vllm-openai:latest
container_name: vllm_homeassistant
runtime: nvidia
environment:
– HUGGING_FACE_HUB_TOKEN=your_optional_hf_token
ports:
– “8000:8000”
volumes:
– ~/.cache/huggingface:/root/.cache/huggingface
deploy:
resources:
reservations:
devices:
– driver: nvidia
count: all
capabilities: [gpu]
command: >
–model Qwen/Qwen2.5-7B-Instruct-AWQ
–quantization awq
–max-model-len 8192
–gpu-memory-utilization 0.90
–enable-auto-tool-choice
–tool-call-parser pythonic
If you prefer using CPU rendering, Apple Silicon, or mixed memory offloading, run Ollama alongside a custom network driver instead:
docker exec -it ollama ollama run qwen2.5:7b
Step 2: Configure the Conversation Integration in Home Assistant
While Home Assistant includes a basic built-in Ollama integration, advanced tool calling, dynamic entity exposure, and multi-turn automation scripts perform significantly better using the popular HACS component: Extended OpenAI Conversation.
- Open HACS in your Home Assistant UI, search for
Extended OpenAI Conversation, and select Install.Conversational Large Language Models & Local Intent ParsersPairing ultra-fast local speech transcription with dedicated on-premise language models enables rich conversational smart home interactions. For Ollama vs vLLM setups, read Whisper STT for Home Assistant: Latency Benchmarks.