AI Assistant is a self-hosted voice stack I own end to end in JS1B/ai-assistant: STT and TTS on edge hosts, LLM inference and PostgreSQL memory on a GPU server. The same compose layout ran a formal benchmark campaign: 78 LLM, 14 STT, and 24 retrieval runs on x86 and ARM.
Scope and ownership
| Workstream | Owner | Where to read |
|---|---|---|
| Repo architecture, memory layer, deployment split | Piotr (sole owner) | This page |
| Formal evaluation and benchmark tables | Piotr | CS Master’s Thesis |
| Full technical build log and numbers | Piotr | Architecture & benchmarks post |
| Thesis benchmark | Scope |
|---|---|
| LLM | 78 end-to-end /chat data points |
| STT | 14 data points across x86 and Raspberry Pi 5 |
| Retrieval | 24 runs measuring memory-mode overhead |
Edge speech, server inference
A spoken turn crosses capture, VAD, transcription, retrieval, generation, synthesis, and playback. Latency and failure handling add at each boundary. Service placement is part of the product.
Decision: STT/TTS on the edge host; LLM, Ollama, and Postgres on the GPU server.
Constraints: microphone and speakers need local ALSA/PulseAudio access. GPU and durable disks sit on the server. HTTP/JSON still has to complete one assistant turn across those hosts.
Outcome: compose.speech.yml runs STT with host networking and direct ALSA capture, TTS with host audio sockets, and Whisper weights in a named volume. compose.llm.yml runs ROCm Ollama with /dev/dri and /dev/kfd, the LLM service behind /health, and pgvector/pgvector:pg17 with schema init from mounted SQL. The LLM service waits on database and Ollama health checks and fails fast if memory does not initialize. There is no silent degraded startup.
Measured trade-off: edge STT on Raspberry Pi 5 ran on the order of 8.7× slower on average than the x86 laptop for matched configurations. Short VAD-filtered turns remain viable. Continuous workloads need an explicit offload story. Details and STT speed ratios are in the benchmark post.
PostgreSQL as the memory system
Early planning assumed a separate vector database. Vectors, episodic rows, relational metadata, and correction signals now share one Postgres schema — one backup path, one operational surface.
Decision: PostgreSQL + pgvector instead of a detached vector sidecar.
Constraints: coding-assistant context needs project namespace, working directory, and git state alongside embeddings. Retrieval must stay bounded before prompt injection.
Outcome: episodes store user/assistant turns plus situational metadata. Hybrid retrieval combines semantic similarity, recency, and situational match. The /chat path trims retrieved snippets so prompts stay within local context limits. llm-service/memory/episodic_memory.py and summaries/POSTGRES_MEMORY_README.md document the implementation.
Measured trade-off: enabling retrieval increased median end-to-end latency by about 2.4× in one thesis configuration versus a no-retrieval baseline. Memory mode shifted latency more than swapping similarly sized chat models. Retrieval has a product-visible systems cost. Full tables live in the thesis repo and benchmark post.
What runs in the repo today
The shipped stack covers:
- distributed STT and TTS services with VAD-filtered segmentation before Whisper,
- an LLM service integrated with Ollama,
- PostgreSQL + pgvector episodic memory with hybrid retrieval and fail-fast init,
- benchmark harnesses and thesis asset generation against the same Docker layout.
The implementation plan sketches richer orchestration, reasoning loops, and longer-horizon assistant behavior. Read it as architecture intent — parts are roadmap, parts are stale. It does not describe current runtime status.
Verified
- Distributed compose split with health-checked startup.
- Episodic memory with hybrid retrieval and fail-fast initialization.
- Formal 78 LLM / 14 STT / 24 retrieval benchmark campaign.
Roadmap: extended orchestration and longer-horizon behavior from implementation-plan.md.
Further reading
- Local voice assistant: architecture, deployment, and what the benchmarks found — pipeline, deployment, and what the numbers mean
- CS Master’s Thesis — citable PDF, full tables, evaluation scope boundaries