AI Assistant

2026-02-11

Code and docs

Distributed self-hosted voice assistant — STT and TTS on edge hosts, GPU-backed inference and PostgreSQL+pgvector memory on a server, with a formal thesis benchmark campaign on x86 and ARM.

python · docker · llm · ai-assistant · postgres · self-hosted

Repository

AI Assistant is a self-hosted voice stack I own end to end in JS1B/ai-assistant: STT and TTS on edge hosts, LLM inference and PostgreSQL memory on a GPU server. The same compose layout ran a formal benchmark campaign: 78 LLM, 14 STT, and 24 retrieval runs on x86 and ARM.

Scope and ownership

WorkstreamOwnerWhere to read
Repo architecture, memory layer, deployment splitPiotr (sole owner)This page
Formal evaluation and benchmark tablesPiotrCS Master’s Thesis
Full technical build log and numbersPiotrArchitecture & benchmarks post
Thesis benchmarkScope
LLM78 end-to-end /chat data points
STT14 data points across x86 and Raspberry Pi 5
Retrieval24 runs measuring memory-mode overhead

Edge speech, server inference

A spoken turn crosses capture, VAD, transcription, retrieval, generation, synthesis, and playback. Latency and failure handling add at each boundary. Service placement is part of the product.

Decision: STT/TTS on the edge host; LLM, Ollama, and Postgres on the GPU server.

Constraints: microphone and speakers need local ALSA/PulseAudio access. GPU and durable disks sit on the server. HTTP/JSON still has to complete one assistant turn across those hosts.

Outcome: compose.speech.yml runs STT with host networking and direct ALSA capture, TTS with host audio sockets, and Whisper weights in a named volume. compose.llm.yml runs ROCm Ollama with /dev/dri and /dev/kfd, the LLM service behind /health, and pgvector/pgvector:pg17 with schema init from mounted SQL. The LLM service waits on database and Ollama health checks and fails fast if memory does not initialize. There is no silent degraded startup.

Measured trade-off: edge STT on Raspberry Pi 5 ran on the order of 8.7× slower on average than the x86 laptop for matched configurations. Short VAD-filtered turns remain viable. Continuous workloads need an explicit offload story. Details and STT speed ratios are in the benchmark post.

PostgreSQL as the memory system

Early planning assumed a separate vector database. Vectors, episodic rows, relational metadata, and correction signals now share one Postgres schema — one backup path, one operational surface.

Decision: PostgreSQL + pgvector instead of a detached vector sidecar.

Constraints: coding-assistant context needs project namespace, working directory, and git state alongside embeddings. Retrieval must stay bounded before prompt injection.

Outcome: episodes store user/assistant turns plus situational metadata. Hybrid retrieval combines semantic similarity, recency, and situational match. The /chat path trims retrieved snippets so prompts stay within local context limits. llm-service/memory/episodic_memory.py and summaries/POSTGRES_MEMORY_README.md document the implementation.

Measured trade-off: enabling retrieval increased median end-to-end latency by about 2.4× in one thesis configuration versus a no-retrieval baseline. Memory mode shifted latency more than swapping similarly sized chat models. Retrieval has a product-visible systems cost. Full tables live in the thesis repo and benchmark post.

What runs in the repo today

The shipped stack covers:

  • distributed STT and TTS services with VAD-filtered segmentation before Whisper,
  • an LLM service integrated with Ollama,
  • PostgreSQL + pgvector episodic memory with hybrid retrieval and fail-fast init,
  • benchmark harnesses and thesis asset generation against the same Docker layout.

The implementation plan sketches richer orchestration, reasoning loops, and longer-horizon assistant behavior. Read it as architecture intent — parts are roadmap, parts are stale. It does not describe current runtime status.

Verified

  • Distributed compose split with health-checked startup.
  • Episodic memory with hybrid retrieval and fail-fast initialization.
  • Formal 78 LLM / 14 STT / 24 retrieval benchmark campaign.

Roadmap: extended orchestration and longer-horizon behavior from implementation-plan.md.

Further reading