The code is in ai-assistant. The thesis that measures it is in CS Master’s Thesis. This post is the technical account: how the system is built, why it is deployed this way, what the memory layer does, and what the benchmark campaign found.
The constraint that shaped everything
The interesting part was never “make Ollama answer over HTTP.”
The hard part was making a voice assistant behave like one coherent system on ordinary hardware with small context windows, visible latency, and services that fail in different ways.
Text chat can hide a lot behind a spinner. Voice cannot. Each stage has its own latency and its own failure modes:
- capture audio,
- detect speech,
- segment audio,
- transcribe it,
- retrieve context,
- build a prompt,
- generate,
- synthesize audio,
- play it back.
A broken pause anywhere in that chain is obvious. That pressure shaped the architecture.
Why the system is split
Speech runs on edge devices. Inference and memory run on a server.
That is not microservices as ideology. It matches the physical use case:
- Speech belongs near the microphone and speakers. The STT service runs in Docker with host networking and direct access to ALSA capture devices. The TTS service mounts the host PulseAudio and PipeWire sockets. Audio I/O stays on the edge, so it does not cross the network on every turn.
- Heavy inference and durable state belong where the GPU and disks are. The LLM service and the database run together with an ROCm-configured Ollama instance and a named Postgres volume.
The two compose files follow that split. compose.speech.yml is the edge side. compose.llm.yml is the server side: Ollama with GPU passthrough (/dev/dri, /dev/kfd), the LLM service, and PostgreSQL with pgvector.
The boundary is HTTP and JSON. That is simple until turn timing, slow-side degradation, and whether the full loop still feels like one assistant start to matter.
Deployment in practice
This is not a design sketch. It runs.
compose.llm.yml shows what that means:
- Ollama is the only service with GPU device access. The LLM service and the database do not. GPU access stays on the model server.
- The LLM service depends on the database via
condition: service_healthyand on Ollama viacondition: service_healthy. Both health checks must pass before the LLM service starts. - The database runs
pgvector/pgvector:pg17and initializes the schema from a mountedinit.sqlon first start. - All three services have health checks. The LLM service exposes a
/healthendpoint.
On the speech side:
- The STT service uses
network_mode: hostso it can reach the microphone without a network bridge in the way. - The TTS service mounts PulseAudio and PipeWire sockets from the host to play audio on the local speakers.
- Whisper models live in a named volume so they are not re-downloaded on every restart.
- Both speech services run as the host user (
HOST_UID:HOST_GID) to avoid audio permission problems.
The LLM service fails fast on startup if the memory subsystem does not initialize. It does not start degraded and fail silently later.
The speech pipeline
STT is not just “Whisper in a container.” It is a small pipeline:
- audio capture,
- preprocessing,
- VAD-based filtering,
- buffering into speech segments,
- Whisper transcription,
- post-processing and stitching,
- reporting.
Preprocessing matters. Raw microphone streams contain silence, noise, and partial utterances. Sending all of that into Whisper wastes compute and makes the interaction worse.
STT preprocessing uses WebRTC VAD plus a speech buffer. Speech frames accumulate. Silence triggers a flush. Continuous speech can keep overlap frames so segments do not break too hard at phrase boundaries.
The transcriber then sees speech segments instead of arbitrary microphone chunks.
The benchmark material includes matched-condition STT comparisons with VAD on and off. The direction was clear enough to change the architecture: filter early, cut wasted transcription work, and improve the responsiveness the user feels.
That is the same order used in larger production voice pipelines: cheap filtering first, then spend model time only on speech.
Memory and retrieval
The main storage choice is PostgreSQL + pgvector.
Early planning assumed a separate vector database. I moved away from that. I wanted one storage system that could hold:
- episodic traces,
- relational metadata,
- correction signals,
- embeddings,
- and future higher-level memory structures.
One database means one operational system to run, monitor, and reason about. No vectors here, facts there, drifting apart over time.
The memory model is built around episodes. An episode carries:
- the user message and assistant response,
- timestamp,
- project namespace,
- working directory,
- git state,
- and other situational metadata.
That situational data matters. Conversations happen somewhere and sometime. In coding workflows, “what repo were we in” and “what were we touching” can matter as much as the literal words. Normal vector similarity search does not capture that.
Retrieval is hybrid: semantic similarity plus recency plus situational match. Retrieved episodes are also bounded before prompt injection. The /chat path trims memory snippets instead of dumping everything found. Retrieval only helps if the resulting prompt stays small enough to be useful.
Context pressure and why it is an architecture constraint
On local hardware, context management is not polish. It is an architectural constraint.
Cloud-scale demos assume you can keep appending conversation to the prompt. On local hardware that fails fast. If the model is small enough to run comfortably, the context window is tight. If the model is large enough to want more context, latency gets ugly.
The design response:
- keep the active prompt bounded,
- compress older interaction into summaries,
- store durable material in the retrieval layer,
- rebuild only the relevant slice when needed.
The foundation for that is in the repo: episode storage, hybrid retrieval, and treating context compression as first-class rather than an afterthought.
What the benchmark numbers actually mean
One thing I specifically avoided in the thesis and in this repo was fake benchmark clarity.
The reported LLM numbers are end-to-end /chat path numbers. They are not decoder-only throughput. They include:
- retrieval when enabled,
- prompt assembly,
- generation,
- and API overhead.
That makes them less flattering than isolated model numbers. They are more useful. End-to-end latency is what the assistant feels like.
The STT metrics use a speed ratio: processed audio duration divided by inference time. Above 1.0 means comfortably faster than real time. Below 1.0 means you are falling behind.
The retrieval benchmarks matter because turning memory on is not a constant tiny tax. Sometimes the overhead is acceptable. Sometimes it is large enough to change the user experience. Measuring it in the same fixed Docker stack made the trade-off visible.
What the thesis formally measured
The benchmark campaign behind the thesis covered:
- 78 LLM data points at the
/chatendpoint level, - 14 STT data points across configurations and platforms,
- 24 retrieval-overhead measurements comparing memory modes.
Platform comparison: matched tests across an x86 laptop and a Raspberry Pi 5.
Main results
1. Architecture was the durable win. The modular system works. STT, LLM, and TTS can be swapped behind stable contracts without rebuilding the database, HTTP surfaces, or benchmark harness around them.
2. Memory mode mattered more than model swapping. In the LLM path, latency shifted more with memory mode than with swapping one similarly sized model for another. That is one of the most useful findings. It breaks the habit of blaming only the chat model.
When retrieval is on, you are measuring vectorization, retrieval, prompt assembly, database access, and generation together. Those have to be engineered as one system.
3. Retrieval overhead can be large. The thesis reports an extreme case where enabling retrieval increased median latency by about 2.4x relative to a no-retrieval baseline.
RAG is not a free feature. It is a systems feature with real cost. Bad pairings between chat model, embedding model, and retrieval strategy can make a voice interaction feel bad fast.
4. Edge STT is viable but constrained. The ARM setup ran on the order of 8.7x slower on average for the measured STT configurations versus the laptop. That does not make edge STT pointless. Short turns with VAD enabled can still be viable. Continuous or more demanding workloads need a deliberate offload story.
5. VAD was a real win, not a detail. The benchmarks confirmed that VAD reduced transcription overhead in a meaningful way. Voice systems are full of these “small” decisions that determine whether the whole chain feels usable.
What the thesis does not claim
- A full semantic-quality study of LLM outputs.
- A formal perceptual evaluation of TTS quality.
- One universally best configuration for every machine and use case.
The work is strongest as a study of latency, architecture, and trade-offs across the full pipeline.
What is shipped vs roadmap
What is in the repo:
- distributed STT and TTS services,
- an LLM service integrated with Ollama,
- PostgreSQL + pgvector episodic memory with hybrid retrieval,
- benchmark and thesis asset generation against the same stack.
The repo also contains an implementation plan with richer orchestration ideas, reasoning loops, and longer-horizon assistant behavior. That document is useful as direction. It is partly roadmap and partly stale. Read it as architecture intent, not a status page.
What I took from the year
The thesis year confirmed some instincts and corrected others.
Benchmarking your own system properly is a good way to lose attachment to your favorite narrative. You expect “pick the smartest model” to be the answer. The data keeps pointing at configuration: memory mode, embedding choice, service boundaries, machine class, whether you are measuring the whole path or just the convenient part.
Writing and building in parallel also created a constraint that was useful: do not describe a system you did not build. Anything that only existed as a design idea became future work. That made the project narrower, but it made the claims honest.
The durable lesson: system quality comes more from architecture, contracts, and measurement than from whatever model release is currently fashionable.
Further reading
- Implementation repo: AI Assistant
- Thesis repo and PDF: CS Master’s Thesis