SemanticSwap — a memory-optimized inference proxy for LLMs

An OpenAI-compatible proxy that gives a language model a searchable long-term memory — it compresses the context window on the fly, offloads it into a semantic store, and reloads only what's relevant. Lower token cost, memory that survives past the context limit.

The challenge

Built on an original concept by Jan Häusle and implemented by Ivo Häusle. Every large-language-model conversation runs into the same wall: the context window is finite and every token in it is paid for on every single turn. As a chat or an agent session grows, you either start dropping earlier messages — and the model "forgets" — or you keep resending the whole history and the cost climbs with each reply. What a client actually wants is simple to say and hard to build: a model that remembers the important facts from far earlier in a long conversation, without paying to carry the entire transcript in context every time.

The honest version of that is not a prompt trick. It means sitting invisibly between any OpenAI-compatible client and the model, deciding what to keep in the live window versus what to file away, storing it so it can be searched by meaning (not just keywords), and pulling the right piece back in exactly when it's needed — all without the client changing a single line of code.

The approach

• A true drop-in proxy. SemanticSwap speaks the OpenAI API, so any existing client (Open WebUI, LangChain, a custom app) switches to it by changing one base URL — no SDK, no code changes. It forwards to any model via a provider abstraction (LiteLLM), local or hosted.
• Session tracking without client cooperation. Sessions are recognised by a prefix-hash chain over the message history, so the proxy can follow a conversation — and even detect forks — with no special client support.
• Swap-out: compress and structure, don't just truncate. When the live context approaches its limit, older turns are summarised and distilled into a structured semantic store — graph triples plus vector embeddings in a single SQLite database — instead of being silently dropped.
• **Swap-in: bring back the original, not a paraphrase.** When an archived fact becomes relevant again, a vector lookup finds it and the original snippet is re-injected, so answers stay faithful to what was actually said. The model can also pull archived transcripts itself through a hidden retrieve_archived_memory tool — invisible to the client.
• A measurement harness, not vibes. A repeatable eval runs a long synthetic conversation with anchored facts and reports recall, compression ratio and proxy overhead against explicit targets — offline (deterministic, no model needed) or against real local models.
• Built to be watched while it works. An event bus streams live activity over Server-Sent Events into a web UI: a /ui dashboard for sessions, segments and extracted graph triples, and a /ui/flow flowchart where each architecture node lights up as its component runs (including background sub-agents).
• Documented as it was decided. Fourteen Architecture Decision Records, a PAD design document and an eval report capture why each choice was made — the kind of trail that lets someone else pick the system up.

The result

• A working proxy with milestones M1–M5 implemented (gateway, session tracking, swap-out, swap-in v1, eval harness), self-hosted and running in production on the operator's own machine as an always-on systemd service, driving local models (gemma / qwen) through Ollama.
• Measured, not marketed: in the offline mechanics eval, 100 % recall of the anchored facts, proxy overhead in the single-digit-millisecond range (~1 ms, far under the 100 ms target), and roughly 9× less context sent upstream on the probed final question. Compression ratio came in at ~4.4:1 against a 5:1 target — a real number, honestly reported, not rounded up.
• 73 tests green, run fully offline against a deterministic fake model, so the mechanics are verified without spending a single token.
• A live-observable system: real-time flowchart and event log make an otherwise invisible pipeline easy to demonstrate and debug.
• Provided as-is; access is currently self-hosted for the operator (reachable over a private network), with a password-gated public route prepared but not yet switched on.

What this demonstrates

• Ownership of a non-trivial infrastructure problem end to end: protocol- compatible gateway, compression pipeline, a semantic store combining graph and vector search, retrieval, an eval harness, a web UI, and self-hosted deployment.
• Comfort with real LLM systems work — context management, provider abstraction, retrieval quality, background sub-agents — rather than a thin wrapper around an API.
• Measure-first engineering: explicit targets, an automated harness, and honest reporting of what passed and what didn't.
• Making infrastructure legible — decision records, a design doc, and a live flowchart that turns internal behaviour into something you can watch.
• Modern, AI-assisted delivery, steered and curated by a human — turning Jan Häusle's original concept into a running system.

Tech at a glance

Python 3.11+ · FastAPI / Uvicorn · OpenAI-compatible gateway · LiteLLM · SQLite (graph triples + vectors) · Vector embeddings · Server-Sent Events · Ollama (local gemma / qwen) · systemd · pytest (73 tests)

---