Blog
← Back to Blog

The Missing Middle: I Built a Proxy to See What My AI Tools Were Doing

/ waniwira / #ai#llm#small-models#go#sse

I've spent a long time trying to figure out how small models can work alongside developers. Not the frontier giants — the small ones. The ones you can run yourself, cheaply, privately, on a laptop or a single GPU. I did research, I ran tests, I built agents. And I'll be honest: small models have too many limitations for big code development.

But I still believe there's more you can do with them. You just have to know them better.

That belief pushed me into an uncomfortable corner: I didn't actually know the tools I was using. I was running a harness — opencode — pointed at a local model server built on llama.cpp and vLLM. And the space between those two pieces was the only part of my stack I couldn't watch.

Not because the information didn't exist. It did. llama.cpp's verbose logs had it. But finding it meant leaving my workflow, sifting raw output, and stitching a story back together from fragments. The middle of my stack had no eyes of its own.

I call that gap The Missing Middle. This is the story of how I built a tool to see it — and what seeing changed.

The Missing Middle

Every AI stack has the same shape:

[harness / opencode]  ──▶  [???]  ──▶  [llama.cpp / vLLM]

The harness is yours to command. The server is yours to log. But between them — the actual traffic, the request bodies, the truncated context, the streamed tokens, the reasoning_content a small model produces before it answers — that space belongs to no one.

I could always find the truth. But finding it meant leaving my workflow, digging through verbose logs, and reconstructing what happened after the fact. I wanted to watch it as it happened, in the tool's own language.

There were already tools for this. Some of them excellent — gpt-load alone has 6.3k stars and does most of what I wanted. But I had two reasons to build from zero:

Reason one: I wanted to experiment while developing. The build itself was the testbed. Every decision I made — how to parse the stream, how to store the conversation, how to structure the growth — was a question I wanted to answer by living it, not by reading a README.

Reason two: more tools keep me blind. This one is subtler. Every layer you add is a layer you don't see. A dashboard is great until you trust the dashboard instead of the data. Building the lens myself meant I could never be fooled by it — I knew exactly what it saw and what it missed.

So I started with a research document. Not a plan — a mind state.

The Mind State

Before I wrote a line of code, I spent a session with a research agent I'd built for learning. The result wasn't a spec. It was a thinking log — a snapshot of where the idea space stood, so every later decision could be traced back to it.

The core identity came first:

The app is first and foremost an observability instrument: a transparent proxy that captures what actually happened between client and inference server.

Then the idea space split into four threads:

Thread Priority Purpose
Capture Core, now Transparent proxy: intercept + log every request/response
Crawl Core, now Review past conversations from the DB — the primary use
Watch Optional Realtime SSE view of live conversations
Modify Future Actively transform requests — a different design space

The key distinction was transparent vs. active. Capture, Crawl, and Watch observe without changing the traffic. Modify changes what the server receives. Those are different worlds, and forcing them into one design would have poisoned both.

The research also caught two truths that shaped everything:

The O(N²) trap. chat.completions is stateless by design — clients resend the full accumulated messages array on every request. A 200-turn conversation becomes ~20k rows of mostly duplicated messages if you log naively. The fix: store each unique message once, content-addressed by SHA-256, and reconstruct conversations by walking the chain. A proven pattern — Simon Willison's llm does exactly this: "re-logging a conversation that grew by one turn only inserts the new tail."

The framework trap. I almost built this on Fiber. Research stopped me: Fiber v3 sits on fasthttp, not net/http, and fasthttp has a known SSE disconnect-detection limitation — RequestCtx.Done() fires only on server shutdown, not client disconnect. Zombie subscribers and goroutine leaks. Called an architectural limitation, "not going to be fixed." The plan locked net/http instead. Not because it was fashionable. Because the physics said so.

Curious about this one? It's a rabbit hole worth falling into:

  • fasthttp package docs — the source of truth. RequestCtx.Done() carries the comment: "only closed when the server is shutting down." One channel, shared by every in-flight request.
  • gofiber/fiber#3307 — an SSE developer hitting the wall live: Hijack either detects disconnects or sends events, never both. Ends with "I solved it by using net/http" — the same conclusion my research reached independently.
  • gofiber/fiber#4194 — the proposal to add an SSE package, which explicitly names the limitation: "fasthttp.RequestCtx.Done() only fires on server shutdown, not per-client disconnect."
  • gofiber/fiber#4263 — the twist: Fiber's own docs claimed Done() fires on client disconnect. This bug report proves it doesn't, quoting the fasthttp source, and even links a community workaround library that wraps the listener to fake per-connection cancellation. As of this writing there's still no first-class fix — the workaround lives outside the framework by design.

The Contract

The research defined the physics. Now the plan defined the laws.

My own scope discipline set the tone — so many ideas in that file... but for now, I want to create the simple one. A proxy that writes everything to a realtime web viewer. No SQLite yet. No crawl API yet. But with the structure ready to grow.

The plan locked the decisions:

Decision Choice
Realtime transport SSE + native EventSource
Backlog on connect Live-only; ring-buffer replay later
Capture scope All endpoints; raw bodies + parsed SSE tokens + assembled message
Auth Optional Bearer check via env
Static assets //go:embed with a -tags dev escape hatch

And at the heart of it, six lines of Go that became the whole architecture:

// Publisher is what the capture layer depends on. Anything that wants to
// observe traffic implements it and registers with the bus.
type Publisher interface {
	// Publish delivers one captured event. Implementations must not block
	// the capture path for long: Hub drops slow subscribers instead.
	Publish(Event)
}

The proxy captures. Anything can observe. Today it's a browser; tomorrow it's a database; it's the same code. That interface was the growth seam — the thing that would let the tool grow without ever being rewritten.

The Build

The plan said: hub first, then proxy, then web, then the viewer. Each step verified before the next.

Here's the first build in one picture — including the seam that lets it grow:

graph LR
    C["OpenAI SDK Client"] -->|"POST /v1/chat/completions"| P["Proxy<br/>capture + optional auth"]
    P -->|"forward unchanged"| U["Upstream<br/>llama.cpp / vLLM"]
    U -->|"SSE token stream"| P
    P -->|"Publish(Event)"| H["Hub<br/>event bus"]
    H -->|"today"| V["Web Viewer<br/>EventSource / SSE"]
    H -.->|"future: second publisher"| S["SQLite Store<br/>implements Publisher"]

The proxy is fully transparent — the exact bytes pass through to the client unchanged. In parallel, the capture layer parses the SSE stream and publishes typed events (request, chunk, message, done) onto the hub; the viewer subscribes over SSE and renders tokens live.

The seam is the Publisher interface: the hub knows only Publish(Event), so the capture layer never needs to know who is listening. Today it's a browser. When the tool needs persistence, a SQLite store implements the same interface and registers on the same bus — the proxy never changes. That was the point of the research and the plan: build the simple thing, but make the seam explicit.

And then it worked. A browser open to the viewer, a request sent through the proxy, and tokens streaming onto the page one by one. The first moment I could see the middle — not in a log file afterward, but live, as it happened.

Growing It — Seeing More

The MVP was a feed. Then the questions got deeper: what is the model actually thinking? So the dashboard grew — Main / Chat / Functions tabs. The Chat tab turned raw JSON into readable role cards. Assistant cards carried collapsible reasoning_content and tool calls, with tool responses embedded into the assistant that called them.

The point wasn't the features. The point was that the shape of the tool kept generating ideas. Room to grow means ideas pop up — and the architecture was ready for them.

The research had already flagged one thread as a different design space: Modify. Capture, Crawl, and Watch observe traffic without touching it — the proxy stays transparent. Modify changes what actually flows: rewriting the request before the server sees it, reshaping the response before the client gets it. Client → server, server → client.

That's the sub-harness idea I keep coming back to. The tool doesn't have to just watch my harness — it can test harness concepts themselves: alter the traffic between harness and model, then observe the effect on the other side. Because the proxy owns the middle, a transform plugs in the same way the viewer and the store did. Watching was step one. The seam is what makes step two cheap.

The tool isn't finished — that's not a confession, it's the design. A growth seam exists to be ready when the next idea knocks, and the ideas keep knocking. The sub-harness experiment is the one at the door today; I can't see the ones behind it. I don't have to. I didn't build a tool. I built a place where tools get born.

What It Unlocked

The tool became the experiment.

I could finally see what actually flows between my harness and my small model server: the full request bodies, the context that got resent every turn, the token stream as it happened, the reasoning the model does before answering, the tools it reaches for. Not reconstructed from logs. Observed.

That changes the relationship you have with your stack. When you know exactly what your harness sends and exactly what your server returns, the "limitations" of small models stop being mysterious walls and start being specific, addressable problems. You stop guessing. You start seeing.

And it loops back to the original belief: I believe there's more you can do with small models. The more I see of what they actually do, the more I think that belief is right — and the more I understand that the tooling around them is the thing holding them back.

Build the lens yourself. You'll never be fooled by it.

Next in this series: the philosophy behind the method — why "prompting" fails and "architecting" survives — in Part 2: The Architecture of Intent.

Case study: jpt_chat_log — a transparent OpenAI-compatible reverse proxy with a realtime SSE web viewer, optional SQLite persistence, and a history API. Built in a single session with zero external dependencies (stdlib + modernc.org/sqlite).