THE INFERENCE RIG

Running a fleet of autonomous agents means running a lot of inference. Not just the frontier model handling complex reasoning — the cheaper, repetitive work: alert vetting, draft scoring, embedding queries, routing decisions. At cloud API rates, that volume adds up fast. More importantly, it adds up unpredictably.

There's a second problem beyond cost. An agent that sends every observation to a cloud endpoint is an agent with no privacy boundary. Security telemetry is sensitive. The knowledge graph stores personal context. Those workloads needed inference that never left the building — not as a preference, but as a design requirement.

The studio already had the GPUs. The question was whether open-weights models at the right size could carry frontier-adjacent quality for the workloads that actually needed it.

A local GPU inference cluster, deployed as a virtual machine on a dedicated AI host. The GPUs are grouped into independent serving pools, each running tensor-parallel across its cards. A custom round-robin router sits in front of the pools as a supervised service. Consumers hit the router endpoint and never touch the backends directly.

Two model lanes run on it today. A mid-size open-weights model is the workhorse — fully GPU-resident, fast enough for interactive agent calls, capable enough for classification, vetting, and draft generation. A larger reasoning model handles the heavier reasoning tasks.

A separate GPU in its own container runs the image-generation workload. Inference and diffusion never compete for the same VRAM. That separation was a deliberate architectural decision: the language cluster serves inference, the dedicated card serves image generation, and each can saturate its hardware independently.

The quantization tradeoff. Not all quants are equal. During setup, an aggressive quantization on a single GPU ran faster than a quality-preserving one split across GPUs — but the output quality was unusable for the downstream agents that depended on it. The lesson: quantization aggressiveness interacts with model architecture in ways that raw throughput benchmarks don't capture. The sweet spot turned out to be a quality-preserving quant with tensor-parallelism across cards. Quality and throughput aren't always in tension — sometimes the right pairing gets you both.

The context-window lesson. The serving layer shipped with a small default context window. Several downstream agents silently started failing — they were sending prompts longer than the default and getting truncated, malformed outputs with no error signal. The fix was straightforward once the cause was found: size the context limit to the real workloads. The lesson was the audit process that found it: agents need explicit context-window contracts, and "it runs" isn't the same as "it runs on the inputs it will actually receive."

vLLM and the pragmatism decision. vLLM was evaluated as an alternative serving layer — better batching, higher theoretical throughput. It was blocked at the time on model-architecture support for the specific model families in use. Ollama shipped, vLLM didn't. The rule that emerged: pick the tool that runs today over the tool that's theoretically better but isn't ready. Infrastructure that works beats infrastructure that's optimal.

The router. A single round-robin proxy in front of the serving pools means consumers don't need to know how many backends exist or which one is healthy. When a backend needs a model reload or a restart, the router keeps the cluster reachable. The router runs as a supervised service — it starts on boot and restarts on failure — and the cluster is never exposed to the internet.

The rig serves three active consumers today: the agent fleet's cheap-token lane (classification, routing, draft scoring), the security alert-triage pipeline (alerts are triaged locally before they reach the operator), and the knowledge graph's embedding endpoint. All three workloads run at zero per-call cost and zero cloud exposure.

The cost-per-token savings versus cloud API are real but UNMEASURED — the studio didn't baseline cloud spend before switching, so a clean comparison doesn't exist. What is measurable: the alert-triage pipeline runs with no cloud dependency and no per-call cost, verified end to end by synthetic event injection. Power draw and utilization percentage are likewise UNMEASURED.

The diffusion / inference separation proved its value immediately: running an image-generation batch on the dedicated card while the language cluster handles agent traffic produces no VRAM contention and no queue stalls. Two workloads, two GPU pools — each at its own ceiling.