THE ALERT-VETTING PIPELINE
An IDS that only speaks when it means it — LLM triage on local GPUs
The Problem
Alert fatigue is how real intrusions get missed. A homelab running a network IDS against live traffic produces a continuous stream of events: DNS lookups that match threat-intelligence patterns, outbound HTTPS to a CDN that happens to share an IP with something flagged last year, internal service chatter that a ruleset written for enterprise environments has never seen before. Most of these are noise. Some are signals buried in the noise. The system cannot tell the difference by volume alone.
The naive response — page on everything, triage manually — fails the moment alert volume exceeds the human's willingness to look. Which happens on day two. The other naive response — raise thresholds until the noise stops — is just selecting which signals to drop unexamined. Neither is detection engineering. Both are avoidance dressed as tuning.
The design requirement was precise: a human should only ever see an alert that has survived two independent filters. The first filter is deterministic and fast. The second is skeptical and slow. Nothing phones home until both agree.
If everything is urgent, nothing is. The signal-to-noise problem is an architecture problem, not a tuning problem.
The Three-Stage Topology
The pipeline has three stages and one guaranteed-delivery backstop. Each stage is independently verifiable. Each discards events the next stage never sees.
Stage 1 — An IDS on a dedicated sensor node
A network IDS runs on a dedicated low-power sensor node on the network path. It produces a continuous JSON event stream — every rule match, suspicious DNS query, TLS handshake to a flagged destination. At this stage the system imposes no filter. Volume is the point: the sensor captures everything so later stages can choose what to discard.
Stage 2 — Deterministic pre-filter
The pre-filter processes the event stream with no machine learning and no remote calls. Four deterministic checks run in order, covering severity tier, destination context, a known-benign baseline, and event rate over a sliding window. Events that fail any check are dropped. Events that pass all four are promoted to the LLM stage.
Speed matters here. A well-calibrated pre-filter means the LLM spends cycles only on genuinely ambiguous cases — the cases where arithmetic cannot give a confident answer. Everything the deterministic layer can handle, it handles without invoking inference.
Stage 3 — Local LLM skeptic
Alerts that survive the pre-filter reach a local open-weights model running on the studio's own GPUs. The model is tasked as a skeptic: its job is to find a reason the alert is a false positive, given the context — destination, time-of-day, event history, the operator's known infrastructure. If it cannot construct a convincing benign explanation, the alert is real.
The zero-marginal-cost argument: every alert the LLM vets costs a GPU cycle on hardware the studio already owns. No per-call API charge, no rate limit from an external provider, no data transmitted off the network. The model runs at 3 AM on the same owned silicon as noon. That is the ownership dividend for running inference locally.
The LLM stage also carries an injection guard: alert payloads are trust-classed before being passed to the model, preventing a maliciously crafted network event from using the alert pipeline itself as a prompt-injection vector.
Stage 4 — Phone push with guaranteed delivery
Only alerts that survive both the pre-filter and the LLM skeptic reach the operator's phone. The delivery path is not assumed — it is verified. A service can report active, dispatch a payload, and silently route it to the wrong endpoint or wrong stream. "Service active" is not the same as "alert received."
End-to-end verification was performed on 2026-07-13 by injecting a synthetic event into the IDS, watching it traverse all three stages, and confirming delivery at the actual receiving device — not inferred from server-side logs. The test is repeatable and is the only proof of delivery the system trusts.
Craft Details — Four Rules Earned From Failures
These are not principles from a textbook. Each one has a postmortem behind it. The system behaved wrong; the investigation produced a precise rule; the rule went into a permanent append-only archive. What follows is that archive, rendered in context.
The pre-filter first ranked alerts by a derived integer — higher numbers, higher urgency. An informational-tier alert for routine DNS behavior scored above a policy-tier alert covering an actual governance violation, purely on arithmetic. The vendor assigned those tier strings for a reason. Deriving your own severity scale on top of them is hubris dressed as engineering. The fix was simple: use the stated tier as the primary axis. Numeric scores can supplement; they cannot override.
Early monitoring surfaced a stream length metric and labelled it "alert rate." The number climbed continuously. The response was escalating urgency. The problem: the length of a queue tells you how much work has accumulated since the last drain, not how many threats arrived in the last minute. Reading accumulated backlog as instantaneous rate produces exactly one outcome: noise treated as signal. The fix was a sliding window with a tier filter. The climbing number stopped being alarming and started being informative.
A data-exfiltration alert fired on outbound HTTPS traffic. The destination was one of the studio's own internal hosts. Routing traffic to owned infrastructure was scoring identically to routing it to an unknown external host. The same bytes crossing the boundary to a stranger are a very different event than those same bytes staying on owned hardware. The pre-filter's outbound scoring stage now checks destination class before elevating severity. Destination matters. An outbound score without it is half a measurement.
This is the incident worth telling in full. A suppression rule was written into the IDS configuration to mute a known-benign alert. It targeted the right signature, with the right suppression type, and read as correct to anyone reviewing the file.
The rule silently failed to load because of a formatting quirk in the config parser. No parse warning. No error in the standard log. No indication at any visible level that the rule failed to apply. The alert kept firing. From the config file, the suppress looked active. From the phone, it looked broken.
The fix was mechanical: normalize the config format. The lesson was the point. A config file containing a directive is not a directive that is applied. The only verification that counts is observing the suppressed alert stop arriving. Anything else is reading the map and assuming the territory matches.
A config file containing a rule is not a rule that is applied. Verify behavior, not configuration. Watch the alert stop.
Stack
Result
The pipeline is live. Every alert the operator receives has survived the pre-filter and the LLM skeptic. The theoretical promise — that only meaningful alerts make it through — is what the design is optimizing for. The numbers below are what the system can actually prove right now.
The six verdicts in the current log prove the pipeline path works end-to-end. They do not prove the false-positive or false-negative rates, both of which remain UNMEASURED until the log accumulates enough volume to characterize. That is the honest state of a recently rotated system at this stage of operation.
The broader self-hosted infrastructure this pipeline protects — virtualization, the watchdog layer, session recovery, and the other detection layers — is covered in The Self-Healing Lab. This page is the deep-dive on alert-vetting specifically.
Six verdicts prove the path. They don't prove the rates. Saying so is the whole point.