THE SELF-HEALING LAB

SRE discipline for a one-person fleet — incidents that never page a human

Site Reliability EngineerVirtualization / Linux services / watchdogs / LLM triage2025 — NOW

The Problem

Traditional SRE wisdom: page someone when something breaks. This fleet has no someone to page. A small self-hosted cluster runs three always-on AI daemons that execute work autonomously, 24 hours a day, with no on-call rotation and no ops team.

The design question was not "how do we page faster" but "how do we stop paging at all." That meant building a system that could detect its own failures, classify them, attempt self-recovery, and — only when it genuinely couldn't proceed — hand a diagnosis to the operator rather than a raw alert.

The goal isn't faster incident response. It's incidents that never reach a human.

Detection Architecture

Alerts are triaged automatically before they reach me. Determinism handles the easy cases, a local language model handles the hard ones, and nothing makes a network call to an external provider.

Why local inference. The classifier runs on local inference (a local open-weights model at interactive speed). No external API call, no data egress, no per-alert cost, no rate limits from an outside provider. The tradeoff is compute weight for a classification task that a smaller model might handle. The payoff is zero keys at risk, deterministic testability, and a classifier that can be interrogated locally before being trusted with live alerts.

Proven, not assumed. The alert path is drill-tested end-to-end: a synthetic event travels from detection through automatic triage to push delivery and is confirmed at the receiving endpoint. A path that has not passed its drill is not considered live.

Self-Healing Patterns

Three patterns compose to keep the system alive without human involvement. None are novel in isolation. What matters is how tightly they are specified and how faithfully they are documented when they fail.

Pattern 01 — Watchdog FSMs for daemon liveness

Each daemon runs as a supervised service with a watchdog that classifies its state into a finite set of transitions: idle → active → frozen → restarted. The FSM prevents restart-loops on transient states — "activating" and "heads-down conductor" are distinct from "frozen" and do not trigger recovery.

A critical production lesson: I found and closed a monitoring blind spot where a daemon could look busy while delivering nothing. The fix was a principle, not a patch — measure the real output, not activity. A process that is working and a process that is delivering are not the same thing.

Pattern 02 — Session-state resume after restarts

Each daemon owns a per-lane state file recording what it is executing right now and the next planned step. The file is written at step boundaries and cleared on completion — not every turn. On restart, the boot brief surfaces the state file first, so a restarted daemon resumes warm rather than cold-starting and dropping in-flight work.

The failure mode this closes: a daemon that hits a session limit mid-turn never auto-resumes after the window resets. Without the state file, queued work would stall with nothing to show for it. With it, restart equals resume.

Pattern 03 — Guaranteed-delivery alert backstop

The alert path has a backstop that verifies delivery at the destination endpoint, not that the sending service is merely active. That distinction matters: a service can report "active," dispatch a payload, and still silently mis-route the alert to the wrong host or stream.

Verification method: synthetic end-to-end injection. A synthetic event traverses the entire pipeline — detection through triage through push delivery — and is confirmed at the receiving device, not inferred from server-side logs. "Service active" is not the same as "alerts delivered."

Liveness is not "process alive." It is "the right output reached the right destination in the last N seconds." Every other definition is an optimistic proxy.

Incident Postmortems

Three real incidents from the permanent archive. Each produced an entry in an append-only lessons corpus — 160 entries spanning March–July 2026 — so the same failure cannot recur without a prior warning. The format is the differentiator: every incident resolves to a headline rule that the next session inherits before acting.

PM-01A host network setting had a side effect that outlived the fix

Timeline

A routine host network change was made. Traffic between machines stopped, while every local health check kept passing. No immediate error surfaced.

Root cause

The change carried a hidden side effect, and reverting the visible setting did not undo it. The side effect outlived the fix.

Fix

Verified the underlying state directly rather than trusting the switch, and restored it. Traffic recovered immediately.

Detection gap

Checks that only looked at one host stayed green throughout the outage. Verification now checks the whole fleet, not just the machine that changed.

Lesson L089 — permanent archiveUndoing a setting does not always undo its effects. Verify the underlying state, not the switch — and verify across the whole fleet, because local checks can stay green while the real output is gone.

PM-02Scheduled maintenance caused fleet-wide false “network down” alarms

Timeline

A scheduled maintenance job briefly paused the fleet. On resume, the liveness watchdog fired “chat / network broken” alerts across multiple daemons simultaneously.

Root cause

The watchdog could not tell a genuine outage from a planned pause. Both looked the same from where it was standing.

Fix

Fixed the false-alarm pattern around scheduled maintenance, so the watchdog now recognises a planned pause for what it is.

Detection gap

The watchdog was answering the wrong question. The root cause was maintenance timing, not a network failure.

Lesson L094b — permanent archivePlanned maintenance can look exactly like an outage to a watchdog. One broken lane is not darkness — require two independent signals before declaring an outage.

PM-03A mute rule that looked right but did nothing

Timeline

A rule was added to mute a noisy alert. The alert kept firing. The mute appeared effective from the config alone.

Root cause

The rule was present on disk but not in effect, so the alert behaved as if no mute existed.

Fix

Corrected the rule and verified the mute by watching the alert stop — not by verifying the config file contained the directive.

Detection gap

Config-level verification (“the rule is there”) is not behavioral verification (“the rule is applied”). The only true test is observing the suppressed alert stop arriving.

Lesson L123 — permanent archiveA rule written down is not a rule in effect. Verify a mute by watching the alert STOP, never by the config containing the line.

Stack

VirtualizationLinux servicesNetwork IDSLocal open-weights LLMWatchdog FSMsLocal inferenceDedicated sensor

Result

What "healthy" means on this fleet is measured, not asserted. The numbers below come from the verified corpus — not from a monitoring dashboard that reports uptime without defining what it is counting.

Measured & verified

160append-only engineering lessons in permanent archive (Mar–Jul 2026)

1,564work receipts on disk (1,002 active + 562 archived)

406persistent memory files; 121-entry curated index

3always-on daemons (restart-safe, session-state resume)

50installed skills; 21 learned beliefs (6 gates, 15 advisory)

1synthetic end-to-end alert injection, delivery confirmed at the device

27findings from independent adversarial Constitution red-team, all addressed

UNMEASURED — labeled honestly

—MTTR before and after system introduction — no baseline exists

—Uptime % per host — monitored but not aggregated or trended

—Alert volume per day — tracked, not yet trended

—Alert triage accuracy as a published rate — drill-tested, not yet trended

—Cost savings in $ — no pre-system labor baseline to compare against

—Daemon restart frequency — watchdog fires not yet aggregated into a trend

—Session-state resume success rate — feature too new; sample too small

The UNMEASURED column is not a failure — it is an honest instrumentation backlog. A metric labeled UNMEASURED tells you exactly where to invest next. A metric invented to fill the gap tells you nothing and erodes trust in everything beside it.

The archive holds 160 lessons in the postmortem format shown above, append-only, indexed by ID with a one-line headline rule. The index is regenerated on a schedule. Every correction becomes a rule the next session inherits before acting. Institutional memory does not expire when a model does.

Radical honesty beats managed optics. Missing data is more useful than invented data.