THE STREAMING HUNT

The rig: a Linux GPU workstation as the host, streaming its desktop at 5120×1440 at 120 Hz over a local network to a lightweight streaming client running integrated graphics. The goal was low-latency game streaming — the GPU lives in one room, the display and controls in another. The baseline was already elite: 2.9 ms round-trip latency. The streaming stack was mature. Nothing should have been wrong.

Three things were wrong. Each one had an obvious suspect. Each obvious suspect was innocent. This is the forensic record of finding the real causes.

The baseline before any of this started: 2.9 ms round-trip latency. That is, by most measures, already elite for local game streaming. During the hunt, several theoretical “optimizations” were applied — encoder parameter tuning, network buffer adjustments — measured carefully, and found to perform worse: 4 ms. They were reverted the same session.

The post-fix result, with the client's decoder stabilized and the correct kernel actually booting: sub-2 ms sustained. The improvement came from fixing the real problem, not from chasing an already-elite metric.

A separate finding during Round 2: the host encoder was running at 98% saturation, outputting approximately 130 Mbps. This wasn't the cause of the green flashes — but it was a real ceiling worth backing off. Found during the hunt, resolved as a separate adjustment.

The methodology held across all three rounds: collect concrete evidence first, form one specific hypothesis, design the smallest possible test to falsify it, run the test, accept the result. When the test falsifies the hypothesis — and it did all three times — return to evidence, not to intuition.

Measure the real quantity. A metric label is not a guarantee that it measures what you think. The fps counter measured on-screen motion — correct for its design, wrong for what the diagnosis needed. Before trusting a diagnostic signal, ask: what exactly does this measure? For what definition of the term it's named after?

Attribute corruption to the consumer, not the producer. Intermittent pixel corruption under load is more likely to be a decode instability at the receiving end than an encode problem at the sending end. The server was running fine. The client was dropping frames in its decoder. Look at where the artifact appears, not where the data was generated.

Installed ≠ booted. A package manager confirming a successful install means the files are on disk. It says nothing about the boot configuration that decides which kernel actually loads. When a fix doesn't take effect after a reboot, read the bootloader config before declaring the install broken.

The rig has been stable since. The streaming stack that was exhibiting three separate failure modes is now operating at sub-2 ms latency with zero decoder hangs. The fixes were targeted: one kernel update, one bootloader repoint, one encoder parameter pullback. Nothing about the host GPU, the network, or the streaming software itself required modification.

The discipline — evidence before hypothesis, one hypothesis at a time, smallest possible falsifying test — is what kept the investigation from sprawling into weeks of misattributed tuning. Each round took less than twenty minutes once the right question was isolated.

The same forensic discipline — evidence before hypothesis, device evidence over emulator evidence — drove the cross-browser debugging story in The Hero Rebuild. That investigation ran seven rounds across three browsers before a single screenshot from the real device ended it.

The self-healing infrastructure that catches system failures automatically is documented in The Self-Healing Lab.