THE HONESTY GATE

AI advice fails most often not through bad reasoning, but through fake confidence. A model with half the relevant data will still return a recommendation at 0.92 confidence if nothing stops it. The structural problem is not that models lie intentionally — it's that nothing in the generation path connects the strength of an assertion to the quality of the evidence beneath it.

A daily strategist that you can't trust to know what it doesn't know is worse than no strategist. It trains you to ignore caveats. It conflates missing data with confident irrelevance. It produces a 0.90-confidence recommendation on a day when the metrics digest was unreachable and three key backlog items were stale — and it presents that recommendation exactly like the one from the day everything was current.

The problem is tractable, but only if you build the ceiling into the pipeline, not into the prompt.

The system produces ONE recommendation per day: a single "do this today" action drawn from an 11-source briefing packet. Sources include the metrics digest, current backlog state, financial indicators, a research drop, activity logs, and related feeds. Any source that fails to load enters the packet honestly — as available: false with a value of "unknown". Nothing is fabricated to fill a gap.

The recommendation is structured into 10 required fields: action, why, what-it-unlocks, what-to-ignore, risks, the 30-minute version, the 2-hour version, done-definition, confidence, and evidence-used. It doesn't ship until it passes the gate.

The gate is deterministic. It does not ask the model to evaluate its own honesty — it measures the recommendation against rules the model cannot override.

The full gate runs five checks in sequence:

1. Structure: all 10 required fields must be present and non-empty.
2. Range: confidence must be a number in [0, 1].
3. Coverage cap: confidence may not exceed 0.30 + 0.70 × (available / total).
4. Evidence citation: every source named in evidence-used must be a source that was actually available in this run's packet. No citing what wasn't there.
5. High-confidence bar: confidence ≥ 0.5 additionally requires at least 2 distinct evidence sources. A high-confidence recommendation backed by a single data point is rejected.

Before the frontier tier took the strategist role, a local open-weight model was tested. It was fast, free to run, and produced structured JSON that parsed cleanly. The audition ran live against a real briefing packet.

The model returned a recommendation at 0.92 confidence. The coverage cap caught the first problem: only 5 of 11 sources were available that day, putting the ceiling at 0.65. Rejected on structural grounds.

But the structural rejection masked a deeper problem. Reading the rejected output, the model had fabricated product specifics and sales figures that appeared nowhere in the briefing packet — inventing plausible-sounding details to fill the gaps where sources were missing. The structural gate caught the confidence violation. It did not catch the invented prose.

This distinction matters: the gate rules out the provably wrong (overconfident on thin data, citing unavailable sources, missing required fields). It cannot rule out the subtly invented. That's a model-quality problem, not a validation problem — and the correct answer is to route synthesis to a model tier that doesn't hallucinate the gaps, while keeping the gate as the structural floor that never relaxes regardless of which model is upstream.

A strategist that never gets graded is a strategist that never gets better. Every recommendation that passes the gate is logged to a prediction ledger before the outcome is known: the expected score, the reasoning, and a confidence reading — all written at decision time, before the week plays out.

Graded outcomes update a belief store. Predictions that land accurately strengthen the weight of the reasoning patterns that produced them. Those that miss, weaken them. The calibration is meant to compound over time.

The page must say this plainly: the calibration data is thin, and what exists is biased.

The house rules that govern this system — "a denominator accompanies every verdict" and "UNMEASURED beats invented" — apply to the system's own self-description. This page is written under the same gate it describes.

The gate has been live since early June. Every recommendation that ships has passed five deterministic checks; recommendations that don't pass are logged with their rejection reason rather than silently dropped or retried without bounds. The local model was rejected after one audition session. Synthesis has stayed on the frontier tier since.

The prediction ledger is accumulating. The calibration signal will be meaningful when the grading coverage is a real denominator rather than 18 of 205. The 1.5-point average error on a 10-point scale is a real number from a small sample. It's reported with that caveat, not without it.