Written in July 2026. Anything that has changed since then is marked with an October update.
I liked the idea in this story, and it failed its test. Nobody required me to publish the result.
The idea
Lookout, the main model in my algorithmic trading program, does one job. It ranks stocks by the probability of an imminent large move. Like any model, it has good days and bad days, and it occurred to me that the good and bad days might themselves be predictable. So the idea was a second, much smaller model whose only task was to grade how far the first model could be trusted, one day at a time, sorting each day into one of three classes (LOW, MID, or HIGH) by its confidence that the big model’s rankings would hold up that day.
The appeal was practical. If the grades were real, position sizes could follow them, larger on the days the layer graded HIGH and smaller on the days it graded LOW.
A model that knows when to distrust itself would be valuable
The rule
Following the program’s standing practice, I pre-registered the trust layer before any code was written. The design doc stated the hypothesis, and it stated an adoption rule. For the layer to be used in trading, the grades had to mean what they claimed on out-of-sample days. Days stamped HIGH had to be better days for the big model, and days stamped LOW had to be worse ones. A grade that works in only one direction is worse than no grade at all, because sizing decisions would rely on it in both directions.
The rule was quantitative. It set four gates, every one required, on at least 200 out-of-sample predictions. A prediction stamped LOW had to be followed by a below-median result for the big model at least 60 percent of the time. The LOW class’s average had to sit at least 0.02 below the HIGH class’s. Both had to hold again in a leak control that graded one window further ahead, so no test day was shared. And weighting by trust had to beat not weighting at all. The evaluation then ran the same way everything in this program runs, on walk-forward windows the layer had never seen. I wrote down sizes in advance because a prediction that only states a direction can count a tiny effect as a win.
I signed the adoption rule while I was still impartial, before LOW and HIGH were results I had feelings about.
The verdict
The layer was built, trained, and evaluated on walk-forward windows like everything else in the program.
The HIGH class worked, somewhat. Days the layer flagged as trustworthy tended to be better days for the big model, and 75.5 percent of those predictions landed above the model’s trailing median, against about 61 percent of predictions in general. It was a small effect, pointing the right way. Had the experiment ended there, I would have been writing a very different and considerably smugger post.
The LOW class failed. On the measure the rule used, the effect pointed the wrong way. Days the layer flagged as low-trust landed below the big model’s trailing median only 31.2 percent of the time, less often than an ordinary prediction did (about 39 percent), when the rule needed at least 60 percent. Their average score did come in below the other classes, by enough to clear the 0.02 gap gate, but on a class of only sixteen predictions, which settles very little. So the layer could partly recognize the big model’s good days and was unreliable about the bad ones, which were the half that risk management needed.
The pre-registered rule required every gate, and the LOW gate failed both here and in the leak control, so the verdict was FAIL, and the ledger records it that way. I shelved the layer before it was wired into the nightly run, so no position size ever depended on its grades.
October update: its code was archived in an October cleanup of closed studies, and its FAIL stays on the record.
I like this result now. The test asked whether Lookout’s bad days could be predicted, and the answer was that its good days could be spotted and its bad days could not. That is something I know about the system I trade, and I would not know it if the test had been designed to be easy to pass.
Then I was tempted to adopt just the HIGH half, since it works. The rule I signed asked about the whole layer, and rewriting a rule after seeing the results is what the rule exists to prevent. If the HIGH class alone is worth having, that is a new hypothesis, and it can have its own design doc, its own bar, and its own entry in the trial ledger like every other idea.
Why publish this
Most public track records are filtered, and you usually cannot see what was filtered out. A notebook that records only wins is essentially just advertising, because the reader has no way to know how many attempts produced the wins on display. The multiple-comparisons arithmetic from the last letter applies to publications the same way it applies to backtests, because the problem is the count of attempts left out.
Publishing this failure shows you part of that count. If a success appears in this notebook, you will be able to judge it against entries like this one, where the test said no and I published it anyway.
This notebook publishes its FAIL verdicts. Subscribe for the whole ledger, failed attempts included.
Writing each loss down keeps the tests hard enough to fail, which is the only reason a pass means anything. When the live diary starts, with real money behind its percentages, every win it reports will sit next to entries like this one. The failed trust layer showed me where my model’s knowledge of its own reliability stops.
October update: the live diary has not started; apart from a few days of hand trading at half size in August, nothing has traded real money. The public record today is The Graded Tape, at fixed paper sizes. Through Oct 21, 2026, 63 trading days and 11,888 signals graded, it reads -$1,348.45.
Lab notebook, education and research purposes only. Not investment advice, not a solicitation; no advisory relationship is created by reading this. I trade a small account and describe my own process, errors included.
Paid subscribers get the build notes below, covering six minutes and forty seconds from rule to verdict, and the rule I forgot to write.
How I built it
The trust layer’s design doc went into git at 2:04 p.m. on July 12, 2026, with 174 lines, the four gates, and a line saying the first version stays display-only whatever the results. The code, 12 tests and the FAIL verdict went in six minutes and forty seconds later. The agent wrote about 700 lines in plain NumPy and wrote the report card; no GPU was involved.




