Verbatim

Verbatim

Why I froze three months of data I'm not allowed to look at

The winner's curse, a holdout used exactly once, and the Sunday evening the sealed months were finally read.

Verbatim's avatar
Verbatim
Oct 29, 2026
∙ Paid

Written in July 2026. Anything that has changed since then is marked with an October update.

Last letter I said that even after you adopt walk-forward evaluation and every grade is earned on unseen days, one problem remains. I ran the program with it for a long time before I noticed it.

The winner’s curse

Suppose you test nine ideas, each one graded correctly on out-of-sample windows. Each measured score is the idea’s true quality plus some noise, because finite test windows always carry noise, and the noise can go in either direction. Then you do the natural thing and pick the highest scorer.

By picking the maximum of nine noisy measurements, you have selected partly for quality and partly for luck. The winner is disproportionately likely to be an idea whose noise happened to go in its favor, which means the winner’s measured score is biased upward even though every individual measurement was correct. The statistics here are old. The expected value of the maximum of several noisy draws sits above the expected value of any single draw, and the gap grows with both the noise in each measurement and the number of ideas compared. Test more ideas or shorten the test windows and you get more of this false edge, without a single wrong number anywhere. Auction theorists named it the winner’s curse, after the finding that the company that wins the bidding for an oilfield of uncertain value is usually the one that most overestimated it. The same holds for the idea that wins a comparison.

Every test in this example was clean. The bias comes in at the moment of choosing, which feels like nothing more than reading the top line of a results table. The arc 1 campaign chose its winner from nine ideas, so its winner’s measured edge is, by construction, a little optimistic, and walk-forward testing cannot fix that.

The holdout

The fix is old and a little painful. Keep some data out of every selection decision, and use it exactly once. Clinical trials have worked this way for decades. My part was to sit down one evening, design the trading-program version, write the protocol into the ledger, and then follow it. I enjoyed the design work. The hard part is writing a rule strict enough that I can’t argue my way around it later.

In my program this took the form of a fixed date. I chose April 2026 as the boundary, and everything after it was excluded from every adjudication decision the program makes, down to the tie-breaks. Design docs may not cite the frozen months, a “quick sanity look” is not permitted, and the trial ledger records that nothing has read them for the purpose of choosing anything.

Lookout, the intraday ensemble this notebook is mostly about, keeps walking forward through the frozen period along with its challengers. The daily walk-forward process continues to train, test, and write grades into the ledgers for those months, so the pipelines keep running, the data stays current, and the eventual reading will reflect the system operating normally. The scores exist on disk. The rule is that no decision is allowed to depend on them, so the freeze is mostly a rule about my own behavior. Knowing the numbers are sitting there, computed and unread, is hard to ignore. Those files hold the answer to whether the whole campaign was real, and I had scheduled the reading for two days later.

One reading, no second attempt

The frozen period gets read at exactly one scheduled event, a final confirmation. The finished, chosen package (the incumbent model plus every adopted change from the campaign, exactly as it would trade) is graded on the frozen months once, with no second attempt.

If a disappointing first reading could be followed by a tweak and a retest, the frozen data would start being used for selection, and the winner’s curse would be back. So I fixed the terms in advance. One reading, of one pre-specified package, and the result publishes either way. If the frozen months come back consistent with the walk-forward results, that is meaningful evidence the campaign’s verdicts were real, because this data played no part in producing them. If they come back badly, then I have learned that my selection process manufactured some of its apparent edge, which is painful and much cheaper to learn here than in the market. Either way the program ends up with a confirmed package or a known flaw, and I found I was looking forward to the reading.

This holdout’s verdict is a few paragraphs down. Subscribe to see the next one when it is read.

The freeze cost three months of the newest, most relevant data, held out of every decision during the stretch when I was making the most decisions. I set the date on the morning of July 10, before any of the round’s verdicts, and read the sealed months on the evening of July

  1. Two days sounds short for a holdout, but those two days held every

decision of the campaign (the nine verdicts, the tie-break and the ensemble’s adoption), all made without the newest three months. The same freeze procedure now applies to every later campaign.

The reading

I read it on a Sunday evening in July. I was nervous, because three months of held-out data could easily have shown the campaign was wrong. The results held.

Each of the four Lookout variants scored higher on the frozen quarter than it had over the sixty windows before the boundary, with every confidence interval sitting entirely above zero. The deltas were +0.033, +0.025, +0.035, and +0.027. The frozen months were evidently an easier market for the model, so I didn’t read much into the levels. The more useful number is the ensemble’s premium over its best single member, which came out to +0.0288 on the frozen data. The value adopted before the freeze, measured on data the selection process had already used, was +0.0287. That frozen figure comes from only the five frozen windows with saved checkpoints, and it has no interval, so the match to the fourth decimal is partly luck. It didn’t shrink. For scale, a premium of that size works out to roughly three extra correct calls in every hundred of the model’s most confident picks.

This holdout is now used up, since it only works once, and a new one starts accumulating for whatever package needs judging next. The model starts the paper diary tomorrow with that offline evidence behind it.

October update: the offline evidence held, but the model has not made money. The paper record that model started is The Graded Tape, which grades its signals every morning at fixed paper sizes; through Oct 21, 2026, 63 trading days and 11,888 signals graded, it shows a 34.0% hit rate and -$1,348.45.


Lab notebook, education and research purposes only. Not investment advice, not a solicitation; no advisory relationship is created by reading this. I trade a small account and describe my own process, errors included.

Paid subscribers get the build notes below, covering the ten lines that set the date, the script that read it once, and the one detail that didn’t line up.

How I built it

The freeze is a few lines of code and one paragraph. On the morning of Friday, July 10, 2026, I set the date at April 10, and the agent committed a ten-line paragraph to the design doc saying every window tested on or after that date is out of every decision, for every variant. The adjudication script carries the date as a constant and drops any window whose test days touch it. It ran at 10:48 that night.

User's avatar

Continue reading this post for free, courtesy of Verbatim.

Or purchase a paid subscription.
© 2026 Verbatim · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for writing, podcasting, and media founders.