Written in July 2026. Anything that has changed since then is marked with an October update.
If you have ever had a strategy look great in a backtest and then lose money in its very first live month, so have I. It took me a long time to see why. The backtest had answered a different question from the one I was asking.
New letters arrive by email, free. Subscribe and the next one comes to you.
What an ordinary backtest measures
An ordinary backtest measures how well a strategy fits the history I used while building it. The question I care about is how the strategy will do on days that have not happened yet. The two stop matching as soon as you start tuning. Every time you adjust a parameter, look at the resulting equity curve, and keep the change when the curve improves, you have used the test days to build the strategy. Do that a few dozen times, which takes one slow afternoon with a laptop, and the strategy has been fitted to the exact days it is graded on. The final backtest score then measures how well you fitted those days and says nothing about how the strategy will do next.
No single adjustment feels like cheating, because each one seems like careful work.
Sliding the window
Walk-forward evaluation is the standard fix. You pick a cutoff date and train the model only on data from before it. The tuning happens inside that same boundary. A short validation period is taken from the end of the training span, the most recent data the model is allowed to see, and every decision that shapes the model (when to stop training, which configuration to keep) uses that period and nothing newer. Most homemade test setups let future data in at this step, and a model tuned with a look at the future has seen part of the test in advance.
Then the model is graded on the days immediately after the cutoff, days it has never seen in any form. The grade is the same strict statistic from arc 1, precision among the model’s most confident names with false alarms held to a one percent budget, computed on those unseen days alone. The grade is written to the ledger, and then the whole process moves forward one day and repeats. It trains on a slightly newer past, tunes on a slightly newer validation period, is graded on a slightly newer future, and records the grade. Because the step is a single day and each window spans much longer, consecutive windows overlap heavily, by design. The same model family is retested under gradually changing conditions, and because the step is one day, no shift between market regimes can fall between two tests.
In my program this procedure produces roughly 869 overlapping windows covering January 2023 through the middle of 2026, and every grade in that walk was earned by Lookout, the intraday ensemble this notebook is mostly about, on days it had never seen. I looked for any way the model could see its test days before being graded and did not find one. The overlap means neighboring grades are correlated, because they share most of their training data, so the program never treats the 869 as independent results. Comparisons between models happen pairwise on matched windows, which the pre-registration letter covers.
Three and a half years covers several kinds of market, including the choppy, directionless stretches of 2023, the steadier market of 2024, and the rougher conditions of 2025 and early 2026. A model that only fits one kind of market shows it, because the windows run through all of them and the ledger keeps each period’s grades where you can see them. Watching those 869 grades come in, each one earned on days the model never saw, is one of the parts of this project I enjoy most.
What walk-forward costs
The first cost is compute. Instead of training a model once and grading it once, you are training and grading it hundreds of times, one fresh fit per window. Every experiment costs that many times more, and a large share of the $423 I spent on the arc 1 campaign went to this procedure. When people ask why I rent data-center GPUs to evaluate ideas for a small account, this is most of the reason. The harness, at least, was a one-time cost. It was written once, reworked at length, and every future idea now runs through it, so each new hypothesis pays for its own compute but not for the engineering.
The second cost is that the score comes out lower. A conventional backtest, tuned on the days it grades, reports flattering numbers; walk-forward reports what the model earns on unseen days, which is almost always lower. The lower number is the one to trust, and the gap between the two shows how much the ordinary backtest overstated the result. When a change to my testing procedure makes the score go up, I check the change before I believe the score.
I don’t mind the lower number, because a score earned this way is one I can rely on. When Lookout’s grade held up across 869 windows it never saw in advance, through three different kinds of market, I could treat it as a property of the system and build plans on it.
Subscribe to get each new grade as it comes in, good or bad.
What it still cannot fix
Walk-forward stops a model from being graded on the days it was tuned on. It does nothing about the person running the experiments, who tests many ideas, watches the grades come in, and then picks the luckiest one. That problem is called the winner’s curse. Clean evaluation does not remove it, so it needs a separate check. The next letter covers that check, which keeps three months of my own data off-limits until one scheduled reading.
October update: a September audit of the data pipeline found three small look-ahead leaks in the model’s feature inputs; removing them moved the grade by at most 0.0015, so the grades stand, but my claim above that I found no way for the model to see its test days was too strong. The plans built on those grades have not made money, and an October test of trading the signals through the day lost money in its main configuration even before costs. One step of that walk also skipped fifteen sessions of June 2025 that the data had silently lost; the last letter of this arc covers it.
Lab notebook, education and research purposes only. Not investment advice, not a solicitation; no advisory relationship is created by reading this. I trade a small account and describe my own process, errors included.
Paid subscribers get the build notes below, covering the 800-line runner, the one-line fix that kept it running, and what a full walk costs.
How I built it
I built the harness in one day, July 8, 2026, with Claude Code doing the typing. Before writing anything new, the agent reran my old training as a control, and my earlier results didn’t reproduce. Those bugs got fixed first.
The new runner is about 800 lines plus 50 tests, and it replaced about 2,100 lines of older scripts. Each window writes one line to a log and flushes it to disk before the next window starts, so a crash costs one window. Every window also grades four dumb baselines beside the model, including a plain logistic regression.
On the first rented H100 run, every job died after about 27 windows with “too many open files”. The fix was one line in the launch command. A full walk takes about 26 hours, roughly $40 to $60 per variant.
Try this: ask your assistant to rerun your best old backtest from a clean checkout before it builds anything new.



