Verbatim

Verbatim

Pre-registration for traders

The one-page design doc that decides a verdict before the results exist, field by field, free to copy.

Verbatim's avatar
Verbatim
Nov 03, 2026
∙ Paid

Written in July 2026. Anything that has changed since then is marked with an October update.

Every verdict in this notebook, the wins and the losses, was decided by a document written before the results existed. I call these documents pre-registrations, a practice from clinical trials and from psychology after its replication crisis. The figure below is a real design doc from my program, with the proprietary details removed. The structure is the useful part, and you are free to copy it.

Prove me wrong, bro

A design doc opens with the hypothesis, and it has to be a single sentence that can be proven wrong. For example, “this specific change, evaluated on these specific windows, will beat the current model by at least this much.” If I cannot state an idea in a form that a number could contradict, it is still a hope, and no result can disprove it. I drop plenty of ideas while writing that sentence, hours before any compute gets rented.

The bar, chosen while you are still impartial

Next comes the success bar, and it gets chosen before any results exist. In my program the standing bar is that a challenger must beat the incumbent by more than +0.02 on the evaluation metric, measured across matched walk-forward windows, with a bootstrap confidence interval clear of zero. The margin exists because a new idea adds complexity, and a challenger that ties the incumbent while being more complex is the worse system. The confidence interval exists because without it, luck alone can clear any bar I set.

The interval comes from a paired bootstrap. Both models are graded on the identical walk-forward windows, so each window yields one number, the challenger’s score minus the incumbent’s on that window. Pairing this way removes the effect of the window itself. Whatever market conditions that test period contained, both models faced them, so they cancel in the difference. The bootstrap then resamples that list of per-window differences with replacement a thousand times, computing the mean difference each time, and the middle 95 percent of those thousand means becomes the interval. So the challenger must beat the incumbent across a thousand reshuffles of the same test history before the verdict counts. The resampler’s random seed is fixed, so anyone replaying the experiment, including a suspicious future me, gets the identical interval to the last digit.

I choose the bar early because once results arrive, I am no longer impartial. I start arguing for whichever outcome I was hoping to see. Choosing the bar first means I set it before I know which side I’m on.

The cost

Every doc states what the experiment will cost to run, in dollars of rented compute. fWith the cost set in advance, a failed experiment is money I already agreed to spend for the answer. It also forces me to rank which questions to ask first, because the budget is finite.

Ties go to the incumbent

This line is standing policy and never negotiated per experiment. When a challenger and the incumbent are statistically indistinguishable, the incumbent wins. New ideas carry the burden of proof, and the older, simpler model stays until something measurably better shows up. This rule has settled more of my close calls than any other clause in the template.

The trial ledger

Every experiment, the moment its design doc is signed, adds one to a count in a ledger, and that count appears in this notebook. Nine experiments in arc 1 means nine ledger entries, including the seven or so I would have preferred to forget.

The ledger exists because of the multiple-comparisons trap, the way backtests mislead even careful people. Run twenty independent tests of worthless ideas at a conventional significance threshold and you should expect roughly one to pass by pure chance. If you then report only the one that passed, you have reported a false discovery without changing a single number. The problem is the count of attempts you left out, the denominator. The ledger keeps that count where I cannot leave it out, so any win I report can be read against the full number of attempts behind it.

The trial ledger is public, so every win I report comes with its count of attempts. Subscribe and hold me to it.

The sweep rule

This is the rule I would have broken first. When an experiment fails, the temptation is to adjust a parameter and run it again, and again, until something clears the bar. So the template carries an explicit rule that a parameter sweep following a failure is a new experiment, requiring its own design doc, its own pre-registered bar, and its own entry in the trial ledger. Retries are allowed, and every one gets recorded, because twenty hidden retries carry the same odds as twenty hidden experiments.

The template fits on one page and needs no mathematics beyond what your backtesting already uses. The hard part is following rules you wrote before you saw the results, which is harder on results day than it sounds in July. Every idea in this program now goes through the same page. The next letter is about a results day when the document said no.

October update: for the second model, the one that trades once a day and is in research now, the document comes at the end. Its exploratory studies are logged and counted as they run, and any configuration that becomes a candidate gets one sealed, pre-registered test on data it has never seen, with the count of configurations searched to find it written into the document.


Lab notebook, education and research purposes only. Not investment advice, not a solicitation; no advisory relationship is created by reading this. I trade a small account and describe my own process, errors included.

Paid subscribers get the build notes below, covering where the template came from and the plain text file that counts every try.

How I built it

The template grew out of the first few design docs. A typical one from July is 61 lines under four headings (the hypothesis, the fixed inputs, the design, and what kills it). Its header carries the price (about $45 to $55 for a 10 to 12 hour walk) and a status line that read AWAITING SIGN-OFF until I signed it. The agent drafts, I sign, then it runs. The sweep rule started as a single parenthetical in a July 9 doc.

User's avatar

Continue reading this post for free, courtesy of Verbatim.

Or purchase a paid subscription.
© 2026 Verbatim · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for writing, podcasting, and media founders.