Research notebooks & reproducibility

~8 min read

Interview occasional

Why it matters

A result nobody else can rerun is a story about a laptop, not a finding. Interviews ask less about notebook fashion and more about whether you can say which data snapshot, which code, and which parameters produced the table in the memo. Notebooks are the usual place that chain breaks: cells run out of order, a global got overwritten, and the chart on the screen is not the chart the file would produce today.

Core concepts

Reproducible means the same numbers from the same inputs. “A similar Sharpe” is not a reproduction. A colleague, or you in three months, should be able to start from a pinned dataset and a pinned revision and regenerate the table. If the path includes an untracked spreadsheet edit, you don’t have that.
Notebooks are for exploration. A rerunnable path is for the claim. Interactive cells are a good way to check a join or plot a residual. They are a bad system of record if the result depends on execution order, on a variable left over from a deleted cell, or on a file that was regenerated by hand. Joel Grus’s objection is practical: hidden state. Use the notebook to discover the check. Then make the check a top-to-bottom script, or a notebook that restarts clean and runs in order.
Pin three things: data, code, parameters. Data: the vendor extract or the query, with the knowledge date, not “whatever the API returned this morning.” Code: a revision id. Parameters: the horizon, lag, cost assumption, and universe rule, in a config the report prints. A chart with none of those in the footer cannot be audited.
Split the pipeline where the bugs are. Separate pulls and cleaning, signal construction, portfolio and backtest, and the report. When the Sharpe changes, you want to know which stage changed. Mixing a SQL pull, a scaler, and a plot in one cell makes that guesswork.
Point-in-time has to be in the data pin, not only in the prose. A “reproducible” notebook that queries latest fundamentals will reproduce the lookahead faithfully. See Data hygiene. Re-running is not the same as being honest about what was knowable.
Randomness and environments are part of the result. A seed, a library version, and a dependency list matter once anything is stochastic or once a library changes a default. You do not need a heavy platform. You need the difference between “it ran on my machine” and a one-line recipe that installs the same stack.
The research log is the multiple-testing record. Notebooks tempt you to keep the last cell that worked. A short log of specs you tried — even a dated list — is what makes the final test interpretable. See Multiple testing. Reproducibility of the winner without a record of the losers is half the job.

Mental model

pinned raw data (as-of / knowledge date)
    -> clean, tested, rerunnable
    -> signal parameters in a config
    -> backtest
    -> report that prints data id, code id, and parameters

If any arrow is “open the notebook and run the cells you remember,” the arrow is not an arrow.

Interview questions

You rerun a teammate’s notebook and the Sharpe moves from 1.4 to 0.6. What are the first three things you check? Answer: Whether cells were run top to bottom from a fresh kernel (hidden state), whether the data pull is “latest” rather than a pinned extract (fundamentals and universe may have changed), and whether the parameters in the notebook match the memo. Library versions are next if those three match.
Why is “the notebook is in git” not enough? Answer: Git pins the code you committed, not the data, not the uncommitted cell outputs you presented, and not the order you executed cells before committing. A clean rerun from the pinned inputs is the test. The filename in the repo is not.
Where should the reporting lag and the cost assumption live so a reviewer can see them? Answer: In a config the report prints next to the result, not in a paragraph that can drift from the code. If the lag is a magic number inside a cell, the memo and the backtest will disagree and nobody will notice.
A researcher iterates in a notebook, then copies the winning snippet into a script. What is still missing for a claim about out-of-sample performance? Answer: A record of the iterations that were not copied. The script reproduces the winner. It does not reproduce the search. The trial count, or a truly untouched holdout, still has to be written down. See Backtest design.

Watch

Quantopian Lecture Series: Introduction to Research — Quantopian. The notebook as a research environment: small, concrete, and the right place to see the habit.
I don’t like notebooks — Joel Grus, JupyterCon 2018. The case against hidden state and out-of-order execution, from a practitioner who still has to ship code.