Skip to content

adding-notebooks.md

Latest commit

 

History

History
271 lines (221 loc) · 13.5 KB

adding-notebooks.md

File metadata and controls

271 lines (221 loc) · 13.5 KB
Raw

Adding a notebook: CI, Colab, and the exclusion lists

This guide explains what happens to a notebook after you open a Pull Request — how it is tested, how it is made runnable in Google Colab, and the three .github/*.txt lists that control that behavior. For the basic file layout, see the README; this doc picks up where that leaves off.

TL;DR — checklist for a new notebook

  • Place it under <dandiset id>/<org or lab>/<mnemonic>/ with a README.md and a requirements.in listing the notebook's direct dependencies (see main README).
  • Run python .github/scripts/lock_notebook.py <notebook>.ipynb to generate the Colab-bootstrap cells (badge → install intro → pinned install cell → restart admonition). See generating the install cell.
  • Stream data directly from the DANDI Archive (remfile/fsspec) — don't download large files or hardcode local paths. See streaming data.
  • Make sure it runs headlessly — no fig.show() on a default plotly renderer, no cv2.imshow, no %matplotlib widget, no input(). See headless gotchas.
  • Open the PR. CI runs the changed notebook end-to-end. Green = good.
  • If it genuinely can't run in CI (or shouldn't get a Colab button), add it to the right exclusion list with a one-line reason.

How CI tests a notebook

Two workflows execute notebooks:

Workflow Trigger Scope
test-changed-notebooks.yml PR touching **.ipynb (or the test harness) Only the notebooks added/modified in the PR
test-all-notebooks-weekly.yml Mondays 06:00 UTC + manual dispatch Every testable notebook; opens an issue on failure

Both call .github/scripts/run_notebook.py on an ubuntu-latest runner with Python 3.13. That script:

  1. Finds the install cell (the first code cell containing !uv pip install --system) and extracts the pinned package list from it.
  2. Installs those exact pins with uv pip install --system.
  3. Runs any !curl/!wget helper-file fetches from that cell.
  4. Stubs out the install cell, converts the notebook to a script with nbconvert --to script, and runs it under ipython (sidestepping the flaky ZMQ kernel).
  5. Exits non-zero — failing CI — on any install error, helper-fetch failure, or notebook exception.

Key consequence: CI is a fresh-env test. It installs only what the install cell pins — there is no preinstalled safety net. If the notebook imports something the install cell doesn't pin, CI fails (even if it "works on Colab," where that package happens to be preinstalled).

For pull requests, CI also executes each changed notebook through a real Jupyter kernel and publishes the executed copy, outputs included, to the PR's preview site, linking it from a checklist comment on the PR so reviewers can read the rendered results without running anything. The checklist lists every notebook under test from the start and each entry turns into a link as soon as that notebook finishes, so a large PR can be reviewed notebook by notebook while the rest are still running. The executed copies live only in the preview and are removed when the PR closes; nothing is committed to the branch. (PRs from forks get the executed notebooks as workflow artifacts instead, since fork workflows cannot deploy the preview.)

test-changed-notebooks.yml only tests notebooks changed in the PR. A notebook already on master that would fail today is not re-run until it is touched again (or until the weekly sweep catches it). Don't assume "it's on master, so it's green."

A brand-new notebook without an install cell fails by design — CI can't know what to install. Add the bootstrap, or add the notebook to an exclusion list with a reason.

The Colab bootstrap cells

Every testable notebook begins with four cells (the pattern established in PR #149, generated by lock_notebook.py — see below):

  1. Colab badge (markdown): 
    [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dandi/example-notebooks/blob/master/<path-to-notebook>.ipynb)
  2. Install intro (markdown): one paragraph explaining the install cell.
  3. Install cell (code) — the one CI keys off: 
    #@title Installing requirements (click ▶ to run) { display-mode: "form" }
# Colab provides Python 3.13. We install with `uv --system` because Colab's
# kernel runs outside a virtualenv. All versions (direct + transitive) are
# pinned below so the notebook is reproducible regardless of resolver drift.
!pip install -q uv
!uv pip install --system \
    "package-a==1.2.3" \
    "package-b==4.5.6" \
    ...
    The { display-mode: "form" } collapses the cell to a one-line bar in Colab so users aren't faced with ~100 lines of pins. If the notebook needs a colocated helper .py, fetch it at the end of this cell: !curl -sL -o helper.py https://raw.githubusercontent.com/dandi/example-notebooks/master/<path>/helper.py
  4. Restart admonition (markdown): tells the user to restart the runtime after install, because upgrading C-extension packages mid-session requires it.

Why fully pinned?

Pinning all transitive deps (not just direct ones) makes the notebook reproducible forever — it can't break later when an upstream release changes a default. It's also what CI installs, so a green CI run means the exact pinned set works on Python 3.13 / linux.

Generating the install cell

Commit a requirements.in next to your notebook listing only its direct dependencies, one per line (e.g. dandi, pynwb, remfile, matplotlib, pandas) — not a full freeze, and not transitive packages. Then run:

python .github/scripts/lock_notebook.py <path/to/notebook>.ipynb

The script resolves a full, pinned lock with uv, constrained to Colab's preinstalled versions so the install is a no-op for packages Colab already ships (faster, and avoids the "RESTART RUNTIME" prompt that changing numpy and other C-extensions triggers), and writes the four bootstrap cells into the notebook — prepending them when absent, or refreshing the pin block in place (existing !curl/!wget helper lines are preserved). Under the hood it runs:

uv pip compile requirements.in \
    --python-version 3.13 \
    --python-platform linux \
    --constraint .github/colab-preinstalled.txt

Conventions:

  • One requirements.in per directory, shared by all its notebooks, unless their needs genuinely differ. Run lock_notebook.py on each notebook after editing it. Notebooks locked from the same file get identical pin sets, so they are also published together in one container image.
  • A notebook that needs a different dependency set can have its own <notebook-stem>.requirements.in, which takes precedence over the shared file.
  • The .in file is committed: it is the source of truth for re-locking when Colab bumps its runtime, a pin breaks, or the periodic image rebuild needs a fresh resolve.
  • If the resolver floats to a version newer than what the notebook was written against (e.g. a matplotlib release removing a kwarg the notebook uses), add an upper bound in requirements.in (matplotlib<3.11) and re-run the script.

.github/colab-preinstalled.txt is a pip-freeze of the current Colab Python 3.13 runtime (numpy 2.1.3, etc.). Using it as a constraint keeps your pins aligned with Colab. The constraint is hard: a package Colab preinstalls is pinned to Colab's version, and a requirement that conflicts with it fails to resolve. When a notebook genuinely needs a different version of one of those packages, declare an override in requirements.in:

# override: click<8.2

The override replaces Colab's pin for that package alone. Colab then downgrades it in the install cell and prompts for a runtime restart, which is the correct trade-off. Use overrides sparingly and only for the package that actually conflicts.

nbformat gotcha: cell id fields require nbformat_minor >= 5. lock_notebook.py bumps this automatically when it prepends cells; only hand-built bootstrap cells need a manual bump.

Stream data from the DANDI Archive

Notebooks should stream their data directly from the DANDI Archive rather than downloading files or reading from a local path. Streaming reads only the bytes a cell actually needs, so the notebook runs unattended in CI and in Colab with no manual data setup, no multi-GB download, and no machine-specific paths.

Resolve the asset's S3 URL through the DANDI API and open it with remfile (or fsspec):

from dandi.dandiapi import DandiAPIClient
import remfile, h5py
from pynwb import NWBHDF5IO

with DandiAPIClient() as client:
    asset = client.get_dandiset("000000", "draft").get_asset_by_path(
        "sub-XX/sub-XX_ecephys.nwb"
    )
    s3_url = asset.get_content_url(follow_redirects=1, strip_query=False)

io = NWBHDF5IO(file=h5py.File(remfile.File(s3_url), "r"), load_namespaces=True)
nwbfile = io.read()

Guidelines:

  • Don't dandi download a dandiset, read ../path/to/local.nwb, or assume a file exists on disk — that fails in CI and gives Colab users a broken notebook. (Hardcoded local paths are a common reason a notebook lands on the exclusion lists.)
  • Do wrap the remote file in a cache so re-runs don't re-fetch: remfile.File(s3_url, disk_cache=remfile.DiskCache("nwb-cache")).
  • Pin remfile (and/or fsspec/s3fs) in the install cell.
  • For Zarr-based assets, stream with fsspec instead of remfile.

Headless gotchas

CI runs notebooks with no display and no browser. Most plotting is fine; a few patterns are not. Before opening a PR, make sure none of these slip in:

Pattern Headless? Notes
plt.show() (matplotlib) ✅ fine Falls back to the Agg backend.
IPython.display.IFrame(...) ✅ fine Produces an inert HTML repr; never fetches or opens anything.
fig.show() (plotly) ❌ fails Default renderer is "browser" → calls webbrowser.open() → Error: could not locate runnable browser. Set a notebook renderer once near the top: import plotly.io as pio; pio.renderers.default = "iframe" (renders in Colab, safe headless).
cv2.imshow / cv2.namedWindow ❌ fails Needs a GUI window.
%matplotlib notebook / widget / tk / qt ❌ fails Interactive backends need a frontend. Use %matplotlib inline.
input() / getpass() ❌ hangs No stdin in CI.
webbrowser.open(...) ❌ fails No browser on the runner.

The plotly case is the most common surprise: a notebook works in Colab (which has a notebook renderer + preinstalled plotly) but fails in CI. The fix is the one-line pio.renderers.default setting above — keep it in CI and Colab.

The three .github lists

These three files control different things and are independent of each other. Add a notebook to whichever applies, always with a one-line reason.

colab-preinstalled.txt

.github/colab-preinstalled.txt — not an exclusion list. It's a snapshot of Colab's preinstalled package versions, used as the uv pip compile --constraint when generating install-cell pins (see above). Refresh it when Colab bumps its runtime.

notebook-test-exclusions.txt → "skip in CI"

.github/notebook-test-exclusions.txt — notebooks the CI test sweep should skip. Add a notebook here when it can't run cleanly on a headless CI runner: needs a database/credentials, hits a headless-incompatible API the slim runner image lacks, a pre-existing content bug, or an upstream break not yet fixed. Lines are fnmatch globs relative to the repo root. Removing the line re-enables testing once the issue is fixed.

notebook-colab-exclusions.txt → "no Colab button"

.github/notebook-colab-exclusions.txt — notebooks that should not show an "Open in Colab" button on the index page (consumed by collect_and_render.py). Add a notebook here when clicking "Open in Colab" would give a broken experience even on a fresh Colab runtime: stale data paths, missing creds, removed upstream APIs, wheels lacking a needed feature, or runtimes too long for a tutorial.

They are independent

A notebook can be on one list but not the other:

  • Test-excluded but Colab-OK: read_avi.ipynb uses OpenCV's libxcb, which the slim CI image lacks but Colab has → on test-exclusions, not on colab-exclusions.
  • Colab-excluded but test-OK: rare, but e.g. a notebook that runs in CI yet points users at data they can't access interactively.
  • Both: the DataJoint examples need a MySQL server that neither CI nor Colab provides → on both lists.

Where to get help

Reach out on the DANDI helpdesk.