Notebook Testing

NVIDIA FLARE uses nbmake to test Jupyter notebooks. This ensures that example notebooks remain functional as the codebase evolves.

Note

Not all notebooks are ready for automated testing yet. Some notebooks require external infrastructure (running FLARE servers, provisioned environments, specific datasets) or contain interactive elements that cannot run in CI. Notebook test coverage is being improved over time.

For general runtest.sh usage (dependency caching, verbose mode, etc.), see Developer Testing.

Quick Start

Use the runtest.sh script to run notebook tests:

# Test default notebook (flare_simulator.ipynb)
./runtest.sh -n

# Test a specific notebook
./runtest.sh -n examples/tutorials/flare_simulator.ipynb

# Test with verbose output
./runtest.sh -n -v examples/tutorials/flare_simulator.ipynb

Notebook-Specific Options

These options are specific to notebook testing (-n):

Argument

Default

Description

--timeout=SECONDS

1200

Timeout in seconds for each notebook execution

--nb-clean=MODE

on-success

When to clean outputs: always, on-success, never

--kernel=NAME

python3

Jupyter kernel name (defaults to python3 if available)

-v / --verbose

off

Pass -v to pytest for verbose output

Examples

# Set a shorter timeout (5 minutes)
./runtest.sh -n --timeout=300 examples/tutorials/flare_simulator.ipynb

# Use a specific kernel
./runtest.sh -n --kernel=python3 examples/tutorials/flare_simulator.ipynb

# Always clean outputs regardless of pass/fail
./runtest.sh -n --nb-clean=always examples/tutorials/

# Combine multiple options with verbose output
./runtest.sh -n -v --timeout=1800 --kernel=python3 examples/tutorials/

Direct pytest Usage

You can also run nbmake directly with pytest:

pytest --nbmake --nbmake-timeout=1200 --nbmake-clean=on-success examples/tutorials/

# With specific kernel
pytest --nbmake --nbmake-timeout=1200 --kernel=python3 examples/tutorials/

Skipping Cells in Notebooks

To skip specific cells during automated testing (e.g., Colab setup cells, interactive visualizations, or cells that require user input), add one of these tags to the cell metadata:

  • skip-execution

  • skip

  • colab

Adding Tags in Jupyter

In Jupyter Lab:

  1. Select the cell you want to skip

  2. Click the gear icon in the right sidebar (or View → Right Sidebar → Show Property Inspector)

  3. Under “Common Tools” → “Cell Tags”, add: skip-execution

In Jupyter Notebook (classic):

  1. Select the cell

  2. View → Cell Toolbar → Tags

  3. Add tag: skip-execution

In VS Code:

  1. Click on the cell

  2. Click “…” menu on the cell

  3. Select “Add Cell Tag”

  4. Enter: skip-execution

How It Works

The testing framework (implemented in conftest.py) automatically:

  1. Before test: Creates a .backup of the original notebook

  2. Filters cells: Removes cells tagged with skip-execution, skip, or colab

  3. Updates kernel: Adjusts kernel spec to match the specified or detected kernel

  4. Executes: nbmake runs the notebook through the Jupyter kernel

  5. Restores: Original notebook is restored from backup

  6. Cleans outputs: Cell outputs are cleared based on --nbmake-clean setting

This ensures:

  • The test does not modify the committed notebook (original is restored from backup)

  • Notebooks can contain Colab-specific or interactive cells that won’t break CI

  • Consistent kernel usage across different development environments

Troubleshooting

Kernel Not Found

If you see “Kernel not found” errors:

  1. Ensure your virtual environment is activated

  2. Install ipykernel: pip install ipykernel

  3. Register your kernel: python -m ipykernel install --user --name=my_env

  4. Or specify an existing kernel: ./runtest.sh -n --kernel=python3

Timeout Errors

For long-running notebooks, increase the timeout:

./runtest.sh -n --timeout=3600 examples/advanced/

Notebook Requires External Infrastructure

Some notebooks (e.g., flare_api.ipynb) require a running FLARE server or provisioned environment. These notebooks will fail in automated testing unless the infrastructure is set up.

For such notebooks, consider:

  1. Running them manually in an interactive environment

  2. Adding skip-execution tags to cells that require external services

  3. Creating a simplified version for automated testing

Best Practices

  1. Tag cells appropriately: Mark Colab setup, interactive widgets, and user-input cells with skip-execution

  2. Keep notebooks focused: Smaller notebooks are faster to test and easier to debug

  3. Use reasonable timeouts: Increase --timeout based on expected execution time plus buffer

  4. Test locally before pushing: Run ./runtest.sh -n on your notebooks before committing

  5. Clean outputs before committing: Use --nb-clean=always or manually clear outputs to keep git diffs clean

  6. Use self-contained examples: Notebooks that use the simulator (like flare_simulator.ipynb) are easier to test than those requiring external servers