Developer Testing

This guide covers the runtest.sh script used for running tests during NVIDIA FLARE development.

Quick Start 

Run the full test suite:

./runtest.sh

Default behavior (no arguments): Runs the complete CI-equivalent test suite:

License header check
Code style check (black, isort, flake8)
Auto-fix formatting
Unit tests with coverage reporting

Individual Commands 

Run specific tests individually:

./runtest.sh -u    # Unit tests only (faster, no style checks)
./runtest.sh -s    # Check code formatting only
./runtest.sh -f    # Fix code formatting only
./runtest.sh -n    # Notebook tests
./runtest.sh -l    # License header check only

Note

./runtest.sh runs the full suite (license + style + unit tests). Use ./runtest.sh -u for faster iteration when you only need unit tests.

Available Commands 

Option	Description
`-u` / `--unit-tests`	Run unit tests with pytest
`-s` / `--check-format`	Check code formatting (black, isort, flake8)
`-f` / `--fix-format`	Auto-fix code formatting issues
`-n` / `--notebook`	Run notebook tests using nbmake (see Notebook Testing)
`-l` / `--check-license`	Check license headers in source files
`-c` / `--coverage`	Enable coverage reporting (use with `-u`)
`-r` / `--test-report`	Generate JUnit XML test report (use with `-u`)
`--clean`	Clean build artifacts

Common Options 

These options can be used with any test command:

Option	Default	Description
`--numprocesses=<N\|auto>`	8	Number of parallel pytest workers (default: 8). Use `auto` to match available CPU cores.
`-d` / `--dry-run`	off	Print commands without executing

Examples 

Unit Tests 

# Run all unit tests
./runtest.sh -u

# Run specific test file or directory
./runtest.sh -u tests/unit_test/fuel/

# Run with coverage report
./runtest.sh -u -c

# Run with coverage and test report
./runtest.sh -u -c -r

# Limit parallelism (e.g. for CI)
./runtest.sh -u --numprocesses=4

Code Quality 

# Check formatting (doesn't modify files)
./runtest.sh -s

# Auto-fix formatting issues
./runtest.sh -f

# Check specific directory
./runtest.sh -s nvflare/apis/

Notebook Tests 

See Notebook Testing for detailed notebook testing options.

# Test default notebook
./runtest.sh -n

# Test specific notebook with verbose output
./runtest.sh -n -v examples/tutorials/flare_simulator.ipynb

Troubleshooting 

Force Clean State 

To start fresh:

./runtest.sh --clean
./runtest.sh -u  # Will reinstall dependencies