<a id="developer-testing"></a>

# Developer Testing

This guide covers the `runtest.sh` script used for running tests during NVIDIA FLARE development.

> ##### Table of Contents
> 
> * [Quick Start](#quick-start)
>   * [Individual Commands](#individual-commands)
> * [Available Commands](#available-commands)
> * [Common Options](#common-options)
> * [Examples](#examples)
>   * [Unit Tests](#unit-tests)
>   * [Code Quality](#code-quality)
>   * [Notebook Tests](#notebook-tests)
> * [Troubleshooting](#troubleshooting)
>   * [Force Clean State](#force-clean-state)

## Quick Start

Run the full test suite:

```bash
./runtest.sh
```

**Default behavior** (no arguments): Runs the complete CI-equivalent test suite:

1. License header check
2. Code style check (black, isort, flake8)
3. Auto-fix formatting
4. Unit tests with coverage reporting

### Individual Commands

Run specific tests individually:

```bash
./runtest.sh -u    # Unit tests only (faster, no style checks)
./runtest.sh -s    # Check code formatting only
./runtest.sh -f    # Fix code formatting only
./runtest.sh -n    # Notebook tests
./runtest.sh -l    # License header check only
```

#### NOTE
`./runtest.sh` runs the **full suite** (license + style + unit tests).
Use `./runtest.sh -u` for **faster iteration** when you only need unit tests.

## Available Commands

| Option                   | Description                                                                                                            |
|--------------------------|------------------------------------------------------------------------------------------------------------------------|
| `-u` / `--unit-tests`    | Run unit tests with pytest                                                                                             |
| `-s` / `--check-format`  | Check code formatting (black, isort, flake8)                                                                           |
| `-f` / `--fix-format`    | Auto-fix code formatting issues                                                                                        |
| `-n` / `--notebook`      | Run notebook tests using nbmake (see [Notebook Testing](notebook_testing.md#notebook-testing)) |
| `-l` / `--check-license` | Check license headers in source files                                                                                  |
| `-c` / `--coverage`      | Enable coverage reporting (use with `-u`)                                                                              |
| `-r` / `--test-report`   | Generate JUnit XML test report (use with `-u`)                                                                         |
| `--clean`                | Clean build artifacts                                                                                                  |

## Common Options

These options can be used with any test command:

| Option                    | Default   | Description                                                                              |
|---------------------------|-----------|------------------------------------------------------------------------------------------|
| `--numprocesses=<N|auto>` | 8         | Number of parallel pytest workers (default: 8). Use `auto` to match available CPU cores. |
| `-d` / `--dry-run`        | off       | Print commands without executing                                                         |

## Examples

### Unit Tests

```bash
# Run all unit tests
./runtest.sh -u

# Run specific test file or directory
./runtest.sh -u tests/unit_test/fuel/

# Run with coverage report
./runtest.sh -u -c

# Run with coverage and test report
./runtest.sh -u -c -r

# Limit parallelism (e.g. for CI)
./runtest.sh -u --numprocesses=4
```

### Code Quality

```bash
# Check formatting (doesn't modify files)
./runtest.sh -s

# Auto-fix formatting issues
./runtest.sh -f

# Check specific directory
./runtest.sh -s nvflare/apis/
```

### Notebook Tests

See [Notebook Testing](notebook_testing.md#notebook-testing) for detailed notebook testing options.

```bash
# Test default notebook
./runtest.sh -n

# Test specific notebook with verbose output
./runtest.sh -n -v examples/tutorials/flare_simulator.ipynb
```

## Troubleshooting

### Force Clean State

To start fresh:

```bash
./runtest.sh --clean
./runtest.sh -u  # Will reinstall dependencies
```