nvflare.app_opt.pt.recipes.scaffold module

class ScaffoldRecipe(*, name: str = 'scaffold', model: Any | dict[str, Any] | None = None, initial_ckpt: str | None = None, min_clients: int, num_rounds: int = 2, train_script: str, train_args: str = '', launch_external_process: bool = False, command: str = 'python3 -u', server_expected_format: ExchangeFormat = ExchangeFormat.NUMPY, params_transfer_type: TransferType = TransferType.FULL, server_memory_gc_rounds: int = 0, client_memory_gc_rounds: int = 0, cuda_empty_cache: bool = False, enable_tensor_disk_offload: bool = False)[source]

Bases: Recipe

A recipe for implementing Scaffold in NVFlare.

Recipe parameters, including train_args, become part of the generated job definition and must never contain actual secret values. Read secrets from site environment variables or mounted files; references are supported only where documented in nvflare.recipe.secrets.

Implements the training algorithm proposed in Karimireddy et al. “SCAFFOLD: Stochastic Controlled Averaging for Federated Learning” (https://arxiv.org/abs/1910.06378).

Client script requirement: Unlike FedAvgRecipe, the client script must use PTScaffoldHelper (nvflare.app_opt.pt.scaffold): call init(model), model_update() during training, terms_update() after training, and include meta[AlgorithmConstants.SCAFFOLD_CTRL_DIFF] = scaffold_helper.get_delta_controls() in the FLModel sent back to the server. A standard flare.receive/send loop without PTScaffoldHelper will cause server-side aggregation to fail.

This recipe sets up a complete federated learning workflow with Scaffold controller.

Parameters:

  • name – Name of the federated learning job. Defaults to “scaffold”.

  • model – Initial model to start federated training with. Can be: - nn.Module instance - Dict config: {“class_path”: “module.ClassName”, “args”: {“param”: value}} - None: no initial model

  • initial_ckpt – Absolute path to a pre-trained checkpoint file. The file may not exist locally as it could be on the server. Used to load initial weights. Note: PyTorch requires model when using initial_ckpt (for architecture).

  • min_clients – Minimum number of clients required to start a training round. Defaults to 2.

  • num_rounds – Number of federated training rounds to execute. Defaults to 2.

  • train_script – Path to the training script that will be executed on each client. Defaults to “client.py”.

  • train_args – Command line arguments to pass to the training script. Defaults to “”.

  • server_memory_gc_rounds – Run memory cleanup (gc.collect + malloc_trim) every N rounds on server. Set to 0 to disable. Defaults to 0.

  • enable_tensor_disk_offload (bool) – Download streamed PyTorch tensors to disk on the server during receive/aggregation instead of holding them in memory, reducing server memory pressure for large models. Requires server_expected_format=ExchangeFormat.PYTORCH. Defaults to False.

Example

```python recipe = ScaffoldRecipe(

name=”my_scaffold_job”, model=pretrained_model, min_clients=2, num_rounds=10, train_script=”client.py”, train_args=”–epochs 5 –batch_size 32”

)

This is base class of a recipe. Recipes are implemented by jobs. A concrete recipe must provide the job for recipe implementation.

Security contract – no secrets in recipe parameters:

Recipe parameters (train_args, task_args, eval_args, per_site_config, config overrides, dicts passed to add_client_config/add_server_config, exec params, etc.) can be written in clear text into generated job configuration. These parameters and their nested values must never contain actual passwords, API keys, tokens, private keys, or other credentials. Instead, read secrets from site environment variables or mounted secret files inside your code, or pass a placeholder created with nvflare.recipe.secrets.secret_ref() or nvflare.recipe.secrets.secret_file_ref() at a supported runtime boundary. See nvflare.recipe.secrets for the supported parameter locations.

Before export or run, recipes scan their parameters with heuristics and emit nvflare.recipe.secrets.PotentialSecretWarning when a value looks like an actual secret. The scan is best-effort: absence of a warning does not prove a parameter is safe to share.

param job:

the job that implements the recipe.