Available Recipes

NVFlare provides a variety of pre-built recipes for common federated learning algorithms and workflows. Recipes are high-level, declarative APIs that simplify job configuration and execution.

Common Recipe Parameters 

Most training recipes accept the following model-related parameters:

model

The model to use for federated training. Accepts:

Class instance: e.g., MyModel() - convenient and Pythonic
Dict config: e.g., {"class_path": "module.MyModel", "args": {"param": value}} - better for large models

Note

Class instances are converted to configuration files before job submission. For large models, use dict config to avoid unnecessary instantiation overhead. For TensorFlow/Keras, class instances should be user-defined subclassed models (for example, tf.keras.Model or tf.keras.Sequential subclasses).

initial_ckpt

Absolute path to a pre-trained checkpoint file. The file may not exist locally but must exist on the server when the model is loaded during job execution.

PyTorch: Requires model for architecture (checkpoint has weights only)
TensorFlow/Keras: Can use initial_ckpt alone (Keras saves full model). If model is provided, use a subclassed Keras class instance or dict config.

See NVFlare Job Recipe for detailed explanations of these options.

Federated Averaging (FedAvg)

The most fundamental federated learning algorithm that aggregates model updates from multiple clients by computing a weighted average.

PyTorch FedAvg 

from nvflare.app_opt.pt.recipes import FedAvgRecipe
from nvflare.recipe import SimEnv

recipe = FedAvgRecipe(
    name="fedavg-pt",
    min_clients=2,
    num_rounds=5,
    model=MyModel(),
    train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

TensorFlow FedAvg 

from nvflare.app_opt.tf.recipes import FedAvgRecipe
from nvflare.recipe import SimEnv

recipe = FedAvgRecipe(
    name="fedavg-tf",
    min_clients=2,
    num_rounds=5,
    model=MyTFModel(),
    train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

Sklearn FedAvg 

For scikit-learn based models.

from nvflare.app_opt.sklearn.recipes import SklearnFedAvgRecipe
from nvflare.recipe import SimEnv

recipe = SklearnFedAvgRecipe(
    name="fedavg-sklearn",
    min_clients=2,
    num_rounds=5,
    train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/advanced/sklearn-linear

FedAvg with Homomorphic Encryption 

FedAvg with secure aggregation using homomorphic encryption.

from nvflare.app_opt.pt.recipes import FedAvgRecipeWithHE
from nvflare.recipe import SimEnv

recipe = FedAvgRecipeWithHE(
    name="fedavg-he",
    min_clients=2,
    num_rounds=5,
    model=MyModel(),
    train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/advanced/kaplan-meier-he

FedProx 

FedProx is FedAvg with a proximal term added to the client loss function to handle data heterogeneity. It uses the standard FedAvgRecipe with the FedProx loss helper on the client side.

PyTorch FedProx 

from nvflare.app_opt.pt.recipes import FedAvgRecipe
from nvflare.recipe import SimEnv

# FedProx uses FedAvgRecipe with FedProxLoss in the client training script
recipe = FedAvgRecipe(
    name="fedprox-pt",
    min_clients=2,
    num_rounds=5,
    model=MyModel(),
    train_script="client.py",
    train_args="--fedproxloss_mu 0.01",  # Pass mu parameter to client
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

In your client training script, use the FedProxLoss helper:

from nvflare.app_opt.pt import PTFedProxLoss

# In training loop:
fedprox_loss = PTFedProxLoss(mu=fedproxloss_mu)
for data, target in train_loader:
    optimizer.zero_grad()
    output = model(data)
    ce_loss = criterion(output, target)
    # Add FedProx regularization term
    prox_loss = fedprox_loss(model)
    loss = ce_loss + prox_loss
    loss.backward()
    optimizer.step()

Examples:

examples/advanced/cifar10/pt/cifar10-sim/cifar10_fedprox

TensorFlow FedProx 

from nvflare.app_opt.tf.recipes import FedAvgRecipe
from nvflare.recipe import SimEnv

recipe = FedAvgRecipe(
    name="fedprox-tf",
    min_clients=2,
    num_rounds=5,
    model=MyTFModel(),
    train_script="client.py",
    train_args="--fedproxloss_mu 0.01",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

In your client training script, use the TensorFlow FedProxLoss:

from nvflare.app_opt.tf.fedprox_loss import TFFedProxLoss

fedprox_loss = TFFedProxLoss(mu=fedproxloss_mu)
# Use in training loop

Examples:

examples/advanced/cifar10/tf/cifar10_fedprox

FedOpt (Federated Optimization)

Federated optimization with server-side optimizer (e.g., SGD, Adam).

PyTorch FedOpt 

from nvflare.app_opt.pt.recipes import FedOptRecipe
from nvflare.recipe import SimEnv

recipe = FedOptRecipe(
    name="fedopt-pt",
    min_clients=2,
    num_rounds=5,
    model=MyModel(),
    train_script="client.py",
    optimizer_args={"class_path": "torch.optim.SGD", "args": {"lr": 1.0, "momentum": 0.6}},
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/advanced/cifar10/pt/cifar10-sim/cifar10_fedopt

TensorFlow FedOpt 

from nvflare.app_opt.tf.recipes import FedOptRecipe
from nvflare.recipe import SimEnv

recipe = FedOptRecipe(
    name="fedopt-tf",
    min_clients=2,
    num_rounds=5,
    model=MyTFModel(),
    train_script="client.py",
    optimizer_args={"class_path": "tensorflow.keras.optimizers.SGD", "args": {"learning_rate": 1.0}},
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/advanced/cifar10/tf/cifar10_fedopt

SCAFFOLD 

SCAFFOLD algorithm for handling data heterogeneity with control variates.

PyTorch SCAFFOLD 

from nvflare.app_opt.pt.recipes import ScaffoldRecipe
from nvflare.recipe import SimEnv

recipe = ScaffoldRecipe(
    name="scaffold-pt",
    min_clients=2,
    num_rounds=5,
    model=MyModel(),
    train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/advanced/cifar10/pt/cifar10-sim/cifar10_scaffold

TensorFlow SCAFFOLD 

from nvflare.app_opt.tf.recipes import ScaffoldRecipe
from nvflare.recipe import SimEnv

recipe = ScaffoldRecipe(
    name="scaffold-tf",
    min_clients=2,
    num_rounds=5,
    model=MyTFModel(),
    train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/advanced/cifar10/tf/cifar10_scaffold

Cyclic Learning 

Sequential training across clients in a cyclic order.

PyTorch Cyclic 

from nvflare.app_opt.pt.recipes import CyclicRecipe
from nvflare.recipe import SimEnv

recipe = CyclicRecipe(
    name="cyclic-pt",
    min_clients=2,
    num_rounds=5,
    model=MyModel(),
    train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/hello-world/hello-cyclic

TensorFlow Cyclic 

from nvflare.app_opt.tf.recipes import CyclicRecipe
from nvflare.recipe import SimEnv

recipe = CyclicRecipe(
    name="cyclic-tf",
    min_clients=2,
    num_rounds=5,
    model=MyTFModel(),
    train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

XGBoost Recipes 

Federated XGBoost for tree-based models.

XGBoost Horizontal (Histogram-based)

Histogram-based federated XGBoost for horizontal data partitioning.

from nvflare.app_opt.xgboost.recipes import XGBHorizontalRecipe
from nvflare.recipe import SimEnv

recipe = XGBHorizontalRecipe(
    name="xgb-horizontal",
    min_clients=2,
    num_rounds=10,
    xgb_params={"max_depth": 6, "eta": 0.1, "objective": "binary:logistic"},
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/advanced/xgboost/fedxgb

XGBoost Bagging (Tree-based)

Tree-based federated XGBoost using bagging.

from nvflare.app_opt.xgboost.recipes import XGBBaggingRecipe
from nvflare.recipe import SimEnv

recipe = XGBBaggingRecipe(
    name="xgb-bagging",
    min_clients=2,
    training_mode="bagging",
    num_rounds=10,
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/advanced/xgboost/fedxgb (job_tree.py)

XGBoost Vertical 

Federated XGBoost for vertical data partitioning.

from nvflare.app_opt.xgboost.recipes import XGBVerticalRecipe
from nvflare.recipe import SimEnv

recipe = XGBVerticalRecipe(
    name="xgb-vertical",
    min_clients=2,
    num_rounds=10,
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

Sklearn Specialized Recipes 

K-Means FedAvg 

Federated K-Means clustering.

from nvflare.app_opt.sklearn.recipes import KMeansFedAvgRecipe
from nvflare.recipe import SimEnv

recipe = KMeansFedAvgRecipe(
    name="kmeans",
    min_clients=2,
    num_rounds=5,
    n_clusters=3,
    train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/advanced/sklearn-kmeans

SVM FedAvg 

Federated Support Vector Machine.

from nvflare.app_opt.sklearn.recipes import SVMFedAvgRecipe
from nvflare.recipe import SimEnv

recipe = SVMFedAvgRecipe(
    name="svm",
    min_clients=2,
    num_rounds=5,
    train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/advanced/sklearn-svm

Logistic Regression FedAvg 

Federated Logistic Regression.

from nvflare.app_common.np.recipes.lr.fedavg import FedAvgLrRecipe
from nvflare.recipe import SimEnv

recipe = FedAvgLrRecipe(
    name="lr",
    min_clients=2,
    num_rounds=5,
    train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/hello-world/hello-lr

Federated Statistics 

Compute federated statistics across distributed data.

from nvflare.recipe import SimEnv
from nvflare.recipe.fedstats import FedStatsRecipe

recipe = FedStatsRecipe(
    name="stats",
    stats_output_path="./output",
    sites=["site-1", "site-2"],
    statistic_configs={"count": {}, "mean": {}, "stddev": {}},
    stats_generator=my_stats_generator,
)
env = SimEnv(clients=["site-1", "site-2"])
run = recipe.execute(env)

Examples:

Federated Evaluation 

Evaluate a pre-trained model across multiple sites.

PyTorch FedEval 

Evaluate a pre-trained PyTorch model by sending it to all clients for evaluation on their local data.

from nvflare.app_opt.pt.recipes.fedeval import FedEvalRecipe
from nvflare.recipe import SimEnv

recipe = FedEvalRecipe(
    name="eval_job",
    model=MyModel(),
    eval_ckpt="/path/to/pretrained_model.pt",
    min_clients=2,
    eval_script="client.py",
    eval_args="--batch_size 32",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Note

eval_ckpt is required. It can be either:

an absolute path on the server to the pre-trained checkpoint (.pt, .pth), or
a relative or absolute path to a local checkpoint file that will be bundled with the job (for example, via utilities such as prepare_initial_ckpt).

When specifying an absolute server-side path, the checkpoint file may not exist locally when building the job.

Examples:

examples/hello-world/hello-lightning-eval

Private Set Intersection (PSI)

Compute intersection of private sets across clients.

from nvflare.app_common.psi.recipes import DhPSIRecipe
from nvflare.recipe import SimEnv

recipe = DhPSIRecipe(
    name="psi",
    min_clients=2,
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/advanced/psi/user_email_match

Flower Integration 

Run Flower-based federated learning jobs.

from nvflare.app_opt.flower.recipe import FlowerRecipe
from nvflare.recipe import SimEnv

recipe = FlowerRecipe(
    name="flower-job",
    min_clients=2,
    flower_app="path/to/flower/app",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)

Examples:

examples/hello-world/hello-flower

Swarm Learning 

Decentralized federated learning without a central server.

from nvflare.app_opt.pt.recipes.swarm import SwarmLearningRecipe
from nvflare.recipe import SimEnv

recipe = SwarmLearningRecipe(
    name="swarm",
    model=MyModel(),
    min_clients=3,
    num_rounds=5,
    train_script="client.py",
    initial_ckpt="/path/to/pretrained.pt",  # Optional: pre-trained weights
    round_timeout=3600,  # P2P model-transfer ACK budget (seconds); increase for large models
)
env = SimEnv(num_clients=3)
run = recipe.execute(env)

Note

For large models (>2 GB), tune the following parameters:

round_timeout (default 3600 s): P2P model-transfer ACK budget between peers. Increase for 7B+ models where P2P tensor streaming can take several minutes.
pipe_type (default "cell_pipe"): set to "file_pipe" when cell networking is unavailable or for third-party subprocess integrations.
submit_result_timeout and tensor_min_download_timeout: set via recipe.add_client_config({...}) — see Timeout Troubleshooting Guide.

Edge Recipes 

Recipes for edge device federated learning.

EdgeFedBuffRecipe 

Federated buffered aggregation for generic edge devices.

from nvflare.edge.tools.edge_fed_buff_recipe import (
    EdgeFedBuffRecipe,
    ModelManagerConfig,
    DeviceManagerConfig,
)

recipe = EdgeFedBuffRecipe(
    job_name="edge-fedavg",
    model=MyModel(),
    model_manager_config=ModelManagerConfig(max_num_active_model_versions=3, max_model_version=20),
    device_manager_config=DeviceManagerConfig(device_selection_size=100),
    initial_ckpt="/path/to/pretrained.pt",  # Optional: pre-trained weights
)

Examples:

examples/advanced/edge/jobs

ETFedBuffRecipe 

Federated buffered aggregation for ExecuTorch-based edge devices. Extends EdgeFedBuffRecipe with a DeviceModel wrapper that handles ExecuTorch model export. Requires the executorch package.

from nvflare.edge.tools.et_fed_buff_recipe import ETFedBuffRecipe
from nvflare.edge.tools.edge_fed_buff_recipe import (
    ModelManagerConfig,
    DeviceManagerConfig,
)
from nvflare.edge.models.model import DeviceModel

recipe = ETFedBuffRecipe(
    job_name="et-fedavg",
    device_model=DeviceModel(MyModel()),
    input_shape=(1, 3, 224, 224),
    output_shape=(1, 10),
    model_manager_config=ModelManagerConfig(max_num_active_model_versions=3, max_model_version=20),
    device_manager_config=DeviceManagerConfig(device_selection_size=100),
    initial_ckpt="/path/to/pretrained.pt",  # Optional: pre-trained weights
    device_wait_timeout=300.0,  # Recommended when device_reuse=False with a finite device pool
)

Note

device_wait_timeout sets how long (seconds) the job waits for sufficient devices to join before stopping. When device_reuse=False with a finite device pool, set an explicit timeout (e.g. 300.0) to avoid the job hanging indefinitely once the pool is exhausted. Defaults to None (wait indefinitely).

Examples:

examples/advanced/edge/jobs

Utility Functions 

Add Experiment Tracking 

Add experiment tracking (MLflow, TensorBoard, W&B) to any recipe.

from nvflare.recipe.utils import add_experiment_tracking

add_experiment_tracking(recipe, tracking_type="tensorboard")
# or
add_experiment_tracking(recipe, tracking_type="mlflow")
# or
add_experiment_tracking(recipe, tracking_type="wandb")

Add Cross-Site Evaluation 

Add cross-site evaluation to any training recipe.

from nvflare.recipe.utils import add_cross_site_evaluation

add_cross_site_evaluation(recipe)

Execution Environments 

Recipes can be executed in different environments:

SimEnv (Simulation)

Run locally for development and testing.

from nvflare.recipe import SimEnv

env = SimEnv(num_clients=2)
run = recipe.execute(env)

PocEnv (Proof of Concept)

Run with multiple processes on a single machine.

from nvflare.recipe import PocEnv

env = PocEnv(num_clients=2)
run = recipe.execute(env)

ProdEnv (Production)

Deploy to production NVFlare infrastructure.

from nvflare.recipe import ProdEnv

env = ProdEnv(startup_kit_location="/path/to/startup_kit")
run = recipe.execute(env)