Available Recipes
NVFlare provides a variety of pre-built recipes for common federated learning algorithms and workflows. Recipes are high-level, declarative APIs that simplify job configuration and execution.
Before You Start
This page is a catalog of available recipe classes and short starting snippets. For model input formats, checkpoint behavior, and execution environments, see NVFlare Job Recipe. For common Recipe methods, helpers, and stable API behavior, see Recipe API Reference.
Important
Recipe arguments and helper configuration become part of the generated job
definition. Never put an actual password, token, API key, private key, or
other credential in any recipe parameter, including nested dictionaries.
Keep secrets in site environment variables or mounted secret files. Use
secret_ref or secret_file_ref only at supported runtime boundaries;
otherwise read the secret directly inside your training code. See
Keeping Secrets Out Of Recipe Parameters for the supported locations.
Fed Task
For one-round workflows that do not have a global model lifecycle, use FedTaskRecipe.
This is useful for client-side embedding extraction, preprocessing, feature generation, local evaluation,
or other federated tasks that only need the server to coordinate one script execution across clients.
from nvflare.recipe import FedTaskRecipe, SimEnv
recipe = FedTaskRecipe(
name="extract-embeddings",
task_name="embed",
min_clients=2,
task_script="client.py",
task_args="--data-root /data --output-root /tmp/embeddings",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
FedTaskRecipe sends one FLModel task to the selected clients and waits for their results.
It does not require model, initial_ckpt, or model_persistor.
Federated Averaging (FedAvg)
The most fundamental federated learning algorithm that aggregates model updates from multiple clients by computing a weighted average.
PyTorch FedAvg
from nvflare.app_opt.pt.recipes import FedAvgRecipe
from nvflare.recipe import SimEnv
recipe = FedAvgRecipe(
name="fedavg-pt",
min_clients=2,
num_rounds=5,
model=MyModel(),
train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
For large PyTorch model updates, FedAvgRecipe also supports
enable_tensor_disk_offload=True to reduce server memory use by materializing
incoming streamed tensors to temporary files. See
Starting Federated Learning Servers for deployment
notes about configuring the server temporary directory.
Examples:
TensorFlow FedAvg
from nvflare.app_opt.tf.recipes import FedAvgRecipe
from nvflare.recipe import SimEnv
recipe = FedAvgRecipe(
name="fedavg-tf",
min_clients=2,
num_rounds=5,
model=MyTFModel(),
train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
NumPy FedAvg
For framework-agnostic or NumPy-based models.
from nvflare.app_common.np.recipes import NumpyFedAvgRecipe
from nvflare.recipe import SimEnv
recipe = NumpyFedAvgRecipe(
name="fedavg-numpy",
min_clients=2,
num_rounds=5,
model=[0.0, 0.0, 0.0],
train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
Sklearn FedAvg
For scikit-learn based models.
from nvflare.app_opt.sklearn.recipes import SklearnFedAvgRecipe
from nvflare.recipe import SimEnv
recipe = SklearnFedAvgRecipe(
name="fedavg-sklearn",
min_clients=2,
num_rounds=5,
train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
FedAvg with Homomorphic Encryption
FedAvg with secure aggregation using homomorphic encryption.
from nvflare.app_opt.pt.recipes import FedAvgRecipeWithHE
from nvflare.recipe import ProdEnv
recipe = FedAvgRecipeWithHE(
name="fedavg-he",
min_clients=2,
num_rounds=5,
model=MyModel(),
train_script="client.py",
)
env = ProdEnv(
startup_kit_location="/path/to/startup_kit/admin@nvidia.com",
username="admin@nvidia.com",
)
run = recipe.execute(env)
Note
FedAvgRecipeWithHE requires provisioned startup kits with homomorphic encryption context files.
Use ProdEnv or PocEnv with HE provisioning; SimEnv is not supported.
Examples:
WEIGHT_DIFF Compatibility
DataKind.WEIGHT_DIFF is supported only when the client executor sends parameter
differences and the server aggregation path accepts differences. A client’s
FLModel.params_type is the authoritative description of its result. Recipe construction
cannot inspect an arbitrary training script, so it validates only recipe-owned server settings,
such as supported data kinds and a custom aggregator’s declared expected_data_kind.
Recipe |
Server aggregation path |
Support |
Required configuration |
|---|---|---|---|
Unified, PyTorch, TensorFlow, and NumPy |
Built-in |
Yes |
Set |
|
User-provided aggregator |
Conditional |
The client returns |
|
|
Yes |
Set |
PyTorch |
|
Yes |
The recipe fixes its aggregation path to |
TensorFlow |
Built-in |
Yes |
Return |
|
|
Yes |
Set |
|
Built-in |
Conditional |
The client script must compute the difference and return
|
The standard InTimeAccumulateWeightedAggregator and
HEInTimeAccumulateWeightedAggregator accept both WEIGHTS and WEIGHT_DIFF
when their expected_data_kind is configured accordingly. Aggregators that declare
expected_data_kind are checked against the recipe setting during construction.
For example, configure PyTorch FedAvg with differences as follows:
from nvflare.apis.dxo import DataKind
from nvflare.app_opt.pt.recipes import FedAvgRecipe
recipe = FedAvgRecipe(
name="fedavg-diff",
min_clients=2,
num_rounds=5,
model=MyModel(),
train_script="client.py",
aggregator_data_kind=DataKind.WEIGHT_DIFF,
)
The client script must compute local minus global parameters and return them explicitly:
import nvflare.client as flare
from nvflare.app_common.abstract.fl_model import ParamsType
flare.send(flare.FLModel(params=model_diff, params_type=ParamsType.DIFF))
If a custom aggregator declares an incompatible expected_data_kind, recipe construction
raises an error naming both the configured and declared kinds and how to align them.
FedProx
FedProx is FedAvg with a proximal term added to the client loss function to handle data heterogeneity.
It uses the standard FedAvgRecipe with the FedProx loss helper on the client side.
Because PyTorch FedProx uses FedAvgRecipe, it also supports enable_tensor_disk_offload=True for
streamed PyTorch tensor updates, with the same behavior and constraints as PyTorch FedAvg.
PyTorch FedProx
from nvflare.app_opt.pt.recipes import FedAvgRecipe
from nvflare.recipe import SimEnv
# FedProx uses FedAvgRecipe with FedProxLoss in the client training script
recipe = FedAvgRecipe(
name="fedprox-pt",
min_clients=2,
num_rounds=5,
model=MyModel(),
train_script="client.py",
train_args="--fedproxloss_mu 0.01", # Pass mu parameter to client
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
In your client training script, use the FedProxLoss helper:
from nvflare.app_opt.pt import PTFedProxLoss
# In training loop:
fedprox_loss = PTFedProxLoss(mu=fedproxloss_mu)
for data, target in train_loader:
optimizer.zero_grad()
output = model(data)
ce_loss = criterion(output, target)
# Add FedProx regularization term
prox_loss = fedprox_loss(model)
loss = ce_loss + prox_loss
loss.backward()
optimizer.step()
Examples:
TensorFlow FedProx
from nvflare.app_opt.tf.recipes import FedAvgRecipe
from nvflare.recipe import SimEnv
recipe = FedAvgRecipe(
name="fedprox-tf",
min_clients=2,
num_rounds=5,
model=MyTFModel(),
train_script="client.py",
train_args="--fedproxloss_mu 0.01",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
In your client training script, use the TensorFlow FedProxLoss:
from nvflare.app_opt.tf.fedprox_loss import TFFedProxLoss
fedprox_loss = TFFedProxLoss(mu=fedproxloss_mu)
# Use in training loop
Examples:
FedOpt (Federated Optimization)
Federated optimization with server-side optimizer (e.g., SGD, Adam).
PyTorch FedOpt
from nvflare.app_opt.pt.recipes import FedOptRecipe
from nvflare.recipe import SimEnv
recipe = FedOptRecipe(
name="fedopt-pt",
min_clients=2,
num_rounds=5,
model=MyModel(),
train_script="client.py",
optimizer_args={"path": "torch.optim.SGD", "args": {"lr": 1.0, "momentum": 0.6}},
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Note
PyTorch FedOpt supports enable_tensor_disk_offload=True for streamed PyTorch tensor updates.
Import ExchangeFormat from nvflare.client.config and configure
server_expected_format=ExchangeFormat.PYTORCH so the server path preserves tensors instead of converting
updates to NumPy before aggregation.
Examples:
TensorFlow FedOpt
from nvflare.app_opt.tf.recipes import FedOptRecipe
from nvflare.recipe import SimEnv
recipe = FedOptRecipe(
name="fedopt-tf",
min_clients=2,
num_rounds=5,
model=MyTFModel(),
train_script="client.py",
optimizer_args={"path": "tensorflow.keras.optimizers.SGD", "args": {"learning_rate": 1.0}},
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
SCAFFOLD
SCAFFOLD algorithm for handling data heterogeneity with control variates.
PyTorch SCAFFOLD
from nvflare.app_opt.pt.recipes import ScaffoldRecipe
from nvflare.recipe import SimEnv
recipe = ScaffoldRecipe(
name="scaffold-pt",
min_clients=2,
num_rounds=5,
model=MyModel(),
train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
TensorFlow SCAFFOLD
from nvflare.app_opt.tf.recipes import ScaffoldRecipe
from nvflare.recipe import SimEnv
recipe = ScaffoldRecipe(
name="scaffold-tf",
min_clients=2,
num_rounds=5,
model=MyTFModel(),
train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
Cyclic Learning
Sequential training across clients in a cyclic order.
PyTorch Cyclic
from nvflare.app_opt.pt.recipes import CyclicRecipe
from nvflare.recipe import SimEnv
recipe = CyclicRecipe(
name="cyclic-pt",
min_clients=2,
num_rounds=5,
model=MyModel(),
train_script="client.py",
task_assignment_timeout=30,
shutdown_timeout=120.0, # External client process only
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
task_assignment_timeout configures the server CyclicController and
shutdown_timeout configures the client ScriptRunner. For controller or
runner options that do not have named recipe parameters, use
server_config_overrides or client_config_overrides. These dictionaries
are shallow-merged after the named parameters, so overlapping dictionary values
take precedence. task_check_period must be positive when overridden:
recipe = CyclicRecipe(
name="cyclic-advanced",
min_clients=2,
model=MyModel(),
train_script="client.py",
task_assignment_timeout=30,
server_config_overrides={"task_check_period": 1.0},
client_config_overrides={"launch_once": False},
)
TensorFlow Cyclic
from nvflare.app_opt.tf.recipes import CyclicRecipe
from nvflare.recipe import SimEnv
recipe = CyclicRecipe(
name="cyclic-tf",
min_clients=2,
num_rounds=5,
model=MyTFModel(),
train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
XGBoost Recipes
Federated XGBoost for tree-based models.
XGBoost Horizontal (Histogram-based)
Histogram-based federated XGBoost for horizontal data partitioning.
from nvflare.app_opt.xgboost.recipes import XGBHorizontalRecipe
from nvflare.recipe import SimEnv
recipe = XGBHorizontalRecipe(
name="xgb-horizontal",
min_clients=2,
num_rounds=10,
xgb_params={"max_depth": 6, "eta": 0.1, "objective": "binary:logistic"},
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
XGBoost Bagging (Tree-based)
Tree-based federated XGBoost using bagging.
from nvflare.app_opt.xgboost.recipes import XGBBaggingRecipe
from nvflare.recipe import SimEnv
recipe = XGBBaggingRecipe(
name="xgb-bagging",
min_clients=2,
training_mode="bagging",
num_rounds=10,
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
XGBoost Vertical
Federated XGBoost for vertical data partitioning.
from nvflare.app_opt.xgboost.recipes import XGBVerticalRecipe
from nvflare.recipe import SimEnv
recipe = XGBVerticalRecipe(
name="xgb-vertical",
min_clients=2,
num_rounds=10,
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
Sklearn Specialized Recipes
K-Means FedAvg
Federated K-Means clustering.
from nvflare.app_opt.sklearn.recipes import KMeansFedAvgRecipe
from nvflare.recipe import SimEnv
recipe = KMeansFedAvgRecipe(
name="kmeans",
min_clients=2,
num_rounds=5,
n_clusters=3,
train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
SVM FedAvg
Federated Support Vector Machine.
from nvflare.app_opt.sklearn.recipes import SVMFedAvgRecipe
from nvflare.recipe import SimEnv
recipe = SVMFedAvgRecipe(
name="svm",
min_clients=2,
num_rounds=5,
train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
Logistic Regression FedAvg
Federated Logistic Regression.
from nvflare.app_common.np.recipes.lr.fedavg import FedAvgLrRecipe
from nvflare.recipe import SimEnv
recipe = FedAvgLrRecipe(
name="lr",
min_clients=2,
num_rounds=5,
train_script="client.py",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
Federated Statistics
Compute federated statistics across distributed data.
from nvflare.recipe import SimEnv
from nvflare.recipe.fedstats import FedStatsRecipe
recipe = FedStatsRecipe(
name="stats",
stats_output_path="./output",
sites=["site-1", "site-2"],
statistic_configs={"count": {}, "mean": {}, "stddev": {}},
stats_generator=my_stats_generator,
)
env = SimEnv(clients=["site-1", "site-2"])
run = recipe.execute(env)
Examples:
Kaplan-Meier Survival Analysis
Federated Kaplan-Meier survival analysis with optional homomorphic encryption over binned event histograms.
The KMRecipe is defined in the Kaplan-Meier example’s job.py rather than exported as a package-level
recipe.
Run the snippet from the Kaplan-Meier example directory so from job import KMRecipe resolves correctly:
cd examples/advanced/kaplan-meier-he
from job import KMRecipe
from nvflare.recipe import SimEnv
# KMRecipe is defined in examples/advanced/kaplan-meier-he/job.py
recipe = KMRecipe(
num_clients=5,
encryption=True,
data_root="/tmp/nvflare/dataset/km_data",
he_context_path_client="/tmp/nvflare/he_context/he_context_client.txt",
he_context_path_server="/tmp/nvflare/he_context/he_context_server.txt",
)
env = SimEnv(num_clients=5)
run = recipe.execute(env)
Examples:
Federated Evaluation
Evaluate a pre-trained model across multiple sites.
PyTorch FedEval
Evaluate a pre-trained PyTorch model by sending it to all clients for evaluation on their local data.
from nvflare.app_opt.pt.recipes.fedeval import FedEvalRecipe
from nvflare.recipe import SimEnv
recipe = FedEvalRecipe(
name="eval_job",
model=MyModel(),
eval_ckpt="/path/to/pretrained_model.pt",
min_clients=2,
eval_script="client.py",
eval_args="--batch_size 32",
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Note
eval_ckpt is required. It can be either:
an absolute path on the server to the pre-trained checkpoint (.pt, .pth), or
a relative or absolute path to a local checkpoint file that will be bundled with the job (for example, via utilities such as
prepare_initial_ckpt).
When specifying an absolute server-side path, the checkpoint file may not exist locally when building the job.
Examples:
Cross-Site Evaluation
Evaluate models across all client sites (compare each client’s model against all datasets).
from nvflare.app_common.np.recipes import NumpyCrossSiteEvalRecipe
from nvflare.recipe import SimEnv
recipe = NumpyCrossSiteEvalRecipe(
name="cross-eval",
min_clients=2,
eval_script="evaluate.py",
eval_args="--data_root /path/to/data",
initial_ckpt="/path/to/pretrained_model.npy", # Optional: evaluate specific model
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Note
Use
eval_scriptto specify custom evaluation logic. If not provided, uses a built-in dummy validator (for testing only).Use
initial_ckptto evaluate a specific pre-trained model. If not provided, the recipe evaluates models from the training run directory.
Examples:
Private Set Intersection (PSI)
Compute intersection of private sets across clients.
from nvflare.app_common.psi.recipes import DhPSIRecipe
from nvflare.recipe import SimEnv
recipe = DhPSIRecipe(
name="psi",
min_clients=2,
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
Flower Integration
Run Flower-based federated learning jobs.
from nvflare.app_opt.flower.recipe import FlowerRecipe
from nvflare.recipe import SimEnv
recipe = FlowerRecipe(
name="flower-job",
min_clients=2,
flower_content="path/to/flower/app",
run_config={"num-server-rounds": 5}, # Optional: used to override default values in pyproject.toml
)
env = SimEnv(num_clients=2)
run = recipe.execute(env)
Examples:
Server-Predeployed Flower App Mode
For production deployments where BYOC (Bring Your Own Code) is restricted, you can use server-predeployed Flower apps. In this mode, the Flower application code is admin-controlled and pre-installed on the server at a known path, and NVFlare distributes it to clients via Flower’s FAB (Flower Application Bundle) mechanism. This eliminates the need for BYOC authorization.
Important: flower_app_path must reference apps that are pre-approved and managed by
server administrators, not user-provided paths. The path must be within the workspace’s
local/custom/ directory to ensure the app is pre-deployed and controlled by the server
admin, not arbitrarily chosen by users at job submission time.
from nvflare.app_opt.flower.recipe import FlowerRecipe
from nvflare.recipe import ProdEnv
recipe = FlowerRecipe(
name="flower-job",
min_clients=2,
flower_app_path="local/custom/preapproved_apps/my_app", # Admin-predeployed on server
run_config={"num-server-rounds": 5},
)
env = ProdEnv(startup_kit_location="/path/to/startup_kit")
run = recipe.execute(env)
Key Differences:
flower_content: Packages Flower app in job ZIP (requires BYOC authorization)flower_app_path: References admin-predeployed app on server (no BYOC needed)
Security Model: When using flower_app_path with BYOC disabled and
flower_predeployed=true:
No user-provided NVFlare custom code is deployed through the job
Flower app code is distributed only from server-admin-controlled/pre-approved app locations
flower_app_pathmust start withlocal/custom/to enforce admin controlArbitrary user-chosen paths are not allowed; only pre-approved apps in designated directories
Authorization Requirements:
Sites using flower_app_path must have the server-predeployed-flwr permission
granted in their authorization.json. By default, this permission is set to "none"
(denied) for all roles. To enable:
{
"format_version": "1.0",
"permissions": {
"lead": {
"server-predeployed-flwr": "any"
}
}
}
See Site Policy Management for details on site authorization policies.
Swarm Learning
Decentralized federated learning without a central server.
from nvflare.app_opt.pt.recipes.swarm import SwarmLearningRecipe
from nvflare.recipe import SimEnv
recipe = SwarmLearningRecipe(
name="swarm",
model=MyModel(),
min_clients=3,
num_rounds=5,
train_script="client.py",
initial_ckpt="/path/to/pretrained.pt", # Optional: pre-trained weights
progress_timeout=7200,
learn_task_timeout=None, # No training-task time limit
learn_task_ack_timeout=3600,
final_result_ack_timeout=3600,
max_concurrent_submissions=1,
)
env = SimEnv(num_clients=3)
run = recipe.execute(env)
Note
For large models (>2 GB), tune the following parameters:
learn_task_timeout(defaultNone): maximum training-task duration.learn_task_ack_timeoutandfinal_result_ack_timeout: P2P model-transfer acknowledgment budgets. Theround_timeoutcompatibility shortcut sets both when their explicit parameters are omitted.progress_timeout(default 3600 s): maximum time without workflow progress.max_concurrent_submissions(default 1, minimum 1): concurrent aggregation submissions.pipe_type(default"cell_pipe"): set to"file_pipe"when cell networking is unavailable or for third-party subprocess integrations.submit_result_timeout,download_complete_timeout,tensor_min_download_timeout, andPEER_READ_TIMEOUT: set viarecipe.add_client_config({...}).max_resendsdefaults to finite value3and can be overridden the same way — see Timeout Troubleshooting Guide.
For advanced controller settings, server_config_overrides and
client_config_overrides are shallow-merged into SwarmServerConfig and
SwarmClientConfig after the named parameters. Overlapping dictionary values
therefore take precedence over the documented named API. Client overrides cannot
replace the recipe-managed executor, aggregator, persistor, shareable generator, or
min_responses_required; use BaseSwarmLearningRecipe for custom components or
quorum settings. Server overrides cannot replace min_clients; set it through the
named parameter so all scheduler and workflow quorum settings remain aligned.
Edge Recipes
Recipes for edge device federated learning.
EdgeFedBuffRecipe
from nvflare.edge.tools.edge_fed_buff_recipe import (
EdgeFedBuffRecipe,
ModelManagerConfig,
DeviceManagerConfig,
)
recipe = EdgeFedBuffRecipe(
job_name="edge-fedavg",
model=MyModel(),
model_manager_config=ModelManagerConfig(max_num_active_model_versions=3, max_model_version=20),
device_manager_config=DeviceManagerConfig(device_selection_size=100),
initial_ckpt="/path/to/pretrained.pt", # Optional: pre-trained weights
)
Examples:
Utility Functions
Add Experiment Tracking
Add experiment tracking (MLflow, TensorBoard, W&B) to any recipe.
from nvflare.recipe.utils import add_experiment_tracking
add_experiment_tracking(recipe, tracking_type="tensorboard")
# or
add_experiment_tracking(recipe, tracking_type="mlflow")
# or
add_experiment_tracking(recipe, tracking_type="wandb")
Add Cross-Site Evaluation
Add cross-site evaluation to any training recipe.
from nvflare.recipe.utils import add_cross_site_evaluation
add_cross_site_evaluation(recipe)
# or limit evaluation to selected clients
add_cross_site_evaluation(recipe, participating_clients=["site-1", "site-3"])