nvflare.app_opt.pt.recipes.fedeval module

class EvalController(persistor_id: str, timeout: int)[source]

Bases: ModelController

Model Controller API for FLModel-based Controller.

Parameters:

persistor_id (str, optional) – ID of the persistor component. Defaults to AppConstants.DEFAULT_PERSISTOR_ID (“persistor”).

run()[source]

Main run routine for the controller workflow.

class FedEvalRecipe(*, name: str = 'eval', model: Any | Dict[str, Any], eval_ckpt: str, min_clients: int, eval_script: str, eval_args: str = '', launch_external_process: bool = False, command: str = 'python3 -u', server_expected_format: ExchangeFormat = ExchangeFormat.NUMPY, validation_timeout: int = 6000, per_site_config: Dict[str, Dict] | None = None, client_memory_gc_rounds: int = 0, cuda_empty_cache: bool = False)[source]

Bases: Recipe

A recipe for federated evaluation of a PyTorch model across multiple sites.

This recipe sets up a federated evaluation workflow where a global model from the server is sent to multiple clients for evaluation. Each client evaluates the model on their local data and reports metrics back to the server.

The recipe configures: - A federated job with an initial model to evaluate - EvalController for coordinating federated evaluation across clients - Script runners for client-side evaluation execution

Parameters:

  • name – Name of the federated evaluation job. Defaults to “eval”.

  • model – Model structure to evaluate. Can be: - An instantiated nn.Module (e.g., Net()) - A dict config: {“class_path”: “module.ClassName”, “args”: {…}}

  • eval_ckpt – Absolute path to pre-trained checkpoint file (.pt, .pth, etc.). Required for evaluation - specifies which weights to evaluate. The file may not exist locally (server-side path).

  • min_clients – Minimum number of clients required to start evaluation.

  • eval_script – Path to the evaluation script that will be executed on each client.

  • eval_args – Command line arguments to pass to the evaluation script. The string is stored in the job definition and must not contain actual secret values; see nvflare.recipe.secrets for safe runtime references. Defaults to “”.

  • launch_external_process – Whether to launch the script in external process. Defaults to False.

  • command – If launch_external_process=True, command to run script (prepended to script). Defaults to “python3 -u”.

  • server_expected_format – What format to exchange the parameters between server and client. Defaults to ExchangeFormat.NUMPY.

  • validation_timeout – Timeout for evaluation task in seconds. Defaults to 6000.

  • per_site_config – Deprecated constructor form of per-site configuration. New code should call set_per_site_config(recipe, config) immediately after construction. Each config dict can contain optional overrides: eval_script, eval_args, launch_external_process, command, server_expected_format. Values are stored in the job definition and must not contain actual secret values. If not provided, the same configuration will be used for all clients. Defaults to None.

Example

Basic usage with model instance:

```python from nvflare.app_opt.pt.recipes.fedeval import FedEvalRecipe from model import Net

recipe = FedEvalRecipe(

name=”eval_job”, model=Net(), eval_ckpt=”/path/to/pretrained_model.pt”, min_clients=2, eval_script=”client.py”, eval_args=”–batch_size 32”,

)

Using dict config:

```python recipe = FedEvalRecipe(

name=”eval_job”, model={“class_path”: “my_module.Net”, “args”: {“num_classes”: 10}}, eval_ckpt=”/path/to/pretrained_model.pt”, min_clients=2, eval_script=”client.py”,

)

This is base class of a recipe. Recipes are implemented by jobs. A concrete recipe must provide the job for recipe implementation.

Security contract – no secrets in recipe parameters:

Recipe parameters (train_args, task_args, eval_args, per_site_config, config overrides, dicts passed to add_client_config/add_server_config, exec params, etc.) can be written in clear text into generated job configuration. These parameters and their nested values must never contain actual passwords, API keys, tokens, private keys, or other credentials. Instead, read secrets from site environment variables or mounted secret files inside your code, or pass a placeholder created with nvflare.recipe.secrets.secret_ref() or nvflare.recipe.secrets.secret_file_ref() at a supported runtime boundary. See nvflare.recipe.secrets for the supported parameter locations.

Before export or run, recipes scan their parameters with heuristics and emit nvflare.recipe.secrets.PotentialSecretWarning when a value looks like an actual secret. The scan is best-effort: absence of a warning does not prove a parameter is safe to share.

param job:

the job that implements the recipe.