nvflare.app_common.executors.client_api_executor module

The public Client API executor, configured by execution mode.

Design: docs/design/client_api_execution_modes.md (“What We Propose”, “Overview”, “Execution Modes”, “Configuration Surface”). This module path is normative - job configs reference nvflare.app_common.executors.client_api_executor.ClientAPIExecutor.

Availability: in_process is fully supported. external_process and attach are not yet implemented; selecting them fails cleanly at job startup.

Unlike the legacy executors (InProcessClientAPIExecutor / ClientAPILauncherExecutor), this surface has no parameter-conversion args (params_exchange_format / params_transfer_type / server_expected_format / converter ids): the Client API boundary passes parameters through unconverted, and format conversion belongs to send/receive filters at the client edge. Transfer type (FULL/DIFF) is a model-registry concern, configured elsewhere.

class ClientAPIExecutor(execution_mode: str, command: str | None = None, task_script_path: str | None = None, task_script_args: str = '', launch_once: bool = True, launch_timeout: float | None = None, shutdown_timeout: float | None = None, stop_grace_period: float = 30.0, heartbeat_interval: float = 5.0, heartbeat_timeout: float = 30.0, task_wait_timeout: float | None = None, result_wait_timeout: float | None = None, train_task_name: str = 'train', evaluate_task_name: str = 'validate', submit_model_task_name: str = 'submit_model', train_with_evaluation: bool = False, memory_gc_rounds: int = 0, cuda_empty_cache: bool = False, attach_timeout: float | None = None, allow_reconnect: bool = False)[source]

Bases: Executor

One executor for all Client API execution modes.

The trainer-facing Client API (flare.init/receive/send/log) is unchanged; this executor replaces the Pipe/launcher integration stack. It delegates to an internal mode-specific backend (ClientAPIBackendSpec) resolved from execution_mode at START_RUN:

  • in_process: trainer runs inside the CJ process over DataBus.

  • external_process: NVFlare launches and owns the trainer process tree; control over Cell.

  • attach: an externally started/owned trainer attaches over Cell.

Initializes the ClientAPIExecutor.

Parameter names are part of the public job-config surface: renames are breaking changes (guarded by the surface-freeze test).

Parameters:

  • execution_mode (str) – One of “in_process”, “external_process”, or “attach”. Required.

  • command (Optional[str]) – The trainer launch command, e.g. “python custom/train.py”, “torchrun …”. Required for (and only valid in) “external_process” mode. An empty/whitespace-only string is treated as unset.

  • task_script_path (Optional[str]) – in_process only. Path to the user training script the in_process backend runs via TaskScriptRunner. An empty/whitespace-only string is treated as unset. (The in_process backend validates presence and “.py” suffix.)

  • task_script_args (str) – in_process only. Arguments appended to task_script_path.

  • launch_once (bool) – external_process only. Launch the trainer once per job (default) vs once per task.

  • launch_timeout (Optional[float]) – external_process only. Bound for the launched trainer to complete its HELLO/session setup (this replaces the legacy external_pre_init_timeout). None means no timeout.

  • shutdown_timeout (Optional[float]) – external_process only. How long to wait for the trainer to exit naturally after an orderly SHUTDOWN before starting forced process-tree termination. None means the backend default.

  • stop_grace_period (float) – external_process only. Grace period between SIGTERM and SIGKILL when terminating the trainer process group (design: “Process-tree termination”).

  • heartbeat_interval (float) – out-of-process only (external_process/attach). Interval (seconds) for session heartbeats.

  • heartbeat_timeout (float) – out-of-process only (external_process/attach). Session lease timeout (seconds) on missed heartbeats. An in-flight payload transfer keeps the lease alive (design: “Heartbeat and Liveness”).

  • task_wait_timeout (Optional[float]) – Bound for the trainer to accept a delivered task. None means no timeout.

  • result_wait_timeout (Optional[float]) – Control-side bound for retrieving the task result. Payload transfer completion is governed by the shared transfer layer, not by this value. None means no timeout.

  • train_task_name (str) – Task name treated as “train” by flare.is_train() (rank contract). Defaults to AppConstants.TASK_TRAIN.

  • evaluate_task_name (str) – Task name treated as “evaluate” by flare.is_evaluate(). Defaults to AppConstants.TASK_VALIDATION.

  • submit_model_task_name (str) – Task name treated as “submit_model” by flare.is_submit_model(). Defaults to AppConstants.TASK_SUBMIT_MODEL.

  • train_with_evaluation (bool) – Whether the trainer also returns evaluation metrics with the trained model.

  • memory_gc_rounds (int) – Force a GC cycle every N rounds (0 disables).

  • cuda_empty_cache (bool) – Whether to also empty the CUDA cache during memory cleanup.

  • attach_timeout (Optional[float]) – attach only. Bound for the externally started trainer to attach. None means no timeout.

  • allow_reconnect (bool) – attach only. Whether a trainer may re-attach to an existing session after a disconnect.

execute(task_name: str, shareable: Shareable, fl_ctx: FLContext, abort_signal: Signal) → Shareable[source]

Executes a task.

Parameters:

  • task_name (str) – task name.

  • shareable (Shareable) – input shareable.

  • fl_ctx (FLContext) – fl context.

  • abort_signal (Signal) – signal to check during execution to determine whether this task is aborted.

Returns:

An output shareable.

fire_log_analytics(fl_ctx: FLContext, dxo: DXO) → None[source]

Converts trainer LOG data into an analytics event. Executor-owned surface.

Backends call this for every LOG control message (flare.log from the trainer), regardless of execution mode - this replaces MetricRelay (ex-process) and the in-process executor’s log callback as the single analytics-event ownership point.

The fire path is mode-selectable via set_analytics_fire_fed_event():

  • False (default): fires the local, un-prefixed ANALYTIC_EVENT_TYPE (“analytix_log_stats”) and relies on the ConvertToFedEvent widget (added by BaseFedJob) to forward it to the server as “fed.analytix_log_stats”.

  • True: fires a federation-scoped event directly to the server, matching what MetricRelay does today for ex-process metrics. Cell backends may select this path during initialize() by calling set_analytics_fire_fed_event(True) when no ConvertToFedEvent widget is configured.

The two paths must land on the same server-side event name. ConvertToFedEvent prefixes the local event with “fed.”, so the fed path must fire the already-prefixed FED_ANALYTIC_EVENT_TYPE (“fed.analytix_log_stats”); firing the un-prefixed name federation-scoped would miss every consumer listening on “fed.analytix_log_stats” (MetricRelay in job_config/script_runner.py, flower_job.py).

Parameters:

  • fl_ctx – an FLContext to fire the event with.

  • dxo – the analytics data (e.g. from create_analytic_dxo) carried by the event.

handle_event(event_type: str, fl_ctx: FLContext)[source]

Handles events.

Parameters:

  • event_type (str) – event type fired by workflow.

  • fl_ctx (FLContext) – FLContext information.

set_analytics_fire_fed_event(enabled: bool) → None[source]

Selects whether trainer LOG data is fired as a federation-scoped analytics event.

Cell backends call this during initialization when no ConvertToFedEvent widget is configured. The default remains the local analytics path used by the in-process backend.

Parameters:

enabled – True to fire federation-scoped events directly; False to fire local events.

class ExecutionMode[source]

Bases: object

Valid values for ClientAPIExecutor’s execution_mode (see design “Overview”).

ATTACH = 'attach'
EXTERNAL_PROCESS = 'external_process'
IN_PROCESS = 'in_process'