nvflare.app_common.executors.client_api_executor module

Client API executor with mode-specific trainer backends.

All three execution modes are supported. Parameter conversion and FULL/DIFF state are handled by the trainer-side Client API.

class ClientAPIExecutor(execution_mode: str, command: str | list[str] | None = None, task_script_path: str | None = None, task_script_args: str = '', launch_once: bool = True, launch_timeout: float | None = 300.0, shutdown_timeout: float | None = None, stop_grace_period: float = 30.0, heartbeat_interval: float = 5.0, heartbeat_timeout: float = 30.0, task_wait_timeout: float | None = None, result_wait_timeout: float | None = None, train_task_name: str = 'train', evaluate_task_name: str = 'validate', submit_model_task_name: str = 'submit_model', train_with_evaluation: bool = False, params_exchange_format: ExchangeFormat = ExchangeFormat.RAW, server_expected_format: ExchangeFormat = ExchangeFormat.NUMPY, params_transfer_type: TransferType = TransferType.FULL, memory_gc_rounds: int = 0, cuda_empty_cache: bool = False, attach_id: str | None = None, attach_timeout: float | None = None, allow_reconnect: bool = False, allow_insecure_attach: bool = False)[source]

Bases: Executor

Delegates Client API task execution to the configured backend.

Initializes the ClientAPIExecutor.

Parameter names are part of the public job-config surface: renames are breaking changes (guarded by the surface-freeze test).

Parameters:

  • execution_mode (str) – One of “in_process”, “external_process”, or “attach”. Required.

  • command (Optional[Union[str, list[str]]]) – The trainer launch command, either as a command string (e.g. “python custom/train.py”) or shell-free argv. Required for (and only valid in) “external_process” mode. An empty/whitespace-only string or empty argv is treated as unset. Use argv when values must retain exact boundaries across platforms.

  • task_script_path (Optional[str]) – in_process only. Path to the user training script the in_process backend runs via TaskScriptRunner. An empty/whitespace-only string is treated as unset. (The in_process backend validates presence and “.py” suffix.)

  • task_script_args (str) – in_process only. Arguments appended to task_script_path.

  • launch_once (bool) – external_process only. Launch the trainer once per job (default) vs once per task.

  • launch_timeout (Optional[float]) – external_process only. Bound for the launched trainer to complete its HELLO/session setup (this replaces the legacy external_pre_init_timeout). Defaults to 300 seconds for compatibility with prior releases; an explicit None means no timeout.

  • shutdown_timeout (Optional[float]) – external_process only. Primarily the natural-exit wait after orderly SHUTDOWN; it also bounds the finalize/execute gate, supplies the accepted-source disconnect grace, bounds the post-settlement process-group exit wait, and feeds the settled per-task reaper budget. None selects the 30-second backend default. Zero skips the direct orderly-exit and finalize-gate waits. In the accepted-source roles, zero is normalized to 30 seconds for disconnect and post-settlement group-exit waits; that fallback also feeds the fixed 30-second settled-reaper budget, which reserves up to 5 seconds for termination.

  • stop_grace_period (float) – external_process only. Grace period between SIGTERM and SIGKILL when terminating the trainer process group (design: “Process-tree termination”). The accepted-result reaper silently caps this phase at 5 seconds; ordinary teardown honors the configured value.

  • heartbeat_interval (float) – out-of-process only (external_process/attach). Interval (seconds) for session heartbeats.

  • heartbeat_timeout (float) – out-of-process only (external_process/attach). Session lease timeout (seconds) on missed heartbeats. An in-flight payload transfer keeps the lease alive (design: “Heartbeat and Liveness”). Zero disables heartbeat checks for external_process but is invalid for attach, whose external trainer requires heartbeat liveness as the fallback for a lost terminal SHUTDOWN.

  • task_wait_timeout (Optional[float]) – Bound for the trainer to accept a delivered task. None means no timeout for external_process; attach applies a 600-second task-delivery budget when unset.

  • result_wait_timeout (Optional[float]) – Control-side bound for retrieving the task result. Payload transfer completion is governed by the shared transfer layer, not by this value. None means no timeout.

  • train_task_name (str) – Task name treated as “train” by flare.is_train() (rank contract). Defaults to AppConstants.TASK_TRAIN.

  • evaluate_task_name (str) – Task name treated as “evaluate” by flare.is_evaluate(). Defaults to AppConstants.TASK_VALIDATION.

  • submit_model_task_name (str) – Task name treated as “submit_model” by flare.is_submit_model(). Defaults to AppConstants.TASK_SUBMIT_MODEL.

  • train_with_evaluation (bool) – Whether a training result is required to include pre-training evaluation metrics. When False, metrics are optional; metrics supplied by the trainer or a framework integration are still returned.

  • params_exchange_format (ExchangeFormat) – Framework-native parameter representation exposed by flare.receive() and accepted by flare.send(). The declaration is transported to the trainer in TASK_EXCHANGE; the executor does not perform conversion. RAW explicitly disables representation adaptation.

  • server_expected_format (ExchangeFormat) – Parameter representation expected by the server. The declaration is transported to the trainer in TASK_EXCHANGE.

  • params_transfer_type (TransferType) – Whether training results contain full parameters or a difference from the received parameters. This is applied by the trainer-side model state and is independent of framework representation conversion.

  • memory_gc_rounds (int) – Force a GC cycle every N rounds (0 disables).

  • cuda_empty_cache (bool) – Whether to also empty the CUDA cache during memory cleanup.

  • attach_id (Optional[str]) – attach only. Pre-provisioned rendezvous ID shared with exactly one externally started trainer. It is a routing name, not a secret.

  • attach_timeout (Optional[float]) – attach only. Bound for the externally started trainer to attach. None means no timeout.

  • allow_reconnect (bool) – attach only. Whether a trainer may re-attach to an existing session after a disconnect.

  • allow_insecure_attach (bool) – attach only. Deprecated compatibility argument with no effect. Network Attach uses the site’s existing CP trust model; a CJ-owned shared-file route must always satisfy its filesystem protection checks.

execute(task_name: str, shareable: Shareable, fl_ctx: FLContext, abort_signal: Signal) → Shareable[source]

Executes a task.

Parameters:

  • task_name (str) – task name.

  • shareable (Shareable) – input shareable.

  • fl_ctx (FLContext) – fl context.

  • abort_signal (Signal) – signal to check during execution to determine whether this task is aborted.

Returns:

An output shareable.

property execution_mode: str

Read-only view of the configured mode, for validation and diagnostics.

fire_log_analytics(fl_ctx: FLContext, dxo: DXO) → None[source]

Emit trainer LOG data through the configured analytics path.

The local path fires analytix_log_stats and ConvertToFedEvent prefixes it to fed.analytix_log_stats. The direct path must fire that prefixed name itself or server-side consumers will miss the event.

handle_event(event_type: str, fl_ctx: FLContext)[source]

Handles events.

Parameters:

  • event_type (str) – event type fired by workflow.

  • fl_ctx (FLContext) – FLContext information.

set_analytics_fire_fed_event(enabled: bool) → None[source]

Select direct federation events instead of the local ConvertToFedEvent path.

class ExecutionMode[source]

Bases: object

Valid values for ClientAPIExecutor’s execution_mode (see design “Overview”).

ATTACH = 'attach'
EXTERNAL_PROCESS = 'external_process'
IN_PROCESS = 'in_process'