nvflare.app_common.executors.client_api.backend_spec module

Backend spec for the Client API execution modes (V1-internal).

Design: docs/design/client_api_execution_modes.md (“Overview”, “Execution Modes”, “Client API Backends”). One ClientAPIExecutor delegates to one mode-specific backend:

• in_process: trainer runs inside the Client Job (CJ) process over DataBus

• external_process: NVFlare launches and owns the trainer process tree over Cell

• attach: externally owned trainer rendezvous over Cell

This module is internal to NVFlare. It is not a user extension point; users configure ClientAPIExecutor(execution_mode=...) only.

class ClientAPIBackendContext(executor: ClientAPIExecutor, task_script_path: str | None = None, task_script_args: str = '', command: str | list[str] | None = None, launch_once: bool = True, launch_timeout: float | None = 300.0, shutdown_timeout: float | None = None, stop_grace_period: float = 30.0, heartbeat_interval: float = 5.0, heartbeat_timeout: float = 30.0, task_wait_timeout: float | None = None, result_wait_timeout: float | None = None, attach_id: str | None = None, attach_timeout: float | None = None, allow_reconnect: bool = False, allow_insecure_attach: bool = False, params_exchange_format: ExchangeFormat = ExchangeFormat.RAW, server_expected_format: ExchangeFormat = ExchangeFormat.NUMPY, params_transfer_type: TransferType = TransferType.FULL, train_task_name: str = 'train', evaluate_task_name: str = 'validate', submit_model_task_name: str = 'submit_model', train_with_evaluation: bool = False, memory_gc_rounds: int = 0, cuda_empty_cache: bool = False)

Bases: object

Immutable config a ClientAPIExecutor hands to its backend at initialize().

Rationale: the executor’s frozen constructor args live in private attributes and the backend factories are zero-arg, so a backend previously had no clean way to read the heartbeat, timeout, model-state, and task-name config it needs, nor a supported reference back to the executor’s analytics hook. This frozen snapshot is that supported channel - a backend reads its config from here rather than reaching into ClientAPIExecutor private attributes.

The fields contain only settings consumed by the implemented backends. executor is a back-reference so a backend can:

• call executor.fire_log_analytics(fl_ctx, dxo) for every trainer LOG message (the single LOG-to-analytics ownership point; see design “Configuration Surface”), and

• select the federation-scoped analytics path when appropriate by calling executor.set_analytics_fire_fed_event(True) in initialize() (Cell backends do this when no ConvertToFedEvent widget is configured), and

• use the executor’s FLComponent logging helpers.

allow_insecure_attach: bool = False

allow_reconnect: bool = False

attach_id: str | None = None

attach_timeout: float | None = None

command: str | list[str] | None = None

cuda_empty_cache: bool = False

evaluate_task_name: str = 'validate'

executor: ClientAPIExecutor

heartbeat_interval: float = 5.0

heartbeat_timeout: float = 30.0

launch_once: bool = True

launch_timeout: float | None = 300.0

memory_gc_rounds: int = 0

params_exchange_format: ExchangeFormat = 'raw'

params_transfer_type: TransferType = 'FULL'

result_wait_timeout: float | None = None

server_expected_format: ExchangeFormat = 'numpy'

shutdown_timeout: float | None = None

stop_grace_period: float = 30.0

submit_model_task_name: str = 'submit_model'

task_script_args: str = ''

task_script_path: str | None = None

task_wait_timeout: float | None = None

train_task_name: str = 'train'

train_with_evaluation: bool = False

class ClientAPIBackendSpec

Bases: ABC

The narrow lifecycle contract that ClientAPIExecutor drives on its backend.

Lifecycle ownership per execution mode:

• in_process: the backend runs the trainer inside the CJ process and owns its thread.

• external_process: the backend launches and owns the external trainer process tree; it must not stop the trainer before the payload transfer of a pending result reaches terminal state.

• attach: the backend owns only its protocol session. The trainer process is externally owned and must never be signalled, terminated, or waited on by the backend.

abort(fl_ctx: FLContext) → None

Notifies the backend that the run’s active task is being aborted.

Most backends observe the task’s abort_signal directly and need no additional action. Backends that retain resources after execute() returns can override this hook to make their later cleanup abort-aware.

abstract execute(task_name: str, shareable: Shareable, fl_ctx: FLContext, abort_signal: Signal) → Shareable

Executes one task on the trainer and returns its result.

The backend delivers the task to the trainer (TASK_READY over Cell, or DataBus for in_process), waits for the result within the executor-configured task/result bounds, and returns the result Shareable.

Contract: this method must always return a Shareable and must not hang past abort: when abort_signal is triggered, the backend notifies/stops the trainer per its mode’s lifecycle ownership and returns make_reply(ReturnCode.TASK_ABORTED). On failure it should return an error reply (e.g. ReturnCode.EXECUTION_EXCEPTION) rather than raise; exceptions that do escape are converted to EXECUTION_EXCEPTION replies by the executor, except UnsafeJobError which the executor lets propagate so ClientRunner can apply its dedicated UNSAFE_JOB handling.

Parameters:

• task_name – name of the task.

• shareable – the task data.

• fl_ctx – the FLContext of the task.

• abort_signal – checked during execution; triggered means the task is aborted.

Returns:

The result Shareable (an error reply on failure/abort - never None).

abstract finalize(fl_ctx: FLContext) → None

Releases backend resources. Called when the executor handles END_RUN.

The backend tears down per its mode’s lifecycle ownership: stop the in-process trainer thread; send SHUTDOWN and stop the owned process tree (honoring the executor’s shutdown_timeout and stop_grace_period, and pending payload terminal state) for external_process.

Contract: must be idempotent and must not raise. Not called if initialize() raised (see the cleanup-on-failure contract on initialize()).

Parameters:

fl_ctx – the FLContext of the END_RUN event.

abstract initialize(context: ClientAPIBackendContext, fl_ctx: FLContext) → None

Prepares the backend for the run. Called once when the executor handles START_RUN.

context is the frozen snapshot of the executor’s configuration plus a back-reference to the executor (for fire_log_analytics and logging). A backend should read all of its config from context rather than from executor private attributes, and should retain what it needs for execute/finalize.

The backend sets up its control plane here (DataBus wiring for in_process; Cell session machinery, bootstrap config, and - per launch_once policy - trainer launch for external_process).

Contract: raise an exception on any setup failure. The executor converts the exception into system_panic so the job fails cleanly instead of hanging while tasks wait on a backend that never became ready.

Cleanup-on-failure contract: initialize() must be exception-safe and self-unwinding - if it raises, it must first release any partial setup it already made (threads started, processes launched, listeners/tokens registered, files written). The executor does NOT call finalize() on a backend whose initialize() raised, because finalize() cannot assume a consistently half-initialized backend. Own your own rollback.

Parameters:

• context – the frozen backend configuration and executor back-reference.

• fl_ctx – the FLContext of the START_RUN event.