Slurm Job Launcher

In this guide, the parent is the long-running NVFlare client parent process (CP) or server parent process (SP). It launches a separate client job process (CJ) or server job process (SJ) for each federated job. The Slurm job launcher submits each CJ or SJ as a Slurm batch job. Slurm selects nodes and enforces the allocation; NVFlare submits, monitors, and cancels jobs owned by the current parent.

The prepare host runs nvflare deploy prepare. The optional submission host runs sbatch to place a client parent in Slurm. The runtime parent host runs the CP or SP, and the compute nodes run its CJ or SJ allocations. A login or service host is a runtime parent host outside a Slurm allocation.

Choose an execution backend with job_launcher.sandbox:

Value	Execution	Use
apptainer	Launcher-managed Apptainer container	Isolation after validation on the target cluster
pyxis	Read-only Pyxis/Enroot container	Trusted container packaging
none	Python directly in the allocation	Site-trusted code; required for multi-node jobs

Prerequisites

Before starting a parent, verify:

• Slurm 23.02 or later provides working sbatch, squeue, sacct, and scancel commands on the runtime parent host. Parent bootstrap resolves these commands and verifies the version; 23.02 is the minimum because the launcher uses sbatch --export=NIL. Production sites should run a Slurm release that is still supported by SchedMD.

• slurmdbd accounting is enabled and sacct responds. The default AccountingStoreFlags is sufficient.

• The cluster is not federated, and submission plugins do not redirect jobs to another cluster.

• The parent uses a dedicated site account with a consistent numeric UID and compatible group access on the parent host, shared filesystem, and compute nodes.

• The runtime workspace, images, datasets, and secret-mount sources are visible at the same absolute paths on all participating nodes. The shared filesystem supports O_EXCL and atomic rename.

• Compute nodes can reach parent_host:internal_port. Multi-node jobs also require connectivity among their allocated nodes.

• Slurm partition, association, QOS, reservation, cgroup, and device policies enforce the site’s resource limits.

The prepare output is the runtime workspace. The parent and worker allocations must see it at the same absolute path. Every workspace must be private, owned by the runtime account, and dedicated to one NVFlare site or federation.

For Apptainer, enable unprivileged user namespaces and install Apptainer on all eligible nodes. Validate its filesystem, process, cgroup, and GPU isolation on the production cluster. For Pyxis, install and configure Pyxis/Enroot on all eligible nodes and ensure srun is available after setup.

The environment selected by python_path must contain a compatible NVFlare installation and the job dependencies. This applies to the host environment in bare mode and to the image for Apptainer and Pyxis. A PYTHONPATH override used only to start the parent does not install NVFlare in the worker environment.

The prepare, submission, and runtime parent hosts may differ. The prepare host does not need Slurm commands. The submission host that runs the generated submit_command needs sbatch on PATH. The runtime parent host needs all four parent commands after its service environment or parent.environment_setup has run.

Configure the Site

Create slurm.yaml beside the startup kit. This example runs the parent on a login or service host:

runtime: slurm
job_launcher:
  sandbox: apptainer
  image: /lustre/images/nvflare-prod.sif
  python_path: /usr/bin/python3
  parent_host: nvflare-site1.internal
  sbatch_directives:
    partition: fl-gpu
    account: proj123
    time: "12:00:00"
  setup: |
    source /etc/profile.d/modules.sh
    module load apptainer
  pending_timeout: 600

Important keys are:

Key	Meaning
sandbox	Required: apptainer, pyxis, or none.
image	Existing absolute image file required by Apptainer and Pyxis. Omit in bare mode.
python_path	Required absolute worker interpreter path in the selected execution environment, with a compatible NVFlare installation.
parent_host	Compute-reachable parent host. Required when the parent is not in a Slurm allocation.
sbatch_directives	Site defaults. Supported keys are partition, account, qos, time, constraint, and reservation.
setup	Trusted host-side Bash run in the batch job before the worker starts.
forward_env	Environment names whose post-setup values should reach the worker.
executables	Optional explicit paths for Slurm and backend commands. Parent-side paths are preserved in the workspace, then resolved and validated on the runtime parent host.
internal_port	Worker-to-parent port; default 8102.
poll_interval	Scheduler polling interval; default 10 seconds.
pending_timeout	Time limit starting at the first observed pending state; default 600 seconds. A job may lower it.

setup starts with the minimal environment from sbatch --export=NIL. Source module initialization explicitly. Put fixed values in study env; use forward_env only for values created by setup.

Prepare and Start

Prepare directly into the shared runtime workspace:

nvflare deploy prepare ./site-1 \
    --config ./slurm.yaml \
    --output /lustre/proj123/nvflare/site-1

Run one parent per workspace. Re-running prepare with the same output replaces the complete workspace. Stop the parent and preserve required runs, snapshots, and server job storage before updating it.

Start a parent on a login or service host:

/lustre/proj123/nvflare/site-1/startup/start_slurm.sh

To run a client parent in a Slurm allocation, add a client-only parent block and omit job_launcher.parent_host:

parent:
  sbatch_directives:
    partition: batch
    account: proj123
    time: "7-00:00:00"
  environment_setup: |
    source /lustre/proj123/venv/bin/activate

Then run the submit_command printed by prepare on a submission host where sbatch is on PATH. It has this form:

sbatch --parsable \
    --output=/lustre/proj123/nvflare/site-1/parent-slurm-%j.out \
    /lustre/proj123/nvflare/site-1/startup/parent.slurm

The parent script runs parent.environment_setup before starting NVFlare. Parent bootstrap then resolves sbatch, squeue, sacct, and scancel once and keeps their canonical paths in memory for that process. An explicitly configured path may therefore point to a cluster-managed stable symlink such as .../slurm/current/bin/sbatch; a restarted parent resolves a new target after a cluster upgrade without re-preparing the workspace.

An explicit parent_host always wins. Otherwise an allocated parent uses SLURMD_NODENAME. A parent outside an allocation without parent_host cannot launch jobs. NVFlare does not guess or resolve a host name.

Server kits reject parent. Run the server parent on a stable host with a stable external NVFlare federation endpoint.

Study Settings

Site-owned study settings belong in local/study_runtime.yaml and are re-read for every launch:

format_version: 2
studies:
  pathology:
    container:
      image: /lustre/images/pathology.sif
    datasets:
      slides:
        source: /lustre/data/pathology/slides
        mode: ro
    env:
      MODEL_FAMILY: vit-large
    secret_env:
      DB_PASSWORD:
        source: PATHOLOGY_DB_PASSWORD
    slurm:
      partition: fl-gpu-large
      account: pathology-project

Each dataset is mounted at /data/<study>/<dataset> inside the container; the example above mounts slides at /data/pathology/slides.

A study can override the site sandbox, setup, partition, account, and QOS. It can also provide an image, environment, dataset mounts, secret environment, and read-only secret mounts. A Slurm secret_env source names a variable in the parent environment; its value is passed through a temporary private file and is not written to the batch script or scheduler command.

Mount sources must be absolute paths outside the runtime workspace and must exist on the compute nodes that can run the job. Bare mode rejects container, dataset, and secret-mount settings because it has no mount namespace. Migrate legacy local/study_data.yaml files to study_runtime.yaml before using Slurm.

Job Settings

Jobs put portable GPU totals in resource_spec and Slurm-only settings in launcher_spec. This is a valid single-node container request:

{
  "resource_spec": {
    "site-1": {"num_of_gpus": 1}
  },
  "launcher_spec": {
    "site-1": {
      "slurm": {
        "image": "/shared/images/nvflare-job.sif",
        "cpus_per_node": 8,
        "mem_per_node": 32768,
        "time": "02:00:00",
        "pending_timeout": 300
      }
    }
  }
}

Supported job keys are image, nodes, gpus_per_node, cpus_per_node, mem_per_node (MiB), time, and pending_timeout. A job image requires normal BYOC authorization and takes precedence over the study and site images. Job and study images are rejected on any site whose effective sandbox is none.

Multi-node jobs require effective sandbox: none and must not specify an image. A positive multi-node num_of_gpus requires gpus_per_node; whenever both are supplied, num_of_gpus must equal nodes * gpus_per_node. The application owns any multi-node process fan-out.

Security and Operations

The worker-to-parent internal channel is clear TCP, matching the Kubernetes launcher. Use it only on a trusted or isolated site network. This does not change the configured security of the external NVFlare federation channel.

Working accounting is mandatory. The parent refuses to start if sacct is unavailable. A later scheduler or accounting outage leaves affected jobs non-terminal and retries; it never assumes that a missing observation means a job has stopped.

NVFlare stores transient launch artifacts under <prepare-output>/.nvflare_slurm. The live parent removes a job’s artifacts after launch failure or terminal completion. User abort and pending timeout verify ownership before scancel; normal framework shutdown terminates running handles through the same path before the Slurm launcher closes launch admission.

Job output is <run-dir>/slurm-<slurm-job-id>.out. Use squeue for live state and sacct for completed jobs. Preserve the complete workspace and relevant scheduler records for investigation.

After a parent crash, the launcher does not cancel surviving allocations or remove their job artifacts; this matches the Docker and Kubernetes launchers. A leftover job directory blocks relaunch of that job ID until an operator verifies that no old allocation uses it and removes the directory.

An sbatch timeout fails the FL dispatch even if Slurm accepted the job. Artifact removal prevents a pending allocation from starting unless the same job ID is relaunched before it starts.

To change the workspace, stop the parent and prepare to a new --output. Preparing to an existing output replaces it, so copy any data that must survive before running prepare.

After a server job starts, clients wait for its SJ to become available. The SJ’s Slurm queue and startup time must fit within the client-side max_runner_sync_timeout (60 seconds by default). Configure this value in the job’s client config_fed_client.json when the site needs a longer bound; see Timeout Troubleshooting Guide.

Before production use, test on the target cluster:

1. Successful, failed, timed-out, pending-aborted, and running-aborted jobs.

2. Parent restart with a live job and workspace replacement behavior.

3. Compute-node connectivity to the parent and multi-node collectives when used.

4. Slurm accounting, association, QOS, partition, cgroup, and GPU enforcement.

5. The selected backend’s filesystem view, environment, secret handling, exit status, and CPU/GPU device visibility.

Do not describe Apptainer as a production sandbox until those isolation checks pass on the deployment cluster.