Deploy Command

nvflare deploy prepares already-created server or client startup kits for a site deployment runtime. The first supported subcommand is nvflare deploy prepare.

deploy prepare does not create identities, certificates, startup kits, or Kubernetes clusters. Use nvflare provision or the distributed nvflare cert / nvflare package workflow first, then run deploy prepare on each server or client kit that should run in Docker, Kubernetes, or Slurm.

Note

The Kubernetes and Slurm runtimes secure the internal parent/job links (SP/SJ and CP/CJ) with mTLS by default, and these links use the participant certificate in both TLS roles. The certificate must therefore allow both clientAuth and serverAuth in its extended key usage (EKU). Startup kits whose certificates are restricted to a single role — for example, kits created by NVFlare 2.8 distributed provisioning, which issues certificates carrying only clientAuth or only serverAuth — must be re-provisioned before using these runtimes. Kits whose certificates carry no EKU (unrestricted) remain compatible and do not need re-provisioning.

Note

In secure mode, jobs run on per-job certificates issued by the server, and these runtimes never ship the site private keys to job processes. The server startup kit must contain the job CA (job_ca.crt / job_ca.key), which nvflare provision adds by default. Server kits provisioned before this feature must be re-provisioned or every job deploy fails.

For Kubernetes deployment workflow, see Running FLARE in Kubernetes. For the Slurm deployment workflow and security checklist, see Slurm Job Launcher. For job-level runtime settings, see Launcher-Specific Execution Settings.

Usage

nvflare deploy prepare <startup-kit-dir> [--output <prepared-kit-dir>] [--config <runtime-config.yaml>]

Arguments:

  • <startup-kit-dir>: existing server or client startup kit directory.

  • --kit: named-option alias for <startup-kit-dir>.

  • --output: directory for the prepared kit copy. Defaults to <startup-kit-dir>/prepared/<runtime>.

  • --config: YAML runtime config. Defaults to <startup-kit-dir>/config.yaml.

  • --schema: print command schema as JSON and exit.

Default convention:

Put the runtime config in the startup kit as config.yaml, then run nvflare deploy prepare <startup-kit-dir>. The command reads runtime from that config and writes the prepared copy to <startup-kit-dir>/prepared/<runtime> (for example, docker, k8s, or slurm). Use --config to read a config file from another path, and use --output to write the prepared kit somewhere else.

Admin startup kits are not supported by deploy prepare because admin kits do not run parent server or client processes.

The input kit is treated as read-only. Runtime-specific files are written to the prepared output directory.

Note

Docker and Kubernetes preparation support only the TCP internal transport. If the input kit’s local/comm_config.json explicitly sets internal.scheme to shared-file, deploy prepare stops with an INVALID_KIT error instead of rewriting the scheme to TCP. Run the original kit in process mode or use the Slurm runtime for shared-file transport (see Shared-File Worker Channel).

Docker Config

Example docker.yaml:

runtime: docker

parent:
  docker_image: registry.example.com/nvflare-site:2.8
  network: nvflare-network
  internal_connection_security: mtls

job_launcher:
  default_python_path: /usr/local/bin/python
  default_job_env:
    NCCL_P2P_DISABLE: "1"
  default_job_container_kwargs:
    shm_size: 8g
    ipc_mode: host

Top-level keys:

  • runtime: required, must be docker.

  • parent: required mapping for the parent server/client container.

  • job_launcher: optional mapping for per-job Docker container defaults.

parent keys:

  • docker_image: required parent image used by startup/start_docker.sh.

  • network: Docker network for parent and job containers. Defaults to nvflare-network.

  • internal_connection_security: security mode for internal parent/job TCP links (SP/SJ and CP/CJ). Accepts mtls or clear and defaults to mtls. Use clear only as an explicit insecure opt-out because it removes certificate authentication while the parent port remains reachable from containers attached to the Docker network.

job_launcher keys:

  • default_python_path: Python executable used in job containers unless a job overrides it with launcher_spec[site]["docker"]["python_path"].

  • default_job_env: environment variables injected into every Docker job container.

  • default_job_container_kwargs: Docker SDK container kwargs applied to every job container. Launcher-controlled keys such as volumes, mounts, network, environment, command, name, detach, auto_remove, user, working_dir, and image are rejected. For a site default job image, set studies.<study>.container.image in local/study_runtime.yaml.

Prepare and start:

nvflare deploy prepare ./site-1 --config docker.yaml --output ./site-1-docker
cd ./site-1-docker
./startup/start_docker.sh

The command writes:

  • startup/start_docker.sh

  • patched local/resources.json.default with DockerJobLauncher

  • patched local/comm_config.json

  • local/study_runtime.yaml template when missing (skipped for legacy kits that already have study_data.yaml)

The Docker launcher preserves stcp:// while replacing the parent hostname with its Docker DNS name. It rejects URL and connection-security mismatches instead of silently downgrading an mTLS parent connection to clear TCP.

K8s Config

Example k8s.yaml:

runtime: k8s
namespace: nvflare

parent:
  docker_image: registry.example.com/nvflare-site:2.8
  internal_connection_security: mtls
  image_pull_secrets:
    - registry-credentials
  parent_port: 8102
  workspace_pvc: nvflws
  workspace_mount_path: /var/tmp/nvflare/workspace
  python_path: /usr/local/bin/python3
  resources:
    requests:
      cpu: "2"
      memory: 8Gi
  pod_security_context: {}

job_launcher:
  config_file_path: null
  pending_timeout: 300
  default_python_path: /usr/local/bin/python3
  image_pull_secrets:
    - job-registry-credentials
  job_pod_security_context: {}

Top-level keys:

  • runtime: required, must be k8s.

  • namespace: Kubernetes namespace for parent and job pods. Defaults to default.

  • server_service_name: optional Kubernetes Service name for the FL server.

  • parent: required mapping for the generated parent Helm chart.

  • job_launcher: optional mapping for dynamically launched job pods.

parent keys:

  • docker_image: required parent image used by the Helm chart.

  • internal_connection_security: security mode for internal parent/job TCP links (SP/SJ and CP/CJ). Accepts mtls or clear and defaults to mtls. Use clear only as an explicit insecure opt-out because it removes certificate authentication from these links.

  • image_pull_secrets: optional list of existing Kubernetes Secret names to render as imagePullSecrets on the parent server/client pod. Create these registry pull Secrets in the target namespace before installing the chart. This setting applies to the generated parent pod chart. Use job_launcher.image_pull_secrets for dynamically launched job pods.

  • parent_port: port that job pods use to reach the parent pod’s FLARE process. Defaults to 8102.

  • workspace_pvc: PVC claim containing the runtime workspace. Defaults to nvflws.

  • workspace_mount_path: parent pod workspace mount path. Defaults to /var/tmp/nvflare/workspace. This is also written into the Kubernetes job launcher config so job pods use the same in-container workspace path.

  • python_path: Python executable used by the parent pod command. Defaults to /usr/local/bin/python3.

  • resources: parent pod resources rendered into values.yaml.

  • pod_security_context: parent pod security context rendered into values.yaml.

job_launcher keys:

  • config_file_path: kubeconfig path used by K8sJobLauncher. Use null for in-cluster config, where the Kubernetes Python client uses the pod’s ServiceAccount token.

  • pending_timeout: seconds to wait for a job pod to leave Pending.

  • default_python_path: Python executable used in job pods unless a job overrides it with launcher_spec[site]["k8s"]["python_path"]. Defaults to /usr/local/bin/python3.

  • image_pull_secrets: optional list of existing Kubernetes Secret names attached to every dynamically launched job pod for this prepared site. This is configured by the deployment owner and does not require job authors to add registry Secret names to meta.json.

  • job_pod_security_context: security context passed to dynamically launched job pods.

Study-specific Pod templates are not launcher arguments. Configure them per study in local/study_runtime.yaml (studies.<study>.pod_template, inline or as a path relative to local/). Matching studies use the template with launcher-owned fields overlaid; template volumes or job-container mounts named workspace-job or startup-kit are replaced by the launcher-generated workspace and startup mounts.

Prepare the parent server or client kit first:

nvflare deploy prepare ./site-1 --config k8s.yaml --output ./site-1-k8s

After deploy prepare and before staging or starting the parent pod, deployment owners may edit the generated local/study_runtime.yaml to configure per-study datasets, env vars, secrets, and Pod templates in one auto-discovered file — no launcher arguments are needed. For legacy kits that still carry a v1 local/study_data.yaml, the generated K8s launcher config instead sets study_data_pvc_file_path to <workspace_mount_path>/local/study_data.yaml so existing data mounts keep working; the two files must not coexist. Stage or copy any referenced files under local/ so the parent process can read them at the in-pod paths.

Then choose one of the following two staging methods before starting the parent pod with Helm.

Method 1: copy ``startup/`` and ``local/`` into the workspace PVC.

Copy the prepared kit’s startup/ and local/ directories into the root of the configured workspace PVC. The chart mounts that PVC at workspace_mount_path. The OpenShift helper examples/devops/openshift/scripts/k8s_deploy.sh is a complete scripted example of this PVC-copy method.

After the copy is complete, install or upgrade the chart:

helm upgrade --install site-1 ./site-1-k8s/helm_chart --namespace nvflare

Method 2: stage ``local/`` as a ConfigMap and ``startup/`` as a Secret.

Use nvflare deploy k8s stage to create the Kubernetes resources and patch the generated Helm values. nvflare deploy k8 stage is accepted as an alias.

nvflare deploy k8s stage ./site-1-k8s --namespace nvflare
helm upgrade --install site-1 ./site-1-k8s/helm_chart --namespace nvflare

This keeps the workspace PVC mounted at workspace_mount_path for writable runtime state, but the parent pod reads local/ from the generated ConfigMap and startup/ from the generated Secret.

nvflare deploy prepare also patches the prepared kit’s internal communication settings so dynamically launched job pods connect to the generated parent Kubernetes Service on parent_port. If you customize the chart’s Service name or port, keep that Service endpoint consistent with the prepared kit. By default, these TCP links use mTLS and preserve stcp:// through the job launcher and runtime arguments. Setting parent.internal_connection_security: clear instead emits tcp:// links without certificate authentication; receiver-side CellNet authorization still applies, but it does not replace peer authentication.

The command writes:

  • helm_chart/ for the parent server or client pod

  • patched local/resources.json.default with K8sJobLauncher

  • patched local/comm_config.json

  • local/study_runtime.yaml template when missing (skipped for legacy kits that already have study_data.yaml)

K8s Staging

nvflare deploy k8s stage creates Kubernetes resources from a prepared K8s kit and patches the generated Helm chart values to mount them:

nvflare deploy k8s stage <prepared-kit-dir> [--namespace <namespace>]

The command requires Kubernetes CLI access to the target cluster. It uses kubectl by default; set --kubectl oc or KUBECTL=oc when staging into OpenShift with oc. It:

  • creates or updates a ConfigMap containing every file under prepared local/

  • creates or updates a Secret containing every file under prepared startup/

  • patches helm_chart/values.yaml so the parent pod mounts the ConfigMap at workspace_mount_path/local and the Secret at workspace_mount_path/startup

  • records the resolved namespace and object names so they can be removed by nvflare deploy k8s unstage

The resource names default to nvflare-local-<site> and nvflare-startup-<site>. Override them with --local-configmap and --startup-secret. The namespace defaults to the namespace written into the prepared kit’s K8sJobLauncher config, or default when unavailable.

--namespace selects the target namespace; it does not create that namespace. Make sure the namespace already exists before staging. On initial staging, an explicit --namespace takes precedence over the namespace in the prepared kit. It controls where the ConfigMap and Secret are applied and the namespace in the printed Helm command, but it does not rewrite the K8sJobLauncher namespace in local/resources.json.default. Normally, use the same namespace that was configured for nvflare deploy prepare so the parent and dynamically launched job pods remain in the same namespace.

The stage command records the resolved namespace and resource names in helm_chart/values.yaml before invoking kubectl. Recording them first preserves the exact cleanup targets if kubectl succeeds, partially succeeds, or fails. A later stage command without overrides reuses these recorded values. Restaging with an explicit namespace or resource name that differs from a recorded value is rejected. For example:

Initial staging:
  stage --namespace old-ns
  -> values.yaml records old-ns
  -> kubectl may succeed, partially succeed, or fail

Restaging:
  stage --namespace new-ns
  -> rejected because values.yaml still records old-ns

In this case, the error includes Prepared kit is already staged with a different namespace. To change the namespace safely, uninstall any Helm release that uses the staged resources, unstage the prepared kit, and then stage it in the new namespace:

# After uninstalling any Helm release:
nvflare deploy k8 unstage <prepared-kit>

nvflare deploy k8 stage <prepared-kit> --namespace new-ns

After this staging command succeeds, run the printed helm_command or the equivalent helm upgrade --install command for the prepared chart to start the parent server or client pod. The command also prints a cleanup_command for use after Helm uninstall.

The generated Helm chart still mounts the configured workspace PVC at the workspace root. The ConfigMap and Secret only replace the local/ and startup/ subdirectories.

K8s Unstaging

The ConfigMap and Secret created by nvflare deploy k8s stage are not part of the generated Helm release. After uninstalling the release, run nvflare deploy k8s unstage so this staged participant identity Secret is not left in the cluster:

helm uninstall site-1 --namespace nvflare
nvflare deploy k8s unstage ./site-1-k8s

unstage reads the exact namespace and resource names recorded by the most recent stage command, deletes the Secret and ConfigMap, and clears their references from helm_chart/values.yaml. Deletion uses exact names and is safe when either object has already been removed.

Run unstage before replacing the same prepared output with another nvflare deploy prepare command. Prepare refuses to overwrite a chart that still records staged resources because doing so would lose their cleanup targets.

For a kit staged by an older NVFlare version that did not record its namespace, pass the original namespace explicitly:

nvflare deploy k8s unstage ./site-1-k8s --namespace nvflare

You can also pass --local-configmap and --startup-secret to clean up legacy or partially staged resources whose names are not recorded. Use --kubectl oc or KUBECTL=oc for OpenShift. Run unstage only after the Helm release has been uninstalled; an installed parent pod still depends on these volumes.

Slurm Config

The Slurm backend requires a stable shared workspace and a launcher policy. A minimal slurm.yaml is:

runtime: slurm
job_launcher:
  sandbox: apptainer
  image: /lustre/images/nvflare-prod.sif
  python_path: /usr/bin/python3
  parent_host: nvflare-site1.internal
  internal_connection_security: mtls

Prepare directly into the shared runtime workspace, then start the parent:

nvflare deploy prepare ./site-1 --config slurm.yaml --output /lustre/proj123/nvflare/site-1
/lustre/proj123/nvflare/site-1/startup/start_slurm.sh

The output is the live workspace and must be visible at the same absolute path on the parent and compute nodes. Preparing to the same output again replaces the complete workspace. A client kit can optionally generate startup/parent.slurm; prepare prints the direct sbatch command that runs it in an allocation. Internal TCP links use mTLS by default and preserve stcp:// while the launcher rewrites only the parent host and port. Set job_launcher.internal_connection_security: clear only as an explicit insecure opt-out; it emits tcp:// links without certificate authentication. Receiver-side CellNet authorization still applies, but it does not replace peer authentication. Shared-file transport is unchanged and remains clear. See Slurm Job Launcher for the complete guide.

Job Images

Docker, Kubernetes, and Slurm jobs can select a job image in meta.json. The preferred form is launcher_spec:

{
  "launcher_spec": {
    "default": {
      "docker": {"image": "registry.example.com/nvflare-job:2.8"},
      "k8s": {"image": "registry.example.com/nvflare-job:2.8"},
      "slurm": {"image": "/shared/images/nvflare-job.sif"}
    },
    "site-1": {
      "docker": {"shm_size": "8g"}
    }
  },
  "resource_spec": {
    "site-1": {
      "num_of_gpus": 1
    }
  }
}

launcher_spec["default"][mode] applies to every site for that mode. launcher_spec[site][mode] overrides the default for one site. Keep resource requests such as num_of_gpus in resource_spec.

A job-supplied image is executable content and requires the site’s normal BYOC authorization. Slurm resolves the effective image as job, then study container.image, then site job_launcher.image. Unlike registry image names used by Docker/Kubernetes, a Slurm image must be an absolute, site-visible existing file; see Slurm Job Launcher.

Exit Status

Validation errors exit with code 4 and report a structured error. Common causes include:

  • missing or invalid runtime config

  • unsupported admin startup kit

  • missing startup/ or local/ directory

  • invalid resources.json.default

  • reserved Docker launcher kwargs

  • --output pointing at or inside the input kit

  • a Slurm --output path that is not valid as a runtime workspace

  • a missing/non-executable Slurm parent CLI, invalid sandbox/image, or invalid Slurm internal_connection_security or server parent configuration