.. _confidential_azure_container_instances_deployment:

#######################################################
Confidential Azure Container Instances Deployment Guide
#######################################################

Overview
========

This guide will walk you through the complete deployment of confidential Azure Container Instances (ACI).
This type of deployment is used to enable NVFlare servers to perform secure aggregation by utilizing features in confidential ACI, such as disabling login,
verifying container images before launching instances, and other confidential computing features.

.. note::

   Launching confidential ACI requires your Azure account to have certain permissions.  Please consult your Azure account and Azure for more information.
   
   
Steps for Launching The NVFlare Server in Confidential ACI
===============================================================

* Login and create one resource group
* Create one Azure container registry
* Build container images
* Publish container images to Azure container registry
* Create and run the confidential ACI launch script

Login and create one resource group
--------------------------------------------

First, you have to login to Azure with az cli.  Then you can create one resource group to host all resources
generated by the following operations.

You can choose another name for the resource group and another location.

.. code-block:: bash

   #!/usr/bin/env bash

   resource_group=cc-prep-rsr-grp
   location=eastus

   az login
   
   az group create --name $resource_group --location $location


Create One Azure Container Registry (ACR)
-----------------------------------------

With the resource group created, we first create one Azure Container Registry (ACR) which our built images are pushed to and pulled from.


.. code-block:: bash

   #!/usr/bin/env bash

   resource_group=cc-prep-rsr-grp
   location=eastus
   reg_name=ccprepreg
   dnl_scope=Unsecure

   az group create --name $resource_group --location $location

   az acr create --resource-group $resource_group \
   --name $reg_name --sku Standard \
   --dnl-scope $dnl_scope

.. note::

   As at the later steps, it is required to operate the Azure Container Registry (ACR) with higher permissions, 
   please check if you have at least "Contributor" role on the ACR.  Roles with lesser permissions may cause the following steps to fail.

Build Docker Container Images
--------------------------------------

The container images we are going to build will include the startup kit for the NVFlare server.  Therefore, please
obtain that set of files and copy the startup kit to one folder inside the current working directory.  In the following
example, the server's startup kit and the Dockerfile are stored in nvflserver.eastus.azurecontainer.io
folder and docker folder under current working folder, respectively, as shown below.

.. code-block ::

   $ tree -d -L 1
   .
   ├── docker
   └── nvflserver.eastus.azurecontainer.io


.. code-block:: bash

   #!/usr/bin/env bash

   tag=0.0.1
   name=cc_prep
   reg_name=ccprepreg
   registry=${reg_name}.azurecr.io
   nvfl_root=nvflserver.eastus.azurecontainer.io

   docker build --build-arg NVFL_ROOT=$nvfl_root -t $registry/$name:$tag -f docker/Dockerfile .

.. code-block:: dockerfile

   FROM python:3.10
   ARG NVFL_ROOT=nvflserver.eastus.azurecontainer.io
   WORKDIR /workspace
   RUN python3 -m pip install --no-cache nvflare
   COPY $NVFL_ROOT nvflare


The following is a sample cc_server.yml file, which is used with project.yml for cc provision.  A sample project.yml file
is shown after the cc_server.yml file.  Note this sample project.yml file also includes cc_site-1.yml, as described in
:ref:`azure_confidential_virtual_machine_deployment` - Creating Azure confidential virtual machines

.. code-block:: yaml

  compute_env: azure_confidential_container
  cc_cpu_mechanism: amd_sev_snp
  role: server
  cc_issuers:
    - id: aci_authorizer
      path: nvflare.app_opt.confidential_computing.aci_authorizer.ACIAuthorizer
      token_expiration: 100 # seconds, needs to be less than check_frequency

The following is the sample project.yml file.

.. code-block:: yaml

  api_version: 3
  name: example_project
  description: NVIDIA FLARE sample project yaml file
  participants:
    # Change the name of the server (server1) to the Fully Qualified Domain Name
    # (FQDN) of the server, for example: server1.example.com.
    # Ensure that the FQDN is correctly mapped in the /etc/hosts file.
    - name: server1
      type: server
      org: nvidia
      fed_learn_port: 8002
      cc_config: cc_server.yml
    - name: site-1
      type: client
      org: nvidia
      cc_config: cc_site-1.yml
      # Specifying listening_host will enable the creation of one pair of
      # certificate/private key for this client, allowing the client to function
      # as a server for 3rd-party integration.
      # The value must be a hostname that the external trainer can reach via the network.
      # listening_host: site-1-lh
    - name: admin@nvidia.com
      type: admin
      org: nvidia
      role: project_admin
  # The same methods in all builders are called in their order defined in builders section
  builders:
    - path: nvflare.lighter.impl.workspace.WorkspaceBuilder
    - path: nvflare.lighter.impl.static_file.StaticFileBuilder
      args:
        # config_folder can be set to inform NVIDIA FLARE where to get configuration
        config_folder: config
        # scheme for communication driver (currently supporting the default, grpc, only).
        # scheme: grpc

        # app_validator is used to verify if uploaded app has proper structures
        # if not set, no app_validator is included in fed_server.json
        # app_validator: PATH_TO_YOUR_OWN_APP_VALIDATOR
    - path: nvflare.lighter.impl.cert.CertBuilder
    - path: nvflare.lighter.cc_provision.impl.cc.CCBuilder
    - path: nvflare.lighter.impl.signature.SignatureBuilder

Publish Container Images to Azure Container Registry
------------------------------------------------------------

After the above steps, we have one container image built and stored locally.  Now we will push it to ACR we created in step 2.  This step
requires you to obtain an access token from ACR and to use it to login to ACR.


.. code-block:: bash

   #!/usr/bin/env bash

   reg_name=ccprepreg
   reg_token_file=reg_token.json

   az acr login --name $reg_name --expose-token > $reg_token_file 

   echo "ACR reg token saved to $reg_token_file"


With the token file available, you can log in to the ACR.

.. code-block:: bash

   #!/usr/bin/env bash

   reg_name=ccprepreg
   registry=${reg_name}.azurecr.io
   reg_token_file=reg_token.json

   reg_token=$(jq -r .accessToken $reg_token_file)

   docker login $registry -u 00000000-0000-0000-0000-000000000000 -p $reg_token

Then you can push your newly built container image to the ACR.

.. code-block:: bash

   #!/usr/bin/env bash

   tag=0.0.1
   name=cc_prep
   reg_name=ccprepreg
   registry=${reg_name}.azurecr.io
   
   docker push $registry/$name:$tag
   docker push $registry/skr:2.7

.. note::

   The skr:2.7 is built from Microsoft open source project at https://github.com/microsoft/confidential-sidecar-containers.
   Please check its document on how to build the skr image and rename it with the registry name.
   
   
Create And Run The Confidential ACI Launch Script
------------------------------------------------------------

This step requires the confcom extension for az cli.  You can install it with this command.

.. code-block:: bash

   az extension add --name confcom


The following is the script to properly inject the credential and verification
information of container images into the Azure Resource Manager (ARM) template file, cce_done_p.json in this case.


.. code-block:: bash

   #!/usr/bin/env bash

   resource_group=cc-prep-rsr-grp
   base_file=ccprep.json
   reg_token_file=reg_token.json
   export registry_token=$(jq -r .accessToken $reg_token_file)

   tmp=$(jq . $base_file)
   tmp=$(echo $tmp | jq '.resources[0].properties.imageRegistryCredentials[0].password = env.registry_token')
   echo $tmp > tmp.json

   az confcom acipolicygen -a tmp.json --print-policy > cce_token.b64
   
   export cce_token=$(cat cce_token.b64)
   cce_done=$(echo $tmp | jq '.resources[0].properties.confidentialComputeProperties.ccePolicy = env.cce_token')

   echo $cce_done > cce_done_p.json

   az deployment group create --resource-group $resource_group --template-file cce_done_p.json

The script requires a base ARM template file, as shown below.  All token or verification information are removed in this sample file.  However, you need
to edit it to match the information used in your confidential ACI and NVFlare server configuration.  Please note the port number, location, container image names and
confidential ACI resources must be updated according to your own account and previous settings.

If you encounter permission issues or the script gets stuck, please check your role in the ACR.

.. literalinclude:: ../../../resources/ccprep.json
   :language: json


This script will take a few minutes to finish.  After it's finished, you can check the confidential ACI is up and running in the Container Instances page of Azure.