Commmand for Proof Of Concept (POC) Mode¶
Introduction to the POC Command¶
The POC command allows users to try out the features of FLARE in a proof of concept deployment on a single machine.
In release 2.2, the poc command has been updated to make it easier to use.
Syntax and Usage¶
nvflare poc -h
usage: nvflare poc [-h] [-n [NUMBER_OF_CLIENTS]] [-p [PACKAGE]] [-gpu [GPU [GPU ...]]]
[--prepare] [--start] [--stop] [--clean]
optional arguments:
-h, --help show this help message and exit
-n [NUMBER_OF_CLIENTS], --number_of_clients [NUMBER_OF_CLIENTS]
number of sites or clients, default to 2
-p [PACKAGE], --package [PACKAGE]
package directory, default to all = all packages, only used for
start/stop-poc commands when specified
-gpu [GPU [GPU ...]], --gpu [GPU [GPU ...]]
gpu device ids will be used as CUDA_VISIBLE_DEVICES. used for poc
start command
--prepare prepare poc workspace
--start start poc
--stop stop poc
--clean cleanup poc workspace
Note
After you clone the NVFlare GitHub repository, you may optionally define an NVFLARE_HOME environment variable to point to the local NVFlare directory so a symbolic link is created to point the transfer directory to the examples in the code base. For example, if the the NVFlare GitHub repository is cloned under ~/projects, then you should set NVFLARE_HOME=~/projects/NVFlare. If the NVFLARE_HOME environment variable is not set, you will need to manually copy the examples to the transfer directory.
Setup poc workspace¶
nvflare poc --prepare
prepare_poc at /tmp/nvflare/poc for 2 clients
This will delete poc folder in current directory and create a new one. Is it OK to proceed? (y/N) y
Successfully creating poc folder at /tmp/nvflare/poc. Please read poc/Readme.rst for user guide.
WARNING:
******* Files generated by this poc command are NOT intended for production environments.
link examples from /path_to_NVFlare/NVFlare/examples to /tmp/nvflare/poc/admin/transfer
This is similar to old command
poc -n 2
except that workspace is defaulted to “/tmp/nvflare/poc”, and the default of 2 clients is not specified.
Note
Examples were automatically linked from /path_to_NVFlare/NVFlare/examples to /tmp/nvflare/poc/admin/transfer in the output above. The transfer directory was set up as a symlink to examples using the NVFLARE_HOME environment variable.
Replace the default POC workspace¶
You can change the default POC workspace to any location. All you need to do is to set environment variable NVFLARE_POC_WORKSPACE:
NVFLARE_POC_WORKSPACE="/tmp/nvflare/poc2"
In this example, the default workspace is set to the location “/tmp/nvfalre/poc2”.
Now running the following command:
nvflare poc --prepare
will generate the POC startup startup kits in the workspace “/tmp/nvfalre/poc2”.
Start Package(s)¶
Once the startup kits are generated with the prepare command, they are ready to be started. If you prepared the POC startup kits using default workspace, then you need to start with the same default workspace, otherwise, you need to specify the workspace.
Start ALL packages¶
Running the following command:
nvflare poc --start
will start ALL clients (site-1, site-2) and server as well as FLARE Console (aka Admin Client) located in the default workspace=”/tmp/nvflare/poc”.
Example Output
start_poc at /tmp/nvflare/poc, gpu_ids=[0], excluded = [], white_list=[]
start: package: server, executing /tmp/nvflare/poc/server/startup/start.sh
WORKSPACE set to /tmp/nvflare/poc/server/startup/..
start: package: site-2, executing /tmp/nvflare/poc/site-2/startup/start.sh
No sp_end_point is provided. Default sp_end_point (localhost:8002:8003) is used
Usage: start.sh <SP_END_POINT <CLIENT_NAME>>
WORKSPACE set to /tmp/nvflare/poc/site-2/startup/..
PYTHONPATH is /local/custom::/workspace/repos/NVFlare_experiment
start fl because of no pid.fl
new pid 12474
start: package: site-1, executing /tmp/nvflare/poc/site-1/startup/start.sh
No sp_end_point is provided. Default sp_end_point (localhost:8002:8003) is used
Usage: start.sh <SP_END_POINT <CLIENT_NAME>>
2022-10-17 15:20:46,601 - FederatedServer - INFO - starting insecure server at localhost:8002
2022-10-17 15:20:46,603 - FederatedServer - INFO - Got the primary sp: localhost fl_port: 8002 SSID: ebc6125d-0a56-4688-9b08-355fe9e4d61a. Turning to hot.
deployed FL server trainer.
2022-10-17 15:20:46,666 - nvflare.fuel.hci.server.hci - INFO - Starting Admin Server localhost on Port 8003
2022-10-17 15:20:46,666 - root - INFO - Server started
WORKSPACE set to /tmp/nvflare/poc/site-1/startup/..
PYTHONPATH is /local/custom::/workspace/repos/NVFlare_experiment
start: package: admin, executing /tmp/nvflare/poc/admin/startup/fl_admin.sh
/tmp/nvflare/poc/admin/startup
Waiting for SP....
2022-10-17 15:20:50,300 - FederatedClient - INFO - Got the new primary SP: localhost:8002
Trying to obtain server address
Obtained server address: localhost:8003
Trying to login, please wait ...
Logged into server at localhost:8003
Type ? to list commands; type "? cmdName" to show usage of a command.
> 2022-10-17 15:20:51,393 - ClientManager - INFO - Client: New client site-2@127.0.0.1 joined. Sent token: c0fff401-ca5e-4c2c-9168-da45e07588a2. Total clients: 1
2022-10-17 15:20:51,394 - FederatedClient - INFO - Successfully registered client:site-2 for project example_project. Token:c0fff401-ca5e-4c2c-9168-da45e07588a2 SSID:ebc6125d-0a56-4688-9b08-355fe9e4d61a
Waiting for SP....
2022-10-17 15:20:53,227 - FederatedClient - INFO - Got the new primary SP: localhost:8002
2022-10-17 15:20:54,306 - ClientManager - INFO - Client: New client site-1@127.0.0.1 joined. Sent token: 89a09d00-06cd-41d0-a382-0d654ce5aea9. Total clients: 2
2022-10-17 15:20:54,307 - FederatedClient - INFO - Successfully registered client:site-1 for project example_project. Token:89a09d00-06cd-41d0-a382-0d654ce5aea9 SSID:ebc6125d-0a56-4688-9b08-355fe9e4d61a
Note
If you run nvflare poc --start before prepare, you will get the following error:
start_poc at /tmp/nvflare/poc, white_list=[] workspace /tmp/nvflare/poc is not ready, please use poc --prepare to prepare poc workspace
Note
If you run nvflare poc --start after having already started the server or any of the clients, you will get errors like:
There seems to be one instance, pid=12458, running. If you are sure it's not the case, please kill process 12458 and then remove daemon_pid.fl in /tmp/nvflare/poc/server/startup/..There seems to be one instance, pid=12468, running. If you are sure it's not the case, please kill process 12468.
Note
If you prefer to have the FLARE Console on a different terminal, you can use nvflare poc --start -ex admin.
Start the server only¶
nvflare poc --start -p server
An example of successful output for starting a server:
WORKSPACE set to /path_to_WORKSPACE
start fl because of no pid.fl
new pid 4191598
2022-10-17 14:46:35,413 - FederatedServer - INFO - starting secure server at keubuntu:8002
2022-10-17 14:46:35,415 - FederatedServer - INFO - Got the primary sp: keubuntu fl_port: 8002 SSID: ebc6125d-0a56-4688-9b08-355fe9e4d61a. Turning to hot.
deployed FL server trainer.
2022-10-17 14:46:35,421 - nvflare.fuel.hci.server.hci - INFO - Starting Admin Server keubuntu on Port 8003
2022-10-17 14:46:35,421 - root - INFO - Server started
Start the FLARE Console (previously called the Admin Client)¶
nvflare poc --start -p admin
Start Clients with GPU assignment¶
If the user provides the GPU device IDs and in certain order, such as:
nvflare poc -gpu 1 0 0 2 --start
The system will try to match the clients with the given GPU devices order, in this example, site-1 with GPU_id = 1, site-2 with GPU_id = 0, site-3 with GPU_id = 0 and site-4 with GPU_id = 2.
If the GPU ID does not exist on the client machine, you will get an error like:
gpu_id provided is not available in the host machine, available GPUs are [0]
If no GPU id is specified, the host GPU ID will be used if available.
If there is no GPU, then there will be no assignments. If there are GPUs, they will be assigned to clients automatically.
Tip
You can check the GPUs available with the following command (assuming you have NVIDIA GPUs with drivers installed):
nvidia-smi --list-gpus
Stop Package(s)¶
To stop packages, issue the command:
nvflare poc --stop
Similarly, you can stop a specific package, for example:
nvflare poc --stop -p server
Note that you may need to exit the FLARE Console yourself.
Clean up¶
There is a command to clean up the POC workspace added in version 2.2 that will delete the POC workspaces:
nvflare poc --clean