# FL Experiment Tracking with TensorBoard Streaming ## Introduction In this exercise, you will learn how to stream TensorBoard events from the clients to the server in order to visualize live training metrics from a central place on the server. This exercise will be working with the `tensorboard` example in the advanced examples folder under experiment-tracking, which builds upon [Hello PyTorch with the Job API](hello_pt_job_api.md) by adding TensorBoard streaming. The setup of this exercise consists of one **server** and two **clients**. #### NOTE This exercise differs from [Hello PyTorch with the Job API](hello_pt_job_api.md), as it uses the `Learner` API along with the `LearnerExecutor`. In short, the execution flow is abstracted away into the `LearnerExecutor`, allowing you to only need to implement the required methods in the `Learner` class. For more about those APIs, see [`Learner`](../apidocs/nvflare.app_common.abstract.learner_spec.md#nvflare.app_common.abstract.learner_spec.Learner) and [`LearnerExecutor`](../apidocs/nvflare.app_common.executors.learner_executor.md#nvflare.app_common.executors.learner_executor.LearnerExecutor). Let’s get started. Make sure you have an environment with NVIDIA FLARE installed as described in [Quick Start Series](../quickstart.md#getting-started). First clone the repo: ```shell $ git clone https://github.com/NVIDIA/NVFlare.git ``` Now remember to activate your NVIDIA FLARE Python virtual environment from the installation guide. And install the required dependencies in the example folder (NVFlare/examples/advanced/experiment-tracking/tensorboard). ```shell (nvflare-env) $ python3 -m pip install -r requirements.txt ``` ## Adding TensorBoard Streaming to Configurations Inside the example, job configuration and TensorBoard setup are defined in: - [job.py](https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/experiment-tracking/tensorboard/job.py) - [client.py](https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/experiment-tracking/tensorboard/client.py) Take a look at the components section of the client config at line 24. The first component is the `pt_learner` which contains the initialization, training, and validation logic. `learner_with_tb.py` (under NVFlare/examples/advanced/experiment-tracking/pt) is where we will add our TensorBoard streaming changes. Next we have the [`TBWriter`](../apidocs/nvflare.app_opt.tracking.tb.tb_writer.md#nvflare.app_opt.tracking.tb.tb_writer.TBWriter), which implements some common methods that follow the signatures from the PyTorch SummaryWriter. This makes it easy for the `pt_learner` to log metrics and send events. Finally, we have the [`ConvertToFedEvent`](../apidocs/nvflare.app_common.widgets.convert_to_fed_event.md#nvflare.app_common.widgets.convert_to_fed_event.ConvertToFedEvent), which converts local events to federated events. This changes the event `analytix_log_stats` into a fed event `fed.analytix_log_stats`, which will then be streamed from the clients to the server. Under the component section in the server config, we have the `TBAnalyticsReceiver` of type [`AnalyticsReceiver`](../apidocs/nvflare.app_common.widgets.streaming.md#nvflare.app_common.widgets.streaming.AnalyticsReceiver). This component receives TensorBoard events from the clients and saves them to a specified folder (default `tb_events`) under the server’s run folder. Notice how the accepted event type `"fed.analytix_log_stats"` matches the output of [`ConvertToFedEvent`](../apidocs/nvflare.app_common.widgets.convert_to_fed_event.md#nvflare.app_common.widgets.convert_to_fed_event.ConvertToFedEvent) in the client config. ## Adding TensorBoard Streaming to your Code In this exercise, TensorBoard logging is implemented in: - [client.py](https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/experiment-tracking/tensorboard/client.py) First we must initialize our TensorBoard writer to the `AnalyticsSender` we defined in the client config: The `LearnerExecutor` passes in the component dictionary into the `parts` parameter of `initialize()`. We can then access the `AnalyticsSender` component we defined in `config_fed_client.json` by using the `self.analytic_sender_id` as the key in the `parts` dictionary. Note that `self.analytic_sender_id` defaults to `"analytic_sender"`, but we can also define it in the client config to be passed into the constructor. Now that our TensorBoard writer is set to `AnalyticsSender`, we can write and stream training metrics to the server in `local_train()`: The script uses `add_scalar(tag, scalar, global_step)` to send training metrics and validation accuracy. You can learn more about other supported writer methods in [`AnalyticsSender`](../apidocs/nvflare.app_common.widgets.streaming.md#nvflare.app_common.widgets.streaming.AnalyticsSender). ## Train the Model, Federated! Now you can use admin command prompt to submit and start this example job. To do this on a proof of concept local FL system, follow the sections [Setting Up the Application Environment in POC Mode](../user_guide/data_scientist_guide/poc.md#setting-up-poc) and [Starting the Application Environment in POC Mode](../user_guide/data_scientist_guide/poc.md#starting-poc) if you have not already. ### Running the FL System With the admin client command prompt successfully connected and logged in, enter the command below. ```default > submit_job tensorboard-streaming ``` Pay close attention to what happens in each of four terminals. You can see how the admin submits the job to the server and how the [`JobRunner`](../apidocs/nvflare.private.fed.server.job_runner.md#nvflare.private.fed.server.job_runner.JobRunner) on the server automatically picks up the job to deploy and start the run. This command uploads the job configuration from the admin client to the server. A job id will be returned, and we can use that id to access job information. #### NOTE If we use submit_job [app] then that app will be treated as a single app job. From time to time, you can issue `check_status server` in the admin client to check the entire training progress. You should now see how the training does in the very first terminal (the one that started the server). ## Viewing the TensorBoard Dashboard during Training On the client side, the `AnalyticsSender` works as a TensorBoard SummaryWriter. Instead of writing to TB files, it actually generates NVFLARE events of type `analytix_log_stats`. The `ConvertToFedEvent` widget will turn the event `analytix_log_stats` into a fed event `fed.analytix_log_stats`, which will be delivered to the server side. On the server side, the `TBAnalyticsReceiver` is configured to process `fed.analytix_log_stats` events, which writes received TB data into appropriate TB files on the server (defaults to `server/[JOB ID]/tb_events`). To view training metrics that are being streamed to the server, run: ```shell tensorboard --logdir=poc/server/[JOB ID]/tb_events ``` #### NOTE if the server is running on a remote machine, use port forwarding to view the TensorBoard dashboard in a browser. For example: ```shell ssh -L {local_machine_port}:127.0.0.1:6006 user@server_ip ``` #### ATTENTION The `server/[JOB ID]` folder only exists when job is running. After the job is finished, please use download_job [JOB ID] to get the workspace data as explained below. ### Accessing the results The results of each job will usually be stored inside the server side workspace. Please refer to [access server-side workspace](../user_guide/core_concepts/workspace.md#access-server-workspace) for accessing the server side workspace. ### Shutdown FL system Once the FL run is complete and the server has successfully aggregated the client’s results after all the rounds, and cross site model evaluation is finished, run the following commands in the fl_admin to shutdown the system (while inputting `admin` when prompted with password): ```shell > shutdown client > shutdown server > bye ``` Congratulations! Now you will be able to see the live training metrics of each client from a central place on the server. The full source code for this exercise can be found in [examples/advanced/experiment-tracking/tensorboard](https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/experiment-tracking/tensorboard). ## Previous Versions of TensorBoard Streaming > - [tensorboard-streaming for 2.3](https://github.com/NVIDIA/NVFlare/tree/2.3/examples/advanced/experiment-tracking/tensorboard-streaming)