Cross Site Model Evaluation / Federated Evaluation¶
cross site model evaluation workflow
uses the data from clients to run evaluation with the models of other clients.
Data is not shared, rather the collection of models is distributed to each client site to run local validation. The
results of local validation are collected by the server to construct an all-to-all matrix of model performance vs.
The server’s global model is also distributed to each client for evaluation on the client’s local dataset for global model evaluation.
The hello-numpy-cross-val example is a simple
example that implements the
cross site model evaluation workflow.
Previously in NVFlare before version 2.0, cross-site validation was built into the framework itself, and there was an
admin command to retrieve cross-site validation results. In NVFlare 2.0, with the ability to have customized
workflows, cross-site validation is no longer in the NVFlare framework but is instead handled by the workflow. The
the cifar10 example is configured to run cross-site
model evaluation and
config_fed_server.json is configured with
to write the results to a JSON file on the server.
Example with Cross Site Model Evaluation / Federated Evaluation Workflow¶
See the Hello Numpy Cross-Site Validation for an example application with the cross site model evaulation / federated evaluation workflow.