Logging Configuration¶

FLARE uses python logging with the dictConfig API following the configuration dictionary schema. FLARE Loggers are designed to follow the package level hierarchy using dot separated logger names in order to faciliate granular control at different levels.

We provide a Default Logging Configuration file log_config.json.default for all NVFLARE sub-systems with pre-configured handlers for console level colors, logs, error logs, structured json logs, and fl training logs.

Overwrite the default configuration by Modifying Logging Configurations files, or change the logging configuration during runtime by using the Dynamic Logging Configuration Commands configure_site_log and configure_job_log.

Logging Configuration and Features¶

Default Logging Configuration¶

The default logging configuration json file (log_config.json.default, LogMode.FULL) is divided into 3 main sections: formatters, handlers, and loggers. This file can be found at log_config.json. See the configuration dictionary schema for more details.

{
    "version": 1,
    "disable_existing_loggers": false,
    "formatters": {
        "baseFormatter": {
            "()": "nvflare.fuel.utils.log_utils.BaseFormatter",
            "fmt": "%(asctime)s - %(name)s - %(levelname)s - %(fl_ctx)s - %(message)s"
        },
        "consoleFormatter": {
            "()": "nvflare.fuel.utils.log_utils.ColorFormatter",
            "fmt": "%(asctime)s - %(name)s - %(levelname)s - %(fl_ctx)s - %(message)s"
        },
        "jsonFormatter": {
            "()": "nvflare.fuel.utils.log_utils.JsonFormatter",
            "fmt": "%(asctime)s - %(name)s - %(fullName)s - %(levelname)s - %(fl_ctx)s - %(message)s"
        }
    },
    "filters": {
        "FLFilter": {
            "()": "nvflare.fuel.utils.log_utils.LoggerNameFilter",
            "logger_names": ["custom", "nvflare.app_common", "nvflare.app_opt"]
        }
    },
    "handlers": {
        "consoleHandler": {
            "class": "logging.StreamHandler",
            "level": "DEBUG",
            "formatter": "consoleFormatter",
            "filters": [],
            "stream": "ext://sys.stdout"
        },
        "logFileHandler": {
            "class": "logging.handlers.RotatingFileHandler",
            "level": "DEBUG",
            "formatter": "baseFormatter",
            "filename": "log.txt",
            "mode": "a",
            "maxBytes": 20971520,
            "backupCount": 10
        },
        "errorFileHandler": {
            "class": "logging.handlers.RotatingFileHandler",
            "level": "ERROR",
            "formatter": "baseFormatter",
            "filename": "log_error.txt",
            "mode": "a",
            "maxBytes": 20971520,
            "backupCount": 10
        },
        "jsonFileHandler": {
            "class": "logging.handlers.RotatingFileHandler",
            "level": "DEBUG",
            "formatter": "jsonFormatter",
            "filename": "log.json",
            "mode": "a",
            "maxBytes": 20971520,
            "backupCount": 10
        },
        "FLFileHandler": {
            "class": "logging.handlers.RotatingFileHandler",
            "level": "DEBUG",
            "formatter": "baseFormatter",
            "filters": ["FLFilter"],
            "filename": "log_fl.txt",
            "mode": "a",
            "maxBytes": 20971520,
            "backupCount": 10,
            "delay": true
        }
    },
    "loggers": {
        "root": {
            "level": "INFO",
            "handlers": ["consoleHandler", "logFileHandler", "errorFileHandler", "jsonFileHandler", "FLFileHandler"]
        }
    }
}

We use different formatters, filters, and handlers to output log records to the console and various log files, which are described in more detail below.

Formatters¶

Formatters are used to specify the format of log records. We provide several useful formatters by default:

BaseFormatter¶

The BaseFormatter is the default formatter serving as the base class for other FLARE formatters.

All the default Formatter arguments such as fmt with log record attributes and the datefmt date format string can be specified.
The record.name is shortened to the logger base name, and record.fullName is set to the logger full name.

Example configuration and output:

"baseFormatter": {
    "()": "nvflare.fuel.utils.log_utils.BaseFormatter",
    "fmt": "%(asctime)s - %(name)s - %(fullName)s - %(levelname)s - %(fl_ctx)s - %(message)s",
    "datefmt": "%m-%d-%Y- %H:%M:%S"
}

01-14-2025 14:44:46 - PTInProcessClientAPIExecutor - nvflare.app_opt.pt.in_process_client_api_executor.PTInProcessClientAPIExecutor - INFO - [identity=site-1, run=fc711945-a7cf-4834-9fc4-aa9cb60e327b, peer=example_project, peer_run=fc711945-a7cf-4834-9fc4-aa9cb60e327b, task_name=train, task_id=a16b7a02-b2ea-4eb5-895a-b40d507b2c5c] - execute for task (train)

ColorFormatter¶

The ColorFormatter uses ANSI color codes to format log records based on log level and/or logger names.

We provide the ANSIColor class for commonly used colors and default mappings for log levels. To customize the colors, use either string of a color name specifed in ANSIColor.COLORS, or an ANSI color code (semicolons can be used for additional ANSI arguments).

level_colors: dict of levelname: ANSI color. Defaults to ANSIColor.DEFAULT_LEVEL_COLORS.
logger_colors: dict of loggername: ANSI color. Defaults to {}.

Example configuration:

"consoleFormatter": {
    "()": "nvflare.fuel.utils.log_utils.ColorFormatter",
    "fmt": "%(asctime)s - %(name)s - %(levelname)s - %(fl_ctx)s - %(message)s",
    "level_colors": {
        "NOTSET": "grey",
        "DEBUG": "grey",
        "INFO": "grey",
        "WARNING": "yellow",
        "ERROR": "red",
        "CRITICAL": "bold_red"
    },
    "logger_colors": {
        "nvflare.app_common": "blue",
        "nvflare.app_opt": "38;5;212"
    }
}

JsonFormatter¶

The JsonFormatter converts the log records into a json string.

Example configuration and output:

"jsonFormatter": {
    "()": "nvflare.fuel.utils.log_utils.JsonFormatter",
    "fmt": "%(asctime)s - %(name)s - %(levelname)s - %(fl_ctx)s - %(message)s"
}

{"asctime": "2025-01-14 14:44:46,559", "name": "PTInProcessClientAPIExecutor", "fullName": "nvflare.app_opt.pt.in_process_client_api_executor.PTInProcessClientAPIExecutor", "levelname": "INFO", "fl_ctx": "[identity=site-1, run=fc711945-a7cf-4834-9fc4-aa9cb60e327b, peer=example_project, peer_run=fc711945-a7cf-4834-9fc4-aa9cb60e327b, task_name=train, task_id=a16b7a02-b2ea-4eb5-895a-b40d507b2c5c]", "message": "execute for task (train)"}

Filters¶

Filters are used to allow certain log records to pass through based on specified criteria.

LoggerNameFilter¶

LoggerNameFilter filters loggers based on a list of logger_names. Filters utilize the logger hierarchy, so any descendants of the specified names will also be allowed through the filter. By default, LoggerNameFilter is configured with allow_all_error_logs to allow all logs with level greater than INFO though even if they are not from a logger in logger_names.

logger_names: list of logger names to allow through filter
exclude_logger_names: list of logger names to disallow through filter (takes precedence over allowing from logger_names)
allow_all_error_logs: allow all log records with levelno > logging.INFO through filter, even if they are not from a logger in logger_names. Defaults to True.

We leverage this in our FLFilter, which filters loggers related to fl training or custom code.

"FLFilter": {
    "()": "nvflare.fuel.utils.log_utils.LoggerNameFilter",
    "logger_names": ["custom", "nvflare.app_common", "nvflare.app_opt"]
}

Handlers¶

Handlers are responsible for sending log records to a destination, while applying any specified Formatter or Filters (applied sequentially).

consoleHandler¶

The consoleHandler uses the StreamHandler to send logging output to a stream, such as sys.stdout.

Example configuration:

"consoleHandler": {
    "class": "logging.StreamHandler",
    "level": "DEBUG",
    "formatter": "consoleFormatter",
    "filters": ["FLFilter"],
    "stream": "ext://sys.stdout"
}

FileHandlers¶

We use FileHandlers to send different formatted and filtered log records to different files.

In the pre-configured handlers, more specifically we utilize the RotatingFileHandler to rollover to backup files after a certain file size is reached. FLARE dynamically interprets the filename to be relative to the either the workspace root directory (for site log files), or the run directory (for job log files).

Example configuration:

"logFileHandler": {
    "class": "logging.handlers.RotatingFileHandler",
    "level": "DEBUG",
    "formatter": "baseFormatter",
    "filename": "log.txt",
    "mode": "a",
    "maxBytes": 20971520,
    "backupCount": 10
}

The following log file handlers are pre-configured:

logFileHandler with baseFormatter to write all logs to log.txt
errorFileHandler with baseFormatter and level “ERROR” to write error level logs to log_error.txt
jsonFileHandler with jsonFormatter to write json formatted logs to log.json
FLFileHandler with baseFormatter and FLFilter to write fl training and custom logs to log_fl.txt

Loggers¶

Loggers can be configured in the logger section to have a level and handlers.

We define the root logger with INFO level and add the desired handlers.

"root": {
    "level": "INFO",
    "handlers": ["consoleHandler", "logFileHandler", "errorFileHandler", "jsonFileHandler", "FLFileHandler"]
}

Given the hierarchical structure of loggers, specific loggers can be configured using their dot separated names. Furthermore, any intermediate logger parents are already created and are configureable.

When creating loggers for custom code, we provide a user custom logger function:

custom_logger: From a logger, return a new logger with “custom” prepended to the logger name. This enables logs from the custom logger to pass through the default FLFilter so the logs will be displayed in “concise” mode.

When creating loggers for FLARE code, we provide several developer functions to help adhere to the package logger hierarchy:

get_obj_logger for classes
get_script_logger for scripts
get_module_logger for modules

Modifying Logging Configurations¶

Log Config Argument¶

We provide a log config argument (-l or log_config in simulator mode, and config in the dynamic logging admin commands for POC and production mode). This argument can be any of the following:

log configuration json file (/path/to/my_log_config.json, my_log_config.json)
predefined console LogMode (concise, full, verbose)
- concise (default for simulator mode): FLFilter for FL training logs with simplified log attributes
- full (default in workspaces in poc and production mode): full info level logs
- verbose: debug level logs with detailed log attributes
log level name or number (debug, info, warning, error, critical, 30)
For admin commands only: read the current log configuration file log_config.json from the workspace (reload)

Simulator log configuration¶

Users can specify a log configuration in the simulator command with the -l simulator Log Config Argument:

nvflare simulator -w /tmp/nvflare/hello-numpy-sag -n 2 -t 2 hello-world/hello-numpy-sag/jobs/hello-numpy-sag -l log_config.json

Or using the log_config argument of the Job API simulator run:

job.simulator_run("/tmp/nvflare/hello-numpy-sag", log_config="log_config.json")

POC log configurations¶

If you search the POC workspace, you will find the following:

find /tmp/nvflare/poc  -name "log_config.json*"

/tmp/nvflare/poc/server/local/log_config.json.default
/tmp/nvflare/poc/site-1/local/log_config.json.default
/tmp/nvflare/poc/site-2/local/log_config.json.default

You can add a log_config.json to make changes.

We also recommend using the Dynamic Logging Configuration Commands.

Startup kits log configurations¶

The log configuration files are located in the startup kits under the local directory.

If you search for the log_config.json.* files in the startup kits workspace, you will find the following files:

find . -name "log_config.json.*"

./site-1/local/log_config.json.default
./site-2/local/log_config.json.default
./server1/local/log_config.json.default

The server log_config.json.default is the default logging configuration used by the FL Server and clients. To overwrite the default, you can change log_config.json.default to log_config.json and modify the configuration.

We also recommend using the Dynamic Logging Configuration Commands.

Dynamic Logging Configuration Commands¶

When running the FLARE system (POC mode or production mode), there are two sets of logs: the site logs and job logs. The current site log configuration will be used for the site logs as well as the log config of any new job started on that site. In order to access the generated logs in the workspaces refer to Accessing server-side workspace and Client.

We provide two admin commands to enable users to dynamically configure the site or job level logging when running the FLARE system. Note these command effects will last until reconfiguration or as long as the corresponding site or job is running. However these commands do not overwrite the log configuration file in the workspace- the log configuration file can be reloaded using “reload”.

target: server, client <clients>..., or all
config: the log config argument can be any of the following (For more details, refer to Log Config Argument above):
- path to a json log configuration file (/path/to/my_log_config.json)
- predefined log mode (concise, full, verbose)
- log level name or number (debug, info, warning, error, critical, 30)
- read the current log configuration file log_config.json from the workspace (reload)

To configure the target site logging (does not affect currently running jobs):

configure_site_log target config

To configure the target job logging (the job must be running):

configure_job_log job_id target config

See Operating NVFLARE - Admin Client, Commands, FLARE API for how to use commands and Command Categories for the default authorization policy.