nvflare.fuel.hci.binary_proto module

This package implements a binary protocol for data exchange between the Admin client and server. This is mainly used for large data exchanges such as job submission and download.

The format of a binary exchange has 4 sections (header, meta, body, and footer), as follows:

[Header] [Meta] [Body] [Footer]

Header section: {binary_marker:1} {meta_size:4} {body_size:8} Meta section: {meta:meta_size} Body section: {body:body_size} Footer section: {end_marker:4} {checksum:4}

The 1-byte binary_marker in Header signifies that the exchange is binary. If this marker is missing, the exchange is text.

A binary exchange can optionally include text-encoded meta information (e.g. a JSON string).

At the end of the exchange is the footer that contains end-of-data marker (four bytes of 0) and the checksum computed over the body bytes.

Note that the binary protocol does not replace the text protocol, which is still used for regular admin commands.

class DataGenerator

Bases: ABC

abstract data_size() → int

Return the size of the exchange body to be generated. This method is called before the generate method is called. Therefore, the DataGenerator must know the size of the exchange body in advance.

Returns: the size of the exchange body to be generated

abstract generate() → bytes

This method is called to generate next chunk of data to be sent.

Returns: bytes to be sent; or None if no more data.

class DataProcessor

Bases: ABC

A DataProcessor is used to process data received from the peer.

abstract finalize()

Finalize the processor. This is called when the exchange processing is finished.

Returns: None

abstract process(data: bytes, content_type: int) → bool

Process the data received from peer.

Parameters:

• data – the data to be processed

• content_type – the content type: CT_TEXT or CT_BINARY

Returns: whether this is the end of process

class ExchangeHandler(receiver: Receiver, processor: DataProcessor)

Bases: object

The ExchangeHandler is used to receive and parse exchange from the peer. It uses the provided Receiver to receive data from the peer, parses the data according to the echange protocl, and calls the provided DataProcessor to process the data.

Constructor of ExchangeHandler

Parameters:

• receiver – the data receiver object to be used to receive data from the peer

• processor – the data processor for data processing

receive_and_parse()

Receive data of the exchange from the peer and parse it according to the protocol definition.

Returns: None

class GenerateDataFromFile(file_name: str)

Bases: DataGenerator

This is a special DataGenerator that generates bytes from a file.

data_size() → int

Return the size of the exchange body to be generated. This method is called before the generate method is called. Therefore, the DataGenerator must know the size of the exchange body in advance.

Returns: the size of the exchange body to be generated

generate() → bytes

This method is called to generate next chunk of data to be sent.

Returns: bytes to be sent; or None if no more data.

class MsgDataProcessor

Bases: DataProcessor

The MsgDataProcessor is a special DataProcessor that can handle both TEXT and BINARY content type. For TEXT, it collects all received text segments and assemble them into one text string; For BINARY, it saves received data into a temporary file so that no memory is used to collect the data. This is necessary to support extremely large data exchanges (e.g. large job submission and download).

finalize()

Finalize the processor. This is called when the exchange processing is finished.

Returns: None

process(data, content_type: int)

Process the data received from peer.

Parameters:

• data – the data to be processed

• content_type – the content type: CT_TEXT or CT_BINARY

Returns: whether this is the end of process

class Receiver

Bases: ABC

A Receiver must be able to receive bytes from the peer.

abstract recv(size: int) → bytes

Receive bytes of up to the specified size. Note that this method is named “recv” to make TCP socket automatically a Receiver (duck typing).

Parameters:

size – the max number of bytes to receive.

Returns: bytes of no more than size; or None if recv is not possible (e.g. peer reset connection)

class Sender

Bases: ABC

abstract sendall(data: bytes)

Send specified data until done. This method must send all bytes, instead of a subset of it! Note that this method is named “sendall” to make TCP socket automatically a Sender (duck typing).

Parameters:

data – data to be sent.

Returns:

binary_footer(checksum: int)

Create bytes for the exchange footer.

Parameters:

checksum – the checksum value.

Returns: encoded bytes of the footer

binary_header(meta_size: int, body_size: int)

Create bytes for the exchange header.

Parameters:

• meta_size – size of the meta info

• body_size – size of the message body

Returns: encoded bytes of the header

receive_all(receiver: Receiver)

Receive all data from the peer via the specified communication socket. This function uses the MsgDataProcessor for memory saving during data exchange.

Parameters:

receiver – the object that is capable of receiving. Note that TCP socket is a Receiver (duck typing)

Returns: a tuple of (content_type, request_text, additional_data)

When content_type is CT_TEXT, the request_text is the request body, and additional_data is None; When content_type is CT_BINARY, the request_text is the meta info, and additional_data is the name of a temporary file that holds the received body data. However, if there is no data (size 0), the value of additional_data is None.

send_binary_data(sender: Sender, generator: DataGenerator, meta: str) → int

Send data in binary exchange protocol.

Parameters:

• sender – the sender that is capable of sending bytes

• generator – the generator that is capable of generating data to be sent

• meta – the meta info to be included in the exchange

Returns: number of body bytes sent

send_binary_file(sender: Sender, file_name: str, meta: str) → int

Send a file using binary protocol.

Parameters:

• sender – the object that is capable of sending. Note that TCP socket is a Sender (duck typing).

• file_name – the file to be sent

• meta – the meta info to be sent

Returns: number of bytes sent (the same as size of the file)