nvflare.tool.agent.dataset_inspect module

Static dataset classification for nvflare agent inspect.

Classifies a data-only directory as a tabular or image dataset and emits a metadata-only evidence block. It never emits values from rows classified as data and never emits pixel data, and follows no symlinks. The one inference in the block is the header row: names are emitted only under the strong rule (every numeric-bodied column non-numeric, names unique and non-empty), and a masked data row that satisfies all of that is emitted as names — by construction no inference can tell it from a header, so datasets with masked/sentinel first rows must declare names. Read bounds: text and image reads are capped at max_file_bytes each (worst case max_files x max_file_bytes); parquet metadata reads parse only the file footer via pyarrow — they are not capped by max_file_bytes and are accounted in scan at file size as the upper bound.

Dataset block contract (all keys always present per modality):

• modality: tabular | image | mixed, decided per site, not by file-count share. Targeted shapes: a tabular dataset tolerates stray images (<= 2 per site, e.g. exported plots); an image dataset tolerates companion tabular metadata (<= 4 files per site, e.g. train/val/test label files, flagged tabular_companions, never a statistics target); tiny sites inside both tolerance windows resolve to the side with more files overall; anything else — an image-only site among tabular sites, materially both modalities, or a dead file-count tie — is mixed, reported but not routed (target_type stays unknown_target).

• layout: per_site_directories | flat | root_and_site_directories (root-level data files coexist with site directories — ambiguous site mapping, an ask/fail-closed input for the consumer; the root group appears as site .).

• file_census: extension -> count (bounded; see counts_approximate).

• counts_approximate: true when a walk limit was hit.

• scan: files_read / bytes_read performed by this classification.

• sites: per top-level directory (or . when flat): - always: name, data_files, tabular_files, image_files; - tabular/parquet: format, header, column_count, features,

  dtypes, row_count, row_count_approximate, and for parquet schema_available. header: present means “names available” (parquet always has names when readable); header: ambiguous means names must be user-supplied. row_count aggregates every tabular file in the site; sharded CSV sites are always approximate (per-shard headers are unknowable; a shard whose header-like first row disagrees with the site schema flags shard_schema_consistent: false), parquet shards are exact metadata sums when the shards agree on schema. Feature names are sanitized: control characters stripped and length capped (feature_names_truncated marks a cap hit); schemas wider than the column cap set columns_truncated — drift past the cap is invisible. header: present requires every numeric-bodied column to carry a non-numeric first value AND the names to be unique and non-empty; parquet names failing that are flagged feature_names_invalid. Dtype classes are inferred from a bounded row sample; a full-file reader (pandas at job runtime) can disagree when anomalies appear past the sample.

  • image: image_formats (the site’s image extensions, so the consumer picks the matching loader) and pixel_depth from one sample when an optional loader exists (PIL formats only; DICOM/NIfTI depth stays null and the loader must be chosen from image_formats).

• schema_agreement (tabular): compares feature names, column counts, AND dtype classes across sites; any drift is a mismatch with per-site issues (feature_names_differ | column_count_differs | header_presence_differs (one site has a header, another is ambiguous — alignment unverifiable even at equal widths) | dtypes_differ | shards_differ, the last when shards inside one site disagree — flagged per site as shard_schema_consistent: false).

Walk bounds: max_files counts data files so non-data clutter cannot hide a dataset, and MAX_WALK_ENTRIES bounds total traversal (files and directories). This deliberately differs from the code walk in inspector.py, which bounds per-file scan work; the two walks bound different costs and are kept separate on purpose.

inspect_dataset(root: Path, max_files: int, max_file_bytes: int) → dict | None

Classify a directory as a dataset; None when it holds no data files.