Skip to content

Data Splits

How Cuvis.AI decides which samples are train, which are validation, and which are test: a small selector language over an explicit sample universe, resolved into a reproducible, committable splits.json.

A split is not a folder layout or a hard-coded id list. It is three separable things:

  1. the universe of samples that exist (state),
  2. a rule that assigns them to stages (train / val / test / predict), and
  3. the baked assignment you commit and ship.

Keeping these apart is what lets one split definition follow your data from raw .cu3s sessions to converted .npz frames without rewriting anything, and lets a GUI edit a split without re-scanning the disk. This page explains what each artifact does, the states it can be in, and how they compose.

The model: state, rule, baked

   STATE (universe)                 RULE (live, expressive)          BAKED (concrete, committed)
   what samples exist               how to assign them               the frozen assignment
   ----------------                 -------------------              ---------------------
   DataModule.enumerate()           splits.json selectors:           splits.json with only
   from a universe.csv              dir_indices / glob / tag /       file_indices / files +
   (cu3s_multi, npz_multi), OR      categories / set-algebra         a pinned universe_hash.
   from disk (tiff_paired pairs).   (resolve live at setup())        GENERATED by resolve-splits
   Fingerprinted by universe_hash.  file_indices = already concrete  (--from-csv / --strategy)
                                                                     or the cu3s->npz converter.
        |                                    |                                 ^
        |        resolve_selectors(selectors, enumerate())  (setup + leakage guard)
        |                                                                      |
        +------- resolve-splits GENERATES a baked file (import CSV / compute split) -----+

  • State (universe). The set of samples that exist and where each physically lives. A module produces it from enumerate(): either by reading an explicit universe file (universe.csv) as cu3s_multi and npz_multi do, or by scanning disk for cube/label pairs as tiff_paired does. universe_hash fingerprints the universe so drift can be detected.

  • Rule. A splits.json of selectors. dir_indices / glob / tag / categories are the live, expressive rules; file_indices / files are already concrete. Every kind resolves at setup() via resolve_selectors, so a rule-form file trains directly with no extra step.

  • Baked. A splits.json that holds only concrete file_indices / files plus a pinned universe_hash. Position-independent, reproducible, and drift-guarded. This is what a GUI edits and what you commit next to a dataset. You obtain a baked file by generating one (see The bake is by generation), not by freezing a hand-authored rule.

The same splits.json is both "rule" and "baked": it is a rule while any live selector kind remains, and baked once every selector is concrete. "Baked" is a property of the file's content, not a separate schema. The universe is never folded into splits.json; it stays on disk or in universe.csv.

States, defined

A splits.json is in exactly one of two states, and the universe file is optional:

  splits.json state          universe file state
  -----------------          -------------------
  RULE   any dir_indices /    ABSENT   a disk-enumerated module (tiff_paired
         glob / tag /                  cube/label pairs) has no universe file;
         categories remains            its universe is implicit
  BAKED  only file_indices /  PRESENT  universe.csv (the explicit map) == the
         files + a pinned              `universe_csv` argument; required for
         universe_hash                 cu3s_multi + npz_multi

There is also a third, separate case: a module whose DataConfig.splits is None owns its split in code and produces neither a splits.json nor a universe file (see Two ways a split is produced).

Selectors

Selectors are the split language. They live in cuvis_ai_schemas.training.data and are resolved by cuvis_ai_core.data.selectors.resolve_selectors. Eleven kinds exist:

Kind Picks Live or concrete
file_indices read positions ids within a source concrete
files whole sources by identity concrete
dir_indices the Nth files under a directory live (position-dependent)
stems samples by filename stem live
glob samples whose source matches a glob live
tag samples carrying a tag live
categories samples containing given COCO category ids live
all the whole universe live
union / except / intersect set algebra over operand selectors composite

resolve_selectors takes a stage's list[Selector] plus the enumerated universe and returns the per-sample list, unioned in selector order with first-occurrence dedup on each sample's uid. A selector that matches zero samples raises, so a typo fails loudly instead of silently shrinking a split.

Two addressing modes matter most. file_indices names samples by identity (source, index); it is position-independent, so inserting a file elsewhere in the dataset does not change which samples it selects. dir_indices names "the Nth files in this directory"; it is compact to author by hand but moves if the directory contents change. A committed split should hold file_indices (baked); dir_indices / glob / tag are for authoring.

The universe_csv argument

When a module cannot list its own samples from disk, you give it an explicit universe: a CSV whose path is the module's universe_csv argument (DataConfig.params.universe_csv). The file is universe.csv. It is format-agnostic and maps each sample's identity to its physical asset:

columns: source, index [, materialized_path, split, annotation, format, group]
  source            posix logical id (matches a splits.json selector `source`)
  index             read position within the source (== COCO image_id / measurement index)
  materialized_path physical file to open, relative to the CSV (posix; no `..` escape).
                    Defaults to `source` for cu3s_multi (a raw .cu3s is its own file);
                    REQUIRED for npz_multi (the physical file is the derived .npz).
  split             OPTIONAL, cu3s_multi only: train/val/test. Present -> module-owned;
                    absent -> a splits.json is required. npz_multi rejects this column.
  annotation        optional paired label path (a per-day COCO json for cu3s)
  format            optional provenance hint (cu3s | npz | tiff | ...); each module has a
                    fixed reader, so it is not used for dispatch
  group             RESERVED: a leakage-grouping key, carried onto the sample but not yet
                    enforced by the leakage check (see Leakage below)

  • Required for both cu3s_multi and npz_multi. Both read a universe.csv (one shared vocabulary, one shared parser; each keeps its own reader). Only tiff_paired enumerates from disk (cube/label pairs), so it needs no universe_csv.

  • One universe, many splits, many materializations. Because source is a stable logical identity (a cu3s-derived npz carries its originating cu3s path), the same splits.json resolves against the raw cu3s data and against a converted npz universe.csv. You can also point several splits (for example splits/dinomaly.json and splits/adaclip.json) at one universe.

  • Nothing is parsed out of filenames. Every path is explicit in the CSV. Duplicate (source, index), a duplicate materialized_path (npz), and any .. path escape are all rejected when the file is read.

  • Provenance columns are welcome. Beyond the core columns, a universe.csv (or a split CSV a dataset ships alongside it) may carry extra columns purely for traceability, for example group or dataset-specific fields like acquisition day or camera. Unknown columns are ignored by the reader, and group is reserved for group-aware leakage. Keeping such a CSV next to a baked splits.json is a valid way to retain provenance the baked selectors would otherwise drop.

splits_io in the workflow

The split machinery in cuvis_ai_core.data is small and composable. Its pieces slot into one flow, run for you by the DataModule base class at setup():

flowchart LR
    A["enumerate(required_attrs)"] --> B["verify_universe<br/>(drift guard)"]
    B --> C["resolve_selectors"]
    C --> D["validate_leakage"]
    D --> E["build_dataset_from_refs"]

  • enumerate(required_attrs) returns the attributed sample universe. It is two-level: the cheap level is the canonically sorted file list; the expensive level (parsing COCO / masks for tag or categories) is only paid for the sources a selector actually touches. required_attrs, computed from the selectors, tells enumerate which attributes are needed.

  • load_splits / save_splits read and write a splits.json as a DataSplitConfig, with shape validation on load and stable pretty JSON on save.

  • universe_hash(refs) is a sha256 over the ordered sample uids: the fingerprint pinned into a baked split.

  • has_dir_indices(cfg) reports whether a split still holds live position-dependent selectors (that is, whether it is a rule or already baked).

  • verify_universe(cfg, refs) is the drift guard: if a split pins a universe_hash and still carries dir_indices, it fails loudly when the universe no longer matches the pin, telling you to regenerate rather than silently resolving against shifted positions.

  • resolve_selectors and validate_leakage turn the selectors into the per-stage sample lists and check the split for leakage.

  • build_dataset_from_refs(refs) is the module's heavy step: it maps the resolved samples to the actual reads (npz loads, cu3s frames, tiff pairs).

The bake is by generation

"Baking" a split means producing a splits.json that holds only concrete file_indices / files plus a pinned universe_hash. You get one by generating it, with the resolve-splits CLI (from the cuvis-ai-dataloader plugin):

# Import an existing CSV's `split` column into per-source file_indices (outcome-equivalent):
resolve-splits --from-csv splits_source.csv --out splits.json

# Or compute a split over a module's enumerated universe:
resolve-splits --data-module cu3s_multi --data-arg universe_csv=sessions.csv \
    --strategy stratified --group-by source --ad-aware --seed 7 --out splits.json

Both modes emit concrete file_indices selectors built from the enumerated universe. The cu3s -> npz converter emits the same baked form as a side effect of conversion.

There is no author-then-lower step

Cuvis.AI does not ship a transform that takes a hand-authored dir_indices / glob file and "lowers" it into a file_indices file in place. A rule-form splits.json resolves fine at training time, but to freeze it you regenerate with resolve-splits. Author with live selectors for convenience, generate a baked file to commit.

Two ways a split is produced

  Selector path (per-sample addressable)        Module-owned (split computed in code)
  --------------------------------------         -------------------------------------
  cu3s_multi, tiff_paired, npz_multi             metal_scrap (cuvis-ai-inspecscrap)
  -> DataConfig.splits = a splits.json           -> DataConfig.splits is None
  arbitrary per-sample assignment                dataset/group-level split, computed
  one universe, many splits                      by the module (e.g. a fixed carve)

  • Selector path is the general mechanism. The universe is addressable per sample, so a splits.json can assign any sample to any stage. Every selector-path module (cu3s_multi, tiff_paired, npz_multi) reads its assignment from DataConfig.splits.

  • Module-owned is for datasets whose split is intrinsically a rule at the group or dataset level (a fixed directory carve, a computed holdout). The module leaves DataConfig.splits unset and computes the split itself. Such a module can still be frozen to a splits.json with a generator when you want a committable, editable artifact.

Leakage, drift, and predict

  • Leakage. DataSplitConfig.leakage_check (error / warn / off, default error) asserts that train, val, and test are pairwise disjoint by sample uid. predict is deliberately excluded (it may legitimately overlap). The group field is carried on each sample (defaulting to its source) and is reserved: group-aware leakage is not yet enforced, so today leakage keys purely on uid.

  • Drift. A baked split pins a universe_hash. If the underlying universe changes and the split still holds position-dependent dir_indices, verify_universe fails loudly. Baked file_indices are position-independent and unaffected.

  • Empty predict. An empty predict stage resolves to the whole universe (predict-everything). That is why the converter mirrors a split into predict: to avoid accidentally predicting the entire dataset when only a subset was intended.

  • Reproducibility. enumerate() returns a canonically sorted universe, so selector resolution and universe_hash are deterministic across machines and platforms (source is normalized to posix).