ConcatDataset

class ConcatDataset(datasets, loop=1)

Bases: Dataset

Concatenate several configured datasets into a single index space.

Builds each child dataset from its config and flattens them into a shared index of (dataset_index, local_index) pairs, so a single sampler can draw from all of them. Items pass through unchanged from the owning child, so the emitted keys (coord, segment, instance, offset after collation, …) are whatever the children emit. Registered as ConcatDataset – use as type under data.train/data.val/ data.test.

Parameters:

• datasets (list[dict]) – List of child dataset configs, each built with build_dataset().

• loop (int) – Epoch multiplier applied to the concatenated length. Also consumed by MultiDatasetDataloader as the main-dataset epoch multiplier; per-child loop values can act as sampling ratios when mixed batches are built. Defaults to 1.

Example

>>> from pimm.datasets.builder import build_dataset
>>> ds = build_dataset(dict(type="ConcatDataset", datasets=[
...     dict(type="PILArNetH5Dataset", revision="v2", split="train", transform=[]),
...     dict(type="PILArNetH5Dataset", revision="v2", split="val", transform=[]),
... ]))
>>> len(ds)                   # sum of child lengths (x loop)  
1234567
>>> sample = ds[0]            # item passes through unchanged from the owning child
>>> # keys are whatever that child emits (here PILArNet: coord, energy,
>>> # segment_motif, instance_particle, name, split, ...)

get_data(idx)

Dispatch a global index to the owning child dataset.

get_data_list()

Return (dataset_index, local_index) pairs for all children.

get_data_name(idx)

Return the child dataset’s sample name for a global index.