Loader

Minibatch iteration over a cell-major store.

loader

Minibatch iteration over a cell-major store.

CellLoader

CellLoader(
    reader: CellReader,
    indices: Any = None,
    *,
    batch_size: int = 256,
    shuffle: bool = True,
    seed: int | None = None,
    layer: Layer = "norm",
    sparse: bool = True,
    prefetch: int = 4,
    drop_last: bool = False,
    genes: Any = None,
)

Iterate a cell-major store in minibatches, decoding ahead of the consumer.

Each pass spawns one feeder thread that decompresses and materialises batches into a bounded queue, so decode overlaps the training step. The queue depth is the backpressure knob, not a parallelism knob: decode is already rayon-parallel inside a single batch.

>>> import bixverse as bv
>>> reader = bv.CellReader("counts.bin")
>>> loader = bv.CellLoader(reader, batch_size=256)
>>> for batch in loader:
...     x = batch.scipy_csr()

Sparse is the default. A dense batch costs a zero-fill and a scatter over batch_size * n_genes floats no matter how few non-zeros it holds, which at droplet sparsity is roughly twenty times the memory traffic.

Build a loader.

Parameters:

Name	Type	Description	Default
reader	CellReader	An open cell-major CellReader.	required
indices	Any	Cells to iterate. None uses every cell in the store.	None
batch_size	int	Cells per batch.	256
shuffle	bool	Reshuffle at the start of every epoch.	True
seed	int | None	Base seed for the shuffle. None draws one at random.	None
layer	Layer	"raw", "norm" or "both".	'norm'
sparse	bool	CSR components when true, a dense buffer when false.	True
prefetch	int	Batches held in flight by the feeder thread.	4
drop_last	bool	Drop a trailing partial batch.	False
genes	Any	Store gene indices to keep, in the column order wanted. None keeps every gene.	None

Raises:

Type	Description
ValueError	For a gene-major store, an unusable parameter, an out-of-range cell or gene index, or a repeated gene.

reader property

reader: CellReader

The store this loader reads from.

indices property

indices: ndarray

The cells this loader iterates, in the order they were given.

Batch

Bases: NamedTuple

One minibatch of cells.

Sparse batches carry indices and indptr and CSR-ordered raw/norm. Dense batches carry (n_cells, n_genes) buffers and leave both None. Whichever layer was not requested is None. cell_ids gives the store index of each row, which is how per-cell covariates are joined back on.

is_sparse property

is_sparse: bool

Whether this batch holds CSR components rather than a dense buffer.

values

values() -> ndarray

The single requested layer.

Returns:

Type	Description
ndarray	raw or norm, whichever was asked for.

Raises:

Type	Description
ValueError	If the batch carries both layers, which is ambiguous.

scipy_csr

scipy_csr(
    layer: Literal["raw", "norm"] | None = None,
) -> csr_matrix

Rebuild this batch as a scipy.sparse.csr_matrix.

Parameters:

Name	Type	Description	Default
layer	Literal['raw', 'norm'] | None	Which layer to use when the batch carries both.	None

Returns:

Type	Description
csr_matrix	A CSR matrix of shape shape, sharing the batch's buffers.

Raises:

Type	Description
ValueError	If the batch is dense, or layer is needed and absent.