ProxyNDSource¶

Interface for NDim sources in Proxy. For example, a NDArray, a HDF5 dataset, etc. For a simpler source, see ProxySource.

class blosc2.ProxyNDSource[source]¶

Base interface for NDim sources in Proxy.

A source may also serve single blocks rather than whole chunks, which is worth doing when a fetch costs a round trip and a slice touches little of the chunks it lands in. Proxy uses that path when the source has all five of blocks_per_chunk, wants_blocks(nchunk, nwanted), chunk_layout(nchunk), block_plan(nchunk, nblocks) and read_range(offset, size); FsspecNDSource implements them over byte ranges and is the worked example.

Having them is not the same as being able to use them for the dataset in hand: a source that knows, without asking anything, that this one is served whole says so with serves_blocks, and Proxy then keeps to chunks from the start rather than to a per-block bitmap it would never fill. It is read once, when the proxy is built, and a source without it counts as True.

A source whose transport can ask for several ranges at once says so with max_ranges and serves read_ranges(spans) and chunk_layouts(nchunks) as well; Proxy then sends a whole wave of reads as one request. Setting wants_wave says that wants_blocks takes a third argument and is to be given the wave – the fetch that chunk belongs to – since a shared round trip is the wave’s to weigh and not the chunk’s. It is an opt-in of its own, in the same shape as max_ranges and read the same way, so a two-argument wants_blocks keeps being called with two. All are optional, and a source without them is asked one range at a time, exactly as before.

A block read that the transport cannot answer raises NotRanged, and Proxy then fetches the chunks it was after whole.

Attributes:

attrs: The user attributes of the source.
blocks: The block shape of the source.
chunks: The chunk shape of the source.
cparams: The compression parameters of the source.
dtype: The dtype of the source.
meta: The fixed-length metadata of the source.
shape: The shape of the source.
vlmeta: The variable-length metadata of the source.

Methods

`aget_chunk`(nchunk)	Return the compressed chunk in `self` asynchronously.
`get_chunk`(nchunk)	Return the compressed chunk in `self`.

async aget_chunk(nchunk: int) → bytes[source]¶

Return the compressed chunk in self asynchronously.

Parameters:: nchunk¶ (int) – The index of the chunk to retrieve.
Returns:: out – The compressed chunk.
Return type:: bytes object

Notes

This method is optional, and only available if the source has an async aget_chunk method.

abstractmethod get_chunk(nchunk: int) → bytes[source]¶

Return the compressed chunk in self.

Parameters:: nchunk¶ (int) – The unidimensional index of the chunk to retrieve.
Returns:: out – The compressed chunk.
Return type:: bytes object

property attrs¶: The user attributes of the source.

abstract property blocks: tuple¶: The block shape of the source.

abstract property chunks: tuple¶: The chunk shape of the source.

property cparams: CParams¶

The compression parameters of the source.

This property is optional and can be overridden if the source has a different compression configuration.

abstract property dtype: dtype¶: The dtype of the source.

property meta: dict¶: The fixed-length metadata of the source.

abstract property shape: tuple¶: The shape of the source.

property vlmeta: dict¶: The variable-length metadata of the source.