Here's a short demonstration of this:

import numpy as np
import zarr


def track_chunk_calls(z: zarr.Array):
    """Wrap a zarr.Array such that it prints out the details of the requested chunk."""

    f = z._chunk_getitem

    def _track(chunk_coords, chunk_sel, out, out_sel, **kwargs):
        print(f"{chunk_coords=}, {chunk_sel=}, {out_sel=}")
        return f(chunk_coords, chunk_sel, out, out_sel, **kwargs)

    z._chunk_getitem = _track


# Create a chunked array
N = 50
C = 5
z = zarr.ones((N,), chunks=(C,))

# Define a sparse slice selection
sl = slice(4, 41, 12)

# Calculate the chunks the requested elements are in
chunk_inds = np.arange(N)[sl] // C
print(f"## Chunks that must be read: {chunk_inds}")

print("## Chunks that are actually read:")
track_chunk_calls(z)
z[sl]

Prior to the fix in this PR, this gives the output:

## Chunks that must be read: [0 3 5 8]
## Chunks that are actually read:
chunk_coords=(0,), chunk_sel=(slice(4, 5, 12),), out_sel=(slice(0, 1, None),)
chunk_coords=(1,), chunk_sel=(slice(11, 5, 12),), out_sel=(slice(1, 1, None),)
chunk_coords=(2,), chunk_sel=(slice(6, 5, 12),), out_sel=(slice(1, 1, None),)
chunk_coords=(3,), chunk_sel=(slice(1, 5, 12),), out_sel=(slice(1, 2, None),)
chunk_coords=(4,), chunk_sel=(slice(8, 5, 12),), out_sel=(slice(2, 2, None),)
chunk_coords=(5,), chunk_sel=(slice(3, 5, 12),), out_sel=(slice(2, 3, None),)
chunk_coords=(6,), chunk_sel=(slice(10, 5, 12),), out_sel=(slice(3, 3, None),)
chunk_coords=(7,), chunk_sel=(slice(5, 5, 12),), out_sel=(slice(3, 3, None),)
chunk_coords=(8,), chunk_sel=(slice(0, 1, 12),), out_sel=(slice(3, 4, None),)

demonstrating that it is reading all the chunks spanning the range, not just the minimal set of [0, 3, 5, 8].

After applying the fix only the minimal chunks are actually read:

## Chunks that must be read: [0 3 5 8]
## Chunks that are actually read:
chunk_coords=(0,), chunk_sel=(slice(4, 5, 12),), out_sel=(slice(0, 1, None),)
chunk_coords=(3,), chunk_sel=(slice(1, 5, 12),), out_sel=(slice(1, 2, None),)
chunk_coords=(5,), chunk_sel=(slice(3, 5, 12),), out_sel=(slice(2, 3, None),)
chunk_coords=(8,), chunk_sel=(slice(0, 1, 12),), out_sel=(slice(3, 4, None),)

Codecov Report

Merging #1154 (b7b25cb) into main (2dcffcd) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1154   +/-   ##
=======================================
  Coverage   99.95%   99.95%           
=======================================
  Files          36       36           
  Lines       14117    14142   +25     
=======================================
+ Hits        14110    14135   +25     
  Misses          7        7           

Thanks so much for this PR @jrs65! It looks like a simple solution with a clear improvement.

It would be feasible to have a test asserting that only the minimal set of chunks are read, but that feels like it's testing an implementation detail.

I'm not sure it's an implementation detail. Not accessing data unnecessarily is an important feature, particularly when the access might be very expensive (think cloud storage, large chunks, etc.). From my point of view, it would be great to have tests for this sort of thing.

On thing we do in Xarray's test suite is to define a special kind of store which errors if you try to access any of its data. This allows us to guard against undesirable data access. We could take a similar approach here in Zarr.

Thanks so much for this PR @jrs65! It looks like a simple solution with a clear improvement.

I'm not sure it's an implementation detail. Not accessing data unnecessarily is an important feature, particularly when the access might be very expensive (think cloud storage, large chunks, etc.). From my point of view, it would be great to have tests for this sort of thing.

On thing we do in Xarray's test suite is to define a special kind of store which errors if you try to access any of its data. This allows us to guard against undesirable data access. We could take a similar approach here in Zarr.

Great. That's along the lines I was thinking of anyway, so I will put together a set of unit tests along those lines, and update this PR with them. I'll add them to zarr/tests/test_indexing.py unless you have another suggestion.

That sounds great! Thanks so much for volunteering to take that on. It's a lot more work than just fixing the bug! 🙏 We welcome your contributions to zarr-python and will do our best to help with your PR. 🙏

@rabernat very happy to do so! Compared to hacking around in the internals of HDF5 and h5py, the zarr codebase is a joy to understand and modify.

I've implemented tests for this case. In the end I used the existing zarr.tests.util.CountingDict implementation rather than reinventing the wheel. I use that as a backing store, and then after each operation I check that the expected chunks have been accessed once and that no other chunks have been touched.

Let me know what you think, and anything else that needs to be done to the PR.

@jrs65: I'll second @rabernat's thanks! I've triggered the GHA actions. Let's see how the tests far. (Also looking forward to what the asv benchmarks look like before & after. cc: @MSanKeys963)

Tests are all green! 🎉 @jrs65, do you want to add yourself to docs/release.rst?

Otherwise, unless there are objections, I could see getting this out for a quick patch release.

Agree this is a nice improvement. Thank you! 🙏

Are there other indexers that would benefit from the same kind of fix?

Tests are all green! 🎉 @jrs65, do you want to add yourself to docs/release.rst?

Sure! I'll add it down in a provisional 2.13.3 patch release, presuming you'll edit that around when you decide when to get it out.

Agree this is a nice improvement. Thank you! 🙏

Are there other indexers that would benefit from the same kind of fix?

I've just read over the code for the other indexers. BoolArrayDimIndexer, IntArrayDimIndexer, CoordinateIndexer, and MaskIndexer (a subclass of CoordinateIndexer) look like they explicitly calculate which chunks contain the requested elements and then ignores those that don't. So I believe those are all fine.

I'm unsure about OrthogonalIndexer it does not seem to explicitly do that, so I'd need to read in more detail.

Regardless, I think it would be very straightforward to extend the tests to ensure that going forward the other selection types continue to only access what they need. If I find a bit of time to spare in the next week I might give that ago and see what happens.

Adding to 2.13.3 sounds good. Thanks! 🙏

Also thanks for taking the time to look into other indexers. No worries if you don't have time to look further. Figured this was a good opportunity to revisit since we are already looking at performance here. Though happy to spin off into a new issue for a future date if that's preferable 🙂

Errrmm... ooops. Sorry. I typo'd my own name in the Release Notes 😆 I'm just going to push a quick fix to that one.

No worries. Thanks again Richard! 🙏

Planning on merging after CI passes.

Re-launched the job after an unrelated failure. I'll keep an eye on it for release.

Thanks Josh! 🙏 Looks like it passed 🟢

Thanks, @jakirkham. I created the GH release. 👍