feat: subchunk write order - #3826

…al reads

Three integration gaps surfaced when the Fused pipeline met main's new
subchunk_write_order (zarr-developers#3826), partial-read coalescing (zarr-developers#3004), and _ShardIndex
refactor. Under Fused these caused 25 sharding/parity failures (data was
correct in the partial-read cases; the failures were write-order layout +
IO-pattern divergence). Fixes:

1. Write order: _encode_shard_dict_sync laid out chunks in hardcoded morton
   order, ignoring subchunk_write_order. Now iterates
   _subchunk_order_iter(self.subchunk_write_order), matching the async
   _encode_shard_dict. Fixes lexicographic/colexicographic/unordered storage.

2. Coalesced sync partial reads: add Store.get_ranges_sync (a synchronous,
   coalescing counterpart of get_ranges, reusing coalesce_ranges) and
   ShardingCodec._load_partial_shard_maybe_sync; route _decode_partial_sync's
   partial branch through it. Sync stores now get zarr-developers#3004's byte-range coalescing
   without an event loop (fewer, merged reads).

3. Non-sync fallback: FusedCodecPipeline.read now routes non-sync stores (e.g.
   ZipStore) through the async partial-decode path when the AB codec supports
   it, instead of _async_read_fallback's whole-shard get(). Matches Batched's
   IO behavior; avoids over-reading whole shards on partial reads.

Tests: the zarr-developers#3004 partial-read tests are made pipeline-aware (assert the active
method family: get/get_ranges vs get_sync/get_ranges_sync, gated on store sync
support). 573 sharding+parity+pipeline+indexing and 657 codec tests pass under
BOTH pipelines (was 25 failing under Fused).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* feat: subchunk write order (#3826)

* feat: subchunk write order

* chore: export `SubchunkWriteOrder`

* chore: docs

* chore: relnote

* rename

* refactor: no enums

* Update docs/user-guide/performance.md

Co-authored-by: Davis Bennett <[email protected]>

* feat: deterministic but random order

* fix: make vectorized fetching less reliant on matching order

* chore: add hypothesis

* refactor: dead code

* refactor: more cleanup

* don't shard unless there is something to shard

* fix: dont mix chunk grid and sharding

---------

Co-authored-by: Davis Bennett <[email protected]>
(cherry picked from commit 093a153)

* refactor: unordered subchunk order means no-promise, not random

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test: pin subchunk_write_order survival through pickle

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* refactor: remove rng from ShardingCodec; carry subchunk_write_order through pickle

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs: describe unordered subchunk order as no-guarantee, drop rng

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test: drive sharding strategy through serializer to exercise subchunk_write_order

The hypothesis arrays() strategy passed both shards= and a ShardingCodec
serializer, which nested the codecs and left subchunk_write_order governing
only a 1-element inner grid. Drive sharding through the serializer alone.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* harden: guard _subchunk_order_iter; document write-order is not persisted

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* cleanup: use np.ndindex for immaterial intermediate order; drop stale FIXME

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* polish: guard scalar arrays in sharding strategy; align doc value ordering

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* Revert strategies.py changes — unrelated to this PR

These changes were patching a latent bug in `arrays()` where
ShardingCodec-as-serializer was being double-stacked with `shards=...`,
producing nested sharding. Splitting to a follow-up PR so this one stays
focused on removing the `rng`/random-subchunk-order surface.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

---------

Co-authored-by: Ilan Gold <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

* perf: cache lexicographic chunk coords in sharding codec

The subchunk_write_order feature (#3826) regressed sharded write
performance: _encode_partial_single rebuilt the full per-shard chunk
coordinate grid on every write via
`np.array(list(_subchunk_order_iter(..., "lexicographic")))`, and
`to_dict_vectorized` rebuilt a tuple key per row with `tuple(coords.ravel())`.
For a single-chunk write into a shard with tens of thousands of chunks this
roughly doubled write time (~22ms -> ~40ms on test_sharded_morton_write_single_chunk,
matching the -44% CodSpeed regression).

Add cached `_lexicographic_order` (array) and `_lexicographic_order_keys`
(tuples) helpers in indexing.py, mirroring `_morton_order`/`_morton_order_keys`,
and pass the cached keys into `to_dict_vectorized` instead of deriving them
row-by-row. This restores write throughput to the pre-#3826 baseline while
preserving identical chunk ordering (verified equal to np.ndindex across
shapes including 0-d and empty).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore(deps): bump the actions group across 1 directory with 8 updates (#176)

Bumps the actions group with 8 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` |
| [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` |
| [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` |
| [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` |
| [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` |
| [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` |



Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6
- [Release notes](https://github.com/prefix-dev/setup-pixi/releases)
- [Commits](prefix-dev/setup-pixi@1b2de7f...5185adf)

Updates `codecov/codecov-action` from 6.0.0 to 6.0.1
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](codecov/codecov-action@57e3a13...e79a696)

Updates `github/issue-metrics` from 4.2.2 to 4.2.7
- [Release notes](https://github.com/github/issue-metrics/releases)
- [Commits](github-community-projects/issue-metrics@c9e9838...1e38d5e)

Updates `j178/prek-action` from 2.0.3 to 2.0.4
- [Release notes](https://github.com/j178/prek-action/releases)
- [Commits](j178/prek-action@6ad8027...bdca6f1)

Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v7...043fb46)

Updates `actions/download-artifact` from 7.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v7...3e5f45b)

Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0
- [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases)
- [Commits](pypa/gh-action-pypi-publish@v1.13.0...cef2210)

Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6
- [Release notes](https://github.com/zizmorcore/zizmor-action/releases)
- [Commits](zizmorcore/zizmor-action@b1d7e1f...5f14fd0)

---
updated-dependencies:
- dependency-name: prefix-dev/setup-pixi
  dependency-version: 0.9.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: codecov/codecov-action
  dependency-version: 6.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: github/issue-metrics
  dependency-version: 4.2.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: j178/prek-action
  dependency-version: 2.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: pypa/gh-action-pypi-publish
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: zizmorcore/zizmor-action
  dependency-version: 0.5.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* refactor(sharding): derive coords inside to_dict_vectorized

Address review feedback: `_ShardReader.to_dict_vectorized` took the
lexicographic coordinate array and key tuples as parameters, even though
the reader already knows its own `chunks_per_shard` and both structures
are `lru_cache`d. Thread nothing in — fetch them inside the method via
`_lexicographic_order`/`_lexicographic_order_keys`. Same cache, so no
perf change; the call site collapses to `to_dict_vectorized()`.

Add a unit test covering the method directly across 0-d, 1-d, and 2-d
shard grids: present chunks map to their stored bytes, empty chunks to
None, and every lexicographic coordinate appears as a key.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* Update src/zarr/core/indexing.py

Co-authored-by: Ilan Gold <[email protected]>

* refactor(sharding): drop redundant lexicographic_order_iter

Address review feedback from @ilan-gold and @chuckwondo on the
`lexicographic_order_iter` helper.

`lexicographic_order_iter` returned a *lazy* iterator over an
*eagerly-built, cached* tuple (`_lexicographic_order_keys`), which
chuckwondo rightly flagged as confusing — and its output is byte-for-byte
identical to the pre-existing, genuinely-lazy `c_order_iter`
(verified across 0-d, empty, and N-d shapes). So the name promised
laziness the implementation didn't provide, over a sequence we could
already produce.

Remove the wrapper and use the cached `_lexicographic_order_keys` tuple
directly at the two `dict.fromkeys` call sites and in
`_subchunk_order_iter`. This keeps the eager/cached coordinate tuples —
which is the actual optimization: `dict.fromkeys` over the cached tuple
is ~1.4x faster than over lazy `c_order_iter` at 32^3 (≈900us vs ≈1300us),
because the cache amortizes tuple construction across repeated writes to
same-shaped shards. Switching to `c_order_iter` would have reintroduced
that cost, so it is deliberately not used here.

Also drop the now-dead `tuple()` wrap in `morton_order_iter` (its
argument is typed `tuple[int, ...]` and every caller passes one), per
ilan-gold.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* refactor(indexing): prefer lexicographic_order_iter, soft-deprecate c_order_iter

`c_order_iter` names a memory layout ("C order") rather than what the
iterator actually yields. Reintroduce `lexicographic_order_iter` as the
clearer name for the same row-major coordinate sequence, and make
`c_order_iter` a thin alias that delegates to it, with a docstring note
steering new code to the preferred name. No runtime warning — these are
internal helpers.

`lexicographic_order_iter` keeps the eager/cached implementation
(iter over the lru_cached `_lexicographic_order_keys` tuple), which is
~1.4x faster than the old lazy `itertools.product` on the
`dict.fromkeys` shard-write path and is the optimization this branch
exists to deliver. The alias therefore changes `c_order_iter` from lazy
to eager/cached; all in-repo callers (_ShardReader.__iter__,
_is_total_shard, _subchunk_order_iter, and two tests) are migrated to
`lexicographic_order_iter`, so nothing in-tree relies on the old laziness.

Output is unchanged: lexicographic_order_iter, the c_order_iter alias,
and np.ndindex all agree across 0-d, empty, and N-d shapes.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* refactor(indexing): make lexicographic_order_iter the lazy primitive

Per review from @mkitti: invert the relationship between the lazy
iterator and the eagerly-collected tuple. `lexicographic_order_iter` is
now a genuine lazy generator over the chunk-grid coordinates, and
`_lexicographic_order_keys` collects it into a cached tuple — the eager
version is "collect the lazy one", not the other way around.

Previously lexicographic_order_iter returned iter() over the cached
tuple, so any consumer that only needed a prefix still paid to
materialize the entire grid. _is_total_shard does exactly that — an
early-exit `all(coord in set for coord in ...)` — and on a cold cache
for a 32^3 shard whose first coordinate is absent this dropped from
~15.8ms to ~24us (the lazy generator builds one coordinate and bails).

The hot path is unchanged: the two dict.fromkeys sites consume the full
grid and use the cached `_lexicographic_order_keys` tuple directly
(~0.9ms at 32^3), so the regression fix this branch delivers is intact.
This also resolves @chuckwondo's point — the iterator is now actually
lazy rather than a thin wrapper over eager data.

Co-authored-by: Mark Kittisopikul <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* refactor(indexing): make morton_order_iter the lazy primitive too

Per @mkitti: the morton pair was backwards in the same way the
lexicographic pair was. Invert it to match — `morton_order_iter` is now
the lazy generator primitive and `_morton_order_keys` collects it into a
cached tuple, mirroring `lexicographic_order_iter` /
`_lexicographic_order_keys`.

No behavioral change for the in-tree consumers (all fully consume the
sequence) and the Z-order is identical; this keeps the two coordinate-
order families symmetric and gives morton the same lazy/early-exit
option lexicographic now has.

Co-authored-by: Mark Kittisopikul <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* refactor(indexing): expose chunk-order coordinates as cached sequences

Replace the morton/lexicographic order iterators (and the c_order_iter
alias) with two cached, numpy-backed sequences:
`morton_order_coords(shape)` and `lexicographic_order_coords(shape)`,
each returning the grid coordinates in that order as a tuple of
coordinate tuples.

This addresses several points from review:

- The earlier "lazy primitive" inversion de-optimized the hot write
  path: `morton_order_iter` rebuilt every coordinate tuple from the
  array on each call, and that path runs in `_encode_shard_dict` on
  every shard write (~16ms/write at 32^3 chunks-per-shard). The coords
  are a finite set of known length reused in full, so they are an
  indexable sequence built once and cached, not a lazily-rebuilt
  generator. (per @mkitti)

- `lexicographic_order_iter` was never genuinely lazy — `_lexicographic_order`
  materializes the whole `np.indices` grid up front — so the early-exit
  framing was inaccurate. (per @Copilot, @chuckwondo)

- Two functions differing only in caching vs laziness was redundant
  (per @ilan-gold); there is now one sequence per order. `_ShardReader.__iter__`
  wraps it in `iter()`, the only site that needs an iterator.

- `_is_total_shard` no longer iterates the order at all: `all_chunk_coords`
  is always a subset of the shard grid (guaranteed by `validate`'s
  shard/chunk divisibility check), so a count check proves totality. A
  subset assertion documents the invariant.

Coordinates are Python int tuples because every consumer uses them as
dict keys / set members, which numpy arrays cannot be (unhashable,
mutable); the numpy array is kept only for the vectorized index lookup
in `to_dict_vectorized`. The per-shape cache holds ~prod(chunks_per_shard)
tuples (~0.07% of shard size for multi-GB shards with (64,64,64) chunks),
capped at 16 shapes per order.

Co-authored-by: Mark Kittisopikul <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* test(bench): add warm-cache shard-write benchmark

The existing test_sharded_morton_write_single_chunk clears the chunk-order
cache before every iteration, so it only measures the cold grid-build cost.
That made it blind to a regression where the per-shard coordinate tuples
were rebuilt on every write instead of being reused from the cache — the
cold benchmark could not distinguish the two (both pay the build each
iteration).

Add test_sharded_morton_write_single_chunk_warm_cache, which warms the
cache once and then times repeated same-shape writes — the amortized
regime the cache exists to optimize (many shards of one shape per array).
Verified it discriminates: with the cached sequence it is ~4x faster than
the cold benchmark, and a rebuild-every-write regression shows up as a ~4x
slowdown here while staying invisible to the cold benchmark.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* docs: update changelog for full-shard write coverage

The fix caches the per-shard coordinate grid for every shard write, not
only partial writes, and the win is amortized across repeated writes to
same-shaped shards. Reword the note accordingly; keep it user-facing
(the internal indexing helper refactor is not part of the public API).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* perf: build order-coord tuples via .tolist(); document dual representation

`morton_order_coords` / `lexicographic_order_coords` built their tuple-of-
tuples with a row-by-row `tuple(int(x) for x in row)` comprehension. Using
`map(tuple, arr.tolist())` instead does the int conversion in a single
C-level call, producing byte-identical native-int tuples ~8-9x faster
(~16ms -> ~1.9ms cold build at 32^3). It is a per-shape cached build, so
this only speeds the first write to each shard shape, but it is free.

Also document in `to_dict_vectorized` why the chunk coordinates are needed
in two forms — a numpy array for the vectorized index lookup and hashable
tuples for the dict keys — since numpy rows are unhashable and a tuple list
can't be used for the vectorized modulo/advanced-indexing.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* perf,test: address code-review findings in the sharding coord cache

- Drop the O(n_chunks) assert in _is_total_shard. It built a fresh
  set(lexicographic_order_coords(...)) on every partial read/write to
  check an invariant `validate` already guarantees, regressing the very
  partial-access hot path this PR optimizes (~673us vs ~112ns at 32^3
  chunks-per-shard) and vanishing under -O. The invariant is documented
  in the comment; the count check alone proves totality.

- Cache the colexicographic subchunk order. The colex branch of
  _subchunk_order_iter rebuilt the grid via uncached np.ndindex on every
  write while its morton/lexicographic siblings hit the cache; add
  colexicographic_order_coords (cached, derived from lexicographic_order_coords
  of the reversed shape) and use it.

- Fix two benchmark docstrings: the cold benchmark now clears the
  lexicographic caches too (the write path builds that grid via
  dict.fromkeys / to_dict_vectorized, so a morton-only clear left it warm
  and under-reported the cold cost); the warm benchmark docstring now
  describes what it actually exercises (repeated writes to one shard,
  which reuse the cache identically to writes across same-shaped shards).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ilan Gold <[email protected]>
Co-authored-by: Mark Kittisopikul <[email protected]>

* feat: define `PreparedWrite` and `SupportsChunkPacking` data structures

`PreparedWrite` models a set of per-chunk changes that would be applied to a stored chunk. `SupportsChunkPacking`
is a protocol for array -> bytes codecs that can use `PreparedWrite` objects to update an existing chunk.

* feat: new codec pipeline that uses sync path

* feat: complete second codecpipeline

* fix: handle rectilinear chunks

* fixup

* feat: SupportsSetRange protocol + sync byte-range writes

Adds a SupportsSetRange protocol to zarr.abc.store for stores that
allow overwriting a byte range within an existing value. Implementations
are added for LocalStore (using file-handle seek+write) and MemoryStore
(in-memory bytearray slice assignment).

This is the prerequisite for the partial-shard write fast path in
ShardingCodec, which can patch individual inner-chunk slots without
rewriting the entire shard blob when the inner codec chain is fixed-size.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat: add sync codec methods to V2 and numcodecs codecs

V2Codec, BytesCodec, BloscCodec, etc. previously only implemented the
async _decode_single / _encode_single methods. Add their sync
counterparts (_decode_sync / _encode_sync) so that the upcoming
SyncCodecPipeline can dispatch through them without spinning up an
event loop.

For codecs that wrap external compressors (numcodecs.Zstd, numcodecs.Blosc,
the V2 fallback chain), the sync versions just call the underlying
compressor's blocking API directly instead of routing through
asyncio.to_thread.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat: SyncCodecPipeline — synchronous codec pipeline with per-chunk parallelism

Adds SyncCodecPipeline alongside BatchedCodecPipeline. The new pipeline
runs codecs through their sync entry points (_decode_sync / _encode_sync)
and dispatches per-chunk work to a module-level thread pool sized by
the codec_pipeline.max_workers config (default = os.cpu_count()).

Each chunk's full lifecycle (fetch + decode + scatter for reads;
get-existing + merge + encode + set/delete for writes) runs as one
pool task — overlapping IO of one chunk with compute of another.
Scatter into the shared output buffer is thread-safe because chunks
have non-overlapping output selections.

The async wrappers (read/write) detect SupportsGetSync/SupportsSetSync
stores and dispatch to the sync fast path, passing the configured
max_workers. Other stores fall through to the async path, which still
uses asyncio.concurrent_map at async.concurrency.

Notes on perf:
- Default (None → cpu_count) is tuned for chunks ≥ ~512 KB.
- Small chunks (≤ 64 KB) regress 1.5-3x because pool dispatch overhead
  (~30-50 µs/task) dominates per-chunk work. Workaround:
  zarr.config.set({"codec_pipeline.max_workers": 1}).
- For large chunks on local/memory stores, IO+compute parallelism
  yields 1.7-2.5x over BatchedCodecPipeline on direct-API reads and
  ~2.5x on roundtrip.

ChunkTransform encapsulates the sync codec chain. It caches resolved
ArraySpecs across calls with the same chunk_spec — combined with the
constant-ArraySpec optimization in indexing, hot-path overhead is
minimized.

Includes test scaffolding for the new pipeline (test_sync_codec_pipeline)
and config plumbing for the max_workers key.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat: partial-shard write support in ShardingCodec

Adds _encode_partial_sync and _decode_partial_sync to ShardingCodec.
For fixed-size inner codec chains and stores that implement
SupportsSetRange, partial writes patch individual inner-chunk slots
in-place instead of rewriting the whole shard:

  - Reads existing shard index (one byte-range get).
  - For each affected inner chunk: decodes the slot, merges the new
    region, re-encodes.
  - Writes each modified slot at its deterministic byte offset, then
    rewrites just the index.

For variable-size inner codecs (e.g. with compression) or stores that
don't support byte-range writes, falls through to a full-shard rewrite
matching BatchedCodecPipeline semantics.

The partial-decode path computes a ReadPlan from the shard index and
issues one byte-range get per overlapping chunk, decoding only what
the read selection touches.

Both paths are dispatched from SyncCodecPipeline via the existing
supports_partial_decode / supports_partial_encode protocol checks.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test: codec invariants + pipeline parity matrix

Two new test files:

  test_codec_invariants — asserts contract-level properties that every
  codec / shard / buffer combination must satisfy: round-trip exactness,
  prototype propagation, fill-value handling, all-empty shard handling.

  test_pipeline_parity — exhaustive matrix asserting that
  SyncCodecPipeline and BatchedCodecPipeline produce semantically
  identical results across codec configs, layouts (including
  nested sharding), write sequences, and write_empty_chunks settings.
  Three checks per cell:
    1. Same array contents on read.
    2. Same set of store keys after writes.
    3. Each pipeline reads the other's output identically (catches
       layout-divergence bugs).

These tests pinned the design throughout the SyncCodecPipeline +
partial-shard development.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore: gitignore local agent/planning notes

Adds .gitignore entries for .claude/, CLAUDE.md, and docs/superpowers/
so local IDE/agent planning artifacts don't get committed by accident.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore: remove unused PreparedWrite and SupportsChunkCodec

Both were exported from zarr.abc.codec.__all__ but never referenced
by either codec pipeline or any test. Artifacts of an earlier design
iteration superseded by the current SyncCodecPipeline.

Also remove now-unused imports of `dataclass` and `ChunkProjection`
that were only needed by the deleted symbols.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore: remove stale phased-pipeline test files

Both tests/test_phased_codec_pipeline.py and tests/test_pipeline_benchmark.py
import PhasedCodecPipeline, which no longer exists in src/. Each failed at
collection. The benchmarking intent of test_pipeline_benchmark.py is replaced
by extensions to tests/benchmarks/test_e2e.py later in this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* refactor: rename SyncCodecPipeline to FusedCodecPipeline

The new name describes what the pipeline does (fuses fetch+decode+scatter
into one task per chunk) rather than the implementation detail of using
sync codec entry points. The name also stays accurate when this pipeline
gains a remote-store / async fast path in a future change.

Mechanical rename across the class, register_pipeline call, dotted-path
strings used by zarr.config, isinstance checks, parametrize values, and
docstrings. tests/test_sync_pipeline.py renamed to tests/test_fused_pipeline.py.

Nothing on this branch is released, so no deprecation alias is needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* refactor: lift _merge_chunk_array to module level

The BatchedCodecPipeline and FusedCodecPipeline classes had identical
copies of _merge_chunk_array (one method, one staticmethod). Extract
once as a module-level free function and call from both. No new base
class or mixin is introduced.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* refactor: extract _async_read_fallback to module level

Both BatchedCodecPipeline.read_batch (non-partial-decode branch) and
FusedCodecPipeline.read (async fallback) duplicate the same sequence:
concurrent_map(get) -> pipeline.decode -> scatter into out. Lift to a
module-level free function and call from both.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* refactor: extract _async_write_fallback to module level

Both BatchedCodecPipeline.write_batch (non-partial-encode branch) and
FusedCodecPipeline.write (async fallback) duplicate the same sequence:
read existing bytes -> decode -> merge -> encode -> set/delete. Lift to
a module-level free function and call from both.

After this change, neither pipeline class carries _merge_chunk_array,
nor the duplicated read/write fallback bodies. Each class is reduced to
its constructor, fast-path methods, and thin async dispatch.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test(bench): parametrize test_e2e over both codec pipelines

Adds a `pipeline` fixture with values ["batched", "fused"] that swaps
codec_pipeline.path for the duration of each benchmark. Both
test_write_array and test_read_array now produce one benchmark cell
per (compression x layout x store x pipeline). CodSpeed will report
comparable numbers for both pipelines on the same workloads.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* test(bench): parametrize test_e2e over a synthetic latency dimension

Adds `latency in {0, 0.001, 0.05, 0.2}` and a bench_store fixture that
wraps the underlying memory store in zarr.testing.store.LatencyStore
when latency > 0. Local-store cells skip nonzero latency — adding
synthetic latency on top of a real filesystem double-counts and is not
the intended measurement.

Combined with the pipeline parameter, the matrix now produces
comparable benchmark numbers for {Batched, Fused} x {0, 1ms, 50ms, 200ms}
on memory-shaped operation. The numbers are signal under one simple
model of remote latency, not absolute predictions of S3 behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* chore: restore deleted comments in V2Codec._decode_sync

Commit 7f45aba9 (which converted _decode_single -> _decode_sync) dropped
two explanatory comment blocks from the dtype-handling branches in
V2Codec.decode. Both comments document non-obvious WHY:

- The TypeError catch is for chunks whose stored dtype doesn't match
  the array spec dtype (e.g. string dtype vs object array).
- The elif branch fires when filters were tampered with: an object array
  needs an object codec in the filter chain to be read correctly.

These were removed as drive-by cleanup during the sync-method rename
without intent to delete the substance. Restoring verbatim.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs: explain non-obvious behaviors in sharding sync methods

Add docstring substance and a couple of inline notes to the new sync
methods on ShardingCodec that landed on this branch. Concretely:

- _decode_sync / _encode_sync: explain how each relates to the async
  counterpart and the partial-* variants, and why inner chunks are
  iterated in Morton order on the encode path.
- _encode_shard_dict_sync: explain the two-pass offset shift in the
  index-at-start branch (offsets are written relative to the data
  section, then bumped by len(index_bytes)) and the MAX_UINT_64
  empty-chunk sentinel that must not be touched.
- _encode_partial_sync byte-range path: explain WHY morton-rank
  determines byte offset deterministically (fixed-size inner chunks =
  every slot at a stable offset regardless of which others are present);
  this is the load-bearing invariant for the byte-range fast path.
- _decode_partial_sync: docstring now lists the two sub-paths
  (full-shard fetch vs. index-then-byte-ranges) and the reason the
  full-shard branch exists (one round trip beats N+1 small ones).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* docs: convert RST inline literals to Markdown-style backticks

Docstrings added on this branch used RST-style ``literal`` markup
(double-backticks). Convert to Markdown-style `literal` (single
backticks) so the docstrings render correctly in Markdown-aware
viewers without needing a separate RST-to-Markdown step.

Two cases worth calling out:

- src/zarr/core/codec_pipeline.py and src/zarr/codecs/sharding.py:
  every ``literal`` in these files came in on this branch, so the
  conversion is global within those files.
- src/zarr/abc/store.py and src/zarr/core/array.py: only docstrings
  added on this branch are converted; pre-existing RST-style
  literals from main are left alone (out of scope).

Also converted one .. note:: directive in
src/zarr/core/array.py (the regular_chunk_array_spec helper) to
a Markdown blockquote, since that directive was added on this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* perf: memoize encoded inner chunk for scalar complete-shard writes (#177)

In ShardingCodec._encode_partial_sync's full-shard-rewrite loop, a scalar
broadcast value produces byte-for-byte identical results for every complete
inner chunk (same fill, same empty-check, same encoded bytes). Compute that
outcome once and reuse it across all complete chunks instead of re-merging,
re-checking write_empty_chunks, and re-encoding tens of thousands of identical
chunks. Incomplete edge chunks still merge against their own data individually.

Target case (fused, memory, chunks=100/shards=1M, no compression):
write 92.26ms -> 21.59ms (4.3x). Pipeline parity (byte-identical to batched)
and 956 tests pass under the fused pipeline; adversarial partial-overwrite/
edge/compression/2D/aliasing checks pass.

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* perf+fix: bulk whole-shard read + repair _ShardIndex construction post-merge

Two things, both scoped to the sync sharding read path:

1. Fix: main's #3975 made _ShardIndex a 2-field NamedTuple (chunks_per_shard,
   offsets_and_lengths), but the Fused sync methods still constructed it with one
   arg, erroring on every Fused sharded read. Pass chunks_per_shard through in
   _decode_shard_index_sync and the byte-range write path.

2. Perf: _decode_full_shard_bulk + _ShardIndex.is_dense. A whole-shard read of a
   dense, fixed-size, uncompressed shard is reconstructed by reshaping/scattering
   the data section in bulk, replacing the per-chunk decode/index/projection loop
   (~78% of a full read). Chunk positions are read from the stored index, so it is
   correct for any subchunk_write_order. Falls through to the per-chunk path for
   compression/filters, non-dense shards, and any read whose output shape != the
   shard shape (strided/partial/fancy).

Full read (memory, 10000 chunks/shard, uint8): ~291ms -> ~21ms (13.9x vs Batched).
Verified: 0 new test failures vs the merge baseline; full reads correct across
dtypes and 2D; partial/strided/gzip fall through. (Pre-existing Fused x
subchunk_write_order gaps remain, tracked separately.)

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: FusedCodecPipeline honors subchunk_write_order + coalesced partial reads

Three integration gaps surfaced when the Fused pipeline met main's new
subchunk_write_order (#3826), partial-read coalescing (#3004), and _ShardIndex
refactor. Under Fused these caused 25 sharding/parity failures (data was
correct in the partial-read cases; the failures were write-order layout +
IO-pattern divergence). Fixes:

1. Write order: _encode_shard_dict_sync laid out chunks in hardcoded morton
   order, ignoring subchunk_write_order. Now iterates
   _subchunk_order_iter(self.subchunk_write_order), matching the async
   _encode_shard_dict. Fixes lexicographic/colexicographic/unordered storage.

2. Coalesced sync partial reads: add Store.get_ranges_sync (a synchronous,
   coalescing counterpart of get_ranges, reusing coalesce_ranges) and
   ShardingCodec._load_partial_shard_maybe_sync; route _decode_partial_sync's
   partial branch through it. Sync stores now get #3004's byte-range coalescing
   without an event loop (fewer, merged reads).

3. Non-sync fallback: FusedCodecPipeline.read now routes non-sync stores (e.g.
   ZipStore) through the async partial-decode path when the AB codec supports
   it, instead of _async_read_fallback's whole-shard get(). Matches Batched's
   IO behavior; avoids over-reading whole shards on partial reads.

Tests: the #3004 partial-read tests are made pipeline-aware (assert the active
method family: get/get_ranges vs get_sync/get_ranges_sync, gated on store sync
support). 573 sharding+parity+pipeline+indexing and 657 codec tests pass under
BOTH pipelines (was 25 failing under Fused).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: address roborev review (job 222) — Fused sharding correctness

HIGH (sharding.py, byte-range write fast path): derived each chunk's physical
slot from self.subchunk_write_order instead of hardcoded morton order, and
excluded 'unordered' (no recoverable rank -> falls through to the index-driven
full-rewrite path). A partial write into a dense shard first written with a
non-default order no longer corrupts data via wrong byte offsets.

HIGH (sharding.py, _decode_full_shard_bulk): build the read-view dtype from the
BytesCodec's endian (as BytesCodec._decode_sync does), not the dtype's native
endianness. A big-endian shard read on a little-endian host (or vice versa) now
decodes correctly instead of silently reinterpreting bytes.

MEDIUM (sharding.py, _decode_full_shard_bulk): the bulk fast path now requires
the inner chain to be exactly one BytesCodec, excluding crc-bearing shards. The
bulk path can't verify per-chunk checksums, so crc shards fall through to the
per-chunk path and keep their corruption detection.

LOW (codec_pipeline.py, ChunkTransform._resolve_specs): key the resolved-spec
cache on the frozen, hashable ArraySpec value instead of (shape, id()), which
could collide after id reuse.

LOW (codec_pipeline.py, _get_pool): don't shutdown(wait=False) the old pool on
grow — a concurrent in-flight pool.map could hit 'cannot schedule new futures
after shutdown'. The orphaned pool drains and is GC'd.

Tests: extended test_pipeline_parity with big-endian + crc32c codec configs and
a dedicated subchunk_write_order x index_location parity test (asserts identical
contents always, identical bytes for deterministic orders). Verified each new
test fails when its corresponding fix is reverted. 1219 tests pass under both
pipelines; mypy clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* docs: correct FusedCodecPipeline framing — sync scheduling, not IO/compute separation

The class docstring claimed it 'separates IO from compute', then immediately
said the ShardingCodec does IO internally — self-contradictory and misleading.
The actual win is replacing per-chunk ASYNC scheduling with synchronous,
batched/coalesced execution; the sharding codec still owns its storage IO
(the zarrs model, unlike tensorstore's storage-free codecs). Rewrite the
docstring to state this plainly and note that a storage-free codec is a
possible future direction, not what this pipeline does. No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* refactor: stop hard-coding assumptions about the 'unordered' write order

After merging #4011 (which made 'unordered' deterministic and warns callers not
to rely on its layout), drop the two places my earlier fixes special-cased it by
name:

- Byte-range write fast path: remove the 'subchunk_write_order != unordered'
  gate. The rank map is derived from _subchunk_order_iter(self.subchunk_write_
  order), which is the single source of truth for physical layout — correct for
  every order without a name check. _subchunk_order_iter is the only place that
  knows a given order's layout.
- Parity test: assert byte-equality across pipelines for ALL orders, not just
  'deterministic' ones. The check verifies the two pipelines AGREE (they share
  _subchunk_order_iter), which holds whatever an order resolves to; it makes no
  assumption about what 'unordered' means.

540 parity+sharding and 862 codec/indexing tests pass under both pipelines; mypy clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* feat: make FusedCodecPipeline the default codec pipeline

Flip codec_pipeline.path default from BatchedCodecPipeline to FusedCodecPipeline.
Fused runs codec compute synchronously/in bulk and gives large speedups on
sharded workloads (up to ~24x write / ~14x read on many-chunks-per-shard, more
with compression) and no regressions on compute-bound cases; it falls back to
the async path for non-sync stores. Batched remains selectable via config.

Test fallout from the flip (all behavior, not stale-assertion churn):
- test_config_defaults_set: expected default path updated.
- test_config_codec_implementation: the mock codec now also overrides
  _encode_sync, so it records a call regardless of which pipeline is default
  (Fused uses the sync entry point).
- StoreExpectingTestBuffer (zarr.testing.buffer): added set_sync/get_sync that
  mirror the async buffer-type guards, so the 'all buffers are TestBuffer'
  invariant is checked on the sync write path too. Verified Fused correctly
  threads a custom BufferPrototype (sharded writes store TestBuffer instances) —
  the test simply wasn't exercising the sync path before.

Full suite: 6346 passed, 0 failed under the new default.

NOTE: changelog fragment filename is a PLACEHOLDER — rename
changes/PLACEHOLDER-fused-default.feature.md to changes/<PR#>.feature.md once the
PR number is known (towncrier keys fragments by issue/PR number).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: ShardingCodec inner pipeline follows the configured default, not hard-coded Batched

The codec_pipeline property hard-coded BatchedCodecPipeline.from_codecs(). main
resolves it against the registry via get_pipeline_class() (#2179); the branch
carried an older hard-coded version and the main-merge kept the branch side.
With FusedCodecPipeline now the default this left the inner sub-chunk pipeline
stuck on Batched while the outer array used Fused — an inconsistency, and stale
relative to main. Restore get_pipeline_class().from_codecs(), matching the rest
of this module (which already uses get_pipeline_class elsewhere).

Verified: sharding + parity + pipeline (596) and codecs+array+indexing+properties
(2161) pass; nested sharding roundtrips correctly under both pipelines; no
functional BatchedCodecPipeline references remain in sharding.py. mypy clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: Fused async decode/encode must evolve codec specs (HIGH-2) + shared CodecPipelineTests

HIGH-2: FusedCodecPipeline.decode()/encode() (the async fallback for non-sync
stores) reused one flat chunk_spec across every codec stage instead of evolving
it per codec via resolve_metadata. Spec-changing array->array codecs broke:
TransposeCodec crashed on read (could not broadcast (2,2) into (2,4));
cast_value/scale_offset would silently corrupt. Reachable on the DEFAULT
pipeline for every non-sync store (S3/GCS/fsspec/zip).

Fix, without re-duplicating spec logic (the duplication caused the bug):
- Extract resolve_aa_specs(): single source of truth for per-stage spec
  evolution (forward-thread resolve_metadata over the AA codecs). Pure metadata.
- Add AsyncChunkTransform: per-chunk ASYNC mirror of ChunkTransform, driving the
  codecs' async _decode_single/_encode_single with the correct per-stage spec.
  No mini-batch concept (that stays a BatchedCodecPipeline concern).
- ChunkTransform._resolve_specs delegates to resolve_aa_specs.
- Fused.decode()/encode() loop per chunk through AsyncChunkTransform.

Also harden the sharding byte-range WRITE fast path: take chunk offsets from the
stored shard index, not from the live subchunk_write_order (which is not
recoverable on reopen by design).

New tests/test_codec_pipeline_suite.py: xUnit CodecPipelineTests base run as
TestBatchedPipeline and TestFusedPipeline over a sync (MemoryStore) AND a
non-sync (LatencyStore) store axis. Reproduces HIGH-2 automatically. 140 pass;
mypy clean; original ZipStore+transpose crash now roundtrips.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* test: dedupe codec-pipeline tests against the shared CodecPipelineTests suite

The shared suite runs every pipeline-agnostic behavior test against BOTH
pipelines x both store paths, so per-file copies of the same behavior are
redundant. Remove confirmed duplicates; keep tests that exercise something the
suite does not.

- Strengthen the suite's write_empty_chunks tests to also assert chunk-key
  presence/absence (absorbing the old _no_store / _persists coverage).
- test_codec_pipeline.py: drop the 8 behavior duplicates now in the suite. KEEP
  test_read_returns_get_results (low-level pipeline.read GetResult API),
  test_write_empty_chunks_false_no_store (store-key shape), and
  test_codec_pipeline_threads_dtype_through_evolve (#3937 regression).
- test_fused_pipeline.py: drop the array-level streaming read/write tests and
  test_partial_shard_write_roundtrip_correctness (array behavior, suite-covered).
  KEEP all pipeline-API / Fused-internal tests (construction, evolve, low-level
  write/read(_sync) roundtrips, sync-write/async-read interop, ChunkTransform
  encode/decode, set_range, inner_codecs_fixed_size, byte-range fast path).

740 pass across suite + codec_pipeline + fused + sync + invariants + parity +
sharding; ruff + mypy clean. No coverage removed without a verified equivalent.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* test: unify the create/write/read suite tests into one Scenario-parametrized test

The bulk of CodecPipelineTests followed one shape: create an array, apply some
writes, optionally assert which chunk keys exist, then assert reads come back
correct. Capture those variables in a frozen Scenario dataclass (array_kwargs,
writes, reads, keys_present/absent) and drive them all through a single
parametrized test_scenario. Correctness is checked against a numpy reference the
scenario derives from its own writes, so cases don't hand-maintain expected
values. 18 scenarios cover the same matrix (layouts, gzip, transpose
spec-evolution, nested sharding, partial-shard overwrite, write_empty key
presence/absence) x both pipelines x sync/async stores.

Kept as separate focused tests the two cases that don't fit the shape:
test_read_missing_chunks_false_raises (asserts an exception) and
test_partial_write_after_reopen_is_correct (has an extra reopen step).

Verified the parametrized form keeps its regression-guard value: reverting the
HIGH-2 spec-evolution fix still fails test_scenario[async-transpose]. 670 pass
across pipeline + sharding suites; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* test: prune test_fused_pipeline.py to its irreducible Fused-specific core

The Fused test file had accumulated tests that either duplicated the
pipeline-agnostic CodecPipelineTests suite or were misfiled. Triage:

- async roundtrip / missing-chunk-fill / partial-shard-write dups: removed;
  the shared test_scenario covers these across both pipelines x sync/async
  stores. Added float32 and zstd Scenarios first so the dtype/codec coverage
  the dups carried transfers to the shared matrix (no net coverage loss).
- store set_range / SupportsSetRange tests: already covered (more thoroughly,
  parametrized) in tests/test_store/test_memory.py; removed as dups.
- ShardingCodec._inner_codecs_fixed_size tests: moved to
  tests/test_codecs/test_sharding_unit.py where the sharding internals live.

What stays is genuinely Fused-only and cannot be pipeline-agnostic: the
synchronous API (write_sync / read_sync / _sync_transform) which Batched has
no equivalent of, and the byte-range fast-path assertions (set_range_sync
fires / falls back) which test a Fused-only optimization.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: dissolve test_codec_invariants.py, redistributing by subject

The "invariants" file grouped tests by their shared motivation (a design
doc) rather than by what they test, which is the wrong axis -- it mixed
pipeline-agnostic behavior, Fused-only internals, and a per-codec property
into one file. Sorted each test into the home its subject implies:

Pipeline-agnostic behavior -> CodecPipelineTests (runs on BOTH pipelines x
sync/async stores via the existing fixtures):
- S2 empty-chunk skipping under default config -> a Scenario (keys_absent).
- S2 shard deleted after overwrite-to-fill -> a base-class method (it needs
  a mid-sequence key assertion the Scenario shape can't express).
- C3 no isinstance(ShardingCodec) branching in read/write -> a base-class
  method that resolves the subclass's configured pipeline and source-scans it.

Fused-only (byte-range fast path / ChunkTransform internals) ->
test_fused_pipeline.py:
- S3 fast path skipped when write_empty_chunks=False (the unique complement
  of the existing uses-set-range test; the write_empty_chunks=True case was a
  dup and is dropped).
- B1 byte-range path copies read-only LocalStore buffers before mutating.
- C2 ChunkTransform passes each codec the runtime chunk_spec prototype.

Per-codec contract -> tests/test_codecs/test_codecs.py:
- C1 resolve_metadata only mutates shape (prototype/dtype/fill_value/config
  stable across the chain) -- a property of individual codecs, no pipeline.

Dropped as a pure duplicate (already in test_store/test_memory.py):
- test_supports_set_range_is_runtime_checkable.

No coverage lost: every kept test moved, and the two genuinely-shared
behaviors now run on both pipelines instead of only whichever was default.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: drop redundant read-parity matrix, move partial-read coverage to shared suite

test_pipeline_read_parity checked Fused vs Batched partial reads against
*each other*. The shared CodecPipelineTests suite already reads partial/strided
selections from sharded arrays against a numpy reference on BOTH pipelines --
which is strictly stronger (it would catch both pipelines diverging from the
spec in the same way, which a pipeline-vs-pipeline check cannot).

The one sliver read-parity covered that the shared suite didn't was scalar
single-element reads from a sharded array (the sharding codec's partial-decode
path). Added two Scenarios (sharded-scalar-reads-1d / -2d) to capture it.
Verified they exercise the partial-decode path on both pipelines: the default
Fused pipeline routes a scalar sharded read through _decode_partial_sync, the
Batched pipeline through _decode_partial_single -- so both variants are now
checked against numpy, not just against each other.

Kept in test_pipeline_parity.py the two checks the per-pipeline suite cannot
express, because its two subclasses run in isolation and never see each other's
output:
- test_pipeline_parity: cross-read interop (write under A, read whole under B)
  + cross-pipeline store-key-set equality.
- test_pipeline_parity_subchunk_write_order: byte-identical shard output across
  pipelines for every subchunk_write_order x index_location.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: rename test_sync_codec_pipeline -> test_chunk_transform; drop cross-file dup

The file named test_sync_codec_pipeline.py tested no pipeline -- it is the unit
test suite for ChunkTransform (the per-chunk synchronous codec chain that
FusedCodecPipeline uses internally). "sync codec pipeline" was an earlier name
for the Fused pipeline; the filename had outlived it. Renamed to
test_chunk_transform.py (git mv preserves history) and added a module docstring
naming what it actually covers.

Also removed test_sync_transform_encode_decode_roundtrip from
test_fused_pipeline.py: it was a weaker cross-file duplicate of this file's
test_encode_decode_roundtrip (which covers the same encode->decode->compare over
five codec chains rather than just bytes-only). Its one extra assertion -- that
evolve_from_array_spec populates _sync_transform -- is already covered by
test_evolve_from_array_spec in the Fused file.

test_codec_pipeline.py left as-is: all three tests are correctly placed and
cover things the Scenario suite can't (the low-level pipeline.read GetResult
API, a plain dict store, and the #3937 cast_value dtype-threading regression).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* feat: remove byte-range-write support pending store-interface decision

The byte-range-write machinery works, but the right store interface for it is
still undecided, so it is removed from this PR and will return once that lands.

Removed:
- SupportsSetRange protocol (abc/store.py) and its __all__ export.
- MemoryStore.set_range / set_range_sync / _set_range_impl and the
  SupportsSetRange base (storage/_memory.py).
- LocalStore.set_range / set_range_sync, the _put_range helper, and the
  SupportsSetRange base (storage/_local.py).
- The sharding codec's byte-range-write fast path in _encode_partial_sync;
  partial shard writes now always take the full-shard-rewrite path (identical
  to BatchedCodecPipeline, verified by the pipeline-parity suite). Also dropped
  the now-dead _chunk_byte_offset helper it relied on.
- changes/3907.feature.md (the byte-range-writes changelog note). The
  byte-range-READ changelog (3004) is unrelated and kept.

Byte-range READS (ByteRequest, get(byte_range=), get_ranges coalescing,
the read-side bulk shard decode) are untouched -- this only removes writes.

The known-good tests that exercise byte-range writes are commented out (not
deleted) in test_store/test_memory.py, test_store/test_local.py, and
test_fused_pipeline.py, to restore once the store design is settled.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* refactor: remove dead _get_default_chunk_spec helper

PR-added module-level helper in array.py with zero callers — an ArraySpec-reuse
optimization that was never wired up. Plain function, no protocol role, safe to
drop. Verified: no references anywhere in src/ or tests/, and the full
array/sharding/pipeline suites stay green.

Note: ShardingCodec._encode_sync, though never *called*, is NOT dead — it is a
required member of the runtime_checkable SupportsSyncCodec protocol. Removing it
drops ShardingCodec from SupportsSyncCodec and breaks the sync read-fallback
routing (16 test failures), so it stays.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs: correct ShardingCodec._encode_sync docstring re: write order

The docstring claimed _encode_sync "iterates inner chunks in Morton order —
that's the canonical layout the shard index expects", which is wrong and a
latent footgun: it implies the method imposes a morton physical layout. It does
not. The morton iteration only populates an intermediate dict whose key order is
immaterial; the on-disk layout is decided downstream by the subchunk_write_order
loop in _encode_shard_dict_sync (same as the async _encode_single sibling).

Also clarified that this method IS reached — via nested sharding, where an inner
ShardingCodec is encoded through the outer codec's ChunkTransform. (It is not
called for top-level sharded writes, which route through _encode_partial_sync.)

Verified empirically: routing through nested _encode_sync, all three
subchunk_write_order values roundtrip correctly AND morton vs lexicographic
produce physically different bytes — i.e. the order is honored, not ignored.
Behavior unchanged; docstring only.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* refactor: remove unused ShardingCodec._load_shard_index wrapper

PR-added thin wrapper (`_load_shard_index_maybe(...) or _ShardIndex.create_empty(...)`)
with zero invocations anywhere in src/ or tests/. Unlike _encode_sync, this is
genuinely removable: confirmed it is NOT a member of any runtime_checkable
protocol or ABC (no reference in src/zarr/abc/, not a base-class override) and is
reached by no dynamic dispatch (no getattr / string reference). main has no
_load_shard_index* methods at all, so it was introduced and left unused by this
PR. The _maybe and _maybe_sync variants it wrapped remain and are used.

Verified: full sharding + nested-sharding + parity + pipeline suites stay green,
ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs: drop stale set_range_sync mention from FusedCodecPipeline docstring

The FusedCodecPipeline class docstring still described sharded writes as using
"byte-range writes via set_range_sync" — but byte-range-write support was removed
from this PR (set_range_sync / SupportsSetRange are gone). Sharded writes now take
the codec's synchronous full-shard-rewrite path. Docstring only; no behavior change.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* docs: use plain single backticks in docstrings, not RST double-backticks

This branch's docstrings/comments had introduced RST-style ``double-backtick``
inline literals, which this project does not use (plain single backticks only —
no RST roles or double-backticks). Converted the 25 occurrences across the
sharding codec, codec_pipeline, and fsspec store docstrings/comments to single
backticks. Style only; no behavior change.

Also confirmed (via git blame, this-branch lines only) there are no remaining
references to removed/outdated designs: the byte-range-write (set_range) mentions
and the "separating IO from compute" framing were already corrected earlier in
this branch.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* feat: default codec_pipeline.max_workers to 1 (sequential), threading opt-in

Pairs with the FusedCodecPipeline default: keep the new pipeline, but do NOT
enable threading by default. `max_workers=None` (auto -> cpu_count) spawned a
thread pool on every read/write, which is a behavior change with real downstream
risk — it runs custom stores/codecs concurrently (thread-safety) and can
oversubscribe many-core nodes whose workloads already parallelize at the
dask/MPI layer. The default is now 1 (fully sequential: the pool is never
created when max_workers <= 1). Parallelism is opt-in via
`codec_pipeline.max_workers` (positive int, or None for auto).

Updates _resolve_max_workers docstring and the config-defaults test accordingly.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* perf: vectorize shard_dict build in _encode_partial_sync (fix write regression)

CodSpeed flagged test_sharded_morton_write_single_chunk regressing ~38-39%
(writing one 1x1x1 chunk into a 32^3 = 32768-chunk shard). Both main and this
branch do a full shard rewrite for a partial write, so the rewrite itself is not
the regression — and it is NOT the removed byte-range fast path (that path was
gated out here anyway: write_empty_chunks defaults to False -> skip_empty=True).

The cause: the sync _encode_partial_sync rebuilt the in-memory shard_dict with a
per-coordinate __getitem__ loop over all 32768 chunks (O(n_chunks) Python
overhead + try/except per chunk), whereas main's async _encode_partial_single
builds the same dict with a single vectorized index lookup via
_ShardReader.to_dict_vectorized. Switched the sync path to to_dict_vectorized
(a plain, non-async method; _shard_reader_from_bytes_sync already returns a
_ShardReader), matching the async path.

The dict's key order is immaterial (the physical layout is decided downstream by
the subchunk_write_order loop in _encode_shard_dict_sync), so the merge loop —
which looks up by coordinate, not order — is unaffected.

Local micro-benchmark (32^3 shard, single 1x1x1 chunk write): 59.4 -> 40.0
ms/write (~1.5x), matching the CodSpeed delta. Correctness: full sharding +
pipeline-parity suites pass (581), so Fused still matches Batched byte-for-byte.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix: thread spec forward in FusedCodecPipeline.evolve_from_array_spec

The deps=optional CI job (where cast_value_rs is installed) failed
test_codec_pipeline_threads_dtype_through_evolve and several test_cast_value
tests with "Invalid endianness: None" / "endian needs to be specified for
multi-byte data types".

Root cause: FusedCodecPipeline.evolve_from_array_spec evolved EVERY codec against
the same original array_spec:

    evolved = tuple(c.evolve_from_array_spec(array_spec=array_spec) for c in self.codecs)

When an array->array codec widens the dtype (e.g. cast_value int8 -> int16), the
BytesCodec serializer was still evolved against the single-byte SOURCE dtype, so
it stripped its `endian` to None (bytes.py treats single-byte dtypes as having no
endianness) and then failed at decode time on the multi-byte data.
BatchedCodecPipeline.evolve_from_array_spec already threads the spec forward
(spec = evolved_codec.resolve_metadata(spec)); the Fused version did not. Fixed by
mirroring the Batched threading.

Also added test_evolve_threads_spec_preserving_serializer_endian: a
dependency-free regression test (uses a minimal dtype-widening AA codec stub, no
cast_value_rs) that runs on BOTH pipelines via the pipeline_class fixture. It
fails on [sync] without this fix and passes with it — closing the gap where the
only coverage required an optional dep and thus ran in no default env.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* refactor: extract shared pipeline logic into freestanding functions

The endian bug fixed in the previous commit existed because the two codec
pipelines duplicated the same conceptual logic and one copy drifted: Batched's
evolve_from_array_spec threaded the spec forward correctly, Fused's did not.
Duplicated logic that can silently diverge is a standing bug source, so extract
the drift-prone pieces into single sources of truth that both pipelines delegate
to (mirroring the existing resolve_aa_specs precedent):

- evolve_codecs(codecs, array_spec): the construction-time spec-threading loop.
  Both BatchedCodecPipeline and FusedCodecPipeline.evolve_from_array_spec now
  call it. There is now exactly one place this logic lives, so it cannot drift.
- pipeline_supports_partial_decode / _encode(ab, *, aa, bb, require_no_aa_bb):
  the partial-decode/encode predicate. Both pipelines delegate. The two pass
  DIFFERENT require_no_aa_bb values (Batched True, Fused False) — that divergence
  is pre-existing and deliberately preserved here (not silently unified); it is
  now explicit at the call sites and documented in one function instead of being
  buried in two slightly-different inline isinstance checks.

Behavior-preserving: each pipeline computes exactly what it did before. Left the
trivial fan-out loops (validate, compute_encoded_size) as-is — deduping those
would require changing the CodecPipeline ABC contract (abstract -> concrete) for
near-zero drift benefit. Full pipeline/sharding/parity suites + the dtype-evolve
regression test pass; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: cover the max_workers>1 thread-pool path and concurrent decode

The pool dispatch in read_sync/write_sync (codec_pipeline.max_workers > 1) had
zero functional test coverage — only config-default assertions existed — even
though threading is the opt-in we point users at. Adds:

- an end-to-end multi-chunk read/write roundtrip with max_workers=4 (verified
  the pool dispatch actually fires, not the sequential branch);
- worker-exception propagation tests for both write_sync (list-consumed
  pool.map) and read_sync (tuple-consumed pool.map): a store error raised in a
  pool worker must surface to the caller;
- a concurrent-decode test: transpose filter (so ChunkTransform._resolve_specs
  cache traffic actually occurs — with no AA codecs the cache is bypassed),
  pool workers decoding concurrently, plus an outer thread pool issuing
  overlapping reads. Pins correctness under concurrency around the shared
  transform's mutable cache.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: add zarr v2 scenarios to the shared codec-pipeline suite

zarr_format=2 appeared in none of the pipeline test files: v2 arrays were only
exercised implicitly through whichever pipeline is the global default. v2 goes
through the V2Codec wrapper (numcodecs filters + compressor) — a different
codec path than the v3 AA/AB/BB chain, with its own _encode_sync/_decode_sync
under FusedCodecPipeline — so it deserves explicit coverage on BOTH pipelines
and BOTH store kinds (sync fast path + async fallback).

Adds v2-roundtrip (uncompressed) and v2-gzip-roundtrip (numcodecs.GZip — the v2
compressor spelling; v3 codec configs are rejected for v2 arrays) to SCENARIOS.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* perf: encode the shard index once, via a shared layout helper

Both _encode_shard_dict_sync and the async _encode_shard_dict encoded the shard
index TWICE when index_location=start: encode to learn the length, shift the
present chunks' offsets by it, then re-encode with corrected offsets. The index
size is knowable without encoding — _shard_index_size() is already the byte-
exact contract every index read path relies on (reads slice exactly that many
bytes) — so the layout can use absolute offsets from the start and the index is
encoded once. Saves a full index encode (including its crc32c over the offsets
array) per shard write with index_location=start.

The layout loop was also duplicated between the sync and async versions — the
same drift surface that produced the evolve_from_array_spec endian bug. Both now
delegate to a shared pure _build_shard_layout (offset math lives once) and
_assemble_shard. A runtime guard verifies the encoded index length matches
_shard_index_size rather than silently corrupting offsets if someone ever
configures variable-size index codecs.

Verified: 606 tests pass including the pipeline-parity byte-identical shard
assertions across index_location=start/end and every subchunk_write_order, and
the sharded reopen tests — the on-disk layout is unchanged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test: pin read_missing_chunks=False semantics on sharded arrays

read_missing_chunks exists to help consumers distinguish a transport error from
a truly missing chunk. That distinction is a STORE-KEY-level concept: a missing
shard key raises ChunkNotFoundError. It does not cleanly apply to inner
subchunks of a shard that was fetched successfully — there is no transport
ambiguity there; the shard index simply records the subchunk as absent — so
those fill with the fill value rather than raising.

Both pipelines already implement exactly this (verified empirically), but
nothing pinned it, so the asymmetry vs unsharded arrays read as a bug in review.
This adds a shared-suite test (both pipelines x sync/async stores) asserting
both sides: missing shard key raises; missing inner subchunk of an existing
shard fills.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* Ig/fused additions (#180)

* chore: add parallelism TODOs

* perf: non-sharded reads

* chore: code duplication

* fix: arguments dont spread themselves!

* fix: bring in suggested guard

* perf: don't block on pool ops

* chore: docstring + materialize early

---------

Co-authored-by: ilan-gold <[email protected]>

* perf: sync IO for sharding byte getters; evolve the nested inner pipeline

Addresses the open question on this PR about sync/async byte getters,
benchmark-guided as discussed.

_ShardingByteGetter/_ShardingByteSetter are in-memory dict wrappers but
presented only an async API, so the nested codec_pipeline.read over inner
chunks fell to the async fallback: one concurrent_map coroutine per inner chunk
for a dict lookup (~2.1 us/chunk pure asyncio overhead, ~8.8 ms per 4096-chunk
shard). On top of that, the nested pipeline (ShardingCodec.codec_pipeline) is
built by from_codecs and never evolved, so its sync transform was always None —
inner chunks also paid per-chunk AsyncChunkTransform coroutines, and the
decode_sync/encode_sync fallback improvements in this PR could not reach them.

Changes:
- SyncByteGetter / SyncByteSetter runtime protocols in zarr.abc.store
  (resurrecting the design from the original perf/prepared-write experiments,
  same names and shape). StorePath matches structurally but is still gated on
  its STORE's sync support; the protocols gate non-StorePath byte getters.
- _ShardingByteGetter/Setter implement get_sync/set_sync/delete_sync; the async
  methods delegate to the sync ones (single implementation).
- ShardingCodec._get_inner_pipeline(shard_spec): the nested pipeline evolved
  against the inner chunk spec (threads specs through the inner chain AND
  builds the sync transform). The four nested read/write call sites use it.
- FusedCodecPipeline.read/write gates accept non-StorePath SyncByteGetter/
  SyncByteSetter, so nested inner-chunk IO takes read_sync/write_sync.
- _decode_shard_index/_encode_shard_index delegate to their sync twins (pure
  compute; kills a per-shard AsyncChunkTransform round-trip and a sync/async
  duplication).

Benchmark (4096 inner chunks per shard, LatencyStore@0 i.e. the async-fallback
path = sharded data on remote stores), vs this PR's head:

  uncompressed read: 44.2 -> 28.9 ms (1.53x)
  gzip         read: 182.0 -> 50.9 ms (3.6x)
  writes: unchanged (~34 / ~70 ms) — already optimized by this PR's
  encode_sync fallback (whole-shard sync encode, no byte setters involved).

Adds a regression test asserting sharded fallback reads route inner chunks
through the sync fast path (read_sync engaged, zero AsyncChunkTransform calls);
verified it fails if the gate is removed. Full sharding + parity + pipeline
suites pass (619).

Assisted-by: ClaudeCode:claude-fable-5

* fix: root-cause the cross-file test flake (pytest-asyncio loop leak); cache inner pipeline

The pipeline test suites have failed intermittently all along on an arbitrary
test that passes in isolation. Root cause (allocation site verified with
PYTHONTRACEMALLOC): pytest-asyncio implicitly creates an event loop in
_get_event_loop_no_warn during fixture setup/teardown and never closes it. When
GC reclaims that loop — or its self-pipe socketpair — mid-test, pytest's
unraisable hook converts the ResourceWarning into a failure of whichever
unrelated test happens to be running. The sync-bytegetter change increased
per-shard-op allocation churn enough to make this near-deterministic, which is
how it was finally traced.

Two changes:
- pyproject filterwarnings: narrowly ignore the two unraisable shapes
  (BaseEventLoop.__del__, AF_UNIX socketpair), mirroring the existing
  s3fs/aiobotocore entry. Not zarr's loops.
- ShardingCodec._get_inner_pipeline is now memoized per (pipeline class,
  shard_spec) — evolving built a ChunkTransform on every shard operation. The
  pipeline class is part of the key so codec_pipeline.path config changes are
  still honored.

Battery that previously failed ~every run now passes 3x consecutively (635).

Assisted-by: ClaudeCode:claude-fable-5

* refactor: dedupe sharding sync/async mirrors; fix un-threaded inner-chain evolve

Continues the anti-skew work: where the sync and async sharding paths
implemented the same logic twice, extract a single source of truth so the
copies cannot drift (the mechanism behind the pipeline-level endian bug).

- _get_inner_chunk_transform / _get_index_chunk_transform now evolve their
  codec chains via evolve_codecs (spec THREADED forward). Both previously
  evolved every codec against the same unthreaded spec — the exact bug shape
  that stripped BytesCodec.endian at the pipeline level, latent here for any
  spec-changing inner codec. Both are also now actually memoized (the inner
  transform's docstring claimed a cache that did not exist; transforms were
  rebuilt per call).
- New regression test (dependency-free dtype-widening stub codec) asserting
  the inner serializer keeps its endian; verified it fails on the unthreaded
  version.
- _shard_index_byte_range(): the index-location byte-range arithmetic existed
  verbatim in both _load_shard_index_maybe and its _sync twin; now one helper.
- _pair_chunks_with_byte_ranges(): the chunk-coord/byte-range pairing loop
  existed verbatim in both _load_partial_shard_maybe and its _sync twin; now
  one helper.

Deliberately NOT unified: the small hand-rolled loops remaining in
_decode_sync/_encode_sync vs their async twins. Post sync-bytegetter work the
async versions are thin delegations to the (shared, evolved) nested pipeline,
so the heavy machinery — evolve, transforms, layout, index codecs — is already
single-sourced; force-merging the residual loops would couple different
missing-chunk/concurrency semantics for little drift-surface gain. The
pipeline-parity suite guards their behavioral equivalence.

Full battery passes twice (636); ruff + mypy clean.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: canonical chunk write-state functions; fix complete-chunk merge copy

Prototype of the "chunk state algebra" direction: the write-side state logic
(maybe-read existing -> merge -> empty-normalize -> encode-or-elide) existed in
four places with divergent inline conventions. It is now three canonical
functions in codec_pipeline.py:

- chunk_is_empty(): THE write_empty_chunks normalization rule (all-fill chunk
  normalizes to missing), previously five scattered inline all_equal checks.
- encode_or_elide_chunk(): normalize-empty + encode; None = must not be stored.
- merge_and_encode_chunk(): the full single-chunk write transition. Used by the
  fused _write_one and both branches of the sharding _encode_partial_sync loop
  (including the scalar-broadcast memoization, which now memoizes the canonical
  function's result). _encode_sync uses encode_or_elide_chunk.

Unification found a real perf bug: _merge_chunk_array's complete-chunk early
return required value.shape == chunk_spec.shape, which never holds for
multi-chunk writes — so every complete chunk of every multi-chunk fused write
paid a create+fill+copy. (Sharding's hand-rolled loop bypassed this with a
view, which is itself how the two copies had drifted.) The guard now returns
value[out_selection] whenever it is exactly chunk-shaped. Measured, two A/B
passes: unsharded full write (1000 chunks) ~1.5x faster; bulk partial shard
write (900 complete inner chunks) ~1.4x faster; sharded single-chunk write
unchanged. Callers pass existing=None for complete chunks so fully-overwritten
data is never decoded.

Also removed the last divergent missing-chunk conventions: _decode_sync's
try/except KeyError is now .get() -> None (None is the single "missing"
convention), and the dead skip_empty/fill_value prologues are gone.

1945 tests pass including pipeline-parity byte-identical assertions; ruff +
mypy clean.

Assisted-by: ClaudeCode:claude-fable-5

* refactor: canonical chunk read functions (decode_and_scatter / scatter_chunk)

The read twin of merge_and_encode_chunk: the fill-on-missing scatter logic
existed in five places (fused _read_one, partial-decode _read_one, the async
fallback scatter loop, and the two sharding decode tails) with two different
missing conventions (None vs try/except KeyError) and three fill spellings
(precomputed batch fill, raw shard_spec.fill_value, inline or-default).

Now two canonical functions in codec_pipeline.py:
- scatter_chunk(): scatter an already-selected region; None = missing ->
  scatter fill, return a "missing" GetResult. POLICY-FREE on purpose: whether
  missing is an error (read_missing_chunks=False) is decided at the array
  layer from the top-level statuses — which is exactly what makes missing
  INNER chunks of a present shard fill rather than raise (the sharding codec
  discards the nested statuses). That semantic, previously implicit in which
  loop happened to run, is now written down where the rule lives.
- decode_and_scatter_chunk(): decode (None = missing) -> select -> scatter.

All five sites converted; the last try/except KeyError missing-convention is
gone (Mapping.get works for both _ShardReader and plain dicts). Read
benchmarks A/B neutral-to-marginally-better on all paths (unsharded full,
sharded full/partial, half-missing fill).

786 tests pass including pipeline-parity and indexing; ruff + mypy clean.

Assisted-by: ClaudeCode:claude-fable-5

* test: property tests asserting fast paths equal general paths

Add tests/test_fastpath_equivalence.py with four hypothesis properties,
one per fast path on the branch:

- _merge_chunk_array complete-chunk view == general merge path (and
  independent of existing chunk content)
- ShardingCodec._decode_full_shard_bulk == _decode_sync for dense
  uncompressed shards across dtypes, endianness, subchunk write order,
  and index location (asserts the bulk path actually applies, so the
  test cannot pass vacuously)
- scalar writes leave the store byte-identical to equivalent broadcast
  array writes (pins the sharded scalar-broadcast memoization)
- Store.get_ranges_sync coalesced reads == one get_sync per range, for
  arbitrary gap/coalesce limits and Range/Offset/Suffix/None requests

Each was verified to catch its bug class by temporary fault injection:
dropping the bulk decode's endian handling and shifting the coalesce
re-slice offset by one both produced shrunk falsifying examples.

Assisted-by: ClaudeCode:claude-fable-5

* fix: address open review findings across sharding, pipeline, and tests

Code fixes:
- ShardingCodec.evolve_from_array_spec now threads the spec through the
  inner chain via evolve_codecs. The unthreaded evolve survived on the
  real array-creation path after the transform builders were fixed,
  baking an endian-stripped BytesCodec into the evolved instance behind
  any dtype-changing inner codec. Regression test goes through
  evolve_from_array_spec and then builds the inner transform.
- Async _decode_shard_index/_encode_shard_index fall back to the async
  pipeline when an index codec is not sync-capable, instead of failing
  every path for third-party async-only index codecs.
- Removed the redundant hand-rolled dict caches inside the chunk
  transform builders; the instance-local lru_cache wrappers are the
  single memoization mechanism. _shard_index_size is now lru_cached too.
- The encoded-index-size guard lives once, in _assemble_shard, instead
  of duplicated in the sync and async encoders.
- _get_inner_pipeline cache key includes codec_pipeline.batch_size,
  which from_codecs captures at construction.
- Coordinate arrays built via np.indices instead of
  np.array(list(np.ndindex(...))) in the partial-write loaders.
- _merge_chunk_array docstring states the view-aliasing contract; the
  guard comment sits on the check it annotates.
- The socketpair unraisable filter covers family=(1|2) (Windows
  emulates socketpair with AF_INET), and its comment owns the tradeoff
  that the patterns cannot scope to pytest-asyncio.

Test hardening:
- Pool tests assert the pool branch actually fires (_resolve_max_workers
  + a _get_pool spy) instead of silently degrading to the sequential
  branch on a config regression; the concurrent-read test re-opens the
  array each round so readers race a cold spec cache.
- New direct test pins the SyncByteSetter write gate in
  FusedCodecPipeline.write (verified to fail with the gate removed).
- New test pins the merge fast path's view-aliasing + source-unmutated
  contract end-to-end on both pipelines.
- read_missing_chunks=False sharded test asserts both halves of the
  asymmetry against the same partially-written array.
- v2 scenario with a numcodecs Delta filter covers the V2Codec filter
  branch; _chunk_keys recognizes v2 metadata keys.

Assisted-by: ClaudeCode:claude-fable-5

* doc: add changelog entries for PR #3885

Rename the fused-default placeholder to the PR-numbered 3885.feature.md and
add a second feature entry for the new SyncByteGetter/SyncByteSetter protocols
and Store.get_ranges_sync.

Assisted-by: ClaudeCode:claude-opus-4.8

* test: cover async fallback paths orphaned by the Fused default

Making FusedCodecPipeline the default left the async (Batched) mirror
paths and the new sync-IO error paths exercised only narrowly, dropping
project coverage. Add targeted tests for the reachable gaps:

- AsyncChunkTransform.decode_chunk/encode_chunk == ChunkTransform across
  aa/ab/bb codec combinations (the async per-chunk chain the default
  sync path never runs), plus FusedCodecPipeline.encode/decode
  None-chunk passthrough.
- ShardingCodec._decode_single/_encode_single whole-shard round-trip and
  all-empty branches. The codec advertises partial decode/encode, so the
  pipeline always picks the partial methods; these whole-shard async
  methods are reached only via the direct ArrayBytesCodec API.
- Async-only index codec fallback in _decode_shard_index/
  _encode_shard_index (#269), via a test-only async-only ArrayBytesCodec
  stub that is not SupportsSyncCodec.
- Store.get_ranges_sync happy path, missing-key BaseExceptionGroup, and
  the non-sync-store TypeError.

Local merged coverage on the touched files: codec_pipeline.py
85.4->92.2%, sharding.py 92.4->96.7%, abc/store.py 93.8->95.3%.

Assisted-by: ClaudeCode:claude-opus-4.8

* refactor: move reused functions around chunks into standalone file

* refactor: use more helpful name and remove comment

* refactor: move more

* refactor: move more

* Ig/as completed read (#194)

* perf: use `as_completed`

* refactor: unfiy branches

* refactor: `concurrent_iter` only handles running

* add note

* add note about blocking

* chore: latency distribution in store

* refactor: as_completed reading

* perf: as_completed writes

* chore: remove dead code

* fix: use concurrent_map

* fix: remove unnecessary list

* fix: revert use of concurrent_map

---------

Co-authored-by: ilan-gold <[email protected]>

* fix: bulk shard decode must not serve reordering reads in natural order

`ShardingCodec._decode_full_shard_bulk_if_uncompressed` gated the
vectorized whole-shard fast path on `indexer.shape == shard_spec.shape`.
For a `CoordinateIndexer` (vindex / integer-array oindex), `.shape` is the
flattened point count, which can equal the shard shape by coincidence
(trivially in 1-D). Such a selection then passed the gate and the bulk
path returned the shard in natural order, silently dropping the
reordering — data corruption on uncompressed, crc-free shards.

Gate instead on `sel_shape`: a gather indexer exposes it, a contiguous
full read (BasicIndexer, or a non-gathering OrthogonalIndexer from
`arr[:]`) does not. `isinstance(indexer, BasicIndexer)` would be too
strict — `arr[:]` produces an OrthogonalIndexer and must keep the fast
path.

Regression guard: test_reordering_read_on_uncompressed_shard_honors_selection
exercises the end-to-end read path (where the gate lives), which the
BasicIndexer-only bulk-decode parity test could not reach. The `arrays()`
hypothesis strategy now also samples the uncompressed single-BytesCodec
sharding config (via `sharding_inner_codecs`) so the fast path is covered
under randomized indexing going forward.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: make ChunkTransform spec cache torn-read-safe under threading

A `ChunkTransform` is shared across thread-pool workers (read_sync /
write_sync with max_workers > 1). Its `_resolve_specs` cache stored the
key and the resolved specs in three separate fields, written
non-atomically: a worker could observe a freshly-set key paired with the
previous (or None) specs, returning the wrong codec chain for a chunk_spec
— silent corruption with mixed specs, or a tripped assert.

Collapse the cache into a single `(key, aa_specs, ab_spec)` tuple field
replaced with one atomic attribute write. Under the GIL a reader now sees
either the complete old entry or the complete new one, never a torn mix;
worst case is a recompute, never a wrong result.

Regression guard: test_shared_transform_decode_alternating_specs pins the
single-slot eviction/refill correctness that underpins the atomicity (it
fails if specs go stale on eviction).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: default Codec.is_fixed_size so sharded vlen/numcodecs reads don't crash

`ShardingCodec._inner_codecs_fixed_size` reads `c.is_fixed_size` on every
inner codec to gate the bulk-decode fast path. `is_fixed_size` was
declared on the Codec ABC as a bare annotation with no default, so codecs
that never set it — VLenUTF8Codec, VLenBytesCodec, the numcodecs wrappers
— raised AttributeError, crashing every sharded read whose inner chain
included such a codec under the default (Fused) pipeline.

Give the ABC a conservative default `is_fixed_size = False`. It stays a
class attribute (not a dataclass field; fixed-size codecs still override
with True), and treating an unknown codec as not-fixed-size only disables
the size-dependent fast path, never correctness.

Regression guard: test_sharding_vlen_inner_codec_roundtrip (Fused +
Batched). Uses StringDType to force the VLenUTF8 inner chain — a
fixed-width <U dtype would use BytesCodec and not reproduce.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* refactor: drop unreachable None branch in _async_write_fallback

`_merge_chunk_array` always returns a real NDBuffer, so the merged-chunk
list never contains None and the `if chunk_array is None` branch (with its
`# type: ignore[unreachable]`) was dead. Replace the loop with a
compreh…