[pull] main from zarr-developers:main #3
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* test(ci): add test environment for upstream dependencies * try PIP_INDEX_URL again * test(ci): add test environment for oldest supported dependency versions * use a matrix again
* fix(remotestore): raise error if path includes scheme * fixup * fixup * strip scheme in from_url in case fsspec fails to * style: pre-commit fixes * disable cache in listing ops * update docs * use listings cache again * no refresh * style: pre-commit fixes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.6.9 → v0.7.0](astral-sh/ruff-pre-commit@v0.6.9...v0.7.0) - [github.com/pre-commit/mirrors-mypy: v1.11.2 → v1.12.1](pre-commit/mirrors-mypy@v1.11.2...v1.12.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feature(array): implement Array.append changes the Array.resize to be an inplace operation * better error message * no more warn * style: pre-commit fixes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix JSON encoding of complex fill values We were not replacing NaNs and Infs with the string versions. * Fix decoding of complex fill values * try excluding `math.inf` * Check complex numbers explicitly * Update src/zarr/core/metadata/v3.py
* support zero-sized chunks * fix imports * add min_side=0 to testing strategies * fix property tests --------- Co-authored-by: Deepak Cherian <[email protected]>
* refactors codec pipeline for v2 * async * rm ensure_bytes
* feature(group): add group setitem api * arrays proxy * rollback to simple version * rollback deprecation * rollback ... * Update tests/test_group.py
* test(ci): test on windows and osx * add os to name * bail on making np upper bound super flexible * remove lmdb * remove types-redis * fixup * tweak timeout * to_posix in LocalStore * no mo windows * Update src/zarr/storage/local.py * Update .github/workflows/test.yml * style: pre-commit fixes --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
) Bumps the requirements group with 1 update: [sphinx-autoapi](https://github.com/readthedocs/sphinx-autoapi). Updates `sphinx-autoapi` from 3.3.2 to 3.3.3 - [Release notes](https://github.com/readthedocs/sphinx-autoapi/releases) - [Changelog](https://github.com/readthedocs/sphinx-autoapi/blob/main/CHANGELOG.rst) - [Commits](readthedocs/sphinx-autoapi@v3.3.2...v3.3.3) --- updated-dependencies: - dependency-name: sphinx-autoapi dependency-type: direct:production update-type: version-update:semver-patch dependency-group: requirements ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.0 → v0.7.1](astral-sh/ruff-pre-commit@v0.7.0...v0.7.1) - [github.com/pre-commit/mirrors-mypy: v1.12.1 → v1.13.0](pre-commit/mirrors-mypy@v1.12.1...v1.13.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Bumps the actions group with 1 update: [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish). Updates `pypa/gh-action-pypi-publish` from 1.10.3 to 1.11.0 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@v1.10.3...v1.11.0) --- updated-dependencies: - dependency-name: pypa/gh-action-pypi-publish dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Make AsyncArray.nchunks_initialized async
This changes the API of AysncArray.nchunks_initialized to
change it from a property to an async function. The motivation
here comes from
1. general cleanliness (a property access calling async functions doing
I/O feels a bit wrong)
2. Work on Array.info, where I hit a strange error, I think from jumping
from a
- sync Array.info_complete ->
- async AsyncArray.info_complete ->
- sync AsyncArray.nchunks_initialzed ->
- sync collect_aiterator (async list_prefix)
With this change, we'll be able to jump from sync to async just once
at the boundary.
```
File "/Users/tom/gh/zarr-developers/zarr-python/src/zarr/core/array.py", line 3011, in info_complete
return sync(self._async_array.info_complete())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tom/gh/zarr-developers/zarr-python/src/zarr/core/sync.py", line 141, in sync
raise return_result
File "/Users/tom/gh/zarr-developers/zarr-python/src/zarr/core/sync.py", line 100, in _runner
return await coro
^^^^^^^^^^
File "/Users/tom/gh/zarr-developers/zarr-python/src/zarr/core/array.py", line 1223, in info_complete
"count_chunks_initialized": self.nchunks_initialized, # this should be async?
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tom/gh/zarr-developers/zarr-python/src/zarr/core/array.py", line 844, in nchunks_initialized
return nchunks_initialized(self)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tom/gh/zarr-developers/zarr-python/src/zarr/core/array.py", line 3035, in nchunks_initialized
return len(chunks_initialized(array))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tom/gh/zarr-developers/zarr-python/src/zarr/core/array.py", line 3061, in chunks_initialized
collect_aiterator(array.store_path.store.list_prefix(prefix=array.store_path.path))
File "/Users/tom/gh/zarr-developers/zarr-python/src/zarr/core/sync.py", line 178, in collect_aiterator
return sync(_collect_aiterator(data))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tom/gh/zarr-developers/zarr-python/src/zarr/core/sync.py", line 128, in sync
raise SyncError("Calling sync() from within a running loop")
zarr.core.sync.SyncError: Calling sync() from within a running loop
```
* fixup
---------
Co-authored-by: Davis Bennett <[email protected]>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.1 → v0.7.2](astral-sh/ruff-pre-commit@v0.7.1...v0.7.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…erarchy nodes (#2430) * fix(array): thread order parameter through to array __init__ * type fixes * move defaults * apply MemoryOrder type to ArrayV2Metadata * more more * more more more * feature(store,group,array): stores learn to delete prefixes when overwriting nodes - add Store.delete_dir and Store.delete_prefix - update array and group creation methods to call delete_dir - change list_prefix to return absolue keys * fixup * fixup * respond to review * fixup * fixup * Update src/zarr/abc/store.py * style: pre-commit fixes --------- Co-authored-by: Davis Bennett <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* mv BatchedCodecPipeline to zarr.core * fix for tests * import codec_pipeline and buffer in zarr.core.__init__
* Apply ruff/Pylint rule PLR1711 PLR1711 Useless `return` statement at end of function * Apply ruff/Pylint rule PLR2044 PLR2044 Line with empty comment * Apply ruff/Pylint rule PLR6104 PLR6104 Use `+=`, `*=`, `/=` to perform an augmented assignment directly --------- Co-authored-by: Joe Hamman <[email protected]>
* add argument check * test * test zarr.save with multiple arrays * fix path for zarr.save args * improve test readability * fix typing * support other array types Co-authored-by: Tom Augspurger <[email protected]> * support NDArrayLike * fix typing * fix typing * fix typing * format --------- Co-authored-by: Tom Augspurger <[email protected]>
* Enforce ruff/flake8-executable rules (EXE) * Enforce ruff/flake8-future-annotations rules (FA) * Enforce ruff/flake8-logging rules (LOG) * Apply ruff/flake8-pie rule PIE790 PIE790 Unnecessary `pass` statement PIE790 Unnecessary `...` literal * Apply ruff/flake8-pie rule PIE800 PIE800 Unnecessary spread `**` * Apply ruff/flake8-pie rule PIE808 PIE808 Unnecessary `start` argument in `range` * Enforce ruff/flake8-pie rules (PIE) * Apply ruff preview rule RUF022 RUF022 `__all__` is not sorted * Sort pyproject.toml * Apply ruff/flake8-simplify rule SIM102 SIM102 Use a single `if` statement instead of nested `if` statements * Apply ruff/flake8-simplify rule SIM114 SIM114 Combine `if` branches using logical `or` operator * Apply ruff/flake8-simplify rule SIM118 SIM118 Use `key in dict` instead of `key in dict.keys()` * Enforce ruff/flake8-simplify rules (SIM) * Enforce ruff/flake8-slots rules (SLOT)
The `None` was added in 0e035fb, let's try to remove it.
* use as_posix for str and for listdir * release the windows workflows * unbreak gha
The keyword argument `shape` is mandatory.
* Add docstrings for array module * style: pre-commit fixes * Add more docstrings * Remove docstrings for internal functions * Fix pre-commit * Minor fix * Precommit fix * Update src/zarr/core/array.py Co-authored-by: Joe Hamman <[email protected]> * Update src/zarr/core/array.py Co-authored-by: Joe Hamman <[email protected]> * Update src/zarr/core/array.py Co-authored-by: Joe Hamman <[email protected]> * Update src/zarr/core/array.py Co-authored-by: Joe Hamman <[email protected]> --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <[email protected]> Co-authored-by: Joe Hamman <[email protected]>
… group (#2479) Bumps the actions group with 1 update: [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish). Updates `pypa/gh-action-pypi-publish` from 1.11.0 to 1.12.2 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@v1.11.0...v1.12.2) --- updated-dependencies: - dependency-name: pypa/gh-action-pypi-publish dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Stateful tests for array/group manipulation Port over some stateful tests from [Arraylake](https://docs.earthmover.io/). Co-authored-by: Sebastián Galkin <[email protected]> * add TODOs --------- Co-authored-by: Sebastián Galkin <[email protected]>
* Added group info * Basic array * fixup * docs, tests * fixup * fixup * docstringsx * fixup * chunk info * fixup * wip - split to chunks_initialized * fixup * fixup * update docs * fixup test * lint * fixup test --------- Co-authored-by: Joe Hamman <[email protected]>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.2 → v0.7.3](astral-sh/ruff-pre-commit@v0.7.2...v0.7.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix typing errors in testing.stateful * Add a dimensionnames type * Add bugfix entry * Fix properties test
* (fix): use `typesize` on `Blosc` codec * (chore): relnote * (fix): intersphinx * (fix): look at that compression ratio! * (fix): add test * (fix): min version * (fix): parenthesis? * (fix): try assertion error * (fix): windows size * (fix): add bytes print * (fix): aghh windows latest is correct, error for non latest * (fix): conditions for sizes * (fix): try clearer data * (fix): awesome! * (fix): pre-commit --------- Co-authored-by: David Stansby <[email protected]>
* unbreak chunks initialized * Update src/zarr/storage/_utils.py Co-authored-by: Tom Augspurger <[email protected]> * update docstring * make relativize_paths kw-only, and add tests --------- Co-authored-by: Tom Augspurger <[email protected]>
* Ignore explicit test store files * Fix test_zip * Fix test_local * Fix test_fsspec * Fix typing in test_memory * Remove walrus Co-authored-by: Davis Bennett <[email protected]> --------- Co-authored-by: Davis Bennett <[email protected]>
* Allow no compressor for v2 arrays * Use typing aliases for compressors * Test v2 array w/ v3 codec errors * Add changelog entry * Update type comment * fix test names Co-authored-by: Davis Bennett <[email protected]> --------- Co-authored-by: Davis Bennett <[email protected]>
* Avoid memory copy in obstore write * Add as_bytes_like method to Buffer * Add changelog entry * No need to take unsigned bytes view following #2738 * Change method name to `as_buffer_like` --------- Co-authored-by: jakirkham <[email protected]> Co-authored-by: Davis Bennett <[email protected]>
* Replace redundant list comprehension with generator * Partially revert There is no such thing as a "tuple comprehension": https://stackoverflow.com/questions/52285419/aggregating-an-async-generator-to-a-tuple#52285420 Fixes CI error: FAILED tests/test_group.py::test_create_hierarchy_existing_nodes[zarr2-async-array-memory] - TypeError: 'async_generator' object is not iterable
* remove duplicate metadata parsing * add test cases * add test cases * tests for different zarr_formats in test_storage_transformers * tests for different zarr_formats in test_storage_transformers * ignore mypy arg-type error for deprecation test * fix typing in tests * test_chunk_key_encoding * test_invalid_v2_arguments * test_array_repr * type annotation for parse_array_metadata * test_v2_and_v3_exist_at_same_path * remove duplicate check for dimension_separator in v3 * tests for invalid arguments in creation * format * revert typing * document changes
* Update pre-commit hooks * Update type checking code
* Don't compress data in hypothesis testing * Add comment
* feat: add function * doc: add docstring * doc: add change log for print_debug_info * Update .github/ISSUE_TEMPLATE/bug_report.yml * feat: expand debug printout + better test * feat: debug - also print out rich * test: print debug test for upstream * feat: better formatting for print_debug_info * restore original issue template --------- Co-authored-by: Davis Bennett <[email protected]> Co-authored-by: David Stansby <[email protected]>
* Fix overwrite modes * Add many more tests that data doesn't disappear * Add bugfix entry. * Fix function name
* Changelog for 3.0.8 * Add warning to top of release notes * fix warning
* (chore): release notes cleanup * (chore): add double space
* move for LocalStore * move for ZipStore * remove redundant check * open and close zipstore * fix zipstore.move * fix localstore.move for ndim>1 * format * remove abstract Store .move * document changes * shutil.move entire folder * fix test for windows
* Error on invalid store mode * Fix tests * Add release note * Fix test
* (feat): use `np.zeros` for buffer creation when possible * (fix): add comment and check * (chore): relnote
* add fill_value output to info * changelog * fix tests * fix example in docs
* issues: add pep-723 to issue template * use zarr main branch + make copyable * corret example link + not uv specific
* The ruff ruleset is TC, not TCH * Apply ruff/flake8-type-checking rule TC006 TC006 Add quotes to type expression in `typing.cast()` * Apply ruff/flake8-type-checking rule TC003 TC003 Move standard library import into a type-checking block --------- Co-authored-by: David Stansby <[email protected]>
* remove insertion of vlen-string codec for v2 metadata creation * changelog * Update changes/3100.bugfix.rst Co-authored-by: David Stansby <[email protected]> * Update changes/3100.bugfix.rst Co-authored-by: David Stansby <[email protected]> * add dtype strings to test_v2_chunk_encoding --------- Co-authored-by: David Stansby <[email protected]>
* Add a GroupNotFoundError * Fixup tests * Add changelog entries * Remove inheritance change
* Delegate logic for chunks to AsyncArray._create * Fix changelog number * Test public API * Add todo
Co-authored-by: David Stansby <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )