diff --git a/.gitattributes b/.gitattributes index 57eb8a8807..f507ce96ca 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,3 +1,9 @@ *.py linguist-language=python *.ipynb linguist-documentation .git_archival.txt export-subst + +# just refuses to format a CRLF file, so `just just-check` would fail on a Windows +# checkout with core.autocrlf=true before the contributor has changed anything. +Justfile text eol=lf +*/justfile text eol=lf +packages/*/justfile text eol=lf diff --git a/.github/labeler.yml b/.github/labeler.yml index 7eb74211ea..482207d6cb 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -2,3 +2,21 @@ needs release notes: - all: - changed-files: - all-globs-to-all-files: '!changes/*.md' + +# Subpackage labels: a pull request whose changes all live under one package +# directory gets that package's label, which .github/release.yml uses to keep +# subpackage work out of the generated `zarr` release notes. +zarr-metadata: + - all: + - changed-files: + - all-globs-to-all-files: 'packages/zarr-metadata/**' + +zarr-indexing: + - all: + - changed-files: + - all-globs-to-all-files: 'packages/zarr-indexing/**' + +zarr-http-server: + - all: + - changed-files: + - all-globs-to-all-files: 'packages/zarr-http-server/**' diff --git a/.github/release.yml b/.github/release.yml new file mode 100644 index 0000000000..a66182c50a --- /dev/null +++ b/.github/release.yml @@ -0,0 +1,22 @@ +# Configuration for GitHub's "Generate release notes" button. +# +# The zarr-python repository also hosts the companion packages under packages/ +# (zarr-metadata, zarr-indexing, zarr-http-server), which have their own release +# cadence, tags and changelogs. Pull requests that only touch a subpackage are +# labelled by the labeler workflow (.github/labeler.yml) and excluded here, so a +# `zarr` release lists only the changes that ship in `zarr`. +changelog: + exclude: + labels: + - zarr-metadata + - zarr-indexing + - zarr-http-server + authors: + - pre-commit-ci[bot] + categories: + - title: Dependency updates + labels: + - dependencies + - title: What's Changed + labels: + - "*" diff --git a/.github/workflows/check_changelogs.yml b/.github/workflows/check_changelogs.yml index f642eb17ca..960451cc4c 100644 --- a/.github/workflows/check_changelogs.yml +++ b/.github/workflows/check_changelogs.yml @@ -22,16 +22,22 @@ jobs: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 + + - name: Install just + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Check zarr-python changelog entries - run: uv run --no-sync python ci/check_changelog_entries.py + run: just check-changelogs - name: Check zarr-metadata changelog entries - run: uv run --no-sync python ci/check_changelog_entries.py packages/zarr-metadata/changes + run: just check-changelogs packages/zarr-metadata/changes - name: Check zarr-indexing changelog entries - run: uv run --no-sync python ci/check_changelog_entries.py packages/zarr-indexing/changes + run: just check-changelogs packages/zarr-indexing/changes - name: Check zarr-http-server changelog entries - run: uv run --no-sync python ci/check_changelog_entries.py packages/zarr-http-server/changes + run: just check-changelogs packages/zarr-http-server/changes diff --git a/.github/workflows/codspeed.yml b/.github/workflows/codspeed.yml index 1441950c3f..a58ce40a9d 100644 --- a/.github/workflows/codspeed.yml +++ b/.github/workflows/codspeed.yml @@ -31,10 +31,19 @@ jobs: uses: pypa/hatch@f647ed70d49adb885f53a27d1c7f5bdaeacf2c60 with: version: '1.16.5' + # No interpreter is set up in this job on purpose. Benchmarks are only comparable + # across runs if the interpreter underneath them does not move: adding a pinned + # Python here measured a uniform ~13% slowdown on every benchmark, because hatch + # then built the environment on that interpreter instead of the runner's own. + - name: Install just + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Run the benchmarks uses: CodSpeedHQ/action@373d6868929f444bc08d901fd0eb0ad52a8875ea # v5.2.1 env: ZARR_BENCHMARK_CLEAR_CACHE: '1' with: mode: walltime - run: hatch run test.py3.12-minimal:pytest tests/benchmarks --codspeed + run: HATCH_ENV=test.py3.12-minimal just benchmark-codspeed diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 0f9c711a60..990866a109 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -22,19 +22,23 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 - - run: uv sync --group docs + - uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 + - name: Install just + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 # Fast source-level guards that need no built site, so they run before the (slower) # build for a quick failure: every public export is in the API reference, and no # docstring/Markdown carries reStructuredText markup that MkDocs won't render. - - run: uv run python ci/check_documented_exports.py docs/api - - run: uv run python ci/lint_docs.py + - run: just check-doc-exports + - run: just lint-docs # --strict turns warnings into errors, so a docs code block that fails to execute # at build time (e.g. a non-exec python fence disrupting a later exec="true" block) # fails CI instead of merging as a silent warning. - - run: uv run mkdocs build --strict + - run: just docs-build env: DISABLE_MKDOCS_2_WARNING: "true" NO_MKDOCS_2_WARNING: "true" - - run: uv run python ci/check_unlinked_types.py + - run: just check-doc-links continue-on-error: true diff --git a/.github/workflows/downstream.yml b/.github/workflows/downstream.yml index 30210cd452..76cfaa0aa8 100644 --- a/.github/workflows/downstream.yml +++ b/.github/workflows/downstream.yml @@ -45,7 +45,7 @@ jobs: python-version: '3.13' - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 - name: Install xarray and test dependencies working-directory: xarray @@ -59,10 +59,11 @@ jobs: working-directory: xarray run: uv pip install --no-deps .. + # Keep the branch override: syncing xarray would reinstall zarr from PyPI. - name: Show versions working-directory: xarray run: | - uv run python -c " + uv run --no-sync python -c " import zarr; print(f'zarr {zarr.__version__}') import xarray; print(f'xarray {xarray.__version__}') " @@ -70,7 +71,7 @@ jobs: - name: Run xarray zarr backend tests working-directory: xarray run: | - uv run python -m pytest --no-header -q \ + uv run --no-sync python -m pytest --no-header -q \ xarray/tests/test_backends.py \ xarray/tests/test_backends_api.py \ xarray/tests/test_backends_datatree.py @@ -102,7 +103,7 @@ jobs: python-version: '3.13' - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 - name: Install numcodecs with test-zarr-main group working-directory: numcodecs diff --git a/.github/workflows/gpu_test.yml b/.github/workflows/gpu_test.yml index c902bfdaa8..a8e989f580 100644 --- a/.github/workflows/gpu_test.yml +++ b/.github/workflows/gpu_test.yml @@ -62,22 +62,29 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 - name: Install Hatch uses: pypa/hatch@f647ed70d49adb885f53a27d1c7f5bdaeacf2c60 with: version: '1.16.5' + - name: Install just + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 + # Two names for the same environment on purpose: `just setup` builds whatever + # HATCH_ENV points at, while `just gpu` reads GPU_HATCH_ENV so that an + # ambient HATCH_ENV can never redirect `pytest -m gpu` into a CPU environment. - name: Set Up Hatch Env env: HATCH_ENV: gputest.py${{ matrix.python-version }} run: | - hatch env create "$HATCH_ENV" - hatch env run -e "$HATCH_ENV" list-env + just setup - name: Run Tests env: - HATCH_ENV: gputest.py${{ matrix.python-version }} + GPU_HATCH_ENV: gputest.py${{ matrix.python-version }} run: | - hatch env run --env "$HATCH_ENV" run-coverage + just gpu - name: Upload coverage uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0 diff --git a/.github/workflows/hypothesis.yaml b/.github/workflows/hypothesis.yaml index 352888f749..3be6f3ba60 100644 --- a/.github/workflows/hypothesis.yaml +++ b/.github/workflows/hypothesis.yaml @@ -60,17 +60,21 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 - name: Install Hatch uses: pypa/hatch@f647ed70d49adb885f53a27d1c7f5bdaeacf2c60 with: version: '1.16.5' + - name: Install just + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Set Up Hatch Env env: HATCH_ENV: test.py${{ matrix.python-version }}-${{ matrix.dependency-set }} run: | - hatch env create "$HATCH_ENV" - hatch env run -e "$HATCH_ENV" list-env + just setup # https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache - name: Restore cached hypothesis directory id: restore-hypothesis-cache @@ -89,7 +93,7 @@ jobs: PYTEST_ADDOPTS: "--report-log=output-${{ matrix.python-version }}-log.jsonl" run: | echo "Using Hypothesis profile: $HYPOTHESIS_PROFILE" - hatch env run --env "$HATCH_ENV" run-hypothesis + just hypothesis # explicitly save the cache so it gets updated, also do this even if it fails. - name: Save cached hypothesis directory diff --git a/.github/workflows/issue-metrics.yml b/.github/workflows/issue-metrics.yml index 7b4cd08a0b..507451e04a 100644 --- a/.github/workflows/issue-metrics.yml +++ b/.github/workflows/issue-metrics.yml @@ -33,7 +33,7 @@ jobs: echo "last_month=$first_day..$last_day" >> "$GITHUB_ENV" - name: Run issue-metrics tool - uses: github-community-projects/issue-metrics@61084fa9599a62c7821f06602e180a42d1c7a205 # v5.0.1 + uses: github-community-projects/issue-metrics@a7dc2fb675661e208d4fc6fa321a6b95fc4a06f9 # v5.0.2 env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} SEARCH_QUERY: 'repo:zarr-developers/zarr-python is:issue created:${{ env.last_month }} -reason:"not planned"' diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 24521aadc5..f47fbcac11 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -27,7 +27,29 @@ jobs: with: python-version: "3.12" - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: true - - uses: j178/prek-action@4e14d07f9231acabce116ccfca13b13dd9755ece # v3.0.0 + - name: Install just + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 + # `uvx prek` builds each hook's environment from scratch, so cache them the + # way the prek action used to. Keyed on the hook config: a new pinned rev or + # a new hook is exactly when the cached environments stop being valid. + - name: Cache prek hook environments + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: ~/.cache/prek + # Justfile is hashed too: prek_version lives there, and a bumped prek + # must not restore a store built by the previous one. No restore-keys + # fallback for the same reason. + key: prek-${{ runner.os }}-${{ hashFiles('.pre-commit-config.yaml', 'Justfile') }} + - name: Lint + run: just lint + - name: Check justfile formatting + # After the linters, and never masking them: this is cosmetic, and a + # mis-formatted recipe should not cost someone their ruff/mypy results. + if: always() + run: just just-check diff --git a/.github/workflows/nightly_wheels.yml b/.github/workflows/nightly_wheels.yml index 2cb913511e..3d7fc60bb4 100644 --- a/.github/workflows/nightly_wheels.yml +++ b/.github/workflows/nightly_wheels.yml @@ -42,7 +42,7 @@ jobs: run: hatch build - name: Upload nightly wheels - uses: scientific-python/upload-nightly-action@e76cfec8a4611fd02808a801b0ff5a7d7c1b2d99 + uses: scientific-python/upload-nightly-action@16fa02eacee1655195143de09f03676e60ef2bf5 with: artifacts_path: dist anaconda_nightly_upload_token: ${{ secrets.ANACONDA_ORG_UPLOAD_TOKEN }} diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 650932309c..0d7d396fff 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -66,21 +66,25 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 - name: Install Hatch run: python -m pip install hatch==1.16.5 + - name: Install just + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Set Up Hatch Env env: HATCH_ENV: test.py${{ matrix.python-version }}-${{ matrix.dependency-set }} run: | - hatch env create "$HATCH_ENV" - hatch env run -e "$HATCH_ENV" list-env + just setup - name: Run Tests env: HYPOTHESIS_PROFILE: ci HATCH_ENV: test.py${{ matrix.python-version }}-${{ matrix.dependency-set }} run: | - hatch env run --env "$HATCH_ENV" run-coverage + just coverage - name: Upload coverage if: ${{ matrix.dependency-set == 'optional' && matrix.os == 'ubuntu-latest' }} uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0 @@ -115,20 +119,24 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 - name: Install Hatch run: python -m pip install hatch==1.16.5 + - name: Install just + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Set Up Hatch Env env: HATCH_ENV: ${{ matrix.dependency-set }} run: | - hatch env create "$HATCH_ENV" - hatch env run -e "$HATCH_ENV" list-env + just setup - name: Run Tests env: HATCH_ENV: ${{ matrix.dependency-set }} run: | - hatch env run --env "$HATCH_ENV" run-coverage + just coverage - name: Upload coverage uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0 with: @@ -150,15 +158,20 @@ jobs: python-version: '3.13' cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 - name: Install Hatch run: python -m pip install hatch==1.16.5 + - name: Install just + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Set Up Hatch Env run: | - hatch run doctest:pip list + HATCH_ENV=doctest just setup - name: Run Tests run: | - hatch run doctest:test + just doctest benchmarks: name: Benchmark smoke test @@ -174,14 +187,19 @@ jobs: python-version: '3.13' cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 - name: Install Hatch run: python -m pip install hatch==1.16.5 + - name: Install just + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Run Benchmarks env: ZARR_BENCHMARK_CLEAR_CACHE: '1' run: | - hatch env run --env "test.py3.13-minimal" run-benchmark + HATCH_ENV=test.py3.13-minimal just benchmark test-complete: name: Test complete diff --git a/.github/workflows/zarr-http-server-release.yml b/.github/workflows/zarr-http-server-release.yml index fb5e0fc21f..703cb7fd26 100644 --- a/.github/workflows/zarr-http-server-release.yml +++ b/.github/workflows/zarr-http-server-release.yml @@ -51,7 +51,7 @@ jobs: path: dist - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: false diff --git a/.github/workflows/zarr-http-server.yml b/.github/workflows/zarr-http-server.yml index a30d990a33..4a8fcca101 100644 --- a/.github/workflows/zarr-http-server.yml +++ b/.github/workflows/zarr-http-server.yml @@ -47,13 +47,16 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: true - name: Set up Python ${{ matrix.python-version }} run: uv python install ${{ matrix.python-version }} - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Sync test dependency groups # The examples group carries the deps the README examples need, so the # test that reads a served array back with a zarr client runs here @@ -74,9 +77,12 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Run ruff run: just lint @@ -92,13 +98,16 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: true - name: Set up Python run: uv python install 3.12 - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Sync test dependency group run: uv sync --group test --python 3.12 - name: Run mypy @@ -116,11 +125,14 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: true - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Build docs run: just docs-check diff --git a/.github/workflows/zarr-indexing-release.yml b/.github/workflows/zarr-indexing-release.yml index 5d95eb1996..2ae363294d 100644 --- a/.github/workflows/zarr-indexing-release.yml +++ b/.github/workflows/zarr-indexing-release.yml @@ -51,7 +51,7 @@ jobs: path: dist - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: false diff --git a/.github/workflows/zarr-indexing.yml b/.github/workflows/zarr-indexing.yml index 3b106e16aa..9af8ed1e3d 100644 --- a/.github/workflows/zarr-indexing.yml +++ b/.github/workflows/zarr-indexing.yml @@ -36,11 +36,14 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: true - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Set up Python ${{ matrix.python-version }} run: uv python install ${{ matrix.python-version }} # The suite imports nothing from `zarr`; it runs against the repo-root @@ -67,9 +70,12 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Run ruff # The ruff version pin lives in packages/zarr-indexing/justfile. run: just lint @@ -86,7 +92,7 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: true - name: Set up Python @@ -94,7 +100,10 @@ jobs: - name: Sync test dependency group run: uv sync --group test --python 3.12 - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Run pyright # The pyright invocation lives in packages/zarr-indexing/justfile. run: just typecheck @@ -111,11 +120,14 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: true - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Build docs # The strict mkdocs build lives in packages/zarr-indexing/justfile. run: just docs-check diff --git a/.github/workflows/zarr-metadata-release.yml b/.github/workflows/zarr-metadata-release.yml index 3125d2529a..d3409a1176 100644 --- a/.github/workflows/zarr-metadata-release.yml +++ b/.github/workflows/zarr-metadata-release.yml @@ -51,7 +51,7 @@ jobs: path: dist - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: false diff --git a/.github/workflows/zarr-metadata.yml b/.github/workflows/zarr-metadata.yml index 069f6d5060..3c42f4810f 100644 --- a/.github/workflows/zarr-metadata.yml +++ b/.github/workflows/zarr-metadata.yml @@ -39,11 +39,14 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: true - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Set up Python ${{ matrix.python-version }} run: uv python install ${{ matrix.python-version }} - name: Sync test dependency group @@ -63,9 +66,12 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Run ruff run: just lint @@ -81,11 +87,14 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: true - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Run pyright # The pyright version and interpreter pins live in the justfile. run: just typecheck @@ -102,11 +111,14 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4 # v10.1.0 with: enable-cache: true - name: Install just - uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4 + uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1 + with: + repo: casey/just + version: 1.58.0 - name: Build docs run: just docs-check diff --git a/.github/workflows/zizmor.yml b/.github/workflows/zizmor.yml index eb9a68d07d..19bdb461ea 100644 --- a/.github/workflows/zizmor.yml +++ b/.github/workflows/zizmor.yml @@ -32,4 +32,4 @@ jobs: persist-credentials: false - name: Run zizmor - uses: zizmorcore/zizmor-action@70fb788f84895a7701f5643d103d587e460b5c99 # v0.6.3 + uses: zizmorcore/zizmor-action@cc914d7f3750a2d13d75c7f184a1060aa0e9d482 # v0.6.4 diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 54345c819e..be96f1551f 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -17,13 +17,13 @@ default_language_version: repos: - repo: https://github.com/astral-sh/ruff-pre-commit - rev: v0.16.0 + rev: v0.16.6 hooks: - id: ruff-check args: ["--fix", "--show-fixes"] - id: ruff-format - repo: https://github.com/codespell-project/codespell - rev: v2.4.2 + rev: v2.4.3 hooks: - id: codespell args: ["-L", "fo,ihs,kake,te", "-S", "fixture"] @@ -34,7 +34,7 @@ repos: exclude: mkdocs.yml - id: trailing-whitespace - repo: https://github.com/DavidAnson/markdownlint-cli2 - rev: v0.22.1 + rev: v0.23.2 hooks: # Markdown structure/hygiene. Rule selection and ignores are in # .markdownlint-cli2.jsonc; complements ci/lint_docs.py (RST residue, @@ -72,7 +72,7 @@ repos: files: ^packages/zarr-http-server/ stages: [pre-push] - repo: https://github.com/scientific-python/cookie - rev: 2026.06.18 + rev: 2026.08.14 hooks: - id: sp-repo-review - repo: https://github.com/numpy/numpydoc @@ -89,10 +89,10 @@ repos: types: [python] files: ^(src|tests)/ - repo: https://github.com/zizmorcore/zizmor-pre-commit - rev: v1.26.1 + rev: v1.30.0 hooks: - id: zizmor - repo: https://github.com/twisted/towncrier - rev: 25.8.0 + rev: 26.9.0 hooks: - id: towncrier-check diff --git a/.readthedocs.yaml b/.readthedocs.yaml index dddf8449a4..872eb2be80 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -19,15 +19,24 @@ build: fi install: - pip install --upgrade pip - - pip install .[remote] --group docs + # The docs recipes resolve their toolchain from uv.lock, so uv is all that is + # needed here besides just itself. + - pip install uv==0.12.9 + # No GitHub Action is available on Read the Docs, so fetch the official just + # release and verify it against the checksum casey/just publishes. Installed + # into the build virtualenv's bin, which is already on PATH. + - | + curl -fsSL -o /tmp/just.tar.gz https://github.com/casey/just/releases/download/1.58.0/just-1.58.0-x86_64-unknown-linux-musl.tar.gz + echo "4a5cc2f53e6f0f8c59092a6cc38291eb729d46a7dd95d3ae582008881b84931d /tmp/just.tar.gz" | sha256sum -c - + tar -xzf /tmp/just.tar.gz -C "$READTHEDOCS_VIRTUALENV_PATH/bin" just pre_build: - | if [ "$READTHEDOCS_VERSION_TYPE" != "tag" ]; then - towncrier build --version Unreleased --yes; + just changelog-build --version Unreleased --yes; fi build: html: - - mkdocs build --strict --site-dir $READTHEDOCS_OUTPUT/html + - just docs-build --site-dir $READTHEDOCS_OUTPUT/html mkdocs: configuration: mkdocs.yml diff --git a/Justfile b/Justfile new file mode 100644 index 0000000000..6e56644207 --- /dev/null +++ b/Justfile @@ -0,0 +1,165 @@ +# Development and CI verbs live here; Hatch owns Python environments in pyproject.toml. +# Install: uv tool install hatch==1.16.5 && uv tool install rust-just==1.58.0 +# Select test dependencies/interpreter: HATCH_ENV=test.py3.13-minimal just test +# On Windows, use Git Bash (the same shell used by the test workflow). +set shell := ["bash", "-eu", "-o", "pipefail", "-c"] +set windows-shell := ["bash", "-eu", "-o", "pipefail", "-c"] +set positional-arguments + +hatch_env := env("HATCH_ENV", "test.py3.12-optional") +# Deliberately a different variable from HATCH_ENV: `just gpu` must not inherit a +# CPU test environment that happens to be exported in the caller's shell, which +# would run `pytest -m gpu` against an environment built without the gpu feature. +gpu_env := env("GPU_HATCH_ENV", "gputest.py3.12") +# Pinned so a prek release cannot change what CI lints without a commit here. +prek_version := "0.5.3" +# Documentation and changelog tooling resolves from uv.lock, not a hatch environment. +# There is only ever one docs toolchain, so it can be locked and hash-verified, and +# dependabot's uv ecosystem keeps it current. Hatch still owns the test environments, +# which exist per interpreter and per dependency set and cannot live in one lockfile. +docs_run := "uv run --frozen --group docs" +# The hatch docs environment used to set these; they belong with the mkdocs calls now. +mkdocs_env := "DISABLE_MKDOCS_2_WARNING=true NO_MKDOCS_2_WARNING=true" + +# List available recipes +default: + @just --list + +# List available Python environments +envs: + hatch env show + +# Create the selected Python environment and list its installed packages +setup: && list-env + hatch env create {{ quote(hatch_env) }} + +# List packages in the selected Python environment +list-env: + hatch run {{ quote(hatch_env) }}:pip list + +# Run unit tests; pass pytest arguments, e.g. just test -k 'array and resize' +test *args: + hatch run {{ quote(hatch_env) }}:pytest --ignore tests/benchmarks "$@" + +# Run unit tests and write coverage.xml and junit.xml +coverage *args: + hatch run {{ quote(hatch_env) }}:coverage run --source=src -m pytest --ignore tests/benchmarks --junitxml=junit.xml -o junit_family=legacy "$@" + hatch run {{ quote(hatch_env) }}:coverage xml + +# Run unit tests and generate an HTML coverage report +coverage-html *args: + hatch run {{ quote(hatch_env) }}:coverage run --source=src -m pytest --ignore tests/benchmarks "$@" + hatch run {{ quote(hatch_env) }}:coverage html + +# Serve the HTML coverage report (default port 8000) +coverage-serve *args: + hatch run {{ quote(hatch_env) }}:python -m http.server -d htmlcov "$@" + +# Run slow Hypothesis tests and write coverage.xml +hypothesis *args: + hatch run {{ quote(hatch_env) }}:coverage run --source=src -m pytest -nauto --run-slow-hypothesis tests/test_properties.py tests/test_store/test_stateful* "$@" + hatch run {{ quote(hatch_env) }}:coverage xml + +# Validate executable documentation code blocks +doctest *args: + hatch run doctest:pytest tests/test_docs.py -v "$@" + +# Run the benchmark suite +benchmark *args: + hatch run {{ quote(hatch_env) }}:pytest --benchmark-enable tests/benchmarks "$@" + +# Run benchmarks under CodSpeed +benchmark-codspeed *args: + hatch run {{ quote(hatch_env) }}:pytest tests/benchmarks --codspeed "$@" + +# Run GPU tests with coverage; select the environment with GPU_HATCH_ENV +gpu *args: + HATCH_ENV={{ quote(gpu_env) }} just coverage -m gpu "$@" + +# Build documentation (warnings are errors) +docs-build *args: + {{ mkdocs_env }} {{ docs_run }} mkdocs build --strict "$@" + +# Serve documentation with live reload +docs-serve *args: + {{ mkdocs_env }} {{ docs_run }} mkdocs serve --watch src "$@" + +# Check that every public export has API documentation +check-doc-exports *args: + {{ docs_run }} python ci/check_documented_exports.py docs/api "$@" + +# Check documentation source conventions +lint-docs *args: + {{ docs_run }} python ci/lint_docs.py "$@" + +# Report unlinked types in built documentation +check-doc-links *args: + {{ docs_run }} python ci/check_unlinked_types.py "$@" + +# Run source documentation checks followed by a strict build +docs-check: check-doc-exports lint-docs docs-build + +# Run all pre-commit hooks (ruff, codespell, mypy, repo-review, ...) +lint *args: + uvx prek@{{ prek_version }} run --show-diff-on-failure --color=always --all-files "$@" + +# Run hooks with a custom selection, e.g. just hooks run --last-commit +hooks +args: + uvx prek@{{ prek_version }} "$@" + +# prek is installed as a persistent uv tool rather than run through uvx: the hook shim +# prek writes into .git/hooks hard-codes the binary path it was installed from and falls +# back to `prek` on PATH, and a uvx archive path stops existing at the next cache prune. +# Install local pre-commit hooks +hooks-install: + uv tool install prek=={{ prek_version }} + prek install + +# Type-check the library using the locked tooling environment +typecheck *args: + uv run --frozen mypy "$@" + +# Check that uv.lock is in sync with pyproject.toml +lock-check: + uv lock --check + +# Update the dependency lockfile +lock *args: + uv lock "$@" + +# Build the source distribution and wheel +build *args: + hatch build "$@" + +# Create a changelog fragment (interactive without arguments) +changelog *args: + {{ docs_run }} towncrier create "$@" + +# Preview the next release's changelog +changelog-draft *args: + {{ docs_run }} towncrier build --draft --version Unreleased "$@" + +# Build release notes; pass --version and --yes when preparing a release +changelog-build *args: + {{ docs_run }} towncrier build "$@" + +# Check changelog filenames (default: changes/; accepts a package changes directory) +check-changelogs *args: + uv run --no-project python ci/check_changelog_entries.py "$@" + +# Check recipe formatting of the root Justfile and every package justfile +just-check: + just --fmt --check + shopt -s nullglob; for f in packages/*/justfile; do just --justfile "$f" --fmt --check; done + +# Run a zarr-metadata recipe, or list its recipes with no arguments +zarr-metadata *args: + just --justfile packages/zarr-metadata/justfile "$@" + +# Run a zarr-indexing recipe, or list its recipes with no arguments +zarr-indexing *args: + just --justfile packages/zarr-indexing/justfile "$@" + +# Run a zarr-http-server recipe, or list its recipes with no arguments +zarr-http-server *args: + just --justfile packages/zarr-http-server/justfile "$@" diff --git a/changes/3285.feature.md b/changes/3285.feature.md deleted file mode 100644 index 35507ea26c..0000000000 --- a/changes/3285.feature.md +++ /dev/null @@ -1,13 +0,0 @@ -JSON metadata validation now delegates to ``msgspec.convert`` for the type -coercions it supports (``Literal`` membership, ``int`` / ``bool`` strictness, -list-to-tuple), replacing the per-field hand-written ``parse_*`` logic. -User-defined attributes retain their existing JSON handling. -A latent generator-exhaustion bug in -``parse_storage_transformers`` is also fixed. See #3285. - -As a result some metadata inputs are now parsed more strictly. The previous -per-field checks compared values with ``==``, which accepts any numerically -equal object, so a float such as ``2.0`` was accepted as ``zarr_format``; it is -now rejected because it is not an ``int``. Booleans are likewise no longer -accepted where an ``int`` is expected, since ``bool`` is an ``int`` subclass. -Metadata that conforms to the Zarr specification is unaffected. diff --git a/changes/4149.doc.md b/changes/4149.doc.md deleted file mode 100644 index 8a473acac9..0000000000 --- a/changes/4149.doc.md +++ /dev/null @@ -1 +0,0 @@ -Added a Roadmap page to the documentation outlining future plans and intended changes to the library. diff --git a/changes/4174.bugfix.md b/changes/4174.bugfix.md deleted file mode 100644 index 37dc10078c..0000000000 --- a/changes/4174.bugfix.md +++ /dev/null @@ -1,13 +0,0 @@ -Array creation is now O(1) in the number of chunks per dimension. Chunk -normalization returns a `ChunkGrid` whose uniform dimensions are stored as a -size + extent pair (`FixedDimension`) instead of being expanded to one entry -per chunk, so creating arrays like -`zarr.create_array(store, shape=(2**62,), chunks=(1,), dtype='int32')` succeeds -instantly instead of raising `ValueError` or allocating gigabytes of memory. -The intermediate `ChunksTuple` representation was removed in the process, and -`ChunksLike` now admits per-dimension specs that mix a bare int (uniform chunk -size) with explicit edge-length sequences, matching what the normalizer and -the rectilinear grid spec already accepted. -This fixes the array-creation half of #4174; the coordinate-selection -allocation reported there is still tracked in that issue (#4172 fixed the -related case of sorted 1-D coordinate selections). diff --git a/changes/4189.bugfix.md b/changes/4189.bugfix.md deleted file mode 100644 index 76ef7e7a5e..0000000000 --- a/changes/4189.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Allow `Group.require_array` to accept a `ZDType` for `dtype`, matching the other array creation methods. Previously an existing array could only be required with a string or NumPy dtype. diff --git a/changes/4193.doc.md b/changes/4193.doc.md deleted file mode 100644 index 0972e8be2c..0000000000 --- a/changes/4193.doc.md +++ /dev/null @@ -1,4 +0,0 @@ -Converted remaining reStructuredText-style double-backtick markup to Markdown -single backticks in the docstrings of `zarr.api.asynchronous`, -`zarr.api.synchronous`, `zarr.core.array`, `zarr.registry`, and -`zarr.storage._common`. No functional changes. diff --git a/changes/4213.misc.md b/changes/4213.misc.md deleted file mode 100644 index 150e60b57b..0000000000 --- a/changes/4213.misc.md +++ /dev/null @@ -1 +0,0 @@ -Updated ruff to 0.16.0 and fixed the violations surfaced by its expanded default rule set: narrowed a blind `except Exception` in `StorePath.__eq__` to `AttributeError`, removed unnecessary `global` declarations in `zarr.core.sync`, made `subprocess.run` calls in tests pass `check=False` explicitly, and applied automatic fixes (`None` moved to the end of type unions, unused `noqa` directives removed). diff --git a/changes/4227.bugfix.md b/changes/4227.bugfix.md deleted file mode 100644 index 18293178bd..0000000000 --- a/changes/4227.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Consolidated metadata is now reconstructed independently of the order the keys appear in on disk. Previously, sibling subtrees whose keys were not adjacent in the persisted mapping lost their children, which made nodes unreachable through consolidated metadata -- most visibly for sibling groups whose names differ only by case. diff --git a/changes/4236.doc.md b/changes/4236.doc.md deleted file mode 100644 index 45f8a2c22c..0000000000 --- a/changes/4236.doc.md +++ /dev/null @@ -1 +0,0 @@ -Document how to reassign Read the Docs version slugs when publishing a subpackage release. diff --git a/changes/4239.bugfix.md b/changes/4239.bugfix.md deleted file mode 100644 index b5bc92f18b..0000000000 --- a/changes/4239.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -`FsspecStore.from_mapper` and `FsspecStore.from_url` no longer fail when converting a synchronous instance of an async-capable filesystem whose storage options contain objects that cannot be serialized to JSON (e.g. an `azure.identity.DefaultAzureCredential`). The async instance is now constructed from the original filesystem arguments instead of a JSON round-trip. diff --git a/changes/4247.doc.md b/changes/4247.doc.md deleted file mode 100644 index 6dbca1d3d6..0000000000 --- a/changes/4247.doc.md +++ /dev/null @@ -1,5 +0,0 @@ -Added a "Related Projects" page to the documentation listing the companion -packages developed in this repository — `zarr-metadata` and `zarr-indexing` — -and linked it from the landing page. Links to those packages now use the -canonical `https://zarr.readthedocs.io/projects/...` URLs, and each companion -package's documentation links back to the `zarr-python` docs. diff --git a/changes/4257.bugfix.md b/changes/4257.bugfix.md deleted file mode 100644 index f643cf31f0..0000000000 --- a/changes/4257.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Numpy integers are accepted as chunk sizes again. Since 3.3.0 a per-dimension chunk size that was a numpy integer (e.g. `chunks=(np.int64(2), np.int64(2))`, as produced by any computed chunk shape) raised `TypeError: 'numpy.int64' object is not iterable`, because the scalar chunk path narrowed on `int` while its caller dispatched on `numbers.Integral`. The same regression had broken numpy arrays as chunk specifications through the legacy `zarr.create` / `zarr.array` / `zarr.open_array` functions, which accepted them in 2.x and 3.2.x; those work again. (`zarr.create_array` and the functions built on it gain numpy-array support separately, in #4329.) A chunk specification that is neither an integer nor iterable now reports the offending value instead of failing with an opaque iteration error. diff --git a/changes/4260.bugfix.md b/changes/4260.bugfix.md deleted file mode 100644 index b703c47a35..0000000000 --- a/changes/4260.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -The `cast_value` codec now requires `cast-value-rs>=0.4.2`. Earlier versions of that backend silently corrupted data when handed an array that was not row-major — the layout the `transpose` codec produces — so a `cast_value` codec next to a `transpose` codec would either write transposed values with no error or fail with `ValueError: Input array must be contiguous`. The minimum version is enforced at runtime as well as in the package metadata, so an environment that already has an older `cast-value-rs` installed now raises `ImportError` when the codec is used, instead of corrupting data. diff --git a/changes/4261.misc.md b/changes/4261.misc.md deleted file mode 100644 index ca2a3fbb1e..0000000000 --- a/changes/4261.misc.md +++ /dev/null @@ -1 +0,0 @@ -The contents of the `zarr` source distribution are now defined by an explicit allowlist rather than a blocklist. Previously the sdist bundled the whole `packages/` tree — `zarr-indexing`, `zarr-metadata` and `zarr-http-server`, which are released as their own distributions — along with CI configuration and other repository files. The sdist also now ships `docs/`, so the test suite it carries can be collected and run from an unpacked sdist. diff --git a/changes/4265.bugfix.md b/changes/4265.bugfix.md deleted file mode 100644 index 6daf0fc7d0..0000000000 --- a/changes/4265.bugfix.md +++ /dev/null @@ -1,13 +0,0 @@ -Accept [universal-pathlib](https://github.com/fsspec/universal_pathlib) `UPath` objects wherever -zarr accepts a `StoreLike` value. A remote `UPath` now creates an `FsspecStore` using the -filesystem and storage options the `UPath` already carries, and a local `UPath` creates a -`LocalStore`, so that `UPath('/data')` and `Path('/data')` behave the same. - -Previously this worked only by accident: in universal-pathlib < 0.3 every `UPath` subclassed -`pathlib.Path` and implemented `__fspath__`, so remote paths were either converted to a URI string -by the caller or wrapped in a `LocalStore` that happened to dispatch through fsspec. Since -universal-pathlib 0.3 remote paths do neither, and passing one raised -`TypeError: Unsupported type for store_like`. - -`FsspecStore.from_upath` also now converts the `UPath`'s filesystem to async mode, instead of -raising `TypeError` for synchronous filesystems and warning for sync-mode instances of async ones. diff --git a/changes/4272.bugfix.md b/changes/4272.bugfix.md deleted file mode 100644 index 90616ca322..0000000000 --- a/changes/4272.bugfix.md +++ /dev/null @@ -1,38 +0,0 @@ -Explicit per-chunk size lists now always produce a rectilinear chunk grid, -even when the sizes happen to describe a regular grid (all equal, or all equal -with a smaller trailing chunk). Previously such input was silently collapsed to -a regular grid, which changed resize semantics: a regular grid grows by -extending the uniform pattern, while a rectilinear grid appends a new edge -chunk — the behavior an append-oriented layout like `(168,) * 13 + (24,)` -relies on. The grid kind now follows the input syntax, matching 3.2.x: -scalar chunk sizes (including numpy integers and the `-1` sentinel) produce a -regular grid, nested sequences produce a rectilinear grid. Rectilinear grids -remain gated behind `zarr.config.set({"array.rectilinear_chunks": True})`. -See #4174 for the accompanying O(1) chunk normalization change. - -One consequence for users who never enable rectilinear chunks: because a -nested sequence now always requests a rectilinear grid, a per-dimension -sequence of edge lengths that happens to be uniform — for example the -`((4,), (4,))` or `[[3, 3, 1]]` form that a dask array's `.chunks` attribute -produces — is no longer quietly accepted as a regular grid when the -`array.rectilinear_chunks` option is off. Such input raises -`ValueError: Rectilinear chunk grids are experimental and disabled by default`, -exactly as it did in 3.2.x; the silent acceptance existed only in 3.3.0. Pass -one integer per dimension (e.g. `chunks=(4, 4)`, or a dask array's -`.chunksize`) to request a regular grid. - -`zarr.from_array` with the default `chunks="keep"` / `shards="keep"` now -reproduces the source's stored grid exactly: a rectilinear grid is passed -through in O(number of dimensions), with uniform dimensions keeping their -bare-int shorthand; sharding under a rectilinear shard grid is preserved -instead of being silently dropped; and the default `write_data=True` copy -works for every grid kind. `Array.chunks` is now defined for any sharded array -(the inner chunks of a shard are always regular), and for sharded arrays with -a rectilinear shard grid `Array.info` no longer raises — it reports the shard -shape as `` — while `Array.nchunks_initialized` counts the chunks of -each initialized shard individually instead of raising. - -Apart from the nested-sequence input form noted above, everything described -here concerns rectilinear chunk grids, which remain an experimental feature -gated behind `zarr.config.set({"array.rectilinear_chunks": True})`; arrays -with regular chunk grids are unaffected. diff --git a/changes/4277.feature.md b/changes/4277.feature.md deleted file mode 100644 index f5c247496d..0000000000 --- a/changes/4277.feature.md +++ /dev/null @@ -1,28 +0,0 @@ -`zarr.registry.get_codec_class` now raises `zarr.errors.UnknownCodecError` instead of `KeyError` -when no implementation is registered for a codec, and `zarr.core.config.BadConfigError` instead of -`KeyError` when the implementation named in `config["codecs"][name]` is not registered. -`zarr.registry.get_numcodec` raises `UnknownCodecError` instead of the `ValueError` numcodecs -raises for an unregistered Zarr format 2 codec id (`numcodecs.errors.UnknownCodecError` on -numcodecs 0.15.1 and later). All of these are subclasses of `ValueError`, so `except ValueError` -is unaffected, but `except KeyError` and `except numcodecs.errors.UnknownCodecError` are. - -These errors now name Python packages known to provide the codec, so that a user who cannot read -an array learns what to install: - -``` -An implementation for codec 'wavpack' is not available. Register one explicitly using the codec -registry (see ...), or install a Python package that registers a codec implementation with -numcodecs. Known packages supporting this codec: wavpack-numcodecs. -``` - -The tables covering this live in `src/zarr/registry.py`, one per Zarr format, and include the -codecs `numcodecs` gates behind its own optional dependencies (`zfpy`, `pcodec`, `crc32c`, -`msgpack2`). Codec authors can add their published package to them. - -A codec whose `from_dict` raises `KeyError` on a malformed configuration now surfaces as -`zarr.errors.MetadataValidationError` naming the codec and the missing key. Previously it was -reported as `UnknownCodecError: Unknown codec: ''`, presenting a configuration -key as though it were a codec name, and on the `zarr.open` path a bare `KeyError` could be -swallowed by the array-then-group fallback and reported as an unrelated group error. - -`zarr.errors.UnknownCodecError` is now exported from `zarr.errors`. diff --git a/changes/4279.bugfix.md b/changes/4279.bugfix.md deleted file mode 100644 index 7d99a49ccf..0000000000 --- a/changes/4279.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -A `scale_offset` codec configured with a string-valued zero scale is now rejected. `scale` accepts strings, and no string is ever equal to `0`, so `"0"`, `"0.0"` and the hex form `"0x0000000000000000"` skipped the "scale must be non-zero" check that the numeric `0` triggers. On float data types the array was created, every chunk was written as zero and read back as `nan` with no error, and the zero scale was persisted to the metadata so reopening the store reproduced it; on integer data types the codec raised `ZeroDivisionError` instead of `ValueError`. The check now runs on the parsed scalar rather than the value as supplied. diff --git a/changes/4284.bugfix.md b/changes/4284.bugfix.md deleted file mode 100644 index 4ad1c456b4..0000000000 --- a/changes/4284.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed a `ValueError` when setting an orthogonal selection on a sharded array where more than one dimension is indexed by an array. The sharding codec re-derives an indexer from the chunk selection it is handed, which turns such a selection into a coordinate selection addressing the value buffer flat, so the write failed on a shape mismatch. Both partial-encode paths are fixed, so the write works under either codec pipeline. diff --git a/changes/4286.bugfix.md b/changes/4286.bugfix.md deleted file mode 100644 index e1ebb0dee2..0000000000 --- a/changes/4286.bugfix.md +++ /dev/null @@ -1,14 +0,0 @@ -Fixed integer array indexing with unsigned index dtypes. An unsorted index such as -`np.array([3, 0], dtype="uint8")` spanning more than one chunk raised `IndexError`, because -the order check used `np.diff`, which wraps on unsigned dtypes and misclassified a -descending selection as increasing. Separately, a `uint64` index raised `IndexError` on both -`array[...]` and `array.vindex[...]` — sorted or not — because `uint64` promotes to -`float64` against a signed chunk offset. Index arrays are now cast to `intp`. - -Unsigned indices are bounds-checked before this conversion, so values such as -`np.uint64(2**64 - 1)` are rejected rather than wrapping to a negative index and -reading or overwriting an element at the end of the array. - -Negative-index normalization copies indices before modifying them, preserving -caller-owned arrays and supporting read-only index arrays. Reusing one index -array across axes of different lengths now normalizes each axis independently. diff --git a/changes/4288.bugfix.md b/changes/4288.bugfix.md deleted file mode 100644 index 4080c2b44c..0000000000 --- a/changes/4288.bugfix.md +++ /dev/null @@ -1,3 +0,0 @@ -`zarr.from_array` now defaults to the fill value and the attributes of the source array. Previously both were silently discarded: the array was created with the data type's default scalar and no attributes. - -An explicit `fill_value=None` now selects the data type's default scalar (Zarr format 3) or a null fill value (Zarr format 2), consistently with `create_array`, and an empty `attributes` dict creates the array with no attributes. diff --git a/changes/4305.bugfix.md b/changes/4305.bugfix.md deleted file mode 100644 index 13221d8e3b..0000000000 --- a/changes/4305.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed an infinite loop when creating a 0-dimensional array with `shards="auto"` while the `array.target_shard_size_bytes` config option is set. Such arrays now resolve to `shards=()`, matching the behavior when no shard size target is configured. diff --git a/changes/4307.bugfix.md b/changes/4307.bugfix.md deleted file mode 100644 index b77c89cfa5..0000000000 --- a/changes/4307.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed `chunks=-1` on a zero-length axis resolving to an invalid chunk size of 0, which caused a `ValueError`, `ZeroDivisionError`, or infinite loop depending on the sharding configuration. Such axes now get chunk size 1, matching `chunks="auto"`. diff --git a/changes/4316.bugfix.md b/changes/4316.bugfix.md deleted file mode 100644 index e4b9bd8d94..0000000000 --- a/changes/4316.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed a `ValueError` when setting an orthogonal selection on a sharded array that mixes an integer index with two or more array indices, such as `a.oindex[[3, 1, 2], 1, [0, 2]] = value`. The fix for the array-only case in #4284 reshaped the value only when its shape matched the coordinate selection exactly; the sharding codec now also ravels a value that is the selection's shape minus the integer-indexed axes. Values of any other rank are left alone, so a write that is invalid on an unsharded array fails the same way on a sharded one. Both partial-encode paths share one helper for this. diff --git a/changes/4324.bugfix.md b/changes/4324.bugfix.md deleted file mode 100644 index d71f105ef8..0000000000 --- a/changes/4324.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Opening a Zarr format 3 array whose codec's ``from_dict`` raised a ``KeyError`` with no arguments used to fail with an unrelated ``IndexError: tuple index out of range`` while formatting the error message. Because that ``IndexError`` is not a ``ValueError``, it also escaped the array-then-group fallback in ``zarr.open`` and broke group operations such as ``Group.members()`` and ``"child" in group`` when any child array used such a codec. The ``KeyError`` is now always reported as a ``MetadataValidationError`` naming the codec, with the offending key included only when the ``KeyError`` carried one. diff --git a/changes/4325.bugfix.md b/changes/4325.bugfix.md deleted file mode 100644 index 19e1fc96aa..0000000000 --- a/changes/4325.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -`zarr.from_array` now deep-copies the source array's attributes instead of sharing nested dicts and lists between the source and the new array. Previously, mutating a nested attribute on the new array (for example ``dst.attrs["meta"]["tags"].append(...)``) silently changed the source array's in-memory attributes too. Deeply nested attributes can raise `RecursionError` during the copy even if they can be stored and reopened; the threshold depends on Python's recursion limit and call stack. Pass `attributes={}` to omit inherited attributes. diff --git a/changes/4326.misc.md b/changes/4326.misc.md deleted file mode 100644 index 500b4cdf8b..0000000000 --- a/changes/4326.misc.md +++ /dev/null @@ -1 +0,0 @@ -Removed the private test helper `_gzip_streams_equal_except_mtime` from `zarr.codecs.gzip`; it now lives in the test suite. diff --git a/changes/4328.bugfix.md b/changes/4328.bugfix.md deleted file mode 100644 index 2751dbd47f..0000000000 --- a/changes/4328.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed `chunks=False` on a zero-length axis resolving to a chunk size of 0, which raised a `ValueError` for Zarr format 3, raised a `ZeroDivisionError` with `shards="auto"`, and silently wrote invalid `chunks` metadata for Zarr format 2. `False` now takes the same path as `chunks=-1`, so such axes get chunk size 1, matching `chunks="auto"`. diff --git a/changes/4329.feature.md b/changes/4329.feature.md deleted file mode 100644 index 65b0bc598f..0000000000 --- a/changes/4329.feature.md +++ /dev/null @@ -1 +0,0 @@ -`zarr.create_array`, `Group.create_array`, `zarr.from_array`, and the entry points built on them now accept a numpy array as the `chunks` or `shards` specification, alongside ints, tuples, and numpy integer scalars. This is new for that API: it has never accepted numpy arrays in any 3.x release, because each entry point compared the specification to the `"auto"` or `"keep"` sentinel string before normalizing it, and for a numpy array that comparison raised numpy's ambiguous-truth-value `ValueError`. Those sentinel checks are now guarded so array-like specifications reach the normalizer, bringing this API in line with the legacy `zarr.create` / `zarr.array` / `zarr.open_array` functions, which have accepted numpy arrays since 2.x. diff --git a/changes/4331.misc.md b/changes/4331.misc.md deleted file mode 100644 index 949ebfffd3..0000000000 --- a/changes/4331.misc.md +++ /dev/null @@ -1 +0,0 @@ -Added `zarr.testing.strategies.sharded_arrays`, a Hypothesis strategy that always generates a sharded Zarr v3 array, drawing the chunk shape, shard shape, subchunk write order and inner codec chain, and — for half of its draws, or as selected by its `nested` argument — one level of recursive sharding, where the chunks are grouped into inner shards that are in turn grouped into the stored shards. The `test_oindex` and `test_vindex` property tests now draw it as a third arm alongside `simple_arrays` and `rectilinear_arrays`, so the sharding codec's write path for orthogonal selections with two or more array-indexed axes is exercised in tens of examples per run instead of about one. diff --git a/changes/4335.bugfix.md b/changes/4335.bugfix.md deleted file mode 100644 index 38e24f466b..0000000000 --- a/changes/4335.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -`from_array` preserves a Zarr source's explicit data type instead of trying to infer it from its NumPy dtype, allowing variable-length bytes arrays to be copied. diff --git a/changes/4339.bugfix.md b/changes/4339.bugfix.md deleted file mode 100644 index 0697e5d232..0000000000 --- a/changes/4339.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -The documentation build and the documentation test suite no longer delete a `data/` directory relative to the current working directory. Two executable docs sessions opened with `shutil.rmtree('data', ignore_errors=True)` to make their examples re-runnable; because executed docs blocks run in the process working directory rather than the docs tree, `mkdocs build -f /mkdocs.yml` or `pytest tests/test_docs.py` started from any directory containing a `data/` folder — a project checkout, or `/` — silently emptied it. The sdist ships `docs/` and `tests/` and `testpaths` collects `docs/user-guide`, so this reached anyone running the shipped test suite, not only contributors. The deletions are gone; the on-disk examples in the quick start, arrays, groups, storage and performance guides now create with `overwrite=True` (or `zarr.save_array(..., mode="w")`), which is also what a reader re-running an example needs, and a new docs test rejects any executed block that calls a filesystem deletion. diff --git a/changes/4363.bugfix.md b/changes/4363.bugfix.md new file mode 100644 index 0000000000..fbccb109f4 --- /dev/null +++ b/changes/4363.bugfix.md @@ -0,0 +1,8 @@ +Removed every `assert` statement from runtime code and enabled ruff's `S101` +rule to keep them out. Asserts are stripped under `python -O`, and two of them +were load-bearing: `GroupMetadata.from_dict` asserted on `node_type` and the +array-to-group fallback in `zarr.open` relied on catching the resulting +`AssertionError`, and `make_store` asserted on `mode` before the real +validation. Both now raise proper errors (`NodeTypeValidationError` and +`ValueError`). Redundant asserts were deleted, type-narrowing asserts were +restructured so mypy narrows without them, and the rest became explicit raises. diff --git a/changes/4368.misc.md b/changes/4368.misc.md new file mode 100644 index 0000000000..bdb87c70cf --- /dev/null +++ b/changes/4368.misc.md @@ -0,0 +1 @@ +Prevent the downstream xarray CI job from replacing the branch version of Zarr with a PyPI release before reporting versions and running tests. diff --git a/changes/4372.misc.md b/changes/4372.misc.md new file mode 100644 index 0000000000..55a800a3f3 --- /dev/null +++ b/changes/4372.misc.md @@ -0,0 +1 @@ +Define development and CI commands in a root Justfile, with Hatch managing Python environments and documentation tooling resolving from `uv.lock`. The Hatch script tables are removed, so invocations like `hatch env run --env test.py3.12-optional run-coverage` become `just coverage`; see the contributing guide for the full set. Extracted from [#4096](https://github.com/zarr-developers/zarr-python/pull/4096). diff --git a/docs/blog/posts/3.3.0-release.md b/docs/blog/posts/3.3.0-release.md index 13368848ae..e64e9f7d85 100644 --- a/docs/blog/posts/3.3.0-release.md +++ b/docs/blog/posts/3.3.0-release.md @@ -167,3 +167,30 @@ We hope these new features are helpful, and we would appreciate any feedback tha The updates in this release are just the first step of a larger performance-oriented direction for Zarr-Python. Landing these two enhancements taught us a *lot* about the performance-sensitive areas of the library. We can and will invest more time in performance tuning, e.g. by adding or changing abstractions, writing code for special cases, etc. We plan to consider including compiled code that should enable significant performance improvements. The [`zarrs`](https://zarrs.dev/) project is an ecosystem of Zarr tools written in Rust, with [extremely high performance](https://book.zarrs.dev/#-zarrs-is-fast-). Is there a `zarrs` binding in Zarr-Python's future? I hope so! We are keenly observing development of [`zarrista`](https://developmentseed.org/zarrista/latest/) as a proof-of-concept for what a Python-`zarrs` binding layer might look like. Stay tuned! + +## Contributors + +Thanks to everyone who contributed to this release (`*` marks a first-time contributor to Zarr-Python): + +- [@aldenks](https://github.com/aldenks) +- [@AMBRA7592](https://github.com/AMBRA7592) * +- [@binggao1230](https://github.com/binggao1230) * +- [@chuckwondo](https://github.com/chuckwondo) * +- [@d-v-b](https://github.com/d-v-b) +- [@DimitriPapadopoulos](https://github.com/DimitriPapadopoulos) +- [@goutamadwant](https://github.com/goutamadwant) * +- [@ilan-gold](https://github.com/ilan-gold) +- [@jhamman](https://github.com/jhamman) +- [@josh-ag2](https://github.com/josh-ag2) * +- [@kabilar](https://github.com/kabilar) * +- [@keewis](https://github.com/keewis) +- [@lhoupert](https://github.com/lhoupert) * +- [@maxrjones](https://github.com/maxrjones) +- [@NIK-TIGER-BILL](https://github.com/NIK-TIGER-BILL) * +- [@oldrobotdev](https://github.com/oldrobotdev) * +- [@SAY-5](https://github.com/SAY-5) * +- [@sehoffmann](https://github.com/sehoffmann) * +- [@selmanozleyen](https://github.com/selmanozleyen) * +- [@stibrew](https://github.com/stibrew) * +- [@TomAugspurger](https://github.com/TomAugspurger) +- [@zkoppert](https://github.com/zkoppert) * diff --git a/docs/blog/posts/3.4.0-release.md b/docs/blog/posts/3.4.0-release.md new file mode 100644 index 0000000000..7612838f3b --- /dev/null +++ b/docs/blog/posts/3.4.0-release.md @@ -0,0 +1,120 @@ +--- +date: 2026-09-10 +authors: + - d-v-b +categories: + - Release +draft: false +--- + +# `zarr` 3.4.0 + +`zarr` 3.4.0 is out! This post covers the highlights of the latest version of `zarr`. We will also introduce some new subpackages we are developing as part of an effort to make `zarr` more modular. See the full [release notes](../../release-notes.md) for the per-PR breakdown, or keep reading to catch the digested version. + +## Stackification + +This first item is not a new `zarr` feature. Rather, it's a **Zarr-Python** feature: Zarr-Python (the project) is becoming a *stack* of related Python packages, foremost among them `zarr`, the Python library. In the last few months Zarr-Python gained the following stand-alone Python packages: + +- [`zarr-metadata`](https://zarr-metadata.readthedocs.io/), a library narrowly scoped to modelling Zarr metadata documents (e.g., `zarr.json`), with precise type definitions and validation routines. Designed for anyone who wants type safety for Zarr V2 or V3 metadata without re-implementing the Zarr specs. `zarr-metadata` is very lightweight -- it does not depend on `zarr` -- so it should be a very cheap dependency for projects to add, either as a runtime or test dependency. We hope to replace most of the metadata logic in `zarr` with the contents of `zarr-metadata`. +- [`zarr-indexing`](https://zarr-indexing.readthedocs.io/), a library concerned with data structures and algorithms for lazy, chunked array indexing routines based on Google's trailblazing [TensorStore](https://google.github.io/tensorstore/) library. We hope to replace the array indexing logic in `zarr` with routines defined in `zarr-indexing`, and we also want to use `zarr-indexing` to support new lazy indexing functionality for arrays in `zarr`. +- [`zarr-http-server`](https://zarr-http-server.readthedocs.io/), a library that serves a `zarr` `Store`, `Array` or `Group` over HTTP via an ASGI app. This allows any array-like data to be dynamically exposed as Zarr. For example, you can take TIFF stacks, access them with `tifffile`, and use `tifffile`'s Zarr layer with `zarr-http-server` to expose the TIFF data as Zarr over HTTP for consumption by clients that don't know how to read TIFF. There are no plans for `zarr` to depend on this package. + +These three packages are just a start. We have in-flight PRs for defining packages for Zarr chunk key encodings ([PR 4298](https://github.com/zarr-developers/zarr-python/pull/4298)), stores ([PR 4318](https://github.com/zarr-developers/zarr-python/pull/4318)), and codecs ([PR 4319](https://github.com/zarr-developers/zarr-python/pull/4319)). + +All of these packages are pre-1.0 and changing rapidly; `zarr` does not yet depend on them. When the relevant APIs stabilize, we plan to gradually replace functionality defined in `zarr` today for the equivalent functionality in a subpackage. For example, the [store PR](https://github.com/zarr-developers/zarr-python/pull/4318) includes a `legacy` module that contains a verbatim copy of `zarr`'s current `Store` API, so that when `zarr` depends on `zarr-storage`, we can swap out functionality defined in `zarr` for identical functionality in `zarr-storage`, and users won't notice any difference beyond an additional installed package. + +The strategy here is to identify the logically separate parts of a Zarr implementation, and build a separate package around each part. This is not an original idea: the Rust [`zarrs`](https://zarrs.dev/) project is composed of many separate packages and this architecture has worked well for that project. + +Why are we doing this? Factoring `zarr` into separate packages may seem like a lot of churn with little upside beyond more maintenance. It's true that there will be new kinds of maintenance, and publishing gets a bit more complicated. But we see the following advantages to organizing Zarr-Python into subpackages: + +- **Modularity:** Installing `zarr` brings in `numcodecs` and other dependencies required for the full Zarr format. If a Python user only needs to parse Zarr metadata documents, e.g. because they are using [`zarrista`](https://developmentseed.org/zarrista/latest/) or [`tensorstore`](https://google.github.io/tensorstore/) to do IO, then the full set of `zarr` dependencies is wasteful. We want to make it as easy as possible for people to use Zarr correctly. That means it should be easy to install a package that defines the syntax and semantics of the Zarr metadata documents, without committing consumers of that package to a full runtime. +- **Agility:** We believe that separate packages will enable more effective development of the Zarr stack. For example, we want to develop a new version of our storage APIs that formally separate sync and async stores. We could carry out this development inside `zarr`, but `zarr` releases would mix storage API developments with ordinary `zarr` development, and so changes to the store API itself would not be transparent to consumers, e.g. implementers of third-party storage backends. If the storage API is defined entirely in a separate package, then we can version that package exactly when the storage API changes, which makes the design history much more visible to consumers. +- **Focus:** Functionality in `zarr` has a tendency to be as good as it needs to be for `zarr`'s needs. This is reasonable, but it can complicate the decision about when to make internal APIs public. For example, the core metadata APIs in `zarr` are not clearly marked public, which is unfortunate because the metadata is a really important part of Zarr! We hope that devoting an entire package to, e.g., Zarr metadata, makes it easier to define a public API surface that `zarr` and any other consumer can depend on. We want these subpackages to treat `zarr` as the main, but not sole, consumer, which means we can implement useful features even if `zarr` doesn't need them today. +- **Packaging:** Defining core APIs in separate packages solves a packaging problem for `zarr`. If `zarr` defines core APIs like storage and codec interfaces internally, third-party packages must depend on `zarr` to access those APIs. This in turn means `zarr` cannot offer those third-party packages as optional dependencies without incurring a circular dependency. Introducing a common dependency for `zarr` and `third-party-zarr-package` removes the circular dependency issue entirely. + +For the full list of subpackages, see the "Related Projects" [page in our docs](../../subprojects.md). + +## Our Roadmap + +With the Zarr V2 -> V3 format transition largely settled, it's worth reflecting on how we want `zarr` to evolve. The previous section outlined one facet of that evolution -- defining `zarr` as the top of a stack of interlocking Zarr tools. There are other big changes ahead, foremost among them our goal of unlocking a vast performance improvement by wrapping the `zarrs` Rust crate via [PyO3](https://pyo3.rs/) bindings defined in [`zarrista`](https://developmentseed.org/zarrista/latest/). You can read more about this direction and other future plans in our new [roadmap document](../../roadmap.md), which is published as part of the 3.4.0 release. + +## Rectilinear Chunk Grid Improvements + +The experimental rectilinear (irregular) chunk grid support (opt in with `zarr.config.set({"array.rectilinear_chunks": True})`) got several API improvements and bug fixes, mostly in [#4218](https://github.com/zarr-developers/zarr-python/pull/4218) and [#4272](https://github.com/zarr-developers/zarr-python/pull/4272): + +- Chunk grid flavor (regular vs rectilinear) now consistently follows input syntax: a nested per-dimension list always requests a rectilinear grid, even when the edges happen to be uniform. This fixes a regression in 3.3.0, where uniform-plus-short-tail lists requested a regular grid, which changed resize semantics. +- `zarr.from_array(..., chunks="keep")` reproduces a rectilinear source grid + exactly (in O(ndim)); sharding under a rectilinear shard grid is preserved + instead of dropped. +- `Array.chunks` is defined for any sharded array; `Array.info` and + `Array.nchunks_initialized` work for rectilinear shard grids instead of + raising (`info` shows the shard shape as ``). +- Zero-length axes: `chunks=-1` and `chunks=False` resolve to chunk size 1 ([#4307](https://github.com/zarr-developers/zarr-python/pull/4307), [#4328](https://github.com/zarr-developers/zarr-python/pull/4328)) instead of `ZeroDivisionError` / invalid metadata. + +## Useful Codec Errors + +`zarr` does not support every possible codec out of the box. Instead, we allow external packages to register their codec implementations with `zarr` at runtime, which broadens the palette of codecs we can support. When `zarr` reads an array with a codec it doesn't recognize, there's a good chance that the fix is to install the package that ships that codec implementation. But how should a user know which package they need? + +Previously, `zarr` handled an unknown codec with an opaque error, conveying only that `zarr` doesn't recognize the codec. Now, thanks to PRs [#4277](https://github.com/zarr-developers/zarr-python/pull/4277) and [#4351](https://github.com/zarr-developers/zarr-python/pull/4351), we maintain a table that relates the names of well-known codecs `zarr` doesn't ship with to packages that provide implementations. We use this table to provide a useful error message when an unknown codec is encountered. For example, opening an array whose metadata names the `imagecodecs_jpeg` codec, without any package that provides it installed: + +```python exec="true" session="codec-errors" source="above" result="ansi" +import json + +import zarr +from zarr.buffer.cpu import buffer_prototype + +metadata = { + "zarr_format": 3, + "node_type": "array", + "shape": [4, 4], + "data_type": "uint8", + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [4, 4]}}, + "chunk_key_encoding": {"name": "default"}, + "fill_value": 0, + "codecs": [{"name": "bytes"}, {"name": "imagecodecs_jpeg"}], + "attributes": {}, +} +store = {"zarr.json": buffer_prototype.buffer.from_bytes(json.dumps(metadata).encode())} + +try: + zarr.open_array(store, mode="r") +except zarr.errors.UnknownCodecError as e: + print(e) +``` + +The table lives in [`zarr.registry`](../../api/zarr/registry.md); if you maintain a package that provides a codec, please add it there. + +## New Dependency: `msgspec` + +Zarr metadata is composed of JSON documents that must comply with various specification documents, e.g. the Zarr V2 and V3 specs. `zarr` should model these JSON documents in a way that's correct, so that we generate interoperable data, and useful for developers, so they can also generate interoperable data. To do that we need to define two things: precise types, and routines for coercing or narrowing input values with unknown shapes into our precise types. + +Writing types for JSON data structures has gotten a lot easier in Python with the development of `TypedDict` utility types, but writing a validation function that checks an unknown input against an arbitrary `TypedDict` is arbitrary work. This isn't a hard programming task; rather, it's one that's very easy to mess up, e.g. by omitting an important check, or failing to model a field as optional. + +Runtime validation libraries like `pydantic`, `beartype`, and `msgspec` use type annotations to generate runtime type checking routines. Using one of these libraries could spare us a lot of hand-written type checking code. We have long discussed adding a runtime type checker to `zarr`, and as of 3.4.0 we are now using [`msgspec`](https://github.com/msgspec/msgspec) in this capacity ([#4063](https://github.com/zarr-developers/zarr-python/pull/4063), resolving [#3285](https://github.com/zarr-developers/zarr-python/issues/3285)). We chose `msgspec` because of its performance and minimal dependency count, and we expect it to deliver big value whenever we introduce new codecs, chunk grids, or data types to `zarr`, as we can just define the type for the metadata and get a validation routine for free. + +## Tell us what you think + +As always, we would love any and all feedback about the work highlighted in this release! We would especially appreciate feedback from Python users who work with Zarr data and **don't** use the `zarr` package, since we are hoping to target their needs with our new family of subpackages. + +## Contributors + +Thanks to everyone who contributed to this release (`*` marks a first-time contributor to Zarr-Python): + +- [@arcusbuilds](https://github.com/arcusbuilds) * +- [@cucuwang](https://github.com/cucuwang) * +- [@d-v-b](https://github.com/d-v-b) +- [@dylanpulver](https://github.com/dylanpulver) * +- [@glaziermag](https://github.com/glaziermag) * +- [@jhamman](https://github.com/jhamman) +- [@JOhnsonKC201](https://github.com/JOhnsonKC201) * +- [@paraseba](https://github.com/paraseba) +- [@selmanozleyen](https://github.com/selmanozleyen) +- [@sheikhayaan](https://github.com/sheikhayaan) * +- [@Tomatokeftes](https://github.com/Tomatokeftes) * +- [@vup903](https://github.com/vup903) * + +__Our [previous blog post](3.3.0-release.md) omitted the contributor list, so we've updated that post to include one.__ + +!!! info "AI Usage Disclaimer" + + The outline for this post was generated by Claude Fable after prompting with the set of unreleased changes since 3.3.0. That outline was then heavily expanded by @d-v-b, introducing a large number of typographical errors. Those errors were removed, code snippets added, and hyperlinks resolved, by Claude. diff --git a/docs/contributing.md b/docs/contributing.md index 369e60110a..45e401e064 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -80,19 +80,52 @@ git remote add upstream git@github.com:zarr-developers/zarr-python.git ### Creating a development environment -To work with the Zarr source code, it is recommended to use [hatch](https://hatch.pypa.io/latest/index.html) to create and manage development environments. Hatch will automatically install all Zarr dependencies using the same versions as are used by the core developers and continuous integration services. Assuming you have a Python 3 interpreter already installed, and you have cloned the Zarr source code and your current working directory is the root of the repository, you can do something like the following: +The root `Justfile` defines development and CI commands. [just](https://just.systems/) +runs these commands, while [Hatch](https://hatch.pypa.io/latest/index.html) manages +the Python environments declared in `pyproject.toml`. Install +[uv](https://docs.astral.sh/uv/getting-started/installation/) first, then the two task +tools. `uv tool install` puts them in their own environments, which a plain `pip install` +cannot do on a Python that marks itself externally managed (Debian, Ubuntu, Homebrew): ```bash -pip install hatch -hatch env show # list all available environments +uv tool install hatch==1.16.5 +uv tool install rust-just==1.58.0 # or any option from https://just.systems/man/en/packages.html +just # list available commands +just envs # list Python environments +just setup # create the default test environment +just test ``` -To verify that your development environment is working, you can run the unit tests for one of the test environments, e.g.: +Test recipes default to `test.py3.12-optional`. Set `HATCH_ENV` to select a different +interpreter or dependency set, just as CI does. `just gpu` reads `GPU_HATCH_ENV` +instead, so an exported `HATCH_ENV` cannot silently send GPU tests to an +environment built without the `gpu` feature, and `just doctest` always runs in the +`doctest` environment. On Windows, run these commands in +Git Bash. ```bash -hatch env run --env test.py3.12-optional run +HATCH_ENV=test.py3.13-minimal just test +HATCH_ENV=min_deps just coverage +HATCH_ENV=upstream just coverage +GPU_HATCH_ENV=gputest.py3.12 just gpu +just test tests/test_array.py -k 'resize and not async' ``` +Arguments after the recipe name are forwarded to the underlying tool. Use +`just --show test` to inspect a command. Package-specific commands live in the +`justfile` inside each package directory. The root recipes delegate to these files, +so you can also run package commands from the repository root: + +```bash +just zarr-metadata # list this package's recipes +just zarr-metadata test +just zarr-indexing test-tensorstore +just zarr-http-server docs-check +``` + +The package justfile sets the working directory and defines the command and its +environment. Root recipes forward arguments without duplicating those definitions. + ### Creating a branch Before you do any new work or submit a pull request, please open an issue on GitHub to report the bug or propose the feature you'd like to add. @@ -125,10 +158,10 @@ Again, any conflicts need to be resolved before submitting a pull request. ### Running the test suite -Zarr includes a suite of unit tests. The simplest way to run the unit tests is to activate your development environment (see [creating a development environment](#creating-a-development-environment) above) and invoke: +Zarr includes a suite of unit tests. The simplest way to run the unit tests is to invoke: ```bash -hatch env run --env test.py3.12-optional run +just test ``` All tests are automatically run via GitHub Actions for every pull request and must pass before code can be accepted. Test coverage is also collected automatically via the Codecov service. @@ -137,46 +170,36 @@ All tests are automatically run via GitHub Actions for every pull request and mu All code must conform to the PEP8 standard. Regarding line length, lines up to 100 characters are allowed, although please try to keep under 90 wherever possible. -`Zarr` uses a set of git hooks managed by [`prek`](https://github.com/j178/prek), a fast, Rust-based pre-commit hook manager that is fully compatible with `.pre-commit-config.yaml` files. `prek` can be installed locally by running: - -```bash -uv tool install prek -``` - -or: - -```bash -pip install prek -``` +`Zarr` uses a set of git hooks managed by [`prek`](https://github.com/j178/prek), a fast, Rust-based pre-commit hook manager compatible with `.pre-commit-config.yaml`. The recipes pin the prek version: `just lint` and `just hooks` run it through `uvx`, and `just hooks-install` installs it as a persistent `uv tool` so the git hook can find it on later commits. The hooks can be installed locally by running: ```bash -prek install +just hooks-install ``` This will run the checks every time a commit is created locally. The checks will by default only run on the files modified by a commit, but the checks can be triggered for all the files by running: ```bash -prek run --all-files +just lint ``` You can also run hooks only for files in a specific directory: ```bash -prek run --directory src/zarr +just hooks run --directory src/zarr ``` Or run hooks for files changed in the last commit: ```bash -prek run --last-commit +just hooks run --last-commit ``` To list all available hooks: ```bash -prek list +just hooks list ``` If you would like to skip the failing checks and push the code for further discussion, use the `--no-verify` option with `git commit`. @@ -188,7 +211,7 @@ If you would like to skip the failing checks and push the code for further discu Zarr strives to maintain 100% test coverage under the latest Python stable release. Both unit tests and docstring doctests are included when computing coverage. Running: ```bash -hatch env run --env test.py3.12-optional run-coverage +just coverage ``` will automatically run the test suite with coverage and produce an XML coverage report. This should be 100% before code can be accepted into the main code base. @@ -196,7 +219,7 @@ will automatically run the test suite with coverage and produce an XML coverage You can also generate an HTML coverage report by running: ```bash -hatch env run --env test.py3.12-optional run-coverage-html +just coverage-html ``` When submitting a pull request, coverage will also be collected across all supported Python versions via the Codecov service, and will be reported back within the pull request. Codecov coverage must also be 100% before code can be accepted. @@ -210,15 +233,17 @@ Zarr uses mkdocs for documentation, hosted on readthedocs.org. Documentation is The documentation can be built locally by running: ```bash -hatch --env docs run build +just docs-build ``` +`just docs-check` also runs the documentation source checks used in CI. + The resulting built documentation will be available in the `site` folder. -Hatch can also be used to serve continuously updating version of the documentation during development at [http://127.0.0.1:8000/](http://127.0.0.1:8000/). This can be done by running: +`just docs-serve` serves a continuously updating version of the documentation during development at [http://127.0.0.1:8000/](http://127.0.0.1:8000/). This can be done by running: ```bash -hatch --env docs run serve +just docs-serve ``` #### Adding executable code blocks in the documentation @@ -320,10 +345,10 @@ Sometimes, you may want the documentation to build quicker. You can disable code ### Changelog -zarr-python uses [towncrier](https://towncrier.readthedocs.io/en/stable/tutorial.html) to manage release notes. Most pull requests should include at least one news fragment describing the changes. To add a release note, you'll need the GitHub issue or pull request number and the type of your change (`feature`, `bugfix`, `doc`, `removal`, `misc`). With that, run `towncrier create` with your development environment, which will prompt you for the issue number, change type, and the news text: +zarr-python uses [towncrier](https://towncrier.readthedocs.io/en/stable/tutorial.html) to manage release notes. Most pull requests should include at least one news fragment describing the changes. To add a release note, you'll need the GitHub issue or pull request number and the type of your change (`feature`, `bugfix`, `doc`, `removal`, `misc`). With that, run `just changelog`, which will prompt you for the issue number, change type, and the news text: ```bash -towncrier create +just changelog ``` Alternatively, you can manually create the files in the `changes` directory using the naming convention `{issue-number}.{change-type}.md`. @@ -437,6 +462,18 @@ Features in `zarr.experimental` carry no stability guarantees. They may be chang Zarr uses [pytest-benchmark](https://pytest-benchmark.readthedocs.io/en/latest/) for running performance benchmarks as part of our test suite. The benchmarks are found in `tests/benchmarks`. By default pytest is configured to run these benchmarks as plain tests (i.e., no benchmarking). To run -a benchmark with timing measurements, use the `--benchmark-enable` when invoking `pytest`. +a benchmark with timing measurements, run `just benchmark`. Pass pytest arguments +to select benchmarks, for example `just benchmark -k test_morton_order`. The benchmarks are run as part of the continuous integration suite through [codspeed](https://app.codspeed.io/zarr-developers/zarr-python). + +## Building distributions and maintaining dependencies + +`just just-check` verifies that the root `Justfile` and each package `justfile` are +formatted the way CI expects; `just --fmt` rewrites them in place if it complains. + +Run `just build` to produce a source distribution and wheel in `dist/`. +Use `just lock-check` to check the dependency lockfile, or `just lock` to update it. +`just typecheck` runs the type checker independently of the other lint hooks. +Preview release notes with `just changelog-draft`; use `just check-changelogs` +to validate fragment names, optionally passing a package's `changes/` directory. diff --git a/docs/release-notes.md b/docs/release-notes.md index 3b54ea993a..a00cc8b7d6 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -4,6 +4,188 @@ +## 3.4.0 (2026-09-15) + +### Features + +- JSON metadata validation now delegates to ``msgspec.convert`` for the type + coercions it supports (``Literal`` membership, ``int`` / ``bool`` strictness, + list-to-tuple), replacing the per-field hand-written ``parse_*`` logic. + User-defined attributes retain their existing JSON handling. + A latent generator-exhaustion bug in + ``parse_storage_transformers`` is also fixed. See #3285. + + As a result some metadata inputs are now parsed more strictly. The previous + per-field checks compared values with ``==``, which accepts any numerically + equal object, so a float such as ``2.0`` was accepted as ``zarr_format``; it is + now rejected because it is not an ``int``. Booleans are likewise no longer + accepted where an ``int`` is expected, since ``bool`` is an ``int`` subclass. + Metadata that conforms to the Zarr specification is unaffected. ([#4063](https://github.com/zarr-developers/zarr-python/pull/4063)) + +- `zarr.registry.get_codec_class` now raises `zarr.errors.UnknownCodecError` instead of `KeyError` + when no implementation is registered for a codec, and `zarr.core.config.BadConfigError` instead of + `KeyError` when the implementation named in `config["codecs"][name]` is not registered. + `zarr.registry.get_numcodec` raises `UnknownCodecError` instead of the `ValueError` numcodecs + raises for an unregistered Zarr format 2 codec id (`numcodecs.errors.UnknownCodecError` on + numcodecs 0.15.1 and later). All of these are subclasses of `ValueError`, so `except ValueError` + is unaffected, but `except KeyError` and `except numcodecs.errors.UnknownCodecError` are. + + These errors now name Python packages known to provide the codec, so that a user who cannot read + an array learns what to install: + + ```text + An implementation for codec 'wavpack' is not available. Register one explicitly using the codec + registry (see ...), or install a Python package that registers a codec implementation with + numcodecs. Known packages supporting this codec: wavpack-numcodecs. + ``` + + The tables covering this live in `src/zarr/registry.py`, one per Zarr format, and include the + codecs `numcodecs` gates behind its own optional dependencies (`zfpy`, `pcodec`, `crc32c`, + `msgpack2`). Codec authors can add their published package to them. + + A codec whose `from_dict` raises `KeyError` on a malformed configuration now surfaces as + `zarr.errors.MetadataValidationError` naming the codec and the missing key. Previously it was + reported as `UnknownCodecError: Unknown codec: ''`, presenting a configuration + key as though it were a codec name, and on the `zarr.open` path a bare `KeyError` could be + swallowed by the array-then-group fallback and reported as an unrelated group error. + + `zarr.errors.UnknownCodecError` is now exported from `zarr.errors`. ([#4277](https://github.com/zarr-developers/zarr-python/pull/4277)) + +- `zarr.create_array`, `Group.create_array`, `zarr.from_array`, and the entry points built on them now accept a numpy array as the `chunks` or `shards` specification, alongside ints, tuples, and numpy integer scalars. This is new for that API: it has never accepted numpy arrays in any 3.x release, because each entry point compared the specification to the `"auto"` or `"keep"` sentinel string before normalizing it, and for a numpy array that comparison raised numpy's ambiguous-truth-value `ValueError`. Those sentinel checks are now guarded so array-like specifications reach the normalizer, bringing this API in line with the legacy `zarr.create` / `zarr.array` / `zarr.open_array` functions, which have accepted numpy arrays since 2.x. ([#4329](https://github.com/zarr-developers/zarr-python/pull/4329)) + +### Bugfixes + +- Array creation is now O(1) in the number of chunks per dimension. Chunk + normalization returns a `ChunkGrid` whose uniform dimensions are stored as a + size + extent pair (`FixedDimension`) instead of being expanded to one entry + per chunk, so creating arrays like + `zarr.create_array(store, shape=(2**62,), chunks=(1,), dtype='int32')` succeeds + instantly instead of raising `ValueError` or allocating gigabytes of memory. + The intermediate `ChunksTuple` representation was removed in the process, and + `ChunksLike` now admits per-dimension specs that mix a bare int (uniform chunk + size) with explicit edge-length sequences, matching what the normalizer and + the rectilinear grid spec already accepted. + This fixes the array-creation half of #4174; the coordinate-selection + allocation reported there is still tracked in that issue (#4172 fixed the + related case of sorted 1-D coordinate selections). ([#4218](https://github.com/zarr-developers/zarr-python/pull/4218)) +- Allow `Group.require_array` to accept a `ZDType` for `dtype`, matching the other array creation methods. Previously an existing array could only be required with a string or NumPy dtype. ([#4189](https://github.com/zarr-developers/zarr-python/pull/4189)) +- Consolidated metadata is now reconstructed independently of the order the keys appear in on disk. Previously, sibling subtrees whose keys were not adjacent in the persisted mapping lost their children, which made nodes unreachable through consolidated metadata -- most visibly for sibling groups whose names differ only by case. ([#4227](https://github.com/zarr-developers/zarr-python/pull/4227)) +- `FsspecStore.from_mapper` and `FsspecStore.from_url` no longer fail when converting a synchronous instance of an async-capable filesystem whose storage options contain objects that cannot be serialized to JSON (e.g. an `azure.identity.DefaultAzureCredential`). The async instance is now constructed from the original filesystem arguments instead of a JSON round-trip. ([#4239](https://github.com/zarr-developers/zarr-python/pull/4239)) +- Numpy integers are accepted as chunk sizes again. Since 3.3.0 a per-dimension chunk size that was a numpy integer (e.g. `chunks=(np.int64(2), np.int64(2))`, as produced by any computed chunk shape) raised `TypeError: 'numpy.int64' object is not iterable`, because the scalar chunk path narrowed on `int` while its caller dispatched on `numbers.Integral`. The same regression had broken numpy arrays as chunk specifications through the legacy `zarr.create` / `zarr.array` / `zarr.open_array` functions, which accepted them in 2.x and 3.2.x; those work again. (`zarr.create_array` and the functions built on it gain numpy-array support separately, in #4329.) A chunk specification that is neither an integer nor iterable now reports the offending value instead of failing with an opaque iteration error. ([#4257](https://github.com/zarr-developers/zarr-python/pull/4257)) +- The `cast_value` codec now requires `cast-value-rs>=0.4.2`. Earlier versions of that backend silently corrupted data when handed an array that was not row-major — the layout the `transpose` codec produces — so a `cast_value` codec next to a `transpose` codec would either write transposed values with no error or fail with `ValueError: Input array must be contiguous`. The minimum version is enforced at runtime as well as in the package metadata, so an environment that already has an older `cast-value-rs` installed now raises `ImportError` when the codec is used, instead of corrupting data. ([#4260](https://github.com/zarr-developers/zarr-python/pull/4260)) +- Accept [universal-pathlib](https://github.com/fsspec/universal_pathlib) `UPath` objects wherever + zarr accepts a `StoreLike` value. A remote `UPath` now creates an `FsspecStore` using the + filesystem and storage options the `UPath` already carries, and a local `UPath` creates a + `LocalStore`, so that `UPath('/data')` and `Path('/data')` behave the same. + + Previously this worked only by accident: in universal-pathlib < 0.3 every `UPath` subclassed + `pathlib.Path` and implemented `__fspath__`, so remote paths were either converted to a URI string + by the caller or wrapped in a `LocalStore` that happened to dispatch through fsspec. Since + universal-pathlib 0.3 remote paths do neither, and passing one raised + `TypeError: Unsupported type for store_like`. + + `FsspecStore.from_upath` also now converts the `UPath`'s filesystem to async mode, instead of + raising `TypeError` for synchronous filesystems and warning for sync-mode instances of async ones. ([#4265](https://github.com/zarr-developers/zarr-python/pull/4265)) + +- Explicit per-chunk size lists now always produce a rectilinear chunk grid, + even when the sizes happen to describe a regular grid (all equal, or all equal + with a smaller trailing chunk). Previously such input was silently collapsed to + a regular grid, which changed resize semantics: a regular grid grows by + extending the uniform pattern, while a rectilinear grid appends a new edge + chunk — the behavior an append-oriented layout like `(168,) * 13 + (24,)` + relies on. The grid kind now follows the input syntax, matching 3.2.x: + scalar chunk sizes (including numpy integers and the `-1` sentinel) produce a + regular grid, nested sequences produce a rectilinear grid. Rectilinear grids + remain gated behind `zarr.config.set({"array.rectilinear_chunks": True})`. + See #4174 for the accompanying O(1) chunk normalization change. + + One consequence for users who never enable rectilinear chunks: because a + nested sequence now always requests a rectilinear grid, a per-dimension + sequence of edge lengths that happens to be uniform — for example the + `((4,), (4,))` or `[[3, 3, 1]]` form that a dask array's `.chunks` attribute + produces — is no longer quietly accepted as a regular grid when the + `array.rectilinear_chunks` option is off. Such input raises + `ValueError: Rectilinear chunk grids are experimental and disabled by default`, + exactly as it did in 3.2.x; the silent acceptance existed only in 3.3.0. Pass + one integer per dimension (e.g. `chunks=(4, 4)`, or a dask array's + `.chunksize`) to request a regular grid. + + `zarr.from_array` with the default `chunks="keep"` / `shards="keep"` now + reproduces the source's stored grid exactly: a rectilinear grid is passed + through in O(number of dimensions), with uniform dimensions keeping their + bare-int shorthand; sharding under a rectilinear shard grid is preserved + instead of being silently dropped; and the default `write_data=True` copy + works for every grid kind. `Array.chunks` is now defined for any sharded array + (the inner chunks of a shard are always regular), and for sharded arrays with + a rectilinear shard grid `Array.info` no longer raises — it reports the shard + shape as `` — while `Array.nchunks_initialized` counts the chunks of + each initialized shard individually instead of raising. + + Apart from the nested-sequence input form noted above, everything described + here concerns rectilinear chunk grids, which remain an experimental feature + gated behind `zarr.config.set({"array.rectilinear_chunks": True})`; arrays + with regular chunk grids are unaffected. ([#4218](https://github.com/zarr-developers/zarr-python/pull/4218)) + +- A `scale_offset` codec configured with a string-valued zero scale is now rejected. `scale` accepts strings, and no string is ever equal to `0`, so `"0"`, `"0.0"` and the hex form `"0x0000000000000000"` skipped the "scale must be non-zero" check that the numeric `0` triggers. On float data types the array was created, every chunk was written as zero and read back as `nan` with no error, and the zero scale was persisted to the metadata so reopening the store reproduced it; on integer data types the codec raised `ZeroDivisionError` instead of `ValueError`. The check now runs on the parsed scalar rather than the value as supplied. ([#4279](https://github.com/zarr-developers/zarr-python/pull/4279)) +- Fixed a `ValueError` when setting an orthogonal selection on a sharded array where more than one dimension is indexed by an array. The sharding codec re-derives an indexer from the chunk selection it is handed, which turns such a selection into a coordinate selection addressing the value buffer flat, so the write failed on a shape mismatch. Both partial-encode paths are fixed, so the write works under either codec pipeline. ([#4284](https://github.com/zarr-developers/zarr-python/pull/4284)) +- Fixed integer array indexing with unsigned index dtypes. An unsorted index such as + `np.array([3, 0], dtype="uint8")` spanning more than one chunk raised `IndexError`, because + the order check used `np.diff`, which wraps on unsigned dtypes and misclassified a + descending selection as increasing. Separately, a `uint64` index raised `IndexError` on both + `array[...]` and `array.vindex[...]` — sorted or not — because `uint64` promotes to + `float64` against a signed chunk offset. Index arrays are now cast to `intp`. + + Unsigned indices are bounds-checked before this conversion, so values such as + `np.uint64(2**64 - 1)` are rejected rather than wrapping to a negative index and + reading or overwriting an element at the end of the array. + + Negative-index normalization copies indices before modifying them, preserving + caller-owned arrays and supporting read-only index arrays. Reusing one index + array across axes of different lengths now normalizes each axis independently. ([#4286](https://github.com/zarr-developers/zarr-python/pull/4286)) + +- `zarr.from_array` now defaults to the fill value and the attributes of the source array. Previously both were silently discarded: the array was created with the data type's default scalar and no attributes. + + An explicit `fill_value=None` now selects the data type's default scalar (Zarr format 3) or a null fill value (Zarr format 2), consistently with `create_array`, and an empty `attributes` dict creates the array with no attributes. ([#4288](https://github.com/zarr-developers/zarr-python/pull/4288)) + +- Fixed an infinite loop when creating a 0-dimensional array with `shards="auto"` while the `array.target_shard_size_bytes` config option is set. Such arrays now resolve to `shards=()`, matching the behavior when no shard size target is configured. ([#4305](https://github.com/zarr-developers/zarr-python/pull/4305)) +- Fixed `chunks=-1` on a zero-length axis resolving to an invalid chunk size of 0, which caused a `ValueError`, `ZeroDivisionError`, or infinite loop depending on the sharding configuration. Such axes now get chunk size 1, matching `chunks="auto"`. ([#4307](https://github.com/zarr-developers/zarr-python/pull/4307)) +- Fixed a `ValueError` when setting an orthogonal selection on a sharded array that mixes an integer index with two or more array indices, such as `a.oindex[[3, 1, 2], 1, [0, 2]] = value`. The fix for the array-only case in #4284 reshaped the value only when its shape matched the coordinate selection exactly; the sharding codec now also ravels a value that is the selection's shape minus the integer-indexed axes. Values of any other rank are left alone, so a write that is invalid on an unsharded array fails the same way on a sharded one. Both partial-encode paths share one helper for this. ([#4316](https://github.com/zarr-developers/zarr-python/pull/4316)) +- Opening a Zarr format 3 array whose codec's ``from_dict`` raised a ``KeyError`` with no arguments used to fail with an unrelated ``IndexError: tuple index out of range`` while formatting the error message. Because that ``IndexError`` is not a ``ValueError``, it also escaped the array-then-group fallback in ``zarr.open`` and broke group operations such as ``Group.members()`` and ``"child" in group`` when any child array used such a codec. The ``KeyError`` is now always reported as a ``MetadataValidationError`` naming the codec, with the offending key included only when the ``KeyError`` carried one. ([#4324](https://github.com/zarr-developers/zarr-python/pull/4324)) +- `zarr.from_array` now deep-copies the source array's attributes instead of sharing nested dicts and lists between the source and the new array. Previously, mutating a nested attribute on the new array (for example ``dst.attrs["meta"]["tags"].append(...)``) silently changed the source array's in-memory attributes too. Deeply nested attributes can raise `RecursionError` during the copy even if they can be stored and reopened; the threshold depends on Python's recursion limit and call stack. Pass `attributes={}` to omit inherited attributes. ([#4325](https://github.com/zarr-developers/zarr-python/pull/4325)) +- Fixed `chunks=False` on a zero-length axis resolving to a chunk size of 0, which raised a `ValueError` for Zarr format 3, raised a `ZeroDivisionError` with `shards="auto"`, and silently wrote invalid `chunks` metadata for Zarr format 2. `False` now takes the same path as `chunks=-1`, so such axes get chunk size 1, matching `chunks="auto"`. ([#4328](https://github.com/zarr-developers/zarr-python/pull/4328)) +- `from_array` preserves a Zarr source's explicit data type instead of trying to infer it from its NumPy dtype, allowing variable-length bytes arrays to be copied. ([#4335](https://github.com/zarr-developers/zarr-python/pull/4335)) +- The documentation build and the documentation test suite no longer delete a `data/` directory relative to the current working directory. Two executable docs sessions opened with `shutil.rmtree('data', ignore_errors=True)` to make their examples re-runnable; because executed docs blocks run in the process working directory rather than the docs tree, `mkdocs build -f /mkdocs.yml` or `pytest tests/test_docs.py` started from any directory containing a `data/` folder — a project checkout, or `/` — silently emptied it. The sdist ships `docs/` and `tests/` and `testpaths` collects `docs/user-guide`, so this reached anyone running the shipped test suite, not only contributors. The deletions are gone; the on-disk examples in the quick start, arrays, groups, storage and performance guides now create with `overwrite=True` (or `zarr.save_array(..., mode="w")`), which is also what a reader re-running an example needs, and a new docs test rejects any executed block that calls a filesystem deletion. ([#4339](https://github.com/zarr-developers/zarr-python/pull/4339)) +- Missing Zarr format 3 imagecodecs now name `imagecodecs-zarr` as a known provider, alongside `virtual-tiff` where both packages register the codec. ([#4351](https://github.com/zarr-developers/zarr-python/pull/4351)) +- `LocalStore` now retries the rename that publishes a written file when Windows + reports the destination as transiently busy (`ERROR_ACCESS_DENIED` or + `ERROR_SHARING_VIOLATION`). Replacing a name that was itself replaced moments + earlier intermittently fails this way in a single process, which aborted + otherwise ordinary writes. The retry is bounded to well under a second, is a + no-op off Windows, and never retries the `FileExistsError` that the `exclusive` + path uses to report an existing node. This mitigates but does not fully resolve + #3522: a second process holding the destination open for longer than the retry + budget will still fail. Zarr v2 had the equivalent retry from #698 and it was + not carried over when atomic writes arrived in #3412. ([#4358](https://github.com/zarr-developers/zarr-python/pull/4358)) + +### Improved Documentation + +- Added a Roadmap page to the documentation outlining future plans and intended changes to the library. ([#4149](https://github.com/zarr-developers/zarr-python/pull/4149)) +- Converted remaining reStructuredText-style double-backtick markup to Markdown + single backticks in the docstrings of `zarr.api.asynchronous`, + `zarr.api.synchronous`, `zarr.core.array`, `zarr.registry`, and + `zarr.storage._common`. No functional changes. ([#4193](https://github.com/zarr-developers/zarr-python/pull/4193)) +- Document how to reassign Read the Docs version slugs when publishing a subpackage release. ([#4236](https://github.com/zarr-developers/zarr-python/pull/4236)) +- Added a "Related Projects" page to the documentation listing the companion + packages developed in this repository — `zarr-metadata` and `zarr-indexing` — + and linked it from the landing page. Links to those packages now use the + canonical `https://zarr.readthedocs.io/projects/...` URLs, and each companion + package's documentation links back to the `zarr-python` docs. ([#4247](https://github.com/zarr-developers/zarr-python/pull/4247)) + +### Misc + +- [#4213](https://github.com/zarr-developers/zarr-python/pull/4213), [#4261](https://github.com/zarr-developers/zarr-python/pull/4261), [#4326](https://github.com/zarr-developers/zarr-python/pull/4326), [#4331](https://github.com/zarr-developers/zarr-python/pull/4331) + + ## 3.3.0 (2026-07-30) ### Features @@ -156,7 +338,7 @@ - Clarify the difference between `zarr.load` and `zarr.open` in their docstrings. `load` eagerly reads data into an in-memory array, while `open` returns a lazy `Array` or `Group` backed by the store, with `See Also` cross-references - linking the two. ([#3984](https://github.com/zarr-developers/zarr-python/pull/3984)) + linking the two. ([31817c68](https://github.com/zarr-developers/zarr-python/commit/31817c68)) - Updated the custom dtype example in `examples/custom_dtype/custom_dtype.py` to use only the public API, eliminating all non-public imports, illustrating what users should do. diff --git a/lychee.toml b/lychee.toml index dccb3001dc..9bb9be5caf 100644 --- a/lychee.toml +++ b/lychee.toml @@ -16,7 +16,7 @@ exclude_path = [ # URL patterns to ignore (regex, matched against the full URL). exclude = [ - # Local docs preview server shown in the contributing guide ("hatch run serve"), + # Local docs preview server shown in the contributing guide ("just docs-serve"), # documentation of a command rather than a reachable link. '^https?://0\.0\.0\.0', '^https?://(localhost|127\.0\.0\.1)(:\d+)?', diff --git a/packages/zarr-http-server/README.md b/packages/zarr-http-server/README.md index 53df0d2f50..c68f2c9ed2 100644 --- a/packages/zarr-http-server/README.md +++ b/packages/zarr-http-server/README.md @@ -30,7 +30,7 @@ store = zarr.storage.MemoryStore() array = zarr.create_array(store, shape=(100,), chunks=(10,), dtype="float64") with serve_background(node_app(array)) as server: - print(server.url) # e.g. http://127.0.0.1:8000 + print(server.url) # e.g. http://127.0.0.1:8000 ``` Building an app and running it are separate steps, and either app works with diff --git a/packages/zarr-http-server/justfile b/packages/zarr-http-server/justfile index d65b1b53b7..5daac00ebc 100644 --- a/packages/zarr-http-server/justfile +++ b/packages/zarr-http-server/justfile @@ -10,6 +10,9 @@ # newest release: when ruff 0.16 began selecting BLE001 under the root # config's `B` prefix, this job failed on rules the pinned ruff never enforced, # with no code change to blame. Bump alongside the pre-commit rev. +# Quoted arguments must survive delegation from the root Justfile. +set positional-arguments + ruff_version := "0.16.0" # List available recipes @@ -21,7 +24,7 @@ default: # silently skipping. # Run the test suite; extra args are passed to pytest test *args: - uv run --group test --group examples pytest tests {{ args }} + uv run --group test --group examples pytest tests "$@" # Lint the package sources and tests lint: diff --git a/packages/zarr-indexing/CONTRIBUTING.md b/packages/zarr-indexing/CONTRIBUTING.md index 632e24929f..bcbe51c079 100644 --- a/packages/zarr-indexing/CONTRIBUTING.md +++ b/packages/zarr-indexing/CONTRIBUTING.md @@ -8,16 +8,16 @@ just test # run the test suite (extra args go to pytest) just lint # ruff, same invocation as CI just typecheck # pyright, same invocation as CI just docs-check # strict build of the docs site -just check # all of the above +just check # the checks above plus TensorStore parity just docs-serve # serve the docs site locally ``` -Run them from this directory, or from anywhere in the repository as +Run them from this directory, or from the repository root as `just packages/zarr-indexing/`. -The test recipe runs against the workspace-root environment, because the -chunk-resolution tests exercise this package against `zarr`'s chunk grids and -`zarr` is deliberately not a dependency of this package. +The test recipe layers this package into the repository-root environment. +Chunk-resolution tests use this package’s own grids; the Dask example also +uses `zarr`, which is not a dependency of the base indexing package. ## License diff --git a/packages/zarr-indexing/README.md b/packages/zarr-indexing/README.md index 4328de1cc1..2d75402664 100644 --- a/packages/zarr-indexing/README.md +++ b/packages/zarr-indexing/README.md @@ -5,19 +5,22 @@ Composable, lazy coordinate transforms for Zarr array indexing. Documentation: This package implements TensorStore-inspired index transforms. The core idea: -every indexing operation (slicing, fancy indexing, etc.) produces a coordinate -mapping from user space to storage space. These mappings compose lazily — no -I/O until you explicitly read or write. +each supported indexing operation (slicing, fancy indexing, etc.) produces a coordinate +mapping from user space to storage space. These mappings compose without reading selected source values. `LazyArray` +materializes them on request; the transform algebra itself performs no source I/O. Key types: -- `LazyArray` — wraps a system-memory/basic-indexing source and adds a `.lazy` - accessor: `LazyArray.from_numpy(numpy_array).lazy[10:50, ::2].lazy.oindex[[3, 1, 1], :]` +- `LazyArray` — wraps a system-memory/basic-indexing source with lazy indexing: + `LazyArray.from_numpy(numpy_array)[10:50, ::2].oindex[[3, 1, 1], :]`. + The key's type picks the frame: NumPy keys are positions relative to the + view, while an `IndexDomain` or `IndexTransform` key addresses the view's + absolute domain, which every view keeps. composes a transform and returns a new view without reading data, and `result()` materializes it into owned system memory. `LazyArray(source)` uses - the conservative basic reader; `from_numpy` explicitly selects NumPy's - optimized reader. Device arrays require an explicit custom reader responsible - for transferring values into the supplied system-memory output buffer. + the basic reader; `from_numpy` selects `numpy_reader`, which currently uses + the same slab-and-gather implementation. Device sources that refuse NumPy + conversion need a custom reader to transfer values into the output buffer. - `Reader` — the explicit backend execution boundary: transforms say which values belong in the result, while readers say how a backend obtains them - `IndexDomain` — a rectangular region of integer coordinates @@ -32,9 +35,11 @@ Key types: dimension can depend on the input - `compose` — chain two transforms into one -The package depends only on NumPy and the standard library; it does not import -`zarr`. It is developed in the [zarr-python](https://github.com/zarr-developers/zarr-python) -repository and consumed by `zarr` to resolve array indexing operations. +The base package depends on NumPy and the standard library; its optional testing +module also requires Hypothesis. The package does not import `zarr`. It is developed +in the [zarr-python](https://github.com/zarr-developers/zarr-python) repository, +and its examples include reading Zarr arrays through Dask. Installing it does not +replace Zarr's indexing implementation. ## Installation diff --git a/packages/zarr-indexing/benchmarks/README.md b/packages/zarr-indexing/benchmarks/README.md index 28ef3e6a16..e5d39b1a89 100644 --- a/packages/zarr-indexing/benchmarks/README.md +++ b/packages/zarr-indexing/benchmarks/README.md @@ -22,6 +22,19 @@ whether input arrays and transform construction are included. Repeated local column access measures cache reuse, which trades allocation against retained memory. Bounded coordinate batches avoid constructing the full coordinate array. +The current script reports `peak_mib` as the incremental peak tracked by +`tracemalloc` during a separate invocation after the timed calls. It is not +process RSS or total retained memory. Inputs and most transforms are constructed +before measurement. The case walks construct plans and consume projections +without retaining them; `all_coordinates` materializes the complete coordinate +array. There is no bounded-batch coordinate workload in this script. + +`local_rows` reuses one prepared table: its first timed invocation populates the +local-coordinate cache, while later timed invocations and the allocation probe +reuse that cache. Its median mixes a cold first call with warm calls, and its +reported peak excludes the already-retained cache. A separate fresh-table +measurement would be needed to quantify cold cache construction. + These scripts measure planning rather than codec or storage throughput. Repeat measurements with alternating operation order before interpreting small timing differences. Preserve raw benchmark output as an experiment artifact rather than diff --git a/packages/zarr-indexing/benchmarks/chunk_planning.py b/packages/zarr-indexing/benchmarks/chunk_planning.py index 934d68dcbc..d7eadf248d 100644 --- a/packages/zarr-indexing/benchmarks/chunk_planning.py +++ b/packages/zarr-indexing/benchmarks/chunk_planning.py @@ -25,6 +25,11 @@ def measure(operation: Callable[[], Any], repeats: int) -> dict[str, float]: + """Time repeated calls, then trace one additional call's incremental peak. + + State retained by earlier calls can affect the extra call; this is not a + measurement of process RSS or all memory owned by the operation's inputs. + """ samples = [] for _ in range(repeats): start = time.perf_counter() @@ -84,6 +89,8 @@ def read_local_rows() -> None: for row in range(len(table)): table.local[table.run(row)] + # The first timed call fills table.local; later calls, including the traced + # allocation probe, reuse it. The table and retained cache are not re-created. results["local_rows"] = measure(read_local_rows, args.repeats) part = plan_chunks( IndexTransform.from_shape((100, 100, 100)), diff --git a/packages/zarr-indexing/changes/4222.feature.2.md b/packages/zarr-indexing/changes/4222.feature.2.md index 722f56db99..d89fb98134 100644 --- a/packages/zarr-indexing/changes/4222.feature.2.md +++ b/packages/zarr-indexing/changes/4222.feature.2.md @@ -1,13 +1,13 @@ Added `LazyArray`, which grafts the full NumPy indexing dialect onto any source exposing `shape`, `dtype`, and basic integer/slice `__getitem__` — a chunked -store, an FFI binding, an HTTP endpoint. `view.lazy[...]`, `.lazy.oindex[...]` -and `.lazy.vindex[...]` each compose an `IndexTransform` and return a new view -without reading anything; `result()` materializes. Selections use positional -NumPy semantics (negatives wrap, scalars drop their axis, coordinate arrays keep -order and duplicates), which the new `zarr_indexing.boundary` module translates -into the algebra's literal coordinates. The wrapper describes reads only, and -behaves as a duck array: eager `__getitem__` and `__array__` make it a -`dask.array.from_array` source. +store, an FFI binding, an HTTP endpoint. `view[...]`, `view.oindex[...]` and +`view.vindex[...]` each compose an `IndexTransform` and return a new view +without reading anything; `result()` and `numpy.asarray(view)` materialize. +Selections use positional NumPy semantics (negatives wrap, scalars drop their +axis, coordinate arrays keep order and duplicates), which the new +`zarr_indexing.boundary` module translates into the algebra's literal +coordinates. Consumers that need indexing to produce data, such as +`dask.array.from_array`, wrap a view in `EagerArrayAdapter`. A read is divided along a **partitioning** — discovered from the wrapped array, or chosen with `with_parts` / `with_parts_per_axis` / `unpartitioned`. diff --git a/packages/zarr-indexing/changes/4345.bugfix.md b/packages/zarr-indexing/changes/4345.bugfix.md new file mode 100644 index 0000000000..928114728f --- /dev/null +++ b/packages/zarr-indexing/changes/4345.bugfix.md @@ -0,0 +1 @@ +Reject invalid wire index-array values and unrepresentable normalized bounds/ranks, and raise an explicit error for unsupported intersections sharing an affine and lookup input axis instead of returning incorrect coordinates. Group negative chunk-coordinate tuples without merging distinct chunks. diff --git a/packages/zarr-indexing/changes/4345.doc.md b/packages/zarr-indexing/changes/4345.doc.md new file mode 100644 index 0000000000..562e114e4e --- /dev/null +++ b/packages/zarr-indexing/changes/4345.doc.md @@ -0,0 +1,3 @@ +Correct indexing, reader, serialization, cache, and integration descriptions to match supported behavior; qualify NumPy/TensorStore compatibility and performance claims. + +Describe current contracts in source and test docstrings instead of narrating prior implementations. Clarify that immutable index coordinates do not snapshot source values. diff --git a/packages/zarr-indexing/changes/4346.misc.md b/packages/zarr-indexing/changes/4346.misc.md new file mode 100644 index 0000000000..05886fa863 --- /dev/null +++ b/packages/zarr-indexing/changes/4346.misc.md @@ -0,0 +1 @@ +Expand planner property tests across signed origins and chunk IDs, custom grids, mixed affine and lookup dependencies, duplicate coordinates, and empty domains. Verify exact request coverage and storage mapping with an independent pointwise oracle. diff --git a/packages/zarr-indexing/changes/4347.bugfix.md b/packages/zarr-indexing/changes/4347.bugfix.md new file mode 100644 index 0000000000..b2eea6c071 --- /dev/null +++ b/packages/zarr-indexing/changes/4347.bugfix.md @@ -0,0 +1 @@ +Validate inclusive `index_array_bounds` against all raw index values when loading transforms and output maps from JSON. Accept valid finite and one-sided constraints, and reject out-of-bounds values eagerly before offset, stride, or map simplification. Validated immutable maps need not retain the constraints; message normalization preserves the original bounds. diff --git a/packages/zarr-indexing/changes/4348.bugfix.md b/packages/zarr-indexing/changes/4348.bugfix.md new file mode 100644 index 0000000000..69a7daf9bc --- /dev/null +++ b/packages/zarr-indexing/changes/4348.bugfix.md @@ -0,0 +1 @@ +Delegate LazyArray source tokenization to Dask, honoring its registered normalizers, source hooks, and deterministic-token requirements. Remove local content-hashing and UUID fallbacks. Dask remains optional for indexing and reading. diff --git a/packages/zarr-indexing/changes/4349.feature.md b/packages/zarr-indexing/changes/4349.feature.md new file mode 100644 index 0000000000..7234374987 --- /dev/null +++ b/packages/zarr-indexing/changes/4349.feature.md @@ -0,0 +1 @@ +Make every `LazyArray` read supply a chunk projection, including unpartitioned reads and independently executed partition views. Partition views retain the source grid and full source base shape, so indexing and repartitioning use the same coordinate frame as other views. diff --git a/packages/zarr-indexing/changes/4350.feature.md b/packages/zarr-indexing/changes/4350.feature.md new file mode 100644 index 0000000000..7b740e9712 --- /dev/null +++ b/packages/zarr-indexing/changes/4350.feature.md @@ -0,0 +1,20 @@ +Make `LazyArray` indexing lazy by default: use `view[...]`, `view.oindex[...]`, +and `view.vindex[...]` directly, without a `.lazy` accessor. Iteration yields +lazy views, so arithmetic over iterated elements no longer works on values; +`result()` and NumPy conversion materialize. Add synchronous `write(values)` +and assignment through composed views, and an explicit `EagerArrayAdapter` for +consumers such as Dask that require eager indexing. + +Writes are planned against the source's write grid, discovered from +`write_chunk_sizes` or `chunks`: an affine selection is one basic assignment, +and any other selection reads, updates, and rewrites each touched cell once, +so storage round trips scale with touched chunks rather than selected +elements. A source with no advertised grid is written one element at a time. +Writes bypass the reader, so a caching reader is not invalidated. + +Views keep their literal domain instead of re-zeroing after every selection: +`a[10:20]` has domain `[10, 20)` and `a[10:20][2:5]` has domain `[12, 15)`, +as in TensorStore, while NumPy keys stay positional. An `IndexDomain` key +restricts a view to literal coordinates and an `IndexTransform` key composes +onto it. Box partitions keep the request's coordinates, so a part view's +domain is a sub-domain of its parent's. diff --git a/packages/zarr-indexing/changes/4364.misc.md b/packages/zarr-indexing/changes/4364.misc.md new file mode 100644 index 0000000000..0ae29424bc --- /dev/null +++ b/packages/zarr-indexing/changes/4364.misc.md @@ -0,0 +1,7 @@ +Removed every `assert` statement from the package's runtime code and dropped +the `zarr-indexing` exemption from the repo-wide ruff `S101` rule, so none can +return. Asserts are stripped under `python -O`; the ones here narrowed types +or guarded internal invariants rather than validating input, so most were +restructured away (branching on the map type, carrying the narrowed value in a +local) and the remainder became explicit `RuntimeError`s for states the +public API cannot reach. No user-visible behavior changes. diff --git a/packages/zarr-indexing/docs/api/eager.md b/packages/zarr-indexing/docs/api/eager.md new file mode 100644 index 0000000000..e3b10998d8 --- /dev/null +++ b/packages/zarr-indexing/docs/api/eager.md @@ -0,0 +1,12 @@ +--- +title: Eager adapter +--- + +# Eager adapter + +`LazyArray` indexing returns views. Consumers that require indexing to +produce data, such as `dask.array.from_array`, wrap a view in +`EagerArrayAdapter`: the adapter reads each indexed block through the view's +reader and partitioning, while the view itself keeps lazy indexing. + +::: zarr_indexing.eager diff --git a/packages/zarr-indexing/docs/api/grid.md b/packages/zarr-indexing/docs/api/grid.md index b7c376eb85..012c9df6dc 100644 --- a/packages/zarr-indexing/docs/api/grid.md +++ b/packages/zarr-indexing/docs/api/grid.md @@ -11,8 +11,9 @@ objects whose `shape` is the valid data size and whose `codec_shape` preserves the full codec-buffer size at a regular-grid boundary. `dimension_grids_from_chunks` returns these compact dimensions: integer chunk -shapes become `FixedDimension` instances and explicit per-axis edge sequences -become `VaryingDimension` instances. `DimensionGridLike` remains the narrow +shapes become `FixedDimension` instances and positive per-axis edge sequences +become `VaryingDimension` instances. An empty-axis sequence of zeros (including +an empty sequence) becomes `FixedDimension(size=0, extent=0)`. `DimensionGridLike` remains the narrow protocol used by the chunk planner, while `EdgeDimensionGrid` is kept for explicit edge-based and coordinate-origin examples. diff --git a/packages/zarr-indexing/docs/api/index.md b/packages/zarr-indexing/docs/api/index.md index a5b7163bbd..a5c5498e8d 100644 --- a/packages/zarr-indexing/docs/api/index.md +++ b/packages/zarr-indexing/docs/api/index.md @@ -25,12 +25,11 @@ and the wire format built on top of it. - [`zarr_indexing.domain`](domain.md) — `IndexDomain`, a rectangular region of integer coordinates with an explicit (possibly non-zero) origin - [`zarr_indexing.output_map`](output_map.md) — `ConstantMap`, `DimensionMap`, - and `ArrayMap`: three representations of a set of integer coordinates, one - per storage dimension + and `ArrayMap`: three coordinate mappings that preserve order and duplicates, + one per storage dimension - [`zarr_indexing.transform`](transform.md) — `IndexTransform`, which pairs a domain with output maps, plus the indexing (`[...]`, `.oindex`, `.vindex`), - `intersect`, and `translate` operations, and `selection_to_transform` - transforms into one + `intersect`, `translate`, and `compose` operations, and `selection_to_transform` **Chunk resolution** @@ -49,7 +48,7 @@ and the wire format built on top of it. **Lazy arrays** - [`zarr_indexing.lazy_array`](lazy_array.md) — `LazyArray`, a wrapper for - system-memory/basic-indexing sources that adds a `.lazy` accessor for + system-memory/basic-indexing sources with lazy indexing for TensorStore-style deferred indexing, plus `Partition` and `parts()` / `with_parts()`, which determine the boxes a read is broken into. Device sources require an explicit custom reader that transfers into the supplied diff --git a/packages/zarr-indexing/docs/api/lazy_array.md b/packages/zarr-indexing/docs/api/lazy_array.md index f855511d46..44175476ab 100644 --- a/packages/zarr-indexing/docs/api/lazy_array.md +++ b/packages/zarr-indexing/docs/api/lazy_array.md @@ -2,26 +2,46 @@ title: lazy_array --- -`LazyArray.lazy[...]` is metadata-only: every derived view keeps the same +`LazyArray[...]` is metadata-only: every derived view keeps the same reader and composes its transform without reading data. `result()` allocates owned system memory, then calls that reader once for each projected part. Rectangular parts write directly into their final slices; advanced placement may first use an owned dense temporary. `LazyArray(source)` assumes only basic -indexing, while `LazyArray.from_numpy(array)` explicitly selects NumPy's -optimized reader. +indexing, while `LazyArray.from_numpy(array)` selects `numpy_reader`. Both +currently use the same slab-and-gather implementation. The built-in readers lower through NumPy system memory and support sources -whose basic reads can be converted there. They do not implicitly transfer -device arrays; a device source needs an explicit custom reader that transfers -into the supplied system-memory output. Derived views and parts share their +whose basic reads can be converted there. A device source that refuses NumPy +conversion needs a custom reader that transfers into the output buffer. Derived views and parts share their reader and part views may be materialized concurrently, so stateful readers must synchronize their own mutable state. -Every public `Partition.view.transform` directly maps that view's zero-origin +The key's type picks the frame: NumPy keys are relative positions, `IndexDomain` +and `IndexTransform` keys are absolute, and every view keeps its absolute +domain. See [the guide](../guide/index.md#the-keys-type-picks-the-frame). + +Every public `Partition.view.transform` directly maps that view's own domain coordinates into its raw `Partition.view.array`, including for non-first -partitions. `Partition.projection.chunk_transform` intentionally stays local to -the selected chunk. During materialization the reader receives both frames in -one `ReadContext`: the public global transform in `context.transform` and the -same local plan in `context.projection`. +partitions; a box part's domain is a sub-domain of the parent view's literal +domain. `Partition.projection.chunk_transform` intentionally stays local to +the selected chunk. During parent materialization (`view.result(parts=parts)`) the reader receives +both frames in one `ReadContext`: the public global transform in `context.transform` and the +local plan in `context.projection`. Every view retains the source grid and plans +its reads, so `part.view.result()` also supplies both frames. Its projection's +result placement is relative to that part view, rather than the parent output. +Further indexing and repartitioning use the same source-global coordinate frame. +Even `unpartitioned()` reads carry a projection for the single source-wide cell. + +`view[key]`, `view.oindex[key]`, and `view.vindex[key]` return lazy views, +and iteration yields lazy first-axis views. `result()` and `numpy.asarray(view)` +materialize values. `view.write(values)` synchronously writes to the original +source through the composed transform and returns `None`; `view[key] = values` +writes a selected sub-view. Writes require a writable source. + +For consumers requiring eager indexing, import `EagerArrayAdapter` from +`zarr_indexing` and wrap the view. The adapter delegates shape, rank, dtype, +NumPy conversion, and tokenization to the view, but materializes each +`adapter[key]`. Use it with `dask.array.from_array`; direct lazy indexing +is not a reliable Dask block-read interface. ::: zarr_indexing.lazy_array diff --git a/packages/zarr-indexing/docs/api/reader.md b/packages/zarr-indexing/docs/api/reader.md index 39129c4d47..6cb49afacd 100644 --- a/packages/zarr-indexing/docs/api/reader.md +++ b/packages/zarr-indexing/docs/api/reader.md @@ -11,9 +11,14 @@ ownership. `Reader.read_into(source, context, out)` receives a `ReadContext` whose `transform` maps zero-origin output-buffer coordinates to global coordinates in -`source`, with `context.transform.domain.shape == out.shape`. Its optional -`projection` is the existing plan for a partitioned read. The projection's -`chunk_transform` remains chunk-local, its `cell_transform` describes result +`source`, with `context.transform.domain.shape == out.shape`. A view's +transform keeps its literal domain; `ReadContext` re-bases it to origin zero on +construction, so readers never see a view's coordinates. Its optional +`projection` describes one planned read. `LazyArray.result()` always supplies +it, including for partition views and unpartitioned reads. Direct callers of +the reader protocol may omit it when their reader supports that. The projection's +`chunk_transform` remains chunk-local, its `cell_transform` places cells in the +zero-origin result buffer of the view that planned the read, which is what result placement, and its `chunk_domain` describes the grid cell. The global read transform and the projection's chunk transform deliberately use different coordinate frames. @@ -34,7 +39,7 @@ class RecordingReader: self.calls = [] def read_into(self, source, context, out, /): - self.calls.append((source, context, out)) + self.calls.append((source, context, out.shape, out.dtype)) self.inner.read_into(source, context, out) @@ -44,7 +49,8 @@ view = LazyArray.from_numpy(array).with_reader(outer) values = view.result() ``` -Both wrappers observe the same three objects, in outer-to-inner order. This +Both wrappers observe the same arguments, in outer-to-inner order, and log +output metadata without retaining the output buffer. This delegation pattern supports policies such as logging and caching without library-defined wrapper primitives. diff --git a/packages/zarr-indexing/docs/api/writer.md b/packages/zarr-indexing/docs/api/writer.md new file mode 100644 index 0000000000..a8eb365954 --- /dev/null +++ b/packages/zarr-indexing/docs/api/writer.md @@ -0,0 +1,16 @@ +--- +title: Writers +--- + +# Writers + +`LazyArray.write(values)` and assignment through a view are implemented by +`write_into`, which writes through a transform using only basic integer/slice +assignment on the source. Independent affine selections become one basic +assignment. Other selections are scattered against the source's write grid: +each touched cell is read once, updated in memory, and written back, so +storage round trips are bounded by touched cells rather than selected +elements. NumPy sources receive one fancy assignment instead, and a source +that advertises no grid is written one element at a time without reading. + +::: zarr_indexing.writer diff --git a/packages/zarr-indexing/docs/design-notes.md b/packages/zarr-indexing/docs/design-notes.md index 59eef4dacd..9efed99769 100644 --- a/packages/zarr-indexing/docs/design-notes.md +++ b/packages/zarr-indexing/docs/design-notes.md @@ -16,7 +16,7 @@ visual guide owns the mechanics of ## Relationship to TensorStore The core is [TensorStore's](https://google.github.io/tensorstore/index_space.html) -index-transform model, reimplemented in Python against NumPy. The visual guide +index-transform model, implemented here in Python against NumPy. The visual guide introduces the shared model in [Coordinates are addresses](guide/index.md#coordinates-are-addresses) and [Lazy views compose](guide/index.md#lazy-views-compose); the comparison here is @@ -28,13 +28,14 @@ about the deliberately matching semantics: - **Slice semantics.** Slice bounds are literal domain coordinates: no clamping, no negative wrapping, non-empty intervals must be contained in the domain, and a strided slice's domain origin is `trunc(start/step)` rounded - toward zero. Every one of those rules was executed against tensorstore 0.1.84 - and is pinned in `tests/test_tensorstore_parity.py`. -- **The wire format.** A canonical [ndsel](ndsel.md) transform body is, - field-for-field, a TensorStore `IndexTransform` minus the `kind` - discriminator, and `tests/test_ndsel_tensorstore.py` loads our bodies into - `tensorstore.IndexTransform(json=...)` and round-trips them back through our - engine layer. + toward zero. `tests/test_tensorstore_parity.py` compares the enumerated cases with + TensorStore when that optional dependency is installed. +- **The wire format.** [ndsel](ndsel.md) uses TensorStore's domain and + output-map field names for transform bodies, with an additional `kind` + discriminator. `tests/test_ndsel_tensorstore.py` checks interoperability for + the tested cases. Their validation rules differ, and loading and + re-emitting a message through the engine can normalize or discard metadata; + see [lowering to a transform](ndsel.md#lowering-to-a-transform). - **Chunk partitioning.** Both factor a transform over a grid before visiting any cell, rather than intersecting the whole transform with each chunk. TensorStore's `IndexTransformGridPartition` holds strided sets and index @@ -47,17 +48,19 @@ about the deliberately matching semantics: components within a mixed request. Both derive the per-chunk transforms from the partition ([the guide](guide/index.md#a-plan-is-a-product-of-per-axis-tables) shows the tables). TensorStore keeps strided sets implicit, while this - library materializes their per-axis rows for vectorized consumers. Diagonals are rejected here; supporting them needs a strided set per - *input* dimension spanning every storage axis that reads it, TensorStore's - representation. + library materializes their per-axis rows for vectorized consumers. Pure + affine diagonals need grouping by input dimension; mixed affine/index-array + dependencies need joint partitioning. TensorStore classifies a connected + component containing index-array edges as an index-array set + ([source](https://github.com/google/tensorstore/blob/66b2ce5290fa2ec5c8019682391421062ce767a2/tensorstore/internal/grid_partition.h#L58-L67)). Four deliberate differences: | | TensorStore | `zarr-indexing` | | --- | --- | --- | -| Dialect | One strict dialect everywhere: literal coordinates, no negative wrapping | The algebra keeps that dialect; each public boundary picks its own. [`LazyArray`](api/lazy_array.md) speaks positional NumPy, `zarr.Array.lazy` speaks literal. [`zarr_indexing.boundary`](api/boundary.md) is the translation | +| Dialect | Coordinate indices are literal; negative coordinates do not wrap | The algebra keeps that dialect; each public boundary picks its own. [`LazyArray`](api/lazy_array.md) speaks positional NumPy, `IndexTransform` speaks literal. [`zarr_indexing.boundary`](api/boundary.md) is the translation | | Scheduling | An internal C++ scheduler owns concurrency and chunk ordering | [`parts()`](api/lazy_array.md) exposes the partition structure so the caller's own scheduler — dask, a thread pool, a task queue — drives it | -| Wire format | Implementation-defined JSON, specified by what the implementation accepts | [ndsel](ndsel.md) is spec-first, with a vendored language-agnostic conformance corpus every implementation runs | +| Wire format | [Documented JSON schema](https://google.github.io/tensorstore/index_space.html#index-transform) | [ndsel](ndsel.md) is spec-first, with a vendored conformance corpus exercised by this implementation | | Backends | A driver ecosystem (zarr, N5, neuroglancer, GCS, …) built into the library | No drivers. The default reader needs `shape`, `dtype`, basic integer/slice indexing, and selected slabs convertible to NumPy system memory; other backends use explicit custom readers. A device reader owns transfer into the supplied system-memory output | The mechanics of a @@ -74,19 +77,18 @@ and caller-supplied grid; it does not own reads, writes, buffers, locks, or scheduling. Zarr can therefore plan reads against an inner codec-chunk grid and writes against an atomic shard grid; napari or dask can turn the same projections into tasks without putting a dask dependency in this package. -`coverage` is relative to that selected grid: `full` proves a blind replacement -safe, `partial` proves it is not, and `unknown` conservatively covers fancy -selections whose duplicates would require additional work to classify. - -The comparison also runs the other way. TensorStore is a mature, heavily -optimized C++ system whose performance this library cannot approach. Independent strided planning -here is per axis, but each materialized `ChunkProjection` is -Python-level bookkeeping over NumPy — two domains, two transforms and the -projection itself — so the per-part overhead of the object view is -significant; a consumer that reads the partition's tables directly pays no -per-chunk object construction. This library is small and depends on nothing beyond -NumPy, so the algebra can be adopted by a Python project that wants the model -without the C++ runtime. +`coverage` describes selection coverage relative to that grid. A `full` +classification can help a writer avoid reading old values, but does not by +itself prove that a write is safe: encoding requirements, conflicts, duplicate +semantics, and concurrency remain consumer responsibilities. `unknown` means +the planner has not established complete or partial coverage. + +This implementation performs Python-level bookkeeping over NumPy. This page +provides no benchmark establishing a general performance ordering against +TensorStore; costs depend on the selection and execution backend. + +The partition tables can be consumed without constructing a `ChunkProjection` +for each chunk. Materializing projections adds Python object construction. ## Bounding-box selections vs query selections @@ -102,11 +104,11 @@ and the coordinates it touches form a regular lattice. Basic indexing produces one, and composing basic indexing with basic indexing keeps one. **A query** is a transform with at least one `ArrayMap` — an explicit lookup -table of coordinates. It costs `O(n)` to store, it has no locality (the -coordinates may repeat, reverse, or scatter arbitrarily), and intersecting it -with a region means scanning it. `oindex`, `vindex`, and boolean masks all -produce one, and once an axis is a query, subsequent basic indexing cannot make -it a box again. A second query composes onto any axis of an existing one — +table of coordinates. Its stored coordinate arrays cost space proportional to their stored size. +Coordinates may repeat or scatter, but can also be contiguous and local. +Current query-resolution paths inspect these arrays. Fancy indexing can produce +a query, but singleton or constant selections can collapse to `ConstantMap`; +subsequent indexing can therefore make a query affine again. A second query composes onto any axis of an existing one — including the axes it merely broadcasts along — by evaluating the existing lookup tables at the new coordinates. @@ -116,9 +118,10 @@ planning and materialization. [ndsel](ndsel.md) encodes the same split in its message kinds: `point`, `box`, and `slice` desugar to constant and affine output maps and are always boxes; -`points` desugars to `index_array` maps, and a `transform` body is a box -exactly when none of its output maps carries an `index_array`. A consumer can -therefore classify a selection off the wire without materializing anything: +`points` desugars to `index_array` maps. A transform without index-array maps +is a box in the engine’s structural classification. Loading can further +simplify degenerate index arrays to constants, so an arbitrary incoming body +with `index_array` fields need not remain a query. For example: ```python from zarr_indexing import IndexTransform @@ -134,16 +137,13 @@ gather.to_json()["output"][0] # 'index_array_bounds': ['-inf', '+inf']} ``` -The distinction matters to consumers of a selection. A box can be tiled into -rectangular dask chunks or passed to a viewer or tile server that only accepts -rectangles; a query cannot, and has to be resolved into a gather. A box can also -be served as a single strided slab read, but the read has to be strided: reading -its bounding box and discarding the rest transfers proportionally more data as -soon as any stride exceeds 1. The two also behave differently under -partitioning: a box touches a regularly-spaced run of parts, in increasing -order, each at most once — a stride larger than a part's extent skips parts -outright, so the run is not contiguous — while a query can touch any subset of -them, in any order, more than once. +The representation helps a consumer choose a lowering strategy. Independent +affine axes can often be read with slices plus reversal, permutation, or +broadcasting. Arbitrary affine maps can also express diagonals, so the absence +of `ArrayMap` alone is not proof of a rectangular slab. Queries may be lowered +through gathers or covers, and can sometimes simplify to slices. Chunk plans +group selected coordinates by chunk while preserving result placement; repeated +coordinates do not imply repeated visits to the same chunk. [`LazyArray`](api/lazy_array.md) exposes the category directly: @@ -157,13 +157,13 @@ arr = zarr.create_array({}, shape=(100, 80), chunks=(30, 40), dtype="int32") arr[:] = np.arange(8000).reshape(100, 80) lazy = LazyArray(arr) -slab = lazy.lazy[10:50, ::4] +slab = lazy[10:50, ::4] slab.is_box # True slab.bounding_box() # ((10, 50), (0, 77)) slab.strides() # (1, 4) slab.shape # (40, 20) -gather = lazy.lazy.oindex[[90, 3, 3], :] +gather = lazy.oindex[[90, 3, 3], :] gather.is_box # False gather.bounding_box() # ((3, 91), (0, 80)) gather.strides() # None @@ -173,16 +173,20 @@ gather.shape # (3, 80) `bounding_box()` is defined for both: it is the hull, the smallest interval per storage dimension containing every coordinate the selection reaches. `strides()` is defined only for a box and gives the step per dimension. -Together the two describe a box selection completely. - -Both are needed, because a box is dense in its hull only when every stride is -1. The slab above spans a 40x77 hull over the 40x20 cells it selects, so a +These summaries omit traversal direction, input-axis correspondence, and +result layout. For example, forward and reversed views have identical bounds +and stride magnitudes but different ordered results. Use the transform for the +complete selection. + +For independent axes with multiple selected coordinates, a stride magnitude +greater than one leaves gaps in the hull. Singleton axes are an exception, +and a query can also cover every cell of its hull. The slab above spans a 40x77 hull over the 40x20 cells it selects, so a consumer that issued one rectangular read of the hull and discarded the rest would transfer 3.85x the data. A query's hull is looser still and carries no stride at all: 88 rows of hull over three selected rows. An empty *box* touches no coordinate to report an interval around, so `bounding_box()` is `None` while `strides()` still answers — the step is a property of the selection's shape, not -of the region it reaches. Only a query returns `None` from both. +of the region it reaches. An empty query returns `None` from both. There is deliberately no separate `BoxView` type today. A statically-typed rectangular-only view is a plausible next step, but it should be introduced by @@ -248,8 +252,8 @@ could accept and finishing the rest elsewhere. A reader lowers the complete transform and can compose through delegation instead. This resembles [zarrita.js store extensions](https://zarrita.dev/packages/zarrita.html), where storage-specific behavior is an explicit extension point rather than an -inferred array capability. The implementation remains independently authored: -no code is shared with TensorStore, xarray, or zarrita.js. +inferred array capability. This is an architectural analogy, not a claim of API or implementation +compatibility. ## Current scope @@ -259,43 +263,41 @@ than a silently empty selection. One consequence: a negative step normally produces a negative domain origin. Reversing a length-20 zero-origin axis gives the domain `[-19, 1)`, because the result stays anchored to the source coordinate frame and a reversing map traverses that frame backwards. `LazyArray` -re-bases every view to origin 0, so the positional dialect never exposes it; a -caller working with `IndexTransform` directly will see it, and re-bases -explicitly with `translate_domain_to` for NumPy-shaped coordinates. +views keep that literal domain, so `view.transform.domain` shows it; positional +keys are normalized against the domain's origin, so the NumPy dialect never +requires typing it. A caller wanting zero-origin coordinates re-bases +explicitly with `translate_domain_to`. -Fancy selections compose without restriction: a second `oindex`/`vindex`/mask +Supported fancy selections compose across already-fancy views: a second `oindex`/`vindex`/mask step may land on any axis of an already-fancy view, including axes an existing index array merely broadcasts along, so -`lazy.oindex[[2, 0], :].lazy.oindex[:, [1, 3]]` selects the outer product it +`lazy.oindex[[2, 0], :].oindex[:, [1, 3]]` selects the outer product it spells. An array-carrying transform is composed — the new selection is applied to an identity transform over the current domain and chained on with `compose`, which evaluates the existing lookup tables at the new coordinates — rather than rewritten in place. Resolution classifies the result by structure (`index_array_structure`): pure per-axis outer products keep the orthogonal resolvers, and everything else — correlated maps, mixtures, index arrays -sharing an input axis (a diagonal gather, reachable only by hand-building a -transform) — takes the general reader/intersection path. Chunk planning +sharing an input axis (as in paired vectorized coordinates) — takes the general reader/intersection path. Chunk planning factors index arrays into connected dependency components before flattening, so independent groups do not expand one another. Vectorized selection preserves broadcast singletons to retain those dependencies. -Three limits remain, all intentional and all expected to be lifted: +Some current limits are: -- **Affine diagonals.** A hand-built transform in which two output maps read - one input dimension — two slice maps, or a slice map and an orthogonal index - array — is rejected at planning with `ValueError`; a correlated index array - varying over a dimension a slice map also reads is rejected with - `NotImplementedError`. No selection dialect produces either. Supporting them - needs a strided set per *input* dimension spanning all dependent storage - axes, TensorStore's connected-component representation. *Planned.* +- **Shared affine dependencies.** Planning rejects two affine output maps + sharing an input axis with `ValueError`. An index array sharing a varying + input axis with an affine map takes the general classification and raises + `NotImplementedError`. Pure affine diagonals would need grouping dependent + storage axes by input dimension; mixed components need joint partitioning. - **Finite explicit bounds only.** `IndexDomain` has no implicit or unbounded dimensions; the message layer will normalize a body with `"-inf"`/`"+inf"` bounds, but the engine layer refuses to lower one into a transform. - TensorStore supports both. *Planned.* + TensorStore supports both. - **Labels are carried, not propagated.** `IndexDomain` holds optional dimension labels and the wire format round-trips them, but indexing operations build new domains without them, so a label does not survive a - slice. *Planned.* + slice. ## Selection to chunk operations diff --git a/packages/zarr-indexing/docs/guide/index.md b/packages/zarr-indexing/docs/guide/index.md index 68192f58cb..f227ca1ac4 100644 --- a/packages/zarr-indexing/docs/guide/index.md +++ b/packages/zarr-indexing/docs/guide/index.md @@ -1,6 +1,6 @@ # Visual guide -The whole model in one sentence: indexing through `LazyArray.lazy` builds a +The whole model in one sentence: indexing through `LazyArray` builds a view, chunk planning partitions its coordinates, and `result()` materializes the view. This page follows one familiar NumPy selection, `source[2:5]`, through those stages. @@ -38,7 +38,7 @@ make the correspondence explicit: those result coordinates receive values `12`, `13`, and `14` from source coordinates `2`, `3`, and `4`. The wrapper below gives the same familiar selection a lazy spelling. Indexing -through `.lazy` creates `view`; the last line asks for its values and checks the +with `[...]` creates `view`; the last line asks for its values and checks the observable NumPy result. ```python @@ -105,11 +105,40 @@ different questions: | Surface | Meaning of an integer index | Meaning of `-1` | | --- | --- | --- | | `IndexDomain` and `IndexTransform` | A literal coordinate in the current domain | The actual address `-1`, if the domain contains it | -| `LazyArray.lazy` | A NumPy-style position in the current view | The last position, normalized before it reaches the transform algebra | +| `LazyArray` with a NumPy key | A NumPy-style position in the current view | The last position, normalized against the domain's origin before it reaches the transform algebra | +| `LazyArray` with an `IndexDomain` or `IndexTransform` key | A literal coordinate in the view's domain | The actual address `-1`, if the domain contains it | -`LazyArray` uses positions because it is an array-like wrapper: each derived -view starts at position zero and negative indices wrap exactly as they do in -NumPy. The lower-level domain and transform types keep literal coordinates. +`LazyArray` reads NumPy keys as positions because it is an array-like wrapper: +negative indices wrap exactly as they do in NumPy. The view itself keeps its +literal domain, as a TensorStore view does: `source[10:20]` has domain +`[10, 20)`, and a further `[2:5]` on it has domain `[12, 15)`. A domain or +transform key addresses those literal coordinates directly. + +### The key's type picks the frame {#the-keys-type-picks-the-frame} + +**The type of a key decides whether it is relative or absolute.** This is the +one rule to remember about indexing a `LazyArray`: + +| Key type | Read as | Example | +| --- | --- | --- | +| slice, integer, `...`, `None`, index array, mask | **Relative**: positions in the current view, NumPy-style | `view[2:5]`, `view[-1]`, `view.oindex[[3, 1]]` | +| `IndexDomain` | **Absolute**: coordinates of the view's domain, restricted to a box | `view[IndexDomain((12,), (15,))]` | +| `IndexTransform` | **Absolute**: composed onto the view; the key's domain becomes the new view's domain | `view[IndexTransform.identity(view.transform.domain)[12:15]]` | + +No key type is valid in both readings, so a key never has two meanings. That is +the property pandas lost with value-based `ix` dispatch and removed in favor of +`loc` and `iloc`; TensorStore keeps it by giving each key type one fixed +reading, and so does this wrapper. The only difference from TensorStore is +which reading the NumPy key gets: absolute there, relative here. + +Whichever key produced a view, **its domain is always absolute**: +`view.transform.domain` reports literal coordinates after `view[2:5]` exactly as +it does after `view[IndexDomain(...)]`. The relative reading exists only at the +moment a NumPy key is interpreted, so relative and absolute steps compose freely. + +```python +--8<-- "snippets/key_types.py:key-types" +``` ### A transform points from the request to the source @@ -209,28 +238,35 @@ metadata is ready to inspect: | Available without reading | Value in this example | | --- | --- | | `composed.shape` | `(2,)` | -| `composed.transform` | One transform mapping request `i` to source `3 - i` | +| `composed.transform` | One transform over the literal domain `[-3, -1)`, mapping request `i` to source `-i` | Neither property needs source values. Composition works only on the coordinate description; the assertion's call to `result()` is the first operation in the example that materializes the selected data. !!! warning "Stop here: the materialization boundary" - Indexing through `.lazy[...]` never reads. These do: + Indexing through `[...]` composes a selection without reading source values. + These operations request values: - `result()` - - eager indexing of the wrapper: `view[...]` - - `numpy.asarray(view)`, or passing the view to any NumPy function - (`numpy.add(view, 1)` converts, and therefore materializes, the view) + - `numpy.asarray(view)` and NumPy operations that convert the view + (`numpy.add(view, 1)` does so; `numpy.shape(view)` and `numpy.ndim(view)` + can use metadata without reading values) + + Dask tokenization may also inspect values, depending on the wrapped source + and tokenization path. Python arithmetic such as `view + 1` raises `TypeError` instead: this wrapper defers indexing, not a general compute graph. - Nor does it write. There is no `__setitem__`, so `view[...] = values` - raises `TypeError` too, and a wrapped source needs no `__setitem__` of - its own. A consumer that writes plans the selection with `plan_chunks` - and performs its own read-modify-write, keeping chunk atomicity and - concurrent-writer policy on the backend's side of the boundary. + Iteration yields lazy first-axis views; call `result()` on each to read it. + + `view.write(values)` writes through the composed transform to the original + writable source, synchronously, and returns `None`. `view[key] = values` + writes the selected sub-view in the same way. These calls do not create + futures or transactions. Read-only sources remain usable for reads; + writes require source assignment support. Storage atomicity and concurrent + writer coordination remain the backend's responsibility. ## An index defines a result array {#an-index-defines-a-result-array} @@ -384,8 +420,11 @@ each bundles a sub-view of the request (`.view`), that chunk's projection Within one `Partition`, the frames divide: `Partition.view.transform` is a different, global transform — it maps the part view directly into the raw wrapped source — while only `Partition.projection.chunk_transform` uses -zero-origin chunk-local coordinates. Readers receive both so the global -source address and the local planning frame cannot be confused. +zero-origin chunk-local coordinates. Parent assembly passes both frames to +the reader. Independently scheduled `part.view.result()` calls plan against the +same source grid and supply both frames too. Their result placement is relative +to the part view being read. A partition view can be indexed or repartitioned +like any other view; its base shape remains the full source shape. | Projection field | What its output coordinates mean | | --- | --- | @@ -399,7 +438,7 @@ rank one and source rank two. ### Order and duplicates need the request-side projection -Orthogonal indexing (`.lazy.oindex`) applies each axis's indexer +Orthogonal indexing (`.oindex`) applies each axis's indexer independently, like `numpy.ix_` — an outer product; the [pattern reference](patterns.md) develops the dialects. It can visit source cells in an order that does not match chunk order, and it can visit one diff --git a/packages/zarr-indexing/docs/guide/integrations.md b/packages/zarr-indexing/docs/guide/integrations.md index f787d11f6c..c96dcf91a5 100644 --- a/packages/zarr-indexing/docs/guide/integrations.md +++ b/packages/zarr-indexing/docs/guide/integrations.md @@ -3,8 +3,9 @@ For the complete path from indexing syntax to chunk coordinates, local selectors, and result positions, start with [From a selection to chunk operations](selection-flow.md). -This package supplies indexing plans. It does **not** supply scheduling, -caching, codecs, or async orchestration. A consumer decides when projections +The core package supplies indexing plans and synchronous readers. It does not +provide a general scheduler, codec pipeline, or async execution engine. A +synchronous cache is included as an example. A consumer decides when projections run, how decoded chunks are obtained, and where completed values are retained. An `IndexTransform` says which source values belong in a result; a `Reader` lowers that complete transform for one backend. The reader does not choose @@ -61,21 +62,21 @@ afterward, so accessing a row does not recalculate every point. ## One slab read or many part reads -A backend with its own native subset read — a Rust or C zarr implementation, -a database, an HTTP range endpoint — resolves a **dense box** (`is_box` with -every stride 1) best as a single read: hand it the whole selection and let it -dispatch to chunks, decode in parallel, and partial-decode shards on its own -side of the boundary. Splitting that read along this library's partitioning -only adds round-trips. Every **other** selection — a strided box, an `oindex` -or `vindex` gather — is where the partitioning earns its keep. The **cover** -of a read is the smallest step-1 slab enclosing every coordinate it needs; -partitioned, each part's cover is bounded by that part's box, so a sparse -selection can never force one read of its whole bounding hull (the smallest -rectangle containing every selected coordinate — a thousand rows for the two -of `oindex[[0, 999]]`). +A backend with an efficient native subset operation may benefit from receiving +one complete dense selection so it can choose its own chunk dispatch. Other +backends may benefit from partitioning, including for strided or fancy +selections. The tradeoff depends on the backend, chunk layout, latency, memory, +and selection; a single read is not universally fastest. + +A read's **cover** is the smallest unit-step slab enclosing its coordinates. +With this package's basic readers, partitioning limits each source read to the +part's selected cover. This may reduce over-reading, but a partition that spans +the source can still require the entire hull. Custom readers choose their own +source operations under the reader contract. The composed view carries enough to make that call at materialization time, -and re-partitioning is a pure setter, so the policy is three lines: +and `with_parts()` returns a view with a new partitioning. This example uses +unit strides as a sufficient condition for its independently mapped selections: ```python --8<-- "snippets/integrations.py:dense-box-repartition" @@ -87,20 +88,19 @@ the dense box becomes exactly one backend call. Both regimes go through ### Sources that accept only unit-step slices -The default `basic_reader` pushes strided and descending selections down as -positive-step slices, which reads the minimum but assumes the source accepts -any step. Many backends do not: FFI bindings and range requests often -support nothing but `slice(start, stop, 1)`. Select -[`unit_step_reader`][zarr_indexing.reader.UnitStepReader] for such a source -and every key it receives is an ascending unit-step slice per axis, with +For affine selections, `basic_reader` uses positive-step slices and applies +reversal or layout changes in memory. Fancy selections can require reading a +cover containing unselected values. The source must accept the emitted steps. +Use [`unit_step_reader`][zarr_indexing.reader.UnitStepReader] for a source that +accepts only unit steps. Every key it receives is an ascending unit-step slice per axis, with strides, reversals, and gathers applied to the in-memory block instead: ```python view = LazyArray(source).with_reader(unit_step_reader) ``` -A strided selection then over-reads its cover by the stride factor, which the -partitioning above bounds by one part. +For strided selections, the ratio of cover cells to selected cells depends on +stride, length, and endpoint alignment. Partitioning can reduce that cover. ## napari-like consumer @@ -205,13 +205,12 @@ decoded chunks. Every read delta follows directly from the viewport request: | 4 | `image[1:5, 2]` | `(0, 0)` | `(0, 0)`, `(1, 0)` | The evicted chunk is reloaded while the required ready chunk is retained. | | 5 | `image[3:5, 4:6]` | `(1, 1)` fails; no repeated read; `(1, 1)` succeeds after retry | `(0, 0)`, `(1, 1)` | Failure is retained until explicit retry; the repaired source then returns `[[28, 29], [36, 37]]`. | -Chunks required by an active request are pinned through assembly, so a request -may temporarily span more chunks than the steady-state capacity. Capacity is -counted in decoded chunks—not records or bytes—and eviction occurs only after -all requested values have been placed. Because pinning and materialization use -the same prepared tuple, those lifecycle decisions cannot drift from the parts -that are actually read, and the cache never has to infer or reconstruct a -projection. +The example defers eviction until a successful outermost request finishes, so +it can temporarily exceed capacity during assembly. Capacity counts decoded +chunks, not bytes, event records, or temporary arrays. Failed requests skip +that eviction step. The example assumes an unchanged source and one source/grid +per reader; it does not implement invalidation or synchronization for concurrent +requests. The prepared tuple supplies the projections used for each read. The event log makes the failure boundary equally explicit: @@ -244,3 +243,20 @@ that implementation or reproduce its full worker/GPU lifecycle. · **API:** [API reference](../api/index.md) + +## Dask tokenization + +`LazyArray.__dask_tokenize__()` combines Dask's token for the wrapped source +with the serialized view transform. Dask owns source hashing, registered +normalizers, custom source hooks, and deterministic-token requirements. +Tokenization may read or hash source values. Dask is optional for indexing +and reading, but required when requesting a Dask token. + +The reader and partitioning are omitted because they must preserve values. +Changing a source after graph construction does not update existing Dask keys. + +For consumers that require eager indexing, such as `dask.array.from_array`, +wrap the view in `EagerArrayAdapter` (importable from `zarr_indexing`). The +adapter materializes each indexed block while the wrapped view keeps lazy +indexing, and its token derives from the view's. See Dask's [from_array +documentation](https://docs.dask.org/en/stable/generated/dask.array.from_array.html). diff --git a/packages/zarr-indexing/docs/guide/patterns.md b/packages/zarr-indexing/docs/guide/patterns.md index 9d22c4eb4e..df9a482c3e 100644 --- a/packages/zarr-indexing/docs/guide/patterns.md +++ b/packages/zarr-indexing/docs/guide/patterns.md @@ -302,17 +302,18 @@ so they equal the zero-origin models after `translate_domain_to`: --8<-- "snippets/indexing_patterns.py:indexing-patterns" ``` -`LazyArray` adds nothing to these semantics: it is a regular array-like API -whose `.lazy`, `.lazy.oindex`, and `.lazy.vindex` accessors compile the same -dialects to the same transforms — the only difference is the return type, a -view instead of an array. The test suite holds the wrapper to this matrix. +`LazyArray` exposes the transform machinery through a positional array-like +API. Its `[...]`, `.oindex[...]`, and `.vindex[...]` operations return views and +normalize positions before composition. This boundary differs from the literal +coordinate semantics of `IndexTransform`, as the following table shows. The +executable matrix checks the documented cases, not every possible NumPy expression. ## Positions vs literal coordinates | Surface | Meaning of an integer index | Meaning of `-1` | | --- | --- | --- | | `IndexDomain` and `IndexTransform` | A literal coordinate in the current domain | The address `-1`, when the domain contains it | -| `LazyArray.lazy` | A NumPy-style position in the current view | The last position, normalized before transform composition | +| `LazyArray` | A NumPy-style position in the current view | The last position, normalized before transform composition | The wrapper's three indexing modes all use positions in the current view. Each derived view begins at position zero, while the transform algebra underneath diff --git a/packages/zarr-indexing/docs/guide/selection-flow.md b/packages/zarr-indexing/docs/guide/selection-flow.md index 571baa3869..749a9c36ae 100644 --- a/packages/zarr-indexing/docs/guide/selection-flow.md +++ b/packages/zarr-indexing/docs/guide/selection-flow.md @@ -171,7 +171,7 @@ pairs: | `(0,)` | `[1]` | `[1]` | | `(1,)` | `[2, 2, 0]` | `[0, 2, 3]` | -Chunk `(1,)` is fetched once, but its local value `2` contributes to two result +Chunk `(1,)` is fetched once, but the value at local coordinate `2` contributes to two result positions. Chunk visitation order need not be result order; `out_selection` restores the requested arrangement. @@ -188,6 +188,9 @@ than one. Repeated coverage is not full; fancy coverage remains conservative. Zarr's merge operation can skip a read for a full data-extent write and allocate a fill-valued codec buffer when the selected data is smaller than that buffer. +That shortcut also requires the consumer's expected value layout and order; +complete coverage by a reversal or reordered gather alone does not establish +that the supplied buffer can be copied directly into codec order. Touching every chunk alone does not prove full coverage of each chunk. Planning does not fetch bytes, decode buffers, choose concurrency, or define diff --git a/packages/zarr-indexing/docs/index.md b/packages/zarr-indexing/docs/index.md index 87e5a4b7a1..03537493f2 100644 --- a/packages/zarr-indexing/docs/index.md +++ b/packages/zarr-indexing/docs/index.md @@ -5,8 +5,7 @@ the *declaration* of an array indexing expression from the result of that expres Developed for use in [`zarr`](https://zarr.readthedocs.io). -Inspired by [TensorStore](https://google.github.io/tensorstore/), which pioneered -the approach used here. +Inspired by [TensorStore's index-transform model](https://google.github.io/tensorstore/index_space.html). ## Install @@ -21,15 +20,16 @@ pip install zarr-indexing ## Quickstart -Wrap an array, compose a lazy view through `.lazy`, and call `result()` when +Wrap an array, compose a lazy view with `view[...]`, and call `result()` when you want its values: ```python --8<-- "snippets/canonical_slice.py:landing-quickstart" ``` -Nothing is read until the `result()` call, however many selections are -composed. [Lazy views compose](guide/index.md#lazy-views-compose) shows how +Composing these selections does not read source values; the example reads them +at `result()`. Construction inspects source metadata, and Dask tokenization can +inspect source values. [Lazy views compose](guide/index.md#lazy-views-compose) shows how the chain stays one description, and where the materialization boundary is. ## Learn more diff --git a/packages/zarr-indexing/docs/ndsel.md b/packages/zarr-indexing/docs/ndsel.md index 94971d49bf..3ba13176c1 100644 --- a/packages/zarr-indexing/docs/ndsel.md +++ b/packages/zarr-indexing/docs/ndsel.md @@ -14,11 +14,13 @@ before a transform is serialized. `zarr-indexing` implements ndsel in two layers | Layer | Module | Depends on | Job | | --- | --- | --- | --- | -| Message | [`zarr_indexing.messages`](api/messages.md) | stdlib only | JSON in, canonical JSON out. Validates and desugars. Never rounds, clamps, or drops information. | +| Message | [`zarr_indexing.messages`](api/messages.md) | stdlib only | Validates and desugars JSON, removing redundant constant-map fields. | | Engine | [`zarr_indexing.json`](api/json.md) | NumPy | Lowers a *canonical* body into an in-memory [`IndexTransform`](api/transform.md), and back. | Constraints that only make sense for a real array — finite bounds, index -arrays as `ndarray`s — live in the engine layer and nowhere else. As a result, +arrays as `ndarray`s — are checked during engine lowering. The message layer +also limits input rank to 32 and checks affine input-dimension references. +These checks are stricter than the draft's required message validation. As a result, `messages` normalizes a message with `"-inf"` bounds that `IndexTransform.from_json` refuses to lower. @@ -47,9 +49,13 @@ normalize_ndsel({"kind": "box", "inclusive_min": [10, 5], "shape": [40, 1]}) ``` Normalization is idempotent: re-tag the output with `kind: "transform"` and -normalizing it again returns the same body. Because the canonical body is -field-for-field a TensorStore `IndexTransform` minus `kind`, a normalized -message loads directly into `tensorstore.IndexTransform(json=...)`. +normalizing it again returns the same body. The canonical body uses +TensorStore's `IndexTransform` field vocabulary, but normalization does not +guarantee acceptance by TensorStore. For example, the message layer leaves +index-array content unchecked, whereas TensorStore validates it; TensorStore +also requires unique nonempty labels and restricts finite index values to +`[-(2**62 - 2), 2**62 - 2]`. See +[TensorStore's index-space constraints](https://google.github.io/tensorstore/index_space.html). Both entry points raise [`NdselError`](api/messages.md#zarr_indexing.messages.NdselError), which @@ -64,12 +70,13 @@ Four are shorthands; the fifth is the canonical form itself. | `kind` | Fields | Selects | | --- | --- | --- | | `point` | `coords` | A single element. Normalizes to rank 0 with one `constant` output map per dimension. | -| `box` | `inclusive_min`, one of `exclusive_max` / `inclusive_max` / `shape`, `labels` | A rectangular region. Exactly one upper-bound spelling may appear. | +| `box` | `inclusive_min`, one of `exclusive_max` / `inclusive_max` / `shape`, `labels` | A rectangular region. At most one upper-bound spelling may appear; omission gives implicit positive infinity. | | `slice` | `start`, `stop`, `step`, `labels` | A strided region, one Python-style slice per dimension. | | `points` | `coords` (a list of coordinate rows) | An explicit list of points — the `vindex` case. Normalizes to one `index_array` output map per dimension over a shared rank-1 input domain. | | `transform` | `input_rank`, `input_inclusive_min`, one of the three `input_*` upper bounds, `input_labels`, `output` | The full canonical form. | -Value rules the message layer enforces throughout: every integer is a 64-bit +Value rules for validated fields (excluding verbatim `index_array` payloads +and discarded constant-map fields): every integer is a 64-bit signed value; JSON booleans are **not** integers (Python's `isinstance(True, int)` is guarded against explicitly); the `"-inf"` / `"+inf"` sentinels are legal only in bound positions; and an implicit bound is the @@ -81,17 +88,35 @@ normalization intact. The engine layer converts between canonical bodies and `IndexTransform`s: ```python -from zarr_indexing import IndexTransform +from zarr_indexing import IndexTransform, normalize_ndsel +canonical = normalize_ndsel({"kind": "box", "shape": [2, 3]}) t = IndexTransform.from_json(canonical) -t.to_json() == canonical +assert t.to_json() == canonical ``` `IndexDomain` carries the same pair for a bare domain body, and each output map kind has a `to_json`; `output_index_map_from_json` dispatches the wire's -tagged union back to the right kind. - -Two engine constraints apply here and only here. A canonical body carrying a +structurally discriminated union back to the right kind. Exact JSON equality +in this example is not a general round-trip guarantee: implicit flags are +removed and degenerate array maps are collapsed. + +`index_array_bounds` constrains raw index-array values before the map's offset +and stride are applied. Both `IndexTransform.from_json` and +`output_index_map_from_json` validate every supplied value against the inclusive +bounds when loading. Finite and one-sided bounds are supported; omitted bounds +and `["-inf", "+inf"]` impose no additional constraint. Values outside the +bounds raise `NdselError("invalid_json", ...)`, including in singleton arrays +and zero-stride maps. Empty arrays satisfy any well-formed, ordered bounds. + +Validation is eager: an invalid entry rejects the entire map even if a later +selection would avoid that entry. After validation the engine owns immutable +index coordinates, so it need not retain the bounds; serialization emits +unbounded constraints for non-degenerate maps. Message normalization preserves +the original bounds without checking array contents. This implementation does +not defer bounds errors until individual positions are accessed. + +A canonical body carrying a `"-inf"` or `"+inf"` bound cannot be lowered — an `IndexDomain` addresses a finite array — so `IndexTransform.from_json` raises. And implicit bounds lower *by value*: the `[n]`-bracket flag is a message-layer concern, and the engine @@ -102,21 +127,17 @@ keeps only the integer. ndsel and TensorStore both **reject** an output map that carries both `input_dimension` and `index_array`. The in-memory [`ArrayMap`](api/output_map.md#zarr_indexing.output_map.ArrayMap), though, -records an `input_dimension` to pin the axis an orthogonal (`oindex`) array -varies over. The serializer bridges that gap in both directions: +records its dependency axes in its full-rank array shape, with no +`input_dimension` field: - **On serialize**, a non-degenerate `index_array` map is emitted *without* `input_dimension`. -- **On load**, the in-memory `input_dimension` is reconstructed from the - full-rank array's dependency axes — its non-singleton axes. An array that - solely owns a single non-singleton axis is orthogonal; arrays that share - non-singleton axes, or vary over several, are correlated (`vindex`), and get - `input_dimension = None`. A single 1-D array over a rank-1 domain is - inherently ambiguous between the two flavors and reconstructs as - orthogonal, which is behaviorally identical in that case. - -There is one deliberate exception, and it is the only place a round trip changes -representation rather than preserving it. An all-singleton `index_array` — size +- **On load**, dependency axes are the full-rank array's non-singleton axes. + Maps sharing these axes describe correlated coordinates. The engine also + accepts lower-rank nonempty arrays by prepending singleton axes; that + convenience is not a guarantee of compatibility with other ndsel consumers. + +An all-singleton `index_array` — size 1 — selects the same coordinate regardless of the input, so it is collapsed to a `constant` map on serialize: @@ -137,6 +158,11 @@ The transform is still valid and the output shape is unchanged. A length-1 `oindex` selection therefore round-trips behaviorally (an `ArrayMap` comes back as a `ConstantMap`) rather than by object identity. +Empty index arrays also serialize as constant maps with offset zero: the +empty input domain carries the fact that no coordinates are selected. This +avoids losing trailing shape information in JSON when an array has a leading +zero-length axis. + ## Conformance The package is checked against the language-agnostic ndsel conformance corpus, @@ -145,8 +171,9 @@ vendored unmodified under — one JSON file per message kind plus `errors.json`, with the source commit recorded in `PROVENANCE.md`. Each fixture is either a *success* case (`input` + expected `normalized` body) or an *error* case (`input` + expected -reason code), and an implementation is conformant iff `normalize` reproduces -every one. `tests/test_conformance.py` runs the whole corpus as one +reason code). Matching every fixture establishes corpus conformance; the +fixtures do not prove correctness for every possible input or universal +TensorStore compatibility. `tests/test_conformance.py` runs the whole corpus as one parametrized test per fixture, so a corpus update reports failures fixture by fixture rather than as a single opaque assertion. diff --git a/packages/zarr-indexing/docs/release-notes.md b/packages/zarr-indexing/docs/release-notes.md index 767cd56a84..d0687d8758 100644 --- a/packages/zarr-indexing/docs/release-notes.md +++ b/packages/zarr-indexing/docs/release-notes.md @@ -1,5 +1,6 @@