From 7689b98f6f37ee44d982052587cb17d2ecc641ee Mon Sep 17 00:00:00 2001 From: Bill Little Date: Fri, 11 Sep 2026 00:31:57 +0100 Subject: [PATCH 1/2] [Agentic] Design spec: making merge and concatenate robust, efficient and configurable (#7274) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add design spec for merge/concatenate improvement programme Specifies a twelve-PR programme to make cube merge and concatenate more robust, efficient, configurable, extensible, lazy and lenient, without a rewrite. Each PR is either a behaviour-preserving refactor or a standalone user-visible improvement; together they grow a shared substrate beneath both engines. Includes a confirmed diagnosis and fix for the separability bug open since 2017 (SciTools/iris#2761), verified end-to-end against the reporter's case. Co-Authored-By: Claude Opus 5 * Record design-spec location in AGENTS.md Co-Authored-By: Claude Opus 5 * docs: publish design specs as MyST Markdown Adopt the spec/plan convention used in bjlittle/tephpy. Agreed design specs now live in `docs/src/developers_guide/specs/` and are published as part of the documentation. They are written in MyST Markdown, are living documents revised as a design evolves, and each declares a citation prefix so its sections can be cited from issues and pull requests. Implementation plans live in the sibling `docs/src/developers_guide/plans/`. A plan is point-in-time and frozen once its pull request merges, so plans are tracked in the repository but excluded from the Sphinx build. This requires `myst-parser`, added to `requirements/py*.yml` and inserted into the three lock files following the precedent of #7095 and #7100. Markdown is used only for specs; all other documentation remains reStructuredText. The merge/concatenate design spec moves here from `docs/superpowers/specs/` and is reworked to the new conventions. Co-Authored-By: Claude Opus 5 * docs: add a plan roadmap and decision register to the merge spec The spec described a twelve-PR programme but had nowhere to record how far it had got, which assumptions it rested on, or what it had left open. Restructure §6 from a static table of work into the programme's live progress record: dependencies as a partial order rather than a chain, a status vocabulary, an "Outside the roadmap" list, and a numbered register of assumptions and open decisions - each item owned by the pull request that must resolve it. Add §10 for the questions the programme raises but does not answer, renumbering References to §11. Also exclude "_build" and "_static" from the Sphinx build. myst-parser makes ".md" a source suffix, so the README of each was being read as a document. Co-Authored-By: Claude Opus 5 * docs: record how agentic contributions are pushed and labelled This work is long-running partly because it is also an exercise in earning core-developer buy-in for agentic contributions, so the mechanics of how it reaches them are worth writing down rather than re-deciding each session. Add a "Contribution Workflow" section to AGENTS.md carrying the general rule: branches to the fork, pull requests to a feature branch rather than main, honest "Agentic" labelling, clear attribution, and ask before doing anything outward-facing when unsure. Drop the "Getting Help" link list to stay inside the file's 200-line budget - the links it held are all reachable from CONTRIBUTING.md. The programme-specific parts - base branch "greenfield" and the "Feature: Merge/Concatenate" label - go in the spec's cross-cutting rules instead, where a reviewer can see why these pull requests do not target main. Co-Authored-By: Claude Opus 5 * docs: add changelog fragments for the design spec Co-Authored-By: Claude Opus 5 * docs: rebase the spec baseline onto greenfield The branch was cut from main at c508ac800; greenfield carries #7270 on top of that, which re-solved all three lock files. The lock additions here are re-derived against those fresh locks, and remain minimal - the myst-parser dependency closure and the input_hash, nothing else. The cube.py line references the spec cites are unaffected: #7270's changes are confined to rolling_window, some 4500 lines below the merge and concatenate methods. Co-Authored-By: Claude Opus 5 * Spell out spec section citation runs Give every cited section its own section sign, and separate a run of them with a comma or a solidus rather than a dash. A dashed range such as "5.6-5.7" reads as a single citation carrying a stray suffix, so only its first section is unambiguously identified -- by a reader now, and by any cross-reference tooling later. Records the rule on the Design Specs index page, next to the citation prefix table it belongs with, and applies it to the one occurrence in the merge/concatenate spec. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 --- AGENTS.md | 214 +++--- changelog/7274.dependency.rst | 3 + changelog/7274.doc.rst | 6 + docs/src/conf.py | 17 + .../contributing_getting_involved.rst | 1 + docs/src/developers_guide/plans/README.md | 10 + .../2026-09-10-merge-concatenate-design.md | 683 ++++++++++++++++++ docs/src/developers_guide/specs/index.rst | 52 ++ requirements/locks/py312-linux-64.lock | 4 +- requirements/locks/py313-linux-64.lock | 4 +- requirements/locks/py314-linux-64.lock | 6 +- requirements/py312.yml | 1 + requirements/py313.yml | 1 + requirements/py314.yml | 1 + 14 files changed, 870 insertions(+), 133 deletions(-) create mode 100644 changelog/7274.dependency.rst create mode 100644 changelog/7274.doc.rst create mode 100644 docs/src/developers_guide/plans/README.md create mode 100644 docs/src/developers_guide/specs/2026-09-10-merge-concatenate-design.md create mode 100644 docs/src/developers_guide/specs/index.rst diff --git a/AGENTS.md b/AGENTS.md index 5ad21e3d45..9e8ac0b487 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,10 +7,10 @@ Iris is a Python package for analysing and visualising Earth science data, built around CF-compliant multi-dimensional arrays ("Cubes"). Subdirectory AGENTS.md files take precedence for their subtrees: -- [`changelog/AGENTS.md`](changelog/AGENTS.md) — documentation on changelog +- [`changelog/AGENTS.md`](changelog/AGENTS.md) — changelog fragments - [`docs/AGENTS.md`](docs/AGENTS.md) — documentation-specific rules -- [`lib/iris/tests/AGENTS.md`](lib/iris/tests/AGENTS.md) — test-specific rules - +- [`lib/iris/tests/AGENTS.md`](lib/iris/tests/AGENTS.md) — testing rules; read + this before writing or running any test ## Project Overview @@ -24,87 +24,51 @@ Subdirectory AGENTS.md files take precedence for their subtrees: | **Test runner** | pytest + pytest-xdist (`-n auto`) | | **Env management** | nox + conda | - ### Main source layout ``` -lib/iris/ - cube.py # Core Cube / CubeList data structures - coords.py # DimCoord, AuxCoord, CellMeasure, AncillaryVariable - loading.py # File-loading entry points - analysis/ # Collapse, regrid, statistics, calculus - fileformats/ # NetCDF, PP, GRIB, NIMROD format handlers - io/ # I/O registry and URI handling - common/ # Shared metadata, mixins, resolvers - mesh/ # Unstructured grid (UGRID) support - experimental/ # Unstable / in-progress features - tests/ # All tests (unit/, integration/, graphics/) -changelog/ # changelog fragments -docs/src/ # Sphinx documentation source -benchmarks/ # ASV performance benchmarks -requirements/ # Conda environment specs and lock files +lib/iris/ cube.py, coords.py, loading.py; analysis/ (collapse, regrid, + statistics), fileformats/ (NetCDF, PP, GRIB, NIMROD), io/, + common/ (metadata, mixins, resolvers), mesh/ (UGRID), + experimental/, tests/ (unit/, integration/, graphics/) +changelog/ changelog fragments +docs/src/ Sphinx documentation source +benchmarks/ ASV performance benchmarks +requirements/ conda environment specs and lock files ``` - ## Setup -### Conda environment (recommended) - -Always use a conda environment, reuse the iris-dev conda environment if it -already exists but confirm with the user before installing or removing packages. - -If a package cannot be installed via conda then you can use pip that is in the -conda environment. - -```bash -# Create and activate a development environment -conda env create -f requirements/iris.yml -conda activate iris-dev -pip install --no-build-isolation -e . -``` - -Alternatively, use lock files for exact reproducibility: +Always work in a conda environment. Reuse `iris-dev` if it exists, but confirm +with the user before installing or removing packages; if a package is not on +conda, use the `pip` inside the conda environment. ```bash +# Preferred - exact reproducibility from a lock file. conda create -n iris-dev --file requirements/locks/py314-linux-64.lock +# Alternative - solve fresh: conda env create -f requirements/iris.yml conda activate iris-dev pip install --no-build-isolation -e . ``` - ### Environment variables ```bash # Disable CPU features that can cause SIGILL in some CI environments export NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX" - # Point to iris-test-data for tests that need external data files export OVERRIDE_TEST_DATA_REPOSITORY=/path/to/iris-test-data/test_data - # Override Cartopy cache directory if needed export CARTOPY_CACHE_DIR=~/.local/share/cartopy ``` - -## Testing - -- [`lib/iris/tests/AGENTS.md`](lib/iris/tests/AGENTS.md) — test-specific - - ## Code Style ```bash -# Lint -ruff check lib/iris - -# Auto-fix safe lint issues -ruff check --fix lib/iris - -# Format -ruff format lib/iris - -# Check formatting without writing -ruff format --check lib/iris +ruff check lib/iris # lint +ruff check --fix lib/iris # auto-fix safe lint issues +ruff format lib/iris # format +ruff format --check lib/iris # check formatting without writing ``` - **Line length**: 88 characters (Ruff default). @@ -121,7 +85,6 @@ ruff format --check lib/iris - **Imports**: Ruff-managed ordering. No direct `import netCDF4` — always use `iris.fileformats.netcdf._thread_safe_nc` for thread safety. - ## Development Conventions ### Core data model @@ -130,108 +93,99 @@ ruff format --check lib/iris - Coordinates: `DimCoord` (regular), `AuxCoord` (auxiliary), `CellMeasure`, `AncillaryVariable`. - Data may be **lazy** (Dask array). Always preserve laziness; never call - `.data` - unnecessarily inside library code. + `.data` unnecessarily inside library code. - Operations return **new** Cubes (functional style); do not mutate in place. - All metadata must be **CF-convention** compliant. - ### Deprecation - Use `iris._deprecation.warn_deprecated()` or issue a custom warning class. - Warning classes live in `iris.warnings` (e.g., `IrisUserWarning`, - `IrisCfWarning`). + `IrisCfWarning`). All `UserWarning` subclasses must ultimately inherit from + `IrisUserWarning`. - Follow the NEP29 deprecation schedule (same as NumPy). -- All `UserWarning` subclasses must ultimately inherit from `IrisUserWarning`. - -### Exception hierarchy - -Base class: `iris.exceptions.IrisError`. Common subclasses: -`CoordinateNotFoundError`, `CoordinateCollapseError`, `IgnoreCubeException`. - - -### Versioning - -Version is derived from git tags via `setuptools_scm`. Do not hard-code version -strings. +### Exceptions and versioning +- Base class: `iris.exceptions.IrisError`. Common subclasses: + `CoordinateNotFoundError`, `CoordinateCollapseError`, `IgnoreCubeException`. +- Version is derived from git tags via `setuptools_scm`. Do not hard-code + version strings. ## Changelog -Changelog fragments lives under `chngelog/` and is built with towncrier via +Changelog fragments live under `changelog/` and are built with towncrier via sphinx. See [`changelog/AGENTS.md`](changelog/AGENTS.md) for full rules. - ## Documentation Documentation lives under `docs/` and is built with Sphinx. See [`docs/AGENTS.md`](docs/AGENTS.md) for full rules. +Agreed design specs live in `docs/src/developers_guide/specs/` as +`YYYY-MM-DD--design.md`. They are **published** MyST Markdown and are +**living documents**, revised as a design evolves. Each declares a citation +prefix (e.g. `merge spec §5.7`) so its sections can be cited from issues and +pull requests, and carries explicit `(prefix-N-N)=` anchors. Markdown is used +*only* for specs; all other documentation is RST. + +Implementation plans live in the sibling `docs/src/developers_guide/plans/` as +`YYYY-MM-DD-.md`. A plan is point-in-time and frozen once its pull +request merges, so plans are tracked in the repository but excluded from the +Sphinx build via `exclude_patterns` in `docs/src/conf.py`. ## Lock-file Maintenance +Adding a dependency means editing `requirements/py*.yml` **and** the matching +`requirements/locks/*.lock`. Repository precedent (PRs #7095, #7100) is a +*minimal* lock edit — insert only the new package URLs and update `input_hash` +— rather than a full re-solve, which bundles dozens of unrelated version bumps +into an otherwise focused PR. Full re-solves are left to the weekly +`refresh-lockfiles` workflow, or run locally with: + ```bash -# Regenerate lock files for all supported Python versions -python tools/update_lockfiles.py -o requirements/locks requirements/py*.yml -# Shortcut via Makefile -make lockfiles +make lockfiles # python tools/update_lockfiles.py -o requirements/locks requirements/py*.yml ``` - -## Pull Request Guidelines - -- When creating a pull request a template is provided to ensure all checks are - considered. -- This project is configured to use pre-commit tht will ensure some checks are - performed automatically. -- Keep changes focused; avoid unrelated refactors in the same PR. -- Add or update tests for every change to production code. -- Ensure a whatsnew fragment is added, see -[`changelog/AGENTS.md`](changelog/AGENTS.md) - +## Contribution Workflow + +- Push development branches to `origin` (the `bjlittle/iris` fork). Raise pull + requests from there against the relevant **feature branch** on `upstream` + (`SciTools/iris`) — never against `upstream/main`. The `upstream` remote is + push-disabled, which enforces this. +- **Attribute agentic work clearly.** Say in the body of every pull request and + issue that it is agentic, and end commit messages with the `Co-Authored-By` + trailer. Never let agentic work read as hand-written. +- Label such pull requests and issues `Agentic` and `Type: Feature Branch`, + plus whichever `Feature: …` label fits the subject. +- Use the pull request template; keep changes focused and avoid unrelated + refactors; add or update tests for every production change; ship a changelog + fragment (see [`changelog/AGENTS.md`](changelog/AGENTS.md)). +- **If in doubt about anything outward-facing** — base branch, labels, whether + to post at all — **ask before pushing.** ## Critical Development Gotchas -1. **xfail_strict Behavior**: Tests marked `@pytest.mark.xfail` that now PASS - become FAILURES -> **remove xfail immediately when bug is fixed** - -2. **Pre-commit Auto-fixes**: Hooks may auto-fix ISC001/COM812 conflicts -> - re-stage files: `git add . && git commit` - -3. **Lockfile Rebuilds**: Updating `requirements/locks/*.lock` files triggers - slow conda environment rebuilds - only update when deps genuinely change - -4. **Lazy Data Pitfalls**: Check `cube.has_lazy_data()` before operations; use - `cube.lazy_data()` when appropriate; `cube.data` materializes arrays - -5. **Coordinate Scope**: Not all coordinates span all dimensions -> use - `cube.coords(dimensions=dim_indices)` to query specific axes - -6. **Dask/NumPy Compatibility**: Some operations fail on lazy (Dask) arrays - -> materialize with `cube.compute()` or check dask support before use - -7. **Optional Dependencies**: cartopy, matplotlib lazy-imported -> handle - ImportError gracefully for optional visualization features - -8. **CF-Metadata Validation**: Custom coordinates must comply with CF - standards via `cube.metadata` - -9. **Cube Immutability**: Operations return new cubes; don't modify in-place - (`cube.data[...] = x` won't affect the cube) - -10. **Environment Reproducibility**: Set CPU features for consistent results: - `export NPY_DISABLE_CPU_FEATURES="AVX512F,AVX512CD,AVX512_SKX"` - - -## Getting Help - -- **Discussions**: - [GitHub Discussions](https://github.com/SciTools/iris/discussions) -- **Contributing Guide**: [.github/CONTRIBUTING.md](.github/CONTRIBUTING.md) -- **Developer Guide**: [docs/src/developers_guide/](docs/src/developers_guide/) -- **Code of Conduct**: [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md) - +1. **xfail_strict**: `@pytest.mark.xfail` tests that now PASS become FAILURES — + remove the xfail immediately when the bug is fixed. +2. **Pre-commit auto-fixes**: hooks may auto-fix ISC001/COM812 conflicts — + re-stage with `git add . && git commit`. +3. **Lock-file rebuilds**: lock changes trigger slow conda environment rebuilds + — only update when deps genuinely change. +4. **Lazy data**: check `cube.has_lazy_data()`; use `cube.lazy_data()` where + appropriate; `cube.data` materialises arrays. +5. **Coordinate scope**: not all coords span all dims — use + `cube.coords(dimensions=dim_indices)` to query specific axes. +6. **Dask/NumPy compatibility**: some operations fail on lazy arrays — + materialise with `cube.compute()` or check dask support first. +7. **Optional dependencies**: cartopy and matplotlib are lazy-imported — handle + `ImportError` gracefully. +8. **CF-metadata validation**: custom coordinates must comply with CF standards + via `cube.metadata`. +9. **Cube immutability**: operations return new cubes; `cube.data[...] = x` + will not affect the cube. +10. **Environment reproducibility**: set `NPY_DISABLE_CPU_FEATURES` as above for + consistent results. ## ⚠️ Meta-Instruction: Auto-Update Rule - **Trigger**: You MUST automatically update this `AGENTS.md` file if a new diff --git a/changelog/7274.dependency.rst b/changelog/7274.dependency.rst new file mode 100644 index 0000000000..812d965ab1 --- /dev/null +++ b/changelog/7274.dependency.rst @@ -0,0 +1,3 @@ +:user:`bjlittle` added `myst-parser `_ as a +documentation dependency, so that developer design specs may be authored in +MyST Markdown. All other documentation remains reStructuredText. diff --git a/changelog/7274.doc.rst b/changelog/7274.doc.rst new file mode 100644 index 0000000000..19000e671b --- /dev/null +++ b/changelog/7274.doc.rst @@ -0,0 +1,6 @@ +:user:`bjlittle` added a design spec for a programme of work to make +:meth:`~iris.cube.CubeList.merge` and :meth:`~iris.cube.CubeList.concatenate` +more robust, efficient, configurable, extensible, lazy and lenient, along with a +new :ref:`developers_guide_specs` section of the Developers Guide to publish it +and any future specs. (:issue:`2761`, :issue:`5375`, :issue:`6790`, +:issue:`7063`, :issue:`7241`) diff --git a/docs/src/conf.py b/docs/src/conf.py index 0e491f3ddc..af2d7189ff 100644 --- a/docs/src/conf.py +++ b/docs/src/conf.py @@ -154,6 +154,7 @@ def _dotv(version): extensions = [ "matplotlib.sphinxext.mathmpl", "matplotlib.sphinxext.plot_directive", + "myst_parser", "readingtime", "readingtime_validator", "sphinx.ext.autodoc", @@ -222,6 +223,15 @@ def _dotv(version): ) ) +# -- myst-parser extension ---------------------------------------------------- +# See https://myst-parser.readthedocs.io/en/latest/configuration.html +# Markdown is reserved for developer design specs, see +# developers_guide/specs/index. All other documentation remains +# reStructuredText. +myst_enable_extensions = [ + "colon_fence", +] + # -- Napoleon extension ------------------------------------------------------- # See https://sphinxcontrib-napoleon.readthedocs.io/en/latest/sphinxcontrib.napoleon.html napoleon_google_docstring = True @@ -458,6 +468,13 @@ def _dotv(version): # list of sources to exclude from the build. exclude_patterns = [ "**/highlights.rst", + # Implementation plans are tracked in the repository but not published; + # see developers_guide/specs/index. + "developers_guide/plans/**", + # myst-parser makes ".md" a source suffix, which would otherwise pull the + # README of each of these non-document directories into the build. + "_build/**", + "_static/**", ] # -- sphinx-gallery config ---------------------------------------------------- diff --git a/docs/src/developers_guide/contributing_getting_involved.rst b/docs/src/developers_guide/contributing_getting_involved.rst index 94383cd3bf..41a2ce674c 100644 --- a/docs/src/developers_guide/contributing_getting_involved.rst +++ b/docs/src/developers_guide/contributing_getting_involved.rst @@ -52,6 +52,7 @@ If you are new to using GitHub we recommend reading the contributing_changes github_app release + specs/index .. toctree:: diff --git a/docs/src/developers_guide/plans/README.md b/docs/src/developers_guide/plans/README.md new file mode 100644 index 0000000000..bcc33d550e --- /dev/null +++ b/docs/src/developers_guide/plans/README.md @@ -0,0 +1,10 @@ +# Implementation plans + +Implementation plans for the design specs in [`../specs/`](../specs). + +Unlike a spec, a plan is **not** a living document. It records the sequence of +steps agreed at the time it was written, and is frozen once the work it +describes has merged. Plans are tracked in the repository for provenance, but +are excluded from the Sphinx build via `exclude_patterns` in `docs/src/conf.py`. + +Naming: `YYYY-MM-DD-.md`. diff --git a/docs/src/developers_guide/specs/2026-09-10-merge-concatenate-design.md b/docs/src/developers_guide/specs/2026-09-10-merge-concatenate-design.md new file mode 100644 index 0000000000..9359f443fb --- /dev/null +++ b/docs/src/developers_guide/specs/2026-09-10-merge-concatenate-design.md @@ -0,0 +1,683 @@ +# Making Iris merge and concatenate robust, efficient and configurable + +```{readingtime} +``` + +> **Living document.** This spec is updated as the design evolves; it is not a +> point-in-time record of what was agreed on any particular day. Cite sections +> of it as `merge spec §N.N`. A bare `§N.N` inside this document refers to this +> document. + +- **Date:** 2026-09-10 +- **Status:** design agreed; no pull request opened yet. §6 is the live progress + record, and is updated as each pull request lands. +- **Issues:** {issue}`2761`, {issue}`5375`, {issue}`6790`, {issue}`7063`, + {issue}`7241` +- **Applies to:** `lib/iris/_merge.py`, `lib/iris/_concatenate.py`, + `lib/iris/_combine.py` +- **Baseline:** `greenfield` at `253510b37`, Iris `3.17.0.dev4`. Line references + in this document are accurate as of that commit. + +(merge-spec-1)= + +## 1. Summary + +`iris.cube.CubeList.merge` and `CubeList.concatenate` are the core of Iris's +input workflow and the source of a long tail of user complaints. They are two +independent engines that share no code, carry an open correctness bug from 2017, +realise lazy arrays repeatedly, and expose almost no configuration. + +This document specifies a programme of twelve small, independently reviewable +pull requests. Each is either a behaviour-preserving refactor or a standalone +user-visible improvement. Together they build a shared substrate beneath both +engines, without requiring anyone to agree to a rewrite up front. + +The six goals, in the requester's words, are that merging datasets into a hyper +cube should be **more robust, efficient, configurable at the API level, +extensible, lazy and lenient**. + +(merge-spec-2)= + +## 2. Why the current design is at its limit + +(merge-spec-2-1)= + +### 2.1 Two engines that share nothing + +`lib/iris/_merge.py` (1863 lines) and `lib/iris/_concatenate.py` (1702 lines) +each define their own private `_CubeSignature`, `_CoordSignature`, `_ProtoCube`, +`_CoordAndDims` and `_CoordMetaData`. They share no comparison routine, no +signature type and no error type. + +`lib/iris/_combine.py` (added 2025) sequences them but unifies nothing — its +`_combine_cubes` loop calls `.merge()` and `.concatenate()` in an order given by +the `merge_concat_sequence` option. + +A third comparison implementation, `iris.common.resolve.Resolve` (2607 lines), +was built to be the single source of truth for cube comparison and combination +and is used only by cube maths. Iris therefore carries three independent answers +to "are these two cubes compatible, and how do I combine them?". + +(merge-spec-2-2)= + +### 2.2 Merge is not lazy + +Merge compares live `Coord` objects and cell measures / ancillary variables with +`==`: + +- `_merge.py:192` — `_CoordPayload._coords_msgs`, reached from + `match_signature` at lines 262 and 267 with tuples of actual `Coord` objects. +- `_merge.py:450-453` — `_CubeSignature.match`, on `cell_measures_and_dims` and + `ancillary_variables_and_dims`. + +`_DimensionalMetadata.__eq__` (`coords.py:659-690`) ends in +`iris.util.array_equal(..., withnans=True)`, so every one of these comparisons +realises lazy arrays. They run once per candidate-cube × ProtoCube pair, with no +caching. + +Concatenate solved exactly this problem in {pull}`5926` using `xxhash` array +hashing and a single batched `dask.compute` (`_compute_hashes`, +`_concatenate.py:479`). Merge never received it. Reported impact on ECMWF +hybrid-pressure loading is roughly 5×, and more than 10× worse again under +`_LAZY_DERIVED_LOADING`, with over 90% of time in +`cubes.combine → merge → coord equality → dask compute` ({issue}`7063`, +{issue}`7241`). + +(merge-spec-2-3)= + +### 2.3 Merge is barely configurable + +`concatenate()` has four escape hatches — `check_aux_coords`, +`check_cell_measures`, `check_ancils`, `check_derived_coords`. `merge()` has one +keyword, `unique`. This asymmetry is directly user-visible in {issue}`6790`, +where cubes refuse to merge but concatenate happily after `new_axis`. + +(merge-spec-2-4)= + +### 2.4 Merge can be silently wrong + +{issue}`2761`, open since 2017. `build_indexes` (`_merge.py:585-604`) records, +for each scalar value, the *set* of values it co-occurs with. Set membership +discards the structure needed to decide separability, so unrelated candidate +dimensions are judged separable. See §5.6 for the confirmed diagnosis and §5.7 +for the fix. + +(merge-spec-2-5)= + +### 2.5 Leniency has been requested for a decade + +{issue}`1987` (2016) → {issue}`4446` (merge, 2021) → {issue}`5392` +(concatenate, 2023). Never delivered, in part because retrofitting it onto two +unrelated engines means doing the work twice in two idioms. + +(merge-spec-3)= + +## 3. Constraints + +1. **No large pull requests.** Iris core developers are sceptical of agentic + contributions, and the AUX-Coord minutes of 2026-01-07 record "Easy to raise + a PR, but hard to get reviewed". Review capacity is the scarce resource being + optimised for. Every item below must stand alone. +2. **Backwards compatible with v3.x.** No breaking changes. The one deliberate + behaviour change (§5.7) is a bug fix and is treated as such. +3. **Honest labelling.** Agentic work is labelled as agentic; precedent exists in + {pull}`7161`. + +(merge-spec-4)= + +## 4. Decisions taken + +(merge-spec-4-1)= + +### 4.1 Where configurability is exposed + +Each capability is implemented once in the shared substrate, exposed as explicit +keywords on `merge()` / `concatenate()`, and reaches load time through +`merge_kwargs` / `concatenate_kwargs` forwarding dicts on `CombineOptions`. + +This mirrors the existing `equalise_cubes_kwargs` rather than inventing a new +mechanism. `CombineOptions.OPTION_KEYS` is already a list carrying the comment +"so we can update it in an inheriting class" (`_combine.py:182`), and dict-valued +options are already special-cased at `_combine.py:232`. + +Rejected: `CombineOptions`-only, which leaves {issue}`6790` unfixed for direct +callers and buries configuration in ambient thread-local state; and +method-keywords-only, which leaves load-time users unable to reach the new +behaviour. + +(merge-spec-4-2)= + +### 4.2 How far leniency goes + +Leniency covers **metadata** (via the existing `metadata.equal(lenient=...)` +semantics) and **structure** (tolerating missing or extra aux coords, cell +measures and ancillary variables). + +Coercion of dtypes, units and calendars stays out of the engine, in +`iris.util.equalise_cubes`, which is the existing extension point for it. + +(merge-spec-4-3)= + +### 4.3 Substrate built bottom-up, not designed top-down + +The substrate is grown one PR at a time starting from the hashing layer, rather +than specified in advance. The alternative — designing a unified v4.0 engine +first — was considered and parked at +. + +Adopting `Resolve` wholesale was also rejected as a *starting* point: it is +pairwise and merge is N-ary, so whether it scales is unresolved research risk, +and nothing user-visible would ship until it was answered. + +(merge-spec-5)= + +## 5. Design + +(merge-spec-5-1)= + +### 5.1 The substrate module + +A single private module, `lib/iris/_combine_common.py`, sibling to `_merge.py`, +`_concatenate.py`, `_combine.py` and `_lazy_data.py`. Promoted to a package only +if it exceeds roughly 800 lines. + +Rejected placements: + +- **`iris/common/`** — that subpackage is documented and public-facing; putting + engine internals there implies an API commitment. +- **Nesting under a `_combine/` package** — this inverts the dependency. + `_merge.py` would import `iris._combine.hashing`, executing + `iris/_combine/__init__.py`, which reaches `iris.cube`. The substrate sits + *below* merge and concatenate; `_combine.py` sits *above* them. + +The substrate must import nothing from Iris. The hashing code's only Iris +dependency is `iris.coords` for type hints in `array_id`, which goes behind +`TYPE_CHECKING`. This is what makes the merge-side import risk-free. + +(merge-spec-5-2)= + +### 5.2 What is shared, and what is not + +The two cube-level signatures are **not** unified. They encode different +questions: merge's asks "same kind of cube, identical shape?", because merge adds +a dimension and all sources have the same shape; concatenate's asks "joinable +along an axis?" and carries extents, dim order and dim mapping to answer it. + +The genuinely shared kernel sits one level down: + +1. **Element signature** — a hashed, comparable representation of a single + dimensional-metadata object (coord, cell measure, ancillary variable): + its `metadata`, its dims, `has_bounds()`, and hashes of points/bounds/data. +2. **Array-collection walk** — the traversal of a cube yielding arrays to hash. + Concatenate's `add_coords` (`_concatenate.py:601-611`) and merge's equivalent + are the same function written twice. +3. **Comparison result** — a value carrying *why* two elements differ, which + feeds diagnostics. + +(merge-spec-5-3)= + +### 5.3 Merge needs a driver function + +Concatenate's loop lives in `iris._concatenate.concatenate()`, which is why it +can hash everything up front. Merge's loop lives in `CubeList.merge()` at +`cube.py:443-467`; there is no `iris._merge.merge()`. + +A new `iris._merge.merge(cubes, unique=True)` takes that loop, and +`CubeList.merge()` delegates — mirroring `CubeList.concatenate()`. This is also +the landing site for every keyword added later, which is why it comes early. + +`merge_cube()` does not share this loop: it builds a single ProtoCube from +`self[0]` with `error_on_mismatch=True` and no name grouping +(`cube.py:364-373`). It is left alone initially and revisited when its error path +becomes relevant. + +(merge-spec-5-4)= + +### 5.4 Hash-based comparison in merge + +Arrays are collected up front in `_merge.merge()`, mirroring concatenate, and +`hashes` is threaded through `ProtoCube.register()` to the two consumption sites +in §2.2. `self.scalar.defns` (`_merge.py:256`) is left alone — it is already +metadata-only and cheap. + +**Exactly one `compute_hashes()` call per invocation.** This is not stylistic: +`ArrayHash.__eq__` raises `ValueError` on same-shape/different-chunks, and only a +single batched call guarantees chunk unification across every input cube. + +**This is semantics-preserving.** `_DimensionalMetadata.__eq__` +(`coords.py:659-690`) is metadata equality, then `has_bounds()` parity, then +`array_equal(..., withnans=True)` on values and bounds. And +`lib/iris/tests/unit/concatenate/test_hashing.py:88` already asserts that hash +equality is equivalent to `array_equal(a, b, withnans=True)` across a shared +corpus. The metadata and `has_bounds()` checks stay exactly where they are; only +the two `array_equal` calls become hash lookups. + +**The `id()` hazard.** `array_id` is `f"{id(coord)}{bound}"`, and merge calls +`_extract_coord_payload` once per *(cube, ProtoCube)* pair, so ids must be stable +across calls. They are: `Cube.dim_coords` and `Cube.aux_coords` +(`cube.py:3158,3182`) return a fresh tuple of the *same* objects. +`Cube.derived_coords` (`cube.py:3195`) does not — it regenerates from factories +on every access. Merge never array-compares derived coords, going through +`factory_defns` (metadata only, `_merge.py:271`), so it is unaffected. The driver +must nonetheless hold a reference to every input cube for the call's duration, +and this needs a code comment. + +**Risks to state in the PR:** metadata comparison must remain ordered first, +because hashes unify numerical dtypes (`float32([1])` and `bool([1])` hash equal +— consistent with `array_equal`, but only if ordering is unchanged); and the +chunk-mismatch `ValueError` becomes reachable from merge for the first time. + +(merge-spec-5-5)= + +### 5.5 Measurement + +`benchmarks/benchmarks/merge_concat.py::Merge` merges two cubes and explicitly +strips cell measures and ancillary variables. It cannot demonstrate this +improvement. The benchmark is extended in a separate PR *before* the change, so +the baseline is committed to `main` and the speedup is reproducible. + +(merge-spec-5-6)= + +### 5.6 Separability: the diagnosis + +Confirmed by direct probe against the reporter's case in {issue}`2761`: + +```text +A = [1, 2, 1, 2, 1, 2] +B = [3, 4, 4, 4, 4, 3] +C = [5, 5, 6, 6, 7, 7] +``` + +Current Iris derives: + +```text +a: separable=['b', 'c'] inseparable=[] +b: separable=['a'] inseparable=['c'] +c: separable=['a'] inseparable=['b'] +``` + +and `CubeList.merge()` raises +`ValueError: You must specify the meta or dtype of the array` — a raw Dask error +with no connection to the actual problem. That is the user experience today. + +The current predicate is `_separable_pair` (`_merge.py:607-631`): X and Y are +separable iff every value of X co-occurs with the same *set* of Y values. + +(merge-spec-5-7)= + +### 5.7 Separability: the fix + +**Replacing the set with a multiset does not work.** The multiset +co-occurrences are also equal in this case (`a=1 → {3:1, 4:2}`, +`a=2 → {3:1, 4:2}`). The correct criterion is about the pair grid, not per-value +co-occurrence: + +> X and Y are separable if and only if the observed (X, Y) pairs form the +> complete Cartesian product `values(X) × values(Y)` with **uniform +> multiplicity**. + +Uniform, not one: in a full three-way product each (X, Y) pair recurs once per +value of the third dimension, and requiring multiplicity 1 would wrongly reject +it. + +Verified against five topologies: + +| topology | expected | result | +|---|---|---| +| 2×2 product | separable | ✓ | +| {issue}`2761` case | only `a`–`c` separable | ✓ | +| 2×2×2 full product | all pairs separable (multiplicity 2, uniform) | ✓ | +| `b = f(a)` | inseparable | ✓ | +| incomplete 2×2 (3 cells) | inseparable | ✓ | + +Verified end-to-end by simulating the predicate through a real +`CubeList.merge()`: six cubes become one, `(a: 2; c: 3; y: 2; x: 2)` with `b` as +a 2-D auxiliary coordinate spanning `a` and `c` — precisely what the reporter +said the answer should be. The rest of the algebra (`derive_groups`, +`_derive_separable_group`, `_derive_consistent_groups`, `_is_dependent`) already +handles it correctly once the predicate is right. + +**Blast radius is one call site**: `_merge.py:1215-1220` in `ProtoCube.merge()`, +where `positions` is already in scope — it is passed to `derive_space` on the +next line. Only `derive_relation_matrix` changes shape. `build_indexes` stays, +because `_define_space` still consumes it (`_merge.py:1446`). + +**This algebra has essentially no unit tests.** Grepping `lib/iris/tests` finds +exactly one reference: `integration/merge/test_merge.py:329` +`test_separable_combination`. That is why a 2017 bug survived to 2026. +Characterisation tests therefore land *before* the fix, so the fix's diff shows +exactly which expectations change. + +**Known edge case to pin with a test:** duplicated source cubes make a pair grid +non-uniform, so a case that was "separable, then duplicate-detected" becomes +"inseparable, then duplicate-detected". Probably the same user-visible outcome, +but it must be verified rather than assumed. + +(merge-spec-5-8)= + +### 5.8 Diagnostics + +`merge()` explains nothing today — it returns N cubes with no indication why they +did not become one. The `msgs` lists built in `_CubeSignature.match` and +`_CoordPayload.match_signature` are constructed and discarded whenever +`error_on_mismatch=False`. + +The work is therefore mostly plumbing: capture and surface what is already +computed, using the comparison-result type from §5.2. `merge()`'s return type is +unchanged; a separate diagnostic entry point is added alongside +`iris.util.describe_diff` (`util.py:205`), which already does pairwise cube +diffs. What is missing is the group-level account: these three formed one cube, +this fourth stayed out because X. + +(merge-spec-5-9)= + +### 5.9 Leniency + +The primitive already exists: every metadata class has +`equal(other, lenient=None)` (`metadata.py:660`), decorated `@lenient_service`. +Merge hand-compares metadata field by field in `_CubeSignature._defn_msgs` +(`_merge.py:350-406`); concatenate has its own equivalent. Both become +`equal(lenient=...)`. + +`lenient=True` is passed **explicitly**. The `LENIENT` thread-local +(`iris/common/lenient.py:667`) is not used, so no ambient state enters the +engine's hot path — consistent with §4.1. + +On {issue}`5394` and {issue}`5395` the claim must be precise: this adopts the +lenient metadata *semantics* that `Resolve` is built on, not `Resolve` itself. +They are partially addressed, not closed. + +Structural leniency — reconciling elements present in some cubes but not others, +by intersection or union — is separated from the `check_*` flags, which only say +"do not compare". It lands last among the behaviour changes because it is the one +item with genuine semantic debate in it. + +(merge-spec-6)= + +## 6. Plan roadmap + +Twelve pull requests deliver the design. Each gets its own implementation plan in +`docs/src/developers_guide/plans/`, and a PR is merged before any PR that +*depends on it* is written. The dependencies form a partial order, not a chain: +PRs 1, 2, 3 and 6 have no predecessors and may proceed in parallel. The ordering +follows §5's layering — the substrate first (PR 1), then the seam that lets merge +use it (PRs 2–3), then the payoff (PR 4), then the correctness and capability +work that the substrate makes cheap. + +This table is the programme's live progress record and is updated as each PR +lands. Status vocabulary: **not started** · **in progress** · **✅ complete** · +**blocked** · **abandoned**, each carrying its {pull} citation once one exists. + +| # | PR | Scope (spec §) | Type | Closes | Depends on | Status | +|---|---|---|---|---|---|---| +| 1 | Hashing machinery → `_combine_common.py` | §5.1 — move `_concatenate.py:305-541` (~237 lines) and `tests/unit/concatenate/test_hashing.py`; names lose their underscore prefix | internal | — | — | not started | +| 2 | `iris._merge.merge()` driver function | §5.3 — lift the loop out of `CubeList.merge()` (`cube.py:443-467`); `CubeList.merge` delegates, mirroring `CubeList.concatenate` | internal | — | — | not started | +| 3 | Extend merge benchmark | §5.5 — `benchmarks/benchmarks/merge_concat.py::Merge` currently strips cell measures and ancillaries, so it cannot show the PR 4 effect | internal | — | — | not started | +| 4 | **Hash-based comparison in merge** | §5.4 — collect arrays up front, thread `hashes` through `ProtoCube.register()` to the two `==` sites of §2.2 | performance | {issue}`7063`, {issue}`7241` | 1, 2, 3 | not started | +| 5 | Shared element signature + collection walk | §5.2 items 1–3 — one hashed element signature, one array-collection walk, one comparison-result type, for both engines | internal | — | 1, 4 | not started | +| 6 | Separability characterisation tests + doctest cleanup | §5.6, §7 — the five-topology table as tests against *current* behaviour; fix the Python-2 doctests at `_merge.py:691` | internal | — | — | not started | +| 7 | **Separability fix** | §5.7 — replace `_separable_pair`'s set criterion with complete-grid-and-uniform-multiplicity, at the single call site `_merge.py:1215-1220` | bugfix | {issue}`2761`, likely {issue}`5768` | 6 | not started | +| 8 | Structured diagnostics | §5.8 — capture the `msgs` already built and discarded; a group-level account beside `iris.util.describe_diff` | feature | {issue}`5375` | 2, 5 | not started | +| 9 | Metadata leniency | §5.9 — `metadata.equal(lenient=...)` replaces hand-rolled field comparison in both engines | feature | {issue}`4446`, {issue}`5392`; part of {issue}`5394` / {issue}`5395` | 2, 5 | not started | +| 10 | `check_*` parity for merge | §2.3 — the four `check_*` keywords on merge, with the `cube.py:427-433` documentation change in the same PR | feature | {issue}`6790` | 2 | not started | +| 11 | Structural leniency | §5.9 — reconcile elements present in some cubes but not others, by intersection or union | feature | — | 9, 10 | not started | +| 12 | `CombineOptions` forwarding dicts | §4.1 — `merge_kwargs` / `concatenate_kwargs` in all four `SETTINGS` dicts, defaulting to `None` | feature | — | 10, 11 | not started | + +**PRs 1–4 are the proving tranche.** They end with a measured speedup against a +committed baseline and no behaviour change whatsoever. If core developers do not +engage with those, the remainder should not be attempted. + +Cross-cutting rules (apply to every row rather than one of them): + +- **Each PR cites this spec** and says which row it is, so a reviewer can see the + shape of the programme without reading twelve descriptions, and says plainly + that it is agentic work — {pull}`7161` is the precedent (§3). +- **Every PR targets the `greenfield` feature branch, not `main`.** The + programme is long-running and deliberately incremental, so its parts land + together rather than trickling into `main` one at a time; `greenfield` is then + merged back once the tranche it carries is coherent. Each PR is labelled + `Feature: Merge/Concatenate`, `Type: Feature Branch` and `Agentic`, so the + agentic ones are filterable — both for reviewers who want them and for those + who would rather not. +- **Changelog fragment ships with its PR**, typed as the Type column says, per + `changelog/AGENTS.md`. +- **Refactor PRs change no test expectations** (rows 1, 2, 5). Test files move and + imports update; assertions do not. Behaviour PRs land their tests with them. +- **Performance claims are quoted, not asserted.** Any row claiming an effect + quotes before/after numbers from `benchmarks/benchmarks/merge_concat.py` + against a baseline already on `main` (§5.5). + +Notes on individual items: + +- **PR 1** moves seven names (`_concatenate.py:305-541`, ~237 lines) and the test + file `tests/unit/concatenate/test_hashing.py`. Names lose their underscore + prefix on the way out, following `_lazy_data.py`; this makes it a move *and* a + rename, which is a reasonable thing for a reviewer to object to and a cheap + thing to concede. +- **PR 1** must state plainly that it is the first step of a sequence and link + the performance issues. Its justification is PR 4. +- **PR 7** will change CML fixtures under `tests/results/cube_merge/`. Cases + previously called separable become inseparable; most raised or produced wrong + cubes, but some may have been accidentally right. Every changed fixture needs + an individual justification. This is the review-cost centre of the programme, + which is why it follows the cheap wins. +- **PR 10** requires a documentation change in the same PR: the note at + `cube.py:427-433` states that aux coords, cell measures and ancillaries "must + be identical in every input cube". `check_*=False` is exactly the relaxation of + that sentence. +- **PR 12** must add each new option to all four `SETTINGS` dicts + (`_combine.py:190-219`), defaulting to `None` in all four so load-time + behaviour is unchanged. + +Outside the roadmap: + +- A v4.0 unified engine is parked at + (§4.3, §9) and is not a row here. +- The coercion work under {issue}`6383` is adjacent and separately owned: §4.2 + routes dtype, unit and calendar coercion to `iris.util.equalise_cubes` rather + than into either engine. +- Core-developer engagement is the gating condition, not a deliverable. PR 1's + reception decides whether PRs 5–12 are attempted (§3). + +### Assumptions and open decisions + +Enumerated so they are visible decisions rather than silent drift. Items 1–5 are +decisions this design makes; the remainder are questions assigned to the PR that +must answer them, ordered by owning PR. Items carry a tracking issue where one +already exists; the rest name their owning row, and get an issue when that PR is +opened. + +1. **Resolved** (2026-09-10) — **Incremental substrate, not a designed engine.** + The alternative — specify a unified v4.0 engine, then build it — was + considered and rejected as a *starting* point, because nothing user-visible + ships until the whole thing is agreed. It is preserved rather than discarded, + at , and §4.3 carries the + reasoning. Adopting `Resolve` wholesale was rejected on the same ground: it is + pairwise, merge is N-ary, and whether it scales is unresolved research risk. +2. **Resolved** (2026-09-10) — **Configurability is keywords plus forwarding + dicts.** Each capability is implemented once in the substrate, exposed as an + explicit keyword on `merge()` / `concatenate()`, and reaches load time through + `merge_kwargs` / `concatenate_kwargs` on `CombineOptions` (§4.1). The two + rejected alternatives are recorded there: `CombineOptions`-only leaves + {issue}`6790` unfixed for direct callers, and method-keywords-only leaves + load-time users unable to reach the behaviour. +3. **Resolved** (2026-09-10) — **Leniency covers metadata and structure; + coercion stays out.** §4.2. Dtype, unit and calendar coercion belongs in + `iris.util.equalise_cubes`, which is the existing extension point for it. +4. **Resolved** (2026-09-10) — **The substrate is a flat private module.** + `lib/iris/_combine_common.py`, sibling to `_merge.py` and `_concatenate.py`, + promoted to a package only above roughly 800 lines. `iris/common/` was + rejected as public-facing, and a `_combine/` package as inverting the + dependency — the substrate sits *below* the two engines, `_combine.py` sits + above them (§5.1). +5. **Resolved** (2026-09-10) — **The two cube-level signatures are not + unified.** They encode different questions, and forcing one type on both would + be a rewrite wearing a refactor's clothes (§5.2). {issue}`3234` therefore + stays open; this programme narrows it rather than closing it. +6. **Open** (PR 1) — **Whether the moved names drop their underscore prefix.** + Following `_lazy_data.py` makes PR 1 a move *and* a rename, which is a + reasonable thing for a reviewer to object to and a cheap thing to concede. The + location is what matters, not the name (§8). +7. **Open** (PR 1) — **Whether the programme proceeds past the proving tranche.** + §3 names review capacity as the scarce resource. PR 1 is a pure move with no + behaviour change and its justification is PR 4, so its reception is the signal + for everything after it. Nothing in PRs 5–12 is load-bearing for PRs 1–4. +8. **Open** (PR 2) — **Whether `merge_cube()` joins the driver.** §5.3 leaves it + alone: it builds a single ProtoCube from `self[0]` with + `error_on_mismatch=True` and no name grouping (`cube.py:364-373`). Its error + path is exactly where PR 8's diagnostics are most visible, so the question + re-opens there rather than being settled once. +9. **Open** (PR 4) — **What merge does if `ArrayHash.__eq__` raises.** §5.4 + argues that exactly one batched `compute_hashes()` call per invocation makes + the same-shape/different-chunks `ValueError` unreachable. PR 4 must prove that + with a test rather than assert it, and decide the fallback if the argument is + wrong — this exception becomes reachable from merge for the first time. +10. **Open** (PR 5, §9) — **Whether `hexdigest` reconciliation stays out of + scope.** `_concatenate.py` uses both `iris.common.metadata.hexdigest` (line + 807, diagnostic path only) and the array-hashing layer. §9 keeps them apart so + that a mechanical move does not become a semantic argument. If PR 5's shared + element signature makes the duplication glaring, revisit it there. +11. **Open** (PR 6) — **How many CML fixtures PR 7 changes.** Unknown until the + characterisation tests exist. Cases previously called separable become + inseparable; most raised or produced wrong cubes, but some may have been + accidentally right, and every changed fixture needs an individual + justification. §5.7 calls this the programme's review-cost centre; PR 6 + measures it and PR 7 justifies it. +12. **Open** (PR 6, PR 7) — **Duplicated source cubes under the new criterion.** + Duplication makes a pair grid non-uniform, so a case that was "separable, + then duplicate-detected" becomes "inseparable, then duplicate-detected". + Believed to be the same user-visible outcome, but §5.7 requires it be pinned + by a test rather than assumed. +13. **Open** (PR 7, {issue}`5768`) — **Whether the separability fix closes + {issue}`5768`.** The roadmap says "likely", which is not good enough for a PR + description. The claim needs a reproduction against the fixed predicate before + it is made. +14. **Open** (PR 10) — **Whether the new keywords also go on `merge_cube()`.** + Downstream of item 8. If `merge_cube()` never joins the driver, it acquires + the keywords separately or not at all, and {issue}`6790` is only partly + answered. +15. **Open** (PR 11) — **Intersection or union for structural leniency.** + Reconciling elements present in some cubes but not others has two defensible + answers, and the choice is user-visible. §5.9 separates this from the + `check_*` flags, which only say "do not compare". It lands last precisely + because it is the one item with genuine semantic debate in it. + +(merge-spec-7)= + +## 7. Testing and validation + +Per `lib/iris/tests/AGENTS.md`: pytest style, plain assertions, +`pytest.raises(..., match=...)`, `pytest.warns`, explicit laziness assertions, no +network. + +- **Refactor PRs (1, 2, 5)** must change no test expectations. Test files move + and imports update; assertions do not. +- **PR 4** needs explicit tests for masked arrays, NaN, and dtype edge cases at + the hash boundary, plus a test that the chunk-mismatch `ValueError` does not + fire in normal use. The existing equivalence test + (`tests/unit/concatenate/test_hashing.py:88`) is the anchor. +- **PR 6** introduces the topology table of §5.7 as characterisation tests + against *current* behaviour, and fixes the dead Python-2 doctests in + `build_indexes` and `derive_relation_matrix` (`_merge.py:691` uses + `matrix.iteritems()`, which cannot ever have run). +- **PR 7** flips those expectations, with each change justified. +- **Benchmarks**: `benchmarks/benchmarks/merge_concat.py`, extended in PR 3, with + before/after numbers quoted in PR 4. +- Every PR carries a `changelog/..rst` fragment per + `changelog/AGENTS.md`. + +(merge-spec-8)= + +## 8. Risks + +| Risk | Mitigation | +|---|---| +| Core devs reject the shared module in PR 1 | PR 1 is a pure move with no behaviour change; if the module name or the rename is contested, concede both — the location matters, not the name | +| PR 7 changes more CML fixtures than expected | Characterisation tests land first (PR 6) so the delta is visible before the fix; each fixture justified individually | +| Hash equality diverges from `Coord.__eq__` in an untested corner | Metadata and `has_bounds()` checks unchanged; only `array_equal` calls replaced; existing equivalence test is the anchor | +| Programme stalls after the proving tranche | PRs 1–4 are independently valuable; nothing later depends on the programme continuing | +| Structural leniency semantics prove contentious | It is last, and nothing else depends on it | + +(merge-spec-9)= + +## 9. Out of scope + +- Unifying `_CubeSignature` across the two engines (§5.2). +- Adopting `Resolve` as the N-ary comparison engine (§4.3). +- Reconciling `iris.common.metadata.hexdigest` with the array-hashing layer. + `_concatenate.py` uses both — `hexdigest` at line 807, but only in the + diagnostic path that names which coords differ, not in the equality path. + Worth doing eventually; bundling it would turn a mechanical move into a + semantic argument. +- Coercion of dtypes, units and calendars (belongs in + `iris.util.equalise_cubes`). +- A v4.0 unified engine — parked at + . + +(merge-spec-10)= + +## 10. Open questions + +Questions raised by the research behind this spec that the roadmap does not +answer. Unlike §6's assumptions — each of which is owned by a PR that must +resolve it — these are open against the programme as a whole, and some will +outlive it. Every one carries a status and, where it exists, a citation; those +without one say so. + +- **Open** ({issue}`3234`) — **Should the two engines ever become one?** §5.2 + says no *now*: the two cube-level signatures encode different questions, and + the shared substrate captures the real duplication without forcing a common + type. Whether that remains the right answer once PRs 1, 5 and 9 have landed is + worth re-asking against the codebase they leave behind, not against today's. +- **Parked** () — **Does `Resolve` + scale from pairwise to N-ary comparison?** `Resolve` is the modern, tested, + public machinery for reconciling two cubes, and merge is inherently N-ary. + Nobody has established what pairwise-folding an N-ary problem costs, in + performance or in semantics. This is the central unknown behind {issue}`5394` + and {issue}`5395`, and it is research risk, not implementation risk — which is + why §4.3 keeps it off the roadmap rather than pretending to have costed it. +- **Open** ({issue}`5375`) — **Should merge diagnostics be public API?** PR 8 + makes the reasons a group failed to merge *available*; it does not settle + whether they are a documented, stable, user-facing surface or an error-message + improvement. The two have very different maintenance costs, and the answer + determines whether the diagnostics need a deprecation policy. +- **Open** ({issue}`6383`) — **Does `equalise_cubes` absorb enough coercion to + keep engine leniency narrow?** §4.2 draws the line at metadata and structure on + the strength of `equalise_cubes` covering dtype, unit and calendar. If its + sub-issues stall, pressure returns to put coercion in the engines, and the line + has to be re-argued rather than silently redrawn. +- **Open** ({issue}`7063`) — **What is the measured speedup from hash-based + comparison in merge?** {issue}`7063` and {issue}`7241` report the problem; + {pull}`5926` reports roughly 5× for the concatenate case. Merge's profile is + different — many small scalar-coord comparisons rather than a few large array + ones — so the concatenate figure is not transferable. PR 3 exists to make this + answerable before PR 4 claims anything (§5.5). +- **Open** (no tracking issue yet; raised with PR 4) — **Can cube registration + stay lazy end to end?** Hashing still computes, so PR 4 replaces + many realisations with one batched one rather than removing realisation. + Whether merge could instead decide most registrations on metadata alone and + touch array values only for genuine ties is a larger question that PR 4 does + not foreclose but also does not answer. +- **Open** (no tracking issue yet; raised with PR 7) — **Does the separability + fix need a deprecation cycle?** §5.7 treats it as a bugfix: the current + criterion produces silently wrong cubes, and there is no defensible way to + keep producing them. But it changes results for existing data without any + opt-in, which is the kind of change core developers reasonably want staged. + Warn-then-change is available; whether it is warranted is a judgement for the + PR 7 review, and the answer is worth recording here when it comes. + +(merge-spec-11)= + +## 11. References + +- {issue}`2761` — Merge Problems (separability) +- {issue}`3234` — Unify merge and concatenate +- {issue}`4446`, {issue}`5392` — `LENIENT` merge / concatenate +- {issue}`5375` — Various cube merge/concatenate issues +- {issue}`5394`, {issue}`5395` — trial `resolve` in merge / concatenate +- {issue}`5768` — dimensions mashed together on merge +- {pull}`5926` — array hashing in concatenate +- {issue}`6383` — additional `equalise_cubes` functionality +- {issue}`6790` — merge/concatenate asymmetry +- {issue}`7063`, {issue}`7241` — merge performance +- {discussion}`6881` — AUX-Coord minutes, 2026-01-07 diff --git a/docs/src/developers_guide/specs/index.rst b/docs/src/developers_guide/specs/index.rst new file mode 100644 index 0000000000..f7470d5cd8 --- /dev/null +++ b/docs/src/developers_guide/specs/index.rst @@ -0,0 +1,52 @@ +.. include:: ../../common_links.inc + +.. _developers_guide_specs: + +Design Specs +============ + +A **design spec** records what a substantial piece of Iris work is building and +why, and is agreed before any of the implementation lands. Specs are written +in `MyST`_ Markdown rather than reStructuredText; they are the only part of the +documentation that is. + +Specs are **living documents**. They are revised as the design evolves, so a +spec always describes the current intent rather than a snapshot of a past +discussion. Each carries a **plan roadmap** recording the pull requests that +deliver it and the status of each, together with the assumptions, open decisions +and open questions the work has surfaced. A spec is therefore the place to look +for how far a programme of work has got, not only what it intends. + +Each spec declares a **citation prefix** so that its sections can be referred to +unambiguously from issues, pull requests and other specs. Within a spec, a bare +``§N.N`` always means that same document. + +Give every cited section its own ``§``, and separate a run of them with a comma +or a solidus — ``§5.6, §5.7`` or ``§5.6 / §5.7``, never ``§5.6-5.7``. A dashed +range reads as a single citation with a stray suffix, so only its first section +is unambiguously identified; spelling both out costs one character and leaves +nothing for a reader, or a future cross-reference tool, to guess at. + +.. list-table:: + :header-rows: 1 + :widths: 25 75 + + * - Citation + - Spec + * - ``merge spec §…`` + - :doc:`2026-09-10-merge-concatenate-design` + +.. note:: + + Implementation plans live alongside these specs, in + ``docs/src/developers_guide/plans/``. Unlike a spec, a plan is a + point-in-time record that is frozen once the work it describes has merged, + so plans are tracked in the repository but are **not** published here. + +.. _MyST: https://myst-parser.readthedocs.io/ + +.. toctree:: + :maxdepth: 1 + :hidden: + + 2026-09-10-merge-concatenate-design diff --git a/requirements/locks/py312-linux-64.lock b/requirements/locks/py312-linux-64.lock index 541a4e0716..9680f4af73 100644 --- a/requirements/locks/py312-linux-64.lock +++ b/requirements/locks/py312-linux-64.lock @@ -1,6 +1,6 @@ # Generated by conda-lock. # platform: linux-64 -# input_hash: ec70083cb9ed06616b8d4368678123e62003bbdc710c1a3214f19caf1ef913ce +# input_hash: dd58c28bc578adba3973a7540a455cdceb68d17e8ac1e4f8704e1dbc4912929c @EXPLICIT https://conda.anaconda.org/conda-forge/linux-64/eigen-abi-5.0.1.80-hf414acd_0.conda#a85c1768af7780d67c6446c17848b76f https://conda.anaconda.org/conda-forge/noarch/font-ttf-dejavu-sans-mono-2.37-hab24e00_0.tar.bz2#0c96522c6bdaed4b1566d11387caaf45 @@ -297,6 +297,7 @@ https://conda.anaconda.org/conda-forge/linux-64/libegl-devel-1.7.0-ha4b6fd6_5.co https://conda.anaconda.org/conda-forge/linux-64/libgdal-core-3.13.3-h5fdb907_0.conda#c0723084e1315a5247d5a033d9eb5522 https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-devel-14.4.0-h23af247_0.conda#26e37e05324d7bb723350d50b33057fc https://conda.anaconda.org/conda-forge/linux-64/libraqm-0.11.0-h6406941_0.conda#3ac89a48d224409739dbf6200e524373 +https://conda.anaconda.org/conda-forge/noarch/mdit-py-plugins-0.6.1-pyhd8ed1ab_0.conda#ad6821df7a98510117db06e9a833281f https://conda.anaconda.org/conda-forge/linux-64/mo_pack-0.3.1-py312h4f23490_2.conda#cec5bc5f7d374f8f8095f8e28e31f6cb https://conda.anaconda.org/conda-forge/linux-64/pandas-3.0.5-py312h8ecdadd_1.conda#85eb29ade84d1bd97be5ec55607efb00 https://conda.anaconda.org/conda-forge/linux-64/pango-1.58.2-hda50119_0.conda#6a2822aaf9a34ac3708904a47ff3dd7e @@ -353,6 +354,7 @@ https://conda.anaconda.org/conda-forge/noarch/pyvista-0.48.4-pyhd8ed1ab_1.conda# https://conda.anaconda.org/conda-forge/noarch/esmpy-8.9.1-pyhdfbf58e_0.conda#0155d4a55415cda7f7f09bc0b7760f51 https://conda.anaconda.org/conda-forge/noarch/geovista-0.5.3-pyhd8ed1ab_1.conda#64348d05eedb1b1b5676f63101d004f2 https://conda.anaconda.org/conda-forge/linux-64/graphviz-14.1.2-h8b86629_0.conda#341fc61cfe8efa5c72d24db56c776f44 +https://conda.anaconda.org/conda-forge/noarch/myst-parser-5.1.0-pyhd8ed1ab_0.conda#580615e59fc7c07741e4d2ab052cfc8b https://conda.anaconda.org/conda-forge/noarch/pydata-sphinx-theme-0.19.0-pyhcf101f3_0.conda#87921f66a4dc56ce92e4ff13be5f63dc https://conda.anaconda.org/conda-forge/noarch/sphinx-copybutton-0.5.2-pyhd8ed1ab_1.conda#bf22cb9c439572760316ce0748af3713 https://conda.anaconda.org/conda-forge/noarch/sphinx-data-viewer-0.1.5-pyhd8ed1ab_1.conda#fbe6437c6899c54e52f9258518035aca diff --git a/requirements/locks/py313-linux-64.lock b/requirements/locks/py313-linux-64.lock index 464c78499a..ba4a363155 100644 --- a/requirements/locks/py313-linux-64.lock +++ b/requirements/locks/py313-linux-64.lock @@ -1,6 +1,6 @@ # Generated by conda-lock. # platform: linux-64 -# input_hash: 744ae3daa77a848be733ec9f02b1ea2eaa61374e7ccd2b0443f3814db0f06f9a +# input_hash: eee198cca4941f1c0b249463a8292b901c06a2608e5f787123cf985f818b3940 @EXPLICIT https://conda.anaconda.org/conda-forge/linux-64/eigen-abi-5.0.1.80-hf414acd_0.conda#a85c1768af7780d67c6446c17848b76f https://conda.anaconda.org/conda-forge/noarch/font-ttf-dejavu-sans-mono-2.37-hab24e00_0.tar.bz2#0c96522c6bdaed4b1566d11387caaf45 @@ -298,6 +298,7 @@ https://conda.anaconda.org/conda-forge/linux-64/libegl-devel-1.7.0-ha4b6fd6_5.co https://conda.anaconda.org/conda-forge/linux-64/libgdal-core-3.13.3-h5fdb907_0.conda#c0723084e1315a5247d5a033d9eb5522 https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-devel-14.4.0-h23af247_0.conda#26e37e05324d7bb723350d50b33057fc https://conda.anaconda.org/conda-forge/linux-64/libraqm-0.11.0-h6406941_0.conda#3ac89a48d224409739dbf6200e524373 +https://conda.anaconda.org/conda-forge/noarch/mdit-py-plugins-0.6.1-pyhd8ed1ab_0.conda#ad6821df7a98510117db06e9a833281f https://conda.anaconda.org/conda-forge/linux-64/mo_pack-0.3.1-py313h29aa505_2.conda#ad53894d278895bf15c8fc324727d224 https://conda.anaconda.org/conda-forge/linux-64/pandas-3.0.5-py313hbfd7664_1.conda#a78866632a871f274cd2015bccdd2ca0 https://conda.anaconda.org/conda-forge/linux-64/pango-1.58.2-hda50119_0.conda#6a2822aaf9a34ac3708904a47ff3dd7e @@ -352,6 +353,7 @@ https://conda.anaconda.org/conda-forge/noarch/pyvista-0.48.4-pyhd8ed1ab_1.conda# https://conda.anaconda.org/conda-forge/noarch/esmpy-8.9.1-pyhdfbf58e_0.conda#0155d4a55415cda7f7f09bc0b7760f51 https://conda.anaconda.org/conda-forge/noarch/geovista-0.5.3-pyhd8ed1ab_1.conda#64348d05eedb1b1b5676f63101d004f2 https://conda.anaconda.org/conda-forge/linux-64/graphviz-14.1.2-h8b86629_0.conda#341fc61cfe8efa5c72d24db56c776f44 +https://conda.anaconda.org/conda-forge/noarch/myst-parser-5.1.0-pyhd8ed1ab_0.conda#580615e59fc7c07741e4d2ab052cfc8b https://conda.anaconda.org/conda-forge/noarch/pydata-sphinx-theme-0.19.0-pyhcf101f3_0.conda#87921f66a4dc56ce92e4ff13be5f63dc https://conda.anaconda.org/conda-forge/noarch/sphinx-copybutton-0.5.2-pyhd8ed1ab_1.conda#bf22cb9c439572760316ce0748af3713 https://conda.anaconda.org/conda-forge/noarch/sphinx-data-viewer-0.1.5-pyhd8ed1ab_1.conda#fbe6437c6899c54e52f9258518035aca diff --git a/requirements/locks/py314-linux-64.lock b/requirements/locks/py314-linux-64.lock index bf0fca9583..de80c91abd 100644 --- a/requirements/locks/py314-linux-64.lock +++ b/requirements/locks/py314-linux-64.lock @@ -1,6 +1,6 @@ # Generated by conda-lock. # platform: linux-64 -# input_hash: 44e0f500cd52311fec1d8114cd6dcb464ded7a96eedaaf6a9daaf7fbb548c949 +# input_hash: 397a587135f480eca53832a6158013d747d96ee14eec2d30e16fb85606052a4a @EXPLICIT https://conda.anaconda.org/conda-forge/noarch/font-ttf-dejavu-sans-mono-2.37-hab24e00_0.tar.bz2#0c96522c6bdaed4b1566d11387caaf45 https://conda.anaconda.org/conda-forge/noarch/font-ttf-inconsolata-3.000-h77eed37_0.tar.bz2#34893075a5c9e55cdafac56607368fc6 @@ -174,6 +174,7 @@ https://conda.anaconda.org/conda-forge/linux-64/libxkbcommon-1.13.2-h51789e4_1.c https://conda.anaconda.org/conda-forge/linux-64/libxml2-devel-2.15.3-h49c6c72_1.conda#f986d9c9de80813d076d235f753889ff https://conda.anaconda.org/conda-forge/noarch/locket-1.0.0-pyhd8ed1ab_0.tar.bz2#91e27ef3d05cc772ce627e51cff111c4 https://conda.anaconda.org/conda-forge/linux-64/markupsafe-3.0.3-py314h67df5f8_1.conda#9a17c4307d23318476d7fbf0fedc0cde +https://conda.anaconda.org/conda-forge/noarch/mdurl-0.1.2-pyhd8ed1ab_1.conda#592132998493b3ff25fd7479396e8351 https://conda.anaconda.org/conda-forge/linux-64/msgpack-python-1.2.2-py314h5383ef5_0.conda#66e5d65a13ac23cd29ca87eb858c7d27 https://conda.anaconda.org/conda-forge/noarch/munkres-1.1.4-pyhd8ed1ab_1.conda#37293a85a0f4f77bbd9cf7aaefc62609 https://conda.anaconda.org/conda-forge/noarch/packaging-26.3-pyhc364b38_0.conda#936687ed80f295a1f5dbcf8bd34c252c @@ -228,6 +229,7 @@ https://conda.anaconda.org/conda-forge/noarch/jinja2-3.1.6-pyhcf101f3_1.conda#04 https://conda.anaconda.org/conda-forge/linux-64/libgl-devel-1.7.0-ha4b6fd6_5.conda#f5f7fc038c611a25b22758c0a40d25c9 https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-14.4.0-h23af247_0.conda#4463210c2a16ff5d3bf7f502af23f1f4 https://conda.anaconda.org/conda-forge/linux-64/libspatialite-5.1.0-gpl_hab3fe16_120.conda#5947bdda89ced07ab0d8a5d6503082c3 +https://conda.anaconda.org/conda-forge/noarch/markdown-it-py-4.2.0-pyhd8ed1ab_0.conda#6d03368f2b2b0a5fb6839df53b2eb5e0 https://conda.anaconda.org/conda-forge/noarch/nodeenv-1.10.0-pyhd8ed1ab_0.conda#eb52d14a901e23c39e9e7b4a1a5c015f https://conda.anaconda.org/conda-forge/linux-64/numpy-2.5.2-py314h2b28147_0.conda#d77631addad93399a90157d2c597e7f3 https://conda.anaconda.org/conda-forge/noarch/partd-1.4.2-pyhd8ed1ab_0.conda#0badf9c54e24cecfb0ad2f99d680c163 @@ -252,6 +254,7 @@ https://conda.anaconda.org/conda-forge/linux-64/libegl-devel-1.7.0-ha4b6fd6_5.co https://conda.anaconda.org/conda-forge/linux-64/libgdal-core-3.13.3-h5fdb907_0.conda#c0723084e1315a5247d5a033d9eb5522 https://conda.anaconda.org/conda-forge/linux-64/libharfbuzz-devel-14.4.0-h23af247_0.conda#26e37e05324d7bb723350d50b33057fc https://conda.anaconda.org/conda-forge/linux-64/libraqm-0.11.0-h6406941_0.conda#3ac89a48d224409739dbf6200e524373 +https://conda.anaconda.org/conda-forge/noarch/mdit-py-plugins-0.6.1-pyhd8ed1ab_0.conda#ad6821df7a98510117db06e9a833281f https://conda.anaconda.org/conda-forge/linux-64/mo_pack-0.3.1-py314hc02f841_2.conda#55ac6d85f5dd8ec5e9919e7762fcb31a https://conda.anaconda.org/conda-forge/linux-64/pandas-3.0.5-py314hb4ffadd_1.conda#15cd0f375b79b47c4048e44396e6e6f1 https://conda.anaconda.org/conda-forge/linux-64/pango-1.58.2-hda50119_0.conda#6a2822aaf9a34ac3708904a47ff3dd7e @@ -294,6 +297,7 @@ https://conda.anaconda.org/conda-forge/noarch/adwaita-icon-theme-49.0-unix_0.con https://conda.anaconda.org/conda-forge/linux-64/esmf-8.9.1-nompi_h39d0119_4.conda#074ca9356f9e916ec51ad6ab47fd55ba https://conda.anaconda.org/conda-forge/noarch/esmpy-8.9.1-pyhdfbf58e_0.conda#0155d4a55415cda7f7f09bc0b7760f51 https://conda.anaconda.org/conda-forge/linux-64/graphviz-14.1.2-h8b86629_0.conda#341fc61cfe8efa5c72d24db56c776f44 +https://conda.anaconda.org/conda-forge/noarch/myst-parser-5.1.0-pyhd8ed1ab_0.conda#580615e59fc7c07741e4d2ab052cfc8b https://conda.anaconda.org/conda-forge/noarch/pydata-sphinx-theme-0.19.0-pyhcf101f3_0.conda#87921f66a4dc56ce92e4ff13be5f63dc https://conda.anaconda.org/conda-forge/noarch/sphinx-copybutton-0.5.2-pyhd8ed1ab_1.conda#bf22cb9c439572760316ce0748af3713 https://conda.anaconda.org/conda-forge/noarch/sphinx-data-viewer-0.1.5-pyhd8ed1ab_1.conda#fbe6437c6899c54e52f9258518035aca diff --git a/requirements/py312.yml b/requirements/py312.yml index 2683ed107a..dc249dcbbf 100644 --- a/requirements/py312.yml +++ b/requirements/py312.yml @@ -52,6 +52,7 @@ dependencies: - requests # Documentation dependencies. + - myst-parser - sphinx <8.2 - sphinxcontrib-apidoc - sphinx-copybutton diff --git a/requirements/py313.yml b/requirements/py313.yml index 1bfbeb3ba7..f5fd30452d 100644 --- a/requirements/py313.yml +++ b/requirements/py313.yml @@ -52,6 +52,7 @@ dependencies: - requests # Documentation dependencies. + - myst-parser - sphinx <8.2 - sphinxcontrib-apidoc - sphinx-copybutton diff --git a/requirements/py314.yml b/requirements/py314.yml index 3f8e3d81ee..f0fba8146e 100644 --- a/requirements/py314.yml +++ b/requirements/py314.yml @@ -52,6 +52,7 @@ dependencies: - requests # Documentation dependencies. + - myst-parser - sphinx <8.2 - sphinxcontrib-apidoc - sphinx-copybutton From c0dc1d8327b21ead78ea62dc82e1dccf38a8d27f Mon Sep 17 00:00:00 2001 From: Bill Little Date: Fri, 11 Sep 2026 11:07:04 +0100 Subject: [PATCH 2/2] [Agentic] Implementation plan: hashing substrate (merge/concatenate roadmap row 1) (#7275) * Credit Claude for agentic changelog entries Attribution should say who did the work. The pull request record already shows who authored and reviewed it, so crediting the human in the changelog as well overstates their hand in it. Records the rule in AGENTS.md and applies it to the two news fragments from #7274, which are merged but unreleased. The ``:user:`` role resolves to https://github.com/%s, and github.com/claude is Anthropic's own account, so the rendered link is correct. Co-Authored-By: Claude Opus 5 * Add the implementation plan for the hashing substrate Covers row 1 of the merge/concatenate roadmap: moving the array hashing machinery out of _concatenate.py into a new private _combine_common.py. Records the convention that a plan ships as its own pull request, ahead of the implementation it describes. A reviewer opening the implementation pull request should see a diff they can hold in their head; a plan bundled alongside is not evidence about the change, and makes a small diff look like a large one. Co-Authored-By: Claude Opus 5 * Add changelog fragment Co-Authored-By: Claude Opus 5 * Correct the hashing substrate plan from implementation findings Three corrections, all found by executing the plan rather than reading it. The import block was wrong. `from __future__ import annotations` makes `Mapping` annotation-only in Ruff's eyes, so `TC003` requires it behind `TYPE_CHECKING`; `_concatenate.py` has no `__future__` import, so the rule never fires there and copying the import across unchanged fails lint. The move is not quite mechanical. Dropping the underscore from `_array_id` collides with three local bindings inside `compute_hashes` itself, not just the five call sites the plan already anticipated in `_concatenate.py`. Spell out the three, and say why the exception to "do not otherwise touch the moved code" is narrower than it looks -- every one of the eight renames exists only to clear the unprefixed name's path, so conceding decision 6 retires all of them at once. Verification 1 therefore no longer expects an empty diff. It expects two hunks and three lines, which is a stronger check than "identical": it names what may differ instead of asserting that nothing does. The `\b` anchors stay, with an accurate reason -- they change nothing over lines 305-541, but protect `bound_array_id` if the range is ever widened. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 --- AGENTS.md | 6 +- changelog/7274.dependency.rst | 2 +- changelog/7274.doc.rst | 2 +- changelog/7275.internal.rst | 3 + .../plans/2026-09-10-hashing-substrate.md | 680 ++++++++++++++++++ docs/src/developers_guide/plans/README.md | 8 + .../2026-09-10-merge-concatenate-design.md | 5 +- 7 files changed, 699 insertions(+), 7 deletions(-) create mode 100644 changelog/7275.internal.rst create mode 100644 docs/src/developers_guide/plans/2026-09-10-hashing-substrate.md diff --git a/AGENTS.md b/AGENTS.md index 9e8ac0b487..57bbc51abf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -155,7 +155,9 @@ make lockfiles # python tools/update_lockfiles.py -o requirements/locks require push-disabled, which enforces this. - **Attribute agentic work clearly.** Say in the body of every pull request and issue that it is agentic, and end commit messages with the `Co-Authored-By` - trailer. Never let agentic work read as hand-written. + trailer. In changelog fragments the contributor cited is always + ``:user:`claude` `` — Anthropic's account — never the human raising the pull + request. Never let agentic work read as hand-written. - Label such pull requests and issues `Agentic` and `Type: Feature Branch`, plus whichever `Feature: …` label fits the subject. - Use the pull request template; keep changes focused and avoid unrelated @@ -184,8 +186,6 @@ make lockfiles # python tools/update_lockfiles.py -o requirements/locks require via `cube.metadata`. 9. **Cube immutability**: operations return new cubes; `cube.data[...] = x` will not affect the cube. -10. **Environment reproducibility**: set `NPY_DISABLE_CPU_FEATURES` as above for - consistent results. ## ⚠️ Meta-Instruction: Auto-Update Rule - **Trigger**: You MUST automatically update this `AGENTS.md` file if a new diff --git a/changelog/7274.dependency.rst b/changelog/7274.dependency.rst index 812d965ab1..297a494904 100644 --- a/changelog/7274.dependency.rst +++ b/changelog/7274.dependency.rst @@ -1,3 +1,3 @@ -:user:`bjlittle` added `myst-parser `_ as a +:user:`claude` added `myst-parser `_ as a documentation dependency, so that developer design specs may be authored in MyST Markdown. All other documentation remains reStructuredText. diff --git a/changelog/7274.doc.rst b/changelog/7274.doc.rst index 19000e671b..3946e1175a 100644 --- a/changelog/7274.doc.rst +++ b/changelog/7274.doc.rst @@ -1,4 +1,4 @@ -:user:`bjlittle` added a design spec for a programme of work to make +:user:`claude` added a design spec for a programme of work to make :meth:`~iris.cube.CubeList.merge` and :meth:`~iris.cube.CubeList.concatenate` more robust, efficient, configurable, extensible, lazy and lenient, along with a new :ref:`developers_guide_specs` section of the Developers Guide to publish it diff --git a/changelog/7275.internal.rst b/changelog/7275.internal.rst new file mode 100644 index 0000000000..490d566833 --- /dev/null +++ b/changelog/7275.internal.rst @@ -0,0 +1,3 @@ +:user:`claude` added the implementation plan for row 1 of the merge and +concatenate design spec roadmap, and established that a plan ships as its own +pull request ahead of the implementation it describes. (:pull:`7274`) diff --git a/docs/src/developers_guide/plans/2026-09-10-hashing-substrate.md b/docs/src/developers_guide/plans/2026-09-10-hashing-substrate.md new file mode 100644 index 0000000000..6dd7527e10 --- /dev/null +++ b/docs/src/developers_guide/plans/2026-09-10-hashing-substrate.md @@ -0,0 +1,680 @@ +# Hashing Substrate Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Move the array-hashing machinery out of `lib/iris/_concatenate.py` into +a new private module `lib/iris/_combine_common.py`, so that `_merge.py` can reach +it without importing `_concatenate.py`. + +**Architecture:** `_combine_common.py` is a flat private module, sibling to +`_merge.py`, `_concatenate.py`, `_combine.py` and `_lazy_data.py`. It sits +*below* both engines and imports nothing from Iris at runtime — the only Iris +names it needs are annotations on `array_id`, guarded by `TYPE_CHECKING`. That +import-freedom is the whole point of the module: it is what makes the merge-side +import in PR 4 risk-free. Everything here is a move; there is no behaviour +change whatsoever. + +**Tech Stack:** Python 3.12+, NumPy, Dask, xxhash, pytest, Ruff, towncrier. + +**Spec:** [`../specs/2026-09-10-merge-concatenate-design.md`](../specs/2026-09-10-merge-concatenate-design.md) +— this plan implements **row 1** of the roadmap in merge spec §6, per merge +spec §5.1. + +## How this ships + +**This plan ships as its own pull request, ahead of the implementation.** The +implementation pull request then carries only the code change under review. + +The reason is reviewer attention, and it is worth stating plainly because it +governs every plan in this programme. A core developer opening the +implementation pull request should see a diff they can hold in their head: a +module moved, an import repointed, a test added. A four-hundred-line plan +describing how an agent was told to do it is not evidence about the change — +it is homework handed in alongside the work, and it makes a small, easily +approved diff look like a large one. Keeping the two apart means the +implementation pull request can be reviewed on its merits by someone who has +never read this file and does not need to. + +Consequences for the executor: + +- Programme bookkeeping — updating this plan, the roadmap row, the decision + register — is **not** part of the implementation pull request, with one + exception: Task 3's changelog fragment and roadmap row, which cannot be + written before the implementation pull request number exists. +- If implementation turns up something that makes this plan wrong, fix the plan. + A plan is frozen when its work merges, not while the work is being done. Put + the fix on the plan branch, not the implementation branch, so the + implementation diff stays clean. + +## Global Constraints + +- **The substrate imports nothing from Iris at runtime.** Annotations that need + Iris names go behind `if TYPE_CHECKING:`, which requires + `from __future__ import annotations`. (merge spec §5.1) +- **No behaviour change.** PR 1 is part of the proving tranche (PRs 1–4), which + "end with a measured speedup against a committed baseline and no behaviour + change whatsoever". Every existing test must pass unmodified except for the + module path it imports from. (merge spec §6) +- **Names lose their underscore prefix** on the move. The module is already + private; a leading underscore on every name inside it is noise, and + `_lazy_data.py` is the precedent. (merge spec §6, row 1). This is **decision 6 + of the register in merge spec §6, and it is still open** — it makes PR 1 a move + *and* a rename, which a reviewer may reasonably object to. The location is what + matters, not the name, so the rename is cheap to concede. Say so in the pull + request body rather than waiting to be asked. +- **Target branch is `upstream/greenfield`, never `main`.** Labels: + `Feature: Merge/Concatenate`, `Type: Feature Branch`, `Agentic`. The pull + request body must say plainly that the work is agentic and cite the spec row. + (merge spec §6, cross-cutting rules) +- **Changelog fragments cite `` :user:`claude` ``**, never the human raising the + pull request. (root `AGENTS.md`, "Contribution Workflow") +- **Ruff, 88-character lines.** Every new Python file starts with the four-line + Iris copyright header. +- Line numbers in this plan are accurate as of `greenfield` at `7689b98f6`. + +--- + +### Task 1: Create the substrate and move the hashing machinery + +This task is atomic and cannot be split: creating the new module without +removing the original leaves duplicated definitions, and removing the original +first breaks `_concatenate.py`. The move is verbatim — the plan gives full text +for the genuinely new header, and an exact mechanical recipe for the block being +relocated, because retyping 237 lines of working code into a plan is pure +transcription risk. + +**Files:** +- Create: `lib/iris/_combine_common.py` +- Create: `lib/iris/tests/unit/combine_common/__init__.py` +- Move: `lib/iris/tests/unit/concatenate/test_hashing.py` → + `lib/iris/tests/unit/combine_common/test_hashing.py` +- Modify: `lib/iris/_concatenate.py` — delete lines 305–543; adjust imports at + lines 7–25; update seven call sites at lines 604, 608, 623, 1141, 1229, 1230, + 1233 + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: `iris._combine_common` exporting + + ```python + ArrayHash # namedtuple("ArrayHash", ["value", "chunks"]) with __eq__ + array_id(coord: DimCoord | AuxCoord | AncillaryVariable | CellMeasure, + bound: bool) -> str + compute_hashes(arrays: Mapping[str, np.ndarray | da.Array]) + -> dict[str, ArrayHash] + hash_array(a: da.Array | np.ndarray) -> np.int64 + ``` + + plus module-internal `hash_ndarray`, `hash_chunk`, `hash_aggregate`. + Task 2 relies on the module path `iris._combine_common`; PR 4 relies on all + four exported names. + +- [ ] **Step 1: Create the test package** + +Create `lib/iris/tests/unit/combine_common/__init__.py`, matching the style of +the sibling `lib/iris/tests/unit/combine/__init__.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Unit tests for the :mod:`iris._combine_common` module.""" +``` + +- [ ] **Step 2: Move the test file and repoint it** + +```bash +git mv lib/iris/tests/unit/concatenate/test_hashing.py \ + lib/iris/tests/unit/combine_common/test_hashing.py +``` + +Then make exactly these four edits to the moved file — nothing else changes, all +25 parametrised cases stay as they are: + +| Line | From | To | +|---|---|---| +| 5 | `"""Test array hashing in :mod:`iris._concatenate`."""` | `"""Test array hashing in :mod:`iris._combine_common`."""` | +| 11 | `from iris import _concatenate` | `from iris import _combine_common` | +| 76, 90 | `_concatenate._compute_hashes(` | `_combine_common.compute_hashes(` | +| 95, 96, 103 | `_concatenate._ArrayHash(` | `_combine_common.ArrayHash(` | + +- [ ] **Step 3: Run the moved tests to verify they fail** + +Run: `pytest lib/iris/tests/unit/combine_common/test_hashing.py -x -q` +Expected: collection error — `ImportError: cannot import name '_combine_common' +from 'iris'` + +- [ ] **Step 4: Create the substrate module** + +Create `lib/iris/_combine_common.py` with exactly this header: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Machinery shared by cube merge and cube concatenate. + +This module is the substrate beneath :mod:`iris._merge` and +:mod:`iris._concatenate`. It deliberately imports nothing from Iris at +runtime, so that either engine may import it without any risk of a circular +import. The only Iris names it requires are annotations, which are guarded +by :data:`typing.TYPE_CHECKING`. + +""" + +from __future__ import annotations + +from collections import namedtuple +import itertools +from typing import TYPE_CHECKING, Any + +import dask +import dask.array as da +import numpy as np +from xxhash import xxh3_64 + +if TYPE_CHECKING: + from collections.abc import Mapping + + from iris.coords import AncillaryVariable, AuxCoord, CellMeasure, DimCoord + +# Restrict the names imported from this namespace. +__all__ = ["ArrayHash", "array_id", "compute_hashes", "hash_array"] +``` + +`Mapping` is annotation-only, and `from __future__ import annotations` is what +makes that fact visible to Ruff — so `TC003` demands it sit behind +`TYPE_CHECKING` alongside the Iris imports. `_concatenate.py` has no `__future__` +import, so the rule never fires there and the original module scopes `Mapping` +normally; copying that import across unchanged fails lint. + +Then append lines 305–541 of `lib/iris/_concatenate.py` **verbatim** — from +`def _hash_ndarray(a: np.ndarray) -> np.ndarray:` through +`return {k: _ArrayHash(*v) for k, v in hashes.items()}` — and apply exactly +these seven renames throughout the appended block, definitions and call sites +alike: + +| From | To | +|---|---| +| `_hash_ndarray` | `hash_ndarray` | +| `_hash_chunk` | `hash_chunk` | +| `_hash_aggregate` | `hash_aggregate` | +| `_hash_array` | `hash_array` | +| `_ArrayHash` | `ArrayHash` | +| `_array_id` | `array_id` | +| `_compute_hashes` | `compute_hashes` | + +`ArrayHash` keeps its `namedtuple("ArrayHash", ...)` typename, which was already +unprefixed. Do not otherwise touch the moved code — not the docstrings, not the +comments, not the `__eq__` that raises. Whether `ArrayHash.__eq__` should raise +at all is an open decision owned by PR 4 (merge spec §6, decision 9); this PR +changes nothing about it. + +There is one exception, forced by the `_array_id` → `array_id` rename. Three +lines inside `compute_hashes` already bind a *local* called `array_id`, which +coexisted with a function called `_array_id` but shadows one called `array_id`. +No scope in the module calls the function, so the shadowing is inert — but it is +the same trap Step 7 defuses in `_concatenate.py`, and leaving it in the module +that *defines* the name is worse than leaving it anywhere else. Rename those +three, and only those three: + +| Original | Becomes | +|---|---| +| `array_id, a = item` (in `group_key`) | `_, a = item` | +| `for array_id, rechunked in zip(array_ids, rechunked_arrays):` | `for key, rechunked in zip(array_ids, rechunked_arrays):` | +| `hashes[array_id] = (hash_array(rechunked), chunks)` | `hashes[key] = (hash_array(rechunked), chunks)` | + +`array_ids` (plural) does not collide and stays as it is. These three lines are +the whole of the difference between the relocated code and the original modulo +the seven renames, which is what Verification 1 checks. + +- [ ] **Step 5: Run the moved tests to verify they pass** + +Run: `pytest lib/iris/tests/unit/combine_common/test_hashing.py -q` +Expected: `59 passed` — the same count the file gives on `greenfield` before the +move (25 `test_compute_hashes` cases, 32 `test_compute_hashes_vs_array_equal` +cases drawn from `iris.tests.unit.util.test_array_equal.TEST_CASES`, and the two +`raises` tests). A different count means the parametrisation was disturbed. + +- [ ] **Step 6: Delete the moved block from `_concatenate.py`** + +Delete lines 305–543 inclusive — the block plus its two trailing blank lines, +which leaves the two blank lines at 303–304 as the separator before +`def concatenate(`. + +- [ ] **Step 7: Repoint `_concatenate.py` at the substrate** + +In the import block, delete these four now-dead imports: + +```python +import itertools # line 9 - only used by _compute_hashes +from typing import Any # line 10 - only used by _ArrayHash.__eq__ +import dask # line 13 - only used by _compute_hashes +from xxhash import xxh3_64 # line 16 - only used by _hash_ndarray +``` + +Keep `import dask.array as da` (still used at lines 1571–1573) and keep +`from iris.coords import AncillaryVariable, AuxCoord, CellMeasure, DimCoord` +(still used at lines 605, 1228, 1232). Add, in Ruff's import order: + +```python +from iris._combine_common import ArrayHash, array_id, compute_hashes +``` + +Then update the seven remaining references: + +| Line | From | To | +|---|---|---| +| 604 | `array_id = _array_id(coord, bound=False)` | `points_id = array_id(coord, bound=False)` | +| 606 | `arrays[array_id] = coord.core_points()` | `arrays[points_id] = coord.core_points()` | +| 608 | `bound_array_id = _array_id(coord, bound=True)` | `bounds_id = array_id(coord, bound=True)` | +| 609 | `arrays[bound_array_id] = coord.core_bounds()` | `arrays[bounds_id] = coord.core_bounds()` | +| 611 | `arrays[array_id] = coord.core_data()` | `arrays[points_id] = coord.core_data()` | +| 623 | `hashes = _compute_hashes(arrays)` | `hashes = compute_hashes(arrays)` | +| 1141 | `hashes: Mapping[str, _ArrayHash],` | `hashes: Mapping[str, ArrayHash],` | +| 1229 | `) -> tuple[_ArrayHash, ...]:` | `) -> tuple[ArrayHash, ...]:` | +| 1230 | `array_id = _array_id(coord, bound=False)` | `points_id = array_id(coord, bound=False)` | +| 1231 | `result = [hashes[array_id]]` | `result = [hashes[points_id]]` | +| 1233 | `bound_array_id = _array_id(coord, bound=True)` | `bounds_id = array_id(coord, bound=True)` | +| 1234 | `result.append(hashes[bound_array_id])` | `result.append(hashes[bounds_id])` | + +**Name collision — this is the one non-mechanical part of the move.** Lines 604 +and 1230 currently assign to a *local variable* called `array_id`. After the +rename that name is also the imported function, and a local assignment makes it +local for the whole scope, so the call on the right-hand side raises +`UnboundLocalError` at run time. Ruff does not flag it and the type checker does +not flag it; only the tests do. Rename the locals to `points_id` and `bounds_id` +in both enclosing scopes, so that the two helpers read in full as: + +At `add_coords` (`_concatenate.py:601-611` before the deletion, `364-374` +after), the body becomes: + +```python + def add_coords(cube_signature: _CubeSignature, coord_type: str) -> None: + for coord_and_dims in getattr(cube_signature, coord_type): + coord = coord_and_dims.coord + points_id = array_id(coord, bound=False) + if isinstance(coord, (DimCoord, AuxCoord)): + arrays[points_id] = coord.core_points() + if coord.has_bounds(): + bounds_id = array_id(coord, bound=True) + arrays[bounds_id] = coord.core_bounds() + else: + arrays[points_id] = coord.core_data() +``` + +At the `_ProtoCube.register` helper (`_concatenate.py:1226-1235` before the +deletion), the body becomes: + +```python + def get_hashes( + coord: DimCoord | AuxCoord | AncillaryVariable | CellMeasure, + ) -> tuple[ArrayHash, ...]: + points_id = array_id(coord, bound=False) + result = [hashes[points_id]] + if isinstance(coord, (DimCoord, AuxCoord)) and coord.has_bounds(): + bounds_id = array_id(coord, bound=True) + result.append(hashes[bounds_id]) + return tuple(result) +``` + +Confirm the exact pre-edit text of that second helper before rewriting it: + +Run: `sed -n '1226,1236p' lib/iris/_concatenate.py` + +- [ ] **Step 8: Verify no stale references remain** + +Run: + +```bash +grep -rn '_hash_ndarray\|_hash_chunk\|_hash_aggregate\|_hash_array\|_ArrayHash\|_array_id\|_compute_hashes' \ + lib/ benchmarks/ docs/src --include='*.py' --include='*.rst' --include='*.md' \ + | grep -v docs/src/_build +``` + +Expected: no output. + +- [ ] **Step 9: Run the concatenate suite** + +Run: `pytest -n auto lib/iris/tests/unit/concatenate lib/iris/tests/unit/combine_common lib/iris/tests/unit/combine -q` +Expected: PASS, no failures, no errors. + +- [ ] **Step 10: Run lint and format** + +Run: + +```bash +ruff check lib/iris && ruff format lib/iris && ruff format --check lib/iris +``` + +Expected: all pass. Re-run Step 9 if `ruff format` rewrote anything. + +- [ ] **Step 11: Commit** + +```bash +git add lib/iris/_combine_common.py lib/iris/_concatenate.py \ + lib/iris/tests/unit/combine_common/ +git commit -m "$(cat <<'EOF' +Move array hashing into a shared _combine_common substrate + +Lifts the array-hashing machinery out of _concatenate.py into a new private +module, lib/iris/_combine_common.py, so that _merge.py can reach it without +importing _concatenate.py. The substrate imports nothing from Iris at +runtime; the coordinate annotations on array_id sit behind TYPE_CHECKING. + +Names lose their underscore prefix on the move, the module itself already +being private. The two local variables that would have shadowed the imported +array_id are renamed to points_id and bounds_id. + +Pure relocation: no behaviour change. Row 1 of the roadmap in +docs/src/developers_guide/specs/2026-09-10-merge-concatenate-design.md. + +Co-Authored-By: Claude Opus 5 +EOF +)" +``` + +--- + +### Task 2: Lock the no-Iris-imports invariant with a test + +The substrate's import-freedom is an assertion the spec makes (merge spec §5.1) +and the reason the module exists at all. It is also exactly the kind of property +that rots silently: any future contributor adding `from iris.cube import Cube` +for a type hint would break PR 4's premise without a single test failing. This +task makes that a test failure. + +A runtime check is not available — `import iris._combine_common` executes +`iris/__init__.py` first, so `sys.modules` cannot distinguish the substrate's own +imports from the package's. The check is therefore static, over the module's AST. + +**Files:** +- Create: `lib/iris/tests/unit/combine_common/test_imports.py` + +**Interfaces:** +- Consumes: the module path `iris._combine_common` from Task 1. +- Produces: nothing later tasks rely on. + +- [ ] **Step 1: Write the failing test** + +Create `lib/iris/tests/unit/combine_common/test_imports.py`: + +```python +# Copyright Iris contributors +# +# This file is part of Iris and is released under the BSD license. +# See LICENSE in the root of the repository for full licensing details. +"""Test the import-freedom of :mod:`iris._combine_common`.""" + +import ast +from pathlib import Path + +from iris import _combine_common + + +class RuntimeImports(ast.NodeVisitor): + """Collect modules imported at runtime, skipping ``TYPE_CHECKING`` blocks.""" + + def __init__(self): + self.modules: set[str] = set() + + def visit_If(self, node: ast.If) -> None: + guard = node.test + type_checking = (isinstance(guard, ast.Name) and guard.id == "TYPE_CHECKING") or ( + isinstance(guard, ast.Attribute) and guard.attr == "TYPE_CHECKING" + ) + if type_checking: + # Only the ``else`` branch runs at runtime. + for statement in node.orelse: + self.visit(statement) + else: + self.generic_visit(node) + + def visit_Import(self, node: ast.Import) -> None: + self.modules.update(alias.name for alias in node.names) + + def visit_ImportFrom(self, node: ast.ImportFrom) -> None: + if node.level: + # A relative import is necessarily an Iris import. + self.modules.add("." * node.level + (node.module or "")) + elif node.module is not None: + self.modules.add(node.module) + + +def test_no_runtime_iris_imports(): + """The substrate sits below merge and concatenate, so imports no Iris.""" + source = Path(_combine_common.__file__).read_text(encoding="utf-8") + visitor = RuntimeImports() + visitor.visit(ast.parse(source)) + offenders = { + module + for module in visitor.modules + if module == "iris" or module.startswith(("iris.", ".")) + } + assert offenders == set(), ( + "iris._combine_common must import nothing from Iris at runtime; " + f"found {sorted(offenders)}. Put annotation-only imports behind " + "`if TYPE_CHECKING:`." + ) + + +def test_type_checking_imports_are_detected(): + """Guard the guard: a runtime Iris import must be seen as one.""" + source = "\n".join( + [ + "from typing import TYPE_CHECKING", + "if TYPE_CHECKING:", + " from iris.coords import DimCoord", + "from iris.cube import Cube", + ] + ) + visitor = RuntimeImports() + visitor.visit(ast.parse(source)) + assert "iris.cube" in visitor.modules + assert "iris.coords" not in visitor.modules +``` + +The second test is not ceremony. Without it, a `RuntimeImports` visitor that +silently collected nothing at all — the classic way an AST check dies — would +leave the first test passing forever and the invariant unguarded. + +- [ ] **Step 2: Run the test to verify the guard test fails first** + +Temporarily add `import iris.cube` at the top of `lib/iris/_combine_common.py`, +below the copyright header. + +Run: `pytest lib/iris/tests/unit/combine_common/test_imports.py -q` +Expected: `test_no_runtime_iris_imports` FAILS with +`AssertionError: iris._combine_common must import nothing from Iris at runtime; +found ['iris.cube']`, and `test_type_checking_imports_are_detected` PASSES. + +- [ ] **Step 3: Remove the temporary import and re-run** + +Delete the `import iris.cube` line just added. + +Run: `pytest lib/iris/tests/unit/combine_common/test_imports.py -q` +Expected: PASS, 2 passed. + +- [ ] **Step 4: Lint and commit** + +```bash +ruff check lib/iris && ruff format --check lib/iris +git add lib/iris/tests/unit/combine_common/test_imports.py +git commit -m "$(cat <<'EOF' +Test that the combine substrate imports nothing from Iris + +The import-freedom of _combine_common is the reason the module exists, and +is the premise of the merge-side import that follows in a later pull request. +It is also the kind of property that rots without a test: a type hint added +carelessly would break it in silence. + +Checks the module's AST rather than sys.modules, since importing a submodule +of iris necessarily executes iris/__init__.py first. + +Co-Authored-By: Claude Opus 5 +EOF +)" +``` + +--- + +### Task 3: Open the pull request, then land the fragment and roadmap row + +A towncrier fragment is named `..rst`, and the roadmap row cites +`{pull}`NNNN``. Neither number exists until the pull request is open, so this +task deliberately runs last and spans the push. + +**Files:** +- Create: `changelog/.internal.rst` +- Modify: `docs/src/developers_guide/specs/2026-09-10-merge-concatenate-design.md` + — roadmap row 1 (merge spec §6) + +**Interfaces:** +- Consumes: the commits from Tasks 1–2. +- Produces: nothing. + +- [ ] **Step 1: Push the branch** + +```bash +git push -u origin pr1-hashing-substrate +``` + +- [ ] **Step 2: Open the pull request against `greenfield`** + +```bash +gh pr create --repo SciTools/iris --base greenfield \ + --head bjlittle:pr1-hashing-substrate \ + --title "[Agentic] Move array hashing into a shared _combine_common substrate" \ + --body-file - +``` + +Body must: + +- State plainly that the work is agentic. +- Cite the spec and its row — "row 1 of the roadmap in the merge/concatenate + design spec, per §5.1" — and reference `#7274` for the spec itself. +- Confirm the change is a pure relocation with no behaviour change, and point at + the two verification checks below. +- Flag the `array_id` local-variable renames as the only non-mechanical part of + the move — the five call sites in `_concatenate.py` and the three shadowed + bindings inside `compute_hashes` — so a reviewer knows where to look. Nothing + else in the diff needs reading closely, and saying so is what makes the rest + of the review cheap. +- **Put decision 6 to the reviewer explicitly**: dropping the underscore prefix + makes this a move *and* a rename. Offer to drop the rename and keep the + underscores if they would rather review a pure `git mv`. Conceding costs + nothing — the module's location is what the rest of the programme depends on. + Note that keeping `_array_id` would also retire all eight local renames above, + since every one of them exists only to get out of the unprefixed name's way. +Keep the body about the code. The plan, the register and the programme +bookkeeping ship separately — see "How this ships" above. + +- [ ] **Step 3: Apply the labels** + +```bash +PR=$(gh pr view --repo SciTools/iris --json number --jq .number \ + --head bjlittle:pr1-hashing-substrate) +gh api -X POST "repos/SciTools/iris/issues/${PR}/labels" \ + -f 'labels[]=Feature: Merge/Concatenate' \ + -f 'labels[]=Type: Feature Branch' \ + -f 'labels[]=Agentic' +``` + +Use `gh api` rather than `gh pr edit --add-label`: the repository's labeler +workflow races `gh pr edit` and the labels do not stick. Verify with +`gh pr view "$PR" --repo SciTools/iris --json labels`. + +- [ ] **Step 4: Write the changelog fragment** + +Create `changelog/.internal.rst` with the real number substituted: + +```rst +:user:`claude` moved the array hashing machinery shared by +:meth:`~iris.cube.CubeList.merge` and :meth:`~iris.cube.CubeList.concatenate` +into a new private ``iris._combine_common`` module, in preparation for merge +adopting it. No behaviour change. +``` + +- [ ] **Step 5: Update roadmap row 1** + +In `docs/src/developers_guide/specs/2026-09-10-merge-concatenate-design.md`, +change row 1 of the §6 table's final column from `not started` to +`✅ complete ({pull}`NNNN`)`, substituting the real number. Change nothing else +in the table — rows 2–12 are updated by their own pull requests. + +Leave decisions 6 and 7 in the register alone. Both are owned by PR 1 but +resolved by the *reviewers' response* to it, not by opening it: 6 is whether the +rename survives review, 7 is whether the programme proceeds at all. Record their +outcomes in a follow-up once that response exists. + +- [ ] **Step 6: Verify the fragment and the docs build** + +```bash +towncrier build --draft --version 3.17.0 | grep -A3 '_combine_common' +cd docs && make html +``` + +Expected: the fragment renders under "💼 Internal"; the docs build succeeds with +no new warnings. + +- [ ] **Step 7: Run the full unit suite before handing over for review** + +Run: `pytest -n auto lib/iris/tests/unit -q` +Expected: PASS. Any failure here is a genuine regression — this pull request +changes no behaviour. + +- [ ] **Step 8: Commit and push** + +```bash +git add changelog/ docs/src/developers_guide/specs/ +git commit -m "$(cat <<'EOF' +Add changelog fragment and mark roadmap row 1 complete + +Co-Authored-By: Claude Opus 5 +EOF +)" +git push +``` + +--- + +## Verification + +The whole of PR 1 is a relocation, so the strongest evidence that it is correct +is that nothing changed. Two checks make that concrete, and both should be +quoted in the pull request body: + +1. **The moved code is byte-identical apart from the renames.** From the branch: + + ```bash + git show upstream/greenfield:lib/iris/_concatenate.py | sed -n '305,541p' \ + | sed -e 's/\b_hash_ndarray\b/hash_ndarray/g' -e 's/\b_hash_chunk\b/hash_chunk/g' \ + -e 's/\b_hash_aggregate\b/hash_aggregate/g' -e 's/\b_hash_array\b/hash_array/g' \ + -e 's/\b_ArrayHash\b/ArrayHash/g' -e 's/\b_array_id\b/array_id/g' \ + -e 's/\b_compute_hashes\b/compute_hashes/g' > /tmp/expected.py + sed -n '/^def hash_ndarray/,$p' lib/iris/_combine_common.py > /tmp/actual.py + diff -u /tmp/expected.py /tmp/actual.py + ``` + + Expected: the three de-shadowing lines from Step 4 and nothing else — two + hunks, three changed lines. Anything further is unintended; if `ruff format` + reflowed something, the diff shows exactly what and why, so inspect it rather + than accepting it. + + Within lines 305–541 the `\b` word boundaries change nothing, because no one + of the seven names is a substring of another. They are there for the moment + someone widens the range to take in the call sites, where `bound_array_id` + *does* contain `_array_id` and an unanchored substitution would quietly + produce `boundarray_id`. + +2. **The concatenate suite passes unmodified.** No test under + `lib/iris/tests/unit/concatenate/` is edited by this pull request; only + `test_hashing.py` moves out of it. + + ```bash + git diff upstream/greenfield --stat -- lib/iris/tests/unit/concatenate/ + ``` + + Expected: `test_hashing.py` shown as deleted, nothing else touched. diff --git a/docs/src/developers_guide/plans/README.md b/docs/src/developers_guide/plans/README.md index bcc33d550e..58a6ae7a07 100644 --- a/docs/src/developers_guide/plans/README.md +++ b/docs/src/developers_guide/plans/README.md @@ -7,4 +7,12 @@ steps agreed at the time it was written, and is frozen once the work it describes has merged. Plans are tracked in the repository for provenance, but are excluded from the Sphinx build via `exclude_patterns` in `docs/src/conf.py`. +A plan ships as its **own** pull request, ahead of the implementation it +describes, so that the implementation pull request carries only the change under +review. A reviewer opening that pull request should see a diff they can hold in +their head; a plan bundled alongside is not evidence about the change, and makes +a small diff look like a large one. If implementation shows the plan to be +wrong, correct it on the plan's own branch — a plan is frozen when its work +merges, not while the work is under way. + Naming: `YYYY-MM-DD-.md`. diff --git a/docs/src/developers_guide/specs/2026-09-10-merge-concatenate-design.md b/docs/src/developers_guide/specs/2026-09-10-merge-concatenate-design.md index 9359f443fb..958dcae3b1 100644 --- a/docs/src/developers_guide/specs/2026-09-10-merge-concatenate-design.md +++ b/docs/src/developers_guide/specs/2026-09-10-merge-concatenate-design.md @@ -397,8 +397,9 @@ item with genuine semantic debate in it. ## 6. Plan roadmap Twelve pull requests deliver the design. Each gets its own implementation plan in -`docs/src/developers_guide/plans/`, and a PR is merged before any PR that -*depends on it* is written. The dependencies form a partial order, not a chain: +`docs/src/developers_guide/plans/`, shipped as its own pull request ahead of the +implementation so that the implementation PR carries only the change under +review, and a PR is merged before any PR that *depends on it* is written. The dependencies form a partial order, not a chain: PRs 1, 2, 3 and 6 have no predecessors and may proceed in parallel. The ordering follows §5's layering — the substrate first (PR 1), then the seam that lets merge use it (PRs 2–3), then the payoff (PR 4), then the correctness and capability