[Agentic] Design spec: making merge and concatenate robust, efficient and configurable - #7274
Merged
Merged
Conversation
Specifies a twelve-PR programme to make cube merge and concatenate more robust, efficient, configurable, extensible, lazy and lenient, without a rewrite. Each PR is either a behaviour-preserving refactor or a standalone user-visible improvement; together they grow a shared substrate beneath both engines. Includes a confirmed diagnosis and fix for the separability bug open since 2017 (SciTools#2761), verified end-to-end against the reporter's case. Co-Authored-By: Claude Opus 5 <[email protected]>
Co-Authored-By: Claude Opus 5 <[email protected]>
Adopt the spec/plan convention used in bjlittle/tephpy. Agreed design specs now live in `docs/src/developers_guide/specs/` and are published as part of the documentation. They are written in MyST Markdown, are living documents revised as a design evolves, and each declares a citation prefix so its sections can be cited from issues and pull requests. Implementation plans live in the sibling `docs/src/developers_guide/plans/`. A plan is point-in-time and frozen once its pull request merges, so plans are tracked in the repository but excluded from the Sphinx build. This requires `myst-parser`, added to `requirements/py*.yml` and inserted into the three lock files following the precedent of SciTools#7095 and SciTools#7100. Markdown is used only for specs; all other documentation remains reStructuredText. The merge/concatenate design spec moves here from `docs/superpowers/specs/` and is reworked to the new conventions. Co-Authored-By: Claude Opus 5 <[email protected]>
The spec described a twelve-PR programme but had nowhere to record how far it had got, which assumptions it rested on, or what it had left open. Restructure §6 from a static table of work into the programme's live progress record: dependencies as a partial order rather than a chain, a status vocabulary, an "Outside the roadmap" list, and a numbered register of assumptions and open decisions - each item owned by the pull request that must resolve it. Add §10 for the questions the programme raises but does not answer, renumbering References to §11. Also exclude "_build" and "_static" from the Sphinx build. myst-parser makes ".md" a source suffix, so the README of each was being read as a document. Co-Authored-By: Claude Opus 5 <[email protected]>
This work is long-running partly because it is also an exercise in earning core-developer buy-in for agentic contributions, so the mechanics of how it reaches them are worth writing down rather than re-deciding each session. Add a "Contribution Workflow" section to AGENTS.md carrying the general rule: branches to the fork, pull requests to a feature branch rather than main, honest "Agentic" labelling, clear attribution, and ask before doing anything outward-facing when unsure. Drop the "Getting Help" link list to stay inside the file's 200-line budget - the links it held are all reachable from CONTRIBUTING.md. The programme-specific parts - base branch "greenfield" and the "Feature: Merge/Concatenate" label - go in the spec's cross-cutting rules instead, where a reviewer can see why these pull requests do not target main. Co-Authored-By: Claude Opus 5 <[email protected]>
Co-Authored-By: Claude Opus 5 <[email protected]>
The branch was cut from main at c508ac8; greenfield carries SciTools#7270 on top of that, which re-solved all three lock files. The lock additions here are re-derived against those fresh locks, and remain minimal - the myst-parser dependency closure and the input_hash, nothing else. The cube.py line references the spec cites are unaffected: SciTools#7270's changes are confined to rolling_window, some 4500 lines below the merge and concatenate methods. Co-Authored-By: Claude Opus 5 <[email protected]>
bjlittle
force-pushed
the
merge-concat-design
branch
from
September 10, 2026 21:29
cd3da7a to
6190899
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## greenfield #7274 +/- ##
===========================================
Coverage 90.35% 90.35%
===========================================
Files 93 93
Lines 25816 25816
Branches 4796 4796
===========================================
Hits 23327 23327
Misses 1709 1709
Partials 780 780 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Give every cited section its own section sign, and separate a run of them with a comma or a solidus rather than a dash. A dashed range such as "5.6-5.7" reads as a single citation carrying a stray suffix, so only its first section is unambiguously identified -- by a reader now, and by any cross-reference tooling later. Records the rule on the Design Specs index page, next to the citation prefix table it belongs with, and applies it to the one occurrence in the merge/concatenate spec. Co-Authored-By: Claude Opus 5 <[email protected]>
Contributor
⏱️ Performance Benchmark Report: a751ecfPerformance shiftsFull benchmark resultsGenerated by GHA run |
Contributor
⏱️ Performance Benchmark Report: 7f1c512Performance shiftsFull benchmark resultsGenerated by GHA run |
7 tasks
bjlittle
added a commit
that referenced
this pull request
Sep 11, 2026
…oadmap row 1) (#7275) * Credit Claude for agentic changelog entries Attribution should say who did the work. The pull request record already shows who authored and reviewed it, so crediting the human in the changelog as well overstates their hand in it. Records the rule in AGENTS.md and applies it to the two news fragments from #7274, which are merged but unreleased. The ``:user:`` role resolves to https://github.com/%s, and github.com/claude is Anthropic's own account, so the rendered link is correct. Co-Authored-By: Claude Opus 5 <[email protected]> * Add the implementation plan for the hashing substrate Covers row 1 of the merge/concatenate roadmap: moving the array hashing machinery out of _concatenate.py into a new private _combine_common.py. Records the convention that a plan ships as its own pull request, ahead of the implementation it describes. A reviewer opening the implementation pull request should see a diff they can hold in their head; a plan bundled alongside is not evidence about the change, and makes a small diff look like a large one. Co-Authored-By: Claude Opus 5 <[email protected]> * Add changelog fragment Co-Authored-By: Claude Opus 5 <[email protected]> * Correct the hashing substrate plan from implementation findings Three corrections, all found by executing the plan rather than reading it. The import block was wrong. `from __future__ import annotations` makes `Mapping` annotation-only in Ruff's eyes, so `TC003` requires it behind `TYPE_CHECKING`; `_concatenate.py` has no `__future__` import, so the rule never fires there and copying the import across unchanged fails lint. The move is not quite mechanical. Dropping the underscore from `_array_id` collides with three local bindings inside `compute_hashes` itself, not just the five call sites the plan already anticipated in `_concatenate.py`. Spell out the three, and say why the exception to "do not otherwise touch the moved code" is narrower than it looks -- every one of the eight renames exists only to clear the unprefixed name's path, so conceding decision 6 retires all of them at once. Verification 1 therefore no longer expects an empty diff. It expects two hunks and three lines, which is a stronger check than "identical": it names what may differ instead of asserting that nothing does. The `\b` anchors stay, with an accurate reason -- they change nothing over lines 305-541, but protect `bound_array_id` if the range is ever widened. Co-Authored-By: Claude Opus 5 <[email protected]> --------- Co-authored-by: Claude Opus 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request adds the agreed design spec for a programme of work to make
CubeList.mergeandCubeList.concatenatemore robust, efficient, configurable,extensible, lazy and lenient — plus the small amount of Sphinx plumbing needed to
publish specs at all.
It contains no library code changes. Nothing in
lib/iris/is touched. Thepurpose is to get the design in front of core developers before any
implementation lands, so the shape of the programme can be argued with while it
is still cheap to change.
What the spec proposes
Twelve small, independently reviewable pull requests. Each is either a
behaviour-preserving refactor or a standalone user-visible improvement, and
together they grow a shared substrate beneath the two engines — without anyone
having to agree to a rewrite up front. §6 is the live roadmap and will be updated
as each one lands; it also carries the assumptions and open decisions, each owned
by the PR that has to resolve it. §10 records the questions the programme raises
but does not answer.
Issues in scope: #2761 (separability), #5375 (diagnostics),
#6790 (merge/concatenate asymmetry), #7063 / #7241
(performance), and parts of #4446 / #5392 (leniency).
The alternative that was considered and rejected — specify a unified v4.0 engine
first, then build it — is preserved rather than discarded, at
bjlittle#333, with §4.3 recording why it is not
the starting point.
Reviewer notes
Three things worth knowing before you open the diff:
AGENTS.mdshows ~214 changed lines, but almost all of it is reflow. Thefile carries its own 200-line budget, so adding three sections meant
compacting what was already there. The genuinely new content is: the
"Documentation" paragraphs on where specs and plans live, "Lock-file
Maintenance", and "Contribution Workflow". The rest is the same rules, said
shorter. Reviewing the three new sections and skimming the reflow is a
reasonable use of your time; reading all 214 lines is not.
myst-parseris a new documentation dependency. Specs are MyST Markdown;everything else in
docs/stays reStructuredText, andconf.pysays so. Thelock files are edited minimally — new package URLs plus
input_hash— ratherthan fully re-solved, following the precedent of DOCS: Support for generating LLM friendy summary for the docs. #7095 and DOCS: Add sitemap support #7100,
so this does not bundle ~90 unrelated version bumps into a docs PR.
docs/src/developers_guide/plans/isexcluded from the Sphinx build via
exclude_patterns: a spec is a livingdocument, a plan is frozen once its PR merges, and only the former belongs on
the rendered site.
Docs build clean — the 6 remaining warnings are all pre-existing
SKIP_API=1artefacts.
pre-commit run --all-filespasses.Why this targets
greenfieldThe programme is long-running and deliberately incremental, so its parts should
land together as a coherent tranche rather than trickling into
mainone at atime.
greenfieldwas created for exactly this work.🤖 Attribution
This pull request is agentic work. It was researched, written and committed
by Claude Opus 5 working with @bjlittle, who
directed the investigation, made every design decision put to him, and reviewed
the result before it was raised. The commits carry
Co-Authored-Bytrailerssaying the same.
Being explicit about this is the point rather than a formality — part of what
this programme is testing is whether agentic contributions can be made reviewable
enough to be worth a core developer's time. Please review it at least as
sceptically as you would any other pull request. If the answer turns out to be
no, that is a useful result and we would rather hear it early.
Design decisions in the spec are @bjlittle's; errors in how they are written up
are Claude's.
Checklist
changelog/7274.doc.rstandchangelog/7274.dependency.rstextends the merge benchmark, deliberately kept separate)
myst-parseradded torequirements/py31{2,3,4}.ymland the matching lock files