Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Set Table primary key when copied indices are attached - #20362

Open
taldcroft wants to merge 1 commit into
astropy:mainfrom
taldcroft:table-primary-key-from-indices
Open

Set Table primary key when copied indices are attached#20362
taldcroft wants to merge 1 commit into
astropy:mainfrom
taldcroft:table-primary-key-from-indices

Conversation

@taldcroft

Copy link
Copy Markdown
Member

Summary

A Table built from columns that already carry an index, most commonly a column-name slice like t[["a", "b"]], keeps the index but has no primary key. Every consumer of primary_key then fails on it: .loc, .iloc, .loc_indices, repr(t.loc), and write(..., write_indices=True). The slice looks fully indexed (t2.indices is populated), so the failure is surprising and the error message points nowhere useful.

>>> t = Table({"a": [3, 1, 2], "b": [4, 5, 6]})
>>> t.add_index("a")
>>> t2 = t[["a", "b"]]
>>> t2.indices
[<SlicedIndex ...>]                                                 # both: the index is there
>>> t2.primary_key
None                                                                # main
('a',)                                                              # this branch
>>> t2.loc[1]
TypeError: list indices must be integers or slices, not NoneType   # main
<Row index=1> a b -> 1 5                                            # this branch

The cause is that primary_key is set only by add_index(), while Table.__init__ attaches copied indices to the new columns without ever setting it. This PR sets the primary key where those copied indices are attached, following the same "first index is primary" rule as add_index(), and additionally keeps the parent's primary key on a column slice when that index survived the slice. Row slices (t[1:], t[mask]) were never affected, since _new_from_slice already copies primary_key.

This is the astropy.table half of the fix for #20297; the astropy.timeseries half is #20361. The two are independent and either one alone fixes the aggregate_downsample example in that issue.

Related to #20297. Supersedes the astropy.table part of #20354.

AI disclosure

This PR includes AI-generated content using Claude Fable 5.1. I have carefully reviewed the content and fully understand the changes in astropy/table/table.py and astropy/table/tests/test_index.py.

  • I certify that I am human and take responsibility for the code and interactions with reviewers.

Details

Click to expand

Set primary_key whenever a new Table receives copied indices

1. How a table gets indices without a primary key

col_copy(col, copy_indices=True) deep-copies a column's SlicedIndex objects and re-points them at the new column, so Table([t["a"]]) and t[["a", "b"]] (which goes through the same constructor) produce columns with live, valid indices. Table._init_from_cols already walks these to deduplicate index objects shared between columns of a multi-column index, but nothing sets self.primary_key, which Table.__init__ initialised to None. Only add_index() sets it, and only when the table has no indices yet.

TableLoc._get_index_id_and_item uses self.table.primary_key as the index id, so None reaches TableIndices.__getitem__, which falls through to list.__getitem__(None).

2. Implementation

Table._init_from_cols, after the existing deduplication loop that fills index_dict (keyed by index id in column order):

if self.primary_key is None and index_dict:
    self.primary_key = next(iter(index_dict))

The first index in column order becomes primary, matching add_index(). This covers Table([indexed_col]), QTable(table_with_indices), and any other constructor path that copies indices. When the input is a Table, __init__ has already copied data.primary_key, so this only fills in a missing key and never overrides one.

Table.__getitem__, in the list-of-column-names branch:

if self.primary_key in [index.id for index in out.indices]:
    out.primary_key = self.primary_key

A reordered slice such as t[["b", "a"]] on a table with indices on both columns would otherwise get ("b",) from the rule above; this keeps the parent's ("a",). The membership test against the slice's actual indices means a stale key (one whose index was dropped by the slice, or one left behind by remove_indices, see below) is never propagated. None in [...] is False, so an unindexed parent is unaffected.

3. Things deliberately left alone

  • A multi-column index whose other column is dropped by the slice (t.add_index(["a", "b"]); t[["a", "c"]]) now becomes the primary key. .loc[1] on it raises IndexError: tuple index out of range, but the unsliced t.loc[1] raises the identical error for a composite index, so this is consistent rather than a regression. Filtering such indices out would be a separate policy change.
  • Table.remove_indices promises in its docstring to reassign the primary key when the primary index is removed but never does, leaving a stale key. That is pre-existing and not touched here; the __getitem__ guard above just avoids spreading it.
  • Table(t, copy_indices=False) already gives a table with a primary_key but no indices. Unchanged.

4. Tests

test_index.py::test_primary_key_from_copied_indices (fails on main, passes here) checks: t[["a", "b"]] has one index and primary_key == ("a",) with working .loc and .iloc; t[["b"]] has no index and no key; Table([t["a"]]) has a working .loc and repr(t4.loc) no longer raises; with a second index on b, t[["b", "a"]] keeps ("a",), t[["b"]] gets ("b",), and Table([t["b"], t["a"]]) gets ("b",).

python -m pytest astropy/table/tests/test_index.py astropy/table/tests/test_df.py docs/table/indexing.rst on this branch: 710 passed, 56 skipped. The full astropy/table, astropy/io/misc and ECSV suites were run with this change applied together with #20361 (6949 passed, 89 skipped, 23 xfailed), not separately. pre-commit run --files on all changed files passes.

5. Backwards compatibility

No API change, but one visible behavior change: to_pandas() (and to_df()) use a single-column primary key as the DataFrame index by default, so t[["a", "b"]].to_pandas() now returns a frame indexed by a with only b as a column, where on main a stayed a regular column with a RangeIndex. That is what the unsliced t.to_pandas() already did, and to_pandas(index=False) restores the old output. Tables without indices, row slices, and tables built through add_index() are bit-for-bit unchanged.

Changelog fragment: docs/changes/table/20362.bugfix.rst.

🤖 Generated with Claude Code

A table built from columns that already carry an index, such as a
column-name slice or Table([indexed_col]), kept the index but had no
primary key, so loc/iloc failed with a TypeError.  Make the first copied
index primary in _init_from_cols, and keep the parent's primary key on a
column slice when that index survived.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
@github-actions github-actions Bot added the table label Sep 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants