Thanks to visit codestin.com
Credit goes to github.com

Skip to content

fix(timeseries): preserve primary_key on sliced tables and fix aggregate_downsample - #20354

Closed
CAOShurong wants to merge 4 commits into
astropy:mainfrom
CAOShurong:fix-timeseries-primary-key-downsample
Closed

fix(timeseries): preserve primary_key on sliced tables and fix aggregate_downsample#20354
CAOShurong wants to merge 4 commits into
astropy:mainfrom
CAOShurong:fix-timeseries-primary-key-downsample

Conversation

@CAOShurong

@CAOShurong CAOShurong commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Fixes #20297
Related to #11704

Description

When slicing a TimeSeries or indexed Table in Astropy by column names (e.g. ts_sub = ts["time", "a"]), Table.__getitem__ copied indices but dropped primary_key, leaving ts_sub.primary_key = None.

Furthermore, in TimeSeries.add_column(), because the copied index existed in self.indices, len(self.indices) was not 0, preventing self.add_index("time") not running, leaving primary_key permanently None.

Consequently, operations relying on primary key indexing like aggregate_downsample(ts_sub, ...) (which calls ts.iloc[:]) triggered TableLoc._get_index_id_and_item(item) with index_id = self.table.primary_key (None), causing TableIndices.__getitem__(None) to fail with:
TypeError: list indices must be integers or slices, not NoneType.

This pull request resolves the issue by:

  1. Preserving primary_key in Table.__getitem__ whenever all primary key columns are retained in the sliced table.
  2. In TimeSeries.add_column() and TimeSeries.add_columns(), ensuring primary_key = ("time",) is properly assigned when "time" is present even if indices were carried over from sliced data.
  3. Adding defensive fallback in TableLoc._get_index_id_and_item(): when primary_key is not set and exactly one index exists, use that index instead of passing None.
  4. In TableIndices.__getitem__(), raising a clear ValueError("Table has no primary key or index specified") instead of letting list.__getitem__(None) non-informatively fail with TypeError.

AI Disclosure

  • I certify that I am human and that I take full responsibility for this pull request including all interactions with reviewers.

Tests

  • Added test_downsample_subset_columns() in astropy/timeseries/tests/test_downsample.py.
  • Added changelog entry in docs/changes/timeseries/20354.bugfix.rst.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@pllim pllim added this to the v8.1.0 milestone Sep 9, 2026
@pllim

pllim commented Sep 9, 2026

Copy link
Copy Markdown
Member

If AI tools were used to develop this pull request, describe the tools including specific model and version, how they were used, and what content is AI generated.

@CAOShurong

Copy link
Copy Markdown
Contributor Author

Hi @pllim,

Thank you! In accordance with the Astropy AI Usage Policy, here is the disclosure for this PR:

  • Tools & Models: Google Gemini 2.5 Pro (via Antigravity coding assistant).
  • How they were used: Used as an assistive tool to trace table indexing paths across astropy.table / astropy.timeseries and draft the initial test function skeleton.
  • AI-generated content: Initial boilerplate draft for the regression tests in test_downsample.py.
  • Human review & accountability: The root cause analysis (index clearing in folded.remove_column vs primary_key preservation on table column slices), the bugfix implementation in table.py, index.py, and sampled.py, doc updates, and all test assertions were authored, refined, and validated locally against pytest astropy/timeseries and astropy/table by me. I take full responsibility for the code and will address any review feedback directly.

@taldcroft

Copy link
Copy Markdown
Member

@CAOShurong - Review verdict from Claude Fable 5.1, which I agree with. This PR has substantial defects and the basic approach needs to be changed. If you'd like to do that then please open a new PR.

Review of PR #20354: preserve primary_key on sliced tables and fix aggregate_downsample

Branch: fix-timeseries-primary-key-downsample (4 commits, 5 files, +42 lines).
Fixes #20297, related to #11704.

Verdict

The fix works for the reported case and all relevant tests pass locally
(astropy/timeseries, astropy/table/tests/test_index.py,
docs/table/indexing.rst, pre-commit). test_downsample_subset_columns fails
on main and passes on the branch, so it is a genuine regression test.

However, the PR patches one invariant violation in four separate places, one of
those patches introduces a new inconsistent state, the astropy.table changes
have no tests or changelog fragment of their own, and the codecov patch check
fails as a result. Requesting changes.

Root cause

primary_key is a plain Table attribute. Any path that copies indexed columns
without going through add_index produces a table with indices but
primary_key is None:

  • Table([col_with_index]) / QTable([t['a'], t['b']]) (table.py:1378, table.py:1432)
  • Table.__getitem__ with a list of column names
  • TimeSeries.__init__: remove_column("time") then add_column(time, ...)
    re-adds the same column object, which still carries the copied
    SlicedIndex. So len(self.indices) == 0 is False, add_index is skipped,
    and primary_key stays None. This is the exact mechanism behind TimeSeries.iloc failed when time comes from another TimeSeries #11704 and
    the reason the new elif was needed. (sampled.py:86-87)
  • BinnedTimeSeries.__init__ has the same pattern and then calls
    add_index("time_bin_start") unconditionally, producing two identical
    indices on that column. (binned.py:202-208)

Fixing this once, where indices are attached, would make the __getitem__
hunk, the TableLoc fallback, and both TimeSeries elif blocks unnecessary.

Findings (most important first)

1. The new elif can assign a primary key that names no index (new defect)

sampled.py:303 and sampled.py:315 set primary_key = ("time",) whenever
primary_key is None and a time column exists, without checking that any
index covers time. When the only carried-in index is on another column, the
result points at a non-existent index. Verified:

qt = QTable({'time': Time(np.arange(2450000, 2450005), format='jd'), 'a': [5,4,3,2,1]})
qt.add_index('a')
ts = TimeSeries([qt['time'], qt['a']])
# ts.indices == [('a',)], ts.primary_key == ('time',)
aggregate_downsample(ts, n_bins=2)
# IndexError: No index found for ['time']

On main this raised the old TypeError, so it is still an error either way,
but the PR replaces one broken state with a more confusing one.

Recommended fix. In TimeSeries.__init__, call self.remove_indices("time")
before remove_column("time") so the re-added column starts clean and the
existing len(self.indices) == 0 guard does its job. Then delete both elif
blocks. Apply the same one-line change in BinnedTimeSeries.__init__ for
time_bin_start, which also removes the duplicate index. If a guard in
add_column/add_columns is kept, the condition should be "no index on
time exists, so call add_index('time')", shared via one private helper
rather than copy-pasted into both methods.

The same wrong condition also means fold() on a series with a second index
drops the time index and never recreates it (pre-existing, same fix).

2. Prefer fixing primary_key at construction over the TableLoc fallback

The fallback at index.py:1157 infers the primary key only inside TableLoc.
Other consumers of primary_key do not get it:

  • TableLoc.__repr__ (index.py:1118) still does len(self.table.primary_key)
    and raises TypeError for a table whose .loc[...] now works.
  • write_indices=True (connect.py:323) and to_pandas(index=True) still
    treat a None primary key as fatal.
  • With two or more indices and no primary key the lookup still fails, now with
    the new ValueError.
  • The fallback can select a multi-column index whose other columns are gone.
    After t.add_index(['a','b']) and t['a','c'], the surviving column still
    carries the two-column index. iloc[:] then sorts on a stale copy of b,
    and loc[1] fails with "tuple index out of range".

Setting primary_key once in Table.__init__ when copied indices are attached
(matching the "first index is primary" policy in add_index) fixes all of these
together. A larger follow-up option is to make primary_key a property that
validates the stored key against current indices, which would also fix the
stale-key bug in remove_indices (see "Out of scope").

If the fallback stays: use the existing SlicedIndex.id property
(index.py:705) instead of re-deriving the tuple, factor the resolution into
one helper used by both __repr__ and _get_index_id_and_item, and check that
every index column is a column of the table.

3. Table.__getitem__ change is fine but changes to_pandas output

table.py:2121-2124 is the hunk that actually fixes #20297. Two notes:

  • It now carries primary_key onto column slices, so
    t[['a','b']].to_pandas() uses a as the DataFrame index and drops it from
    the columns. On main it stayed a regular column. This matches unsliced
    behavior and is arguably correct, but it is a public-API change that the
    changelog does not mention.
  • The condition reads more directly as a subset test:
    if self.primary_key and set(self.primary_key) <= set(item):.

4. TableIndices.__getitem__ None guard

index.py:1065 turns a TypeError into a ValueError. Reachable only with
two or more indices and no primary key. Harmless, but untested, and it puts
index-resolution policy into a generic list subclass. If a shared resolution
helper is introduced, raise there instead.

5. Missing tests

No test exercises the TableLoc fallback, the ValueError guard, or
primary_key preservation on a plain Table slice. That is why codecov/patch
fails. Ablation: both new tests pass with the elif blocks removed, and with
the fallback removed. Suggested additions:

  • astropy/table/tests/test_index.py: column slice preserves primary_key
    (t[['a','b']].primary_key == ('a',), t[['b']].primary_key is None);
    Table([indexed_col]).loc[...] works; two-index no-primary-key raises.
  • astropy/timeseries/tests/test_sampled.py: the TimeSeries.iloc failed when time comes from another TimeSeries #11704 case
    TimeSeries(time=ts.time).iloc[:], and TimeSeries(QTable([ts['time'], ts['a']])).
  • astropy/timeseries/tests/test_binned.py: column-sliced BinnedTimeSeries
    has exactly one index on time_bin_start.

Also, test_downsample_folded_timeseries_with_new_columns passes unchanged on
main, so it does not guard anything in this PR. Keep it if desired but reword
the comment, or replace it with one of the constructor-based tests above.

6. Changelog

The table.py and index.py changes are user-visible in astropy.table, but
the only fragment is under docs/changes/timeseries/. The towncrier draft
renders it solely under the astropy.timeseries heading. Add
docs/changes/table/20354.bugfix.rst covering the Table.__getitem__ and
Table.loc changes (and the to_pandas effect), and trim the timeseries
fragment to the aggregate_downsample fix.

7. PR description

This PR does fix #11704 (verified: TimeSeries(time=ts.time).iloc[:] fails on
main and works on the branch). Change "Related to #11704" to "Fixes #11704"
and add the regression test.

Out of scope but noticed

  • Table.remove_indices docstring promises to reassign the primary key when
    the primary index is removed, but the body never touches primary_key.
    Result: a stale key naming a removed index, and .loc fails with
    IndexError even though a usable index exists. The new __getitem__ hunk
    propagates that stale key into slices. Worth a separate issue.
  • Table(t, copy_indices=False) and index_mode('discard_on_copy') already
    produce tables with a primary_key but no indices; the new __getitem__
    hunk extends that existing inconsistency rather than creating it.

Verification performed

  • conda run -n astropy-dev python -m pytest astropy/timeseries astropy/table/tests/test_index.py docs/table/indexing.rst passes.
  • pre-commit run --files on all changed files passes.
  • New tests run against main source: test_downsample_subset_columns fails,
    test_downsample_folded_timeseries_with_new_columns passes.
  • Probes for findings 1, 2, and 3 run on the branch as described above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aggregate_downsample() on a TimeSeries object with a subset of columns failed with TypeError

3 participants