set stcal to main, allow python 3.13 #9105

braingram · 2025-01-29T15:53:04Z

This PR adds support for python 3.13 (and sets the stcal dependency to main as that's the only version that supports 3.13).

Tasks

news fragment change types...

changes/<PR#>.general.rst: infrastructure or miscellaneous change
changes/<PR#>.docs.rst
changes/<PR#>.stpipe.rst
changes/<PR#>.datamodels.rst
changes/<PR#>.scripts.rst
changes/<PR#>.fits_generator.rst
changes/<PR#>.set_telescope_pointing.rst
changes/<PR#>.pipeline.rst

stage 1

changes/<PR#>.group_scale.rst
changes/<PR#>.dq_init.rst
changes/<PR#>.emicorr.rst
changes/<PR#>.saturation.rst
changes/<PR#>.ipc.rst
changes/<PR#>.firstframe.rst
changes/<PR#>.lastframe.rst
changes/<PR#>.reset.rst
changes/<PR#>.superbias.rst
changes/<PR#>.refpix.rst
changes/<PR#>.linearity.rst
changes/<PR#>.rscd.rst
changes/<PR#>.persistence.rst
changes/<PR#>.dark_current.rst
changes/<PR#>.charge_migration.rst
changes/<PR#>.jump.rst
changes/<PR#>.clean_flicker_noise.rst
changes/<PR#>.ramp_fitting.rst
changes/<PR#>.gain_scale.rst

stage 2

changes/<PR#>.assign_wcs.rst
changes/<PR#>.badpix_selfcal.rst
changes/<PR#>.msaflagopen.rst
changes/<PR#>.nsclean.rst
changes/<PR#>.imprint.rst
changes/<PR#>.background.rst
changes/<PR#>.extract_2d.rst
changes/<PR#>.master_background.rst
changes/<PR#>.wavecorr.rst
changes/<PR#>.srctype.rst
changes/<PR#>.straylight.rst
changes/<PR#>.wfss_contam.rst
changes/<PR#>.flatfield.rst
changes/<PR#>.fringe.rst
changes/<PR#>.pathloss.rst
changes/<PR#>.barshadow.rst
changes/<PR#>.photom.rst
changes/<PR#>.pixel_replace.rst
changes/<PR#>.resample_spec.rst
changes/<PR#>.residual_fringe.rst
changes/<PR#>.cube_build.rst
changes/<PR#>.extract_1d.rst
changes/<PR#>.resample.rst

stage 3

changes/<PR#>.assign_mtwcs.rst
changes/<PR#>.mrs_imatch.rst
changes/<PR#>.tweakreg.rst
changes/<PR#>.skymatch.rst
changes/<PR#>.exp_to_source.rst
changes/<PR#>.outlier_detection.rst
changes/<PR#>.tso_photometry.rst
changes/<PR#>.stack_refs.rst
changes/<PR#>.align_refs.rst
changes/<PR#>.klip.rst
changes/<PR#>.spectral_leak.rst
changes/<PR#>.source_catalog.rst
changes/<PR#>.combine_1d.rst
changes/<PR#>.ami.rst

other

changes/<PR#>.wfs_combine.rst
changes/<PR#>.white_light.rst
changes/<PR#>.cube_skymatch.rst
changes/<PR#>.engdb_tools.rst
changes/<PR#>.guider_cds.rst

tapastro · 2025-01-29T16:03:52Z

I just started regtests on this old PR: #8864

Should I kill that off and start one with this?

codecov · 2025-01-29T16:27:11Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.73%. Comparing base (4ed4db8) to head (c15be58).
Report is 873 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9105      +/-   ##
==========================================
- Coverage   73.73%   73.73%   -0.01%     
==========================================
  Files         373      372       -1     
  Lines       37276    37285       +9     
==========================================
+ Hits        27486    27492       +6     
- Misses       9790     9793       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

braingram · 2025-01-29T16:27:25Z

I just started regtests on this old PR: #8864

Should I kill that off and start one with this?

Thanks for checking. I'd say leave them running. I expect failures which would be reason to not rush allowing 3.13.

tapastro · 2025-01-29T21:01:18Z

Perhaps worth adding an upper pin on 3.14 until we can test with it, once it exists? Opened #9110 to track reversion of #9106 - not sure if I did that in the proper git-wizard method but did my best.

tapastro · 2025-01-29T21:01:56Z

Would it be overloading the PR to also drop python 3.10 support here?

braingram · 2025-01-29T22:15:58Z

Would it be overloading the PR to also drop python 3.10 support here?

I'd say yes. Is there anything preventing dropping python 3.10? The python 3.13 bump requires further regtest investigation right?

Perhaps worth adding an upper pin on 3.14 until we can test with it, once it exists? Opened #9110 to track reversion of #9106 - not sure if I did that in the proper git-wizard method but did my best.

Good idea on the upper pin! I'll add that to this PR.

tapastro · 2025-01-30T15:59:30Z

The test set run on the closed PR ( https://github.com/spacetelescope/RegressionTests/actions/runs/13037094545 ) were fairly promising - perhaps worth investigating a possible change in numerical behavior for WFSS background subtraction, but the bulk differences look fairly innocuous to me.

braingram · 2025-01-31T17:20:08Z

A fresh set of regtests with this PR and the python 3.13 regtest branch: https://github.com/spacetelescope/RegressionTests/actions/runs/13077580972

braingram · 2025-01-31T20:42:08Z

Regtests show same failures as main
(example of the fgs failures on main: https://github.com/spacetelescope/RegressionTests/actions/runs/13080327495)

emolter

It looks like there are two additional failing regtests on the first link you sent (PR branch) than on the second link (main), namely test_associations_sdp_pools. What's happening with that?

Also, should we add/update one of the CI jobs to build with 3.13?

braingram · 2025-02-03T14:28:59Z

Thanks!

It looks like there are two additional failing regtests on the first link you sent (PR branch) than on the second link (main), namely test_associations_sdp_pools. What's happening with that?

The 'main' link I shared ran only the fgs tests. Here's the most recent scheduled run on main:
https://github.com/spacetelescope/RegressionTests/actions/runs/13103702612
showing both fgs and association failures.

Also, should we add/update one of the CI jobs to build with 3.13?

The build job runs cibuildwheel for all supported versions. With this PR this includes 3.13:
https://github.com/spacetelescope/jwst/actions/runs/13077462299/job/36493160941?pr=9105#step:8:1295
Is there something more you are looking for?

tapastro · 2025-02-03T14:31:55Z

The PR causing the failing association tests is now ready for review: #9098

The FGS failures are due to a CRDS delivery prompted by yours truly, which has backfired. Working on a PR now to get those fixed up.

emolter · 2025-02-03T15:08:59Z

The 'main' link I shared ran only the fgs tests. Here's the most recent scheduled run on main:
https://github.com/spacetelescope/RegressionTests/actions/runs/13103702612
showing both fgs and association failures.

Thanks, I realized that once I started going through the rest of my notifications this morning.

The build job runs cibuildwheel for all supported versions. With this PR this includes 3.13:
https://github.com/spacetelescope/jwst/actions/runs/13077462299/job/36493160941?pr=9105#step:8:1295
Is there something more you are looking for?

Thanks, I think this is good enough. I was looking at the environments where the actual unit test suite is run, but I suppose if it runs on 3.12 it'll run on 3.13

emolter

LGTM but we should probably wait for Tyler too

braingram · 2025-02-03T15:16:08Z

Thanks, I think this is good enough. I was looking at the environments where the actual unit test suite is run, but I suppose if it runs on 3.12 it'll run on 3.13

Thanks for the clarification. py313-xdist: https://github.com/spacetelescope/jwst/actions/runs/13077462294/job/36493092566?pr=9105 is running in 3.13.

tapastro

One last request, more housekeeping than anything: The quickstart and installation guide documentation both specify python=3.11 during conda environment creation - we should probably either remove the specification, now that we're compatible with the latest release, or at least increment to 3.13.

braingram · 2025-02-03T15:43:53Z

Thanks @tapastro. How does e604dc0 look?

tapastro · 2025-02-03T15:44:13Z

docs/getting_started/install.rst

 You can install the latest development version (not as well tested) from the
 Github main branch:

-    | >> conda create -n <env_name> python=3.11


I think we need to keep the python specification on all of these, just remove the version pin. Sorry for the imprecise request.

To clarify, do you want this to say python=3.13 or python=3.12? I don't think an unversioned qualifer (python) is supported (but I'm not a conda user so am not sure).

In the past, we did have an unqualified conda create -n <env_name> python, but I'm fine avoiding that to prevent undefined behavior. We can use either, but let's do 3.12 just because it's been thoroughly tested? 🤷

We've gathered some conflicts during my lack of a response - once this is clean I'll get it merged.

…elescope#9106)" This reverts commit 72f99a5.

github-actions bot added the installation label Jan 29, 2025

braingram closed this Jan 29, 2025

braingram changed the title ~~set stcal to main~~ set stcal to main, allow python 3.13 Jan 29, 2025

braingram reopened this Jan 29, 2025

github-actions bot added assign_wcs testing automation Continuous Integration (CI) and testing automation tools labels Jan 29, 2025

braingram force-pushed the fix_build branch from 8ed7392 to 72173e9 Compare January 29, 2025 16:01

braingram force-pushed the fix_build branch 3 times, most recently from 743910f to 260e63a Compare January 29, 2025 22:22

braingram force-pushed the fix_build branch from 260e63a to b612255 Compare January 31, 2025 17:00

github-actions bot added the documentation label Jan 31, 2025

braingram marked this pull request as ready for review January 31, 2025 20:40

braingram requested a review from a team as a code owner January 31, 2025 20:40

emolter reviewed Feb 3, 2025

View reviewed changes

emolter approved these changes Feb 3, 2025

View reviewed changes

braingram requested a review from tapastro February 3, 2025 15:24

tapastro requested changes Feb 3, 2025

View reviewed changes

tapastro reviewed Feb 3, 2025

View reviewed changes

melanieclarke mentioned this pull request Feb 3, 2025

Point stcal to main #9132

Closed

10 tasks

braingram added 7 commits February 3, 2025 16:03

set stcal to main

cd2bcd3

allow python 3.13

6f3b7d2

Revert "build sdist with python 3.13 (point to workflow fork) (spacet…

75e8ff0

…elescope#9106)" This reverts commit 72f99a5.

update install doc

9c09c34

update readme

3540307

add changelog

9863bf8

bump docs mentioned python version to 3.12

729baa2

braingram force-pushed the fix_build branch from e604dc0 to 729baa2 Compare February 3, 2025 21:06

Merge branch 'main' into fix_build

c15be58

tapastro self-requested a review February 3, 2025 21:47

tapastro approved these changes Feb 3, 2025

View reviewed changes

tapastro merged commit 51f6907 into spacetelescope:main Feb 3, 2025
30 checks passed

braingram deleted the fix_build branch February 4, 2025 14:02

set stcal to main, allow python 3.13 #9105

set stcal to main, allow python 3.13 #9105

Uh oh!

Conversation

braingram commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

stage 1

stage 2

stage 3

other

Uh oh!

tapastro commented Jan 29, 2025

Uh oh!

codecov bot commented Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

braingram commented Jan 29, 2025

Uh oh!

tapastro commented Jan 29, 2025

Uh oh!

tapastro commented Jan 29, 2025

Uh oh!

braingram commented Jan 29, 2025

Uh oh!

tapastro commented Jan 30, 2025

Uh oh!

braingram commented Jan 31, 2025

Uh oh!

braingram commented Jan 31, 2025

Uh oh!

emolter left a comment

Choose a reason for hiding this comment

Uh oh!

braingram commented Feb 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tapastro commented Feb 3, 2025

Uh oh!

emolter commented Feb 3, 2025

Uh oh!

emolter left a comment

Choose a reason for hiding this comment

Uh oh!

braingram commented Feb 3, 2025

Uh oh!

tapastro left a comment

Choose a reason for hiding this comment

Uh oh!

braingram commented Feb 3, 2025

Uh oh!

tapastro Feb 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

braingram Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

tapastro Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

tapastro Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

braingram commented Jan 29, 2025 •

edited

Loading

codecov bot commented Jan 29, 2025 •

edited

Loading

braingram commented Feb 3, 2025 •

edited

Loading

tapastro Feb 3, 2025 •

edited

Loading