Thanks to visit codestin.com
Credit goes to github.com

Skip to content

restructure test_miri_mrs_badpix_selfcal_bkg to allow it to be okified #8971

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Nov 20, 2024
Merged

restructure test_miri_mrs_badpix_selfcal_bkg to allow it to be okified #8971

merged 2 commits into from
Nov 20, 2024

Conversation

@braingram
Copy link
Collaborator

@braingram braingram commented Nov 20, 2024 •
edited
Loading

Closes #8968

This PR makes 2 changes to test_miri_mrs_badpix_selfcal_bkg to allow it's results to be okified.

  1. The multiple comparisons are split into different tests using parametrization
  2. The background comparisons set rtdata.output which is required to allow okify_regtests to know what files to replace

To provide some more details on 2 above. Currently this test is failing due to expected header comment differences:

   a: result_bkgasn_badpix_selfcal_bkg_0.fits
   b: /runner/_work/_temp/pytest_basetemp/popen-gw0/test_miri_mrs_badpix_selfcal_rtdata_module0/truth/result_bkgasn_badpix_selfcal_bkg_0.fits
   HDU(s) not to be compared:
...
     Headers contain differences:
       Keyword DRPFRMS1 has different comments:
          a> Number of frames dropped prior to first integra
           ? ^^^^^^^^^^^
          b> Frames dropped prior to first integration
           ? ^                                    ++++

see: https://github.com/spacetelescope/RegressionTests/actions/runs/11937489373
Note that the failed comparison is for result_bkgasn_badpix_selfcal_bkg_0.fits. If I run okify_regtests on this run I see the following:

OK: jwst-pipeline-results/2024-11-20_GITHUB_CI_Linux-X64-py3.12-1009/2553_test_miri_mrs_badpix_selfcal_bkg/result_bkgasn_badpix_selfcal.fits
--> jwst-pipeline/dev/truth/test_miri_mrs_badpix_selfcal/result_bkgasn_badpix_selfcal_bkg_0.fits

As okify regtests is incorrectly attempting to overwrite the truth result_bkgasn_badpix_selfcal_bkg_0.fits file with result_bkgasn_badpix_selfcal.fits due to the test not setting rtdata.output prior to the call to FITSDIFF.

Regtest run with just this regtest: https://github.com/spacetelescope/RegressionTests/actions/runs/11938033481
show the expected 4 failures due to header keyword comment differences introduced in spacetelescope/stdatamodels#354
If I run okify_regtests I see 4 failures to okify and the reported moves are now correct with this PR. For example:

OK: jwst-pipeline-results/2024-11-20_GITHUB_CI_Linux-X64-py3.12-1013/2554_test_miri_mrs_badpix_selfcal_bkg/result_bkgasn_badpix_selfcal_bkg_0.fits
--> jwst-pipeline/dev/truth/test_miri_mrs_badpix_selfcal/result_bkgasn_badpix_selfcal_bkg_0.fits

Tasks

  • request a review from someone specific, to avoid making the maintainers review every PR
  • add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
  • Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
    • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
    • update or add relevant tests
    • update relevant docstrings and / or docs/ page
    • start a regression test and include a link to the running job (click here for instructions)
      • Do truth files need to be updated ("okified")?
        • after the reviewer has approved these changes, run okify_regtests to update the truth files
  • if a JIRA ticket exists, make sure it is resolved properly
news fragment change types...
  • changes/<PR#>.general.rst: infrastructure or miscellaneous change
  • changes/<PR#>.docs.rst
  • changes/<PR#>.stpipe.rst
  • changes/<PR#>.datamodels.rst
  • changes/<PR#>.scripts.rst
  • changes/<PR#>.fits_generator.rst
  • changes/<PR#>.set_telescope_pointing.rst
  • changes/<PR#>.pipeline.rst

stage 1

  • changes/<PR#>.group_scale.rst
  • changes/<PR#>.dq_init.rst
  • changes/<PR#>.emicorr.rst
  • changes/<PR#>.saturation.rst
  • changes/<PR#>.ipc.rst
  • changes/<PR#>.firstframe.rst
  • changes/<PR#>.lastframe.rst
  • changes/<PR#>.reset.rst
  • changes/<PR#>.superbias.rst
  • changes/<PR#>.refpix.rst
  • changes/<PR#>.linearity.rst
  • changes/<PR#>.rscd.rst
  • changes/<PR#>.persistence.rst
  • changes/<PR#>.dark_current.rst
  • changes/<PR#>.charge_migration.rst
  • changes/<PR#>.jump.rst
  • changes/<PR#>.clean_flicker_noise.rst
  • changes/<PR#>.ramp_fitting.rst
  • changes/<PR#>.gain_scale.rst

stage 2

  • changes/<PR#>.assign_wcs.rst
  • changes/<PR#>.badpix_selfcal.rst
  • changes/<PR#>.msaflagopen.rst
  • changes/<PR#>.nsclean.rst
  • changes/<PR#>.imprint.rst
  • changes/<PR#>.background.rst
  • changes/<PR#>.extract_2d.rst
  • changes/<PR#>.master_background.rst
  • changes/<PR#>.wavecorr.rst
  • changes/<PR#>.srctype.rst
  • changes/<PR#>.straylight.rst
  • changes/<PR#>.wfss_contam.rst
  • changes/<PR#>.flatfield.rst
  • changes/<PR#>.fringe.rst
  • changes/<PR#>.pathloss.rst
  • changes/<PR#>.barshadow.rst
  • changes/<PR#>.photom.rst
  • changes/<PR#>.pixel_replace.rst
  • changes/<PR#>.resample_spec.rst
  • changes/<PR#>.residual_fringe.rst
  • changes/<PR#>.cube_build.rst
  • changes/<PR#>.extract_1d.rst
  • changes/<PR#>.resample.rst

stage 3

  • changes/<PR#>.assign_mtwcs.rst
  • changes/<PR#>.mrs_imatch.rst
  • changes/<PR#>.tweakreg.rst
  • changes/<PR#>.skymatch.rst
  • changes/<PR#>.exp_to_source.rst
  • changes/<PR#>.outlier_detection.rst
  • changes/<PR#>.tso_photometry.rst
  • changes/<PR#>.stack_refs.rst
  • changes/<PR#>.align_refs.rst
  • changes/<PR#>.klip.rst
  • changes/<PR#>.spectral_leak.rst
  • changes/<PR#>.source_catalog.rst
  • changes/<PR#>.combine_1d.rst
  • changes/<PR#>.ami.rst

other

  • changes/<PR#>.wfs_combine.rst
  • changes/<PR#>.white_light.rst
  • changes/<PR#>.cube_skymatch.rst
  • changes/<PR#>.engdb_tools.rst
  • changes/<PR#>.guider_cds.rst

@braingram braingram marked this pull request as ready for review November 20, 2024 17:03
@braingram braingram requested a review from a team as a code owner November 20, 2024 17:03
@braingram braingram requested a review from emolter November 20, 2024 17:05
tapastro
tapastro approved these changes Nov 20, 2024
View reviewed changes
Copy link
Contributor

@tapastro tapastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending typo

braingram
braingram commented Nov 20, 2024
View reviewed changes
@braingram braingram enabled auto-merge November 20, 2024 17:14
@codecov
Copy link

codecov bot commented Nov 20, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 64.52%. Comparing base (72cb6f0) to head (5a2d0cd).
Report is 706 commits behind head on main.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #8971   +/-   ##
=======================================
  Coverage   64.52%   64.52%           
=======================================
  Files         375      375           
  Lines       38739    38739           
=======================================
  Hits        24997    24997           
  Misses      13742    13742

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@braingram braingram merged commit ac642ca into spacetelescope:main Nov 20, 2024
31 checks passed
@braingram braingram deleted the fix_selfcal_regtests branch November 20, 2024 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tapastro tapastro tapastro approved these changes

@emolter emolter Awaiting requested review from emolter

Assignees

No one assigned

Labels

MIRI no-changelog-entry-needed regression_testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

test_miri_mrs_badpix_selfcal_bkg fails okify

2 participants

@braingram @tapastro