Rollforward stpipe version in setup #7400

zacharyburnett · 2022-12-16T15:15:03Z

Reverts #7399

The most recent stpipe version improperly packaged scripts; this unpins the release to the new release with a fix.

Checklist for maintainers

added entry in CHANGES.rst within the relevant release section
updated or added relevant tests
updated relevant documentation
added relevant milestone
added relevant label(s)
ran regression tests, post a link to the Jenkins job below.
How to run regression tests on a PR
Make sure the JIRA ticket is resolved properly

hbushouse · 2022-12-16T16:16:45Z

Running regtests at https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/496/

codecov · 2022-12-16T16:17:25Z

Codecov Report

Base: 80.06% // Head: 76.30% // Decreases project coverage by -3.75% ⚠️

Coverage data is based on head (eff8f03) compared to base (350281a).
Patch has no changes to coverable lines.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7400      +/-   ##
==========================================
- Coverage   80.06%   76.30%   -3.76%     
==========================================
  Files         417      417              
  Lines       38045    38045              
==========================================
- Hits        30460    29031    -1429     
- Misses       7585     9014    +1429

Flag	Coverage Δ
nightly	`76.29% <ø> (-3.71%)`	⬇️
unit	`52.06% <ø> (ø)`

Impacted Files	Coverage Δ
jwst/extract_1d/soss_extract/soss_boxextract.py	`10.44% <0.00%> (-79.11%)`	⬇️
jwst/extract_1d/soss_extract/soss_centroids.py	`17.85% <0.00%> (-75.00%)`	⬇️
jwst/extract_1d/soss_extract/soss_extract.py	`17.85% <0.00%> (-72.17%)`	⬇️
jwst/extract_1d/soss_extract/atoca.py	`9.97% <0.00%> (-68.44%)`	⬇️
jwst/extract_1d/soss_extract/soss_solver.py	`21.42% <0.00%> (-64.29%)`	⬇️
jwst/extract_1d/soss_extract/atoca_utils.py	`8.32% <0.00%> (-61.33%)`	⬇️
jwst/extract_1d/soss_extract/soss_utils.py	`22.22% <0.00%> (-19.05%)`	⬇️
jwst/extract_1d/extract_1d_step.py	`69.79% <0.00%> (-6.72%)`	⬇️
jwst/residual_fringe/utils.py	`77.80% <0.00%> (-2.25%)`	⬇️
jwst/transforms/converters/jwst_models.py	`96.10% <0.00%> (-1.30%)`	⬇️
... and 5 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

hbushouse · 2022-12-16T16:20:23Z

Some of the CI tests are failing with:

ERROR: Could not find a version that satisfies the requirement stpipe<1.0,>=0.4.4 (from jwst[test]) (from versions: 0.0.0, 0.1.0, 0.2.0, 0.2.1, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.4.0, 0.4.1, 0.4.2, 0.4.3)
ERROR: No matching distribution found for stpipe<1.0,>=0.4.4

hbushouse · 2022-12-16T16:22:02Z

And the regtest run is already throwing lots of test errors in the console log. Something's not right.

zacharyburnett · 2022-12-16T16:35:36Z

Some of the CI tests are failing with:

ERROR: Could not find a version that satisfies the requirement stpipe<1.0,>=0.4.4 (from jwst[test]) (from versions: 0.0.0, 0.1.0, 0.2.0, 0.2.1, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.4.0, 0.4.1, 0.4.2, 0.4.3) ERROR: No matching distribution found for stpipe<1.0,>=0.4.4

The first call to PyPI for stpipe failed to find 0.4.4, but the subsequent calls did resolve. I seems like PyPI was not serving it in the first run; I'll rerun this job.

hbushouse · 2022-12-19T19:59:22Z

Latest regtest run is https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/506/

zacharyburnett · 2022-12-19T20:48:30Z

there are a few errors in the regtests with AttributeError: module 'numpy' has no attribute 'int':

==================================== ERRORS ====================================
______________ ERROR at setup of test_niriss_soss_stage2[calints] ______________
[gw12] linux -- Python 3.9.15 /data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/miniconda/envs/tmp_env0/bin/python

jail = local('/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/test_outputs/popen-gw12/test_niriss_soss_rtdata_module0')
rtdata_module = {'input': '/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/test_outputs/popen-gw12/test_niriss_soss_...ut': None,
 'remote_results_path': None,
 'test_name': None,
 'traceback': None,
 'truth': None,
 'truth_remote': None}

    @pytest.fixture(scope="module")
    def run_tso_spec2(jail, rtdata_module):
        """Run stage 2 pipeline on NIRISS SOSS data."""
        rtdata = rtdata_module
    
        # Run tso-spec2 pipeline on the first _rateints file, saving intermediate products
        rtdata.get_data("niriss/soss/jw01091002001_03101_00001-seg001_nis_short_rateints.fits")
        args = ["calwebb_spec2", rtdata.input,
                "--steps.flat_field.save_results=True",
                "--steps.srctype.save_results=True",
                "--steps.extract_1d.soss_atoca=False",
                ]
>       Step.from_cmdline(args)

/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/jwst/regtest/test_niriss_soss.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/miniconda/envs/tmp_env0/lib/python3.9/site-packages/stpipe/step.py:179: in from_cmdline
    return cmdline.step_from_cmdline(args)
/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/miniconda/envs/tmp_env0/lib/python3.9/site-packages/stpipe/cmdline.py:343: in step_from_cmdline
    step.run(*positional)
/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/miniconda/envs/tmp_env0/lib/python3.9/site-packages/stpipe/step.py:443: in run
    step_result = self.process(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <jwst.pipeline.calwebb_spec2.Spec2Pipeline object at 0x7fefd0b37250>
data = '/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/test_outputs/popen-gw12/test_niriss_soss_rtdata_module0/jw01091002001_03101_00001-seg001_nis_short_rateints.fits'

    def process(self, data):
        """Entrypoint for this pipeline
    
        Parameters
        ----------
        input: str, Level2 Association, or ~jwst.datamodels.DataModel
            The exposure or association of exposures to process
        """
        self.log.info('Starting calwebb_spec2 ...')
    
        # Setup step parameters required by the pipeline.
        self.resample_spec.save_results = self.save_results
        self.resample_spec.suffix = 's2d'
        self.cube_build.output_type = 'multi'
        self.cube_build.save_results = False
        self.cube_build.skip_dqflagging = True
        self.extract_1d.save_results = self.save_results
    
        # Retrieve the input(s)
        asn = self.load_as_level2_asn(data)
    
        # Each exposure is a product in the association.
        # Process each exposure.  Delay reporting failures until the end.
        results = []
        failures = []
        for product in asn['products']:
            self.log.info('Processing product {}'.format(product['name']))
            self.output_file = product['name']
            try:
                getattr(asn, 'filename')
            except AttributeError:
                asn.filename = "singleton"
            try:
                result = self.process_exposure_product(
                    product,
                    asn['asn_pool'],
                    asn.filename
                )
            except NoDataOnDetectorError as exception:
                # This error merits a special return
                # status if run from the command line.
                # Bump it up now.
                raise exception
            except Exception:
                traceback.print_exc()
                failures.append(traceback.format_exc())
            else:
                if result is not None:
                    results.append(result)
    
        if len(failures) > 0 and self.fail_on_exception:
>           raise RuntimeError('\n'.join(failures))
E           RuntimeError: Traceback (most recent call last):
E             File "/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/jwst/pipeline/calwebb_spec2.py", line 119, in process
E               result = self.process_exposure_product(
E             File "/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/jwst/pipeline/calwebb_spec2.py", line 301, in process_exposure_product
E               x1d = self.extract_1d(resampled)
E             File "/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/miniconda/envs/tmp_env0/lib/python3.9/site-packages/stpipe/step.py", line 443, in run
E               step_result = self.process(*args)
E             File "/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/jwst/extract_1d/extract_1d_step.py", line 392, in process
E               result, ref_outputs, atoca_outputs = soss_extract.run_extract1d(
E             File "/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/jwst/extract_1d/soss_extract/soss_extract.py", line 1157, in run_extract1d
E               transform = solve_transform(scidata_bkg, scimask, xref_o1, yref_o1,
E             File "/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/jwst/extract_1d/soss_extract/soss_solver.py", line 224, in solve_transform
E               xdat_o1, ydat_o1, _ = get_centroids_com(scidata_bkg, mask=mask,
E             File "/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/jwst/extract_1d/soss_extract/soss_centroids.py", line 77, in get_centroids_com
E               dimx, dimy, xos, yos, xnative, ynative, padding, refpix_mask = get_image_dim(scidata_bkg, header=header)
E             File "/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/jwst/extract_1d/soss_extract/soss_utils.py", line 125, in get_image_dim
E               if np.int(dimy / xos) in [96, 256, 252, 2040, 2048]:
E             File "/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/miniconda/envs/tmp_env0/lib/python3.9/site-packages/numpy/__init__.py", line 284, in __getattr__
E               raise AttributeError("module {!r} has no attribute "
E           AttributeError: module 'numpy' has no attribute 'int'

/data1/jenkins/workspace/RT/JWST-Developers-Pull-Requests@3/clone/jwst/pipeline/calwebb_spec2.py:137: RuntimeError

hbushouse · 2022-12-20T13:44:49Z

All 12 errors in the most recent regtest run are unrelated. 11 are due to numpy 1.20.0 being released, which completely deprecated the np.int datatype and the remaining one appears to be just numerical precision issues (possibly also due to the new numpy pkg). The exact same 12 errors also showed up in the regular regtest run last night, which used the older stpipe. So I think this is good to go.

zacharyburnett self-assigned this Dec 16, 2022

github-actions bot added the installation label Dec 16, 2022

zacharyburnett added the no-changelog-entry-needed label Dec 16, 2022

zacharyburnett requested review from hbushouse and kmacdonald-stsci December 16, 2022 15:51

zacharyburnett marked this pull request as ready for review December 16, 2022 15:51

kmacdonald-stsci approved these changes Dec 16, 2022

View reviewed changes

zacharyburnett added 2 commits December 19, 2022 13:35

Rollforward stpipe version in setup

50234f2

anticipate new version of stpipe with fix

eff8f03

zacharyburnett force-pushed the rollforward_stpipe branch from 0decdf0 to eff8f03 Compare December 19, 2022 18:35

hbushouse added this to the Build 9.1 milestone Dec 19, 2022

hbushouse approved these changes Dec 20, 2022

View reviewed changes

hbushouse merged commit f8cd884 into spacetelescope:master Dec 20, 2022

zacharyburnett deleted the rollforward_stpipe branch December 20, 2022 13:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rollforward stpipe version in setup #7400

Rollforward stpipe version in setup #7400

Uh oh!

zacharyburnett commented Dec 16, 2022 •

edited by hbushouse

Loading

Uh oh!

hbushouse commented Dec 16, 2022

Uh oh!

codecov bot commented Dec 16, 2022 •

edited

Loading

Uh oh!

hbushouse commented Dec 16, 2022

Uh oh!

hbushouse commented Dec 16, 2022

Uh oh!

zacharyburnett commented Dec 16, 2022

Uh oh!

hbushouse commented Dec 19, 2022

Uh oh!

zacharyburnett commented Dec 19, 2022

Uh oh!

hbushouse commented Dec 20, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Rollforward stpipe version in setup #7400

Rollforward stpipe version in setup #7400

Uh oh!

Conversation

zacharyburnett commented Dec 16, 2022 • edited by hbushouse Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hbushouse commented Dec 16, 2022

Uh oh!

codecov bot commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hbushouse commented Dec 16, 2022

Uh oh!

hbushouse commented Dec 16, 2022

Uh oh!

zacharyburnett commented Dec 16, 2022

Uh oh!

hbushouse commented Dec 19, 2022

Uh oh!

zacharyburnett commented Dec 19, 2022

Uh oh!

hbushouse commented Dec 20, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zacharyburnett commented Dec 16, 2022 •

edited by hbushouse

Loading

codecov bot commented Dec 16, 2022 •

edited

Loading