Generalize handling of resource-intensive tests #12350

astrofrog · 2021-11-01T22:53:22Z

Description

This renames the environment variable for the big table tests to be more general and usable by other tests, and adds a tox option to more easily run these tests without setting an environment variable.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

Do the proposed changes actually accomplish desired goals?
Do the proposed changes follow the Astropy coding guidelines?
Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see "When to rebase and squash commits".
Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the Extra CI label.
Is a change log needed? If yes, did the change log check pass? If no, add the no-changelog-entry-needed label. If this is a manual backport, use the skip-changelog-checks label unless special changelog handling is necessary.
Is a milestone set? Milestone must be set but astropy-bot check might be missing; do not let the green checkmark fool you.
At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate backport-X.Y.x label(s) before merge.

…on, and document how to write tests that need a lot of resources

dhomeier · 2021-11-02T15:52:35Z

Would any of the weekly crons be using the intensive option?

dhomeier

Thanks, this looks very useful. Apart from the stylistic and usage notes, good to go in once the readthedocs failure is fixed.

dhomeier · 2021-11-02T17:20:34Z

docs/development/testguide.rst

+Tests that have large disk and/or memory requirements
+====================================


Suggested change

Tests that have large disk and/or memory requirements

====================================

Tests that have large disk and/or memory requirements

=====================================================

Sphinx warning/readthedocs failure

dhomeier · 2021-11-02T17:44:22Z

astropy/io/ascii/tests/test_c_reader.py

+@pytest.mark.skipif(not os.getenv('TEST_INTENSIVE'),
+                    reason='Environment variable TEST_INTENSIVE must be '
                    'defined to run this test')


Suggested change

@pytest.mark.skipif(not os.getenv('TEST_INTENSIVE'),

reason='Environment variable TEST_INTENSIVE must be '

'defined to run this test')

@pytest.mark.skipif(os.getenv('TEST_INTENSIVE', 0) == 0,

reason='Environment variable TEST_INTENSIVE must be defined to run this test')

Just found that I intuitively expected the env var set to 0 to disable the tests; this would also be more consistent with how we are using the ASTROPY_USE_ vars (also noticed that setup is using os.environ.get(), whatever the correct Zen is here ;-).
I prefer it without further linebreak, but unfortunately this would make the docs example break the line...

I don't think we have any consistent zen in this department:

astropy/.github/workflows/ci_workflows.yml

Lines 11 to 13 in 73c57e1

env:

ARCH_ON_CI: "normal"

IS_CRON: "false"

I would have named it SHOW_ME_THE_MONEY but this works too.

But those are checking the value as well.
TEST_INTENSIVE="False" or TEST_INTENSIVE=0 did let the tests run.

Re: values -- Yes, I agree with @dhomeier . The check should be more strict to only let it run when the value is 1, as documented.

I'd be willing to allow anything that could be interpreted as True, but in fact my suggestion will still accept 'false' as well...

dhomeier · 2021-11-02T17:45:05Z

astropy/io/ascii/tests/test_c_reader.py

@@ -1014,8 +1014,8 @@ def test_read_big_table(tmpdir):
    assert len(t) == NB_ROWS


-@pytest.mark.skipif(not os.getenv('TEST_READ_HUGE_FILE'),
-                    reason='Environment variable TEST_READ_HUGE_FILE must be '
+@pytest.mark.skipif(not os.getenv('TEST_INTENSIVE'),


Suggested change

@pytest.mark.skipif(not os.getenv('TEST_INTENSIVE'),

@pytest.mark.skipif(os.getenv('TEST_INTENSIVE', 0) == 0,

dhomeier · 2021-11-02T18:03:59Z

Not sure about the changelog – do we generally include them for functionality only visible to developers?

astrofrog · 2021-11-03T14:29:46Z

Would any of the weekly crons be using the intensive option?

No, the GitHub actions workers don't have enough RAM

taldcroft

Deferring final review to @dhomeier , but this looks good to me!

dhomeier · 2021-11-04T13:33:06Z

Marked this as OK when readthedocs is fixed, but I think with @pllim there are at least 2 votes for checking
os.getenv('TEST_INTENSIVE', 0) != 0 ?

taldcroft · 2021-11-04T14:24:36Z

Marked this as OK when readthedocs is fixed, but I think with @pllim there are at least 2 votes for checking
os.getenv('TEST_INTENSIVE', 0) != 0 ?

No opinion here one way or the other.

saimn · 2021-11-23T17:05:39Z

I think just checking if an environment variable is set or unset is enough, the value doesn't matter and having a complex condition in every test isn't ideal.
But I wanted to propose something else: why not using pytest markers instead ? I think there is a need for more specific markers (e.g. memory intensive, cpu intensive, the thread safety ones would be good candidates: #12522 (comment)), and pytest markers seems more flexible for that. We could exclude those markers by default with addopts, and run the cpu intensive one in a dedicated cron job for example.

Cadair · 2021-11-24T11:49:03Z

+1 for markers, that's definitely the pytest tool for this job.

astrofrog · 2021-11-24T13:49:17Z

I don't have much time to work on this so happy for someone to take the lead on pytest markers (and we can then close this PR)

nstarman · 2022-01-12T14:45:08Z

This looks very useful. I've been looking for something like this for the new cosmology test suite.
What's the upstream-fix-required for?

pllim · 2022-01-12T15:00:46Z

What's the upstream-fix-required for?

Ideally, the pytest marker would be upstream from pytest-astropy or something.

pllim · 2022-02-10T23:51:59Z

Closing in favor of #12764 . Thanks anyway!

Rename TEST_READ_HUGE_FILE to TEST_INTENSIVE, add tox -intensive opti…

7e21f9d

…on, and document how to write tests that need a lot of resources

astrofrog added the no-changelog-entry-needed label Nov 1, 2021

astrofrog added this to the v5.1 milestone Nov 1, 2021

astrofrog requested review from taldcroft and dhomeier November 1, 2021 22:53

github-actions bot added Docs io.ascii Release testing labels Nov 1, 2021

dhomeier mentioned this pull request Nov 2, 2021

Fix unclosed file warning in io.ascii tests for big tables #12367

Merged

9 tasks

dhomeier requested changes Nov 2, 2021

View reviewed changes

taldcroft reviewed Nov 4, 2021

View reviewed changes

dhomeier added the 💤 merge-when-ci-passes Do not use: We have auto-merge option now. label Nov 4, 2021

saimn mentioned this pull request Nov 23, 2021

TST: Update Python versions in CI #12522

Merged

9 tasks

pllim added Upstream Fix Required and removed 💤 merge-when-ci-passes Do not use: We have auto-merge option now. labels Dec 6, 2021

This was referenced Jan 21, 2022

Add plugin for cpu/memory intensive tests astropy/pytest-astropy#48

Merged

Use pytest markers for intensive tests #12764

Merged

pllim closed this Feb 10, 2022

		Tests that have large disk and/or memory requirements
		====================================

	@pytest.mark.skipif(not os.getenv('TEST_INTENSIVE'),
	@pytest.mark.skipif(os.getenv('TEST_INTENSIVE', 0) == 0,

Uh oh!

Generalize handling of resource-intensive tests #12350

Generalize handling of resource-intensive tests #12350

Uh oh!

Conversation

astrofrog commented Nov 1, 2021 • edited by dhomeier Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist for package maintainer(s)

Uh oh!

dhomeier commented Nov 2, 2021

Uh oh!

dhomeier left a comment

Choose a reason for hiding this comment

Uh oh!

dhomeier Nov 2, 2021

Choose a reason for hiding this comment

Uh oh!

dhomeier Nov 2, 2021

Choose a reason for hiding this comment

Uh oh!

pllim Nov 3, 2021

Choose a reason for hiding this comment

Uh oh!

dhomeier Nov 3, 2021

Choose a reason for hiding this comment

Uh oh!

pllim Nov 3, 2021

Choose a reason for hiding this comment

Uh oh!

dhomeier Nov 3, 2021

Choose a reason for hiding this comment

Uh oh!

dhomeier Nov 2, 2021

Choose a reason for hiding this comment

Uh oh!

dhomeier commented Nov 2, 2021

Uh oh!

astrofrog commented Nov 3, 2021

Uh oh!

taldcroft left a comment

Choose a reason for hiding this comment

Uh oh!

dhomeier commented Nov 4, 2021

Uh oh!

taldcroft commented Nov 4, 2021

Uh oh!

saimn commented Nov 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Cadair commented Nov 24, 2021

Uh oh!

astrofrog commented Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstarman commented Jan 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pllim commented Jan 12, 2022

Uh oh!

pllim commented Feb 10, 2022

Uh oh!

Uh oh!

astrofrog commented Nov 1, 2021 •

edited by dhomeier

Loading

saimn commented Nov 23, 2021 •

edited

Loading

astrofrog commented Nov 24, 2021 •

edited

Loading

nstarman commented Jan 12, 2022 •

edited

Loading