feat: env var to control exception wrapping #2142
Conversation
benhoyt
left a comment
There was a problem hiding this comment.
Looks like a good start, but I think we should structure this differently.
d1ccc5c to
04a2b98
Compare
|
I've refactored as per @benhoyt's suggestions:
I've parametrized the existing tests that caught |
dimaqq
left a comment
There was a problem hiding this comment.
How will this env var be documented?
The other env var used to be documented: https://pypi.org/project/ops-scenario/4.0.4.1/
It's no longer documented today.
|
I've implemented the alternative approach that avoids environment variables, and instead controls this through |
benhoyt
left a comment
There was a problem hiding this comment.
Thanks. This looks better to me. I agree we should probably leave it out of the ops tests. I'll leave off an approval stamp for now, to remind us to finalise which approach we're go for in the end (env var or @TheJJ's Context attribute approach -- let's discuss during engineering week.
|
Thanks for reconsidering. I'm convinced solving this via environment variable is the wrong place to solve this - the test execution becomes dependent on an environment variable, instead of being self-contained and self-descriptive. Just executing the tests directly without the "correct env" will lead to failures that are not obvious to resolve (and then in the wrong layer: env instead of test configuration). Instead this exception handling should be configurable per-test (imagine having hundreds to migrate), and be self-contained from within the tests. If you'd like, we can discuss this in person during the engineering sprint :) |
|
I lean towards the environment variable as I think it will give us a better pathway to changing the default behavior in future, but I'm definitely open to discussing alternatives. Regarding the developer experience, it's worth noting that you can still define the behavior in your tests with an environment variable, for example with an autouse fixture. To control the behavior per-test with relatively low boilerplate you could instead use pytest.mark.usefixtures. |
|
@james-garner-canonical For the record, decision is to use this (env var) approach. |
This PR adds a new environment variable,
SCENARIO_WRAP_CHARM_ERRORS, which controls whether charm errors are wrapped inUncaughtCharmError. The name is up for debate, but was chosen to follow the format ofSCENARIO_SKIP_CONSISTENCY_CHECKS.If unset, it defaults to
True, preserving the current behaviour. The idea would be to switch to the other behaviour by default someday.The environment variable logic is encapsulated in the new private
scenario._environmodule. This allows the same logic to be used in our internal tests, to handle both cases. Our tests have been updated as needed.CI has been modified to add a row to our unit test matrix, running with this variable toggled on or off. This would require our expected checks for this repo to be updated, as the job names have changed. An alternative approach might be to have the relevant tests set and unset the environment variable themselves, based on a parametrized input.
tox.inihas been updated to always pass the scenario environment variables along.Unrelated to these changes, the tracing tests have been updated to run in CI even if the main ops tests fail.
Resolves #1428.