test: reduce testing to 2 python versions per sample #8875
Conversation
|
removed earlier comment, realized we were editing the noxfile configs per product not the base test image. sorry! |
|
FYI: App Engine does not fully support 3.11 yet. |
|
Composer LGTM |
|
I'm going to merge with main to kick off another build but I'm thinking that... 3.7
3.11
Update
|
|
I believe some products only serve 3.8 or 3.9 (Run and Data* products). Fundamentally, I'd strongly push back against making this change. If users run into problems before we do, simply because we reduced the number of versions being tested, that'd be a really bad experience for everyone. If we want to save resources, can we run the nightly / continuous less frequently instead? |
|
@dandhlee I believe this change does not affect samples that have a dependency on a specific version (e.g. 3.8 or 3.9). It should only address projects with open-ended 3.x support. If there are spots where the PR is not meeting this criteria, we should definitely update it. This PR is aligned with a risk-based testing strategy that is followed in other languages (e.g. Java 8, 11, 17 and not 8..17). If the endpoints are both passing, the risk of a deterministic language-specific issue in 3.8..3.10 is very low and acceptable for sample code given tradeoffs IMO. We can incrementally add versions back to the matrix if such issues arise -- e.g. a midpoint version (3.9) next. Also, in addition to this change, as you suggested we can review the frequency of periodic builds. FYI @nicain |
@ace-n the doc you linked is for app engine standard, but iiuc these tests are for app engine flexible - does flexible also not support 3.11? |
|
@leahecole here's my understanding: GAE FlexFlexible supports Python 3.4 - 3.7. GAE StandardThe App Engine standard template ( GAE
|
| # To save resources, please consider testing 2 versions at most. | ||
| # Test min and max supported versions which are python 3.7 and 3.11 | ||
| # Ignoring python 2.7 which is no longer supported. | ||
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], |
There was a problem hiding this comment.
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.11"], |
| # To save resources, please consider testing 2 versions at most. | ||
| # Test min and max supported versions which are python 3.7 and 3.11 | ||
| # Ignoring python 2.7 which is no longer supported. | ||
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], |
There was a problem hiding this comment.
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.11"], |
| # To save resources, please consider testing 2 versions at most. | ||
| # Test min and max supported versions which are python 3.7 and 3.11 | ||
| # Ignoring python 2.7 which is no longer supported. | ||
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], |
There was a problem hiding this comment.
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.11"], |
| # To save resources, please consider testing 2 versions at most. | ||
| # Test min and max supported versions which are python 3.7 and 3.11 | ||
| # Ignoring python 2.7 which is no longer supported. | ||
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], |
There was a problem hiding this comment.
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.11"], |
| # To save resources, please consider testing 2 versions at most. | ||
| # Test min and max supported versions which are python 3.7 and 3.11 | ||
| # Ignoring python 2.7 which is no longer supported. | ||
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], |
There was a problem hiding this comment.
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.11"], |
| # To save resources, please consider testing 2 versions at most. | ||
| # Test min and max supported versions which are python 3.7 and 3.11 | ||
| # Ignoring python 2.7 which is no longer supported. | ||
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], |
There was a problem hiding this comment.
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.11"], |
| # To save resources, please consider testing 2 versions at most. | ||
| # Test min and max supported versions which are python 3.7 and 3.11 | ||
| # Ignoring python 2.7 which is no longer supported. | ||
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], |
There was a problem hiding this comment.
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.11"], |
| "ignored_versions": ["2.7", "3.6"], | ||
| # You can opt out specific Python versions from the test. | ||
| # To save resources, please consider testing 2 versions at most. | ||
| # Test min and max supported versions which are python 3.7 and 3.11 |
There was a problem hiding this comment.
| # Test min and max supported versions which are python 3.7 and 3.11 | |
| # Test min and max supported versions which are python 3.7 and 3.10 |
| "ignored_versions": ["2.7", "3.6"], | ||
| # You can opt out specific Python versions from the test. | ||
| # To save resources, please consider testing 2 versions at most. | ||
| # Test min and max supported versions which are python 3.7 and 3.11 |
There was a problem hiding this comment.
| # Test min and max supported versions which are python 3.7 and 3.11 | |
| # Test min and max supported versions which are python 3.7 and 3.10 |
| "ignored_versions": ["2.7", "3.6"], | ||
| # You can opt out specific Python versions from the test. | ||
| # To save resources, please consider testing 2 versions at most. | ||
| # Test min and max supported versions which are python 3.7 and 3.11 |
There was a problem hiding this comment.
| # Test min and max supported versions which are python 3.7 and 3.11 | |
| # Test min and max supported versions which are python 3.7 and 3.10 |
| "ignored_versions": ["2.7", "3.6"], | ||
| # You can opt out specific Python versions from the test. | ||
| # To save resources, please consider testing 2 versions at most. | ||
| # Test min and max supported versions which are python 3.7 and 3.11 |
There was a problem hiding this comment.
| # Test min and max supported versions which are python 3.7 and 3.11 | |
| # Test min and max supported versions which are python 3.7 and 3.10 |
| # Test min and max supported versions which are python 3.7 and 3.11 | ||
| # Ignoring python 2.7 which is no longer supported. | ||
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], |
There was a problem hiding this comment.
| # Test min and max supported versions which are python 3.7 and 3.11 | |
| # Ignoring python 2.7 which is no longer supported. | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], | |
| # Test min and max supported versions which are python 3.7 and 3.10 | |
| # Ignoring python 2.7 which is no longer supported. | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.11"], |
| # Test min and max supported versions which are python 3.7 and 3.11 | ||
| # Ignoring python 2.7 which is no longer supported. | ||
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], |
There was a problem hiding this comment.
| # Test min and max supported versions which are python 3.7 and 3.11 | |
| # Ignoring python 2.7 which is no longer supported. | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], | |
| # Test min and max supported versions which are python 3.7 and 3.10 | |
| # Ignoring python 2.7 which is no longer supported. | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.11"], |
| # Test min and max supported versions which are python 3.7 and 3.11 | ||
| # Ignoring python 2.7 which is no longer supported. | ||
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], |
There was a problem hiding this comment.
| # Test min and max supported versions which are python 3.7 and 3.11 | |
| # Ignoring python 2.7 which is no longer supported. | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.10"], | |
| # Test min and max supported versions which are python 3.7 and 3.10 | |
| # Ignoring python 2.7 which is no longer supported. | |
| "ignored_versions": ["2.7", "3.8", "3.9", "3.11"], |
|
A lot has changed since this PR was proposed, and there are a lot of items for discussion. Could we close this for now and revisit the whole thing following some planned fixes? |
Please review #8872 first
Once #8872 is merged, the versions of python tested will be 2.7, 3.7, 3.8, 3.9, 3.10 and 3.11
To save on resources, let's restrict testing to the end points. So, 3.7 and 3.11 (plus 2.7 when needed).