Skip test on SkipTestException at fixture's constructor #24186

dkurt · 2023-08-22T12:38:33Z

Pull Request Readiness Checklist

In case of SkipTestException throws in the fixture's constructor, test should be skipped. Reproducer added to core module tests.

related: #24157
related: opencv/ci-gha-workflow#109

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

force_builders=Win64 OpenCL

opencv-alalek · 2023-08-22T19:25:09Z

modules/core/test/test_misc.cpp

+// Expected that SkipTestException thrown in the constructor should skip test but not fail
+struct TestFixtureSkip: public ::testing::Test {
+    TestFixtureSkip() {
+        throw SkipTestException("");


Should we cover SetUp() overload too?

Added overload for SetUp and some workaround for a static method SetUpTestCase

opencv-alalek · 2023-08-22T19:25:42Z

modules/core/test/test_misc.cpp

 #endif

+#ifdef _MSC_VER
+# pragma warning(disable:4702)  // unreachable code


Do we still need this with FAIL()?

I wanted to keep it to verify that test body won't start in case of skip exception in the constructor.

Do we still need to disable this warning?

Only on windows there is a warning: https://pullrequest.opencv.org/buildbot/builders/precommit_windows64/builds/102522. And it points to line in the constructor, not test body: 6d934f0#diff-4a53133499f4421656d6e02cc691c986d7ac26c94235f7480d81c156aebdab07R924

It is dangerous to disable this warning for whole file.
Please try to disable it locally through warning push/pop.

CV_NORETURN solved this issue

There is warning on MSVS 2019: https://pullrequest.opencv.org/buildbot/builders/precommit_opencl/builds/100186

Added if condition to avoid warnings.

opencv-alalek

Well done! Thank you 👍

dkurt · 2023-08-25T12:06:01Z

Well done! Thank you 👍

Thanks for review!

asmorkalov · 2023-09-05T10:59:25Z

@dkurt @opencv-alalek The PR breaks "skip" in OpenCV test system at least for DNN. Subset of log after the patch:

./bin/opencv_test_dnn --gtest_filter=Test_Caffe_layers*
...
[       OK ] Test_Caffe_layers.DeConvolution/3 (3 ms)
[ RUN      ] Test_Caffe_layers.InnerProduct/0, where GetParam() = NGRAPH/CPU
Test_Caffe_layers, InnerProduct call
prototxt: /home/alexander/opencv_extra/testdata/dnn/layers/layer_inner_product.prototxt
caffemodel: /home/alexander/opencv_extra/testdata/dnn/layers/layer_inner_product.caffemodel
Load 1 inputs
Load 1 refs
After check backend
[       OK ] Test_Caffe_layers.InnerProduct/0 (8 ms)
[ RUN      ] Test_Caffe_layers.InnerProduct/1, where GetParam() = OCV/OCL
Test_Caffe_layers, InnerProduct call
prototxt: /home/alexander/opencv_extra/testdata/dnn/layers/layer_inner_product.prototxt
caffemodel: /home/alexander/opencv_extra/testdata/dnn/layers/layer_inner_product.caffemodel
Load 1 inputs
Load 1 refs
After check backend
[       OK ] Test_Caffe_layers.InnerProduct/1 (4 ms)
[ RUN      ] Test_Caffe_layers.InnerProduct/2, where GetParam() = OCV/OCL_FP16
Test_Caffe_layers, InnerProduct call
[     SKIP ] Test with tag 'dnn_skip_ocl_fp16' is skipped ('dnn_skip_ocl_fp16' is in skip list)
[       OK ] Test_Caffe_layers.InnerProduct/2 (0 ms)
[ RUN      ] Test_Caffe_layers.InnerProduct/3, where GetParam() = OCV/CPU
[     SKIP ] 
[       OK ] Test_Caffe_layers.InnerProduct/3 (0 ms)
[ RUN      ] Test_Caffe_layers.Pooling_max/0, where GetParam() = NGRAPH/CPU
[     SKIP ] 
[       OK ] Test_Caffe_layers.Pooling_max/0 (0 ms)
[ RUN      ] Test_Caffe_layers.Pooling_max/1, where GetParam() = OCV/OCL
[     SKIP ] 
[       OK ] Test_Caffe_layers.Pooling_max/1 (0 ms)
[ RUN      ] Test_Caffe_layers.Pooling_max/2, where GetParam() = OCV/OCL_FP16
[     SKIP ] 
[       OK ] Test_Caffe_layers.Pooling_max/2 (0 ms)
[ RUN      ] Test_Caffe_layers.Pooling_max/3, where GetParam() = OCV/CPU
[     SKIP ] 
[       OK ] Test_Caffe_layers.Pooling_max/3 (0 ms)

All further tests are marked as skipped and even test body is not called.

./bin/opencv_test_dnn --gtest_filter=Test_Caffe_layers.Pooling_max*
...
[ RUN      ] Test_Caffe_layers.Pooling_max/0, where GetParam() = NGRAPH/CPU
[       OK ] Test_Caffe_layers.Pooling_max/0 (23 ms)
[ RUN      ] Test_Caffe_layers.Pooling_max/1, where GetParam() = OCV/OCL
[       OK ] Test_Caffe_layers.Pooling_max/1 (4 ms)
[ RUN      ] Test_Caffe_layers.Pooling_max/2, where GetParam() = OCV/OCL_FP16
[       OK ] Test_Caffe_layers.Pooling_max/2 (2 ms)
[ RUN      ] Test_Caffe_layers.Pooling_max/3, where GetParam() = OCV/CPU
[       OK ] Test_Caffe_layers.Pooling_max/3 (1 ms)
[----------] 4 tests from Test_Caffe_layers (30 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (30 ms total)
[  PASSED  ] 4 tests.

Focused test invocation works correctly.

dkurt · 2023-09-05T11:01:05Z

@asmorkalov, may I ask to revert?

asmorkalov · 2023-09-05T11:24:05Z

Yes, I'll revert the PR in several minutes.

Skip test on SkipTestException at fixture's constructor (version 2) #24250 ### Pull Request Readiness Checklist Another version of #24186 (reverted by #24223). Current implementation cannot handle skip exception at `static void SetUpTestCase` but works on `virtual void SetUp`. See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

Skip test on SkipTestException at fixture's constructor * Skip test on SkipTestException at fixture's constructor * Add warning supression * Skip Python tests if no test file found * Skip instances of test fixture with exception at SetUpTestCase * Skip test with exception at SetUp method * Try remove warning disable * Add CV_NORETURN * Remove FAIL assertion * Use findDataFile to throw Skip exception * Throw exception conditionally

Skip test on SkipTestException at fixture's constructor (version 2) opencv#24250 ### Pull Request Readiness Checklist Another version of opencv#24186 (reverted by opencv#24223). Current implementation cannot handle skip exception at `static void SetUpTestCase` but works on `virtual void SetUp`. See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

Skip test on SkipTestException at fixture's constructor * Skip test on SkipTestException at fixture's constructor * Add warning supression * Skip Python tests if no test file found * Skip instances of test fixture with exception at SetUpTestCase * Skip test with exception at SetUp method * Try remove warning disable * Add CV_NORETURN * Remove FAIL assertion * Use findDataFile to throw Skip exception * Throw exception conditionally

Skip test on SkipTestException at fixture's constructor (version 2) opencv#24250 ### Pull Request Readiness Checklist Another version of opencv#24186 (reverted by opencv#24223). Current implementation cannot handle skip exception at `static void SetUpTestCase` but works on `virtual void SetUp`. See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

dkurt added the test label Aug 22, 2023

dkurt force-pushed the ts_fixture_constructor_skip branch from 87fec42 to e8bbb0c Compare August 22, 2023 12:38

Skip test on SkipTestException at fixture's constructor

6d934f0

dkurt force-pushed the ts_fixture_constructor_skip branch from e8bbb0c to 6d934f0 Compare August 22, 2023 13:34

Add warning supression

c58eb31

dkurt requested a review from opencv-alalek August 22, 2023 14:18

opencv-alalek reviewed Aug 22, 2023

View reviewed changes

dkurt added 8 commits August 23, 2023 11:36

Skip Python tests if no test file found

81b5182

Skip instances of test fixture with exception at SetUpTestCase

acf403c

Skip test with exception at SetUp method

9782598

Try remove warning disable

97629dd

Add CV_NORETURN

1acf1df

Remove FAIL assertion

7beb1f0

Use findDataFile to throw Skip exception

768864a

Throw exception conditionally

7896769

opencv-alalek approved these changes Aug 25, 2023

View reviewed changes

opencv-alalek self-assigned this Aug 25, 2023

opencv-alalek added this to the 4.9.0 milestone Aug 25, 2023

opencv-alalek merged commit 588ddf1 into opencv:4.x Aug 25, 2023

dkurt deleted the ts_fixture_constructor_skip branch August 25, 2023 12:06

dkurt restored the ts_fixture_constructor_skip branch September 5, 2023 11:02

asmorkalov mentioned this pull request Sep 5, 2023

Revert PR 24186 as it forces skipping tests #24223

Merged

6 tasks

dkurt mentioned this pull request Sep 9, 2023

Skip test on SkipTestException at fixture's constructor (version 2) #24250

Merged

6 tasks

asmorkalov mentioned this pull request Sep 11, 2023

(5.x) Merge 4.x #24254

Merged

dkurt deleted the ts_fixture_constructor_skip branch September 20, 2023 06:12

Uh oh!

Skip test on SkipTestException at fixture's constructor #24186

Skip test on SkipTestException at fixture's constructor #24186

Uh oh!

Conversation

dkurt commented Aug 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Readiness Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkurt Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opencv-alalek left a comment

Choose a reason for hiding this comment

Uh oh!

dkurt commented Aug 25, 2023

Uh oh!

asmorkalov commented Sep 5, 2023

Uh oh!

dkurt commented Sep 5, 2023

Uh oh!

asmorkalov commented Sep 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dkurt commented Aug 22, 2023 •

edited

Loading

dkurt Aug 23, 2023 •

edited

Loading