Feature/1042 add `and` to tags #1250

lwasylow · 2023-03-31T15:31:01Z

Introduction of expressions for running tests with tags

Tag Expressions

Tag expressions are boolean expressions created by combining tags with the !, &, | operators. Tag expressions can be grouped using ( and ) braces. Grouping tag expressions affects operator precedence.

Two reserved keywords, any and none, can be used when creating a tag expression to run tests.

any keyword represents tests and suites with any tags
none keyword represents tests and suites without tags

These keywords may be combined with other expressions just like normal tags.

Note: When specifying none, be aware that it will exclude any tests/suites/contexts contained within a tagged suite.

Operator	Meaning
!	not
&	and
\|	or

If you are tagging your tests across multiple dimensions, tag expressions help you to select which tests to execute. When tagging by test type (e.g., micro, integration, end-to-end) and feature (e.g., product, catalog, shipping), the following tag expressions can be useful.

Tag Expression	Selection
product	all tests for product
catalog \| shipping	all tests for catalog plus all tests for shipping
catalog & shipping	all tests that are tagged with both `catalog` and `shipping` tags
product & !end-to-end	all tests tagged `product`, except the tests tagged `end-to-end`
(micro \| integration) & (product \| shipping)	all micro or integration tests for product or shipping

Taking the last expression above (micro | integration) & (product | shipping)

--%tags	included in run
micro	no
integration	no
micro	no
product	no
shipping	no
micro	no
micro, integration	no
product, shipping	no
micro, product	yes
micro, shipping	yes
integration, product	yes
integration, shipping	yes
integration, micro, shipping	yes
integration, micro, product	yes
integration, shipping ,product	yes
micro, shipping ,product	yes
integration, micro, shipping ,product	yes

Excluding tests/suites by tags

It is possible to exclude parts of test suites with tags.
In order to do so, prefix the tag name to exclude with a ! (exclamation) sign when invoking the test run which is equivalent of - (dash) in legacy notation.
Examples:

select * from table(ut.run(a_tags => '(api|fast)&!complex'));

select * from table(ut.run(a_tags => '(api|fast)&!complex&!test1'));

Address flaky expression checker to be reworked.

This reverts commit b5ad747. Revert "Adding validation for tag expression" This reverts commit 1478b0d.

…ture/1042_add_and_to_tags

codecov · 2023-03-31T15:36:39Z

Codecov Report

Patch coverage: 99.24% and project coverage change: +0.11 🎉

Comparison is base (8b01a39) 94.78% compared to head (46ffe73) 94.89%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1250      +/-   ##
===========================================
+ Coverage    94.78%   94.89%   +0.11%     
===========================================
  Files          109      111       +2     
  Lines         5173     5286     +113     
  Branches        24       24              
===========================================
+ Hits          4903     5016     +113     
- Misses         262      263       +1     
+ Partials         8        7       -1

Flag	Coverage Δ
11XE	`94.27% <99.24%> (+0.15%)`	⬆️
12.1EE	`94.27% <98.48%> (+0.13%)`	⬆️
18XE	`94.48% <99.24%> (+0.10%)`	⬆️
21XE	`94.65% <99.24%> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
source/core/types/ut_run.tpb	`93.93% <ø> (ø)`
source/core/ut_utils.pkb	`94.69% <ø> (+0.24%)`	⬆️
source/core/ut_suite_tag_filter.pkb	`99.09% <99.09%> (ø)`
source/api/ut_runner.pkb	`95.91% <100.00%> (-0.13%)`	⬇️
source/core/types/ut_stack.tpb	`100.00% <100.00%> (ø)`
source/core/ut_suite_cache_manager.pkb	`97.01% <100.00%> (-0.43%)`	⬇️
source/core/ut_suite_manager.pkb	`96.72% <100.00%> (+0.01%)`	⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

github-actions · 2023-03-31T15:37:09Z

Unit Test Results

      1 files       2 suites 5m 24s ⏱️
1 047 tests 1 043 ✔️ 4 💤 0 ❌
1 050 runs 1 042 ✔️ 8 💤 0 ❌

Results for commit 46ffe73.

♻️ This comment has been updated with latest results.

lwasylow · 2023-04-01T17:44:16Z

So seems like there is no hit in perf:
New code:
Finished in 9.393748 seconds
2000 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

First run took+000000000 00:01:26.587663000

Finished in 16.495174 seconds
2000 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

Second run took+000000000 00:01:17.189142000

Finished in 13.658072 seconds
2000 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

third run took+000000000 00:01:20.499584000

old

Finished in 16.167675 seconds
2000 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

First run took+000000000 00:03:37.830094000

Finished in 13.127262 seconds
2000 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

Second run took+000000000 00:03:29.262099000

Finished in 10.98218 seconds
2000 tests, 0 failed, 0 errored, 0 disabled, 0 warning(s)

Third run took+000000000 00:03:01.665203000

…e moment.

lwasylow · 2023-04-07T05:32:20Z

@jgebal I'm also considering to add keywords none() and any(). What you think? It will take an extra time to implement.
Also not sure if that would have any usage so maybe keep it simple :)

jgebal · 2023-04-07T06:16:32Z

I can review this PR after Wednesday.
Sorry for delay.

What is the idea behind any() and none()?
Can you explain how it would work?

lwasylow · 2023-04-07T06:31:01Z

Aye, no rush. As for keyword as per junit5
Two special expressions are supported, any() and none(), which select all tests with any tags at all, and all tests without any tags, respectively. These special expressions may be combined with other expressions just like normal tags. docs

lwasylow · 2023-04-07T17:15:24Z

Got some small changes to do still.
Want to validate using a shunting algorithm and polish reverse notation.
That should allow me to put a back later on to SQL filter of tag members string in infix notation.
The initial idea of using regex was flaky and unreliable.
The RPN supports all complex expressions.

…ix ( Reverse Polish Notation). This allows us to more flexibility of using boolean expressions and not limited to flaky regex.

Fix object name length.

jgebal

I really like the code you wrote. It's realitvely easy to digest. The except rnp and infix/postfix logic is of course quite complex and hard to grasp without some reading.

I added quite few comments but they are mostly cosmetic/readability and maintainability oriented.

docs/userguide/annotations.md

source/api/ut_runner.pkb

source/core/types/ut_stack.tps

source/core/types/ut_stack.tpb

source/core/ut_utils.pkb

source/core/ut_utils.pks

source/core/ut_suite_cache_manager.pkb

test/ut3_tester/core/test_ut_utils.pkb

jgebal

Not a complete review yet, but few comments.
I'll need a bit more time to look through it all.

docs/userguide/running-unit-tests.md

docs/userguide/annotations.md

source/core/ut_suite_cache_manager.pkb

source/core/ut_suite_cache_manager.pks

jgebal · 2023-04-18T19:45:43Z

Aye, no rush. As for keyword as per junit5 Two special expressions are supported, any() and none(), which select all tests with any tags at all, and all tests without any tags, respectively. These special expressions may be combined with other expressions just like normal tags. docs

Sounds like a great idea.
You want to do it as part of this PR or separate after this one is done?

docs/userguide/running-unit-tests.md

lwasylow · 2023-04-19T01:08:41Z

Aye, no rush. As for keyword as per junit5 Two special expressions are supported, any() and none(), which select all tests with any tags at all, and all tests without any tags, respectively. These special expressions may be combined with other expressions just like normal tags. docs

Sounds like a great idea. You want to do it as part of this PR or separate after this one is done?

Let's get that in new PR.
It will be cleaner to review and write tests as well as update documentation.
I started already working on change and seems like it small one but need to write more complex tests and scenarios.

lwasylow · 2023-04-25T16:31:54Z

Aye, no rush. As for keyword as per junit5 Two special expressions are supported, any() and none(), which select all tests with any tags at all, and all tests without any tags, respectively. These special expressions may be combined with other expressions just like normal tags. docs

Sounds like a great idea. You want to do it as part of this PR or separate after this one is done?

Let's get that in new PR. It will be cleaner to review and write tests as well as update documentation. I started already working on change and seems like it small one but need to write more complex tests and scenarios.

I have added into existing PR as the change was minimal.

source/core/ut_suite_cache_manager.pkb

docs/userguide/annotations.md

docs/userguide/running-unit-tests.md

source/core/ut_suite_tag_filter.pkb

sonarqubecloud · 2023-04-27T18:58:01Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

99.3% Coverage
0.0% Duplication

alexeyhimself · 2023-04-28T10:58:34Z

👏👏👏

lwasylow added 9 commits March 27, 2023 21:56

Checkpoint

0daab33

Address too long identified in 11g.

adbc76e

Address flaky expression checker to be reworked.

Adding validation for tag expression

1478b0d

Comment out to see why its failing.

b5ad747

Revert "Comment out to see why its failing."

06cb054

This reverts commit b5ad747. Revert "Adding validation for tag expression" This reverts commit 1478b0d.

Adding validate function, with no calls

e87d39f

Remove a & from text

97537de

Extra changes and added tests

2a0f99a

Merge branch 'develop' of https://github.com/utPLSQL/utPLSQL into fea…

5b46140

…ture/1042_add_and_to_tags

lwasylow added 2 commits April 1, 2023 08:31

Address sonar coverage issues.

b30688c

Adding tests covering exception of invalid tags

0c41a0f

lwasylow added 2 commits April 1, 2023 11:26

Removing that , we will not implement that, there is no benefit at th…

543685d

…e moment.

Removing force

0d3cfa1

lwasylow requested a review from jgebal April 7, 2023 05:29

lwasylow added 9 commits April 9, 2023 18:47

Changing to use Dijkstra algorithm to parse infix notation into postf…

20e3177

…ix ( Reverse Polish Notation). This allows us to more flexibility of using boolean expressions and not limited to flaky regex.

Missing slash at end of type

f51cc99

Cleanup.

4b8e2ab

Fix object name length.

Update tests after removed function

84e8684

Tidy up tests

2e7a766

Added ut_stack to uninstall

cbdf83a

Addressing test failures and sonar smells

436eb5b

Update name

3d77514

Update tests and code

bf6959f

lwasylow added 2 commits April 11, 2023 20:33

fixing typo in docs

d8233ff

Removed unused variable

bd860f6

jgebal reviewed Apr 12, 2023

View reviewed changes

lwasylow added 5 commits April 12, 2023 18:40

Stage 1 Resolving PR comments

313d5e9

Separate tag logic.

02a071c

Fix uninstall

b8b66ee

Various PR fixe

077fdb1

Update tests and code

01e5364

jgebal reviewed Apr 17, 2023

View reviewed changes

Addressing changes via PR review.

dc0b4a6

jgebal reviewed Apr 18, 2023

View reviewed changes

docs/userguide/running-unit-tests.md Show resolved Hide resolved

Update docs

ef1c02b

Adding any and none

1551ea5

Update docs

9dee7e0

jgebal reviewed Apr 26, 2023

View reviewed changes

source/core/ut_suite_cache_manager.pkb Show resolved Hide resolved

docs/userguide/annotations.md Outdated Show resolved Hide resolved

docs/userguide/running-unit-tests.md Outdated Show resolved Hide resolved

jgebal reviewed Apr 26, 2023

View reviewed changes

source/core/ut_suite_tag_filter.pkb Show resolved Hide resolved

lwasylow added 2 commits April 26, 2023 17:30

Resolving PR

beb9a3a

Update note

46ffe73

jgebal approved these changes Apr 28, 2023

View reviewed changes

jgebal merged commit 46e0a8e into develop Apr 28, 2023

jgebal deleted the feature/1042_add_and_to_tags branch April 28, 2023 09:09

jgebal changed the title ~~Feature/1042 add and to tags~~ Feature/1042 add and to tags May 21, 2023

jgebal added this to the v3.1.14 milestone May 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/1042 add `and` to tags #1250

Feature/1042 add `and` to tags #1250

lwasylow commented Mar 31, 2023 •

edited by jgebal

Loading

codecov bot commented Mar 31, 2023 •

edited

Loading

github-actions bot commented Mar 31, 2023 •

edited

Loading

lwasylow commented Apr 1, 2023

lwasylow commented Apr 7, 2023

jgebal commented Apr 7, 2023

lwasylow commented Apr 7, 2023

lwasylow commented Apr 7, 2023 •

edited

Loading

jgebal left a comment

jgebal left a comment

jgebal commented Apr 18, 2023

lwasylow commented Apr 19, 2023

lwasylow commented Apr 25, 2023

sonarqubecloud bot commented Apr 27, 2023

alexeyhimself commented Apr 28, 2023

Feature/1042 add and to tags #1250

Feature/1042 add and to tags #1250

Conversation

lwasylow commented Mar 31, 2023 • edited by jgebal Loading

Introduction of expressions for running tests with tags

Tag Expressions

Excluding tests/suites by tags

codecov bot commented Mar 31, 2023 • edited Loading

Codecov Report

github-actions bot commented Mar 31, 2023 • edited Loading

Unit Test Results

lwasylow commented Apr 1, 2023

lwasylow commented Apr 7, 2023

jgebal commented Apr 7, 2023

lwasylow commented Apr 7, 2023

lwasylow commented Apr 7, 2023 • edited Loading

jgebal left a comment

Choose a reason for hiding this comment

jgebal left a comment

Choose a reason for hiding this comment

jgebal commented Apr 18, 2023

lwasylow commented Apr 19, 2023

lwasylow commented Apr 25, 2023

sonarqubecloud bot commented Apr 27, 2023

alexeyhimself commented Apr 28, 2023

Feature/1042 add `and` to tags #1250

Feature/1042 add `and` to tags #1250

lwasylow commented Mar 31, 2023 •

edited by jgebal

Loading

codecov bot commented Mar 31, 2023 •

edited

Loading

github-actions bot commented Mar 31, 2023 •

edited

Loading

lwasylow commented Apr 7, 2023 •

edited

Loading