Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Added ability to exclude items by tags. #1008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 23, 2019
Merged

Added ability to exclude items by tags. #1008

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cdb6d3e
Added ability to exclude items by tags.
jgebal Oct 20, 2019
0d64c08
Changed syntax to avoid Oracle internal errors on 11.2 and 12.1
jgebal Oct 21, 2019
49c0768
Merge remote-tracking branch 'origin/develop' into feature/exclude_paths
jgebal Oct 21, 2019
f48e7e5
Improve documentation around tags
pesse Oct 22, 2019
2caa749
Use cli 3.1.8
pesse Oct 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ env:
#utPLSQL released version directory
- UTPLSQL_DIR="utPLSQL_latest_release"
- SELFTESTING_BRANCH=${TRAVIS_BRANCH}
- UTPLSQL_CLI_VERSION="3.1.7"
- UTPLSQL_CLI_VERSION="3.1.8"
# Maven
- MAVEN_HOME=/usr/local/maven
- MAVEN_CFG=$HOME/.m2
Expand Down
73 changes: 46 additions & 27 deletions docs/userguide/annotations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1225,11 +1225,11 @@ Finished in .035261 seconds

### Tags

Tag is a label attached to the test or a suite path. It is used for identification and execution a group of tests / suites that share same tag.
Tag is a label attached to the test or a suite. It is used for identification and execution of a group of tests / suites that share the same tag.

It allows us to group a tests / suites using a various categorization and place a test / suite in multiple buckets. Same tests can be group with other tests based on the functionality , frequency, type of output etc.
It allows for grouping of tests / suites using various categorization and place tests / suites in multiple buckets. Same tests can be grouped with other tests based on the functionality , frequency, type of output etc.

e.q.
e.g.

```sql
--%tags(batch,daily,csv)
Expand All @@ -1238,29 +1238,31 @@ e.q.
or

```sql
--%tags(api,online,json)
--%tags(online,json)
--%tags(api)
```

Tags are defined as a comma separated list within the `--%tags` annotation.

When executing a test run with tag filter applied, the framework will find all tests associated with the given tags and execute them.
The framework applies `OR` logic to all specified tags so any test / suite that matches at least one tag will be included in the test run.

Tags are defined as a coma separated list. When executing a test run with tag filter applied, framework will find all tests associated with given tags and execute them. Framework applies `OR` logic when resolving a tags so any tests / suites that match at least one tag will be included in the test run.
When a suite/context is tagged, all of its children will automatically inherit the tag and get executed along with the parent. Parent suite tests are not executed, but a suitepath hierarchy is kept.

When a suite gets tagged all of its children will automatically inherit a tag and get executed along the parent. Parent suit tests are not executed. but a suitepath hierarchy is kept.

Sample tag package.

Sample test suite package with tags.
```sql
create or replace package ut_sample_test IS
create or replace package ut_sample_test is

--%suite(Sample Test Suite)
--%tag(suite1)
--%tags(api)

--%test(Compare Ref Cursors)
--%tag(test1,sample)
--%tags(complex,fast)
procedure ut_refcursors1;

--%test(Run equality test)
--%tag(test2,sample)
--%tags(simple,fast)
procedure ut_test;

end ut_sample_test;
Expand All @@ -1287,30 +1289,47 @@ end ut_sample_test;
/
```

Execution of the test is done by using a parameter `a_tags`
Execution of the test is done by using the parameter `a_tags`

```sql
select * from table(ut.run(a_path => 'ut_sample_test',a_tags => 'suite1'));
select * from table(ut.run(a_tags => 'test1,test2'));
select * from table(ut.run(a_tags => 'sample'));
select * from table(ut.run(a_path => 'ut_sample_test',a_tags => 'api'));
```
The above call will execute all tests from `ut_sample_test` package as the whole suite is tagged with `api`

begin
ut.run(a_path => 'ut_sample_test',a_tags => 'suite1');
end;
/
```sql
select * from table(ut.run(a_tags => 'complex'));
```
The above call will execute only the `ut_sample_test.ut_refcursors1` test, as only the test `ut_refcursors1` is tagged with `complex`

exec ut.run('ut_sample_test', a_tags => 'sample');
```sql
select * from table(ut.run(a_tags => 'fast'));
```
The above call will execute both `ut_sample_test.ut_refcursors1` and `ut_sample_test.ut_test` tests, as both tests are tagged with `fast`

#### Tag naming convention

Tags must follow the below naming convention:

- tag is case sensitive
- tag can contain special characters like `$#/\?-!` etc.
- tag cannot be an empty string
- tag cannot start with a dash, e.g. `-some-stuff` is **not** a valid tag
- tag cannot contain spaces, e.g. `test of batch`. To create a multi-word tag use underscores or dashes, e.g. `test_of_batch`, `test-of-batch`
- leading and trailing spaces are ignored in tag name, e.g. `--%tags( tag1 , tag2 )` becomes `tag1` and `tag2` tag names


#### Excluding tests/suites by tags

Tags should adhere to following rules:
It is possible to exclude parts of test suites with tags.
In order to do so, prefix the tag name to exclude with a `-` (dash) sign when invoking the test run.

- tags are case sensitive
- tags cannot be an empty string
- tags cannot contain spaces e.g. to create a multi-word `tag` please use underscores,dashes, dots etc. e.g. `test_of_batch`
- tags with empty spaces will be ignored during execution
- tags can contain special characters
Examples (based on above sample test suite)

```sql
select * from table(ut.run(a_tags => 'api,fast,-complex'));
```
The above call will execute all suites/contexts/tests that are marked with any of tags `api` or `fast` except those suites/contexts/tests that are marked as `complex`.
Given the above example package `ut_sample_test`, only `ut_sample_test.ut_test` will be executed.



Expand Down
10 changes: 8 additions & 2 deletions docs/userguide/running-unit-tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -284,8 +284,9 @@ select * from table(ut.run('hr.test_apply_bonus', a_random_test_order_seed => 30

# Run by Tags

In addition to the path, you can filter the tests to be run by specifying tags. Tags are defined in the test with the `--%tags`-annotation ([Read more](annotations.md#tags)).
Multiple tags are separated by comma. If multiple tags are set, all tests with __any__ of them specified are run.
In addition to the path, you can filter the tests to be run by specifying tags. Tags are defined in the test / context / suite with the `--%tags`-annotation ([Read more](annotations.md#tags)).
Multiple tags are separated by comma.
The framework applies `OR` logic to all specified tags so any test / suite that matches at least one tag will be included in the test run.

```sql
begin
Expand All @@ -296,6 +297,11 @@ end;
select * from table(ut.run('hr.test_apply_bonus', a_tags => 'suite1'))
```

You can also exclude specific tags by adding a `-` (dash) in front of the tag

```sql
select * from table(ut.run('hr.test_apply_bonus', a_tags => '-suite1'))
```

# Keeping uncommitted data after test-run

Expand Down
24 changes: 18 additions & 6 deletions source/core/ut_suite_builder.pkb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,8 @@ create or replace package body ut_suite_builder is
a_procedure_name t_object_name := null
) is
l_annotation_pos binary_integer;
l_tag_list ut_varchar2_list := ut_varchar2_list();
l_tags_list ut_varchar2_list := ut_varchar2_list();
l_tag_items ut_varchar2_list;
begin
l_annotation_pos := a_tags_ann_text.first;
while l_annotation_pos is not null loop
Expand All @@ -331,14 +332,25 @@ create or replace package body ut_suite_builder is
|| get_object_reference( a_suite, a_procedure_name, l_annotation_pos )
);
else
l_tag_list := l_tag_list multiset union distinct ut_utils.trim_list_elements(
ut_utils.string_to_table(a_tags_ann_text(l_annotation_pos),',')
);
l_tag_items := ut_utils.trim_list_elements(ut_utils.string_to_table(a_tags_ann_text(l_annotation_pos),','));
if l_tag_items is not empty then
for i in 1 .. l_tag_items.count loop
if regexp_like(l_tag_items(i),'^[^-](\S)+$') then
l_tags_list.extend();
l_tags_list(l_tags_list.last) := l_tag_items(i);
else
a_suite.put_warning(
'Invalid value "'||l_tag_items(i)||'" for "--%tags" annotation. See documentation for details on valid tag values. Annotation value ignored.'
|| get_object_reference( a_suite, a_procedure_name, l_annotation_pos )
);
end if;
end loop;
end if;
end if;
l_annotation_pos := a_tags_ann_text.next(l_annotation_pos);
end loop;
--remove empty strings from table list e.g. tag1,,tag2 and conver to rows
a_list := ut_utils.convert_collection( ut_utils.filter_list(l_tag_list,ut_utils.gc_word_no_space) );
--remove empty strings from table list e.g. tag1,,tag2 and convert to rows
a_list := ut_utils.convert_collection( ut_utils.filter_list(set(l_tags_list),ut_utils.gc_word_no_space) );
end;

procedure set_seq_no(
Expand Down
44 changes: 33 additions & 11 deletions source/core/ut_suite_cache_manager.pkb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ create or replace package body ut_suite_cache_manager is
and ( {:path:}
and {:object_name:}
and {:procedure_name:}
)
)
)
),
{:tags:}
suitepaths as (
Expand Down Expand Up @@ -106,8 +106,8 @@ create or replace package body ut_suite_cache_manager is
function get_path_sql(a_path in varchar2) return varchar2 is
begin
return case when a_path is not null then q'[
:l_path||'.' like c.path || '.%' /*all children and self*/
or ( c.path||'.' like :l_path || '.%' --all parents
:l_path||'.' like c.path || '.%' /*all parents and self*/
or ( c.path||'.' like :l_path || '.%' /*all children and self*/
]'
else ' :l_path is null and ( :l_path is null ' end;
end;
Expand All @@ -129,22 +129,31 @@ create or replace package body ut_suite_cache_manager is
function get_tags_sql(a_tags_count in integer) return varchar2 is
begin
return case when a_tags_count > 0 then
q'[filter_tags as (
q'[included_tags as (
select c.obj.path as path
from suite_items c
where c.obj.tags multiset intersect :a_tag_list is not empty
),
where c.obj.tags multiset intersect :a_include_tag_list is not empty or :a_include_tag_list is empty
),
excluded_tags as (
select c.obj.path as path
from suite_items c
where c.obj.tags multiset intersect :a_exclude_tag_list is not empty
),
suite_items_tags as (
select c.*
from suite_items c
where exists (
select 1 from filter_tags t
where t.path||'.' like c.obj.path || '.%' /*all children and self*/
or c.obj.path||'.' like t.path || '.%' --all parents
select 1 from included_tags t
where t.path||'.' like c.obj.path || '.%' /*all parents and self*/
or c.obj.path||'.' like t.path || '.%' /*all children and self*/
)
and not exists (
select 1 from excluded_tags t
where c.obj.path||'.' like t.path || '.%' /*all children and self*/
)
),]'
else
q'[dummy as (select 'x' from dual where :a_tag_list is null ),]'
q'[dummy as (select 'x' from dual where :a_include_tag_list is null and :a_include_tag_list is null and :a_exclude_tag_list is null),]'
end;
end;

Expand Down Expand Up @@ -187,10 +196,23 @@ create or replace package body ut_suite_cache_manager is
l_sql varchar2(32767);
l_suite_item_name varchar2(20);
l_tags ut_varchar2_rows := coalesce(a_tags,ut_varchar2_rows());
l_include_tags ut_varchar2_rows;
l_exclude_tags ut_varchar2_rows;
l_object_owner varchar2(250) := ut_utils.qualified_sql_name(a_object_owner);
l_object_name varchar2(250) := ut_utils.qualified_sql_name(a_object_name);
l_procedure_name varchar2(250) := ut_utils.qualified_sql_name(a_procedure_name);
begin

select column_value
bulk collect into l_include_tags
from table(l_tags)
where column_value not like '-%';

select ltrim(column_value,'-')
bulk collect into l_exclude_tags
from table(l_tags)
where column_value like '-%';

if a_path is null and a_object_name is not null then
select min(c.path)
into l_path
Expand All @@ -216,7 +238,7 @@ create or replace package body ut_suite_cache_manager is

execute immediate l_sql
bulk collect into l_results
using upper(l_object_owner), l_path, l_path, upper(a_object_name), upper(a_procedure_name), l_tags, a_random_seed;
using upper(l_object_owner), l_path, l_path, upper(a_object_name), upper(a_procedure_name), l_include_tags, l_include_tags, l_exclude_tags, a_random_seed;
return l_results;
end;

Expand Down
62 changes: 61 additions & 1 deletion test/ut3_tester/core/test_suite_builder.pkb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1431,7 +1431,67 @@ create or replace package body test_suite_builder is
'%</UT_LOGICAL_SUITE>%'
);

end;
end;

procedure test_spaces_in_tag is
l_actual clob;
l_annotations ut3.ut_annotations;
begin
--Arrange
l_annotations := ut3.ut_annotations(
ut3.ut_annotation(2, 'suite','testsuite', null),
ut3.ut_annotation(3, 'tags',' good_tag , bad tag , good-tag ', null),
ut3.ut_annotation(8, 'test','Some test', 'test_procedure'),
ut3.ut_annotation(9, 'tags',' good_tag , bad tag , good-tag ', 'test_procedure')
);
--Act
l_actual := invoke_builder_for_annotations(l_annotations, 'SOME_PACKAGE');
--Assert
ut.expect(l_actual).to_be_like(
'%<TAGS><VARCHAR2>good_tag</VARCHAR2><VARCHAR2>good-tag</VARCHAR2></TAGS>%'||
'<TAGS><VARCHAR2>good_tag</VARCHAR2><VARCHAR2>good-tag</VARCHAR2></TAGS>%'
);
ut.expect(l_actual).to_be_like(
'%<WARNINGS><VARCHAR2>Invalid value &quot;bad tag&quot; for &quot;--%tags&quot; annotation.'||
' See documentation for details on valid tag values. Annotation value ignored.
at package &quot;UT3_TESTER.SOME_PACKAGE&quot;, line 3</VARCHAR2><VARCHAR2>%'
);
ut.expect(l_actual).to_be_like(
'%<VARCHAR2>Invalid value &quot;bad tag&quot; for &quot;--%tags&quot; annotation.'||
' See documentation for details on valid tag values. Annotation value ignored.
at package &quot;UT3_TESTER.SOME_PACKAGE.TEST_PROCEDURE&quot;, line 9</VARCHAR2></WARNINGS>%'
);
end;

procedure test_minus_in_tag is
l_actual clob;
l_annotations ut3.ut_annotations;
begin
--Arrange
l_annotations := ut3.ut_annotations(
ut3.ut_annotation(2, 'suite','testsuite', null),
ut3.ut_annotation(3, 'tags',' good_tag , -invalid_tag , good-tag ', null),
ut3.ut_annotation(8, 'test','Some test', 'test_procedure'),
ut3.ut_annotation(9, 'tags',' good_tag , -invalid_tag , good-tag ', 'test_procedure')
);
--Act
l_actual := invoke_builder_for_annotations(l_annotations, 'SOME_PACKAGE');
--Assert
ut.expect(l_actual).to_be_like(
'%<TAGS><VARCHAR2>good_tag</VARCHAR2><VARCHAR2>good-tag</VARCHAR2></TAGS>%'||
'<TAGS><VARCHAR2>good_tag</VARCHAR2><VARCHAR2>good-tag</VARCHAR2></TAGS>%'
);
ut.expect(l_actual).to_be_like(
'%<WARNINGS><VARCHAR2>Invalid value &quot;-invalid_tag&quot; for &quot;--%tags&quot; annotation.'||
' See documentation for details on valid tag values. Annotation value ignored.
at package &quot;UT3_TESTER.SOME_PACKAGE&quot;, line 3</VARCHAR2><VARCHAR2>%'
);
ut.expect(l_actual).to_be_like(
'%<VARCHAR2>Invalid value &quot;-invalid_tag&quot; for &quot;--%tags&quot; annotation.'||
' See documentation for details on valid tag values. Annotation value ignored.
at package &quot;UT3_TESTER.SOME_PACKAGE.TEST_PROCEDURE&quot;, line 9</VARCHAR2></WARNINGS>%'
);
end;

end test_suite_builder;
/
Loading