Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Support "AND" logic to tags for the tests runner #1042

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
alexeyhimself opened this issue Jan 31, 2020 · 2 comments · Fixed by #1250
Closed

Support "AND" logic to tags for the tests runner #1042

alexeyhimself opened this issue Jan 31, 2020 · 2 comments · Fixed by #1250
Assignees
Milestone

Comments

@alexeyhimself
Copy link

Is your feature request related to a problem? Please describe.
We use different kinds of tags to formulate a scope for the tests run. We have different kinds of tests: post deployment verification (pdv), functional, data quality evaluation (dqe) tests, unit tests and so on. We have different kinds of releases (relatively defined: current, next; explicitly defined: v63_0, v63_1, v64_0) as well. And we'd wanted to run test cases among these categories.

For example, we'd wanted to run all "p0 tests in v64_0 release". We have tests tagged with priorities: "p0", "p1", "p2", "p3". We have tests marked with category: "pdv", "dqe", "functional", etc. We have special tags like: "long_running", "not_for_production", etc. We have release tags: "v63_0", "v64_0", etc. And we tag our tests, for example, as follows:

--%tag(v64_0, pdv, not_for_production)
--%tag(current, pdv)
--%tag(current, p0, functional)
--%tag(current, p0, performance)
--%tag(current, p1, functional)
--%tag(v63_1, dqe, long_running)

When we want to run all "p0" tests in "v64_0" release, we have no way to do it now. We either run all "p0" tests or all "v64_0" tests, or both - due to "OR" logic for the tests run. We want both "OR" and "AND" logic as well - to form different kinds of tests scope.

Describe the solution you'd like
We'd wanted you support this kind of separators to apply different kinds of logic to the tests runner:

select * from table(ut.run(a_tags=>'p0||p1')); -- "OR" logic
select * from table(ut.run(a_tags=>'p0&&v64_0')); -- "AND" logic
select * from table(ut.run(a_tags=>'(p0||p1)&&v64_0')); -- mixed logic with priority defined by ()

Describe alternatives you've considered
Alternative to this - is to use self-written tests installer to cleanup all test procedures before tests run and install only required tests and run all installed tests. I.e. alternative - is to delegate this scope defining function to the self-written installer. This is not useful and not nice. And this installation-deinstallation packages activity produces db system logs that we don't want. We'd better installed our tests once and then run subsets of them on demand.

Additional context
No

@jgebal
Copy link
Member

jgebal commented Feb 2, 2020

Hi @alexeyhimself

Thanks for raising this request.

This kind of logic will require quite a bit of parsing and error handling.
The logic for building included items (suites/parent suites/tests) would also become quite complex.

Though the request you're making is absolutely valid, I don't see an easy way of implementing it at the moment.
Also, we currently suport comma-separated lists of tags to include/exclude (as or logic) we would need to maintain backward-compatibility.

@jgebal
Copy link
Member

jgebal commented Mar 13, 2020

@alexeyhimself

In our current implementation we use the following logic:

  1. Split the tags into wo lists (include/exclude) based on the - prefix in command.
  2. Filter suite items to be executed by using IN / NOT IN logic

The type of filtering you propose would mean building a complex SQL condition dynamically. So instead of having a list of items that should be used as filters we would have a list of expressions.

Each of expressions would need to be evaluated separately.
Only whole expression could be used for inclusion or exclusion.

It is an interesting challenge.

Do you have any idea on how would you apply such filter criteria to a SQL table or a nested table/varray?

@lwasylow lwasylow self-assigned this Mar 19, 2023
@jgebal jgebal added this to the v3.1.14 milestone May 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants