Feature/cursor pk join #666

lwasylow · 2018-05-11T17:30:01Z

Changes to implement join by cursor and unordered compare.
fixes #453

Adding compare for join by

…rsor_pk_join

…into feature/cursor_pk_join

Update if statement to simplify Remove synonyms for temp tables

updated docs

…into feature/cursor_pk_join

Capture PK Hash to improve lookup

fix to get columns diff

…rsor_pk_join

Small fixes and optimizations

This reverts commit b9ca972. for one file

…into feature/cursor_pk_join

jgebal · 2018-05-12T17:15:21Z

@lwasylow , can you update the branch and resolve conflicts

…rsor_pk_join

…ture/cursor_pk_join

…into feature/cursor_pk_join

jgebal · 2018-05-14T12:04:33Z

test/core/expectations/compound_data/test_expectations_cursor.pkb

+    order by 1,2,3 desc;
+
+    --Act
+    ut3.ut.expect(l_actual).to_equal(l_expected).join_by('OWNER');


How is this test working? I guess the OWNER field is not unique in the dataset so we get some sort of cartesian join here.
Is this the test working by lock or is the logic so clever to somehow make it work?

We are partition row number over PK HASH and join on pk_hash and duplicate no.
Because duplicate_no in both sets match we got success.
If we have an extra row we would get extra row error.
But agree that test is probably not the best example of join on uniquness.

Right, so in fact we are preventing cartesian joins and if key is not unique we would do a join by a rownum within join key value.
Worth documenting - I already forgot it was supposed to work like this

Documentation has been updated with more details

jgebal · 2018-05-14T12:08:11Z

test/core/expectations/compound_data/test_expectations_cursor.pkb

+ l_expected_message := q'[%Actual: refcursor [ count = 3 ] was expected to equal: refcursor [ count = 3 ]%
+Diff:%
+%Unable to join sets:%
+%Unknown key to join by in expected:/*/OWNER%


Maybe better to name it like
Join key /*/OWNER does not exist in expected

Updated to refactor new name

jgebal · 2018-05-14T12:11:39Z

test/core/expectations/compound_data/test_expectations_cursor.pkb

+    ut.expect(l_actual_message).to_be_like(l_expected_message);
+  end;
+
+  procedure cur_joinby_comp_twocols_exkey is


This behaviour should be mentioned in documentation

Updated docs

jgebal · 2018-05-14T12:17:47Z

test/core/expectations/compound_data/test_expectations_cursor.pkb

+ l_expected_message := q'[%Actual: refcursor [ count = 3 ] was expected to equal: refcursor [ count = 3 ]%
+Diff:%
+%Unable to join sets:%
+%Unknown key to join by in actual:/*/RNI%]';


Lets remove the /*/ prefix so it is aligned with KEY: TEST in line 1287 (Expected: <USER_ID>-600</USER_ID> for key: TEST)
I think we should be consistent when showing messages to users.

Challenge here is that we do not know what column name is missing only what XPATH expression to join by is not present. Because we allow user to pass in Xpath expression.
The message

Expected: <USER_ID>-600</USER_ID> for key: TEST

Is extracted from piece of XML using xpath.

In scenario where there is missing key we cannot use that XPATH to extract information about column and we cannot assume that xpath expression is a column name and not some xpath search string with wildcard.

Can you do a testcase for join on nested table element in cursor column?
Would be interesting to see such join happening.

Few tests added for nested table element of test_dummy_object from test helper package

Updated documentation Shift packages around for current user id

…sor_pk_join

jgebal · 2018-05-20T21:01:41Z

test/core/expectations/compound_data/test_expectations_cursor.pkb

+ l_expected_message := q'[%Actual: refcursor [ count = % ] was expected to equal: refcursor [ count = % ]
+%Diff:%
+%Rows: [ 2 differences ]%
+%Missing:  <USERNAME>TEST</USERNAME><USER_ID>-600</USER_ID> for key: -600:TEST%


I have a doubt about readability of this message to the user.
Missing: <USERNAME>TEST</USERNAME><USER_ID>-600</USER_ID> for key: -600:TEST
If we don't join by PK, we use syntax like
Row No. 3 - Missing: <RN>3</RN>
How about if we would present it to the user in a similar way:
PK <USERNAME>TEST</USERNAME><USER_ID>-600</USER_ID> missing.

I think we should exclude PK columns from the Missing/Extra data list - this is duplicated info.

Agree, will update

jgebal · 2018-05-20T21:05:40Z

test/core/expectations/compound_data/test_expectations_cursor.pkb

+ l_expected_message := q'[%Actual: refcursor [ count = % ] was expected to equal: refcursor [ count = % ]
+%Diff:%
+%Rows: [ 1 differences ]%
+%Expected: <USER_ID>-600</USER_ID> for key: TEST:Y%


I believe, this could be more readable if it was:
PK <USERNAME>TEST</USERNAME><IS_VALID>Y</IS_VALID> - Expected: <USER_ID>-600</USER_ID>
PK <USERNAME>TEST</USERNAME><IS_VALID>Y</IS_VALID> - Actual: <USER_ID>-610</USER_ID>

jgebal · 2018-05-20T21:06:33Z

test/core/expectations/compound_data/test_expectations_cursor.pkb

+    --Act
+    ut3.ut.expect(l_actual).to_equal(l_expected).unordered;
+    --Assert
+    ut.expect(expectations.failed_expectations_data()).to_be_not_null();


What data will the user see in this case?
Can you test for an expected message?

jgebal · 2018-05-20T21:08:38Z

test/core/expectations/compound_data/test_expectations_cursor.pkb

+ l_expected_message := q'[%Actual: refcursor [ count = 3 ] was expected to equal: refcursor [ count = 3 ]%
+Diff:%
+%Unable to join sets:%
+%Join key /*/RN does not exists in expected%


Can we remove the leading /*/ from the message?
That way we get actual column name rather than XPATH.
Not sure if it makes sense but probably 98% of use-cases will be joins by regular columns (not nested elements)

Yeah we can make that assumption just have to updated docs to mention that.

jgebal · 2018-05-20T21:11:03Z

test/core/expectations/compound_data/test_expectations_cursor.pkb

+    ut.expect(expectations.failed_expectations_data()).to_be_empty();
+  end;
+
+  procedure compare_nested_tab_col_un is


The test is actually for an object in a cursor column, not a nested table.
Can you rename those tests and add an additional test for a nested table (like ut_varchar2_list or so)?
I wonder how it will work with a nested table containing more than one row.
Maybe we should not support that at all?

that looks fine select * from table(ut_varchar2_list) for one or two columns ut_key_value_pairs

update tests to include table type Update key value pairs type

…ned with new lines char in 11g

This reverts commit b457d69.

…ults."" This reverts commit 7577a95.

jgebal · 2018-05-23T22:57:57Z

test/core/expectations/compound_data/test_expectations_cursor.pkb

+      from table(l_expected_tab) order by 1 desc;
+
+    --Act
+    ut3.ut.expect(l_actual).to_equal(l_expected).join_by('KEY');


What will happen when your cursor l_actual is:

select rownum rn, l_actual_tab from dual connect by level <=2; ut3.ut.expect(l_actual).to_equal(l_expected).join_by('RN,L_ACTUAL_TAB/KEY');

The dataset is then something like:

<?xml version="1.0"?> <ROWSET> <ROW> <RN>1</RN> <L_ACTUAL_TAB> <UT_KEY_VALUE_PAIR> <KEY>1</KEY> <VALUE>Something 1</VALUE> </UT_KEY_VALUE_PAIR> <UT_KEY_VALUE_PAIR> <KEY>2</KEY> <VALUE>Something 2</VALUE> </UT_KEY_VALUE_PAIR> </L_ACTUAL_TAB> </ROW> <ROW> <RN>2</RN> <L_ACTUAL_TAB> <UT_KEY_VALUE_PAIR> <KEY>1</KEY> <VALUE>Something 1</VALUE> </UT_KEY_VALUE_PAIR> <UT_KEY_VALUE_PAIR> <KEY>2</KEY> <VALUE>Something 2</VALUE> </UT_KEY_VALUE_PAIR> </L_ACTUAL_TAB> </ROW> </ROWSET>

That will not be (and probably should not be) supporter but might be still worth documenting in test.
We just don't know what will happen as there is not test case to document how the ut.expect will behave.

jgebal

This looks really good now!
Will you be able to add one more test case (as per comment)?

lwasylow · 2018-05-24T05:21:10Z

Sure thing

lwasylow added 21 commits April 23, 2018 19:44

Initial check in of unordered comparision

b9ca972

Added methods for join_by

41ae0ff

Adding compare for join by

Merge branch 'develop' of github.com:lwasylow/utPLSQL into feature/cu…

bcab19c

…rsor_pk_join

Updated JoinBy query

a3406ee

Updated JoinBy query

3ca16f7

Merge branch 'feature/cursor_pk_join' of github.com:lwasylow/utPLSQL …

d1b4fe0

…into feature/cursor_pk_join

fixes to test name length

a5e6aae

Update documentation

8a33c25

Update if statement to simplify Remove synonyms for temp tables

Update order by to be done before max row nums are pulled

0fc68d1

Updated join by

7f3d12e

updated docs

Updated join by

b8840a1

updated docs

Merge branch 'feature/cursor_pk_join' of github.com:lwasylow/utPLSQL …

8133f11

…into feature/cursor_pk_join

Update to sort random order cause of optimizer

aaa0cc4

Capture PK Hash to improve lookup

Update pk join cursor

784c509

fix to get columns diff

Merge branch 'develop' of github.com:lwasylow/utPLSQL into feature/cu…

fdff62a

…rsor_pk_join

Added failure on missing join by key in cursor.

ac042d0

Added unique key name as part of compare results

38aba43

Small fixes and optimizations

Added unique key name as part of compare results

66974e0

Small fixes and optimizations

Revert "Initial check in of unordered comparision"

0fd8be8

This reverts commit b9ca972. for one file

Update blank line

1bf5bad

Merge branch 'feature/cursor_pk_join' of github.com:lwasylow/utPLSQL …

278dc68

…into feature/cursor_pk_join

lwasylow added 5 commits May 12, 2018 18:18

Merge branch 'develop' of github.com:lwasylow/utPLSQL into feature/cu…

ec258dc

…rsor_pk_join

Merge branch 'develop' of github.com:lwasylow/utPLSQL into feature/cu…

5eeb7f6

…rsor_pk_join

Merge branch 'develop' of https://github.com/utPLSQL/utPLSQL into fea…

ad345f1

…ture/cursor_pk_join

Merge branch 'develop' of https://github.com/utPLSQL/utPLSQL into fea…

5c894c6

…ture/cursor_pk_join

Merge branch 'feature/cursor_pk_join' of github.com:lwasylow/utPLSQL …

44cbcc8

…into feature/cursor_pk_join

jgebal reviewed May 14, 2018

View reviewed changes

lwasylow added 2 commits May 18, 2018 00:14

Added changes to cater for nested tables

cc718d9

Updated documentation Shift packages around for current user id

Merge branch 'develop' of github.com:utPLSQL/utPLSQL into feature/cur…

61a334a

…sor_pk_join

jgebal reviewed May 20, 2018

View reviewed changes

lwasylow added 6 commits May 22, 2018 20:18

Updated code with a new display

77e4468

update tests to include table type Update key value pairs type

Update to handle new line in xmltype that caused pk value being retur…

a7f167c

…ned with new lines char in 11g

Updates to the ordering of sets to get consistent results.

b457d69

Revert "Updates to the ordering of sets to get consistent results."

7577a95

This reverts commit b457d69.

Refactoring to simplify code

890c1d2

Revert "Revert "Updates to the ordering of sets to get consistent res…

4d8b964

…ults."" This reverts commit 7577a95.

jgebal reviewed May 23, 2018

View reviewed changes

lwasylow and others added 2 commits May 24, 2018 20:33

Update to test and docs

f77f13a

Merge branch 'develop' into feature/cursor_pk_join

2f662ed

jgebal approved these changes May 25, 2018

View reviewed changes

jgebal added this to the 3.1.2 milestone May 25, 2018

jgebal added the enhancement label May 25, 2018

jgebal merged commit 3acfb6c into develop May 25, 2018

lwasylow deleted the feature/cursor_pk_join branch May 29, 2018 21:50

Feature/cursor pk join #666

Feature/cursor pk join #666

Uh oh!

Conversation

lwasylow commented May 11, 2018

Uh oh!

jgebal commented May 12, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgebal left a comment

Choose a reason for hiding this comment

Uh oh!

lwasylow commented May 24, 2018

Uh oh!

Uh oh!