ADBDEV-5487: gplogfilter: fix timerange validation #1134

silent-observer · 2024-11-27T15:07:45Z

gplogfilter: fix timerange validation

Previously the time range filtering of log files was incorrect: there was a check for belonging a log file to specified time range which worked like a binary AND (if time < begin AND time > end - skip file), but something like binary OR is needed here. Thus file filtering never worked. Also this validation was executed after the file was opened, so in case of failed validation there was a redundant open/close operations. This patch fixes such behavior, by:

moving the logic of belonging log file name the user specified range to the separate function (it's also helps to cover corner cases)
skip file processing, if it doesn't belong to range, at the begin of loop to prevent redundant open/close operations

Changes from original commit:

Fix print() syntax for Python 3
Add StringIO.seek(0) after StringIO.truncate(0) for Python 3
Make sure gplogfilter doesn't skip the first line.
Update behave test for GPDB 7 (use COORDINATOR_DATA_DIRECTORY and /log)
Add file content checking to behave test (counting lines that were read)

(cherry picked from commit e971286)

Note: do not squash to preserve authorship

silent-observer · 2024-11-29T10:36:05Z

That's the default way cherry-picked commits work (preserving the PR link from the old PR). Should it always be changed to the new PR?

hilltracer · 2024-12-03T14:00:19Z

Look's like --find (-f) is not working in GPDB 7

GPDB 6 behaviour

gpadmin@gpdb6u:~/src/arenadata.sh$ ls -l /home/gpadmin/.data/qddir/demoDataDir-1/pg_log/
total 68
-rw------- 1 gpadmin gpadmin 14836 Dec  3 13:37 gpdb-2024-12-03_163647.csv
-rw------- 1 gpadmin gpadmin 11744 Dec  3 13:37 gpdb-2024-12-03_163710.csv
-rw------- 1 gpadmin gpadmin 34809 Dec  3 13:37 gpdb-2024-12-03_163713.csv
-rw------- 1 gpadmin gpadmin   714 Dec  3 13:37 startup.log
gpadmin@gpdb6u:~/src/arenadata.sh$ head -n 3 /home/gpadmin/.data/qddir/demoDataDir-1/pg_log/gpdb-2024-12-03_163647.csv
2024-12-03 16:36:47.681750 MSK,,,p18626,th704555456,,,,0,,,seg-1,,,,,"LOG","00000","database system was shut down at 2024-12-03 16:36:46 MSK",,,,,,,0,,"xlog.c",6602,
2024-12-03 16:36:47.681838 MSK,,,p18626,th704555456,,,,0,,,seg-1,,,,,"LOG","00000","end of transaction log location is 0/5DF4550",,,,,,,0,,"xlog.c",7715,
2024-12-03 16:36:47.681849 MSK,,,p18626,th704555456,,,,0,,,seg-1,,,,,"LOG","00000","latest completed transaction id is 713 and next transaction id is 714",,,,,,,0,,"xlog.c",8089,
gpadmin@gpdb6u:~/src/arenadata.sh$ gplogfilter -b "2024-12-03 16:36:46" -e "2024-12-03 16:36:48" -f 'database system was shut down'
requested timestamp range from 2024-12-03 16:36:46 to 2024-12-03 16:36:48
----------  /home/gpadmin/.data/qddir/demoDataDir-1/pg_log/gpdb-2024-12-03_163647.csv ---------- 
2024-12-03 16:36:47.681750 MSK|||p18626|th704555456||||0|||seg-1|||||LOG: |00000|database system was shut down at 2024-12-03 16:36:46 MSK|||||||0||xlog.c|6602|
       in:      56 lines,      56 log entries; timestamps from 2024-12-03 16:36:47.681750 to 2024-12-03 16:37:03.185090
  time ok:       9 lines,       9 log entries; timestamps from 2024-12-03 16:36:47.681750 to 2024-12-03 16:36:47.686136
    match:       1 lines,       1 log entries; timestamps from 2024-12-03 16:36:47.681750 to 2024-12-03 16:36:47.681750
      out:       1 lines,       1 log entries; timestamps from 2024-12-03 16:36:47.681750 to 2024-12-03 16:36:47.681750
SKIP file: /home/gpadmin/.data/qddir/demoDataDir-1/pg_log/gpdb-2024-12-03_163710.csv
SKIP file: /home/gpadmin/.data/qddir/demoDataDir-1/pg_log/gpdb-2024-12-03_163713.csv
----------  /home/gpadmin/.data/qddir/demoDataDir-1/pg_log/startup.log ---------- 
       in:       3 lines,       3 log entries; timestamps from 2024-12-03 16:36:47.680281 to 2024-12-03 16:37:13.336233
  time ok:       1 lines,       1 log entries; timestamps from 2024-12-03 16:36:47.680281 to 2024-12-03 16:36:47.680281
    match:       0 lines
      out:       0 lines,       0 log entries

GPDB 7 behaviour

gpadmin@gpdb7u:~/src/arenadata.sh$ ls -l /home/gpadmin/.data/qddir/demoDataDir-1/log/
total 40
-rw------- 1 gpadmin gpadmin  7158 Dec  3 13:40 gpdb-2024-12-03_164049.csv
-rw------- 1 gpadmin gpadmin  4823 Dec  3 13:41 gpdb-2024-12-03_164059.csv
-rw------- 1 gpadmin gpadmin 19975 Dec  3 13:41 gpdb-2024-12-03_164102.csv
-rw------- 1 gpadmin gpadmin  3235 Dec  3 13:41 startup.log
gpadmin@gpdb7u:~/src/arenadata.sh$ head -n 3 /home/gpadmin/.data/qddir/demoDataDir-1/log/gpdb-2024-12-03_164049.csv
2024-12-03 16:40:49.925554 MSK,,,p234151,th-1828210240,,,,0,,,seg-1,,,,,"LOG","00000","database system was shut down at 2024-12-03 16:40:49 MSK",,,,,,,0,,"xlog.c",6626,
2024-12-03 16:40:49.925582 MSK,,,p234151,th-1828210240,,,,0,,,seg-1,,,,,"LOG","00000","end of transaction log location is 0/471D7A0",,,,,,,0,,"xlog.c",7861,
2024-12-03 16:40:49.925591 MSK,,,p234151,th-1828210240,,,,0,,,seg-1,,,,,"LOG","00000","latest completed transaction id is 524 and next transaction id is 525",,,,,,,0,,"xlog.c",8250,
gpadmin@gpdb7u:~/src/arenadata.sh$ gplogfilter -b "2024-12-03 16:40:48" -e "2024-12-03 16:40:50" -f 'database system was shut down'
requested timestamp range from 2024-12-03 16:40:48 to 2024-12-03 16:40:50
---------- /home/gpadmin/.data/qddir/demoDataDir-1/log/gpdb-2024-12-03_164049.csv ----------
       in:      30 lines,       1 log entries; timestamps from 2024-12-03 16:40:49.925554 to 2024-12-03 16:40:49.925554
  time ok:       0 lines
    match:       0 lines
      out:       0 lines,       0 log entries
SKIP file: /home/gpadmin/.data/qddir/demoDataDir-1/log/gpdb-2024-12-03_164059.csv
SKIP file: /home/gpadmin/.data/qddir/demoDataDir-1/log/gpdb-2024-12-03_164102.csv
---------- /home/gpadmin/.data/qddir/demoDataDir-1/log/startup.log ----------
       in:      15 lines,      15 log entries; timestamps from 2024-12-03 16:40:49.912866 to 2024-12-03 16:41:02.150204
  time ok:       4 lines,       4 log entries; timestamps from 2024-12-03 16:40:49.913130 to 2024-12-03 16:40:49.923857
    match:       0 lines
      out:       0 lines,       0 log entries

Is it expected behavior and what we will do?

silent-observer · 2024-12-04T07:47:12Z

This is actually -b/-e failing to filter lines because CsvFlatten outputs nonsense on Python 3. I suggest fixing it here, even though this probably affected other things too.

silent-observer · 2024-12-05T07:36:19Z

Upgraded the behave tests to check the actual log lines as well (this should detect the previous issue)

Previously the time range filtering of log files was incorrect: there was a check for belonging a log file to specified time range which worked like a binary `AND` (if time < begin AND time > end - skip file), but something like binary `OR` is needed here. Thus file filtering never worked. Also this validation was executed after the file was opened, so in case of failed validation there was a redundant `open/close` operations. This patch fixes such behavior, by: - moving the logic of belonging log file name the user specified range to the separate function (it's also helps to cover corner cases) - skip file processing, if it doesn't belong to range, at the begin of loop to prevent redundant `open/close` operations Changes from original commit: 1. Fix print() syntax for Python 3 2. Add StopIteration handler (required by Python 3.7) 3. Add StringIO.seek(0) after StringIO.truncate(0) for Python 3 4. Make sure gplogfilter doesn't skip the first line (with a chain iterator) 5. Update behave test for GPDB 7 (use COORDINATOR_DATA_DIRECTORY and /log) 6. Add file content checking to behave test (counting lines that were read) (cherry picked from commit e971286)

gpMgmt/bin/gppylib/logfilter.py

Previously the time range filtering of log files was incorrect: there was a check for belonging a log file to specified time range which worked like a binary `AND` (if time < begin AND time > end - skip file), but something like binary `OR` is needed here. Thus file filtering never worked. Also this validation was executed after the file was opened, so in case of failed validation there was a redundant `open/close` operations. This patch fixes such behavior, by: - moving the logic of belonging log file name the user specified range to the separate function (it's also helps to cover corner cases) - skip file processing, if it doesn't belong to range, at the begin of loop to prevent redundant `open/close` operations Changes from original commit: 1. Fix print() syntax for Python 3 2. Add StopIteration handler (required by Python 3.7) 3. Add StringIO.seek(0) after StringIO.truncate(0) for Python 3 4. Make sure gplogfilter doesn't skip the first line (with a chain iterator) 5. Update behave test for GPDB 7 (use COORDINATOR_DATA_DIRECTORY and /log) 6. Add file content checking to behave test (counting lines that were read) (cherry picked from commit e971286)

silent-observer · 2024-12-06T13:17:56Z

Yes, probably. This probably wasn't noticed because the first line is normally "database system was shut down", and nobody searches for that. I think we shouldn't create a separate ticket for GPDB 7 though, only one to backport the new fixes to GPDB 6.

gpMgmt/bin/gppylib/logfilter.py

Previously the time range filtering of log files was incorrect: there was a check for belonging a log file to specified time range which worked like a binary `AND` (if time < begin AND time > end - skip file), but something like binary `OR` is needed here. Thus file filtering never worked. Also this validation was executed after the file was opened, so in case of failed validation there was a redundant `open/close` operations. This patch fixes such behavior, by: - moving the logic of belonging log file name the user specified range to the separate function (it's also helps to cover corner cases) - skip file processing, if it doesn't belong to range, at the begin of loop to prevent redundant `open/close` operations Changes from original commit: 1. Fix print() syntax for Python 3 2. Add StringIO.seek(0) after StringIO.truncate(0) for Python 3 3. Make sure gplogfilter doesn't skip the first line. 4. Update behave test for GPDB 7 (use COORDINATOR_DATA_DIRECTORY and /log) 5. Add file content checking to behave test (counting lines that were read) (cherry picked from commit e971286)

silent-observer · 2024-12-06T13:43:42Z

Correction: GPDB 6 isn't affected because it uses a while True loop. It was broken in GPDB 7 by commit f5c285a. I think no additional ticket is needed after all

gpMgmt/bin/gppylib/logfilter.py

Previously the time range filtering of log files was incorrect: there was a check for belonging a log file to specified time range which worked like a binary `AND` (if time < begin AND time > end - skip file), but something like binary `OR` is needed here. Thus file filtering never worked. Also this validation was executed after the file was opened, so in case of failed validation there was a redundant `open/close` operations. This patch fixes such behavior, by: - moving the logic of belonging log file name the user specified range to the separate function (it's also helps to cover corner cases) - skip file processing, if it doesn't belong to range, at the begin of loop to prevent redundant `open/close` operations Changes from original commit: 1. Fix print() syntax for Python 3 2. Add StringIO.seek(0) after StringIO.truncate(0) for Python 3 3. Make sure gplogfilter doesn't skip the first line. 4. Update behave test for GPDB 7 (use COORDINATOR_DATA_DIRECTORY and /log) 5. Add file content checking to behave test (counting lines that were read) (cherry picked from commit e971286)

silent-observer · 2024-12-09T06:38:55Z

No, already known (ADBDEV-6013)

hilltracer

Ready to approve, please just fix typos and add comments.

gpMgmt/bin/gppylib/test/unit/test_unit_logfilter.py

gpMgmt/test/behave/mgmt_utils/gplogfilter.feature

gpMgmt/bin/gplogfilter

Previously the time range filtering of log files was incorrect: there was a check for belonging a log file to specified time range which worked like a binary `AND` (if time < begin AND time > end - skip file), but something like binary `OR` is needed here. Thus file filtering never worked. Also this validation was executed after the file was opened, so in case of failed validation there was a redundant `open/close` operations. This patch fixes such behavior, by: - moving the logic of belonging log file name the user specified range to the separate function (it's also helps to cover corner cases) - skip file processing, if it doesn't belong to range, at the begin of loop to prevent redundant `open/close` operations Changes from original commit: 1. Fix print() syntax for Python 3 2. Add StringIO.seek(0) after StringIO.truncate(0) for Python 3 3. Make sure gplogfilter doesn't skip the first line. 4. Update behave test for GPDB 7 (use COORDINATOR_DATA_DIRECTORY and /log) 5. Add file content checking to behave test (counting lines that were read) (cherry picked from commit e971286)

silent-observer force-pushed the ADBDEV-5487 branch from e3c6ae7 to 6afd8a8 Compare November 28, 2024 08:10

silent-observer marked this pull request as ready for review November 28, 2024 11:38

This comment was marked as resolved.

Sign in to view

RekGRpth previously approved these changes Nov 29, 2024

View reviewed changes

This comment was marked as resolved.

Sign in to view

silent-observer force-pushed the ADBDEV-5487 branch from 6afd8a8 to 87b3a20 Compare November 29, 2024 10:52

silent-observer changed the title ~~ADBDEV-4349: gplogfilter: fix timerange validation (#627)~~ ADBDEV-4349: gplogfilter: fix timerange validation Nov 29, 2024

hilltracer self-requested a review December 2, 2024 10:42

silent-observer dismissed RekGRpth’s stale review via f8311ea December 4, 2024 08:08

silent-observer force-pushed the ADBDEV-5487 branch from 32753f4 to f8311ea Compare December 4, 2024 08:08

silent-observer changed the title ~~ADBDEV-4349: gplogfilter: fix timerange validation~~ ADBDEV-5487: gplogfilter: fix timerange validation Dec 5, 2024

silent-observer force-pushed the ADBDEV-5487 branch from f8311ea to 795dc93 Compare December 5, 2024 07:35

This comment was marked as resolved.

Sign in to view

silent-observer force-pushed the ADBDEV-5487 branch from 795dc93 to 5e668a3 Compare December 5, 2024 07:47

This comment was marked as resolved.

Sign in to view

silent-observer force-pushed the ADBDEV-5487 branch from 5e668a3 to c7dac99 Compare December 6, 2024 05:57

RekGRpth reviewed Dec 6, 2024

View reviewed changes

gpMgmt/bin/gppylib/logfilter.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

silent-observer force-pushed the ADBDEV-5487 branch from c7dac99 to c8a154d Compare December 6, 2024 13:15

RekGRpth reviewed Dec 6, 2024

View reviewed changes

gpMgmt/bin/gppylib/logfilter.py Outdated Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

silent-observer force-pushed the ADBDEV-5487 branch from c8a154d to 3c819cb Compare December 6, 2024 13:33

RekGRpth reviewed Dec 6, 2024

View reviewed changes

gpMgmt/bin/gppylib/logfilter.py Outdated Show resolved Hide resolved

silent-observer force-pushed the ADBDEV-5487 branch from 3c819cb to 05fb8c2 Compare December 6, 2024 14:17

This comment was marked as resolved.

Sign in to view

RekGRpth previously approved these changes Dec 9, 2024

View reviewed changes

hilltracer reviewed Dec 11, 2024

View reviewed changes

silent-observer dismissed RekGRpth’s stale review via be17384 December 11, 2024 09:22

hilltracer approved these changes Dec 11, 2024

View reviewed changes

This comment was marked as resolved.

Sign in to view

silent-observer force-pushed the ADBDEV-5487 branch from 29220f3 to 2db4324 Compare December 12, 2024 06:47

RekGRpth approved these changes Dec 12, 2024

View reviewed changes

silent-observer enabled auto-merge (rebase) December 12, 2024 07:04

silent-observer merged commit 7e49463 into adb-7.2.0 Dec 12, 2024
1 check passed

silent-observer deleted the ADBDEV-5487 branch December 12, 2024 09:11

ADBDEV-5487: gplogfilter: fix timerange validation #1134

ADBDEV-5487: gplogfilter: fix timerange validation #1134

Uh oh!

Conversation

silent-observer commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

silent-observer commented Nov 29, 2024

Uh oh!

This comment was marked as resolved.

hilltracer commented Dec 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

silent-observer commented Dec 4, 2024

Uh oh!

silent-observer commented Dec 5, 2024

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

silent-observer commented Dec 6, 2024

Uh oh!

Uh oh!

This comment was marked as resolved.

silent-observer commented Dec 6, 2024

Uh oh!

Uh oh!

This comment was marked as resolved.

silent-observer commented Dec 9, 2024

Uh oh!

hilltracer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

silent-observer commented Nov 27, 2024 •

edited

Loading

hilltracer commented Dec 3, 2024 •

edited

Loading