Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Comments

Store first seen time in block audit#5567

Merged
softsimon merged 5 commits intomasterfrom
natsoni/block-first-seen-audit
Oct 14, 2024
Merged

Store first seen time in block audit#5567
softsimon merged 5 commits intomasterfrom
natsoni/block-first-seen-audit

Conversation

@natsoni
Copy link
Contributor

@natsoni natsoni commented Oct 5, 2024 •
edited
Loading

Partially implements #5543

This PR introduces a new first_seen field to the block audit, which records the timestamp of when a block was first observed. The timestamp is extracted from the Saw new header ... entries in Core's debug.log, with support for both second and microsecond precision.

To enable this functionality:

  • add "DEBUG_LOG_PATH": "/path/to/bitcoin/debug.log" to the backend config
  • add -logtimemicros=1 to bitcoin.conf for microsecond precision (optional)

I will make a follow-up PR to use the first seen data in the frontend later on when this has been running for a few days.

@natsoni natsoni requested a review from mononaut October 5, 2024 09:09
@natsoni natsoni requested review from softsimon and wiz as code owners October 5, 2024 09:09
@cla-bot cla-bot bot added the cla-signed label Oct 5, 2024
@natsoni natsoni force-pushed the natsoni/block-first-seen-audit branch from 5cdf3c7 to 21b3ed2 Compare October 5, 2024 09:19
@natsoni natsoni force-pushed the natsoni/block-first-seen-audit branch from 21b3ed2 to 1a75e3e Compare October 5, 2024 10:26
mononaut
mononaut requested changes Oct 5, 2024
View reviewed changes
Copy link
Contributor

@mononaut mononaut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested @ [1a75e3e]

A couple of minor issues in the comments, but otherwise looks good!

(1) I ended up with some very late firstSeen times on a bunch of recent blocks while Core was catching up after a restart.

I guess that's technically correct since those were the times my node first saw the block headers, but it might be worth applying some sanity checks to the timestamps we choose to index.

(2) It would be neat to backfill the firstSeen timestamps for older audits as far as we have data available in the debug.log, perhaps in a new indexing task. We can save that for another PR though.

@mononaut
Copy link
Contributor

mononaut commented Oct 5, 2024

also, are we sure we want to store the timestamp in the blocks_audits table rather than the regular blocks table?

@softsimon
Copy link
Member

softsimon commented Oct 5, 2024 •
edited
Loading

also, are we sure we want to store the timestamp in the blocks_audits table rather than the regular blocks table?

Isn't the long term plan to integrate the block table into electrs itself, which makes it a bad option to store dynamic data that can't be re-indexed from the blockchain, although we already store pool-Ids. Or what's your take on this.

@mononaut
Copy link
Contributor

mononaut commented Oct 5, 2024 •
edited
Loading

also, are we sure we want to store the timestamp in the blocks_audits table rather than the regular blocks table?

Isn't the long term plan to integrate the block table into electrs itself, which makes it a bad option to store dynamic data that can't be re-indexed from the blockchain, although we already store pool-Ids. Or what's your take on this.

yeah that's a good point. I was just thinking that this feature doesn't necessarily overlap with the rest of the block audit table.

e.g. I could imagine wanting to index block first seen times even with audits disabled, or to index that data for blocks we didn't/couldn't audit, like those added during a reorg.

@natsoni
Copy link
Contributor Author

natsoni commented Oct 6, 2024

also, are we sure we want to store the timestamp in the blocks_audits table rather than the regular blocks table?

Right, the rationale for storing the timestamp inblocks_audit is that it's kind of an audit feature, but as you said one might want to store first seen data without having audits enabled. Also storing first seen times for reorg'd blocks is a nice use case, so I do think it makes sense to move the first seen field to the blocks table.

@natsoni natsoni force-pushed the natsoni/block-first-seen-audit branch from 83e05a6 to 2eb3f47 Compare October 8, 2024 08:46
@natsoni natsoni marked this pull request as draft October 8, 2024 09:49
@natsoni
Copy link
Contributor Author

natsoni commented Oct 8, 2024

(1) I ended up with some very late firstSeen times on a bunch of recent blocks while Core was catching up after a restart.
I guess that's technically correct since those were the times my node first saw the block headers, but it might be worth applying some sanity checks to the timestamps we choose to index.

Good point, but how strict should we be with the sanitizing? Even if we require the first seen time to be at most 1-2 hours after the block timestamp, we could end up with some false positives. But with such a loose interval I’m wondering if the sanitizing would be useful at all.

@mononaut
Copy link
Contributor

mononaut commented Oct 8, 2024

fair enough. in that case let's index the data as it appears in the logs, and we can always clean it up on the client side later.

@natsoni natsoni force-pushed the natsoni/block-first-seen-audit branch 3 times, most recently from cbbce03 to 229a756 Compare October 9, 2024 07:37
@natsoni
Copy link
Contributor Author

natsoni commented Oct 9, 2024

Added first_seen to the blocks table as datetime(6)

@natsoni natsoni requested a review from mononaut October 9, 2024 08:56
mononaut
mononaut reviewed Oct 9, 2024
View reviewed changes
Copy link
Contributor

@mononaut mononaut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good! just a few small issues left

@natsoni natsoni force-pushed the natsoni/block-first-seen-audit branch from 229a756 to dd0542b Compare October 10, 2024 09:47
@natsoni natsoni marked this pull request as ready for review October 13, 2024 03:58
mononaut
mononaut approved these changes Oct 13, 2024
View reviewed changes
Copy link
Contributor

@mononaut mononaut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested ACK @ [198d79f]

softsimon
softsimon approved these changes Oct 14, 2024
View reviewed changes
Copy link
Member

@softsimon softsimon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested ACK @ [198d79f]

@softsimon softsimon merged commit 26e3a24 into master Oct 14, 2024
@softsimon softsimon deleted the natsoni/block-first-seen-audit branch October 14, 2024 01:10
@natsoni natsoni mentioned this pull request Jan 28, 2025
Merged
@mononaut mononaut mentioned this pull request Oct 14, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@softsimon softsimon softsimon approved these changes

@mononaut mononaut mononaut approved these changes

@wiz wiz Awaiting requested review from wiz wiz is a code owner

Assignees

No one assigned

Labels

cla-signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@natsoni @mononaut @softsimon