Thanks to visit codestin.com
Credit goes to github.com

Skip to content

algod importer: check if node needs catchup.#70

Merged
winder merged 11 commits intoalgorand:masterfrom
winder:will/better-skip-check
May 9, 2023
Merged

algod importer: check if node needs catchup.#70
winder merged 11 commits intoalgorand:masterfrom
winder:will/better-skip-check

Conversation

@winder
Copy link
Contributor

@winder winder commented May 4, 2023

Summary

The catchup logic was too aggressive. It should not run unless it's needed. With this change the target block / state delta are queried. If they are available the catchup phase is skipped.

Test Plan

Update unit tests.

@winder winder requested review from Eric-Warehime and tzaffi May 4, 2023 23:02
assert.NoError(t, err)
}

func TestInitSuccess(t *testing.T) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Entirely redundant with other tests.

logs: []string{},
},
}, {
name: "monitor catchup success",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test was accidentally deleted in an earlier PR.

Comment on lines -25 to -38
var (
logger *logrus.Logger
ctx context.Context
cancel context.CancelFunc
pRound sdk.Round
)

func init() {
logger = logrus.New()
logger.SetOutput(os.Stdout)
logger.SetLevel(logrus.InfoLevel)
ctx, cancel = context.WithCancel(context.Background())
pRound = sdk.Round(1)
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing these globals fixed the parallel test race conditions.

@codecov
Copy link

codecov bot commented May 4, 2023

Codecov Report

Merging #70 (cfcd661) into master (442791a) will increase coverage by 1.39%.
The diff coverage is 74.38%.

@@            Coverage Diff             @@
##           master      #70      +/-   ##
==========================================
+ Coverage   67.66%   69.05%   +1.39%     
==========================================
  Files          32       36       +4     
  Lines        1976     2414     +438     
==========================================
+ Hits         1337     1667     +330     
- Misses        570      654      +84     
- Partials       69       93      +24     
Impacted Files Coverage Δ
conduit/data/block_export_data.go 100.00% <ø> (+92.30%) ⬆️
conduit/plugins/importers/algod/metrics.go 100.00% <ø> (ø)
pkg/cli/internal/list/list.go 20.75% <ø> (ø)
...lugins/exporters/postgresql/postgresql_exporter.go 66.66% <51.21%> (-11.54%) ⬇️
pkg/cli/cli.go 65.97% <65.97%> (ø)
conduit/pipeline/pipeline.go 66.09% <72.10%> (+0.64%) ⬆️
conduit/data/config.go 76.47% <76.47%> (ø)
conduit/plugins/importers/algod/algod_importer.go 87.26% <87.82%> (-1.06%) ⬇️
conduit/pipeline/errors.go 100.00% <100.00%> (ø)
conduit/plugins/exporters/exporter_factory.go 100.00% <100.00%> (ø)
... and 3 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Comment on lines -35 to +37
if responder(req.URL.Path, w) {
if responder(req, w) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched interface to pass the entire request so that they can check the method. Ended up not using GetSyncRound, so it isn't actually used.

@winder winder marked this pull request as ready for review May 5, 2023 13:04
@winder winder changed the title algod importer: check if node needs catchup before running one. algod importer: check if node needs catchup. May 5, 2023
@winder winder force-pushed the will/better-skip-check branch from 4705f12 to b2e3e64 Compare May 5, 2023 15:31
@winder winder force-pushed the will/better-skip-check branch from b2e3e64 to 3986626 Compare May 8, 2023 13:23
Copy link
Contributor

@tzaffi tzaffi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking through code, I'm pointing out the one nit that I had. Am now playing around with the branch to validate the importing will work against an archival-caught-up algod.

Copy link
Contributor

@tzaffi tzaffi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified locally that this works!

Suggesting an additional log.

@winder winder requested a review from tzaffi May 8, 2023 22:32
@winder winder force-pushed the will/better-skip-check branch from 1ccbd67 to 1da853e Compare May 8, 2023 22:38
Copy link
Contributor

@tzaffi tzaffi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@winder winder merged commit 2a0537c into algorand:master May 9, 2023
@winder winder deleted the will/better-skip-check branch May 9, 2023 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

Comments