feat: Add migration backfill for mptoken_issuance_history#3128
Open
BryanJ1ang wants to merge 12 commits into
Open
feat: Add migration backfill for mptoken_issuance_history#3128BryanJ1ang wants to merge 12 commits into
BryanJ1ang wants to merge 12 commits into
Conversation
f6677be to
a0b2942
Compare
a0b2942 to
891846e
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #3128 +/- ##
===========================================
- Coverage 82.65% 82.50% -0.15%
===========================================
Files 398 399 +1
Lines 16118 16169 +51
Branches 8443 8476 +33
===========================================
+ Hits 13323 13341 +18
- Misses 1674 1707 +33
Partials 1121 1121 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Statement.hpp and Collection.hpp each carried a private static copy of the same CassError-to-logic_error helper; both now use one shared free function in data::cassandra::impl. Co-Authored-By: Claude Fable 5 <[email protected]>
…nst ref A transaction that fails to deserialize is a deterministic data error, so rerunning the migration could never get past it; log and skip the row instead of permanently wedging the backfill. Callback and DB read errors still fail closed. The scan callback now takes STTx/TxMeta by const ref, removing a deep copy per scanned row. Co-Authored-By: Claude Fable 5 <[email protected]>
Buffer extracted records across transactions and flush in batches (mirroring the live ETL path's per-ledger accumulation) so the backend coalesces them into full-size write batches instead of submitting one tiny write pair per transaction. Co-Authored-By: Claude Fable 5 <[email protected]>
The warning was not operator-actionable and per-tx fanout beyond the threshold should not occur in practice; the Prometheus counter still tracks index rows written. Co-Authored-By: Claude Fable 5 <[email protected]>
Key the prepared-statement cache by table name alone (partition key and select columns are fixed per TableDesc), replace the rowsRead counter with a bool, and author the read-failure message once for both the log and the thrown exception. Co-Authored-By: Claude Fable 5 <[email protected]>
The TableSpec concept now requires the kSelectColumns list to have exactly one column per Row tuple element, so the two cannot silently drift when columns are added or removed. Co-Authored-By: Claude Fable 5 <[email protected]>
Contributor
Author
|
/ai-reviewer |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: this PR depends on previous ETL indexing PR here
Summary
Adds
MPTTransactionHistoryMigrator, a Cassandra migration that backfills historical MPT issuance transaction history into the new MPT index tables:mptoken_issuance_transactionsaccount_mptoken_issuance_transactionsThe migrator full-scans the
transactionstable, reuses the live ETL MPT extractor, writes the same deterministic index rows produced by live indexing, and waits for queued async writes before marking the migration as complete.Details
MPTTransactionHistoryMigratorwith the Cassandra migration manager.migration.full_scan_threadsmigration.full_scan_jobsmigration.cursors_per_jobTesting
Added coverage for: