Convert token transfer models to microbatch incremental strategy#266
Convert token transfer models to microbatch incremental strategy#266amishas157 wants to merge 1 commit into
Conversation
- int_token_transfer_enrichment and token_transfers move from merge/unique_key to the microbatch strategy (event_time=closed_at, begin=2015-09-30), which compiles to a dynamic insert_overwrite on BigQuery. - stg_token_transfers_raw declares event_time so microbatch consumers get the batch-window filter pushed down automatically. - The stg_history_operations join keeps an explicit batch_run_date filter padded by one day on each side via model.batch, since ingestion date can differ slightly from closed_at (join-side read only). - token_transfers keeps month partitioning, so its batch_size floors at month; sub-month batches would overwrite whole month partitions with partial data. - New microbatch_size var (MICROBATCH_SIZE env var) allows coarser batches for backfills, with a root override of bq_validate_microbatch_config that permits batch_size coarser than the partition granularity (dynamic insert_overwrite replaces exactly the partitions present in each batch's output, so this is safe; finer than partition granularity remains an error).
There was a problem hiding this comment.
Pull request overview
This pull request converts the token transfer pipeline from merge-based incremental models to dbt’s microbatch incremental strategy on BigQuery, aiming to speed up daily runs and make backfills significantly cheaper by using dynamic insert_overwrite partition replacement instead of MERGE.
Changes:
- Switched
int_token_transfer_enrichmentandtoken_transfersfrommergetomicrobatch, addingevent_time,batch_size, andbeginconfiguration. - Added
event_timetostg_token_transfers_rawto enable microbatch window filter pushdown for downstream consumers. - Introduced a root-project override macro to relax BigQuery microbatch config validation, allowing
batch_sizeto be coarser thanpartition_by.granularity, plus a newmicrobatch_sizeproject var.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| models/staging/stg_token_transfers_raw.sql | Adds event_time=closed_at so downstream microbatch models can get batch-window filtering pushed down. |
| models/marts/tokens/transfers/token_transfers.sql | Converts to microbatch incremental; enforces month-sized batches for a month-partitioned table and removes manual batch date filters. |
| models/intermediate/tokens/transfers/int_token_transfer_enrichment.sql | Converts to microbatch incremental; removes manual batch date filters and adds a padded join-side filter for stg_history_operations based on model.batch windows. |
| macros/bq_validate_microbatch_config.sql | Overrides dbt-bigquery’s microbatch validation to allow coarser batch_size than partition granularity (while still disallowing finer). |
| dbt_project.yml | Adds microbatch_size var sourced from MICROBATCH_SIZE env var (default day). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8afa7c570a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| -- microbatch auto-filters this ref to the batch window via event_time | ||
| from {{ ref('stg_token_transfers_raw') }} as tt |
There was a problem hiding this comment.
Preserve the previous closed_at lookback window
When Airflow switches to the --event-time-start/--event-time-end path described for this conversion, this ref is filtered to only that exact microbatch window, but the old model always re-read closed_at >= batch_start_date - 1 day (and up to batch_end_date + 1 day). That padding is what let a daily run pick up token transfers that arrived late for the prior day; with the new exact closed_at auto-filter, those rows remain missing unless someone manually backfills the old batch.
Useful? React with 👍 / 👎.
Why
Full refreshes and backfills of the token transfer models are slow under the merge strategy: every batch pays a MERGE on
unique_keyagainst the full destination scan window, and backfill windows are replayed day by day viaBATCH_START_DATE/BATCH_END_DATE. Converting to dbt's microbatch strategy turns each batch into a dynamicinsert_overwrite(partition replacement, no MERGE) and lets backfills run with coarse batches (MICROBATCH_SIZE=month/year), amortizing per-job overhead.This mirrors the same conversion being done for the entity attribution
operationsmart in stellar-dbt.What
int_token_transfer_enrichment→ microbatch (event_time=closed_at,begin=2015-09-30,batch_sizefrom the newmicrobatch_sizevar, defaultday). Day partitioning unchanged.stg_history_operationsjoin keeps an explicitbatch_run_datefilter padded ±1 day viamodel.batch.event_time_start/end, since ingestion date can differ slightly fromclosed_at. This is a join-side read only — output rows are still bounded by the auto-filter onstg_token_transfers_raw.closed_at, so the padding cannot leak rows into neighbor partitions.token_transfers→ microbatch. Month partitioning kept, so itsbatch_sizefloors atmonth: BigQuery microbatch is a dynamic insert_overwrite that replaces every partition present in a batch's output, and a sub-month batch would overwrite a whole month partition with partial data. dbt truncates/ceilings--event-timewindows to batch-size boundaries, so any window passed by Airflow expands to full months and is safe. Daily runs now rewrite the current month partition (~month-to-date) instead of merging a day.stg_token_transfers_raw: declaresevent_time=closed_atso microbatch consumers get the batch-window filter pushed down automatically.dbt_project.yml: newmicrobatch_sizevar (MICROBATCH_SIZEenv var, defaultday).macros/bq_validate_microbatch_config.sql: root override of the dbt-bigquery built-in validation, allowingbatch_sizecoarser than the partition granularity (safe with dynamic insert_overwrite; finer than partition granularity remains an error). See [Feature] microbatch_size should not be aligned to partition_by dbt-labs/dbt-core#11278.Deployment notes
--event-time-start/--event-time-endfor the token_transfer group instead ofBATCH_START_DATE/BATCH_END_DATEenv vars (without flags, dbt processes the current batch +lookbackrelative to run time).MICROBATCH_SIZE=year dbt run --select int_token_transfer_enrichment token_transfers --full-refresh.Validation
dbt compileof all three models against the test projectmodel.batch.event_time_*render as BQ-parseable timestamp literals (dry-run validated)pre-commit run(sqlfluff, prettier, dbt-checkpoint) passes on all changed files