-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Labels
backlog/starterGood issue for newcomersGood issue for newcomers
Description
For streaming sources that cannot guarantee data consistency until all files are applied - which is the case for Synapse entities with lots of deletes, current behaviour of backfill-merge will cause target table to hold lots of deleted records, until delete batches are applied. Even though this doesn't take much time, it can be enough for a client to read the target table and get confused about the result.
Thus, proposal to change from AS-IS:
stage -> merge into target
to TO-BE:
target_clone <- clone(target) // create or replace ... as select. * from target
stage -> merge into target_clone
on_finish(replace(target, target_clone)) // create or replace target as select * from target_clone
Metadata
Metadata
Assignees
Labels
backlog/starterGood issue for newcomersGood issue for newcomers
Type
Projects
Status
No status