Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Backfill MERGE should merge into target clone #141

@george-zubrienko

Description

@george-zubrienko

For streaming sources that cannot guarantee data consistency until all files are applied - which is the case for Synapse entities with lots of deletes, current behaviour of backfill-merge will cause target table to hold lots of deleted records, until delete batches are applied. Even though this doesn't take much time, it can be enough for a client to read the target table and get confused about the result.

Thus, proposal to change from AS-IS:
stage -> merge into target
to TO-BE:

target_clone <- clone(target) // create or replace ... as select. * from target
stage -> merge into target_clone
on_finish(replace(target, target_clone)) // create or replace target as select * from target_clone

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions