Yep, thanks for the review!

Thanks for the PR!

Perhaps a simpler and more efficient solution here could be to share the ColumnDataCollection between all BufferedIndexData insert / delete nodes. We really only store two different collections:

• Insert data, holding new data to be inserted
• Delete data, holding row ids to be deleted

We could have two separate ColumnDataCollection nodes for these, and have each BufferedIndexData refer to a range within the ColumnDataCollection. These ranges will then always be consecutive. For example, if we have the following operations:

INSERT INTO tbl VALUES (2);
DELETE FROM tbl WHERE rowid=1;
COMMIT;

INSERT INTO tbl VALUES (3);
DELETE FROM tbl WHERE rowid=2;
COMMIT;

INSERT INTO tbl VALUES (4);
DELETE FROM tbl WHERE rowid=3;
COMMIT;

We would have the following collections:

InsertCollection

i: [2, 3, 4]

DeleteCollection

rowids: [1, 2, 3]

With the following nodes:

BufferedIndexData
    type: INSERT
    start: 0
    end: 1

BufferedIndexData
    type: DELETE
    start: 0
    end: 1
    
BufferedIndexData
    type: INSERT
    start: 1
    end: 2

BufferedIndexData
    type: DELETE
    start: 1
    end: 2

BufferedIndexData
    type: INSERT
    start: 2
    end: 3

BufferedIndexData
    type: DELETE
    start: 2
    end: 3

This has a number of advantages:

• We only need to scan two ColumnDataCollections, and we do so in-order, so this will all be memory-adjacent and efficient
• Constructing these will also be efficient, as we're not constantly allocating tiny batches
• I think we will likely end up using less memory in most cases, as the (wasted) empty space is capped to the empty space in the two ColumnDataCollections - versus having potentially much more (wasted) empty space spread across different chunks and collections
• From a code perspective I think this might also be simpler and easier to test fully - given we don't have as many special cases as the proposed solution here adds

Thanks @Mytherin that's a great idea, going to rewrite it!

@taniabogatsch Thank you for the review! I will run the CI now

Looks great, thanks for the changes!