This is a massive PR. Difficult to review.

Before doing anything of the sort, please make sure you have good benchmark results. Changing thousands of lines of code is not something that should be done lightly.

This is a massive PR. Difficult to review.

Thats why I structured the commits to enable to review in bits. The first 5 commits are easy to review small enablers, but removing a data structure in the middle of the implementation had knock on effects. THe 6th commit is bit though

Before doing anything of the sort, please make sure you have good benchmark results. Changing thousands of lines of code is not something that should be done lightly.

I did try to discuss this as a design issue, but all I go was "you need benchmarks". No design discussion

As I said in the summary I will add benchmarks shortly. Benchmarks don't run on code concept, they require an implementation. I offered to benchmark a slice but that was rejected

@mkeskells You need to sync with the main branch.

@blacelle @richardstartin and others: would you chime in ?

I did try to discuss this as a design issue, but all I go was "you need benchmarks". No design discussion

I have a general bias against far reaching pull requests. My bias is not unusual nor is it controversial. Please read again what I wrote: This is a massive PR. Difficult to review. Before doing anything of the sort, please make sure you have good benchmark results. Changing thousands of lines of code is not something that should be done lightly.

Let us go back to how you justify it: It speed up access, by removing a layer if indirection on access to most operation
and then This also removes the inherent unbounded memory leak.

These are two objective and important statements. I am asking you to document and justify these claims.

MArking as Draft as I will not have time on this PR - to sync up, tidy up or benchmark for at least this week

The majority of the new code, other than tests is for the change to the serialisation format (I think. Haven't counted then though)

That change is required by this PR but could be implemented against the current main branch. Its not seperate entirely, but mostly

Thus would make the serialized form smaller, and less tied to the internal memory structure than the current serialised form. The change is largely orthogonal, but dependent

It would also make the serialised form deterministic based on the content, which it isn't currently, as the memory leaks in Containers are reflected in the serialised form, and dehydrated form

The numbers are easier to review as well (serialised size, time, garbage). Retained size should be unaffected

Changing the serialization format certainly adds friction to this PR. Do you confirm this PR does break binary compatibility ? Or does it change the binary output because of the optimizations, while keeping compatibility?

#598 refers to some matters around Roaring64 serialization format. (It is mostly focused on Map-based, but it refers to ART ones too).

Changing the serialization format certainly adds friction to this PR. Do you confirm this PR does break binary compatibility ? Or does it change the binary output because of the optimizations, while keeping compatibility?
Sorry if I was not clear. The PR requires that the serialisation format is changed, as the current format contains the data structures that are removed

The new, or another format that just serialised the required data could be made. I haven't looked at the NavigableMap based format, but will have a look

The format that I proposed here (but am not too fussed about) retains the Trie structure, and therefore is implementation dependent, cos we could change the tree. It doesn't contain the internal fields of the implementing, so just the node with prefix, and count of children, and key/child array

#598 refers to some matters around Roaring64 serialization format. (It is mostly focused on Map-based, but it refers to ART ones too).

Will have a look ... when i can.
I had a look. It seems from a quick read that the proposal is that we write a sequence of 32bit high vae -> 32bit roaringBitmap. Thats quite a way away from the structure of the 64BitRoaringMap, bit not unachievable. It would churn memory a bit as I seet if, converting the lower 2 bytes of the Trie into a data structure, or at least some scanning to work out number of entries etc
Am I reading that correctly.
as Seq<32bit int, Seq<32bit roaringMap>>
which I think is Seq<32bit int, Seq<16 bit int, 16bit container>

I would imagine that having a portable format would be useful, but also a fast format has some attractions. Its not black and white. Maybe there is a need for both.
Certainly if we were to do this I would like to get rid of the duplication of strucure with DataOutput and ByteBuffer!

Is there a specific view if what the format should be?

copied the serialisation format to #805
I havent checked it, but it passes tests. Probably some dead code there still

WIll check it in more details later, but sharing for the conversation, but its very draft

Let us go back to how you justify it: It speed up access, by removing a layer if indirection on access to most operation and then This also removes the inherent unbounded memory leak.

These are two objective and important statements. I am asking you to document and justify these claims.

Sorry I thought this was obvious. I have mentioned this memory leak many times in our conversations, and I presume this was well know and accepted. Every time that a LafNode is removed (e.g. empty) the Containers null the reference bu doesnt recliam or reuse that slot
Trim doesnt combact this either (I was writing a beter trip when I realised that the whole HighLow structure in Roaring64 doesnt add anything (unlike the HighLow in the 32 bit implementation which is a wrapper around parellel arrays)
Serilisation doesnt free the slot either

In terms of the access, it can be seem that most operation that need to do any bitmap operation operation traverses the graph of obects Roaring64Bitmap -> HighLowContainer -> Art -> Node .... LeafNode -> [then to access the container] highLowContainer -> Containers -> array -> Container

This can be shortned to Roaring64Bitmap -> Art -> Node .... LeafNode -> Container
Probably could remove Art as well eventualy, but the bigest traversal difference in the containers/highlow

Every time that a LafNode is removed (e.g. empty) the Containers null the reference bu doesnt recliam or reuse that slot

I do not consider this a memory leak.

If we were to consider that it is a memory leak, then you'd have to consider the standard HashMap in Java to be plagued by memory leaks. The HashMap implementation does not include any mechanism to reduce the capacity of the table when keys are removed. The resize() method is designed to either initialize the table or increase its size (typically doubling it). There is no code path that creates a smaller table when the number of entries decreases.

This does not mean that the current design cannot be improved, of course.

In terms of the access, it can be seem that most operation that need to do any bitmap operation operation traverses the (...)

I get that your claim is that you can remove an indirection and thus improve the performance. However, if that is your claim, then you should be able to support it with benchmark results.

I do not consider this a memory leak.

If we were to consider that it is a memory leak, then you'd have to consider the standard HashMap in Java to be plagued by memory leaks. The HashMap implementation does not include any mechanism to reduce the capacity of the table when keys are removed. The resize() method is designed to either initialize the table or increase its size (typically doubling it). There is no code path that creates a smaller table when the number of entries decreases.

IMHO You have a very narrow definition of a memory leak, lets look a a "accepted" definition of a memory leak - https://en.wikipedia.org/wiki/Memory_leak

"In computer science, a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations in a way that memory which is no longer needed is not released"

That pretty much exactly match the situation described.
It inappropriate to say this isn't a memory leak because some other implementation has the same characteristic. By that definition there are no memory leak in software, because there are similar bugs elsewhere. - And yes HashMap, HashSet have memory leaks. No-one said they didn't until now

Its also not an equivalent comparison. Repeated addition and removal for HashMap doesn't grow memory use (it caps it to the maximum size). Repeated additional and removal for this structure leak more memory. Serialisation and deserialization clean the memory for a HashMap, it doesn't for this structure.

In terms of the access, it can be seem that most operation that need to do any bitmap operation operation traverses the (...)

I get that your claim is that you can remove an indirection and thus improve the performance. However, if that is your claim, then you should be able to support it with benchmark results.

As I said in my initial post, and subsequently, I will provide benchmarks (I agree this is more that a few days), when I have time, and access to a machine which is stable. Laptops are not suitable for CPU comparison due to thermal management and other issues, and this is not my day job, Its not a focus as this is not a library that I will be using as I have a smaller faster closed source direct mapped implementation that meets my much narrower needs

Its also not an equivalent comparison. Repeated addition and removal for HashMap doesn't grow memory use (it caps it to the maximum size). Repeated additional and removal for this structure leak more memory. Serialisation and deserialization clean the memory for a HashMap, it doesn't for this structure.

I did not analyzed the potential memory-leak. @mkeskells Could you provide a unit-test demonstrating the issue, so we can agree current behavior is acceptable or not (independantly of any design analysis). (Similarly to benchmarks on the performance aspect, we're just expecting factual elements before moving forwards on such evolutions). Such a unit-test would also prevent future regressions (standard software practise).

I did not analyzed the potential memory-leak. @mkeskells Could you provide a unit-test demonstrating the issue, so we can agree current behavior is acceptable or not

I can provide a example. Not a unit test (as there isnt a pass/fail) but some code that demmonstrates one of the causes

Anything reproducing the behavior would be useful. Thanks

Some performance results

Some of the results are a little out of step so some more digging may be needed (e.g. addEachMising(10k, 10k, true) but generally positive

For each case ther eis a summary, csv, and full text in the gists below
command line to run this was
java -Xms8g -Xmx8g -jar benchmarks.jar -rf csv -rff data.csv -o run.txt -prof gc -bm avgt -tu ns -w 5 -wf 0 -wi 2 -i 5 -f 1 -r 5 -jvmArgs -Xmx8g 'org.roaringbitmap.longlong.(AddRoaring64|BuildFromEmpty).*'

• before https://gist.github.com/mkeskells/1b232f8c9b54e183fea6ec6f61b0f3ea
• after https://gist.github.com/mkeskells/8123a74c9684de96c22a643bc7af105f

as a comparison of the effect of the PR

Anything reproducing the behavior would be useful. Thanks

Here is a simple example where a bitmap containing one bit consumes all heap

import org.roaringbitmap.art.Containers;
import org.roaringbitmap.longlong.HighLowContainer;
import org.roaringbitmap.longlong.Roaring64Bitmap;

import java.lang.reflect.Field;

public class HighLowMemory {
    public static void main(String[] args) throws Exception {
        new HighLowMemory().run();
    }
    private final Field highLow;
    private final Field containers;
    private final Field containers_containerSize;
    private final Field containers_firstLevelIndex;
    private final Field containers_secondLevelIndex;
    HighLowMemory() throws Exception {
        highLow = Roaring64Bitmap.class.getDeclaredField("highLowContainer");
        highLow.setAccessible(true);

        containers = HighLowContainer.class.getDeclaredField("containers");
        containers.setAccessible(true);

        containers_containerSize = Containers.class.getDeclaredField("containerSize");
        containers_containerSize.setAccessible(true);

        containers_firstLevelIndex = Containers.class.getDeclaredField("firstLevelIdx");
        containers_firstLevelIndex.setAccessible(true);

        containers_secondLevelIndex = Containers.class.getDeclaredField("secondLevelIdx");
        containers_secondLevelIndex.setAccessible(true);
    }
    public void run() throws Exception {
        int count = 10000;
        Roaring64Bitmap rb = new Roaring64Bitmap();
        for (long i = 0; i < 100; i++) {
            for (long j = 0; j < 10000000; j++) {
                rb.add(j << 16);
                rb.remove((j - 1) << 16);
            }
            System.out.println("After cycle " + i + " size: " + rb.getLongCardinality());
            System.out.println("slots used " + getSlotsUsed(rb));
        }
    }

    private String getSlotsUsed(Roaring64Bitmap rb) throws Exception {
        Containers container  = (Containers)containers.get(highLow.get(rb));
        return "containerSize=" + containers_containerSize.getLong(container) +
                ", firstLevelIdx=" + containers_firstLevelIndex.getInt(container) +
                ", secondLevelIdx=" + containers_secondLevelIndex.getInt(container);
    }
}

produces

After cycle 0 size: 1
slots used containerSize=1, firstLevelIdx=0, secondLevelIdx=9999999
After cycle 1 size: 1
slots used containerSize=1, firstLevelIdx=0, secondLevelIdx=19999998
After cycle 2 size: 1
slots used containerSize=1, firstLevelIdx=0, secondLevelIdx=29999997
After cycle 3 size: 1

...

slots used containerSize=1, firstLevelIdx=0, secondLevelIdx=439999956
After cycle 44 size: 1
slots used containerSize=1, firstLevelIdx=0, secondLevelIdx=449999955
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.lang.reflect.Array.newArray(Native Method)
	at java.base/java.lang.reflect.Array.newInstance(Array.java:78)
	at java.base/java.util.Arrays.copyOf(Arrays.java:3721)
	at java.base/java.util.Arrays.copyOf(Arrays.java:3689)
	at org.roaringbitmap.art.Containers.grow(Containers.java:184)
	at org.roaringbitmap.art.Containers.addContainer(Containers.java:107)
	at org.roaringbitmap.longlong.HighLowContainer.put(HighLowContainer.java:78)
	at org.roaringbitmap.longlong.Roaring64Bitmap.addLong(Roaring64Bitmap.java:67)
	at org.roaringbitmap.longlong.Roaring64Bitmap.add(Roaring64Bitmap.java:1028)
	at HighLowMemory.run(HighLowMemory.java:37)
	at HighLowMemory.main(HighLowMemory.java:9)

Some of the results are a little out of step so some more digging may be needed (e.g. addEachMising(10k, 10k, true) but generally positive

After looking at the inconsistent results, it seems that approx 1 time in 20 to one of the benchmarks seems out of line with the norm. I expect the JIT made a different choice that was good/bad.
The difference doesn't seem to be related to the actual parameters for the performance test, just a different fork

Anything reproducing the behavior would be useful. Thanks

Here is a simple example where a bitmap containing one bit consumes all heap

Did anyone have a chance to look at this?