any opinions on the serialisation

About this change on the serialization format of Roaring64Bitmap:

• Define serialisation format for 64 bit bitmaps #351 is a good start regarding some expectations: some users may not expect the serialization format to change. Hence, this may need a major release. (@lemire may confirm the way to go)
• Is there any documentation about the new format, and how it changes with the old? One may go through the PR, but it may then call to is there any specification of the new format?. A synthetic note would be a good first step @mkeskells .
• If this format was to change, should it follows https://github.com/RoaringBitmap/RoaringFormatSpec?tab=readme-ov-file#extension-for-64-bit-implementations ? (probably not as current specification targets rather implementation, which is probably inefficient for ART-based bitmaps)
• Should this new format be documented itself in https://github.com/RoaringBitmap/RoaringFormatSpec?tab=readme-ov-file#extension-for-64-bit-implementations in some ART section? (@mkeskells Indeed, I'm aware this serialization matter is a piece of a larger scheme ; but this piece is already a big one).
• Is the new format compatible with the previous one? (i.e. I understand this relates with some leaks: this new format may prevent such leaks. But maybe the new code may be able to load from old binaries, which would dramatically simplify this matter).
• Did one reviewed if the old/new ART serialization format is compatible with alternative implementations? (In the vein of Roaring64NavigableMap serialized binary format is not compatible with CRoaring's 64bit version #366)
• Was there any specific effort in other implementations regarding such a specification/format, suggesting we should comply into it? (@lemire probably a good overview)
• Should we enable different serialization formats in given implementation, not to break existing applications willing to upgrade without losing their existing serialized bitmaps (in the vein of Enable Portable format in Roaring64NavigableMap #598 for Raoring64NavigableMap)?

@weijietong As initial contributor of the 64bits ART implementations, do you have any opinion on this matter?

(Given the amount of changes in this PR and related matters, especially on the serialization format, I wonder if it would be simpler to go with an alternative implementation, than changing the existing one. The alternative implementation may at some point replace the legacy one (with additional mecanisms to manage the changes of serialization format)).

(@mkeskells I guess it may feel overwhelming to push such changes into such an established library, making some changes quite laborious. Hence this suggestion of an alternative implementation. Benchmarks would remain very useful to confirm strength of weaknesses of various designs. And an additional implementation may make such benchamrks easier to conduct).

#803 (comment)

I had a look. It seems from a quick read that the proposal is that we write a sequence of 32bit high vae -> 32bit roaringBitmap. Thats quite a way away from the structure of the 64BitRoaringMap, bit not unachievable. It would churn memory a bit as I seet if, converting the lower 2 bytes of the Trie into a data structure, or at least some scanning to work out number of entries etc
Am I reading that correctly.
as Seq<32bit int, Seq<32bit roaringMap>>
which I think is Seq<32bit int, Seq<16 bit int, 16bit container>

I would imagine that having a portable format would be useful, but also a fast format has some attractions. Its not black and white. Maybe there is a need for both.
Certainly if we were to do this I would like to get rid of the duplication of strucure with DataOutput and ByteBuffer!

Is there a specific view if what the format should be?

I personally did not study ART-based serialization format. The Map-based is pretty straightforward. My view would be to have a look into other implementation (CRoaring, GoRoaring) to check their ART-implementation (if any), and try (...) to converge a portable format.

I would be very curious to know if the Map-based format would induce such a big penalty or not when read/written into ART structures. Having a dedicated ART format seems very legitimate too. Though, it should not be too much tied with the implementation (to prevent one big issue here: changing the format due to changing the implementation).

I'm not very keen not to do the specification effort, as it has been demonstrated that some users have serialization usages through libraries, and have strong expectations in term of stability. I underatand you own use-cases @mkeskells involves desrialization/deserialization, supposedly long-term scenarios, hence expecting stability on this matter (or with retro-compatibility).

[Check failure on line 11 in roaringbitmap/src/main/java/org/roaringbitmap/art/BranchNode.java](https://github.com/RoaringBitmap/RoaringBitmap/pull/805/files#annotation_38523723317) 

Code scanning
/ CodeQL

Inconsistent equals and hashCode
Error

Class BranchNode overrides  but not hashCode.

@mkeskells Is this legit?

Class BranchNode overrides but not hashCode.

@mkeskells Is this legit?

I only added equals for the tests. These are not user visible classes, and along with most mutuble collections dont have a sensible hashcode

happy to leave as it is, or change to use another method and adjext the test to not use assertEquals if that is easuer for maintainers

It feels awkward to have .equals and not .hascode. It feels awkward to have .equals if it is used only for tests.

along with most mutuble collections dont have a sensible hashcode

I do not get your point.

I would feel better if .equals was not added, if it is relevant only for tests-purposes, as it may lead to confusion to maintainers. I guess we could introduce some test helper methods, dedicated for test purposes.

I personally did not study ART-based serialization format. The Map-based is pretty straightforward. My view would be to have a look into other implementation (CRoaring, GoRoaring) to check their ART-implementation (if any), and try (...) to converge a portable format.

Agreed - the ART serialisation is very fragile

I would be very curious to know if the Map-based format would induce such a big penalty or not when read/written into ART structures. Having a dedicated ART format seems very legitimate too. Though, it should not be too much tied with the implementation (to prevent one big issue here: changing the format due to changing the implementation).

The issue that I see is that the current map serialisation format is based on 32 bit roaring bitmaps, whichwould have to be constructed and deconstructed on the fly,and the structure for that is really just a sequnce of (16 bit address, container)

I think that for the 64 bit roaring bitmaps it map be better to consider an interchange format based on 16 bit containers

Effectively that could make the 32 and 64 bit solutions are similar (conceptually)
i.e. a Sequence of (address, container)

for the 32 bit the address is 16 bit, and for the 64 bit its 48 bits

you could potentially add different prefix sizes in the future/when valhalla delivers

From a quick look at the code I think this should be easy for both of the 64 bit implementations. It would add a bit of time to the serialisation and deserialisation, but [I assume] in reality its dominated by the containers both for time and space

I'm not very keen not to do the specification effort, as it has been demonstrated that some users have serialization usages through libraries, and have strong expectations in term of stability. I underatand you own use-cases @mkeskells involves desrialization/deserialization, supposedly long-term scenarios, hence expecting stability on this matter (or with retro-compatibility).

at least fo this implementation the docs show the dragons -

  * Unlike RoaringBitmap, there is no specification for now: it may change from one java version to
  * another, and from one RoaringBitmap version to another.

I would feel better if .equals was not added, if it is relevant only for tests-purposes, as it may lead to confusion to maintainers. I guess we could introduce some test helper methods, dedicated for test purposes.

Done. On checking it wasnt used in this PR, it got copied over from the other one, but I will remove it there as well

 * Unlike RoaringBitmap, there is no specification for now: it may change from one java version to
  * another, and from one RoaringBitmap version to another.

this comment is a bit out-dated. We worked into a specification of a portable format. It is compatible with either CRoaring or GoRoaring, and compatible with Java-Roaring if some flag is toggled. Though, it is definitely quite some burden for seemingly limited (but not inexistant) use.

 * Unlike RoaringBitmap, there is no specification for now: it may change from one java version to
  * another, and from one RoaringBitmap version to another.

this comment is a bit out-dated. We worked into a specification of a portable format. It is compatible with either CRoaring or GoRoaring, and compatible with Java-Roaring if some flag is toggled. Though, it is definitely quite some burden for seemingly limited (but not inexistant) use.

This is the comment from Roaring64Bitmap. I doubt it would chnage from one java version to another, but would chnage in the internal details of the BranchNodes change,
Also if the Containers, or LeafNode change, which I am proposing

This is the comment from Roaring64Bitmap. I doubt it would chnage from one java version to another, but would chnage in the internal details of the BranchNodes change,

Oh, OK. Then I suppose this could be easier to merge. @lemire I was not aware we were explicitely providing less guarantee on Roaring64Bitmap serialization format.

@lemire I was not aware we were explicitely providing less guarantee on Roaring64Bitmap serialization format.

Please see

https://github.com/RoaringBitmap/RoaringFormatSpec?tab=readme-ov-file#extension-for-64-bit-implementations

Java Roaring bitmaps implementation offers an Map-based 64-bit implementation handling signed longs. It is not compatible with this serialization format (which does not handle signed keys).

@lemire It is clear to me Map-based 64-bit implementation has no clear serialization format (I'm the author of this part of the spec :D). I meant here I feel weird we do not provide any guarantee about the stability of the actual format (even if not specificed) (i.e. zero guarantee in term of retrocompatbility), given:

it may change from one java version to another, and from one RoaringBitmap version to another

in Roaring64Bitmap Javadoc. In practise, I would expect some guarantee (hence some feedback to @mkeskells around issues raised by changing the format). But I may be zeal from myself, given the Javadoc states the contrary.

Also happy to work to a format that is stable

My preference would be to have something based on some metadat and a sequence of the 16 bit Containers, as that seems to be the basis for everything

I am not tied to the format in this PR, so if the change was nessessary, but not sufficient, let move to something that is portable, extensible, and reasonable efficient

Or have 2 formats, a native fast format with constraints, and a portable format, that is stable

Any more thoughts on what to do here?

@mkeskells

We currently have one documented 64-bit format. We even have a Kaitai formal definition.

https://github.com/RoaringBitmap/RoaringFormatSpec

Sadly, people have been designing their own 64-bit formats left and right in various programming languages which makes interoperability difficult and it is generally difficult to rely or test any one format.

I encourage everyone to adopt a common portable standard, as much as possible.