Reported by community:
hazelcast_example.txt
hazelcast_frame_buffer_log.txt

C++ compiler version:
Hazelcast Cpp client version: 4.2
Hazelcast server version: 4.2
Number of the clients:
Cluster size, i.e. the number of Hazelcast cluster members:
OS version (Windows/Linux/OSX):

Please attach relevant logs and files for client and server side.

Expected behaviour

Actual behaviour

Encountered an issue with publishing of large messages from Hazelcast C++ API. Our calculation nodes publish messages 100KB to 200KB in size using the C++ API. Occasionally, these messages arrive at the Hazelcast cluster in a garbled form. It turns out that they are interleaved with data from heartbeat / ping messages.

I believe it is due to a heartbeat messages and data being written from separate threads. This may not be a problem for smaller messages (<65K) because each write is atomic. For larger messages, boost::asio library splits them into smaller chunks resulting in interleaving with ping messages.

ttaching an example program that we ran on our systems that reproduced the problem. This generates a large payload and repeatedly publishes it to different keys in a map. We are setting it to time out if the publish takes longer than 30 seconds – and we got timeouts after about 70 publishes. In the test code, we also increased the heartbeat to happen every 10ms in order to force the problem to happen more frequently. We are using hazelcast-cpp-client 4.2.0, and hazelcast node 4.2.

Attaching a log snippet from the hazelcast node during one instance of this problem. As the data is passed into hazelcast, everything up through frame 3 is received correctly. However, once frame 4 arrives, the hazelcast node interprets the frame buffer length to be an enormously large number. It then incorrectly parses the future data received as if it were all part of frame 4. In the course of our investigation, we discovered that this happens because a ping message is interleaved into frame 3, making it longer than expected. We were able to identify the location of the ping message in one case, remove it from the stream of bytes received, and correctly deserialize the data that was expected to be in frame 3.

Steps to reproduce the behaviour

Please run the attached C++ code for reproducing.