Shovel: use message containers #14096
Conversation
|
I think the only breaking change is the translation of |
There was a problem hiding this comment.
When a message is shovelled from RabbitMQ to another AMQP 1.0 broker, I would expect the following to happen:
RabbitMQ encodes and sends the message.
This can be achieved by using the rabbit_queue_type API directly, as suggested by @kjnilsson initially and as described in #13957 (comment): The shovel plugin would consume directly from the queue, encode and then send the message.
If I read the code correctly, the following happens with this PR:
RabbitMQ encodes the message -> sends the message to shovel plugin -> shovel plugin decodes the message in the AMQP 1.0 client -> shovel plugin encodes the message to create a message container -> mc_amqp on the shovel plugin then decodes the message -> in rabbit_amqp10_shovel.erl:forward/3 the shovel plugin then encodes the message -> the shovel plugin then decodes the message for a format suitable by the AMQP 1.0 client, the AMQP 1.0 client then encodes the message before finally sending it.
Therefore, with this PR there are 7 (!) different encode/decode steps instead of the final goal to have only 1 step (and instead of 3 steps prior to this PR).
So, while this PR uses message containers for conversions and therefore solves one problem, it seems that performance gets worse compared to prior to this PR.
Have you done any performance tests?
That said, it's up to you to decide whether we want to accept a potentially severe performance drop as an interim solution until using rabbit_queue_type API.
|
@ansd How we are going to use the |
| case is_amqp10_body(Body) orelse (not is_list(Body)) of | ||
| true -> | ||
| #amqp10_msg{ | ||
| transfer = #'v1_0.transfer'{ | ||
| delivery_tag = {binary, DeliveryTag}, | ||
| settled = Settled, | ||
| message_format = {uint, ?MESSAGE_FORMAT}}, | ||
| %% This lib is safe by default. | ||
| header = #'v1_0.header'{durable = true}, | ||
| body = Body}; | ||
| false -> | ||
| Transfer = #'v1_0.transfer'{ | ||
| delivery_tag = {binary, DeliveryTag}, | ||
| settled = Settled, | ||
| message_format = {uint, ?MESSAGE_FORMAT}}, | ||
| from_amqp_records([Transfer | Body]) | ||
| end. |
There was a problem hiding this comment.
The function spec is now wrong: The function spec expects currently a body, but not other sections.
There was a problem hiding this comment.
The creation of the 'v1_0.transfer' record got duplicated. Avoid code duplication.
Correct, the assumption of using The case where |
I think that would need to be a new "internal" shovel protocol impl which may be slightly beyond the scope of this PR. But I agree that is where we want to be. |
See #13957