Add mqtt.max_connections per-node connection limit#16367
Conversation
1ab47ab to
1f0df89
Compare
ansd
left a comment
There was a problem hiding this comment.
If mqtt.max_connections is set, does the limit as proposed in your PR work correctly in any of the following cases:
- Port to Virtual Host Mapping: https://www.rabbitmq.com/docs/mqtt#port-to-virtual-host-mapping
- Dual stack (e.g.
0.0.0.0for IPv4 and::for IPv6) - A mix of MQTT and MQTTS connections
Because Ranch creates independent connection supervisors for every unique protocol, IP family, and port combination, ranch:info(RanchRef) only counts connections for that specific listener. In the scenarios above, the limit will multiply rather than act as a global node limit.
Additionally, the MQTT implementation in RabbitMQ is designed to scale to over 1 million concurrent connections per node. The connection limit check should be done as efficiently as possible.
To summarise, just following the same pattern for MQTT as done for Stream connections may not be sufficient. MQTT works differently because:
- Connections can be mapped to vhosts by port.
- MQTT use cases typically need to handle an order of magnitude more connections than Stream use cases.
My idea is to use the following to query the current number of MQTT connections on a given node:
ets:info(persistent_term:get(?PG_SCOPE), size).This approximation should be good enough since the MQTT spec defines:
Each Client connecting to the Server has a unique ClientID.
RabbitMQ enforces this. This solution is more correct globally and more efficient. What do you think?
baa588d to
17ae4f2
Compare
ansd
left a comment
There was a problem hiding this comment.
Thank you @lukebakken !
Your new approach is simpler, more efficient, and more correct.
|
The test coverage of this feature is very limited, I will add a few more basic tests. |
Add the `mqtt.max_connections` configuration key, which sets a
node-wide limit on the number of concurrent MQTT connections.
The limit is checked in `rabbit_mqtt_processor:process_connect/5`
as the first step in the CONNECT packet handler, before
authentication. When exceeded, the broker returns a CONNACK with
reason code `quota_exceeded` (MQTT 5) or return code
`not_authorized` (MQTT 3.x/4).
The active connection count is obtained via
`ets:info(persistent_term:get(?PG_SCOPE), size)`. The MQTT plugin
creates a node-local PG scope in `rabbit_mqtt_sup`; each connection
joins it via `pg:join/3` in `register_client_id/4`, using
`{VHost, ClientId}` as the group key. Since the MQTT spec requires
unique client IDs per vhost, which RabbitMQ enforces, each group
has exactly one member. The PG scope ETS table therefore has one
row per active connection node-wide, covering all listeners and
transports (plain TCP, TLS, and Web MQTT). `ets:info/2` reads this
count in O(1) - important given that MQTT nodes can host a very
large number of connections.
The check runs before `register_client_id/4`, so the current
connection is not yet counted; the limit comparison is `>= Limit`.
When `mqtt.max_connections` is absent, `application:get_env/3`
returns `infinity` and no check is performed.
A `node_connection_limit` test is added to the `limit` group in
`auth_SUITE`.
6aaa47a to
7584804
Compare
7584804 to
a4d56ad
Compare
(cherry picked from commit 1291866)
A cosmetic follow-up to #16367
Add `mqtt.max_connections` per-node connection limit (backport #16367)
Closes #16347 (partial - adds
mqtt.max_connections).What this PR does
Adds
mqtt.max_connections, a per-node connection limit for the MQTT plugin. When the limit is reached, the broker sends a CONNACK with return codenot_authorized(MQTT 3.x/4) or reason codequota_exceeded(MQTT 5) and closes the connection.How it works
The limit is checked in
rabbit_mqtt_processor:process_connect/5as the first step in the CONNECT packet handler, before authentication. It does not operate at the Ranch transport layer, so the TCP listen queue is unaffected and Ranch continues accepting connections normally.The active connection count uses
ets:info(persistent_term:get(?PG_SCOPE), size). The MQTT plugin creates a node-local PG scope inrabbit_mqtt_sup; each connection joins it viapg:join/3inregister_client_id/4, using{VHost, ClientId}as the group key. Since the MQTT spec requires unique client IDs per vhost, which RabbitMQ enforces, each group has exactly one member. The PG scope ETS table therefore has one row per active connection node-wide, covering all listeners and transports (plain TCP, TLS, and Web MQTT).ets:info/2reads this count in O(1) - important given that MQTT nodes can host a very large number of connections.The check runs before
register_client_id/4, so the current connection is not yet counted; the limit comparison is>= max_connections.Configuration
mqtt.max_connections = 1000The default is
infinity(no limit). The setting accepts a non-negative integer orinfinity.Testing
Adds
node_connection_limitto thelimitgroup inauth_SUITE. The test sets the limit to 0 via RPC so the first CONNECT attempt is rejected, covering both MQTT v4 and v5 via the existingexpected_connection_limit_error/1helper.