If I understood the patch correctly, I doubt that it is able to prevent the mentioned DDoS - who says, that a "smart" attacker will always use the same port? Furthermore if there is a Distributed attack, IMHO it is even counterproductive. Also a whole organization/company may sit behind a single firewall, so defaults are probably not reasonable. Last but not least option names are far too long, should be shortened and the request number limit (0 | > 0) should be used to decide, whether to apply a limit or not (the intended option --no-ratelimit-401 is redundant) - not, i.e. req-limit=0 sounds reasonable to me.

Lets remember that in this situation the diagram of the attack would look as follows:

Attacker's Spoofed UDP Packet -> Coturn -> Victim

This patch is meant to mitigate coturn's ability to be used to launch DDoS reflection/amplification attacks (not protect the server from a DDoS). By doing this it allows those running coturn to be good netizens by preventing their servers from being used in a malicious way.

Let me try to clarify things a little bit and elaborate on what we're trying to do here:

If I understood the patch correctly, I doubt that it is able to prevent the mentioned DDoS

In my opinion it will because:

a) The UDP packets are spoofed on behalf of the victim, the 401 response is reflected at the victim
b) By rate limiting the amount of packets (IE: 401 responses in this case) we are mitigating this type of abuse by preventing the attacker from continuing to reflect unlimited traffic at their targets.

who says, that a "smart" attacker will always use the same port?

This patch does not use port numbers, but rather the IP address of the source of the UDP packet. This patch does not use the port. Also note the *_no_port versions of the map functions added to facilitate these comparisons. Additionally source ports for UDP are randomized so using them for this would be pointless.

Furthermore if there is a Distributed attack, IMHO it is even counterproductive

I assume you have made the D bold here to suggest that the source addresses of the UDP packets would be distributed, however in this case the UDP packet source is spoofed. As far as the coturn server sees, there is only one source address (even if the attack is distributed from multiple sources).

This attack would be distributed by using multiple coturn servers to attack a single target (not multiple attackers attacking the coturn service, but rather using it to reflect the 401 requests at a victim).

Please also consider that we are not rate limiting all traffic. Only 401 Unauthorized responses. By doing this we should mitigate any unintentional abuse (For example spoofing a target to get them banned/rate limited).

Also a whole organization/company may sit behind a single firewall, so defaults are probably not reasonable

Understood. What do you think better defaults would be? Remember this code only runs when a 401 response is sent. Do we believe there would be more than 100 unauthorized responses in a 60 second period of time. That would be a 401 unauthorized responses every .6 seconds, sustained for a full 60 seconds straight.

It should also be noted that we're currently working on an allowlist that would exempt IP addresses from this rate limit. However in my opinion this should be broken up into a second patch (which I will submit when completed).

Last but not least option names are far too long, should be shortened and the request number limit (0 | > 0) should be used to decide, whether to apply a limit or not (the intended option --no-ratelimit-401 is redundant) - not, i.e. req-limit=0 sounds reasonable to me.

This makes sense to me. I see no problem making these changes! As far as shortened, what do you think of: --401-window and --401-req-limit?

Finally for now I think, using tools like fail2ban might be the better option to apply rate limits if one is convinced, that this changes anything or helps somehow.

A few things here:

a) fail2ban is not really designed to mitigate (D)DoS style attacks, as it periodically reads the logs
b) In an attack situation where the logs are filling up (as detailed in the tickets linked above) fail2ban will cause more stress to the system as it process 10s of 1000s of additional lines of logs
c) Using fail2ban would ban ALL traffic from a source IP address. This could be used to DoS attack a victim by banning them from the coturn server. This patch only rate limits responding with 401 Unauthorized responses, not all traffic.
d) It would be best practice for coturn to deny this tool for use by DDoS for hire groups/tools/etc. As we've seen with NTPd, these types of abuses are only taken away from malicious actors (ie: DDoS for hire) if the software shuts these abused features down by default

In closing I will try to address your code review as soon as possible. Thank you for taking the time to properly review the code!

For an immediate small fix I suggest using --no-software-attribute when running coturn which will reduce the size of the packets (non-data). It will not solve the problem but will make the exploit less useful for the attacker (reduce the amplification factor).

For some reason this config is an opt-in and not enabled by default....

Any news on this?

Today there were several hundrets of coturn servers used for an amplification attack on hosts of a french hosting company. One of our correctly configured servers has been used for this aswell. Our German hosting provider acknowledged that hundrets of servers in their ip space were used for this attack and that they had to enable a manual mittigation.

CoTURN is behaving correctly in rejecting unauthorized requests. But this enables amplification attacks if spoofed UDP packets are used...

Hello everyone.... My coturn server was also part of this attack mentioned by @lordwebbie . Please prioritize this PR as the whole world would benefit from this. Thank you.

Same here on hetzner, after years of running quietly.

Right now mitigating with blocking various malicous netblocks and with

no-rfc5780
no-stun-backward-compatibility
response-origin-only-with-rfc5780
no-software-attribute

parameters set. But yeah, this PR would definitely help.

Same here on hetzner, after years of running quietly.

Right now mitigating with blocking various malicous netblocks and with

no-rfc5780
no-stun-backward-compatibility
response-origin-only-with-rfc5780
no-software-attribute

parameters set. But yeah, this PR would definitely help.

Same here. Thanks for this. 🫶
Subscribed to see progress on this.

@e-lisa Thank you very much for working on this. A couple of questions/clarifications before we try to make more changes:

Could this attack also occur with STUN messages (Binding Requests) that don’t require authentication and therefore don’t trigger 401 responses?

It seems the attack might work without needing to spoof IP addresses—simply by exploiting STUN backward compatibility. Specifically, there's an attribute in the messages that allows the response to be sent to an arbitrary address: https://datatracker.ietf.org/doc/html/rfc3489#section-11.2.2

The recommendations to disable certain options (which, as far as I know, are disabled by default in the config file) seem like an effective countermeasure 👏

This PR came from Wire's attempt to mitigate abuses last year (we have been seeing them also for quite a while).
We are running a fork of coturn with this PR included in production on Hetzner, and it successfully mitigated this specific attack on the 27th.
But we are not 100% satisfied with the approach as well. On top of this PR, an allow-list was added after seeing the impact the rate-limiting had in production. But the allow-list is also not good enough for our use case, at least being static.
So we see this approach more as a bandaid fix that also comes with some downsides.

On your questions @ggarber - since we authenticate the binding requests, and these were specifically targeted by the attacks, we worked on those. But probably unauthenticated ones are also affected.

We have set the following options on our coturns, but the amplification factor of roughly 2.5x is still worth it for the attackers.

no-stun-backward-compatibility
secure-stun
no-rfc5780

Also, there is another kind of attack that is not caught by the rate limiting. Attackers are also targeting whole ranges of IP addresses (and not all ports on a given IP address, like in the latest Hetzner incident).

As the developer of this patch, I am still looking at this and am open to suggestions. I will try to merge this branch with the latest code next week so this fix does not become too bitrot as I do think many people can still benfit by running this fix.

At the end of the day people are abusing couturn, and anything we can do to stop it is a win for all of us.

@mastaab - I think to ratelimit entire netblocks, would be a separate feature, however I don't think it is an unreasonable idea, but probably would build on this work (but not block it). I also think this feature would need its own settings as the you would not want to apply the same rules for a single IP to an entire netblock (or vice versa)

Sorry if I am being absolutely naive: would it help using a non standard port and in an obscure subdomain?

stun.mydomain.com:3478 vs notudp.mydomain.com:4546

What about Crowdsec to prevent abuse?

What are the patterns that should be looked for at syslog?

If I understood the patch correctly, I doubt that it is able to prevent the mentioned DDoS - who says, that a "smart" attacker will always use the same port? Furthermore if there is a Distributed attack, IMHO it is even counterproductive. Also a whole organization/company may sit behind a single firewall, so defaults are probably not reasonable. Last but not least option names are far too long, should be shortened and the request number limit (0 | > 0) should be used to decide, whether to apply a limit or not (the intended option --no-ratelimit-401 is redundant) - not, i.e. req-limit=0 sounds reasonable to me.

Lets remember that in this situation the diagram of the attack would look as follows:

Attacker's Spoofed UDP Packet -> Coturn -> Victim

This patch is meant to mitigate coturn's ability to be used to launch DDoS reflection/amplification attacks (not protect the server from a DDoS). By doing this it allows those running coturn to be good netizens by preventing their servers from being used in a malicious way.

Let me try to clarify things a little bit and elaborate on what we're trying to do here:

If I understood the patch correctly, I doubt that it is able to prevent the mentioned DDoS

In my opinion it will because:

a) The UDP packets are spoofed on behalf of the victim, the 401 response is reflected at the victim b) By rate limiting the amount of packets (IE: 401 responses in this case) we are mitigating this type of abuse by preventing the attacker from continuing to reflect unlimited traffic at their targets.

who says, that a "smart" attacker will always use the same port?

This patch does not use port numbers, but rather the IP address of the source of the UDP packet. This patch does not use the port. Also note the *_no_port versions of the map functions added to facilitate these comparisons. Additionally source ports for UDP are randomized so using them for this would be pointless.

Furthermore if there is a Distributed attack, IMHO it is even counterproductive

I assume you have made the D bold here to suggest that the source addresses of the UDP packets would be distributed, however in this case the UDP packet source is spoofed. As far as the coturn server sees, there is only one source address (even if the attack is distributed from multiple sources).

This attack would be distributed by using multiple coturn servers to attack a single target (not multiple attackers attacking the coturn service, but rather using it to reflect the 401 requests at a victim).

Please also consider that we are not rate limiting all traffic. Only 401 Unauthorized responses. By doing this we should mitigate any unintentional abuse (For example spoofing a target to get them banned/rate limited).

Also a whole organization/company may sit behind a single firewall, so defaults are probably not reasonable

Understood. What do you think better defaults would be? Remember this code only runs when a 401 response is sent. Do we believe there would be more than 100 unauthorized responses in a 60 second period of time. That would be a 401 unauthorized responses every .6 seconds, sustained for a full 60 seconds straight.

It should also be noted that we're currently working on an allowlist that would exempt IP addresses from this rate limit. However in my opinion this should be broken up into a second patch (which I will submit when completed).

Last but not least option names are far too long, should be shortened and the request number limit (0 | > 0) should be used to decide, whether to apply a limit or not (the intended option --no-ratelimit-401 is redundant) - not, i.e. req-limit=0 sounds reasonable to me.

This makes sense to me. I see no problem making these changes! As far as shortened, what do you think of: --401-window and --401-req-limit?

Finally for now I think, using tools like fail2ban might be the better option to apply rate limits if one is convinced, that this changes anything or helps somehow.

A few things here:

a) fail2ban is not really designed to mitigate (D)DoS style attacks, as it periodically reads the logs b) In an attack situation where the logs are filling up (as detailed in the tickets linked above) fail2ban will cause more stress to the system as it process 10s of 1000s of additional lines of logs c) Using fail2ban would ban ALL traffic from a source IP address. This could be used to DoS attack a victim by banning them from the coturn server. This patch only rate limits responding with 401 Unauthorized responses, not all traffic. d) It would be best practice for coturn to deny this tool for use by DDoS for hire groups/tools/etc. As we've seen with NTPd, these types of abuses are only taken away from malicious actors (ie: DDoS for hire) if the software shuts these abused features down by default

In closing I will try to address your code review as soon as possible. Thank you for taking the time to properly review the code!

Sidenote: We've observed some interesting interactions between this patch and nextcloud-talk-recording. Our req-limit is set to 20, and without setting 401-allowlist for the recording container it would fail if more than one or two calls were placed within the time set by 401-window. Our current working theory is that talk-recording tries to authenticate in several different ways that are unsupported by coturn, which results in 401's being generated, and finally getting limited.

This behavior should be documented somewhere, as it's extremely hard to debug (very little logs, the application itself not giving helpful output, etc).

Also, IMO 401-allowlist should be in a similar format to denied-peer-ip / allowed-peer-ip: IP ranges specified directly in the .conf file, not a separate file with IPs one by one. (I started working on refactoring this, but got confused by some abstractions around the IP lists, with hardcoded allowed/denied values.)

@eakraly @ggarber this PR looks like mergeable and the functionality is already tested by @mastaab 's Wireapp and some other people. Could you please merge the PR before the next attack wave?

@e-lisa can you please add prometheus metrics?
I am thinking of 2:

• Packets rate limited (counter)
• Size of rate-limit map (IPs rate limited)

Happy Anniversary! This critical security PR is now over a year old, and has been in the field, for over a year, defending coturn's more adventurous users. Could we merge this soon, please?

@e-lisa can you please add prometheus metrics? I am thinking of 2:

* Packets rate limited (counter)

* Size of rate-limit map (IPs rate limited)

I can see if I can't fit this in this weekend, should be very useful.

Could this attack also occur with STUN messages (Binding Requests) that don’t require authentication and therefore don’t trigger 401 responses?

I honestly have not looked at this. In the wild all I've seen is reports of using 401 responses for amplification. Someone should look into this.

CC @ggarber

Seems i lost a bit the path now..in which branche are the mentioned patches now included? I tried to get the actual master branch, compiled it,replaced my existing files from the ubuntu apt package and tried to include the additional parameters in the config file but got a "bad format" message in the logs,

FYI, this branch is now updated to the latest master

I have now had a second large network operator where I host coturn point me to this PR, to ensure I had applied it.

I ran into compilation issues with this PR on a system running Debian Trixie and GCC 14.2.0, and another running Arch Linux / GCC 15.2.1.

src/server/ns_turn_maps.h:58:1: error: parameter names (without types) in function declaration [-Wdeclaration-missing-parameter-type]
   58 | typedef int (*ur_addr_map_cond_func)(ur_addr_map_value_type);
      | ^~~~~~~

[...trimmed...]

src/server/ns_turn_maps.c:828:25: error: assignment to ‘ur_addr_map_value_type’ {aka ‘long unsigned int’} from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
  828 |             elem->value = NULL;
      |                         ^
src/server/ns_turn_maps.c:841:27: error: assignment to ‘ur_addr_map_value_type’ {aka ‘long unsigned int’} from ‘void *’ makes integer from pointer without a cast [-Wint-conversion]
  841 |               elem->value = NULL;
      |                           ^

To fix these errors, I had to make the following small changes to ns_turn_maps.c and ns_turn_maps.h:

diff --git a/src/server/ns_turn_maps.c b/src/server/ns_turn_maps.c
index 9f9c43d..c195336 100644
--- a/src/server/ns_turn_maps.c
+++ b/src/server/ns_turn_maps.c
@@ -825,7 +825,7 @@ int addr_list_foreach_del_condition(ur_addr_map *map, ur_addr_map_cond_func func
           if (func(elem->value)) {
             free((void *)elem->value);
             memset(&(elem->key), 0, sizeof(ioa_addr));
-            elem->value = NULL;
+            elem->value = (ur_addr_map_value_type)NULL;
             count++;
           }
         }
@@ -838,7 +838,7 @@ int addr_list_foreach_del_condition(ur_addr_map *map, ur_addr_map_cond_func func
             if (func(elem->value)) {
               free((void *)elem->value);
               memset(&(elem->key), 0, sizeof(ioa_addr));
-              elem->value = NULL;
+              elem->value = (ur_addr_map_value_type)NULL;
               count++;
             }
           }
diff --git a/src/server/ns_turn_maps.h b/src/server/ns_turn_maps.h
index bc17a3c..795b5bf 100644
--- a/src/server/ns_turn_maps.h
+++ b/src/server/ns_turn_maps.h
@@ -53,6 +53,7 @@ typedef struct _ur_map ur_map;
 
 typedef uint64_t ur_map_key_type;
 typedef uintptr_t ur_map_value_type;
+typedef uintptr_t ur_addr_map_value_type;
 
 typedef void (*ur_map_del_func)(ur_map_value_type);
 typedef int (ur_addr_map_cond_func)(ur_addr_map_value_type);
@@ -191,8 +192,6 @@ bool lm_map_foreach_arg(lm_map *map, foreachcb_arg_type func, void *arg);
 
 //////////////// UR ADDR MAP //////////////////
 
-typedef uintptr_t ur_addr_map_value_type;
-
 #define ADDR_MAP_SIZE (1024)
 #define ADDR_ARRAY_SIZE (4)

Hopefully this is useful to anyone else encountering the same problem. I suspect there were some breaking changes in GCC that triggered it.