Add rate-limiting for channel gain change messages #2535

softins · 2022-03-22T16:49:35Z

Short description of changes

Adds rate-limiting for channel gain change messages.
This avoids a backlog of messages being queued due to ACK latency,
particularly if using a MIDI controller that send fine-grained
level changes. After sending a gain change message, further gain
changes will be updated locally for ~~300ms~~ a time and then the latest
value sent to the server. This time will be 50ms by default, or double the
current ping time, whichever is greater.

CHANGELOG: Client: Fix potential long delay in sending fader changes to the server.

Context: Fixes an issue?

Fixes #2492

Does this change need documentation? What needs to be documented and how?

No, bug fix only

Status of this Pull Request

Tested and working

What is missing until this pull request can be merged?

Review and testing on other platforms, and by @bawbgale, who reported the issue

Checklist

I've verified that this Pull Request follows the general code principles
I tested my code and it does what I want
My code follows the style guide
I waited some time after this Pull Request was opened and all GitHub checks completed without errors.
I've filled all the content above

softins · 2022-03-22T17:06:28Z

Tested with single and grouped channels, and the Wireshark trace examined.

src/client.h

src/client.cpp

pljones · 2022-03-22T17:16:32Z

src/client.cpp

+        if ( newGain[iId] != oldGain[iId] )
+        {
+            // send new gain and record as old gain
+            Channel.SetRemoteChanGain ( iId, oldGain[iId] = newGain[iId] );


When does the assignment happen? Before the call or after? I'd rather see it on a separate line so I don't have to think about it.

The assignment expression happens first, and the assigned value is used as the parameter. To me it's a common C idiom, and in this case it also avoids repeating the array lookup again (which the compiler may possibly optimise out, but I like to write optimal code of my own).

One could do:

fGain = oldGain[iId] = newGain[iId]; Channel.SetRemoteChanGain ( iId, fGain );

but to me the original is perfectly readable.

I find it readable, but it's easy to miss, IMO.
I would have gone with a simple two-step assignment / method call as I don't think this is in a performance-critical path.
The alternative (reuse fGain) also looks fine to me.

In the end, I don't feel strongly.

OK, I've updated it for clarity to use fGain.

softins · 2022-03-22T17:45:27Z

Harmless code alerts in Oboe, but Github won't let me dismiss them.

ghost · 2022-03-22T19:42:24Z

I think waiting for 50 milliseconds would give a more realistic response because 300 ms is too long.
Also, the server should make finer level changes for the same amount of time. For example, the client updates level changes every 50 ms if commanded, and the server may smoothly update the received level with several smaller level changes within the following 50 ms.

hoffie · 2022-03-23T07:27:50Z

src/client.cpp

    // start timer so that elapsed time works
    PreciseTime.start();

+    // set gain delay timer to single-shot and connect handler function


That comment almost repeats the variable/method names and provides little added value, but I guess it's ok.. :)

hoffie · 2022-03-23T07:33:06Z

src/client.cpp

+// but just stored in newGain[iId], and the minGainId and maxGainId updated to note the range of
+// IDs that must be checked when the time expires (this will usually be a single channel
+// unless channel grouping is being used). This avoids having to check all possible channels.


As we are talking about worst case n=200 or something and this is not part of the critical path (sound/network processing) and not part of something which scales exponentially (e.g. server-side per-client-per-client stuff), I think I would have gone with the check-all-channels approach in order to keep the logic simpler and shorter.

PS: Now this whole block comment provides value and is well-written! :)

Yes, but I figured if it is possible at little cost to limit the number of checks needed to the specific one or a few out of the 200 channels, it's worth doing. And of course include an explanation, as I did :)

hoffie · 2022-03-23T07:35:20Z

src/client.cpp

+        if ( newGain[iId] != oldGain[iId] )
+        {
+            // send new gain and record as old gain
+            Channel.SetRemoteChanGain ( iId, oldGain[iId] = newGain[iId] );


I find it readable, but it's easy to miss, IMO.
I would have gone with a simple two-step assignment / method call as I don't think this is in a performance-critical path.
The alternative (reuse fGain) also looks fine to me.

In the end, I don't feel strongly.

softins · 2022-03-23T09:42:25Z

@softins I just tried that build with a server 160ms away and it works much better! MIDI controller changes take effect very quickly. Thank you!

Originally posted by @bawbgale in #2492 (comment)

This avoids a backlog of messages being queued due to ACK latency, particularly if using a MIDI controller that send fine-grained level changes. After sending a gain change message, further gain changes will be updated locally for 300ms and then the latest value sent to the server.

src/client.cpp

hoffie · 2022-03-23T10:54:45Z

I've tested this PR and it does what it claims to do. Wireshark looks as expected.

At the same time, this causes very noticable loudness jumps (similar to artifacts) when changing the fader level (tested via streaming music and listening to the sound coming back from the server while playing with the slider of my channel).
One could argue that the most common slider adjustments would be:

Slow, but not by large amounts
Quick, but only to make a (maybe new) participant less loud quickly
Therefore, it might not be too relevant.

With 50ms, it's much smoother again, but obviously causes more protocol messages.
@DavidSavinkoff's proposal to add server-side fading would probably work around that, but I don't see a way to add that without introducing new logic into the very critical sound mixing code path (we do have fading for new channels, but I'm not sure if it's possible to adapt).

In essence, I think

we should have this PR as it solves a rather nasty problem
we should try to use the lowest possible constant for delayed updates

softins · 2022-03-23T12:49:40Z

I think any server-side smoothing of gain changes might be a worthwhile improvement, but should be a separate feature and PR, as it would only apply to servers running new code. The current PR addresses the root issue at the client, which is where it originates, and works when connected to any version of server.

I did wonder about having a smaller default delay period, and then increasing it if necessary based on the ping time of the connected server. Maybe 50ms or 100ms by default, and 1.5x or 2x the ping time if greater (to allow a little extra time)? It sounds like that might be a worthwhile improvement.

softins · 2022-03-23T12:55:45Z

The ping time multiplied by a suitable constant could easily be stored in CClient by CClient::OnCLPingReceived() for use when starting the timer. Or probably just store it and then multiply by the constant when starting the timer.

softins · 2022-03-23T15:13:35Z

I've added a second commit to default the timer to max(50ms, pingtime*2), and refactored the timer starting into a separate function to be DRY. I tested against my private server in London and a server in Hong Kong and compared the packet traces. As expected, when changing a fader, the London server was sent updates at 50ms intervals, and the HK server at 400ms intervals.

hoffie

Thanks! This behaves much better and looks clean. Nice work :)

PR jamulussoftware#2535 introduced rate limiting for gain change messages. The logic required storing the previously used gain value per channel. This logic had two issues: 1. The previously used gain value defaulted to 0, despite the server-side view of the channel being set to 1 (as the default). Therefore, gain(0) changes during a series of gain changes would be lost. The most common scenario would be the initial connection, which always triggers the rate limit and therefore the faulty logic. This also affected New Client Level = 0. 2. The previously used gain values were not reset upon changing servers. This might have caused losing arbitrary gain change messages, e.g. stored fader values. This commit introduces a gain level memory reset to 1 (100%) on connect to fix both of these issues. Fixes: jamulussoftware#2730

PR jamulussoftware#2535 introduced rate limiting for gain change messages. The logic required storing the previously used gain value per channel. This logic had some flaws: 1. The previously used gain value defaulted to 0, despite the server-side view of the channel being set to 1 (as the default). Therefore, gain(0) changes during a series of gain changes would be lost. The most common scenario would be the initial connection, which always triggers the rate limit and therefore the faulty logic. This also affected New Client Level = 0. 2. The previously used gain values were not reset upon changing servers. This might have caused losing arbitrary gain change messages, e.g. stored fader values. 3. The previously used gain values were not reset upon a channel disconnect. This might have caused missing fader level restores. This commit introduces a gain level memory reset to 1 (100%) on connect as well as on channel disconnects to fix these issues. Fixes: jamulussoftware#2730

PR jamulussoftware#2535 introduced rate limiting for gain change messages. The logic required storing the previously used gain value per channel. This logic had some flaws: 1. The previously used gain value defaulted to 0, despite the server-side view of the channel being set to 1 (as the default). Therefore, gain(0) changes during a series of gain changes would be lost. The most common scenario would be the initial connection, which always triggers the rate limit and therefore the faulty logic. This also affected New Client Level = 0. 2. The previously used gain values were not reset upon changing servers. This might have caused losing arbitrary gain change messages, e.g. stored fader values. 3. The previously used gain values were not reset upon a channel disconnect. This might have caused missing fader level restores. This commit introduces a gain level memory reset to 1 (100%) on connect as well as on channel disconnects to fix these issues. Fixes: jamulussoftware#2730 Co-authored-by: ann0see <[email protected]>

softins force-pushed the gain-rate-limit branch from 6ebc768 to f32c241 Compare March 22, 2022 17:00

pljones reviewed Mar 22, 2022

View reviewed changes

src/client.h Outdated Show resolved Hide resolved

pljones reviewed Mar 22, 2022

View reviewed changes

src/client.cpp Outdated Show resolved Hide resolved

pljones reviewed Mar 22, 2022

View reviewed changes

hoffie reviewed Mar 23, 2022

View reviewed changes

softins force-pushed the gain-rate-limit branch from f32c241 to 2db5156 Compare March 23, 2022 09:48

hoffie reviewed Mar 23, 2022

View reviewed changes

src/client.cpp Outdated Show resolved Hide resolved

Improve delay time based on ping to server

68d5612

pljones approved these changes Mar 23, 2022

View reviewed changes

hoffie added this to the Release 3.9.0 milestone Mar 23, 2022

hoffie approved these changes Mar 23, 2022

View reviewed changes

hoffie force-pushed the gain-rate-limit branch from 94e3cd4 to 68d5612 Compare March 23, 2022 21:17

softins merged commit c72195f into jamulussoftware:master Mar 23, 2022

hoffie mentioned this pull request Jul 17, 2022

Regression: Gain changes not applied in all cases (e.g. new client level = 0) #2730

Closed

4 tasks

hoffie mentioned this pull request Jul 17, 2022

Client: Avoid losing gain changes #2737

Merged

5 tasks

softins deleted the gain-rate-limit branch August 30, 2023 17:29

This was referenced Nov 26, 2024

Pan change messages need to be rate-limited similarly to Gain. #3428

Closed

Apply rate limiting to Pan as well as Gain #3429

Merged

Add rate-limiting for channel gain change messages #2535

Add rate-limiting for channel gain change messages #2535

Uh oh!

Conversation

softins commented Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

softins commented Mar 22, 2022

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

softins commented Mar 22, 2022

Uh oh!

ghost commented Mar 22, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

softins commented Mar 23, 2022

Uh oh!

Uh oh!

hoffie commented Mar 23, 2022

Uh oh!

softins commented Mar 23, 2022

Uh oh!

softins commented Mar 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

softins commented Mar 23, 2022

Uh oh!

hoffie left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

softins commented Mar 22, 2022 •

edited

Loading

softins commented Mar 23, 2022 •

edited

Loading