Would you be open to putting up the benchmarking changes as a separate PR, to be landed before this one?

I'd love to keep the history clean/separate.

Yup: #86

Nice. thanks. I'll wait for you to rebase.

Rebased

Wait, this is not rebased, right? I'm seeing a bunch of changes that are already on main.

I'm not intimately familiar with github workflows, but I'm assuming there's a way to:

• rebase your changes on top of main
• potentially squash the commit

I wonder if I can figure out how to do this myself.

Hey, I squashed all commits into one.

I don't understand github then - I expected it's possible to have this PR ONLY list the things we're actually changing here. I expected all the makefile, go.sum, etc changes to be gone/not visible.

Is that not possible? If so, do we need a new PR rebased on the changes already in main?

Okay, I think I clicked the right button now.

Lemme take one last look later today, I'll merge it in after.

Could you explain this one a bit more? Since we have other fields of the struct below, how does postpadding work here exactly?

Every time when we perform CAS on state field, we mark the whole cache line as "dirty". So when some other goroutine wants to read perRequest, maxSlack or clock caching protocol will force us to reload the whole cache line even though perRequest, maxSlack or clock are constants in our case. So we introduce postpadding to move perRequest, maxSlack and clock or separate cache line. More about it here https://en.wikipedia.org/wiki/False_sharing

Benchmarks that show perf degradations when we remove postpadding check that atomic_int64_no_padding and atomic_int64_no_sched_no_padding are slower than their counterparts.

Fascinating. This sounds like an even more advanced case of https://pkg.go.dev/golang.org/x/tools/go/analysis/passes/fieldalignment, where we want to separate read/write variables.

Questions:
(1) do you know if there's any runtime variable that defines cache size line and int64 line? Perhaps we could avoid the hardcodes ints, and thus make the code self documenting.
(2) The comment needs updating, it's no longer "statePointerSize"

Unrelated:
(3) Are you using same padding techniques in some other projects? Could you paste on some links? I'd be curious to see how often, and to what effect these are used.

Do you know if there's any runtime variable that defines cache size line and int64 line?

I didn't know about it before you asked, but apparently golang.org/x/sys/cpu package has CacheLinePad struct for it https://pkg.go.dev/golang.org/x/sys/cpu#CacheLinePad
I can you that, but I'd prefer to add it in a separate PR if you don't mind. I wanted to keep this one as is, not to remeasure everything from scratch.

The comment needs updating, it's no longer "statePointerSize"

Done

Are you using same padding techniques in some other projects? Could you paste on some links?

Yes, it is a widely used technic. I can find some good examples and post them here if you want. In some languages, like Java, for example, AtomicReference or AtomicInteger already have padding built-in.

Did you happen to have run a test with and without the yielding?

Naively thinking, the fact that we failed to load means that another go-routine has updated the value. N-1 go-routines will then try to re-try, each of them yielding ... meaning the go-routine that has successfully written before, is more likely to write again?

This seems like it would be affecting the "fairness" of the rate-limiting. While we don't explicitly say we want to be fair, I wonder if this can be unexpected in some cases.

Good that you asked about it. Because when I basically copied current atomic implementation added yielding, measured that it improves things and didn't go back to re-check when switched to int64 based implementation. So to compare different techniques used in this branch I created separate branch

On that graph we can see that yields improve performance of atomic pointer based rate limiter, I suspect because cost of contention and retry is very high in that implementation, we basically allocate new state for every try.

But in case with int64 based rate limiter the cost of retry is much lower, so it is OK to try immediately after failure.

OK, according to this benchmarks I'll change atomicInt64Limiter to remove yielding.

I think this is equivalent (?), and I like that we only have a single CompareAndSwapInt64 - WDYT?

Separately, I was thinking about changing the initialization of the limiter so that it always inits with valid now. This way we don't have to handle the 0 case in the "hot" path.
This would change the startup behavior - we'd probably start with "maxSlack" accumulated requests on first request, where now we don't. The one "if/else" didn't seem worth it though.

I'll try to rewrite it in such way

If you do, please just do the switch.

For the second part (initialization change) let's keep it separate - I'd like to keep the initialization consistent across all implementation. I was just asking for your opinion about that.

@rabbbit
Hey, I made the following implementation in a separate branch it is a bit simpler, but also a 3.3 ns slower, 27.4ns vs 24.1ns. I think it is because of using time.Time.Sub() or time.Time.Add(), that make some additional checks and handle internal state that int64 math doesn't do.

But this implementation handles zero state better. What do you think if I use that approach for state handling, but leave int64 math around?

We can also merge this PR as is and deal with refactoring in subsequent PR.

I think we're conflating 3 different changes here, right?

1. initial state change
2. single vs multiple CAS
3. int64 vs time types

I'm not okay with changing (1) (sorry, I left a note on this above too) - if we decide to change the initial state, we should change it in all implementations, and potentially add a test to formalize the startup behavior. I would definitely do this separately, or perhaps not at all.

I think I'd like us to decide on (2) in the current PR. As of my current understanding, there's no performance benefit in the original version, and the code seems clearer. So this seems like something we should just do.

I'm okay with leaving (3) as a follow up. Seems like a tradeoff between a tiny bit of speed (2ns-3ns) vs developer experience (compiler help). There's also a question of correctness, but I'm not right now sure how the serialization to int64 plays with the monotonic clock. We can potentially iterate on that separately.

Sorry for the amount of questions/comments :)

I agree on all 3 points. I'll quickly double check if switch has OK perf and make this change. For 2 other things we can discuss later in issues maybe.

Results are definitely strange, but switch is slower:

I'm sorry about the number of comments. If this is getting annoying, let me know, and I can take over.

I'm okay with landing the switch version, and then iterating. For now:

• I prefer the readability over 5ns
• we're now significantly faster than the previous iteration anyway.

However, ^ made me realize is that I don't necessarily care about switch being present, but I'd like to have a single CAS operation. This means we could probably do something like:

		if timeOfNextPermissionIssue == 0 {
	                 // If this is our first request, then we allow it.	
			newTimeOfNextPermissionIssue = now - int64(t.perRequest)
                 }

		if now-timeOfNextPermissionIssue > int64(t.maxSlack) {
			// a lot of nanoseconds passed since the last Take call
			// we will limit max accumulated time to maxSlack
			newTimeOfNextPermissionIssue = now - int64(t.maxSlack)
                } else {
			// calculate the time at which our permission was issued
			newTimeOfNextPermissionIssue = timeOfNextPermissionIssue + int64(t.perRequest)
                }

		if atomic.CompareAndSwapInt64(&t.state, timeOfNextPermissionIssue, newTimeOfNextPermissionIssue) {
			break
		}

Which I think is largely equivalent to your original version, but easier to parse? We're doing exactly one more subtraction on the startup phase, but that shouldn't matter.

But, again, I'm okay with the switch version if you'd like to land that. And we can iterate from that.

Yes, let's land switch for now and then iterate. It can be that only one CAS is causing the slowdown, or maybe we won't need handling of timeOfNextPermissionIssue == 0 in the future. We will see.

Done

thinking about this more, I think this can be [56] as well?

prepadding is usually used to avoid accidental colocations with other contended objects on the heap, since we don't know their size, we just use full cache line size

Am I mistaken thinking that [56] achieves the same target?

It seem to me that prepadding of 56, size of 8, and post-padding of 56 guarantees that the object-in-question will always be "alone" in the cache line?

I think we can reduce the number of substractions here potentially too.

maxTimeOfNextPermissionIssue = now - int64(t.maxSlack)
if timeOfNextPermissionIssue < maxTimeOfNextPermissionIssue {
     newTimeOfNextPermissionIssue = maxTimeOfNextPermissionIssue
} else {
    newTimeOfNextPermissionIssue = timeOfNextPermissionIssue + int64(t.perRequest)

we can micro-optimize this separately though, in follow-ups.
}