fix: actually handle sessions in parallel #308
Conversation
There was a problem hiding this comment.
LGTM, but some questions so that I understand.
According to https://cloud.google.com/functions/docs/concepts/exec#auto-scaling_and_concurrency
Each instance of a function handles only one concurrent request at a time. This means that while your code is processing one request, there is no possibility of a second request being routed to the same instance. Thus the original request can use the full amount of resources (CPU and memory) that you requested.
So in the context of Cloud Functions, this parallelism will never be used, is that right? If CF is all we cared about, we wouldn't need this change.
However, this parallelism may still be useful in cases where this framework is deployed directly to Cloud Run, which by default has a concurrency value of 80. Is that the thinking?
Since I'm guessing that 32 and 64 were chosen as nice round numbers, we might want to consider using 80 somewhere to match Cloud Run's default concurrency.
Codecov Report
@@ Coverage Diff @@
## main #308 +/- ##
==========================================
+ Coverage 55.47% 55.48% +0.01%
==========================================
Files 562 562
Lines 15105 15205 +100
==========================================
+ Hits 8380 8437 +57
- Misses 6725 6768 +43
Continue to review full report at Codecov.
|
Probably.
Yes.
Doh. I should have looked this up, completely forgot. Will change. |
We were creating a thread to handle each request, and then promptly blocking until the thread completed. With this change we leave the thread running. Before accepting a new connection we cleanup the memory resources for old sessions, not that they are that big. We block and stop accepting requests if there are 64 sessions running, at that point it is probably better to let the hosting environment (Cloud Run, Cloud Functions, whatever) create new instances to handle the additional load.
We were creating a thread to handle each request, and then promptly
blocking until the thread completed. With this change we leave the
thread running. Before accepting a new connection we cleanup the memory
resources for old sessions, not that they are that big. We block and
stop accepting requests if there are 64 sessions running, at that point
it is probably better to let the hosting environment (Cloud Run, Cloud
Functions, whatever) create new instances to handle the additional load.
Fixes #309