-
-
Notifications
You must be signed in to change notification settings - Fork 454
Fix capturing OOM errors when very memory constrained #1633
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
5adfccf
to
82dd661
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a very risky approach...
In containerized environment, we may risk to make the kernel unhappy, which in turn can decide to kill the whole process (and our event with it). But the amount (and time) we're risking is low, and the opposite risk is to lose the message due to the limit, so I think it may be worth it.
Yeah. It's definitely not ideal. However I felt like the relatively small memory increase for a process that is being killed anyway (because of the fatal error) seemed like the best of the worst options. I'd be open to other but the only other one would be to take the 5 MiB beforehand but that seems even worse and unnecessary. I also made it possible to configure / disable (just like you can with the current memory reserve) to allow users that cannot risk the increase to disable it if they wish. |
The only other option I see is to reserve more memory (the str_repeat) such that we always have enough space for Sentry to complete its task. i.e. keep 5MB for all processes. This also has trade-offs. Mainly, 99.9% of processes will never need this extra overhead. Only a few processes that hit an OOM would. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should update the docs prior merging, https://docs.sentry.io/platforms/php/integrations/#fatalerrorlistenerintegration.
Thanks for investigating this!
We currently reserve 10 KiB of memory which we free when handling a OOM error, however this is in many cases way to small. This is especially try when extremely memory constrained and the OOM is triggered by allocating just a tiny bit of extra memory from PHP.
For example when testing with:
This doesn't need the reserved memory since a huge block of memory is being allocated and fails which results in the full memory never being allocated.
In contrary when testing with:
Here the memory is being allocated in tiny chunks, so when PHP runs out of memory it becomes prominent that the 10 KiB of reserved memory is way to little. From my testing we'd need 30-60x that amount to handle sending a relatively simple payload.
So the solution is to increase the memory limit (once) and use that extra headroom to send the OOM event, I've chosen for 2 MiB, from my testing I think 2 MiB is usually just enough but to be a little on the safe side I've opted for 5 MiB. This value is configurable if needed and this behavior can also be completely disabled if required.
To configure this behavior you can do the following before running
\Sentry\init(...)
:I think we would also be able to shrink the reserved memory we are currently holding but since it's already very little I think it's safe to leave the value like it is.As it turns out we actually need a little more for the regex matching to work correctly on some configurations so I've upped it very slightly by default.Once accepted I will also port this to the new 4.x branch which suffers from the same problems.
I've also had to ignore some phpstan errors caused by a dependency update.