Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Fix capturing OOM errors when very memory constrained #1633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Nov 10, 2023
Merged

Conversation

stayallive
Copy link
Collaborator

@stayallive stayallive commented Nov 8, 2023

We currently reserve 10 KiB of memory which we free when handling a OOM error, however this is in many cases way to small. This is especially try when extremely memory constrained and the OOM is triggered by allocating just a tiny bit of extra memory from PHP.

For example when testing with:

$foo = str_repeat('x', 1024 * 1024 * 1024);

This doesn't need the reserved memory since a huge block of memory is being allocated and fails which results in the full memory never being allocated.

In contrary when testing with:

$array = [];
for ($i = 0; $i < 100000000; ++$i) {
    $array[] = str_repeat('a', 100);
}

Here the memory is being allocated in tiny chunks, so when PHP runs out of memory it becomes prominent that the 10 KiB of reserved memory is way to little. From my testing we'd need 30-60x that amount to handle sending a relatively simple payload.

So the solution is to increase the memory limit (once) and use that extra headroom to send the OOM event, I've chosen for 2 MiB, from my testing I think 2 MiB is usually just enough but to be a little on the safe side I've opted for 5 MiB. This value is configurable if needed and this behavior can also be completely disabled if required.

To configure this behavior you can do the following before running \Sentry\init(...):

// The amount of reserved memory every request (in bytes), this has to be a positive integer amount of bytes. Defaults to 16 KiB
$errorHandler = \Sentry\ErrorHandler::registerOnceFatalErrorHandler(16 * 1024);
// The amount of memory in bytes to increase the `memory_limit` to when handling a out of memory error, can also be set to `null` to disable increasing the `memory_limit`. Defaults to 5 MiB
$errorHandler->setMemoryLimitIncreaseOnOutOfMemoryErrorInBytes(5 * 1024 * 1024);

I think we would also be able to shrink the reserved memory we are currently holding but since it's already very little I think it's safe to leave the value like it is. As it turns out we actually need a little more for the regex matching to work correctly on some configurations so I've upped it very slightly by default.

Once accepted I will also port this to the new 4.x branch which suffers from the same problems.

I've also had to ignore some phpstan errors caused by a dependency update.

@stayallive stayallive force-pushed the oom-fixes branch 4 times, most recently from 5adfccf to 82dd661 Compare November 8, 2023 14:45
@stayallive stayallive self-assigned this Nov 8, 2023
@stayallive stayallive marked this pull request as ready for review November 8, 2023 14:53
Copy link
Contributor

@Jean85 Jean85 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very risky approach...

In containerized environment, we may risk to make the kernel unhappy, which in turn can decide to kill the whole process (and our event with it). But the amount (and time) we're risking is low, and the opposite risk is to lose the message due to the limit, so I think it may be worth it.

@stayallive
Copy link
Collaborator Author

Yeah. It's definitely not ideal. However I felt like the relatively small memory increase for a process that is being killed anyway (because of the fatal error) seemed like the best of the worst options. I'd be open to other but the only other one would be to take the 5 MiB beforehand but that seems even worse and unnecessary.

I also made it possible to configure / disable (just like you can with the current memory reserve) to allow users that cannot risk the increase to disable it if they wish.

@jarstelfox
Copy link
Contributor

jarstelfox commented Nov 8, 2023

This is a very risky approach...

In containerized environment, we may risk to make the kernel unhappy, which in turn can decide to kill the whole process (and our event with it).

The only other option I see is to reserve more memory (the str_repeat) such that we always have enough space for Sentry to complete its task. i.e. keep 5MB for all processes.

This also has trade-offs. Mainly, 99.9% of processes will never need this extra overhead. Only a few processes that hit an OOM would.

Copy link
Member

@cleptric cleptric left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should update the docs prior merging, https://docs.sentry.io/platforms/php/integrations/#fatalerrorlistenerintegration.

Thanks for investigating this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants