Increase resource limits to prevent crashes #887

tomasfratrik · 2025-01-16T10:49:38Z

If Leapp was run with low resource limits, a crash would occur. This commit adjusts the resource limits to higher values to prevent such crashes:

The first limit increased is the maximum number of open file descriptors.
The second is the limit for the maximum writable file size.

Note that this fix was first implemented into leapp-repository, but with fsize values being too low, the leapp tool started to crash, thats why it has to be implemented into the framework.

Successfully tested with very low fzise values (ulimit -f 1 && leapp preugrade passes).

Jira: RHEL-79411

github-actions · 2025-01-16T10:49:48Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please @oamg/developers to notify leapp developers of the review request
/packit copr-build to submit a public copr build using packit

To launch regression testing public members of oamg organization can leave the following comment:

/rerun to schedule basic regression tests using this pr build and leapp-repository*main* as artifacts
/rerun 42 to schedule basic regression tests using this pr build and leapp-repository*PR42* as artifacts
/rerun-sst to schedule sst tests using this pr build and leapp-repository*main* as artifacts
/rerun-sst 42 to schedule sst tests using this pr build and leapp-repository*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please consider rerunning the CI by commenting leapp-ci build (might require several comments). If the problem persists, contact leapp-infra.

tomasfratrik · 2025-01-16T11:02:04Z

/packit copr-build

tomasfratrik · 2025-01-16T11:05:41Z

/packit retest-failed

MichalHe · 2025-02-10T14:50:37Z

/packit retest-failed

leapp/utils/workarounds/resources.py

MichalHe · 2025-02-10T16:22:03Z

/packit build

MichalHe · 2025-02-10T17:09:04Z

Steps to reproduce the original issue, for reviewers

ulimit -Sf unlimited
dd if=/dev/urandom of=/var/log/leapp/leapp-upgrade.log bs=1M count=20 -- use random so that tarfile is guaranteed to be large
ulimit -Sf 10000 -- set the limit to 10_000 * 1024 bytes = ~10MB
leapp upgrade --debug

Results in:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/util.py", line 70, in archive_logfiles
    tar.add(file_to_add)
  File "/usr/lib64/python3.6/tarfile.py", line 2147, in add
    self.addfile(tarinfo, f)
  File "/usr/lib64/python3.6/tarfile.py", line 2175, in addfile
    copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
  File "/usr/lib64/python3.6/tarfile.py", line 263, in copyfileobj
    dst.write(buf)
  File "/usr/lib64/python3.6/gzip.py", line 264, in write
    self.fileobj.write(self.compress.compress(data))
OSError: [Errno 27] File too large

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/leapp", line 11, in <module>
    load_entry_point('leapp==0.18.0', 'console_scripts', 'leapp')()
  File "/usr/lib/python3.6/site-packages/leapp/cli/__init__.py", line 51, in main
    cli.command.execute('leapp version {}'.format(VERSION))
  File "/usr/lib/python3.6/site-packages/leapp/utils/clicmd.py", line 111, in execute
    args.func(args)
  File "/usr/lib/python3.6/site-packages/leapp/utils/clicmd.py", line 133, in called
    self.target(args)
  File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/breadcrumbs.py", line 166, in wrapper
    return f(*args, breadcrumbs=breadcrumbs, **kwargs)
  File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/__init__.py", line 80, in upgrade
    util.archive_logfiles()
  File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/util.py", line 79, in archive_logfiles
    tar.add(cfg.get('database', 'path'))
  File "/usr/lib64/python3.6/tarfile.py", line 2760, in __exit__
    self.fileobj.close()
  File "/usr/lib64/python3.6/gzip.py", line 309, in close
    fileobj.write(self.compress.flush())
OSError: [Errno 27] File too large

Originally the version of the leapp module has been defined inside `leapp/__init__.py` module which has been loaded when building the leapp python module/package. Part of the `leapp/__init__.py` file is also load and application of workarounds right now, which is unsafe and adding new workarounds can lead to crash of the build process as the required runtime python modules do not have to be installed on the system that time. Moving the definition into the separate `leapp/__version__.py` file which could be used safely during the build time. Note that nowadays suggested python way requires Python 3.8+ but this project is going to support Python3.6 for another years, so we go this old way still.

If Leapp was run with low resource limits, a crash would occur. This commit adjusts the resource limits to higher values to prevent such crashes: * The first limit increased is the maximum number of open file descriptors. * The second is the limit for the maximum writable file size. Resolves: RHEL-26459 and RHEL-16881

pirat89 · 2025-02-12T18:43:29Z

leapp/utils/workarounds/resources.py

+        except ValueError as err:
+            raise FrameworkInitializationError(
+                'Unable to set the required soft limit of resource due to system restrictions. '
+                'This limit is necessary to prevent the program from crashin. '


Suggested change

'This limit is necessary to prevent the program from crashin. '

'This limit is necessary to prevent the program from crashing. '

MichalHe · 2025-02-12T19:08:40Z

Testing the PR - fsize

Leapp without the changes introduced by this PR

Using the reproducer provided in my earlier comment on the current (main) version of leapp, we get:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/util.py", line 71, in archive_logfiles
    tar.add(file_to_add)
  File "/usr/lib64/python3.6/tarfile.py", line 2147, in add
    self.addfile(tarinfo, f)
  File "/usr/lib64/python3.6/tarfile.py", line 2175, in addfile
    copyfileobj(fileobj, self.fileobj, tarinfo.size, bufsize=bufsize)
  File "/usr/lib64/python3.6/tarfile.py", line 263, in copyfileobj
    dst.write(buf)
  File "/usr/lib64/python3.6/gzip.py", line 264, in write
    self.fileobj.write(self.compress.compress(data))
OSError: [Errno 27] File too large

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/leapp", line 11, in <module>
    load_entry_point('leapp==0.18.0', 'console_scripts', 'leapp')()
  File "/usr/lib/python3.6/site-packages/leapp/cli/__init__.py", line 51, in main
    cli.command.execute('leapp version {}'.format(VERSION))
  File "/usr/lib/python3.6/site-packages/leapp/utils/clicmd.py", line 111, in execute
    args.func(args)
  File "/usr/lib/python3.6/site-packages/leapp/utils/clicmd.py", line 133, in called
    self.target(args)
  File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/breadcrumbs.py", line 166, in wrapper
    return f(*args, breadcrumbs=breadcrumbs, **kwargs)
  File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/__init__.py", line 80, in upgrade
    util.archive_logfiles()
  File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/util.py", line 80, in archive_logfiles
    tar.add(cfg.get('database', 'path'))
  File "/usr/lib64/python3.6/tarfile.py", line 2760, in __exit__
    self.fileobj.close()
  File "/usr/lib64/python3.6/gzip.py", line 309, in close
    fileobj.write(self.compress.flush())
OSError: [Errno 27] File too large

This PR's version of leapp

leapp upgrade --debug --no-rhsm does not crash when using the above reproducer and the archive .tar file is created without any issues.

MichalHe · 2025-02-12T19:31:03Z

Testing limit on the number of opened file descriptors

I have added the following code

    fds = []
    for i in range(200):
        fds.append(open('/tmp/leapp-test/0.txt', 'w'))

to the function achive_logfiles in commands/upgrade/util.py, and calling ulimit -Sn 100 before executing leapp.

The patch works as expected - without it I get an error, with it everything goes smoothly.

Also note that we do not need to set the limit on the number of opened file descriptors. This issue has been encountered previously, because we were leaking file descriptors, not because the system had low limits.

MichalHe · 2025-02-12T19:39:40Z

The e2e tests are failing because 9.2 is not a valid target. I will merge this in a second, while my own 8.10 > 9.6 upgrade finishes.

## Announce This is the last Python 2.7 compatible release. Putting Py2 on its earned rest, all the code is going to keep Python 3.6 compatibility. In the same time we are going to stop the testing and building on RHEL 7. ## Packaging - Bump leapp-framework to 6.0 (oamg#870) - Bump leapp-framework-dependencies to 6 (oamg#870) - Require PyAML (on RHEL 7) and python3-PyYAML (on RHEL 8) packages (oamg#870) - Introduce /etc/leapp/actor_conf.d/ in the leapp rpm (oamg#885) ## Framework ### Fixes - Increase soft limits on the number of opened file descriptors and maximum size of manipulated files during initialization (oamg#887) ### Enhancements - [Tech preview] Introduce configurability for leapp actors (oamg#870) ## Snactor ### Enhancements - snactor(run): allow using an existing execution context in used leapp.db (oamg#873) ## stdlib ### Enhancements - [Tech preview] Introduce `api.retrieve_config` function to obtain configuration for the actor based on the specified configuration schema (oamg#870)

…rned rest, all the code is going to keep Python 3.6 compatibility. In the same time we are going to stop the testing and building on RHEL 7. - Bump leapp-framework to 6.0 (oamg#870) - Bump leapp-framework-dependencies to 6 (oamg#870) - Require PyAML (on RHEL 7) and python3-PyYAML (on RHEL 8) packages (oamg#870) - Introduce /etc/leapp/actor_conf.d/ in the leapp rpm (oamg#885) - Increase soft limits on the number of opened file descriptors and maximum size of manipulated files during initialization (oamg#887) - [Tech preview] Introduce configurability for leapp actors (oamg#870) - snactor(run): allow using an existing execution context in used leapp.db (oamg#873) - [Tech preview] Introduce `api.retrieve_config` function to obtain configuration for the actor based on the specified configuration schema (oamg#870)

## Announce This is the last Python 2.7 compatible release. Putting Py2 on its earned rest, all the code is going to keep Python 3.6 compatibility. In the same time we are going to stop the testing and building on RHEL 7. ## Packaging - Bump leapp-framework to 6.0 (oamg#870) - Bump leapp-framework-dependencies to 6 (oamg#870) - Require PyAML (on RHEL 7) and python3-PyYAML (on RHEL 8) packages (oamg#870) - Introduce /etc/leapp/actor_conf.d/ in the leapp rpm (oamg#885) ## Framework ### Fixes - Increase soft limits on the number of opened file descriptors and maximum size of manipulated files during initialization (oamg#887) ### Enhancements - [Tech preview] Introduce configurability for leapp actors (oamg#870) ## Snactor ### Enhancements - snactor(run): allow using an existing execution context in used leapp.db (oamg#873) ## stdlib ### Enhancements - [Tech preview] Introduce `api.retrieve_config` function to obtain configuration for the actor based on the specified configuration schema (oamg#870)

## Announce This is the last Python 2.7 compatible release. Putting Py2 on its earned rest, all the code is going to keep Python 3.6 compatibility. In the same time we are going to stop the testing and building on RHEL 7. ## Packaging - Bump leapp-framework to 6.0 (#870) - Bump leapp-framework-dependencies to 6 (#870) - Require PyAML (on RHEL 7) and python3-PyYAML (on RHEL 8) packages (#870) - Introduce /etc/leapp/actor_conf.d/ in the leapp rpm (#885) ## Framework ### Fixes - Increase soft limits on the number of opened file descriptors and maximum size of manipulated files during initialization (#887) ### Enhancements - [Tech preview] Introduce configurability for leapp actors (#870) ## Snactor ### Enhancements - snactor(run): allow using an existing execution context in used leapp.db (#873) ## stdlib ### Enhancements - [Tech preview] Introduce `api.retrieve_config` function to obtain configuration for the actor based on the specified configuration schema (#870)

MichalHe requested changes Feb 10, 2025

View reviewed changes

leapp/utils/workarounds/resources.py Outdated Show resolved Hide resolved

leapp/utils/workarounds/resources.py Outdated Show resolved Hide resolved

tomasfratrik force-pushed the resource-fix branch from 1f67c62 to 51e485b Compare February 12, 2025 09:37

pirat89 force-pushed the resource-fix branch 2 times, most recently from 51e485b to 656e7b9 Compare February 12, 2025 12:03

tomasfratrik force-pushed the resource-fix branch from 656e7b9 to 82d31b8 Compare February 12, 2025 12:24

pirat89 force-pushed the resource-fix branch from 82d31b8 to 128d210 Compare February 12, 2025 12:30

tomasfratrik force-pushed the resource-fix branch from 128d210 to 0b774e3 Compare February 12, 2025 12:36

pirat89 added the bug label Feb 12, 2025

pirat89 added this to the 8.10/9.6 milestone Feb 12, 2025

pirat89 reviewed Feb 12, 2025

View reviewed changes

MichalHe approved these changes Feb 12, 2025

View reviewed changes

MichalHe merged commit 1d01d4c into oamg:main Feb 12, 2025
24 of 28 checks passed

MichalHe added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label Feb 12, 2025

pirat89 mentioned this pull request Feb 14, 2025

Release 0.19.0 #891

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Increase resource limits to prevent crashes #887

Increase resource limits to prevent crashes #887

Uh oh!

tomasfratrik commented Jan 16, 2025 •

edited by pirat89

Loading

Uh oh!

github-actions bot commented Jan 16, 2025

Uh oh!

tomasfratrik commented Jan 16, 2025

Uh oh!

tomasfratrik commented Jan 16, 2025

Uh oh!

MichalHe commented Feb 10, 2025

Uh oh!

Uh oh!

Uh oh!

MichalHe commented Feb 10, 2025

Uh oh!

MichalHe commented Feb 10, 2025 •

edited

Loading

Uh oh!

pirat89 Feb 12, 2025

Uh oh!

MichalHe commented Feb 12, 2025

Uh oh!

MichalHe commented Feb 12, 2025

Uh oh!

MichalHe commented Feb 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	'This limit is necessary to prevent the program from crashin. '
	'This limit is necessary to prevent the program from crashing. '

Increase resource limits to prevent crashes #887

Increase resource limits to prevent crashes #887

Uh oh!

Conversation

tomasfratrik commented Jan 16, 2025 • edited by pirat89 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 16, 2025

Thank you for contributing to the Leapp project!

Uh oh!

tomasfratrik commented Jan 16, 2025

Uh oh!

tomasfratrik commented Jan 16, 2025

Uh oh!

MichalHe commented Feb 10, 2025

Uh oh!

Uh oh!

Uh oh!

MichalHe commented Feb 10, 2025

Uh oh!

MichalHe commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps to reproduce the original issue, for reviewers

Uh oh!

pirat89 Feb 12, 2025

Choose a reason for hiding this comment

Uh oh!

MichalHe commented Feb 12, 2025

Testing the PR - fsize

Leapp without the changes introduced by this PR

This PR's version of leapp

Uh oh!

MichalHe commented Feb 12, 2025

Testing limit on the number of opened file descriptors

Uh oh!

MichalHe commented Feb 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tomasfratrik commented Jan 16, 2025 •

edited by pirat89

Loading

MichalHe commented Feb 10, 2025 •

edited

Loading