Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Close all ReadClosers explicitly retrieved from resolvers #414

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 12, 2021
Merged

Close all ReadClosers explicitly retrieved from resolvers #414

merged 1 commit into from
May 12, 2021

Conversation

@wagoodman
Copy link
Contributor

When analyzing images with several files opened the following warning was being seen in the logs:

[0025]  WARN cataloger 'javascript-package-cataloger' failed to parse entries ....: failed to parse package.json file: open

Where it seems that syft was exhausting all file descriptor resources on the system. The cause was from PR #339 which pulled in several impactful stereoscope changes, including how you retrieve contents from underlying images... which now returns ReadClosers, not just Readers. The caller is now responsible for explicitly calling Close(), though stereoscope attempts to close when an EOF is observed, not all cataloger parsers read till the end of every file.

This PR adds a corresponding Close() call for every ReadCloser provided by stereoscope.

@wagoodman wagoodman added the bug Something isn't working label May 12, 2021
@wagoodman wagoodman requested a review from a team May 12, 2021 21:16
@wagoodman wagoodman self-assigned this May 12, 2021
@github-actions
Copy link

github-actions bot commented May 12, 2021 •
edited
Loading

Benchmark Test Results

Benchmark results from the latest changes vs base branch 
name                                                   old time/op    new time/op    delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2          1.00ms ± 4%    1.00ms ± 2%    ~     (p=0.690 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2        1.37ms ± 2%    1.35ms ± 4%    ~     (p=0.421 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     512µs ± 2%     523µs ± 4%    ~     (p=0.151 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 516µs ± 3%     493µs ± 2%  -4.41%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                  545µs ± 2%     508µs ± 2%  -6.82%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                  6.89ms ± 4%    6.49ms ± 1%  -5.81%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                  753µs ± 4%     725µs ± 3%    ~     (p=0.151 n=5+5)
ImagePackageCatalogers/go-cataloger-2                     262µs ± 2%     255µs ± 2%  -2.74%  (p=0.032 n=5+5)
ImagePackageCatalogers/rust-cataloger-2                   417µs ± 5%     404µs ± 3%    ~     (p=0.222 n=5+5)

name                                                   old alloc/op   new alloc/op   delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2          97.4kB ± 0%    97.5kB ± 0%    ~     (p=0.095 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2         579kB ± 0%     579kB ± 0%  +0.05%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     112kB ± 0%     112kB ± 0%  +0.14%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 115kB ± 0%     116kB ± 0%  +0.15%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                  134kB ± 0%     134kB ± 0%  +0.04%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                  1.78MB ± 0%    1.78MB ± 0%    ~     (p=0.841 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                 1.14MB ± 0%    1.14MB ± 0%  +0.01%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-cataloger-2                    48.4kB ± 0%    48.4kB ± 0%    ~     (p=0.310 n=5+5)
ImagePackageCatalogers/rust-cataloger-2                  88.9kB ± 0%    88.9kB ± 0%  +0.06%  (p=0.008 n=5+5)

name                                                   old allocs/op  new allocs/op  delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2           1.96k ± 0%     1.96k ± 0%  +0.10%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2         5.88k ± 0%     5.89k ± 0%  +0.14%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     1.93k ± 0%     1.93k ± 0%  +0.05%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 2.37k ± 0%     2.37k ± 0%  +0.13%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                  3.19k ± 0%     3.19k ± 0%  +0.03%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                   22.3k ± 0%     22.3k ± 0%  +0.01%  (p=0.016 n=4+5)
ImagePackageCatalogers/apkdb-cataloger-2                  1.85k ± 0%     1.85k ± 0%  +0.05%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-cataloger-2                     1.40k ± 0%     1.40k ± 0%  +0.07%  (p=0.008 n=5+5)
ImagePackageCatalogers/rust-cataloger-2                   2.74k ± 0%     2.75k ± 0%  +0.04%  (p=0.008 n=5+5)

@wagoodman wagoodman merged commit baa859d into main May 12, 2021
@wagoodman wagoodman deleted the fix-leaky-fds branch May 12, 2021 21:28
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
@wagoodman
Merge pull request anchore#414 from anchore/fix-leaky-fds
8a05cf7 
Close all ReadClosers explicitly retrieved from resolvers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@luhring luhring luhring approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@wagoodman wagoodman

Labels

bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wagoodman @luhring