Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Speed up XZ decompression by 5x with bufio wrapper #520

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Mar 19, 2025
Merged

Speed up XZ decompression by 5x with bufio wrapper #520

merged 1 commit into from
Mar 19, 2025
+4 −2

Conversation

vsarunas
Copy link
Contributor

Nomad was taking over 7.5 seconds just to extract an 11MB tar.xz archive for our use case.
Investigation led to XZ library which performs poorly without buffered I/O (ulikunitz/xz#23).

Decompression performance can be improved dramatically by introducing buffered I/O as suggested in the case.

Benchmark results (using hyperfine with identical test files served from a local python3 -m http.server 8888 web server):

Before:

./go-getter http://localhost:8888/test.tar.xz tmp-xz
Time (mean ± σ): 7.721s ± 0.019s [User: 3.541s, System: 4.225s]
Range (min … max): 7.685s … 7.749s (10 runs)

After:

./go-getter http://localhost:8888/test.tar.xz tmp-xz
Time (mean ± σ): 1.429s ± 0.025s [User: 1.378s, System: 0.061s]
Range (min … max): 1.411s … 1.495s (10 runs)

This represents a 5.4x speedup and should significantly improve performance when working with XZ-compressed artifacts.

@vsarunas vsarunas requested a review from a team as a code owner March 17, 2025 10:01
@hashicorp-cla-app HashiCorp CLA App
Copy link

hashicorp-cla-app bot commented Mar 17, 2025 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@vsarunas vsarunas mentioned this pull request Mar 17, 2025
Open
mukeshjc
mukeshjc approved these changes Mar 18, 2025
View reviewed changes
Copy link
Contributor

@mukeshjc mukeshjc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@crw
Copy link
Contributor

crw commented Mar 18, 2025

Hi @vsarunas, are you making this change to target a specific HashiCorp product? Terraform is using the 1.x tag, and Packer is using the 2.x tag, so this change will likely need to be backported to one or the other (I suspect backported to 1.x but not entirely sure which tag main builds natively.)

@vsarunas
Copy link
Contributor Author

Hi @crw, yes interested in Nomad which uses currently v1.7.8. I do not see a v1 specific branch. I should branch from the tag v1.7.8 release and have 2 PRs?

@crw
Copy link
Contributor

crw commented Mar 19, 2025

Hi @vsarunas, I got caught up on the strategy here. main builds to v1, so this change should be fine as it is in this PR. If we wanted to port this to v2 as well, there is a v2 code branch that would need to be updated. So, to solve your goals, this PR is fine as-is.

tgross
tgross approved these changes Mar 19, 2025
View reviewed changes
Copy link
Member

@tgross tgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

We'll be happy to accept this into Nomad as well, but we'll need the repo owners to merge here first.

@crw
Copy link
Contributor

crw commented Mar 19, 2025

Hi @hashicorp/team-ip-compliance -- none of the project teams have merge permissions anymore. Can you please merge this change? Thanks!

@mallikabandaru mallikabandaru merged commit c831ccd into hashicorp:main Mar 19, 2025
1 check passed
@picatz picatz mentioned this pull request Mar 21, 2025
Open
@dduzgun-security
Copy link
Collaborator

@vsarunas a new release is now available v1.7.9, thanks for the contribution. 🙏

This was referenced Aug 15, 2025
Merged
Merged
Merged
@hc-github-team-nomad-core hc-github-team-nomad-core mentioned this pull request Aug 18, 2025
Merged
6 tasks
@renovate renovate bot mentioned this pull request Aug 22, 2025
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
2 more reviewers

@tgross tgross tgross approved these changes

@mukeshjc mukeshjc mukeshjc approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@vsarunas @crw @dduzgun-security @tgross @mukeshjc @mallikabandaru