Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add Zstandard compression support to the compress plugin #12211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 11 commits into
base: 10.0.x
Choose a base branch
from

Conversation

JakeChampion
Copy link

This is a backport of #12201

  • Updated Makefile and configuration files to include Zstandard as a compression algorithm.
  • Enhanced compress.cc to handle Zstandard compression and added necessary initialization and cleanup functions.
  • Modified misc.cc to normalize accept-encoding headers for Zstandard.
  • Updated tests to verify Zstandard functionality alongside existing compression algorithms.

@JakeChampion JakeChampion force-pushed the jake/9.2.10-zstd branch 19 times, most recently from 5f12f8e to 85ec4ab Compare April 23, 2025 15:03
@bryancall bryancall added the compress compress plugin label Apr 23, 2025
@bryancall bryancall modified the milestones: 10.2.0, 9.2.11 Apr 23, 2025
@bneradt
Copy link
Contributor

bneradt commented Apr 28, 2025

@JakeChampion : Thank you for working on this. It looks like a few autests have output expectation issues.

The compress autest fails due to zstd beign in the output now. Should be a simple gold file update required (src/tests/gold_tests/pluginTest/compress/compress_userver.gold):

   Run: 31-tr: Failed
     file compress_userver.log : Checking that compress_userver.log matches compress_userver.gold - Failed
        Reason: File differences
         Gold File : /home/jenkins/workspace/Github_Builds/autest/src/tests/gold_tests/pluginTest/compress/compress_userver.gold
         Data File : /tmp/sandbox/compress/compress_userver.log
         - 0/gzip, deflate, sdch, br
         + 0/zstd, gzip, deflate, sdch, br
         ?   ++++++
         
           0/gzip
           0/br
           0/deflate
         + 0/zstd
         - 1/gzip, deflate, sdch, br
         + 1/zstd, gzip, deflate, sdch, br
         ?   ++++++
         
           1/gzip
           1/br
           1/deflate
         + 1/zstd
         - 2/gzip, deflate, sdch, br
         + 2/zstd, gzip, deflate, sdch, br
         ?   ++++++
         
           2/gzip
           2/br
           2/deflate
         + 2/zstd
         + 3/zstd, gzip, deflate, sdch, br
         + 3/gzip
         + 3/br
         + 3/deflate
         + 3/zstd
           0/gzip;q=0.666
           0/gzip;q=0.666x
           0/gzip;q=#0.666
           0/gzip; Q = 0.666
           0/gzip;q=0.0
           0/gzip;q=-0.1
           0/aaa, gzip;q=0.666, bbb
           0/ br ; q=0.666, bbb
           0/aaa, gzip;q=0.666 , 
           3/gzip

Also, the normalized_ae_match_vary_cache autest fails, it looks like also for output expecation update needs:

       file /tmp/sandbox/normalized_ae_match_vary_cache/_output/0-tr-Default/stream.stdout.txt : There should be no Proxy Verifier violation errors. - Failed
          Reason: Contents of /tmp/sandbox/normalized_ae_match_vary_cache/_output/0-tr-Default/stream.stdout.txt contains expression: "Violation|Invalid status"
             Details:
               [INFO]: Equals Violation: Different. Key: "34", Field Name: "x-response-identifier", Correct Value: "Zstd-Accept-Encoding", Actual Value: "Br-Accept-Encoding" : 80
               [INFO]: Equals Violation: Different. Key: "34", Field Name: "x-cache", Correct Value: "miss", Actual Value: "hit-fresh" : 81
               [INFO]: Equals Violation: Different. Key: "38", Field Name: "x-response-identifier", Correct Value: "Zstd-Accept-Encoding", Actual Value: "Br-Accept-Encoding" : 98

@bryancall
Copy link
Contributor

bryancall commented Apr 28, 2025

We don't add new features to patch releases and we are not planning on having another minor release of 9.x. We are happy for you to use CI to verity the change and we can have someone look over the code for you. 9.2.x is currently our LTS release and we are doing new development on ATS 10.x.

For example on out release versioning: major.minor.patch

@JakeChampion JakeChampion changed the base branch from 9.2.x to 10.0.x July 15, 2025 10:35
@JakeChampion JakeChampion force-pushed the jake/9.2.10-zstd branch 2 times, most recently from 36cb81a to 0547b04 Compare July 15, 2025 10:41
This patch fixes the Accept-Encoding quality calculation logic in the
HTTP transaction cache and updates test files to reflect the corrected
behavior, removing workarounds for previously broken cache matching.

Cache quality calculation improvements:
- Replace multiplicative quality combining with minimum quality
  selection for multiple content encodings
- Simplify wildcard matching to return quality directly
- Remove gzip-specific fallback logic that bypassed normal quality
  calculation and caused inconsistent cache behavior

Test updates for corrected behavior:
- Remove NOTICE comments describing broken cache matching behavior
- Update test expectations to match correct cache responses instead
  of workaround responses
- Fix cache hit/miss expectations for multi-encoding scenarios
- Add missing Content-Encoding headers in server responses
- Update response identifiers to match the actual cached alternates

Specific test corrections:
- "br, compress, gzip" now correctly matches "Br-Gzip-Accept-Encoding"
  instead of falling back to "Gzip-Accept-Encoding"
- "zstd, compress, gzip" now correctly matches "Zstd-Gzip-Accept-Encoding"
  instead of falling back to "Gzip-Accept-Encoding"
- "zstd, gzip;q=0.8" now correctly matches "Zstd-Gzip-Accept-Encoding"
  instead of falling back to "Gzip-Accept-Encoding"
- Individual encoding requests now create proper cache alternates
  instead of incorrectly matching empty encoding alternates

The previous multiplicative quality calculation (q_a * q_b) was causing
unexpectedly low quality scores and incorrect cache alternate selection.
The new minimum quality approach ensures that multi-encoding responses
are properly cached and matched according to HTTP content negotiation
standards.

All test cases now pass without workarounds and demonstrate correct
cache behavior for all compression algorithms (gzip, brotli, zstd)
across various Accept-Encoding
@JakeChampion JakeChampion force-pushed the jake/9.2.10-zstd branch 6 times, most recently from 1eec47d to 24eecbd Compare July 15, 2025 17:04
@JakeChampion JakeChampion force-pushed the jake/9.2.10-zstd branch 2 times, most recently from c576eb3 to 5c1c85e Compare July 15, 2025 21:07
@JakeChampion JakeChampion force-pushed the jake/9.2.10-zstd branch 7 times, most recently from 630575e to c66d5db Compare July 15, 2025 23:09
- Add support for configurable compression levels per host:
  * gzip-compression-level (1-9, default 6)
  * brotli-compression-level (0-11, default 6)
  * brotli-lgwin (10-24, default 16)
  * zstd-compression-level (1-22, default 12)

This enables fine-tuning compression performance vs. speed trade-offs
on a per-host basis
@ezelkow1 ezelkow1 removed this from the 9.2.11 milestone Aug 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants