Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Performance drop after upgrading from 1.8.0 to 1.10.1 #30

@pan3793

Description

@pan3793

Recently, Apache Spark followed the suggestion to upgrade lz4-java from 1.8.0 to 1.10.1 for security purpose, but from my testing, this upgrade introduces considerable performance regression - lz4-java 1.10.1 is about 10~15% slower on lz4 compression than 1.8.0, and is about 5% slower on lz4 decompression even with migrating to the suggested safeDecompressor

The test cases are simple, which compress/decompress a CSV file (size ~283M) generated by TPC-DS (SF1)

You can find my test code and report at apache/spark#53453 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions