Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Decompress in 8-byte blocks#82

Merged
gatesn merged 19 commits into
developfrom
ngates/8-bytes
Mar 12, 2025
Merged

Decompress in 8-byte blocks#82
gatesn merged 19 commits into
developfrom
ngates/8-bytes

Conversation

@gatesn

@gatesn gatesn commented Mar 12, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

NOTE: we may want to fall back to a 4-byte unrolling for WASM / 4-byte architectures, but it may not matter.

@codspeed-hq

codspeed-hq Bot commented Mar 12, 2025 •
edited
Loading

Copy link
Copy Markdown

CodSpeed Performance Report

Merging #82 will improve performances by 31.01%

Comparing ngates/8-bytes (d580c00) with develop (3f1c495)

Summary

⚡ 4 improvements
✅ 12 untouched benchmarks

Benchmarks breakdown

Benchmark BASE HEAD Change
⚡ decompress 6.5 ms 5 ms +31.01%
⚡ decompress 17.5 ms 13.3 ms +30.96%
⚡ decompress 9.5 ms 7.3 ms +30.81%
⚡ decompress 1,159.2 µs 959.2 µs +20.85%

robert3005
robert3005 reviewed Mar 12, 2025
View reviewed changes
Comment thread src/lib.rs
a10y
a10y reviewed Mar 12, 2025
View reviewed changes
Comment thread src/lib.rs Outdated
@gatesn gatesn enabled auto-merge (squash) March 12, 2025 13:51
@gatesn gatesn disabled auto-merge March 12, 2025 14:22
@gatesn gatesn merged commit 0c3333e into develop Mar 12, 2025
@gatesn gatesn deleted the ngates/8-bytes branch March 12, 2025 14:22
@github-actions github-actions Bot mentioned this pull request Mar 12, 2025
Merged
gatesn
gatesn commented Mar 12, 2025
View reviewed changes
Comment thread src/lib.rs
while out_ptr.cast_const() <= block_out_end && in_ptr < block_in_end {
// Note that we load a little-endian u64 here.
let next_block = in_ptr.cast::<u64>().read_unaligned();
let escape_mask = (next_block & 0x8080808080808080)

@gatesn gatesn Mar 12, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude did a great job of explaining this:

This bit manipulation trick is used to detect special characters (typically escape characters or null terminators) in a 32-bit word. Let me break it down step by step:

let escape_mask = (next_block & 0x80808080)
    & ((((!next_block) & 0x7F7F7F7F) + 0x7F7F7F7F) ^ 0x80808080);

  1. next_block & 0x80808080 - This extracts the most significant bit (MSB) from each byte in the 32-bit word. The MSB will be 1 for bytes with values ≥ 128.

  2. (!next_block) & 0x7F7F7F7F - This inverts all bits in next_block, then masks off the MSBs, keeping only the 7 lower bits of each byte.

  3. ((!next_block) & 0x7F7F7F7F) + 0x7F7F7F7F - This adds 0x7F (127) to each byte. If a byte in the original next_block was 0 or very small, this addition will cause a carry into the MSB position.

  4. (...) ^ 0x80808080 - This XORs with 0x80808080, which flips the MSB of each byte. As a result, bytes that didn't cause a carry will now have their MSB set to 1.

  5. The final & operation combines the results: it will only keep bits set in both expressions. This identifies bytes that:

    • Have their MSB set in the original data (from step 1)
    • Did NOT cause a carry in step 3 (which happens for null bytes or small values)

This trick is commonly used in optimized string processing to identify null terminators (0x00) and other special characters (typically ASCII control characters with values < 32) without doing byte-by-byte comparisons.

In essence, it creates a bit mask where each set bit indicates the position of a special character in the 32-bit word.

gatesn pushed a commit that referenced this pull request Mar 12, 2025
@github-actions
chore: release v0.5.1 (#72)
3fce913 
## 🤖 New release

* `fsst-rs`: 0.5.0 -> 0.5.1 (✓ API compatible changes)

<details><summary><i><b>Changelog</b></i></summary><p>

<blockquote>

## [0.5.1](v0.5.0...v0.5.1) -
2025-03-12

### Other

- Decompress in 8-byte blocks
([#82](#82))
- *(deps)* lock file maintenance
([#83](#83))
- Assert enough room in decoded buffer
([#79](#79))
- *(deps)* update rust crate criterion to v2.9.1
([#80](#80))
- *(deps)* update mozilla-actions/sccache-action action to v0.0.8
([#78](#78))
- *(deps)* lock file maintenance
([#77](#77))
- Add codspeed ([#76](#76))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@gatesn gatesn mentioned this pull request Mar 12, 2025
Closed
a1412744807 added a commit to a1412744807/rs-fsst that referenced this pull request Oct 27, 2025
@a1412744807 @github-actions
chore: release v0.5.1 (#72)
c05624d 
## 🤖 New release

* `fsst-rs`: 0.5.0 -> 0.5.1 (✓ API compatible changes)

<details><summary><i><b>Changelog</b></i></summary><p>

<blockquote>

## [0.5.1](spiraldb/fsst@v0.5.0...v0.5.1) -
2025-03-12

### Other

- Decompress in 8-byte blocks
([#82](spiraldb/fsst#82))
- *(deps)* lock file maintenance
([#83](spiraldb/fsst#83))
- Assert enough room in decoded buffer
([#79](spiraldb/fsst#79))
- *(deps)* update rust crate criterion to v2.9.1
([#80](spiraldb/fsst#80))
- *(deps)* update mozilla-actions/sccache-action action to v0.0.8
([#78](spiraldb/fsst#78))
- *(deps)* lock file maintenance
([#77](spiraldb/fsst#77))
- Add codspeed ([#76](spiraldb/fsst#76))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Goodbai-1206 added a commit to Goodbai-1206/fsst-rs that referenced this pull request Oct 29, 2025
@Goodbai-1206 @github-actions
chore: release v0.5.1 (#72)
d77bd38 
## 🤖 New release

* `fsst-rs`: 0.5.0 -> 0.5.1 (✓ API compatible changes)

<details><summary><i><b>Changelog</b></i></summary><p>

<blockquote>

## [0.5.1](spiraldb/fsst@v0.5.0...v0.5.1) -
2025-03-12

### Other

- Decompress in 8-byte blocks
([#82](spiraldb/fsst#82))
- *(deps)* lock file maintenance
([#83](spiraldb/fsst#83))
- Assert enough room in decoded buffer
([#79](spiraldb/fsst#79))
- *(deps)* update rust crate criterion to v2.9.1
([#80](spiraldb/fsst#80))
- *(deps)* update mozilla-actions/sccache-action action to v0.0.8
([#78](spiraldb/fsst#78))
- *(deps)* lock file maintenance
([#77](spiraldb/fsst#77))
- Add codspeed ([#76](spiraldb/fsst#76))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
akkjs887 added a commit to akkjs887/rt-rsst that referenced this pull request Oct 29, 2025
@akkjs887 @github-actions
chore: release v0.5.1 (#72)
497d030 
## 🤖 New release

* `fsst-rs`: 0.5.0 -> 0.5.1 (✓ API compatible changes)

<details><summary><i><b>Changelog</b></i></summary><p>

<blockquote>

## [0.5.1](spiraldb/fsst@v0.5.0...v0.5.1) -
2025-03-12

### Other

- Decompress in 8-byte blocks
([#82](spiraldb/fsst#82))
- *(deps)* lock file maintenance
([#83](spiraldb/fsst#83))
- Assert enough room in decoded buffer
([#79](spiraldb/fsst#79))
- *(deps)* update rust crate criterion to v2.9.1
([#80](spiraldb/fsst#80))
- *(deps)* update mozilla-actions/sccache-action action to v0.0.8
([#78](spiraldb/fsst#78))
- *(deps)* lock file maintenance
([#77](spiraldb/fsst#77))
- Add codspeed ([#76](spiraldb/fsst#76))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
yelirekhmon added a commit to yelirekhmon/fstrs that referenced this pull request Oct 30, 2025
@yelirekhmon @github-actions
chore: release v0.5.1 (#72)
01dc71f 
## 🤖 New release

* `fsst-rs`: 0.5.0 -> 0.5.1 (✓ API compatible changes)

<details><summary><i><b>Changelog</b></i></summary><p>

<blockquote>

## [0.5.1](spiraldb/fsst@v0.5.0...v0.5.1) -
2025-03-12

### Other

- Decompress in 8-byte blocks
([#82](spiraldb/fsst#82))
- *(deps)* lock file maintenance
([#83](spiraldb/fsst#83))
- Assert enough room in decoded buffer
([#79](spiraldb/fsst#79))
- *(deps)* update rust crate criterion to v2.9.1
([#80](spiraldb/fsst#80))
- *(deps)* update mozilla-actions/sccache-action action to v0.0.8
([#78](spiraldb/fsst#78))
- *(deps)* lock file maintenance
([#77](spiraldb/fsst#77))
- Add codspeed ([#76](spiraldb/fsst#76))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robert3005 robert3005 robert3005 left review comments

@a10y a10y a10y approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gatesn @a10y @robert3005