OffsetScan

A standalone, native, corpus-scale companion to OffsetInspect.

OffsetInspect's static-triage helpers (Get-OffsetPEInfo, Get-OffsetEntropy, Get-OffsetString, Get-OffsetIOC) are cross-platform PowerShell and work great for single-file, interactive analysis. OffsetScan exists for the other end of the workload: thousands of files, where PowerShell's per-file overhead adds up and a parallel, no-GC native core pays for itself.

OffsetScan does not touch AMSI or Microsoft Defender - that stays exactly where it belongs, as Windows-only functionality in OffsetInspect. OffsetScan only does the read-only, cross-platform static-analysis layer: PE parsing, entropy, string extraction, hashing, and (optionally) YARA matching.

Design contract

Every OffsetScan output struct (src/schema.rs) mirrors the equivalent OffsetInspect PowerShell object field-for-field, so the two tools are interchangeable at the JSON layer:

OffsetScan (Rust)	OffsetInspect (PowerShell) equivalent
offsetscan pe	Get-OffsetPEInfo
offsetscan entropy	Get-OffsetEntropy
offsetscan strings	Get-OffsetString
offsetscan ioc	Get-OffsetIOC
offsetscan yara (feature-gated)	Invoke-OffsetYaraScan

The struct definitions have been verified field-for-field against the authoritative docs/OUTPUT-SCHEMA.md and the real OffsetInspect 3.x objects, and cross-checked at runtime: offsetscan ioc and Get-OffsetIOC produce identical panels for the same file - including the imphash. The serialized field names (MD5/SHA1/SHA256/IsPE/IsPE32Plus) are locked by unit tests so the interchange contract can't silently drift.

Two subcommands deliberately sit outside this contract - neither has an OffsetInspect equivalent, so they live only here: offsetscan cluster (TLSH similarity clustering, inherently corpus-scale) and offsetscan rich (PE Rich-header build-toolchain fingerprint). See Similarity clustering and Rich header.

Verified parity

Both engines produce the same IOC panel for a file - captured at runtime against the same Windows system DLL.

offsetscan ioc C:\Windows\System32\kernel32.dll:

[
  {
    "File": "C:/Windows/System32/kernel32.dll",
    "FileSize": 836232,
    "MD5": "46e3ab50afcb6d871b70676f562e01ce",
    "SHA1": "732af3ed087e2033d7e7ccaa0f4498deb2e46e8c",
    "SHA256": "26410f4948e0ed66936880596e2b5a59efce481a7c97086049b385f00325c341",
    "OverallEntropy": 6.361191,
    "HighEntropyWindows": 25,
    "PrintableStringCount": 5665,
    "IsPE": true,
    "Machine": "x64 (AMD64)",
    "ImpHash": "a6c6d5a8f6e13c556e2c3fbc4a3dc407",
    "ImportedDllCount": 104,
    "HasOverlay": true,
    "OverlaySize": 17032
  }
]

Get-OffsetIOC C:\Windows\System32\kernel32.dll | ConvertTo-Json:

{
  "File": "C:\\Windows\\System32\\kernel32.dll",
  "FileSize": 836232,
  "MD5": "46e3ab50afcb6d871b70676f562e01ce",
  "SHA1": "732af3ed087e2033d7e7ccaa0f4498deb2e46e8c",
  "SHA256": "26410f4948e0ed66936880596e2b5a59efce481a7c97086049b385f00325c341",
  "OverallEntropy": 6.361191,
  "HighEntropyWindows": 25,
  "PrintableStringCount": 5665,
  "IsPE": true,
  "Machine": "x64 (AMD64)",
  "ImpHash": "a6c6d5a8f6e13c556e2c3fbc4a3dc407",
  "ImportedDllCount": 104,
  "HasOverlay": true,
  "OverlaySize": 17032
}

Every field value matches, including the imphash and entropy to six decimal places. The only differences are cosmetic serialization: OffsetScan always wraps results in an array (its JSON-mode convention, even for a single file) and normalizes path separators to /, while ConvertTo-Json emits a bare object with Windows \ paths.

String extraction is compared as a set, not just a count. For ntdll.dll (2,517,928 bytes), offsetscan strings and Get-OffsetString both return 32,506 hits that agree exactly on offset, encoding, and value - zero entries unique to either engine.

One caveat when reproducing this: Get-OffsetString reads in 1 MiB windows by default and a string straddling a window seam is split in two, which inflates PowerShell's count by one per affected seam on files larger than the window. That is an artifact of the windowed read, not a disagreement about content - pass a -WindowSize large enough to hold the file to compare like for like. OffsetScan reads the whole file and is unaffected.

Build

cargo build --release
# With optional YARA support (requires the YARA engine installed):
cargo build --release --features yara-scan

Usage

offsetscan pe ./sample.exe
offsetscan pe ./sample.exe --offset 0x5F85   # map a byte offset to its PE section
offsetscan entropy ./payload.bin --window 256 --high-threshold 7.2
offsetscan strings ./sample.bin --min-length 6
offsetscan ioc ./sample.exe

# Corpus mode (any subcommand):
offsetscan ioc ./samples --recurse

# Streaming output for large corpora - one compact JSON object per line,
# emitted as each file finishes so peak memory stays flat:
offsetscan ioc ./samples --recurse --ndjson

# Flat CSV for spreadsheets/SIEM (ioc only; columns match the JSON field names):
offsetscan ioc ./samples --recurse --csv > iocs.csv

# Similarity clustering - group variants/repacks/near-duplicates across a corpus:
offsetscan cluster ./samples --recurse
offsetscan cluster ./samples --recurse --threshold 40   # stricter (fewer, tighter clusters)

# Rich-header build-toolchain fingerprint (RichHash + comp.id records):
offsetscan rich ./sample.exe
offsetscan rich ./samples --recurse --ndjson

By default all commands emit a pretty-printed JSON array to stdout, matching OffsetInspect's JSON-mode convention (always an array, even for one result). Add --ndjson for newline-delimited JSON - pipe-friendly and constant-memory over hundreds of thousands of files. Add --csv (the ioc subcommand only, as the others produce nested data) for a flat header + one-row-per-file table whose columns match the JSON field names - ready for Excel or a SIEM import.

Consuming from PowerShell

OffsetInspect 3.1.0+ ingests OffsetScan's IOC JSON directly, so a corpus report runs off the native engine instead of re-scanning each file in PowerShell:

offsetscan ioc ./samples --recurse > ./ioc.json
$results | Export-OffsetThreatReport -Path ./engagement.md -IocJsonPath ./ioc.json

Because the JSON shape matches Get-OffsetIOC field-for-field, any consumer of that shape accepts OffsetScan's output as a drop-in, faster-at-scale alternative.

Similarity clustering

offsetscan cluster computes a canonical TLSH locality-sensitive digest for each file and groups files whose pairwise TLSH distance is within --threshold (default 70; lower = stricter, 0 = identical) into single-linkage clusters. It answers the question single-file inspection can't: which of these thousands of samples are variants of one another? - repacks, malware families, near-duplicates. A modified copy has a different SHA-256, so exact-hash dedup misses it; TLSH still places it beside its original.

One record per file:

[
  { "File": "a.bin", "FileSize": 836232, "SHA256": "26410f49…",
    "Tlsh": "T187056B2E…", "ClusterId": 1, "Warnings": [] },
  { "File": "b_modified.bin", "FileSize": 836232, "SHA256": "06b26733…",
    "Tlsh": "T187056A2E…", "ClusterId": 1, "Warnings": [] },
  { "File": "unrelated.bin", "FileSize": 1881576, "SHA256": "6fcd8218…",
    "Tlsh": "T1B8956C5D…", "ClusterId": null, "Warnings": [] }
]

ClusterId is shared across near-duplicates and null for a singleton. Files too small or too uniform for TLSH (and unreadable files) come back with Tlsh: null and a Warnings entry rather than failing the run. Digests carry the T1 version prefix, so they compare directly with other canonical TLSH tooling. Clustering is deterministic - cluster numbering is ordered by each cluster's smallest member path, so output doesn't depend on input order or thread timing.

Notes: cluster is a whole-corpus operation (every file is hashed before ids are assigned), so unlike the other subcommands it isn't streamed as files finish; --ndjson still emits one record per line, and --csv isn't supported for it. Pairwise comparison is O(n²) - sized for the few-thousand-file corpora this targets. It depends only on the pure-Rust tlsh2 crate, so cargo install offsetscan needs no system prerequisites (unlike the yara-scan feature).

Rich header

offsetscan rich parses the undocumented Rich header the Microsoft linker writes between the DOS stub and the PE header, and reports a build-toolchain fingerprint. The header lists @comp.id records - which build tool (ProductId), which build (BuildId), and how many objects it contributed (Count) - so two samples compiled in the same environment share it even when little else lines up. Useful for clustering and attribution alongside imphash and TLSH.

[
  { "File": "sample.exe", "FileSize": 836232, "HasRichHeader": true,
    "RichHash": "1f67254359760775d87ef504d2f80fb1", "XorKey": "0x54B34DDD",
    "EntryCount": 10,
    "Entries": [ { "ProductId": 257, "BuildId": 33145, "Count": 4 }, … ],
    "Warnings": [] }
]

RichHash is the MD5 of the decrypted Rich array (the DanS marker, its padding, and every comp.id/count record - pefile's RICH_HEADER.clear_data). It deliberately excludes the per-file XOR key and the Rich/checksum trailer, so it is invariant across binaries built with the same toolchain, which is what makes it a clustering key. It is verified byte-identical to pefile: md5(pe.RICH_HEADER.clear_data) matched OffsetScan's RichHash on 118/118 System32 DLLs that carry a Rich header. A binary without one (non-MSVC or stripped) comes back with HasRichHeader: false and no error; a stray Rich byte sequence with no valid DanS behind it is skipped in favour of the genuine header.

What's intentionally NOT here

• AMSI / Microsoft Defender scanning - stays in OffsetInspect (Windows-only, needs the actual providers).
• Detection-boundary bisection search - that's a stateful, provider-driven workflow (Invoke-OffsetThreatScan), not a stateless corpus pass.
• ClamAV integration - clamscan process-spawning has no real parallel-corpus benefit from a Rust rewrite; left in PowerShell.

Status

Validated against OffsetInspect and covered by a unit-test suite (Shannon entropy vectors, string offsets, PE helpers, and the parity-critical schema field names) that runs in CI on Linux and Windows. resource_size is computed from the PE resource data directory.

The yara-scan feature adds an offsetscan yara subcommand whose records (File/Rule/StringId/Offset/OffsetHex/Data) are verified identical to Invoke-OffsetYaraScan's on the same rules and sample. It is feature-gated and off by default - building it needs a C toolchain and libclang (the yara crate compiles a vendored libyara via bindgen), so the default binary (and everything on crates.io / the release page) does not include it. A CI job builds and tests the feature on Linux (installing libclang for bindgen). Non-printable match bytes are decoded lossily into Data, which can differ from the YARA CLI's textual rendering for binary matches.

# needs: cargo build --release --features yara-scan
offsetscan yara ./sample.bin --rules ./rules/malware.yar
offsetscan yara ./corpus --recurse --rules ./a.yar --rules ./b.yar