📦 Normy

Ultra-fast, zero-copy text normalization — built for Rust NLP pipelines & tokenizers. Flexible enough for any high-throughput multilingual text processing (search, logs, APIs, data pipelines, …)

Normy delivers extreme performance through automatic iterator fusion and precise early-exit checks, while respecting language-specific rules (e.g., Turkish dotted/dotless I, German ß folding).

• Zero-copy → Immediately returns without allocation when input needs no changes.
• Automatic fusion → Can fuse eligible stages (>1 fusable stage) into a single pass for better cache locality.
• Locale-accurate → Built-in rules for correctness across scripts.
• Format-aware → Clean HTML/Markdown while preserving content.

Why Normy?

Traditional normalizers allocate on every call—even for clean text. Normy eliminates this overhead:

• On already-normalized text (common in production streams): up to 51× higher throughput than HuggingFace tokenizers normalizers due to true zero-copy.
• On text requiring transformation: 3.7–4.1× faster through fusion and optimized stages.

🏆 Performance Comparison

Measured against HuggingFace tokenizers normalizers on 64 KiB inputs (200 samples each).

Complex Pipeline Bert-like (Chinese + Strip + Whitespace + NFD + Diacritics + Lowercase)

Already Normalized Text

Needs Transform

Simple Pipeline (French + Lowercase + Transliterate)

Already Normalized Text

With Accents/Diacritics

Numbers represent geometric mean over 200 samples. Hardware, OS, and input distribution can affect results. See /benches/comparison_tokenizers_bench.rs for reproducible results.

💾 Installation

Add Normy to your project:

cargo add normy

⚡ Quickstart

Normy uses a fluent builder pattern with automatic fusion detection.

use std::error::Error;

use normy::{
    COLLAPSE_WHITESPACE_UNICODE, CaseFold, DEU, FRA, JPN, LowerCase, Normy, RemoveDiacritics, SegmentWords,
    TUR, Transliterate, UnifyWidth, ZHO,
};

fn main() -> Result<(), Box<dyn Error>> {
    // ────────────────────────────────────────────────────────────────
    // TURKISH (Turkic) – famous for its dotted/dotless I distinction
    // ────────────────────────────────────────────────────────────────
    let tur = Normy::builder()
        .lang(TUR)
        .add_stage(LowerCase) // Critical: İ → i, I → ı
        .build();

    println!(
        "Turkish : {}",
        tur.normalize("KIZILIRMAK NEHRİ TÜRKİYE'NİN EN UZUN NEHRİDİR.")?
    );
    // → kızılırmak nehri türkiye'nin en uzun nehridir.

    // ────────────────────────────────────────────────────────────────
    // GERMAN (Germany/Austria/Switzerland) – ß and umlaut handling
    // ────────────────────────────────────────────────────────────────
    let deu = Normy::builder()
        .lang(DEU)
        .add_stage(CaseFold) // ß → ss
        .add_stage(Transliterate) // Ä → ae, Ö → oe, Ü → ue
        .build();

    println!(
        "German  : {}",
        deu.normalize("Grüße aus München! Die Straße ist sehr schön.")?
    );
    // → gruesse aus muenchen! die strasse ist sehr schoen.

    // ────────────────────────────────────────────────────────────────
    // FRENCH (France/Belgium/Canada/etc.) – classic accented text
    // ────────────────────────────────────────────────────────────────
    let fra = Normy::builder()
        .lang(FRA)
        .add_stage(CaseFold)
        .add_stage(RemoveDiacritics) // é → e, ç → c, etc.
        .build();

    println!(
        "French  : {}",
        fra.normalize("Bonjour ! J'adore le café et les croissants à Paris.")?
    );
    // → bonjour ! j'adore le cafe et les croissants a paris.

    // ────────────────────────────────────────────────────────────────
    // CHINESE (Simplified – China) – fullwidth & word segmentation
    // ────────────────────────────────────────────────────────────────
    let zho = Normy::builder()
        .lang(ZHO)
        .add_stage(UnifyWidth)
        .add_stage(COLLAPSE_WHITESPACE_UNICODE)
        .add_stage(SegmentWords) // unigram segmentation
        .build();

    println!(
        "Chinese : {}",
        zho.normalize("北京的秋天特别美丽，长城非常壮观！")?
    );
    // → 北 京 的 秋 天 特 别 美 丽 , 长 城 非 常 壮 观 !

    // ────────────────────────────────────────────────────────────────
    // JAPANESE (Japan) – script transitions + width unification
    // ────────────────────────────────────────────────────────────────
    let jpn = Normy::builder()
        .lang(JPN)
        .add_stage(UnifyWidth)
        .add_stage(COLLAPSE_WHITESPACE_UNICODE)
        .add_stage(SegmentWords) // script boundary segmentation
        .build();

    println!(
        "Japanese: {}",
        jpn.normalize("東京は本当に素晴らしい街です！桜がとてもきれい。")?
    );
    // → 東京は本当に素晴らしい街です ! 桜がとてもきれい 。

    Ok(())
}

When text is already normalized, Normy returns Cow::Borrowed — zero allocation.

✨ Features

💼 Available Normalization Stages

Normy provides a rich set of composable, high-performance normalization stages.
Most stages support static iterator fusion for maximum speed (single-pass, zero-copy when possible).

Key notes

• Fusion = static single-pass iterator fusion (zero-copy + minimal allocation when conditions met)
• Non-fusable stages (NFC/NFD/NFKC/NFKD, StripHtml, StripMarkdown) use optimized batch processing and should usually be placed early in the pipeline

🆎 Supported Languages

Features Key:

• Word Segmentation: Automatic boundary detection for non-space-delimited scripts
• CJK Unigram: Character-level tokenization for Chinese ideographs
• Transliteration: Script→Latin conversion (e.g., Cyrillic, ligatures)
• Diacritics: Intelligent spacing/combining mark handling

📖 Documentation

• Full API docs: docs.rs/normy
• Linguistic rules: LINGUISTIC_POLICY.md
• Pipeline guidelines: PIPELINE_GUIDELINES.md
• Examples are in the examples/ directory
• Generate local docs:

cargo doc --open

🤝 Contributing

Contributions are very welcome! See CONTRIBUTING.md

📜 License

Dual-licensed under MIT or Apache-2.0, at your option.

See LICENSE-MIT and LICENSE-APACHE.

Normy — Ultra-fast, linguistically correct normalization — the next-generation layer for Rust NLP & tokenizers