Veloxx: Ultra-High Performance Data Processing & Analytics Library

🚀 v0.4.0 Released! Major performance overhaul with SIMD acceleration, Pivot, and Outer Join support.

Veloxx is a blazing-fast, ultra-lightweight data processing and analytics library in Rust, with seamless bindings for Python and WebAssembly. Built from the ground up for maximum performance, featuring advanced SIMD acceleration, memory optimization, and parallel processing that often outperforms industry leaders.

🏆 Performance Highlights

• SIMD Acceleration: Vectorized aggregation (sum, mean, min, max) now 30-90x faster than scalar implementations.
• Parallel Processing: Hybrid execution strategy using Rayon for large datasets (>500k rows), achieving near-linear scaling.
• Optimized I/O: Multi-threaded memory-mapped CSV reading and zero-copy Parquet integration.
• Lazy Evaluation: Refined Query Optimizer with predicate pushdown for efficient filtering.

✨ New Features (v0.4.0)

• Pivot: Reshape DataFrames from long to wide format with aggregation.
• Outer Join: Full support for Left, Right, Inner, and Outer joins.
• Deterministic Columns: Refactored internal storage to guarantee consistent column ordering.
• Python Bindings: Updated PyDataFrame with pivot and outer_join support.

✨ Project Links

• 🦀 Rust crate (crates.io)
• 🐍 Python package (PyPI)
• 📦 JavaScript package (npm)
• 🌐 GitHub
• 📖 Online Documentation

🧩 Core Principles & Design Goals

• 🚀 Performance First: Advanced SIMD, parallel processing, cache-optimized algorithms
• 🪶 Lightweight: Minimal dependencies, optimized memory footprint
• 🦺 Safety & Reliability: Memory-safe Rust, comprehensive testing
• 🧑‍💻 Developer Experience: Intuitive APIs, excellent documentation
• 🔧 Production Ready: Zero-warning compilation, extensive benchmarking

🚩 Key Features

Core Data Structures

• DataFrame and Series for lightning-fast tabular data processing
• SIMD-optimized operations with AVX2/NEON acceleration
• Memory-efficient storage with advanced compression

High-Performance Operations

• 🚀 Ultra-fast analytics: filtering, joining, grouping, aggregation, pivoting
• 📊 Advanced statistics: correlation, regression, time-series analysis
• Parallel processing: Multi-threaded execution with work-stealing
• 🧮 Vectorized math: SIMD-accelerated arithmetic operations

Advanced I/O & Integration

• 📂 Multiple formats: CSV, JSON, Parquet support
• 🔌 Database connectivity: SQLite, PostgreSQL, MySQL
• 🌊 Streaming operations: Memory-efficient large dataset processing
• ⚡ Async I/O: Non-blocking file and network operations

Data Quality & ML

• 🧹 Data cleaning: Automated outlier detection, validation
• 🤖 Machine learning: Linear/logistic regression, clustering, preprocessing
• 📈 Visualization: Charts, plots, statistical graphics
• 🔍 Data profiling: Schema inference, quality metrics

Multi-Language Support

• 🦀 Rust: Native, zero-cost abstractions
• Python: PyO3 bindings with NumPy integration
• 🌐 WebAssembly: Browser and Node.js support
• 📦 Easy installation: Available on crates.io, PyPI, npm

⚡ Quick Start

Rust

[dependencies]
veloxx = "0.4.0"

use veloxx::dataframe::DataFrame;
use veloxx::series::Series;

let df = DataFrame::new_from_csv("data.csv")?;
let filtered = df.filter(&your_condition)?;
let grouped = df.group_by(vec!["category"]).agg(vec![("amount", "sum")])?;

Python

import veloxx

df = veloxx.PyDataFrame({"name": veloxx.PySeries("name", ["Alice", "Bob"])})
filtered = df.filter(...)
pivoted = df.pivot(values="score", index=["name"], columns="subject", agg_fn="mean")

JavaScript/Wasm

const veloxx = require("veloxx");
const df = new veloxx.WasmDataFrame({name: ["Alice", "Bob"]});
const filtered = df.filter(...);

🛠️ Feature Flags

Enable only what you need:

• advanced_io – Parquet, databases, async
• data_quality – Schema checks, anomaly detection
• window_functions – Window analytics
• visualization – Charting
• ml – Machine learning
• python – Python bindings
• wasm – WebAssembly

📚 Documentation

• Getting Started Guide
• API Guide
• Rust API Docs
• Python API Docs
• JavaScript/Wasm Docs
• Online Docs

🧑‍💻 Examples

Run ready-made examples:

cargo run --example basic_dataframe_operations
cargo run --example advanced_io --features advanced_io
# ... more in the examples/ folder

🤝 Contributing

See CONTRIBUTING.md for guidelines. Please review our Code of Conduct.

💬 Support

• Issues: https://github.com/Conqxeror/veloxx/issues
• Discussions: https://github.com/Conqxeror/veloxx/discussions
• Documentation: https://conqxeror.github.io/veloxx/

📝 License

MIT License. See LICENSE.