Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a toolkit of libraries (Ray AIR) for accelerating ML workloads.

Apache Hadoop

Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core.

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

verl: Volcano Engine Reinforcement Learning for LLMs

DLRover: An Automatic Distributed Deep Learning System

Apache Iceberg

Apache Iceberg

A port of Snappy, LZO, LZ4, and Zstandard to Java

Apache Iceberg C++

Spark integrations for working with Lance datasets

Rust implementation of Apache ORC

Open-source simulator for autonomous driving research.

Apache ORC - the smallest, fastest columnar storage for Hadoop workloads

Scalable, redundant, and distributed object store for Apache Hadoop

Apache Hive

Open source Java implementation for Raft consensus protocol.

🚀2.3x faster than MinIO for 4KB object payloads. RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platfor…

A toolkit to run Ray applications on Kubernetes

A JAX-based simulator for autonomous driving research.

Mirror of Apache Toree (Incubating)

Apache Paimon(incubating) is a streaming data lake platform that supports high-speed data ingestion, change data tracking and efficient real-time analytics.

Gluten: Plugin to Double SparkSQL's Performance

Apache DataFusion SQL Query Engine

Arctic is a streaming lake warehouse service open sourced by NetEase

Apache Kyuubi is a distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark

etcd java client

Apache Hadoop Thirdparty

Apache Spark - A unified analytics engine for large-scale data processing

Upserts, Deletes And Incremental Processing on Big Data.