Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

An orchestration platform for the development, production, and observation of data assets.

🧙 Build, run, and manage data pipelines for integrating and transforming data.

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

A system for agentic LLM-powered data processing and ETL

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

Scalable and efficient data transformation framework - backwards compatible with dbt.

🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Implementing best practices for PySpark ETL jobs and applications.

A Python stream processing engine modeled after Yahoo! Pipes

Postgres to Elasticsearch/OpenSearch sync

Enterprise-grade and API-first LLM workspace for unstructured documents, including data extraction, redaction, rights management, prompt playground, and more!

Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton

The Semantic Operating System for AI Apps

Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch

🍁 Sycamore is an LLM-powered search and analytics platform for unstructured data.