data-pipelines repositories
Trending repositories tagged data-pipelines, ranked by trending score. Updated daily.
-
1
apache/airflow
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
GitHub · Python · 47,569 stars · 18,419 forks · +46 stars in 24 hours · Trending score 3.37
-
2
dagster-io/dagster
An orchestration platform for the development, production, and observation of data assets.
GitHub · Python · 16,444 stars · 2,427 forks · +9 stars in 24 hours · Trending score 2.36
-
3
ucbepic/docetl
A system for agentic LLM-powered data processing and ETL
GitHub · Python · 4,259 stars · 471 forks · +2 stars in 24 hours · Trending score 1.58
-
4
bruin-data/bruin
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
GitHub · Go · 1,815 stars · 101 forks · +6 stars in 24 hours · Trending score 1.55
-
5
feldera/feldera
The Feldera Incremental Computation Engine
GitHub · Rust · 2,181 stars · 181 forks · +3 stars in 24 hours · Trending score 1.54
-
6
apache/dolphinscheduler
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
GitHub · Java · 14,558 stars · 5,122 forks · +7 stars in 24 hours · Trending score 1.54
-
7
Unstructured-IO/unstructured
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
GitHub · HTML · 15,759 stars · 1,404 forks · 0 stars in 24 hours · Trending score 1.38
-
8
cubert-hyperspectral/cuvis-ai
Cuvis.AI is an open-source, extensible framework for building modular AI pipelines that process, analyze, and deploy hyperspectral video data in real-time applications.
GitHub · Jupyter Notebook · 20 stars · 0 forks · +1 stars in 24 hours · Trending score 0.68
-
9
meltano/meltano
Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
GitHub · Python · 2,683 stars · 282 forks · 0 stars in 24 hours · Trending score 0.57
-
10
astronomer/agents
AI agent tooling for data engineering workflows.
GitHub · Python · 464 stars · 72 forks · 0 stars in 24 hours · Trending score 0.55
-
11
openivm/openivm
Incremental View Maintenance on DuckDB
GitHub · C++ · 71 stars · 7 forks · 0 stars in 24 hours · Trending score 0.30
-
12
cernodatainc/cernodata
Open-source control layer over PDF parsing that picks the right method, verifies output quality, and falls back automatically when results fall short
GitHub · Python · 24 stars · 6 forks · 0 stars in 24 hours · Trending score 0.30
-
13
debezium/debezium-platform
An opinionated data-centric view of Debezium components. Please log issues at https://github.com/debezium/dbz/issues.
GitHub · TypeScript · 55 stars · 46 forks · 0 stars in 24 hours · Trending score 0.19
-
14
linkedin/Hoptimator
Multi-hop declarative data pipelines
GitHub · Java · 128 stars · 18 forks · 0 stars in 24 hours · Trending score 0.15
-
15
KafScale/platform
Kafka-compatible streaming that keeps your data in S3. Stateless brokers, standard Kafka clients, Apache 2.0.
GitHub · Go · 59 stars · 15 forks · 0 stars in 24 hours · Trending score 0.10
-
16
pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
GitHub · Python · 61,982 stars · 1,704 forks · −8 stars in 24 hours · Trending score 0.10
-
17
eschizoid/kpipe
Composable Kafka consumer library for building modular, testable JVM data pipelines.
GitHub · Java · 58 stars · 7 forks · 0 stars in 24 hours · Trending score 0.10
-
18
DidactHQ/didact
The open core .NET job orchestrator that we've been missing
GitHub · C# · 162 stars · 3 forks · 0 stars in 24 hours · Trending score 0.10
-
19
akshaysharma1088/dataranger
Open-source AI agent that patrols your data, fixes only what it can prove against the source, and sends the rest to a person with the evidence. Works with Snowflake, dbt and DuckDB, runs as a GitHub Action, and speaks MCP.
GitHub · Python · 5 stars · 10 forks · 0 stars in 24 hours · Trending score 0.08
-
20
databricks-solutions/lakeflow_framework
Metadata-driven framework for Databricks Spark Declarative Pipelines. Config-driven, pattern based approach to batch & streaming across the medallion architecture. Deploys via Declarative Automation Bundles. Built for simplicity, extensibility, and alignment with the Databricks product roadmap.
GitHub · Python · 30 stars · 20 forks · 0 stars in 24 hours · Trending score 0.07
-
21
CogStack/CogStack-NiFi
Building data processing pipelines for documents processing with NLP using Apache NiFi and related services
GitHub · Python · 61 stars · 20 forks · 0 stars in 24 hours · Trending score 0.05
-
22
combust/mleap
MLeap: Deploy ML Pipelines to Production
GitHub · Scala · 1,545 stars · 312 forks · 0 stars in 24 hours · Trending score 0.03
-
23
datajoint/datajoint-python
Relational Workflows: where database schemas define executable data pipelines.
GitHub · Python · 197 stars · 101 forks · 0 stars in 24 hours · Trending score 0.00
-
24
wingfoil-io/wingfoil
ultra low latency graph based stream processing framework
GitHub · Rust · 230 stars · 35 forks · 0 stars in 24 hours · Trending score 0.00