data-pipeline repositories
Trending repositories tagged data-pipeline, ranked by trending score. Updated daily.
-
1
datazip-inc/olake-go
OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
GitHub · Go · 2,062 stars · 314 forks · +196 stars in 24 hours · Trending score 4.21
-
2
qiantongtech/qData
qData is a lightweight, feature-complete, AI-native open-source data platform. It covers data integration, development, modeling, governance, assets and API services, with self-hosted deployment, flexible scaling and commercial use.
GitHub · Java · 940 stars · 140 forks · +21 stars in 24 hours · Trending score 2.59
-
3
airbytehq/airbyte
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
GitHub · Python · 22,411 stars · 5,550 forks · +5 stars in 24 hours · Trending score 2.32
-
4
dagucloud/dagu
Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.
GitHub · Go · 4,635 stars · 406 forks · +18 stars in 24 hours · Trending score 2.27
-
5
superlinked/sie
Open-source inference server and production cluster for all the models your agent needs.
GitHub · Python · 3,533 stars · 339 forks · +8 stars in 24 hours · Trending score 1.98
-
6
debezium/debezium
Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.
GitHub · Java · 13,361 stars · 3,133 forks · +3 stars in 24 hours · Trending score 1.63
-
7
bruin-data/ingestr
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
GitHub · Go · 4,048 stars · 170 forks · +5 stars in 24 hours · Trending score 1.34
-
8
yuyuejia/datay
DataY 是一套轻量、高性能的数据平台,一个 JAR、自研流批一体处理引擎,自研分布式调度框架,在 2 核 4G 的机器上 就能跑通 数据集成 → 数据建模 → 数据开发 → 任务调度 → 数据应用 的完整链路。
GitHub · Java · 50 stars · 23 forks · +2 stars in 24 hours · Trending score 1.28
-
9
simonesiega/european-tech-opportunities-2027
Your open-source hub for 2027 tech internships and New Grad opportunities across Europe.
GitHub · Python · 55 stars · 0 forks · +5 stars in 24 hours · Trending score 1.26
-
10
apache/shardingsphere
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
GitHub · Java · 20,831 stars · 6,911 forks · +2 stars in 24 hours · Trending score 1.18
-
11
galaxy-io/filament
Pluggable data replication with checkpointing, batching, and integrity events
GitHub · Go · 335 stars · 47 forks · 0 stars in 24 hours · Trending score 1.06
-
12
clidey/whodb
Where data access meets operational intelligence
GitHub · Go · 5,061 stars · 243 forks · +2 stars in 24 hours · Trending score 1.04
-
13
datajuicer/data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
GitHub · Python · 7,263 stars · 456 forks · 0 stars in 24 hours · Trending score 0.93
-
14
vaquarkhan/data-engineering-agent-skills
Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.
GitHub · Python · 65 stars · 9 forks · 0 stars in 24 hours · Trending score 0.80
-
15
estuary/flow
🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊
GitHub · Rust · 992 stars · 101 forks · +1 stars in 24 hours · Trending score 0.73
-
16
Hebbian-Robotics/hflow
SDK for robotics teams to verify the quality of their data used for AI model training.
GitHub · Python · 310 stars · 171 forks · 0 stars in 24 hours · Trending score 0.68
-
17
analyticsdurgesh/DBT-Project-Pipeline
Public Snowflake and dbt pipeline transforming Blinkit retail sales data into staging and mart analytics models.
GitHub · Python · 25 stars · 4 forks · +1 stars in 24 hours · Trending score 0.68
-
18
iflytek/dolphin-mcp-pilot
Production-ready MCP server for Apache DolphinScheduler (小海豚) — lets AI agents build, schedule, run and recover workflows
GitHub · Python · 24 stars · 22 forks · 0 stars in 24 hours · Trending score 0.65
-
19
DataSQRL/sqrl
Agentic data engineering for operational data: real-time pipelines, data APIs, and data products on Kafka, Flink, Postgres, and Iceberg.
GitHub · Java · 237 stars · 32 forks · +1 stars in 24 hours · Trending score 0.63
-
20
apache/flink-cdc
Flink CDC is a streaming data integration tool
GitHub · Java · 6,495 stars · 2,209 forks · +1 stars in 24 hours · Trending score 0.56
-
21
SomethingObvious/tiktok-activity-data-scraper
Turns your TikTok data export into an HTML report of the creators, hashtags and topics you watch most. It looks up every video you liked, sorts hashtags into about 600 topics with an editable 50,000-tag table, and saves every post to JSON and CSV for your own analysis. Runs locally in Python with your own cookies.
GitHub · Python · 11 stars · 0 forks · +1 stars in 24 hours · Trending score 0.56
-
22
faucet-hq/faucet-stream
The fast, config-driven way to move data in Rust — a native CLI and an embeddable Rust ETL library
GitHub · Rust · 23 stars · 8 forks · +1 stars in 24 hours · Trending score 0.54
-
23
slothflowlabs/duckle
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 446 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
GitHub · Rust · 1,557 stars · 121 forks · −16 stars in 24 hours · Trending score 0.41
-
24
rudderlabs/rudder-server
Privacy and Security focused Segment-alternative, in Golang and React
GitHub · Go · 4,520 stars · 84 forks · 0 stars in 24 hours · Trending score 0.40