data-pipeline repositories

Trending repositories tagged data-pipeline, ranked by trending score. Updated daily.

  1. 1
    datazip-inc/olake-go

    OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.

    GitHub · Go · 2,062 stars · 314 forks · +196 stars in 24 hours · Trending score 4.21

  2. 2
    qiantongtech/qData

    qData is a lightweight, feature-complete, AI-native open-source data platform. It covers data integration, development, modeling, governance, assets and API services, with self-hosted deployment, flexible scaling and commercial use.

    GitHub · Java · 940 stars · 140 forks · +21 stars in 24 hours · Trending score 2.59

  3. 3
    airbytehq/airbyte

    Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

    GitHub · Python · 22,411 stars · 5,550 forks · +5 stars in 24 hours · Trending score 2.32

  4. 4
    dagucloud/dagu

    Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.

    GitHub · Go · 4,635 stars · 406 forks · +18 stars in 24 hours · Trending score 2.27

  5. 5
    superlinked/sie

    Open-source inference server and production cluster for all the models your agent needs.

    GitHub · Python · 3,533 stars · 339 forks · +8 stars in 24 hours · Trending score 1.98

  6. 6
    debezium/debezium

    Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

    GitHub · Java · 13,361 stars · 3,133 forks · +3 stars in 24 hours · Trending score 1.63

  7. 7
    bruin-data/ingestr

    ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

    GitHub · Go · 4,048 stars · 170 forks · +5 stars in 24 hours · Trending score 1.34

  8. 8
    yuyuejia/datay

    DataY 是一套轻量、高性能的数据平台,一个 JAR、自研流批一体处理引擎,自研分布式调度框架,在 2 核 4G 的机器上 就能跑通 数据集成 → 数据建模 → 数据开发 → 任务调度 → 数据应用 的完整链路。

    GitHub · Java · 50 stars · 23 forks · +2 stars in 24 hours · Trending score 1.28

  9. 9
    simonesiega/european-tech-opportunities-2027

    Your open-source hub for 2027 tech internships and New Grad opportunities across Europe.

    GitHub · Python · 55 stars · 0 forks · +5 stars in 24 hours · Trending score 1.26

  10. 10
    apache/shardingsphere

    Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

    GitHub · Java · 20,831 stars · 6,911 forks · +2 stars in 24 hours · Trending score 1.18

  11. 11
    galaxy-io/filament

    Pluggable data replication with checkpointing, batching, and integrity events

    GitHub · Go · 335 stars · 47 forks · 0 stars in 24 hours · Trending score 1.06

  12. 12
    clidey/whodb

    Where data access meets operational intelligence

    GitHub · Go · 5,061 stars · 243 forks · +2 stars in 24 hours · Trending score 1.04

  13. 13
    datajuicer/data-juicer

    Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

    GitHub · Python · 7,263 stars · 456 forks · 0 stars in 24 hours · Trending score 0.93

  14. 14
    vaquarkhan/data-engineering-agent-skills

    Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.

    GitHub · Python · 65 stars · 9 forks · 0 stars in 24 hours · Trending score 0.80

  15. 15
    estuary/flow

    🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

    GitHub · Rust · 992 stars · 101 forks · +1 stars in 24 hours · Trending score 0.73

  16. 16
    Hebbian-Robotics/hflow

    SDK for robotics teams to verify the quality of their data used for AI model training.

    GitHub · Python · 310 stars · 171 forks · 0 stars in 24 hours · Trending score 0.68

  17. 17
    analyticsdurgesh/DBT-Project-Pipeline

    Public Snowflake and dbt pipeline transforming Blinkit retail sales data into staging and mart analytics models.

    GitHub · Python · 25 stars · 4 forks · +1 stars in 24 hours · Trending score 0.68

  18. 18
    iflytek/dolphin-mcp-pilot

    Production-ready MCP server for Apache DolphinScheduler (小海豚) — lets AI agents build, schedule, run and recover workflows

    GitHub · Python · 24 stars · 22 forks · 0 stars in 24 hours · Trending score 0.65

  19. 19
    DataSQRL/sqrl

    Agentic data engineering for operational data: real-time pipelines, data APIs, and data products on Kafka, Flink, Postgres, and Iceberg.

    GitHub · Java · 237 stars · 32 forks · +1 stars in 24 hours · Trending score 0.63

  20. 20
    apache/flink-cdc

    Flink CDC is a streaming data integration tool

    GitHub · Java · 6,495 stars · 2,209 forks · +1 stars in 24 hours · Trending score 0.56

  21. 21
    SomethingObvious/tiktok-activity-data-scraper

    Turns your TikTok data export into an HTML report of the creators, hashtags and topics you watch most. It looks up every video you liked, sorts hashtags into about 600 topics with an editable 50,000-tag table, and saves every post to JSON and CSV for your own analysis. Runs locally in Python with your own cookies.

    GitHub · Python · 11 stars · 0 forks · +1 stars in 24 hours · Trending score 0.56

  22. 22
    faucet-hq/faucet-stream

    The fast, config-driven way to move data in Rust — a native CLI and an embeddable Rust ETL library

    GitHub · Rust · 23 stars · 8 forks · +1 stars in 24 hours · Trending score 0.54

  23. 23
    slothflowlabs/duckle

    Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 446 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

    GitHub · Rust · 1,557 stars · 121 forks · −16 stars in 24 hours · Trending score 0.41

  24. 24
    rudderlabs/rudder-server

    Privacy and Security focused Segment-alternative, in Golang and React

    GitHub · Go · 4,520 stars · 84 forks · 0 stars in 24 hours · Trending score 0.40