etl-pipeline repositories

Trending repositories tagged etl-pipeline, ranked by trending score. Updated daily.

  1. 1
    DataWithBaraa/sql-data-warehouse-project

    Build a data warehouse in SQL Server from two source systems, a CRM and an ERP, through bronze, silver and gold. The files and scripts of the free full project.

    GitHub · TSQL · 1,089 stars · 751 forks · +27 stars in 24 hours · Trending score 2.27

  2. 2
    qiantongtech/qData

    qData is a lightweight, feature-complete, AI-native open-source data platform. It covers data integration, development, modeling, governance, assets and API services, with self-hosted deployment, flexible scaling and commercial use.

    GitHub · Java · 954 stars · 140 forks · +10 stars in 24 hours · Trending score 2.05

  3. 3
    apache/hamilton

    Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

    GitHub · Jupyter Notebook · 2,631 stars · 235 forks · +6 stars in 24 hours · Trending score 1.60

  4. 4
    yuyuejia/datay

    DataY 是一套轻量、高性能的数据平台,一个 JAR、自研流批一体处理引擎,自研分布式调度框架,在 2 核 4G 的机器上 就能跑通 数据集成 → 数据建模 → 数据开发 → 任务调度 → 数据应用 的完整链路。

    GitHub · Java · 50 stars · 23 forks · +1 stars in 24 hours · Trending score 1.07

  5. 5
    risingwavelabs/risingwave

    Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

    GitHub · Rust · 9,425 stars · 872 forks · 0 stars in 24 hours · Trending score 0.88

  6. 6
    Open-Source-Legal/OpenContracts

    The open document intelligence platform for builders and hackers - DMS for the agentic world

    GitHub · Python · 1,550 stars · 201 forks · 0 stars in 24 hours · Trending score 0.73

  7. 7
    AgentSwarms-fyi/agentswarms

    Unified Agentic AI and Data Platform

    GitHub · TypeScript · 302 stars · 67 forks · 0 stars in 24 hours · Trending score 0.60

  8. 8
    Edwardvaneechoud/Flowfile

    Flowfile is a visual ETL tool and Python library combining drag-and-drop workflows with Polars dataframes. Build data pipelines visually, define flows programmatically with a Polars-like API, and export to standalone Python code. Perfect for fast, intuitive data processing from development to production.

    GitHub · Python · 413 stars · 34 forks · 0 stars in 24 hours · Trending score 0.42

  9. 9
    ebonnal/streamable

    sync/async iterable streams for Python (concurrency, throttling, buffering, ...)

    GitHub · Python · 337 stars · 8 forks · 0 stars in 24 hours · Trending score 0.33

  10. 10
    goldsky-io/streamling

    Data streaming runtime focused on performance, consistency, and extensibility. Write plugins in Rust or WASM and process data with data guarantees.

    GitHub · Rust · 111 stars · 9 forks · 0 stars in 24 hours · Trending score 0.30

  11. 11
    MTSWebServices/onetl

    One ETL tool to rule them all

    GitHub · Python · 91 stars · 8 forks · 0 stars in 24 hours · Trending score 0.20

  12. 12
    bkc39/rkt-polars

    Racket bindings to polars

    GitHub · Racket · 11 stars · 0 forks · 0 stars in 24 hours · Trending score 0.19

  13. 13
    techascent/tech.ml.dataset

    A Clojure high performance data processing system

    GitHub · Clojure · 764 stars · 34 forks · 0 stars in 24 hours · Trending score 0.14

  14. 14
    koralium/flowtide

    High-performance streaming SQL query engine designed for real-time data processing. Use cases include event-driven architectures, ETL pipelines, and modern data-intensive applications.

    GitHub · C# · 110 stars · 8 forks · 0 stars in 24 hours · Trending score 0.10

  15. 15
    opencrawling/opencrawling

    Reference Java & Spring implementation of the Open Ingestion Standard (OIS). It provides a secure, decoupled, and vendor-neutral enterprise data integration platform leveraging modern Java 25 features, Spring Boot, Spring AI and vector search infrastructure to orchestrate data flows

    GitHub · Java · 29 stars · 5 forks · 0 stars in 24 hours · Trending score 0.10

  16. 16
    LAMP-LUCAS/AutoSINAPI

    Solução open source para simplificar o acesso, tratamento e gestão dos dados do SINAPI (Sistema Nacional de Pesquisa de Custos e Índices da Construção Civil)

    GitHub · Python · 12 stars · 5 forks · 0 stars in 24 hours · Trending score 0.10

  17. 17
    keboola/mcp-server

    Model Context Protocol (MCP) Server for the Keboola Platform

    GitHub · Python · 84 stars · 23 forks · −1 stars in 24 hours · Trending score 0.09

  18. 18
    conductor-oss/python-sdk

    Conductor OSS SDK for Python programming language

    GitHub · Python · 104 stars · 46 forks · 0 stars in 24 hours · Trending score 0.09

  19. 19
    ciaren-labs/Ciaren

    Open-source, local-first visual ETL and data pipeline builder. Drag-and-drop data prep and ML workflows, preview every step, schedule runs, and export clean pandas or Polars Python code. Self-hosted, no lock-in.

    GitHub · Python · 37 stars · 12 forks · 0 stars in 24 hours · Trending score 0.08

  20. 20
    wp-labs/warp-parse

    Focusing on building industry-leading ETL engines.

    GitHub · Rust · 186 stars · 13 forks · 0 stars in 24 hours · Trending score 0.08

  21. 21
    CogStack/CogStack-NiFi

    Building data processing pipelines for documents processing with NLP using Apache NiFi and related services

    GitHub · Python · 61 stars · 20 forks · 0 stars in 24 hours · Trending score 0.07

  22. 22
    marciok/gust

    A Task-Based Workflow Orchestration Engine

    GitHub · Elixir · 392 stars · 25 forks · 0 stars in 24 hours · Trending score 0.07

  23. 23
    realdatadriven/etlx

    ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.

    GitHub · Go · 57 stars · 3 forks · 0 stars in 24 hours · Trending score 0.03

  24. 24
    swadhinbiswas/eu-air-traffic

    Live and historical European air traffic: a VPS collector streams ADS-B, OpenSky movements, schedules and weather through Kafka into a Bronze/Silver lake, a dbt warehouse and a read-only serving copy behind a public dashboard. Budget-aware polling and atomic publishes keep it correct while unattended.

    GitHub · Python · 6 stars · 0 forks · 0 stars in 24 hours · Trending score 0.02