etl-pipeline repositories
Trending repositories tagged etl-pipeline, ranked by trending score. Updated daily.
-
1
DataWithBaraa/sql-data-warehouse-project
Build a data warehouse in SQL Server from two source systems, a CRM and an ERP, through bronze, silver and gold. The files and scripts of the free full project.
GitHub · TSQL · 1,089 stars · 751 forks · +27 stars in 24 hours · Trending score 2.27
-
2
qiantongtech/qData
qData is a lightweight, feature-complete, AI-native open-source data platform. It covers data integration, development, modeling, governance, assets and API services, with self-hosted deployment, flexible scaling and commercial use.
GitHub · Java · 954 stars · 140 forks · +10 stars in 24 hours · Trending score 2.05
-
3
apache/hamilton
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
GitHub · Jupyter Notebook · 2,631 stars · 235 forks · +6 stars in 24 hours · Trending score 1.60
-
4
yuyuejia/datay
DataY 是一套轻量、高性能的数据平台,一个 JAR、自研流批一体处理引擎,自研分布式调度框架,在 2 核 4G 的机器上 就能跑通 数据集成 → 数据建模 → 数据开发 → 任务调度 → 数据应用 的完整链路。
GitHub · Java · 50 stars · 23 forks · +1 stars in 24 hours · Trending score 1.07
-
5
risingwavelabs/risingwave
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
GitHub · Rust · 9,425 stars · 872 forks · 0 stars in 24 hours · Trending score 0.88
-
6
Open-Source-Legal/OpenContracts
The open document intelligence platform for builders and hackers - DMS for the agentic world
GitHub · Python · 1,550 stars · 201 forks · 0 stars in 24 hours · Trending score 0.73
-
7
AgentSwarms-fyi/agentswarms
Unified Agentic AI and Data Platform
GitHub · TypeScript · 302 stars · 67 forks · 0 stars in 24 hours · Trending score 0.60
-
8
Edwardvaneechoud/Flowfile
Flowfile is a visual ETL tool and Python library combining drag-and-drop workflows with Polars dataframes. Build data pipelines visually, define flows programmatically with a Polars-like API, and export to standalone Python code. Perfect for fast, intuitive data processing from development to production.
GitHub · Python · 413 stars · 34 forks · 0 stars in 24 hours · Trending score 0.42
-
9
ebonnal/streamable
sync/async iterable streams for Python (concurrency, throttling, buffering, ...)
GitHub · Python · 337 stars · 8 forks · 0 stars in 24 hours · Trending score 0.33
-
10
goldsky-io/streamling
Data streaming runtime focused on performance, consistency, and extensibility. Write plugins in Rust or WASM and process data with data guarantees.
GitHub · Rust · 111 stars · 9 forks · 0 stars in 24 hours · Trending score 0.30
-
11
MTSWebServices/onetl
One ETL tool to rule them all
GitHub · Python · 91 stars · 8 forks · 0 stars in 24 hours · Trending score 0.20
-
12
bkc39/rkt-polars
Racket bindings to polars
GitHub · Racket · 11 stars · 0 forks · 0 stars in 24 hours · Trending score 0.19
-
13
techascent/tech.ml.dataset
A Clojure high performance data processing system
GitHub · Clojure · 764 stars · 34 forks · 0 stars in 24 hours · Trending score 0.14
-
14
koralium/flowtide
High-performance streaming SQL query engine designed for real-time data processing. Use cases include event-driven architectures, ETL pipelines, and modern data-intensive applications.
GitHub · C# · 110 stars · 8 forks · 0 stars in 24 hours · Trending score 0.10
-
15
opencrawling/opencrawling
Reference Java & Spring implementation of the Open Ingestion Standard (OIS). It provides a secure, decoupled, and vendor-neutral enterprise data integration platform leveraging modern Java 25 features, Spring Boot, Spring AI and vector search infrastructure to orchestrate data flows
GitHub · Java · 29 stars · 5 forks · 0 stars in 24 hours · Trending score 0.10
-
16
LAMP-LUCAS/AutoSINAPI
Solução open source para simplificar o acesso, tratamento e gestão dos dados do SINAPI (Sistema Nacional de Pesquisa de Custos e Índices da Construção Civil)
GitHub · Python · 12 stars · 5 forks · 0 stars in 24 hours · Trending score 0.10
-
17
keboola/mcp-server
Model Context Protocol (MCP) Server for the Keboola Platform
GitHub · Python · 84 stars · 23 forks · −1 stars in 24 hours · Trending score 0.09
-
18
conductor-oss/python-sdk
Conductor OSS SDK for Python programming language
GitHub · Python · 104 stars · 46 forks · 0 stars in 24 hours · Trending score 0.09
-
19
ciaren-labs/Ciaren
Open-source, local-first visual ETL and data pipeline builder. Drag-and-drop data prep and ML workflows, preview every step, schedule runs, and export clean pandas or Polars Python code. Self-hosted, no lock-in.
GitHub · Python · 37 stars · 12 forks · 0 stars in 24 hours · Trending score 0.08
-
20
wp-labs/warp-parse
Focusing on building industry-leading ETL engines.
GitHub · Rust · 186 stars · 13 forks · 0 stars in 24 hours · Trending score 0.08
-
21
CogStack/CogStack-NiFi
Building data processing pipelines for documents processing with NLP using Apache NiFi and related services
GitHub · Python · 61 stars · 20 forks · 0 stars in 24 hours · Trending score 0.07
-
22
marciok/gust
A Task-Based Workflow Orchestration Engine
GitHub · Elixir · 392 stars · 25 forks · 0 stars in 24 hours · Trending score 0.07
-
23
realdatadriven/etlx
ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.
GitHub · Go · 57 stars · 3 forks · 0 stars in 24 hours · Trending score 0.03
-
24
swadhinbiswas/eu-air-traffic
Live and historical European air traffic: a VPS collector streams ADS-B, OpenSky movements, schedules and weather through Kafka into a Bronze/Silver lake, a dbt warehouse and a read-only serving copy behind a public dashboard. Budget-aware polling and atomic publishes keep it correct while unattended.
GitHub · Python · 6 stars · 0 forks · 0 stars in 24 hours · Trending score 0.02