data-pipelines repositories

Trending repositories tagged data-pipelines, ranked by trending score. Updated daily.

  1. 1
    apache/airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    GitHub · Python · 47,569 stars · 18,419 forks · +46 stars in 24 hours · Trending score 3.37

  2. 2
    dagster-io/dagster

    An orchestration platform for the development, production, and observation of data assets.

    GitHub · Python · 16,444 stars · 2,427 forks · +9 stars in 24 hours · Trending score 2.36

  3. 3
    ucbepic/docetl

    A system for agentic LLM-powered data processing and ETL

    GitHub · Python · 4,259 stars · 471 forks · +2 stars in 24 hours · Trending score 1.58

  4. 4
    bruin-data/bruin

    Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

    GitHub · Go · 1,815 stars · 101 forks · +6 stars in 24 hours · Trending score 1.55

  5. 5
    feldera/feldera

    The Feldera Incremental Computation Engine

    GitHub · Rust · 2,181 stars · 181 forks · +3 stars in 24 hours · Trending score 1.54

  6. 6
    apache/dolphinscheduler

    Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

    GitHub · Java · 14,558 stars · 5,122 forks · +7 stars in 24 hours · Trending score 1.54

  7. 7
    Unstructured-IO/unstructured

    Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

    GitHub · HTML · 15,759 stars · 1,404 forks · 0 stars in 24 hours · Trending score 1.38

  8. 8
    cubert-hyperspectral/cuvis-ai

    Cuvis.AI is an open-source, extensible framework for building modular AI pipelines that process, analyze, and deploy hyperspectral video data in real-time applications.

    GitHub · Jupyter Notebook · 20 stars · 0 forks · +1 stars in 24 hours · Trending score 0.68

  9. 9
    meltano/meltano

    Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

    GitHub · Python · 2,683 stars · 282 forks · 0 stars in 24 hours · Trending score 0.57

  10. 10
    astronomer/agents

    AI agent tooling for data engineering workflows.

    GitHub · Python · 464 stars · 72 forks · 0 stars in 24 hours · Trending score 0.55

  11. 11
    openivm/openivm

    Incremental View Maintenance on DuckDB

    GitHub · C++ · 71 stars · 7 forks · 0 stars in 24 hours · Trending score 0.30

  12. 12
    cernodatainc/cernodata

    Open-source control layer over PDF parsing that picks the right method, verifies output quality, and falls back automatically when results fall short

    GitHub · Python · 24 stars · 6 forks · 0 stars in 24 hours · Trending score 0.30

  13. 13
    debezium/debezium-platform

    An opinionated data-centric view of Debezium components. Please log issues at https://github.com/debezium/dbz/issues.

    GitHub · TypeScript · 55 stars · 46 forks · 0 stars in 24 hours · Trending score 0.19

  14. 14
    linkedin/Hoptimator

    Multi-hop declarative data pipelines

    GitHub · Java · 128 stars · 18 forks · 0 stars in 24 hours · Trending score 0.15

  15. 15
    KafScale/platform

    Kafka-compatible streaming that keeps your data in S3. Stateless brokers, standard Kafka clients, Apache 2.0.

    GitHub · Go · 59 stars · 15 forks · 0 stars in 24 hours · Trending score 0.10

  16. 16
    pathwaycom/pathway

    Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

    GitHub · Python · 61,982 stars · 1,704 forks · −8 stars in 24 hours · Trending score 0.10

  17. 17
    eschizoid/kpipe

    Composable Kafka consumer library for building modular, testable JVM data pipelines.

    GitHub · Java · 58 stars · 7 forks · 0 stars in 24 hours · Trending score 0.10

  18. 18
    DidactHQ/didact

    The open core .NET job orchestrator that we've been missing

    GitHub · C# · 162 stars · 3 forks · 0 stars in 24 hours · Trending score 0.10

  19. 19
    akshaysharma1088/dataranger

    Open-source AI agent that patrols your data, fixes only what it can prove against the source, and sends the rest to a person with the evidence. Works with Snowflake, dbt and DuckDB, runs as a GitHub Action, and speaks MCP.

    GitHub · Python · 5 stars · 10 forks · 0 stars in 24 hours · Trending score 0.08

  20. 20
    databricks-solutions/lakeflow_framework

    Metadata-driven framework for Databricks Spark Declarative Pipelines. Config-driven, pattern based approach to batch & streaming across the medallion architecture. Deploys via Declarative Automation Bundles. Built for simplicity, extensibility, and alignment with the Databricks product roadmap.

    GitHub · Python · 30 stars · 20 forks · 0 stars in 24 hours · Trending score 0.07

  21. 21
    CogStack/CogStack-NiFi

    Building data processing pipelines for documents processing with NLP using Apache NiFi and related services

    GitHub · Python · 61 stars · 20 forks · 0 stars in 24 hours · Trending score 0.05

  22. 22
    combust/mleap

    MLeap: Deploy ML Pipelines to Production

    GitHub · Scala · 1,545 stars · 312 forks · 0 stars in 24 hours · Trending score 0.03

  23. 23
    datajoint/datajoint-python

    Relational Workflows: where database schemas define executable data pipelines.

    GitHub · Python · 197 stars · 101 forks · 0 stars in 24 hours · Trending score 0.00

  24. 24
    wingfoil-io/wingfoil

    ultra low latency graph based stream processing framework

    GitHub · Rust · 230 stars · 35 forks · 0 stars in 24 hours · Trending score 0.00