data-ingestion repositories

Trending repositories tagged data-ingestion, ranked by trending score. Updated daily.

  1. 1
    datazip-inc/olake-go

    OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.

    GitHub · Go · 2,154 stars · 320 forks · +163 stars in 24 hours · Trending score 4.18

  2. 2
    apache/seatunnel

    SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

    GitHub · Java · 9,821 stars · 2,474 forks · +4 stars in 24 hours · Trending score 1.76

  3. 3
    oracle-samples/oracle-aidp-samples

    Oracle AI Data Platform Workbench Samples

    GitHub · Python · 61 stars · 44 forks · +5 stars in 24 hours · Trending score 1.46

  4. 4
    bruin-data/ingestr

    ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

    GitHub · Go · 4,049 stars · 170 forks · +3 stars in 24 hours · Trending score 1.16

  5. 5
    bruin-data/bruin

    Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

    GitHub · Go · 1,816 stars · 100 forks · +1 stars in 24 hours · Trending score 0.91

  6. 6
    CrunchyData/pg_parquet

    Copy to/from Parquet in S3, Azure Blob Storage, Google Cloud Storage, http(s) stores, local files or standard inout stream from within PostgreSQL

    GitHub · Rust · 696 stars · 40 forks · +2 stars in 24 hours · Trending score 0.81

  7. 7
    apache/paimon

    Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

    GitHub · Java · 3,454 stars · 1,438 forks · 0 stars in 24 hours · Trending score 0.80

  8. 8
    apache/paimon-rust

    Apache Paimon Rust The rust implementation of Apache Paimon.

    GitHub · Rust · 215 stars · 115 forks · 0 stars in 24 hours · Trending score 0.43

  9. 9
    cernodatainc/cernodata

    Open-source control layer over PDF parsing that picks the right method, verifies output quality, and falls back automatically when results fall short

    GitHub · Python · 24 stars · 6 forks · 0 stars in 24 hours · Trending score 0.31

  10. 10
    opencrawling/opencrawling

    Reference Java & Spring implementation of the Open Ingestion Standard (OIS). It provides a secure, decoupled, and vendor-neutral enterprise data integration platform leveraging modern Java 25 features, Spring Boot, Spring AI and vector search infrastructure to orchestrate data flows

    GitHub · Java · 29 stars · 5 forks · 0 stars in 24 hours · Trending score 0.24

  11. 11
    datris/datris-platform-oss

    The Data Control Plane for AI Agents — acquire, validate, and land data over MCP with scoped keys, approval gates, and an audit log. Self-hosted, open source.

    GitHub · Scala · 14 stars · 1 forks · 0 stars in 24 hours · Trending score 0.22

  12. 12
    elastic/LogstashUI

    A control plane and visual pipeline editor for Logstash

    GitHub · Python · 34 stars · 7 forks · 0 stars in 24 hours · Trending score 0.18

  13. 13
    emiliano-go/rypipe

    Format- and source-agnostic ingestion framework: a common execution runtime for turning record-oriented data sources into typed columnar data. Rust core, Python bindings; adapters live in separate packages.

    GitHub · Rust · 23 stars · 2 forks · 0 stars in 24 hours · Trending score 0.00