data-processing repositories

Trending repositories tagged data-processing, ranked by trending score. Updated daily.

  1. 1
    NVIDIA-NeMo/Curator

    Scalable data pre processing and curation toolkit for LLMs

    GitHub · Python · 1,866 stars · 379 forks · +3 stars in 24 hours · Trending score 1.41

  2. 2
    cocoindex-io/cocoindex

    Incremental engine for long horizon agents 🌟 Star if you like it!

    GitHub · Rust · 11,918 stars · 930 forks · +3 stars in 24 hours · Trending score 1.34

  3. 3
    johnkerl/miller

    Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

    GitHub · Go · 10,068 stars · 261 forks · +1 stars in 24 hours · Trending score 1.00

  4. 4
    datajuicer/data-juicer

    Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

    GitHub · Python · 7,263 stars · 456 forks · 0 stars in 24 hours · Trending score 0.93

  5. 5
    MDSplus/mdsplus

    The MDSplus data management system

    GitHub · Java · 99 stars · 56 forks · +2 stars in 24 hours · Trending score 0.81

  6. 6
    mech-lang/mech

    🦾 Mech is a programming language for building data-driven systems like robots, games, and interfaces. Start here!

    GitHub · Rust · 309 stars · 23 forks · 0 stars in 24 hours · Trending score 0.50

  7. 7
    NVIDIA/DALI

    A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

    GitHub · C++ · 5,797 stars · 691 forks · −1 stars in 24 hours · Trending score 0.47

  8. 8
    bytewax/bytewax

    Python Stream Processing

    GitHub · Python · 2,074 stars · 114 forks · 0 stars in 24 hours · Trending score 0.41

  9. 9
    numaproj/numaflow

    Kubernetes-native platform to run massively parallel data/streaming jobs

    GitHub · Rust · 2,838 stars · 187 forks · 0 stars in 24 hours · Trending score 0.33

  10. 10
    senbox-org/snap-engine

    ESA Earth Observation Toolbox and Java Development Platform

    GitHub · Java · 217 stars · 106 forks · 0 stars in 24 hours · Trending score 0.33

  11. 11
    datachain-ai/datachain

    The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

    GitHub · Python · 2,832 stars · 180 forks · 0 stars in 24 hours · Trending score 0.32

  12. 12
    LiberTEM/LiberTEM

    Open pixelated STEM framework

    GitHub · Python · 132 stars · 74 forks · 0 stars in 24 hours · Trending score 0.32

  13. 13
    datumbrain/otters

    🦦 Pandas-style DataFrame library for Go — fluent API for filtering, grouping, sorting, and statistical analysis with type-safe operations and zero-copy typed slice access.

    GitHub · Go · 18 stars · 6 forks · 0 stars in 24 hours · Trending score 0.32

  14. 14
    kubeflow/mcp-apache-spark-history-server

    MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.

    GitHub · Python · 207 stars · 84 forks · 0 stars in 24 hours · Trending score 0.31

  15. 15
    LibreCat/Catmandu

    Catmandu - a data processing toolkit

    GitHub · Perl · 199 stars · 35 forks · 0 stars in 24 hours · Trending score 0.22

  16. 16
    Puchaczov/Musoq

    SQL Runtime without any database

    GitHub · C# · 507 stars · 22 forks · 0 stars in 24 hours · Trending score 0.22

  17. 17
    TangleML/tangle

    Tangle is a web app that allows the users to build and run Machine Learning pipelines without having to set up development environment.

    GitHub · Python · 516 stars · 41 forks · 0 stars in 24 hours · Trending score 0.14

  18. 18
    pathwaycom/pathway

    Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

    GitHub · Python · 61,982 stars · 1,704 forks · −8 stars in 24 hours · Trending score 0.10

  19. 19
    deverte/awesome-science

    A currated list of awesome scientific software, libraries and services.

    GitHub · 38 stars · 11 forks · 0 stars in 24 hours · Trending score 0.10

  20. 20
    GabrielMarquezMatte/ExcelReader

    High-performance, low-allocation Excel (.xlsx, .xlsb, .xls, .csv) reader and writer for .NET 10, with Native AOT support and C++/Rust/Python bindings.

    GitHub · C# · 21 stars · 1 forks · 0 stars in 24 hours · Trending score 0.09

  21. 21
    senbox-org/snap-desktop

    Desktop GUI for SNAP based on NetBeans Platform

    GitHub · Java · 151 stars · 67 forks · 0 stars in 24 hours · Trending score 0.09

  22. 22
    melihbirim/csvql

    Run SQL on CSV files in place. Read-only CSV query engine and MCP server for AI agents. Single Zig binary, no import.

    GitHub · Zig · 31 stars · 6 forks · 0 stars in 24 hours · Trending score 0.07

  23. 23
    polyaxon/haupt

    Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

    GitHub · Python · 452 stars · 204 forks · 0 stars in 24 hours · Trending score 0.05

  24. 24
    apache/wayang

    Apache Wayang is the first cross-platform data processing system.

    GitHub · Java · 284 stars · 143 forks · 0 stars in 24 hours · Trending score 0.04