data-processing repositories
Trending repositories tagged data-processing, ranked by trending score. Updated daily.
-
1
NVIDIA-NeMo/Curator
Scalable data pre processing and curation toolkit for LLMs
GitHub · Python · 1,866 stars · 379 forks · +3 stars in 24 hours · Trending score 1.41
-
2
cocoindex-io/cocoindex
Incremental engine for long horizon agents 🌟 Star if you like it!
GitHub · Rust · 11,918 stars · 930 forks · +3 stars in 24 hours · Trending score 1.34
-
3
johnkerl/miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
GitHub · Go · 10,068 stars · 261 forks · +1 stars in 24 hours · Trending score 1.00
-
4
datajuicer/data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
GitHub · Python · 7,263 stars · 456 forks · 0 stars in 24 hours · Trending score 0.93
-
5
MDSplus/mdsplus
The MDSplus data management system
GitHub · Java · 99 stars · 56 forks · +2 stars in 24 hours · Trending score 0.81
-
6
mech-lang/mech
🦾 Mech is a programming language for building data-driven systems like robots, games, and interfaces. Start here!
GitHub · Rust · 309 stars · 23 forks · 0 stars in 24 hours · Trending score 0.50
-
7
NVIDIA/DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
GitHub · C++ · 5,797 stars · 691 forks · −1 stars in 24 hours · Trending score 0.47
-
8
bytewax/bytewax
Python Stream Processing
GitHub · Python · 2,074 stars · 114 forks · 0 stars in 24 hours · Trending score 0.41
-
9
numaproj/numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
GitHub · Rust · 2,838 stars · 187 forks · 0 stars in 24 hours · Trending score 0.33
-
10
senbox-org/snap-engine
ESA Earth Observation Toolbox and Java Development Platform
GitHub · Java · 217 stars · 106 forks · 0 stars in 24 hours · Trending score 0.33
-
11
datachain-ai/datachain
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
GitHub · Python · 2,832 stars · 180 forks · 0 stars in 24 hours · Trending score 0.32
-
12
LiberTEM/LiberTEM
Open pixelated STEM framework
GitHub · Python · 132 stars · 74 forks · 0 stars in 24 hours · Trending score 0.32
-
13
datumbrain/otters
🦦 Pandas-style DataFrame library for Go — fluent API for filtering, grouping, sorting, and statistical analysis with type-safe operations and zero-copy typed slice access.
GitHub · Go · 18 stars · 6 forks · 0 stars in 24 hours · Trending score 0.32
-
14
kubeflow/mcp-apache-spark-history-server
MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.
GitHub · Python · 207 stars · 84 forks · 0 stars in 24 hours · Trending score 0.31
-
15
LibreCat/Catmandu
Catmandu - a data processing toolkit
GitHub · Perl · 199 stars · 35 forks · 0 stars in 24 hours · Trending score 0.22
-
16
Puchaczov/Musoq
SQL Runtime without any database
GitHub · C# · 507 stars · 22 forks · 0 stars in 24 hours · Trending score 0.22
-
17
TangleML/tangle
Tangle is a web app that allows the users to build and run Machine Learning pipelines without having to set up development environment.
GitHub · Python · 516 stars · 41 forks · 0 stars in 24 hours · Trending score 0.14
-
18
pathwaycom/pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
GitHub · Python · 61,982 stars · 1,704 forks · −8 stars in 24 hours · Trending score 0.10
-
19
deverte/awesome-science
A currated list of awesome scientific software, libraries and services.
GitHub · 38 stars · 11 forks · 0 stars in 24 hours · Trending score 0.10
-
20
GabrielMarquezMatte/ExcelReader
High-performance, low-allocation Excel (.xlsx, .xlsb, .xls, .csv) reader and writer for .NET 10, with Native AOT support and C++/Rust/Python bindings.
GitHub · C# · 21 stars · 1 forks · 0 stars in 24 hours · Trending score 0.09
-
21
senbox-org/snap-desktop
Desktop GUI for SNAP based on NetBeans Platform
GitHub · Java · 151 stars · 67 forks · 0 stars in 24 hours · Trending score 0.09
-
22
melihbirim/csvql
Run SQL on CSV files in place. Read-only CSV query engine and MCP server for AI agents. Single Zig binary, no import.
GitHub · Zig · 31 stars · 6 forks · 0 stars in 24 hours · Trending score 0.07
-
23
polyaxon/haupt
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
GitHub · Python · 452 stars · 204 forks · 0 stars in 24 hours · Trending score 0.05
-
24
apache/wayang
Apache Wayang is the first cross-platform data processing system.
GitHub · Java · 284 stars · 143 forks · 0 stars in 24 hours · Trending score 0.04