spark repositories
Trending repositories tagged spark, ranked by trending score. Updated daily.
-
1
DataTalksClub/data-engineering-zoomcamp
Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼
GitHub · Jupyter Notebook · 46,985 stars · 9,082 forks · +36 stars in 24 hours · Trending score 3.25
-
2
apache/spark
Apache Spark - A unified analytics engine for large-scale data processing
GitHub · Scala · 44,590 stars · 29,593 forks · +30 stars in 24 hours · Trending score 2.84
-
3
tobymao/sqlglot
Python SQL Parser and Transpiler
GitHub · Python · 9,809 stars · 1,443 forks · +13 stars in 24 hours · Trending score 2.52
-
4
metabrainz/listenbrainz-server
Server for the ListenBrainz project, including the front-end (javascript/react) code that it serves and all of the data processing components that LB uses.
GitHub · Python · 1,071 stars · 399 forks · +11 stars in 24 hours · Trending score 2.06
-
5
ssanjaychandra123/data-engineering-patterns
600+ free patterns and concepts for data engineers on Azure Databricks, Microsoft Fabric, and PySpark. 12 books covering the full stack. Free forever.
GitHub · 93 stars · 39 forks · +10 stars in 24 hours · Trending score 1.67
-
6
Luce-Org/lucebox
Lucebox inference engine for heterogeneous hardware & consumer GPUs
GitHub · C++ · 3,018 stars · 299 forks · +3 stars in 24 hours · Trending score 1.63
-
7
apache/amoro
Apache Amoro(incubating) is a Lakehouse management system built on open data lake formats.
GitHub · Java · 1,207 stars · 402 forks · +4 stars in 24 hours · Trending score 1.48
-
8
delta-io/delta-sharing
An open protocol for secure data sharing
GitHub · Scala · 963 stars · 245 forks · +4 stars in 24 hours · Trending score 1.34
-
9
apache/paimon
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
GitHub · Java · 3,452 stars · 1,438 forks · +2 stars in 24 hours · Trending score 1.34
-
10
apache/auron
Apache Auron is an accelerator for big data engines, leveraging native vectorized execution to accelerate query processing.
GitHub · Rust · 1,849 stars · 252 forks · +3 stars in 24 hours · Trending score 1.23
-
11
runzhliu/aik8s
AI/LLM on Kubernetes 基础设施知识库:涵盖 GPU、大数据、调度、分布式训练、LLM 推理、RAG、Agent 与生产运维。| AI/LLM on Kubernetes Infrastructure Knowledge Base: Covering GPUs, big data, scheduling, distributed training, LLM inference, RAG, agents, and production operations.
GitHub · Python · 44 stars · 4 forks · +4 stars in 24 hours · Trending score 1.22
-
12
NVIDIA/cudf-spark
NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs
GitHub · Scala · 1,020 stars · 435 forks · 0 stars in 24 hours · Trending score 1.20
-
13
apache/datafusion-spark
Apache Spark-compatible expressions for Apache DataFusion
GitHub · Rust · 11 stars · 21 forks · +1 stars in 24 hours · Trending score 1.17
-
14
kubeflow/spark-operator
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
GitHub · Python · 3,162 stars · 1,562 forks · +2 stars in 24 hours · Trending score 1.13
-
15
projectnessie/nessie
Nessie: Transactional Catalog for Data Lakes with Git-like semantics
GitHub · Java · 1,534 stars · 205 forks · +2 stars in 24 hours · Trending score 1.07
-
16
delta-io/delta
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
GitHub · Scala · 9,083 stars · 2,225 forks · −1 stars in 24 hours · Trending score 1.03
-
17
apache/datafusion-comet
Apache DataFusion Comet Spark Accelerator
GitHub · Scala · 1,325 stars · 419 forks · +1 stars in 24 hours · Trending score 1.02
-
18
lakehq/sail
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
GitHub · Rust · 3,504 stars · 243 forks · 0 stars in 24 hours · Trending score 0.69
-
19
microsoft/LakeBench
A multi-modal Python library for benchmarking lakehouse engines and ELT scenarios, supporting both industry-standard and novel benchmarks.
GitHub · Python · 57 stars · 24 forks · 0 stars in 24 hours · Trending score 0.68
-
20
databrickslabs/dqx
Databricks framework to validate Data Quality of pySpark DataFrames and Tables
GitHub · Python · 486 stars · 166 forks · 0 stars in 24 hours · Trending score 0.65
-
21
slackhq/spark-gateway
REST API for interacting with batch Spark Applications on Kubernetes
GitHub · Go · 16 stars · 3 forks · +1 stars in 24 hours · Trending score 0.64
-
22
apache/incubator-graphar
An open source, standard data file format for graph data storage and retrieval.
GitHub · Java · 384 stars · 97 forks · 0 stars in 24 hours · Trending score 0.64
-
23
databrickslabs/geobrix
GeoBrix is a high-performance spatial processing library.
GitHub · Python · 36 stars · 6 forks · 0 stars in 24 hours · Trending score 0.62
-
24
YanivZalach/IceGraph
Interactive Apache Iceberg debugger for exploring production tables and tracing metadata evolution through AI and real time graph visualization.
GitHub · TypeScript · 26 stars · 8 forks · +1 stars in 24 hours · Trending score 0.55