spark repositories

Trending repositories tagged spark, ranked by trending score. Updated daily.

  1. 1
    DataTalksClub/data-engineering-zoomcamp

    Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

    GitHub · Jupyter Notebook · 46,985 stars · 9,082 forks · +36 stars in 24 hours · Trending score 3.25

  2. 2
    apache/spark

    Apache Spark - A unified analytics engine for large-scale data processing

    GitHub · Scala · 44,590 stars · 29,593 forks · +30 stars in 24 hours · Trending score 2.84

  3. 3
    tobymao/sqlglot

    Python SQL Parser and Transpiler

    GitHub · Python · 9,809 stars · 1,443 forks · +13 stars in 24 hours · Trending score 2.52

  4. 4
    metabrainz/listenbrainz-server

    Server for the ListenBrainz project, including the front-end (javascript/react) code that it serves and all of the data processing components that LB uses.

    GitHub · Python · 1,071 stars · 399 forks · +11 stars in 24 hours · Trending score 2.06

  5. 5
    ssanjaychandra123/data-engineering-patterns

    600+ free patterns and concepts for data engineers on Azure Databricks, Microsoft Fabric, and PySpark. 12 books covering the full stack. Free forever.

    GitHub · 93 stars · 39 forks · +10 stars in 24 hours · Trending score 1.67

  6. 6
    Luce-Org/lucebox

    Lucebox inference engine for heterogeneous hardware & consumer GPUs

    GitHub · C++ · 3,018 stars · 299 forks · +3 stars in 24 hours · Trending score 1.63

  7. 7
    apache/amoro

    Apache Amoro(incubating) is a Lakehouse management system built on open data lake formats.

    GitHub · Java · 1,207 stars · 402 forks · +4 stars in 24 hours · Trending score 1.48

  8. 8
    delta-io/delta-sharing

    An open protocol for secure data sharing

    GitHub · Scala · 963 stars · 245 forks · +4 stars in 24 hours · Trending score 1.34

  9. 9
    apache/paimon

    Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.

    GitHub · Java · 3,452 stars · 1,438 forks · +2 stars in 24 hours · Trending score 1.34

  10. 10
    apache/auron

    Apache Auron is an accelerator for big data engines, leveraging native vectorized execution to accelerate query processing.

    GitHub · Rust · 1,849 stars · 252 forks · +3 stars in 24 hours · Trending score 1.23

  11. 11
    runzhliu/aik8s

    AI/LLM on Kubernetes 基础设施知识库:涵盖 GPU、大数据、调度、分布式训练、LLM 推理、RAG、Agent 与生产运维。| AI/LLM on Kubernetes Infrastructure Knowledge Base: Covering GPUs, big data, scheduling, distributed training, LLM inference, RAG, agents, and production operations.

    GitHub · Python · 44 stars · 4 forks · +4 stars in 24 hours · Trending score 1.22

  12. 12
    NVIDIA/cudf-spark

    NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs

    GitHub · Scala · 1,020 stars · 435 forks · 0 stars in 24 hours · Trending score 1.20

  13. 13
    apache/datafusion-spark

    Apache Spark-compatible expressions for Apache DataFusion

    GitHub · Rust · 11 stars · 21 forks · +1 stars in 24 hours · Trending score 1.17

  14. 14
    kubeflow/spark-operator

    Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

    GitHub · Python · 3,162 stars · 1,562 forks · +2 stars in 24 hours · Trending score 1.13

  15. 15
    projectnessie/nessie

    Nessie: Transactional Catalog for Data Lakes with Git-like semantics

    GitHub · Java · 1,534 stars · 205 forks · +2 stars in 24 hours · Trending score 1.07

  16. 16
    delta-io/delta

    An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

    GitHub · Scala · 9,083 stars · 2,225 forks · −1 stars in 24 hours · Trending score 1.03

  17. 17
    apache/datafusion-comet

    Apache DataFusion Comet Spark Accelerator

    GitHub · Scala · 1,325 stars · 419 forks · +1 stars in 24 hours · Trending score 1.02

  18. 18
    lakehq/sail

    Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

    GitHub · Rust · 3,504 stars · 243 forks · 0 stars in 24 hours · Trending score 0.69

  19. 19
    microsoft/LakeBench

    A multi-modal Python library for benchmarking lakehouse engines and ELT scenarios, supporting both industry-standard and novel benchmarks.

    GitHub · Python · 57 stars · 24 forks · 0 stars in 24 hours · Trending score 0.68

  20. 20
    databrickslabs/dqx

    Databricks framework to validate Data Quality of pySpark DataFrames and Tables

    GitHub · Python · 486 stars · 166 forks · 0 stars in 24 hours · Trending score 0.65

  21. 21
    slackhq/spark-gateway

    REST API for interacting with batch Spark Applications on Kubernetes

    GitHub · Go · 16 stars · 3 forks · +1 stars in 24 hours · Trending score 0.64

  22. 22
    apache/incubator-graphar

    An open source, standard data file format for graph data storage and retrieval.

    GitHub · Java · 384 stars · 97 forks · 0 stars in 24 hours · Trending score 0.64

  23. 23
    databrickslabs/geobrix

    GeoBrix is a high-performance spatial processing library.

    GitHub · Python · 36 stars · 6 forks · 0 stars in 24 hours · Trending score 0.62

  24. 24
    YanivZalach/IceGraph

    Interactive Apache Iceberg debugger for exploring production tables and tracing metadata evolution through AI and real time graph visualization.

    GitHub · TypeScript · 26 stars · 8 forks · +1 stars in 24 hours · Trending score 0.55