apache-spark repositories

Trending repositories tagged apache-spark, ranked by trending score. Updated daily.

  1. 1
    mlflow/mlflow

    The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

    GitHub · Python · 28,909 stars · 6,644 forks · +40 stars in 24 hours · Trending score 3.23

  2. 2
    itsmekhoathekid/RecSys-MLops

    Highly Scalable RecSys ML & Data system

    GitHub · Python · 137 stars · 29 forks · +9 stars in 24 hours · Trending score 1.66

  3. 3
    kubeflow/spark-operator

    Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

    GitHub · Python · 3,162 stars · 1,562 forks · +2 stars in 24 hours · Trending score 1.13

  4. 4
    vaquarkhan/data-engineering-agent-skills

    Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.

    GitHub · Python · 65 stars · 9 forks · 0 stars in 24 hours · Trending score 0.80

  5. 5
    treeverse/lakeFS

    lakeFS - Data version control for your data lake | Git for data

    GitHub · Go · 5,586 stars · 493 forks · −3 stars in 24 hours · Trending score 0.72

  6. 6
    lakehq/sail

    Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

    GitHub · Rust · 3,504 stars · 243 forks · 0 stars in 24 hours · Trending score 0.69

  7. 7
    sparkdq-community/sparkdq

    A lightweight, declarative PySpark framework for data quality validation — check columns, rows, and entire datasets directly in your Spark pipelines

    GitHub · Python · 77 stars · 10 forks · +1 stars in 24 hours · Trending score 0.61

  8. 8
    tech-pw/cryo

    Self-hosted Apache Iceberg maintenance and health platform — scans, grades A-F, and runs compaction, snapshot expiry, and orphan cleanup on Trino or Spark.

    GitHub · Python · 12 stars · 2 forks · 0 stars in 24 hours · Trending score 0.51

  9. 9
    kubeflow/mcp-apache-spark-history-server

    MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.

    GitHub · Python · 207 stars · 84 forks · 0 stars in 24 hours · Trending score 0.31

  10. 10
    aws-samples/iceberg-streaming-examples

    This repo contains examples of high throughput ingestion using Apache Spark and Apache Iceberg. These examples cover IoT and CDC scenarios using best practices. The code can be deployed into any Spark compatible engine like Amazon EMR Serverless or AWS Glue. A fully local developer environment is also provided.

    GitHub · Java · 32 stars · 8 forks · 0 stars in 24 hours · Trending score 0.24

  11. 11
    mahmoudparsian/big-data-modeling

    Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University

    GitHub · HTML · 167 stars · 144 forks · 0 stars in 24 hours · Trending score 0.20

  12. 12
    stackabletech/spark-k8s-operator

    Kubernetes operator for Apache Spark. Run Spark applications on Kubernetes with the Stackable Data Platform (SDP).

    GitHub · Rust · 74 stars · 6 forks · 0 stars in 24 hours · Trending score 0.08

  13. 13
    GyuyeongKim/Tessera-public

    Fast join-aggregates over skewed sparse relations inside DuckDB, PostgreSQL, SQLite, and Spark

    GitHub · C++ · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05

  14. 14
    swadhinbiswas/eu-air-traffic

    Live and historical European air traffic: a VPS collector streams ADS-B, OpenSky movements, schedules and weather through Kafka into a Bronze/Silver lake, a dbt warehouse and a read-only serving copy behind a public dashboard. Budget-aware polling and atomic publishes keep it correct while unattended.

    GitHub · Python · 6 stars · 0 forks · 0 stars in 24 hours · Trending score 0.04

  15. 15
    DataWithBaraa/databricks_bootcamp_2026

    End-to-end Data Lakehouse project built on Databricks, following the Medallion Architecture (Bronze, Silver, Gold). Covers real-world data engineering and analytics workflows using Spark, PySpark, SQL, Delta Lake, and Unity Catalog. Designed for learning, portfolio building, and job interviews.

    GitHub · Jupyter Notebook · 460 stars · 201 forks · 0 stars in 24 hours · Trending score 0.00

  16. 16
    adrianulbona/osm-parquetizer

    A converter for the OSM PBFs to Parquet files

    GitHub · Java · 98 stars · 31 forks · 0 stars in 24 hours · Trending score 0.00

  17. 17
    sparklyr/sparklyr

    R interface for Apache Spark

    GitHub · R · 971 stars · 310 forks · 0 stars in 24 hours · Trending score 0.00

  18. 18
    lresende/ansible-spark-cluster

    Ansible roles to install an Spark Standalone cluster (HDFS/Spark/Jupyter Notebook) or Ambari based Spark cluster

    GitHub · Shell · 62 stars · 35 forks · 0 stars in 24 hours · Trending score 0.00

  19. 19
    sparklabx/sparklabx

    Self-hosted multi-user Apache Spark notebooks — PySpark + Scala, per-user kernel pods, MinIO IAM storage, OAuth. One Helm install. Apache-2.0.

    GitHub · TypeScript · 28 stars · 9 forks · 0 stars in 24 hours · Trending score 0.00

  20. 20
    geoHeil/awesome-tools

    curated list of awesome tools and libraries for specific domains

    GitHub · 87 stars · 22 forks · 0 stars in 24 hours · Trending score 0.00

  21. 21
    qdrant/qdrant-spark

    Qdrant's Apache Spark connector

    GitHub · Java · 50 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00

  22. 22
    jaceklaskowski/learn-databricks

    Notebooks to learn Databricks Lakehouse Platform

    GitHub · Jupyter Notebook · 51 stars · 30 forks · 0 stars in 24 hours · Trending score 0.00

  23. 23
    jrlasak/awesome-databricks

    170+ curated resources every Databricks Data Engineer should bookmark - tools, courses, creators, labs, and communities

    GitHub · 58 stars · 25 forks · 0 stars in 24 hours · Trending score 0.00

  24. 24
    japila-books/apache-spark-internals

    The Internals of Apache Spark

    GitHub · 1,551 stars · 457 forks · 0 stars in 24 hours · Trending score 0.00