apache-spark repositories
Trending repositories tagged apache-spark, ranked by trending score. Updated daily.
-
1
mlflow/mlflow
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
GitHub · Python · 28,909 stars · 6,644 forks · +40 stars in 24 hours · Trending score 3.23
-
2
itsmekhoathekid/RecSys-MLops
Highly Scalable RecSys ML & Data system
GitHub · Python · 137 stars · 29 forks · +9 stars in 24 hours · Trending score 1.66
-
3
kubeflow/spark-operator
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
GitHub · Python · 3,162 stars · 1,562 forks · +2 stars in 24 hours · Trending score 1.13
-
4
vaquarkhan/data-engineering-agent-skills
Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.
GitHub · Python · 65 stars · 9 forks · 0 stars in 24 hours · Trending score 0.80
-
5
treeverse/lakeFS
lakeFS - Data version control for your data lake | Git for data
GitHub · Go · 5,586 stars · 493 forks · −3 stars in 24 hours · Trending score 0.72
-
6
lakehq/sail
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
GitHub · Rust · 3,504 stars · 243 forks · 0 stars in 24 hours · Trending score 0.69
-
7
sparkdq-community/sparkdq
A lightweight, declarative PySpark framework for data quality validation — check columns, rows, and entire datasets directly in your Spark pipelines
GitHub · Python · 77 stars · 10 forks · +1 stars in 24 hours · Trending score 0.61
-
8
tech-pw/cryo
Self-hosted Apache Iceberg maintenance and health platform — scans, grades A-F, and runs compaction, snapshot expiry, and orphan cleanup on Trino or Spark.
GitHub · Python · 12 stars · 2 forks · 0 stars in 24 hours · Trending score 0.51
-
9
kubeflow/mcp-apache-spark-history-server
MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.
GitHub · Python · 207 stars · 84 forks · 0 stars in 24 hours · Trending score 0.31
-
10
aws-samples/iceberg-streaming-examples
This repo contains examples of high throughput ingestion using Apache Spark and Apache Iceberg. These examples cover IoT and CDC scenarios using best practices. The code can be deployed into any Spark compatible engine like Amazon EMR Serverless or AWS Glue. A fully local developer environment is also provided.
GitHub · Java · 32 stars · 8 forks · 0 stars in 24 hours · Trending score 0.24
-
11
mahmoudparsian/big-data-modeling
Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University
GitHub · HTML · 167 stars · 144 forks · 0 stars in 24 hours · Trending score 0.20
-
12
stackabletech/spark-k8s-operator
Kubernetes operator for Apache Spark. Run Spark applications on Kubernetes with the Stackable Data Platform (SDP).
GitHub · Rust · 74 stars · 6 forks · 0 stars in 24 hours · Trending score 0.08
-
13
GyuyeongKim/Tessera-public
Fast join-aggregates over skewed sparse relations inside DuckDB, PostgreSQL, SQLite, and Spark
GitHub · C++ · 7 stars · 0 forks · 0 stars in 24 hours · Trending score 0.05
-
14
swadhinbiswas/eu-air-traffic
Live and historical European air traffic: a VPS collector streams ADS-B, OpenSky movements, schedules and weather through Kafka into a Bronze/Silver lake, a dbt warehouse and a read-only serving copy behind a public dashboard. Budget-aware polling and atomic publishes keep it correct while unattended.
GitHub · Python · 6 stars · 0 forks · 0 stars in 24 hours · Trending score 0.04
-
15
DataWithBaraa/databricks_bootcamp_2026
End-to-end Data Lakehouse project built on Databricks, following the Medallion Architecture (Bronze, Silver, Gold). Covers real-world data engineering and analytics workflows using Spark, PySpark, SQL, Delta Lake, and Unity Catalog. Designed for learning, portfolio building, and job interviews.
GitHub · Jupyter Notebook · 460 stars · 201 forks · 0 stars in 24 hours · Trending score 0.00
-
16
adrianulbona/osm-parquetizer
A converter for the OSM PBFs to Parquet files
GitHub · Java · 98 stars · 31 forks · 0 stars in 24 hours · Trending score 0.00
-
17
sparklyr/sparklyr
R interface for Apache Spark
GitHub · R · 971 stars · 310 forks · 0 stars in 24 hours · Trending score 0.00
-
18
lresende/ansible-spark-cluster
Ansible roles to install an Spark Standalone cluster (HDFS/Spark/Jupyter Notebook) or Ambari based Spark cluster
GitHub · Shell · 62 stars · 35 forks · 0 stars in 24 hours · Trending score 0.00
-
19
sparklabx/sparklabx
Self-hosted multi-user Apache Spark notebooks — PySpark + Scala, per-user kernel pods, MinIO IAM storage, OAuth. One Helm install. Apache-2.0.
GitHub · TypeScript · 28 stars · 9 forks · 0 stars in 24 hours · Trending score 0.00
-
20
geoHeil/awesome-tools
curated list of awesome tools and libraries for specific domains
GitHub · 87 stars · 22 forks · 0 stars in 24 hours · Trending score 0.00
-
21
qdrant/qdrant-spark
Qdrant's Apache Spark connector
GitHub · Java · 50 stars · 1 forks · 0 stars in 24 hours · Trending score 0.00
-
22
jaceklaskowski/learn-databricks
Notebooks to learn Databricks Lakehouse Platform
GitHub · Jupyter Notebook · 51 stars · 30 forks · 0 stars in 24 hours · Trending score 0.00
-
23
jrlasak/awesome-databricks
170+ curated resources every Databricks Data Engineer should bookmark - tools, courses, creators, labs, and communities
GitHub · 58 stars · 25 forks · 0 stars in 24 hours · Trending score 0.00
-
24
japila-books/apache-spark-internals
The Internals of Apache Spark
GitHub · 1,551 stars · 457 forks · 0 stars in 24 hours · Trending score 0.00