pyspark repositories

Trending repositories tagged pyspark, ranked by trending score. Updated daily.

  1. 1
    sparklabx/sparklabx

    Self-hosted multi-user Apache Spark notebooks — PySpark + Scala, per-user kernel pods, MinIO IAM storage, OAuth. One Helm install. Apache-2.0.

    GitHub · TypeScript · 32 stars · 9 forks · +4 stars in 24 hours · Trending score 1.06

  2. 2
    MrPowers/chispa

    PySpark test helper methods with beautiful error messages

    GitHub · Python · 778 stars · 86 forks · +2 stars in 24 hours · Trending score 1.04

  3. 3
    ibis-project/ibis

    the portable Python dataframe library

    GitHub · Python · 6,705 stars · 790 forks · +1 stars in 24 hours · Trending score 0.96

  4. 4
    narwhals-dev/narwhals

    Lightweight and extensible compatibility layer between dataframe libraries!

    GitHub · Python · 1,751 stars · 276 forks · 0 stars in 24 hours · Trending score 0.75

  5. 5
    lakehq/sail

    Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

    GitHub · Rust · 3,504 stars · 243 forks · 0 stars in 24 hours · Trending score 0.69

  6. 6
    sparkdq-community/sparkdq

    A lightweight, declarative PySpark framework for data quality validation — check columns, rows, and entire datasets directly in your Spark pipelines

    GitHub · Python · 77 stars · 10 forks · +1 stars in 24 hours · Trending score 0.61

  7. 7
    ssanjaychandra123/data-engineering-patterns

    600+ free patterns and concepts for data engineers on Azure Databricks, Microsoft Fabric, and PySpark. 12 books covering the full stack. Free forever.

    GitHub · 93 stars · 39 forks · 0 stars in 24 hours · Trending score 0.58

  8. 8
    DataFabric-Academy/ms-fabric-data-science

    Hands-on Microsoft Fabric Data Science labs (FreshMart churn) with Thai study notes, PySpark/MLflow, and DP-604-aligned learning path.

    GitHub · Jupyter Notebook · 99 stars · 3 forks · 0 stars in 24 hours · Trending score 0.14

  9. 9
    opentargets/gentropy

    Open Targets python framework for post-GWAS analysis

    GitHub · Python · 62 stars · 17 forks · 0 stars in 24 hours · Trending score 0.10

  10. 10
    andfanilo/pyspark-tutorial

    Jupyter notebooks for pyspark tutorials given at University

    GitHub · Jupyter Notebook · 111 stars · 74 forks · 0 stars in 24 hours · Trending score 0.08

  11. 11
    khuyentran1401/codecut-blog

    45+ production-ready tutorials on data science, MLOps, and AI tools. All code is executable and adaptable for real projects.

    GitHub · Jupyter Notebook · 42 stars · 12 forks · 0 stars in 24 hours · Trending score 0.05

  12. 12
    Matt17BR/openwrangler

    Open Wrangler is an open-source visual dataframe editor for VS Code and editors based on it. Explore files or data loaded in memory, make changes through a GUI, and generate Python or R code.

    GitHub · TypeScript · 7 stars · 1 forks · 0 stars in 24 hours · Trending score 0.05

  13. 13
    animetanver-commits/fabric-unified-data-blueprint

    Microsoft Fabric Unified Data Foundation with Databricks & Purview 2026

    GitHub · HTML · 114 stars · 0 forks · 0 stars in 24 hours · Trending score 0.04

  14. 14
    apache/incubator-graphar

    An open source, standard data file format for graph data storage and retrieval.

    GitHub · Java · 384 stars · 97 forks · 0 stars in 24 hours · Trending score 0.04

  15. 15
    mahmoudparsian/big-data-modeling

    Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University

    GitHub · HTML · 167 stars · 144 forks · 0 stars in 24 hours · Trending score 0.04

  16. 16
    datasifaw/databricks-projet-bancaire-lakehouse

    Projet Data Engineering bancaire de bout en bout avec Databricks : ingestion des données clients, comptes et transactions depuis SQL Server, traitement en architecture Bronze-Silver-Gold, transformations PySpark, stockage Delta Lake, contrôles de qualité, orchestration via Workflows et création d’indicateurs financiers pour un dashboard BI fiable

    GitHub · TSQL · 35 stars · 0 forks · 0 stars in 24 hours · Trending score 0.03

  17. 17
    zaratsian/Spark

    Apache Spark (Scala, PySpark, SparkR) Code, Tricks, and References

    GitHub · Jupyter Notebook · 69 stars · 37 forks · 0 stars in 24 hours · Trending score 0.03

  18. 18
    maurinruneacco/luau-diagrams-and-defects

    Production Data Engineering Cloud Architecture Workflow Automation Portfolio 2026

    GitHub · HTML · 69 stars · 0 forks · 0 stars in 24 hours · Trending score 0.02

  19. 19
    pavanbadempet/AI-Healthcare-System

    AI Healthcare System & Clinical Data Platform. Combines a Databricks Medallion Lakehouse (OHDSI OMOP CDM v5.4 & FHIR R4) with TabICLv2 tabular foundation models, calibrated tree ensembles (CatBoost, XGBoost), a 10-year coupled ODE digital twin simulator, LangGraph multi-agent RAG, and an offline HIPAA-compliant clinical decision support portal.

    GitHub · Python · 66 stars · 16 forks · 0 stars in 24 hours · Trending score 0.01

  20. 20
    mlverse/pysparklyr

    Extension to {sparklyr} that allows you to interact with Spark & Databricks Connect

    GitHub · R · 19 stars · 10 forks · 0 stars in 24 hours · Trending score 0.00

  21. 21
    mitchelllisle/sparkdantic

    ✨ A Pydantic to PySpark schema library

    GitHub · Python · 134 stars · 22 forks · 0 stars in 24 hours · Trending score 0.00

  22. 22
    JohnSnowLabs/spark-nlp

    State of the Art Natural Language Processing

    GitHub · Scala · 4,160 stars · 743 forks · 0 stars in 24 hours · Trending score 0.00

  23. 23
    shauryashaurya/The-Silmaril

    Practice #ontology engineering with Python (and other languages).

    GitHub · Jupyter Notebook · 38 stars · 9 forks · 0 stars in 24 hours · Trending score 0.00

  24. 24
    zorost/AI-Engineering-Lab

    A free, self-paced 24-week AI engineering course: Python, machine learning, LLMs, RAG, fine-tuning, agents and MCP, Azure and Vertex and Bedrock, and Databricks. 43 runnable notebooks, one continuous case study. MIT licensed, no signup. By Zorost Intelligence AI Lab.

    GitHub · Jupyter Notebook · 316 stars · 256 forks · 0 stars in 24 hours · Trending score 0.00