interpretability repositories

Trending repositories tagged interpretability, ranked by trending score. Updated daily.

  1. 1
    lyramakesmusic/neuron-explorer

    Steering-first neuron explorer for Qwen3.5-4B: push every MLP neuron, keep every rollout, browse the results

    GitHub · Python · 2,030 stars · 253 forks · +29 stars in 24 hours · Trending score 2.65

  2. 2
    divagr18/qwenputer

    A computer, compiled into the weights of a language model. An exact CPU inside Qwen3.5-4B, with Gated DeltaNet state as RAM.

    GitHub · Python · 830 stars · 113 forks · +3 stars in 24 hours · Trending score 1.78

  3. 3
    stdlogits/neuron-lens

    A catalog of every MLP neuron and attention head of Qwen3-8B and Gemma 4 12B, read from the weights through two Jacobian lenses (R and S), from activations, and from their exact token drivers

    GitHub · Python · 57 stars · 5 forks · +4 stars in 24 hours · Trending score 1.39

  4. 4
    google-deepmind/detectorscope

    Build AI text detectors whose decisions are mechanistically grounded and human-auditable (EMNLP 2026)

    GitHub · Python · 496 stars · 44 forks · +2 stars in 24 hours · Trending score 1.38

  5. 5
    stdlogits/neuron-reflection

    Swap, fuse, remove or split a concept in Qwen3-8B's weights, without training, by mirroring the neurons and attention heads that carry it

    GitHub · Python · 16 stars · 2 forks · +11 stars in 24 hours · Trending score 1.30

  6. 6
    whp199/theodolite

    An atlas of all 1,290,240 neurons in Gemma-4-31B-it, a census of what they do when pushed, and the instrument to read them with.

    GitHub · Python · 39 stars · 5 forks · +2 stars in 24 hours · Trending score 1.27

  7. 7
    manuelgscom/inside-llm

    See inside any GGUF: where two quantizations part ways, token by token, and what each sampler cuts. Live in your browser, powered by llama.cpp (CUDA or Vulkan).

    GitHub · TypeScript · 108 stars · 9 forks · +11 stars in 24 hours · Trending score 1.20

  8. 8
    di37/animated-machine-learning-algorithm

    Watch scikit-learn models learn and predict: live training and inference animations, narrated lessons and Manim videos for California Housing (regression) and Titanic (classification)

    GitHub · Python · 28 stars · 0 forks · +9 stars in 24 hours · Trending score 1.16

  9. 9
    Mattbusel/Every-Other-Token

    See how sure an LLM is about each word.

    GitHub · Rust · 31 stars · 1 forks · +4 stars in 24 hours · Trending score 1.13

  10. 10
    microsoft/responsible-ai-toolbox

    Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.

    GitHub · TypeScript · 1,858 stars · 511 forks · 0 stars in 24 hours · Trending score 1.04

  11. 11
    EthicalML/awesome-production-machine-learning

    A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning

    GitHub · 21,098 stars · 2,618 forks · 0 stars in 24 hours · Trending score 0.95

  12. 12
    shap/shap

    A game theoretic approach to explain the output of any machine learning model.

    GitHub · Jupyter Notebook · 25,862 stars · 3,796 forks · 0 stars in 24 hours · Trending score 0.95

  13. 13
    eomcaleb/weight-space-bench

    Benchmark: can an LLM make a language model better at math by editing its weights directly? (Qwen3.5-0.8B, MATH-500/GSM8K, via OpenRouter)

    GitHub · Python · 18 stars · 2 forks · +2 stars in 24 hours · Trending score 0.92

  14. 14
    mainlp/explaind

    A unified framework for attributing model behavior to model components, training data, and training dynamics.

    GitHub · Python · 12 stars · 0 forks · +2 stars in 24 hours · Trending score 0.73

  15. 15
    liusida/ica-lens-paper

    ICA Lens: compact ICA-based interpretability tools for exploring LLM activations. Code release for the paper.

    GitHub · Python · 61 stars · 7 forks · +1 stars in 24 hours · Trending score 0.64

  16. 16
    Miyamoto-Renya/LaplacesScissors

    Laplace's Scissors: protected readout rotations for open LLMs. Diagnose weak output directions, repair answers, preserve successful generations, and analyze scaling deviations with Rust and Triton on CUDA/HIP.

    GitHub · Rust · 33 stars · 8 forks · +1 stars in 24 hours · Trending score 0.61

  17. 17
    conceptcountry/weight-space-programming

    Gradient-free weight editing reduces overthinking in Qwen3.8-27B — analysis code and evaluation artifacts

    GitHub · Typst · 31 stars · 3 forks · 0 stars in 24 hours · Trending score 0.56

  18. 18
    chessformer-lens/chessformer_lens

    A toolkit+visualizer that is designed for mech interp researchers working with chess models that use transformers tokenized by square.

    GitHub · Python · 10 stars · 0 forks · +1 stars in 24 hours · Trending score 0.55

  19. 19
    OpenBMB/ParamMute

    [NeurIPS 2025] ParamMute (a.k.a. PIP-KAG): suppressing knowledge-critical FFNs for faithful retrieval-augmented generation (RAG knowledge conflicts)

    GitHub · Python · 64 stars · 5 forks · +1 stars in 24 hours · Trending score 0.55

  20. 20
    oumi-ai/weights-lab

    Agent skill for weight-space programming: edit LLM weights

    GitHub · Python · 8 stars · 0 forks · +1 stars in 24 hours · Trending score 0.47

  21. 21
    johnmarktaylor91/torchlens

    Inspect, visualize and intervene on any operation in any neural network. Capture every activation and gradient, the full computational graph and 800+ metadata fields without changing model code. Supports PyTorch, JAX, TensorFlow, MLX, PaddlePaddle and tinygrad. Validated on 6400+ models. Built for researchers and their AI agents.

    GitHub · Python · 714 stars · 35 forks · 0 stars in 24 hours · Trending score 0.44

  22. 22
    csinva/weight-space-programming-autoresearch

    🧠 Let the agent write the weights, why not?

    GitHub · Python · 48 stars · 6 forks · 0 stars in 24 hours · Trending score 0.44

  23. 23
    EleutherAI/bergson

    Mapping out the "memory" of neural nets with data attribution

    GitHub · Python · 110 stars · 35 forks · 0 stars in 24 hours · Trending score 0.42

  24. 24
    astro-weights/astro-weights

    Small course corrections for open LLMs. Agent-driven weight edits. Democratizing AI for all.

    GitHub · Python · 8 stars · 3 forks · +1 stars in 24 hours · Trending score 0.42