interpretability repositories
Trending repositories tagged interpretability, ranked by trending score. Updated daily.
-
1
lyramakesmusic/neuron-explorer
Steering-first neuron explorer for Qwen3.5-4B: push every MLP neuron, keep every rollout, browse the results
GitHub · Python · 2,030 stars · 253 forks · +29 stars in 24 hours · Trending score 2.65
-
2
divagr18/qwenputer
A computer, compiled into the weights of a language model. An exact CPU inside Qwen3.5-4B, with Gated DeltaNet state as RAM.
GitHub · Python · 830 stars · 113 forks · +3 stars in 24 hours · Trending score 1.78
-
3
stdlogits/neuron-lens
A catalog of every MLP neuron and attention head of Qwen3-8B and Gemma 4 12B, read from the weights through two Jacobian lenses (R and S), from activations, and from their exact token drivers
GitHub · Python · 57 stars · 5 forks · +4 stars in 24 hours · Trending score 1.39
-
4
google-deepmind/detectorscope
Build AI text detectors whose decisions are mechanistically grounded and human-auditable (EMNLP 2026)
GitHub · Python · 496 stars · 44 forks · +2 stars in 24 hours · Trending score 1.38
-
5
stdlogits/neuron-reflection
Swap, fuse, remove or split a concept in Qwen3-8B's weights, without training, by mirroring the neurons and attention heads that carry it
GitHub · Python · 16 stars · 2 forks · +11 stars in 24 hours · Trending score 1.30
-
6
whp199/theodolite
An atlas of all 1,290,240 neurons in Gemma-4-31B-it, a census of what they do when pushed, and the instrument to read them with.
GitHub · Python · 39 stars · 5 forks · +2 stars in 24 hours · Trending score 1.27
-
7
manuelgscom/inside-llm
See inside any GGUF: where two quantizations part ways, token by token, and what each sampler cuts. Live in your browser, powered by llama.cpp (CUDA or Vulkan).
GitHub · TypeScript · 108 stars · 9 forks · +11 stars in 24 hours · Trending score 1.20
-
8
di37/animated-machine-learning-algorithm
Watch scikit-learn models learn and predict: live training and inference animations, narrated lessons and Manim videos for California Housing (regression) and Titanic (classification)
GitHub · Python · 28 stars · 0 forks · +9 stars in 24 hours · Trending score 1.16
-
9
Mattbusel/Every-Other-Token
See how sure an LLM is about each word.
GitHub · Rust · 31 stars · 1 forks · +4 stars in 24 hours · Trending score 1.13
-
10
microsoft/responsible-ai-toolbox
Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.
GitHub · TypeScript · 1,858 stars · 511 forks · 0 stars in 24 hours · Trending score 1.04
-
11
EthicalML/awesome-production-machine-learning
A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning
GitHub · 21,098 stars · 2,618 forks · 0 stars in 24 hours · Trending score 0.95
-
12
shap/shap
A game theoretic approach to explain the output of any machine learning model.
GitHub · Jupyter Notebook · 25,862 stars · 3,796 forks · 0 stars in 24 hours · Trending score 0.95
-
13
eomcaleb/weight-space-bench
Benchmark: can an LLM make a language model better at math by editing its weights directly? (Qwen3.5-0.8B, MATH-500/GSM8K, via OpenRouter)
GitHub · Python · 18 stars · 2 forks · +2 stars in 24 hours · Trending score 0.92
-
14
mainlp/explaind
A unified framework for attributing model behavior to model components, training data, and training dynamics.
GitHub · Python · 12 stars · 0 forks · +2 stars in 24 hours · Trending score 0.73
-
15
liusida/ica-lens-paper
ICA Lens: compact ICA-based interpretability tools for exploring LLM activations. Code release for the paper.
GitHub · Python · 61 stars · 7 forks · +1 stars in 24 hours · Trending score 0.64
-
16
Miyamoto-Renya/LaplacesScissors
Laplace's Scissors: protected readout rotations for open LLMs. Diagnose weak output directions, repair answers, preserve successful generations, and analyze scaling deviations with Rust and Triton on CUDA/HIP.
GitHub · Rust · 33 stars · 8 forks · +1 stars in 24 hours · Trending score 0.61
-
17
conceptcountry/weight-space-programming
Gradient-free weight editing reduces overthinking in Qwen3.8-27B — analysis code and evaluation artifacts
GitHub · Typst · 31 stars · 3 forks · 0 stars in 24 hours · Trending score 0.56
-
18
chessformer-lens/chessformer_lens
A toolkit+visualizer that is designed for mech interp researchers working with chess models that use transformers tokenized by square.
GitHub · Python · 10 stars · 0 forks · +1 stars in 24 hours · Trending score 0.55
-
19
OpenBMB/ParamMute
[NeurIPS 2025] ParamMute (a.k.a. PIP-KAG): suppressing knowledge-critical FFNs for faithful retrieval-augmented generation (RAG knowledge conflicts)
GitHub · Python · 64 stars · 5 forks · +1 stars in 24 hours · Trending score 0.55
-
20
oumi-ai/weights-lab
Agent skill for weight-space programming: edit LLM weights
GitHub · Python · 8 stars · 0 forks · +1 stars in 24 hours · Trending score 0.47
-
21
johnmarktaylor91/torchlens
Inspect, visualize and intervene on any operation in any neural network. Capture every activation and gradient, the full computational graph and 800+ metadata fields without changing model code. Supports PyTorch, JAX, TensorFlow, MLX, PaddlePaddle and tinygrad. Validated on 6400+ models. Built for researchers and their AI agents.
GitHub · Python · 714 stars · 35 forks · 0 stars in 24 hours · Trending score 0.44
-
22
csinva/weight-space-programming-autoresearch
🧠 Let the agent write the weights, why not?
GitHub · Python · 48 stars · 6 forks · 0 stars in 24 hours · Trending score 0.44
-
23
EleutherAI/bergson
Mapping out the "memory" of neural nets with data attribution
GitHub · Python · 110 stars · 35 forks · 0 stars in 24 hours · Trending score 0.42
-
24
astro-weights/astro-weights
Small course corrections for open LLMs. Agent-driven weight edits. Democratizing AI for all.
GitHub · Python · 8 stars · 3 forks · +1 stars in 24 hours · Trending score 0.42