Citing NDIF

ICLR 2025Conference Paper

NNsight and NDIF: Democratizing Access to Foundation Model Internals

If you use NNsight or NDIF resources in your research, please cite the following:

BibTeX

@inproceedings{fiotto-kaufman2025nnsight,
  title={{NNsight} and {NDIF}: Democratizing Access to Foundation Model Internals},
  author={Jaden Fried Fiotto-Kaufman and Alexander Russell Loftus and Eric Todd and Jannik Brinkmann and Koyena Pal and Dmitrii Troitskii and Michael Ripa and Adam Belfki and Can Rager and Caden Juang and Aaron Mueller and Samuel Marks and Arnab Sen Sharma and Francesca Lucchetti and Nikhil Prakash and Carla E. Brodley and Arjun Guha and Jonathan Bell and Byron C Wallace and David Bau},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025},
  url={https://openreview.net/forum?id=MxbEiFRf39}
}

When you publish work using NNsight or NDIF resources, we'd love you to email us directly at info@ndif.us to tell us about your work. This helps us track our impact and supports our continued efforts to provide open-source resources for reproducible and transparent research on large-scale AI systems.

Research Using NDIF

32 papers across top venues including ICLR, NeurIPS, ICML, EMNLP, and more.

ICLR 2025 Workshop on Foundation Models in the WildReferences NDIF

DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products

Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, Riccardo Grazzi

Presents DeltaProduct, a linear RNN that improves expressivity while maintaining efficiency by using a diagonal plus rank-nh transition matrix built from Householder transforms.

ArXiv 2025References NDIF

Robustly identifying concepts introduced during chat fine-tuning using crosscoders

Julian Minder, Clement Dumas, Caden Juang, Bilal Chugtai, Neel Nanda

Improves model diffing techniques for understanding how fine-tuning alters language model behavior, introducing Latent Scaling and a new BatchTopK loss.

ArXiv 2025References NDIF

The Dual-Route Model of Induction

Sheridan Feucht, Eric Todd, Byron Wallace, David Bau

Identifies concept-level induction heads—attention heads that copy entire lexical units rather than individual tokens—and shows they specialize in semantic tasks like translation.

ICLR 2025 Workshop on Building Trust in Language ModelsReferences NDIF

Steering Fine-Tuning Generalization with Targeted Concept Ablation

Helena Casademunt, Caden Juang, Samuel Marks, Senthooran Rajamanoharan, Neel Nanda

Introduces a method for steering fine-tuned models toward intended generalizations by identifying and ablating sparse autoencoder latents linked to undesired concepts.

ICLR 2025References NDIF

ICLR: In-Context Learning of Representations

Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, Hidenori Tanaka

Explores whether LLMs can reorganize internal concept representations based on in-context examples that conflict with pretrained semantics.

ArXiv 2025References NDIF

The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence

Tom Wollschläger, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan Günnemann, Johannes Gasteiger

Introduces a gradient-based method to identify and analyze internal mechanisms behind refusal in large language models, finding multiple mechanistically independent directions.

NeurIPS 2024 Workshop on Symmetry and Geometry in Neural RepresentationsReferences NDIF

Structured In-Context Task Representations

Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Kento Nishi, Maya Okawa, Hidenori Tanaka

Investigates whether language models develop interpretable internal representations during in-context learning using synthetic data based on geometric structures.

ArXiv 2025References NDIF

Elucidating Mechanisms of Demographic Bias in LLMs for Healthcare

Hiba Ahsan, Arnab Sen Sharma, Silvio Amir, David Bau, Byron C. Wallace

Uncovers how large language models encode sociodemographic information in the context of healthcare, representing the first application of interpretability methods to study such biases.

ArXiv 2025References NDIF

Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models

Zeping Yu, Yonatan Belinkov, Sophia Ananiadou

Explores how LLMs perform multi-hop reasoning using logit flow, proposing back attention to improve reasoning accuracy across multiple models.

ArXiv 2025References NDIF

Language Models Use Trigonometry to Do Addition

Subhash Kantamneni, Max Tegmark

Discovers that LLMs represent numbers as a generalized helix and proposes the "Clock" algorithm to explain how LLMs perform mathematical operations.

ArXiv 2025References NDIF

Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages

Jannik Brinkmann, Chris Wendler, Christian Bartelt, Aaron Mueller

Finds that abstract grammatical features like number, gender, and tense are encoded in shared multilingual directions in LLMs.

ICMLA 2024References NDIF

Hidden Pieces: An Analysis of Linear Probes for GPT Representation Edits

Austin L Davis, Gita Sukthankar

Explores the use of probing classifiers to analyze and intervene in the internal representations of a chess-playing transformer model.

ArXiv 2024References NDIF

Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models

Michael Hanna, Aaron Mueller

Investigates how autoregressive language models process garden path sentences, revealing that LMs use both syntactic features and shallow heuristics.

ArXiv 2024References NDIF

Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers

Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, Robert West

Investigates whether LLMs develop language-agnostic concept representations by analyzing latent activations during translation tasks.

BiorXiv 2024References NDIF

InterPLM: Discovering Interpretable Features in Protein Language Models via Sparse Autoencoders

Elana Simon, James Zou

Introduces a method for interpreting protein language models by using sparse autoencoders to extract human-interpretable biological features.

EMNLP 2024References NDIF

Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs

Sheridan Feucht, David Atkinson, Byron Wallace, David Bau

Investigates how LLMs transform token groups into higher-level representations, identifying a pronounced "erasure" effect in early layers.

NeurIPS 2024References NDIF

Evidence of Learned Look-Ahead in a Chess-Playing Neural Network

Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, Stuart Russell

Presents evidence of learned look-ahead in the policy network of Leela Chess Zero, showing it internally represents future optimal moves.

ArXiv 2024References NDIF

Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing

Kento Nishi, Maya Okawa, Rahul Ramesh, Mikail Khona, Ekdeep Singh Lubana, Hidenori Tanaka

Investigates why Knowledge Editing methods in LLMs can degrade factual recall and reasoning, proposing the concept of representation shattering.

ACM 2024References NDIF

A Generative Benchmark Creation Framework for Detecting Common Data Table Versions

Daniel C. Fox, Aamod Khatiwada, Roee Shraga

Introduces a framework using LLMs to generate benchmarks for data versioning, releasing VerLLM-v1 with detailed documentation and complex transformations.

NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep LearningReferences NDIF

Emergence of Hierarchical Emotion Representations in Large Language Models

Bo Zhao, Maya Okawa, Eric J Bigelow, Rose Yu, Tomer Ullman, Hidenori Tanaka

Explores how LLMs represent and predict human emotions, revealing that larger models develop more complex hierarchical emotion structures.

ArXiv 2024References NDIF

Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small

Maheep Chaudhary, Atticus Geiger

Evaluates high-dimensional sparse autoencoders for causal analysis in mechanistic interpretability using the RAVEL benchmark on GPT-2 small.

COLM 2024References NDIF

Locating and Editing Factual Associations in Mamba

Arnab Sen Sharma, David Atkinson, David Bau

Investigates factual recall mechanisms in the Mamba state space model, finding key components localized in middle layers similar to transformers.

ArXiv 2024References NDIF

Benchmarking Mental State Representations in Language Models

Matteo Bortoletto, Constantin Ruhdorfer, Lei Shi, Andreas Bulling

Conducts a benchmark study on internal representation of mental states in language models, analyzing model sizes, fine-tuning strategies, and prompt designs.

ICML 2024 Workshop on Mechanistic InterpretabilityReferences NDIF

How do Llamas process multilingual text? A latent exploration through activation patching

Clément Dumas, Veniamin Veselovsky, Giovanni Monea, Robert West, Chris Wendler

Analyzes Llama-2's forward pass during word translation to explore language-agnostic concept representations, providing evidence for universal concept representation.

ICML 2024References NDIF

Language Models Represent Beliefs of Self and Others

Wentao Zhu, Zhining Zhang, Yizhou Wang

Investigates Theory of Mind abilities in LLMs, identifying internal representations of self and others' beliefs through neural activations.

ArXiv 2024References NDIF

The Quest for the Right Mediator: A History, Survey, and Theoretical Grounding of Causal Interpretability

Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, Yonatan Belinkov

A comprehensive history, survey, and theoretical grounding of causal interpretability methods for neural networks.

ArXiv 2024References NDIF

Penzai + Treescope: A Toolkit for Interpreting, Visualizing, and Editing Models As Data

Daniel D. Johnson

A toolkit for interpreting, visualizing, and editing neural network models treated as structured data.

ArXiv 2024References NDIF

Comgra: A Tool for Analyzing and Debugging Neural Networks

Florian Dietz, Sophie Fellenz, Dietrich Klakow, Marius Kloft

A tool for comprehensive analysis and debugging of neural network computations.

ACM 2024References NDIF

Black-Box Access is Insufficient for Rigorous AI Audits

Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Krueger, Dylan Hadfield-Menell

Argues that black-box access alone is insufficient for rigorous audits of AI systems and makes the case for open access to model internals.

ArXiv 2024References NDIF

A Primer on the Inner Workings of Transformer-based Language Models

Javier Ferrando, Gabriele Sarti, Arianna Bisazza, Marta R. Costa-jussà

An accessible primer on how transformer-based language models work internally, covering attention mechanisms, representations, and interpretability.

ArXiv 2024References NDIF

What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation

Aaditya K. Singh, Ted Moskovitz, Felix Hill, Stephanie C.Y. Chan, Andrew M. Saxe

A mechanistic study of in-context learning circuits, specifically induction heads, and the conditions required for their formation during training.

ArXiv 2024References NDIF

Pyvene: A Library for Understanding and Improving PyTorch Models via Interventions

Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah D. Goodman, Christopher D. Manning, Christopher Potts

A Python library for understanding and improving PyTorch models through causal interventions on model internals.