Llama

The block, in order

h   = x + self_attn(input_layernorm(x))
out = h + mlp(post_attention_layernorm(h))

Two RMSNorms per block, each before its sublayer, none after. post_attention_layernorm is named for where it sits on the stream, after the attention's add: it is the MLP's input norm. Some families with a norm after each sublayer give the name to a different module (on Gemma-2 it is the norm after the attention), so check the block's forward before porting a hook by name.

The contributions are the modules' outputs

attention_output is self_attn.output[0] and mlp_output is mlp.output: the same tensors, added to the stream with nothing in between. The identity holds exactly, in float32 on the pinned checkpoint and in bfloat16 on Llama-3.2-1B:

with model.trace(prompt):
    x = model.layers[5].input.save()
    attn = model.layers[5].self_attn.attention_output.save()
    mlp = model.layers[5].mlp.mlp_output.save()
    out = model.layers[5].layer_output.save()

torch.testing.assert_close(x + attn + mlp, out)

So scaling a module's output scales its contribution: model.layers[i].mlp.output[:] *= 0.5 and model.layers[i].mlp.mlp_output[:] *= 0.5 give the same logits. The inputs are the norms' outputs: self_attn.input is input_layernorm.output, mlp.input is post_attention_layernorm.output, which nnterp serves as attention_norm_output and mlp_norm_output. The stream between the two adds is model.layers[i].layer_mid, read as post_attention_layernorm.input; it equals model.layers[i].input + model.layers[i].self_attn.attention_output.

The default load runs the same attention as eager

A default load runs sdpa, and the six attention interior values report unavailable; attn_implementation="eager" serves them. Llama has no score softcap, window or sink, so the two paths compute the same function: on Llama-3.2-1B in float32 their logits differ by about 2e-5, and in bfloat16 by up to 0.25 with the same top token at every position of a test prompt. The query scale is head_dim ** -0.5 (128 ** -0.5 on 8B, 64 ** -0.5 on 3.2-1B).

Grouped-query attention on Llama-3 and Llama-2-70B

8B and 3.2-1B have 32 query heads over 8 key/value heads, 3.2-3B 24 over 8, 70B (Llama-2 and Llama-3) 64 over 8, 405B 128 over 8. attention_keys and attention_values are read before repeat_kv, [batch, 8, seq, head_dim], and query head h reads key/value head h // 4 on 8B: an edit to key/value head j reaches query heads 4j to 4j + 3, a contiguous block. Llama-2-7B, Code Llama 7B and the pinned checkpoint have one key/value head per query head, so the same code edits one head there.

attn = model.layers[5].self_attn
groups = model.num_heads // model.num_kv_heads   # 4 on 8B and 3.2-1B
with model.trace(prompt):
    attn.attention_values[:, 1] = 0             # key/value head 1
    heads = attn.attention_head_outputs.save()  # only query heads 4 to 7 change

Queries and keys are read after the rotary embedding

attention_queries and attention_keys are rotated by position; the projections before it are the outputs of q_proj and k_proj. The two agree at position 0, where the rotation is the identity, and nowhere else. A key or query moved to another position keeps the rotation of the position it was read at. The cosines and sines are computed once per forward by the native model.model.rotary_emb and passed to every block. Llama-3.1, 3.2 and 3.3 use rope_type llama3 (rope_theta 500000, low frequencies rescaled by factor), Meta-Llama-3 the plain rotary with rope_theta 500000, and Llama-2 the plain rotary with rope_theta 10000. Both sides can be read in one trace:

attn = model.layers[5].self_attn
with model.trace(prompt):
    q_raw = attn.source.self_q_proj_0.output.save()   # before the rotary
    q = attn.attention_queries.save()                 # after it

Position 0 is a sink with a very large norm

On Llama-3.2-1B the residual stream at position 0 has a norm near 420 after blocks 1 and 5, against a mean of 2.4 and 4.5 at the other positions, and the heads of blocks 0, 1, 5, 8 and 15 put 0.6 to 0.85 of their attention on it on average, by block and prompt. The Llama-3 tokenizer prepends its beginning-of-text token (id 128000); without it the first real token takes the role (norm 865 after block 5). A mean over positions that includes position 0 is dominated by it: slice [:, 1:] before averaging activations for steering vectors, mean ablation or probes.

with model.trace(prompt):
    resid = model.layers[5].layer_output.save()

resid[0].float().norm(dim=-1)        # 422 at position 0, 4 to 5 elsewhere on 3.2-1B

Target tokens differ by tokenizer

On Llama-3, "Paris" (60704) and " Paris" (12366) are two whole-word tokens, and get_first_tokens(["Paris"], model) returns both. Llama-2's SentencePiece tokenizer prefixes the word marker itself: "Paris" and " Paris" both encode to ▁Paris (3681), and get_first_tokens(["Paris"], model) returns [2177, 3681], where 2177 is the mid-word fragment Par. Decode the target ids on the checkpoint you run.

The readout and the embeddings

model.logits is model.lm_head.output: no cap and no scale, so project_on_vocab is lm_head(norm(hidden)) and a logit lens at the last block equals logits. The final norm's gain is model.norm.weight as stored. Llama-3.2-1B and 3B tie lm_head to embed_tokens (one parameter), so an edit to embed_tokens.weight is an edit to the unembedding; the other Meta checkpoints do not.

What this family module covers

The module serves every checkpoint whose config says model_type llama: besides Meta's Llama-2, Llama-3, 3.1, 3.2 and 3.3, that includes Code Llama, TinyLlama, DeepSeek-R1-Distill-Llama-8B, SmolLM2 and Helium-1-2B. Their settings differ: SmolLM2-135M ties its embeddings, has 9 query heads over 3 key/value heads, and its tokenizer adds no beginning-of-sequence token; Code Llama 7B has rope_theta 1000000. Llama 4 is the llama4_text family.

Sparse autoencoders and transcoders

Llama Scope (OpenMOSS-Team/Llama-Scope), on Llama-3.1-8B, every block. The R SAEs read the stream after the block, model.layers[i].layer_output; A reads model.layers[i].self_attn.attention_output; M reads model.layers[i].mlp.mlp_output. The TC transcoders read the normed stream, model.layers[i].mlp_input, and predict mlp_output. Inputs are scaled to a norm of sqrt(hidden_size) before encoding.
EleutherAI EleutherAI/sae-llama-3-8b-32x, on Meta-Llama-3-8B: residual-stream SAEs whose hook points layers.i are the block outputs, model.layers[i].layer_output, and embed_tokens is model.token_embeddings.
circuit-tracer transcoders on Llama-3.2-1B, per-layer (mntss/transcoder-Llama-3.2-1B) and cross-layer (mntss/clt-llama-3.2-1b-524k). They read hook_resid_mid, the stream before the MLP's norm, model.layers[i].layer_mid, and write hook_mlp_out, model.layers[i].mlp.mlp_output:

with model.trace(prompt):
    mid = model.layers[i].layer_mid.save()           # hook_resid_mid
    out = model.layers[i].mlp.mlp_output.save()      # hook_mlp_out

The CLIP vision encoder

The CLS token comes first

vision.patch_embeddings is the patch grid alone, [images, grid, vision_hidden_size]. The class embedding is prepended after it, so from vision.layers[0].input on the stream is one row longer: [:, 0] is the CLS token and [:, 1:] the patches in raster order. Slice [:, 1:] before mapping a row to a patch.

A norm before block 0, none after the last

The position embeddings are added to the CLS token and the patches, and pre_layrnorm norms the result: vision.layers[0].input is pre_layrnorm's output, not patch_embeddings plus positions. After the last block, post_layernorm norms the pooled CLS token only, never the patches, so it is not vision.norm, and vision.tower_output equals vision.layers[-1].layer_output.

Llava 1.5

The projector reads block -2, without the CLS token

vision_feature_layer is -2 and vision_feature_select_strategy is "default": the projector receives vision.layers[-2].layer_output[:, 1:], so the last block and vision.tower_output are computed and discarded. On llava-hf/llava-1.5-7b-hf, asked the colour of a red square, zeroing vision.tower_output leaves Red at 0.990, and zeroing vision.layers[-2].layer_output drops it to 0.001.

with model.trace(prompt, images=[image]):
    stream = model.vision.layers[-2].layer_output.save()
    fed = model.projector.input.save()

torch.equal(fed, stream[:, 1:])   # True

One token per patch

The projector maps each patch to one token, so an image is 576 image tokens on 7B and 13B (24 × 24 patches of 14 pixels at 336), and vision.image_features is model.projector.output flattened over the images: [576, hidden_size] per image.

The CLIP vision encoder

The CLS token comes first

A norm before block 0, none after the last

VipLlava

The projector reads five blocks

vision_feature_layers is [-2, -5, -8, -11, 6] on vip-llava-7b-hf, indices into the vision encoder's hidden states, where 0 is the stream entering block 0 and k > 0 is block k - 1's layer_output. So the projector reads vision.layers[i].layer_output of blocks 22, 19, 16, 13 and 5, each without its CLS token, concatenated on the last axis: projector.input is [images, 576, 5120], and projector_layernorm norms it before the MLP. The last block and vision.tower_output are computed and discarded. vision.image_features is model.projector.output flattened over the images.

streams = []
with model.trace(prompt, images=[image]):
    for k in (5, 13, 16, 19, 22):
        streams.append(model.vision.layers[k].layer_output.save())
    fed = model.projector.input.save()

order = [4, 3, 2, 1, 0]   # the config's order: blocks 22, 19, 16, 13, 5
torch.equal(fed, torch.cat([streams[j][:, 1:] for j in order], -1))   # True

The CLIP vision encoder

The CLS token comes first

A norm before block 0, none after the last

LLaVA-NeXT

Crops as rows, and features that are not the projector's output

The processor cuts an image into a base image and crops at a resolution from image_grid_pinpoints, each a row of the vision encoder's batch, and the projector reads block -2 without the CLS token as on Llava 1.5 (vision_feature_layer is -2 on llava-v1.6-vicuna-7b-hf). The wrapper then unpads the projector's output to the image's aspect ratio and appends image_newline after each row of patches, so vision.image_features has a different number of rows from model.projector.output. On the pinned tiny checkpoint the projector returns 8 patch rows (the base image and one crop, 4 patches each) and vision.image_features 10, 2 of them image_newline. Edit vision.image_features for what the text model receives.

The SigLIP vision encoder

The patch axis is the patch grid

No class token: vision.patch_embeddings, every block's layer_output and vision.tower_output have the same length, (image_size // patch_size) ** 2 patches per image or tile, in raster order. Row r, column c of the grid is index r * (image_size // patch_size) + c on every value of the vision encoder.

The final norm is `vision.norm`

post_layernorm norms every patch after the last block, so vision.tower_output == vision.norm(vision.layers[-1].layer_output). Each wrapper's notes say whether its projector reads tower_output or a block's layer_output before the norm.

DeepSeek-VL

The aligner reads `tower_output`

model.aligner is the projector: linear1, GELU, linear2, applied to vision.tower_output, the patches after post_layernorm. So model.projector.input == model.vision.tower_output, a write to the last block reaches the text model through the norm, and vision.image_features is model.projector.output flattened. An image is 576 image tokens on 1.3B (24 × 24 patches of 16 pixels at 384), the processor's num_image_tokens.

The SigLIP vision encoder

The patch axis is the patch grid

The final norm is `vision.norm`

Idefics 3

Tiles as rows, pixel-shuffled into tokens

The processor splits an image into tiles of image_size pixels and appends the whole image resized to one tile, each a row of the vision encoder's batch. model.connector (the projector) reads vision.tower_output, folds each scale_factor × scale_factor block of neighbouring patches into one token, their widths concatenated, and projects it with modality_projection, a linear without bias. On Idefics3-8B-Llama3 (scale_factor 2) a 364-pixel tile's 26 × 26 = 676 patches become 169 tokens; HuggingFaceTB/SmolVLM-Instruct is an idefics3 checkpoint with scale_factor 3, so its 384-pixel tile's 729 patches become 81.

The features go in through `inputs_merger`

vision.image_features is read at inputs_merger's image_hidden_states argument, which is model.projector.output flattened, tiles in order: layers[0].input[vision.image_token_mask] == vision.image_features holds as on every wrapper.

The SigLIP vision encoder

The patch axis is the patch grid

The final norm is `vision.norm`

SmolVLM

Idefics 3's layout, with a 3 × 3 shuffle

SmolVLM keeps Idefics 3's vision encoder, connector and merge: tiles as rows of the vision encoder's batch, the connector reading vision.tower_output, and vision.image_features read at inputs_merger's image_hidden_states, model.projector.output flattened. On SmolVLM2-2.2B-Instruct the scale_factor is 3, so a 384-pixel tile's 27 × 27 = 729 patches become 81 tokens.

Llama (LlamaForCausalLM), the family the standard vocabulary is taken from.

Block names (input_layernorm, self_attn, post_attention_layernorm, mlp) and lm_head are already the standard ones. The only change is lifting the containers out of model.model: model.layers instead of model.model.layers. Multimodal wrappers around a Llama text model keep it at model.language_model (Llava, DeepSeek-VL, Janus) or model.text_model (Idefics 3, SmolVLM); RENAME carries those spellings too.

Llava 1.5 (LlavaForConditionalGeneration, model_type llava), VipLlava and LLaVA-NeXT add a CLIP tower at model.vision_tower, which is vision (a Vision), and the projector model.multi_modal_projector, which is projector. Llava feeds the projector vision.layers[-2].layer_output without its CLS token (vision_feature_layer=-2, vision_feature_select_strategy="default"), not the tower's output; VipLlava concatenates several blocks' streams; LLaVA-NeXT unpads the projector's output and adds a newline token per row. vision.image_features is read at the scatter (ImageScatter), so it is what enters the text model on all three. CLIP's post_layernorm norms only the pooled CLS token, never the patches, so it is not vision.norm.

DeepSeek-VL keeps a SigLIP tower at model.vision_model with the projector model.aligner; Idefics 3 and SmolVLM keep their SigLIP-shaped ViT at model.vision_model (one row per image tile) with the pixel-shuffling model.connector as projector, and merge the features in a helper, inputs_merger. Those towers' post_layernorm does norm the patches; since a rename key cannot tell it from CLIP's, SiglipVision serves it as vision.norm.

The block

The vision encoder

The vision encoder

The vision encoder

The vision encoder

The vision encoder

The vision encoder

The API

Values, by host

Printout

Config

Config

Config

Config

Config

Config

Config

Config

Config

Config

Config

Vision Config

Config

Vision Config

Config

Vision Config

Config

Vision Config

Config

Vision Config

Config

Vision Config

Config

Vision Config

Config

Vision Config

The Notes

The block, in order

The contributions are the modules' outputs

The default load runs the same attention as eager

Grouped-query attention on Llama-3 and Llama-2-70B

Queries and keys are read after the rotary embedding

Position 0 is a sink with a very large norm

Target tokens differ by tokenizer

The readout and the embeddings

What this family module covers

Sparse autoencoders and transcoders

Vision notes

The CLIP vision encoder

The CLS token comes first

A norm before block 0, none after the last

Llava 1.5

The projector reads block -2, without the CLS token

One token per patch

Vision notes

The CLIP vision encoder

The CLS token comes first

A norm before block 0, none after the last

VipLlava

The projector reads five blocks

Vision notes

The CLIP vision encoder

The CLS token comes first

A norm before block 0, none after the last

LLaVA-NeXT

Crops as rows, and features that are not the projector's output

Vision notes

The SigLIP vision encoder

The patch axis is the patch grid

The final norm is vision.norm

DeepSeek-VL

The aligner reads tower_output

Vision notes

The SigLIP vision encoder

The patch axis is the patch grid

The final norm is vision.norm

Idefics 3

Tiles as rows, pixel-shuffled into tokens

The features go in through inputs_merger

The final norm is `vision.norm`

The aligner reads `tower_output`

The final norm is `vision.norm`

The features go in through `inputs_merger`

The final norm is `vision.norm`