family llama4_text

Llama 4

Llama's pre-norm block with a mixture of experts and a shared expert as the MLP (every block on Scout, every other block on Maverick), routed by a sigmoid of the top-1 logit; every fourth block attends over the whole context without rotary, and the others attend within fixed chunks.

Query/key norms Blocks without rotary Chunked attention Mixture of experts Grouped-query attention Vision-language CLS token Tiled images Pooled projector

One block of the stack. Hover any part for its nnterp name; click to pin the card. The slider picks a block; this family's blocks differ by layer_types.

embed_tokenstoken_embeddings

→

layers× 48

→

normfinal_norm_output

→

lm_headlm_head_output

→

logitsthe output

block i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The vision encoder

· Llama 4 ViT · Llama4VisionModel

The image's path into the text model, then one block of the vision encoder. Hover any part for its nnterp name; click to pin the card.

imagepixel_values

→

patch_embedpatch_embeddings

→

layers× 34

→

normtower_output

→

projectorinput: the pixel-shuffle adapter's output over vision.tower_output, flattened over the tiles

→

image_featuresat the scatter

→

layers[0].inputat image_token_mask

rows: One row per image tile: the processor fits an image onto a grid of up to max_patches (16) tiles of 336 pixels and adds a global tile, the whole image resized, when there is more than one. A row is the tile's 576 patches, then the CLS token last: patches + 1 rows.
positions: patch_embeddings is the unfold-and-linear output, before the CLS token is appended. A learned embedding per row (positional_embedding_vlm, the CLS's last) is added and layernorm_pre norms the rows; every attention rotates queries and keys by the patch's 2D position, and leaves the CLS row unrotated.
masking: No mask: each row attends to every patch of its own tile and to its CLS token; tiles do not see one another.
final norm: layernorm_post is vision.norm, a LayerNorm over every row, CLS included; vision.tower_output is its output. The tower then drops the CLS row and runs vision.vision_adapter.

identity vision.layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Llama4ForConditionalGenerationlayoutwhat it isread at

logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output

token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output

final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output

lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output

next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output

input_ids Tokens [batch seq] The token ids the model was called with

the module's own input

attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input

input_size scalar [batch, seq] of the current call; read-only

the module's own input

model.layers[i] Llama4TextDecoderLayerlayoutwhat it isread at

layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input

attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output

layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input

mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output

mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input

layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output

model.layers[i].self_attn Llama4TextAttentionlayoutwhat it isread at

⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1

⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2

⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3

⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1

attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input

attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output

⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1

⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0

model.layers[i].mlp Llama4TextMoelayoutwhat it isread at

⚠︎mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output

⚠︎mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input

mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream: its output viewed in the residual's shape

the sibling module source.hidden_states_view_0's output

router_logits RouterLogits [batch seq num_experts] The router's logits, one per expert, before the scoring

inside router's forward: the output of forward_0

⚠︎expert_weights scalar Unavailable: Llama 4's router scales each expert's input by a dense score over every expert; not mapped yet

nowhere: the family does not serve it

expert_indices ExpertIndices [batch seq top_k] The expert each routing slot sends the token to

inside router's forward: the output of torch_topk_0, element 1

⚠︎expert_outputs scalar Unavailable: Llama 4's router scales each expert's input by a dense score over every expert; not mapped yet

nowhere: the family does not serve it

routed_output Residual [batch seq hidden_size] The routed experts' combined output

inside the forward: the output of sum_0

shared_expert_output Residual [batch seq hidden_size] The shared expert's output (a copy, since the mixture adds the routed sum into the live tensor in place)

the child module shared_experts's output

the vision encoder · Llama 4 ViT

model.vision Llama4VisionModellayoutwhat it isread at

image_token_mask ImageTokenMask [batch seq] Which positions hold image tokens: input_ids == the config's image_token_id; read-only

the model's own inputs, at the root

patch_embeddings Patches [images patches vision_hidden_size] The patch embedding's output, one row per patch

the child module patch_embed's output

tower_output Patches [images patches vision_hidden_size] The last block's stream after the tower's final norm, before any pooling or adapter

the child module norm's output

image_features ImageFeatures [image_tokens hidden_size] The image features the text model receives at the image tokens, flat over them

inside the forward: the arguments of inputs_embeds_masked_scatter_0, element 1

model.vision.layers[i] Llama4VisionEncoderLayerlayoutwhat it isread at

layer_input Patches [images patches vision_hidden_size] The tower's stream entering the block

the module's own input

attention_norm_output Patches [images patches vision_hidden_size] The normed stream entering the attention

the child module input_layernorm's output

layer_mid Patches [images patches vision_hidden_size] The tower's stream after the attention add

the child module post_attention_layernorm's input

mlp_norm_output Patches [images patches vision_hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output

mlp_input Patches [images patches vision_hidden_size] What the MLP is called with

the child module mlp's input

layer_output Patches [images patches vision_hidden_size] The tower's stream leaving the block

the module's own output

model.vision.layers[i].self_attn Llama4VisionAttentionlayoutwhat it isread at