Skip to content

llama4_text

llama4_text

Llama 4, text (Llama4ForCausalLM, model_type llama4_text): Scout and Maverick's language model.

model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, feed_forward}, norm} and lm_head: Llama's tree with the feed-forward called feed_forward, renamed to mlp. A llama4 checkpoint (Llama4ForConditionalGeneration, the published Scout and Maverick repos) nests its config's text_config, of this type; the text-generation task builds Llama4ForCausalLM from it, and a wrapper module passed in already loaded keeps the text model at language_model.{model.*, lm_head}, so RENAME carries both spellings and whichever the tree has binds.

The block adds both residuals itself, attention through the shared eager forward (Llama 4's own copy, which keeps the softmax in the model dtype). What differs from Llama, inside the attention and the feed-forward:

  • iRoPE. no_rope_layers[i] == 0 marks a NoPE block: no rotary embedding, no qk-norm, full causal attention, and attention temperature tuning (attn_temperature_tuning) scales its queries by a factor that grows with the position. The other blocks apply the rotary embedding, then a weightless RMS qk-norm (Llama4TextL2Norm, use_qk_norm), and attend within chunks of attention_chunk_size tokens (config.layer_types). The queries and keys served are what the interface receives, after all of that; a chunked block's pattern is zero across chunk boundaries.
  • Interleaved mixture of experts. Blocks in moe_layers carry a Llama4TextMoe (a shared expert plus routed experts) returning (out, router_logits) with out flattened to [batch * seq, hidden_size]; the others a dense Llama4TextMLP. The block views either back into the residual's shape before adding it, so mlp_output is read at that view in the block's forward: [batch, seq, hidden_size] on both kinds, and edits to it reach the feed-forward's own output tensor. The shared expert is a Llama4TextMLP too, so it is an Mlp; its mlp_output is unavailable, since the block adds the mixture's sum. The mixture is a Moe whose router returns dense scores [tokens, num_experts] (the sigmoid of the top-k logits, zero elsewhere): the experts run on every token, each scaled on its input by its score, and the routed sum is added into the shared expert's output tensor in place. So router_logits (the router's projection) and expert_indices (its top-k) are the router's own; routed_output is the mixture's sum over the experts; shared_expert_output is a copy taken as the shared expert returns, whose edits are carried back; and expert_weights / expert_outputs are unavailable.
  • Sizes. intermediate_size in this config is the experts' width (and the shared expert's); the dense MLP's is intermediate_size_mlp, which the root's intermediate_size reports. Scout has a mixture on every block, so there it names a width the model never uses, as on Qwen3-MoE; Maverick alternates dense and mixture blocks.

On the wrapper the vision tower vision_model is vision (a Vision) and multi_modal_projector, the last module before the scatter, is projector. The tower is a ViT over each image tile: the patch embedding (patch_embedding, an unfold and a linear) is vision.patch_embed, the blocks (model.layers, pre-norm, on the shared attention interface with a 2D rotary embedding) are vision.layers, and layernorm_post is vision.norm. A CLS token is appended after the patches, so the tower's stream is [tiles, patches + 1, vision_hidden_size] with the CLS last (CLIP's is first); the tower drops it after layernorm_post, then pixel-shuffles and projects the patches in vision.vision_adapter, so vision.tower_output is layernorm_post's output, CLS included, and projector.input is the adapter's output, flattened over the tiles. Loaded with task="image-text-to-text" (the processor), the tower serves vision.image_token_mask and vision.image_features. The text-generation task builds Llama4ForCausalLM, which has no tower.

RENAME module-attribute 

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'language_model.model.embed_tokens': 'embed_tokens', 'language_model.model.layers': 'layers', 'language_model.model.norm': 'norm', 'language_model.lm_head': 'lm_head', 'feed_forward': 'mlp', 'shared_expert': 'shared_experts', 'vision_model': 'vision', 'multi_modal_projector': 'projector', 'patch_embedding': 'patch_embed', 'layernorm_post': 'norm', 'activation_fn': 'act_fn', 'fc2': 'down_proj'}

ROOT_SCATTER module-attribute 

ROOT_SCATTER = 'inputs_embeds_masked_scatter_0'

FEED_FORWARD_VIEW module-attribute 

FEED_FORWARD_VIEW = 'hidden_states_view_0'

DENSE_SCORES module-attribute 

DENSE_SCORES = "Llama 4's router scales each expert's input by a dense score over every expert; not mapped yet"

ENVOYS module-attribute 

ENVOYS = {Llama4TextDecoderLayer: Layer, Llama4TextAttention: Attention, Llama4TextMLP: Mlp, Llama4TextMoe: Moe, Llama4VisionModel: Vision, Llama4VisionEncoderLayer: VisionLayer, Llama4VisionAttention: VisionAttention, Llama4VisionMLP: VisionMlp}

intermediate_size 

intermediate_size(model: StandardizedTransformer) -> int

The dense MLP's width, intermediate_size_mlp; this config's intermediate_size is the experts'.