gemma3_text¶
Gemma 3, text (Gemma3ForCausalLM, model_type gemma3_text).
Llama's names plus a sandwich block: every sublayer is normed before and
after, x + post_attention_layernorm(attn(input_layernorm(x))) and then
+ post_feedforward_layernorm(mlp(pre_feedforward_layernorm(x))). What the
block adds is the post-norm's output, not the module's, so the contributions
point at the sibling norms. A gemma3 checkpoint (Gemma3ForConditionalGeneration,
gemma-3-4b/12b/27b) nests its config's text_config, of this type; the
text-generation task builds the wrapper, whose text stack sits at
model.language_model.{embed_tokens, layers, norm} with lm_head at the root, so
RENAME carries both spellings and whichever the tree has binds. On the
wrapper the SigLIP tower model.vision_tower is vision (a Vision; its
post_layernorm over the patches is vision.norm) and the pooling
projector model.multi_modal_projector, the last module before the scatter,
is projector; image_features is read at the scatter in
Gemma3Model's forward (ImageScatter). Loaded with task="image-text-to-text" (the
processor), the tower serves vision.image_token_mask and vision.image_features. Only
Gemma3ForCausalLM softcaps its logits where the config sets
final_logit_softcapping (no released checkpoint does); the wrapper never
does, and project_on_vocab follows the class.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'model.vision_tower': 'vision', 'model.multi_modal_projector': 'projector', 'embeddings.patch_embedding': 'patch_embed', 'encoder.layers': 'layers', 'post_layernorm': 'norm', 'layer_norm1': 'input_layernorm', 'layer_norm2': 'post_attention_layernorm', 'out_proj': 'o_proj', 'activation_fn': 'act_fn', 'fc2': 'down_proj'}
module-attribute
¶
ENVOYS = {Gemma3DecoderLayer: Layer, Gemma3Attention: Attention, Gemma3MLP: Mlp, SiglipVisionModel: Vision, SiglipEncoderLayer: VisionLayer, SiglipAttention: VisionAttention, SiglipMLP: VisionMlp, Gemma3Model: ImageScatter}
project_on_vocab(model: StandardizedTransformer, hidden: Tensor) -> Tensor
The logit lens as the model makes its logits: the final norm, lm_head, and the softcap only where the class applies one.
Gemma3ForCausalLM caps with final_logit_softcapping when the config
sets it; Gemma3ForConditionalGeneration never does, whatever its
text_config says, so neither does the lens on it.