gemma2¶
Gemma 2 (Gemma2ForCausalLM).
Llama's names plus a sandwich block: every sublayer is normed before and
after, x + post_attention_layernorm(attn(input_layernorm(x))) and then
+ post_feedforward_layernorm(mlp(pre_feedforward_layernorm(x))). What the
block adds is the post-norm's output, not the module's, so the contributions
point at the sibling norms. final_logit_softcapping applies to the logits after lm_head.
PaliGemma 2 wraps this text model (its text_config.model_type is gemma2) as PaliGemma 1
wraps Gemma's: the text stack at model.language_model, a SigLIP tower at
model.vision_tower, which is vision (its post_layernorm over the patches is
vision.norm), and a linear projector model.multi_modal_projector, which is
projector; vision.image_features is read at the scatter (ImageScatter). Its processor
refuses a prompt without an image: a text-only trace takes the tokenizer's encoding. The wrapper
never softcaps its logits, whatever its text_config says, and project_on_vocab follows the class.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'model.vision_tower': 'vision', 'model.multi_modal_projector': 'projector', 'embeddings.patch_embedding': 'patch_embed', 'encoder.layers': 'layers', 'post_layernorm': 'norm', 'layer_norm1': 'input_layernorm', 'layer_norm2': 'post_attention_layernorm', 'out_proj': 'o_proj', 'activation_fn': 'act_fn', 'fc2': 'down_proj'}
module-attribute
¶
ENVOYS = {Gemma2DecoderLayer: Layer, Gemma2Attention: Attention, Gemma2MLP: Mlp, SiglipVisionModel: Vision, SiglipEncoderLayer: VisionLayer, SiglipAttention: VisionAttention, SiglipMLP: VisionMlp, PaliGemmaModel: ImageScatter}
project_on_vocab(model: StandardizedTransformer, hidden: Tensor) -> Tensor
The logit lens as the model makes its logits: the final norm, lm_head, and the softcap only where the class applies one.
Gemma2ForCausalLM caps with final_logit_softcapping; PaliGemma 2's
PaliGemmaForConditionalGeneration never does, though its text_config
sets 30, so neither does the lens on it.