gemma4_unified_text¶
Gemma 4 unified, text (Gemma4UnifiedForCausalLM, model_type gemma4_unified_text): Gemma-4-12B.
Gemma-4's tree and block without per-layer embeddings or a mixture of experts:
the sandwich, then out = (x + post_attention_layernorm(...) +
post_feedforward_layernorm(...)) * layer_scalar, in place. The contributions are the
post-norms' outputs, served unscaled, so the identity is
(input + attention_output + mlp_output) * layer_scalar == layer_output. KV
sharing, attention_k_eq_v and the per-layer head_dim are Gemma-4's
(gemma4_text), and so is the attention envoy (keys and values served as
copies private to the block), and so are the root's head_dim and
num_kv_heads: the config's top-level values. A gemma4_unified checkpoint
(Gemma4UnifiedForConditionalGeneration) keeps the text stack at
model.language_model, so RENAME carries both spellings.
The wrapper is encoder-free: no vision tower, no blocks. model.embed_vision
embeds the processor's merged patches (model_patch_size pixels square)
directly: patch_ln1, patch_dense, patch_ln2, a factorized 2D position
embedding, pos_norm, then multimodal_embedder (an RMS norm and a linear
onto the text width, the same projection Gemma-4's embed_vision is). So
vision is the embedder (a Vision with no layers), vision.patch_embed
its patch_dense, and projector its multimodal_embedder, the module
mapping onto the text width as on every other wrapper. The processor pads each
image's patches to max_soft_tokens rows at position (-1, -1), and the
embedder runs on every row; the wrapper strips the padded rows of the
projector's output before scattering, so vision.image_features is read at
the scatter (the first inputs_embeds.masked_scatter of
Gemma4UnifiedModel.forward, an ImageScatter). Audio is embedded the same
way, without blocks: model.embed_audio keeps its native name, and there is no
audio_tower.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'model.embed_vision': 'vision', 'model.embed_vision.multimodal_embedder': 'projector', 'patch_dense': 'patch_embed'}
module-attribute
¶
ENVOYS = {Gemma4UnifiedTextDecoderLayer: Layer, Gemma4UnifiedTextAttention: Attention, Gemma4UnifiedTextMLP: Mlp, Gemma4UnifiedVisionEmbedder: Vision, Gemma4UnifiedModel: ImageScatter}
Bases: Attention
Gemma-4 unified's attention: Gemma-4's, with its post-norm contribution and its keys and values private to the block.