llama¶
Llama (LlamaForCausalLM), the family the standard vocabulary is taken from.
Block names (input_layernorm, self_attn, post_attention_layernorm,
mlp) and lm_head are already the standard ones. The only change is
lifting the containers out of model.model: model.layers instead of
model.model.layers. Multimodal wrappers around a Llama text model keep it at
model.language_model (Llava, DeepSeek-VL, Janus) or model.text_model
(Idefics 3, SmolVLM); RENAME carries those spellings too.
Llava 1.5 (LlavaForConditionalGeneration, model_type llava), VipLlava
and LLaVA-NeXT add a CLIP tower at model.vision_tower, which is vision (a
Vision), and the projector model.multi_modal_projector, which is
projector. Llava feeds the projector vision.layers[-2].layer_output
without its CLS token (vision_feature_layer=-2,
vision_feature_select_strategy="default"), not the tower's output; VipLlava
concatenates several blocks' streams; LLaVA-NeXT unpads the projector's output and
adds a newline token per row. vision.image_features is read at the scatter
(ImageScatter), so it is what enters the text model on all three. CLIP's
post_layernorm norms only the pooled CLS token, never the patches, so it is
not vision.norm.
DeepSeek-VL keeps a SigLIP tower at model.vision_model with the projector
model.aligner; Idefics 3 and SmolVLM keep their SigLIP-shaped ViT at
model.vision_model (one row per image tile) with the pixel-shuffling
model.connector as projector, and merge the features in a helper,
inputs_merger. Those towers' post_layernorm does norm the patches; since a
rename key cannot tell it from CLIP's, SiglipVision serves it as vision.norm.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'model.text_model.embed_tokens': 'embed_tokens', 'model.text_model.layers': 'layers', 'model.text_model.norm': 'norm', 'model.vision_tower': 'vision', 'model.vision_model': 'vision', 'model.multi_modal_projector': 'projector', 'model.aligner': 'projector', 'model.connector': 'projector', 'embeddings.patch_embedding': 'patch_embed', 'encoder.layers': 'layers', 'layer_norm1': 'input_layernorm', 'layer_norm2': 'post_attention_layernorm', 'out_proj': 'o_proj', 'activation_fn': 'act_fn', 'fc2': 'down_proj'}
module-attribute
¶
ENVOYS = {LlamaDecoderLayer: Layer, LlamaAttention: Attention, LlamaMLP: Mlp, CLIPVisionModel: Vision, CLIPEncoderLayer: VisionLayer, CLIPAttention: VisionAttention, CLIPMLP: VisionMlp, SiglipVisionModel: SiglipVision, SiglipEncoderLayer: VisionLayer, SiglipAttention: VisionAttention, SiglipMLP: VisionMlp, Idefics3VisionTransformer: SiglipVision, Idefics3EncoderLayer: VisionLayer, Idefics3VisionAttention: VisionAttention, Idefics3VisionMLP: VisionMlp, SmolVLMVisionTransformer: SiglipVision, SmolVLMEncoderLayer: VisionLayer, SmolVLMVisionAttention: VisionAttention, SmolVLMVisionMLP: VisionMlp, LlavaModel: ImageScatter, VipLlavaModel: ImageScatter, LlavaNextModel: ImageScatter, DeepseekVLModel: ImageScatter, Idefics3Model: InputsMerger, SmolVLMModel: InputsMerger}
Bases: ImageScatter
Idefics 3's and SmolVLM's model: the features go in through inputs_merger(..., image_hidden_states=...).