mistral¶
Mistral (MistralForCausalLM).
Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm,
self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual
added in the block, attention through the shared eager forward.
Two towers sit at model.vision_tower, which is vision, with the projector
model.multi_modal_projector as projector: Pixtral (a PixtralVision, every
image's patches packed in one row, no final norm) on the mistral3 wrapper
(Mistral Small 3.1 / 3.2) and on Pixtral-12B's llava wrapper; CLIP on LLaVA-NeXT
(llava-v1.6-mistral) and BakLLaVA (llava). The towers' inner names differ, so
both sets are keyed. vision.image_features is read at the scatter
(ImageScatter), so on LLaVA-NeXT it is the unpadded projector output with its
newline tokens, as the text model receives it. Pixtral-12B reads
vision_feature_layer -1 with strategy "full", so its projector.input is
vision.tower_output; BakLLaVA reads block -2 without the CLS token, as Llava 1.5.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'model.vision_tower': 'vision', 'model.multi_modal_projector': 'projector', 'patch_conv': 'patch_embed', 'transformer.layers': 'layers', 'attention': 'self_attn', 'feed_forward': 'mlp', 'attention_norm': 'input_layernorm', 'ffn_norm': 'post_attention_layernorm', 'embeddings.patch_embedding': 'patch_embed', 'encoder.layers': 'layers', 'layer_norm1': 'input_layernorm', 'layer_norm2': 'post_attention_layernorm', 'out_proj': 'o_proj', 'activation_fn': 'act_fn', 'fc2': 'down_proj'}
module-attribute
¶
ENVOYS = {MistralDecoderLayer: Layer, MistralAttention: Attention, MistralMLP: Mlp, PixtralVisionModel: PixtralVision, PixtralAttentionLayer: VisionLayer, PixtralAttention: VisionAttention, PixtralMLP: VisionMlp, CLIPVisionModel: Vision, CLIPEncoderLayer: VisionLayer, CLIPAttention: VisionAttention, CLIPMLP: VisionMlp, Mistral3Model: ImageScatter, LlavaModel: ImageScatter, LlavaNextModel: ImageScatter}