ministral3¶
Ministral 3 (Ministral3ForCausalLM).
Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm,
self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual
added in the block, attention through the shared eager forward.
The queries are scaled by position (llama_4_scaling_beta) after the rotary
embedding and before the interface, so attention_queries are the scaled ones.
The mistral3 wrapper puts a Pixtral tower at model.vision_tower, which is
vision (a PixtralVision: every image's patches packed in one row, no final
norm), and the projector model.multi_modal_projector (a norm, the
patch_merger that merges each 2x2 patch block, two linears), which is
projector; vision.image_features is read at the scatter (ImageScatter).
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'model.vision_tower': 'vision', 'model.multi_modal_projector': 'projector', 'patch_conv': 'patch_embed', 'transformer.layers': 'layers', 'attention': 'self_attn', 'feed_forward': 'mlp', 'attention_norm': 'input_layernorm', 'ffn_norm': 'post_attention_layernorm'}
module-attribute
¶
ENVOYS = {Ministral3DecoderLayer: Layer, Ministral3Attention: Attention, Ministral3MLP: Mlp, PixtralVisionModel: PixtralVision, PixtralAttentionLayer: VisionLayer, PixtralAttention: VisionAttention, PixtralMLP: VisionMlp, Mistral3Model: ImageScatter}