nnterp · encyclopedia ministral3 · nnterp.families.ministral3

family ministral3

Ministral 3

Llama's pre-norm block whose queries are multiplied, after the rotary embedding, by a factor that grows with the position past 16384 tokens; every checkpoint is a vision-language model on the Pixtral encoder.

Grouped-query attentionVision-languagePacked vision encoderVariable resolutionPooled projector

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card.

embed_tokenstoken_embeddings
→
layers× 26
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The vision encoder

· Pixtral · PixtralVisionModel

The image's path into the text model, then one block of the vision encoder. Hover any part for its nnterp name; click to pin the card.

imagepixel_values
→
patch_embedpatch_embeddings
→
layers× 24 · tower_output
→
projectorinput: vision.tower_output[0]
→
image_featuresat the scatter
→
layers[0].inputat image_token_mask
rows
One row, packed: every image's patches, image after image, [1, patches, vision_hidden_size]. Image j has (height // patch_size) * (width // patch_size) patches in raster order, height and width being its entry in the processor's image_sizes. vision.patch_embeddings is the packed row entering ln_pre; the convolution's own output, patch_embed.output, is the padded grid.
positions
A 2D rotary embedding (the patch's row and column in its image) turns the queries and keys inside each attention; nothing is added to the stream. ln_pre norms the packed row before block 0.
masking
Block-diagonal: a patch attends to every patch of its own image and to none of another's, so the pattern is zero between images.
final norm
None after the last block, so there is no vision.norm, and vision.tower_output is the last block's layer_output.

identity vision.layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Mistral3ForConditionalGenerationlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] Ministral3DecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn Ministral3Attentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp Ministral3MLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output

the vision encoder · Pixtral

model.vision PixtralVisionModellayoutwhat it isread at
image_token_mask ImageTokenMask [batch seq] Which positions hold image tokens: input_ids == the config's image_token_id; read-only

the model's own inputs, at the root
patch_embeddings Patches [images patches vision_hidden_size] Every image's patch embeddings packed in one row, entering the pre-norm

the child module ln_pre's input
tower_output Patches [images patches vision_hidden_size] The last block's stream after the tower's final norm, before any pooling or adapter

the module's own output
image_features ImageFeatures [image_tokens hidden_size] The image features the text model receives at the image tokens, flat over them

inside model.model's forward: the arguments of inputs_embeds_masked_scatter_0, element 1
model.vision.layers[i] PixtralAttentionLayerlayoutwhat it isread at
layer_input Patches [images patches vision_hidden_size] The tower's stream entering the block

the module's own input
attention_norm_output Patches [images patches vision_hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Patches [images patches vision_hidden_size] The tower's stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Patches [images patches vision_hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Patches [images patches vision_hidden_size] What the MLP is called with

the child module mlp's input
layer_output Patches [images patches vision_hidden_size] The tower's stream leaving the block

the module's own output
model.vision.layers[i].mlp PixtralMLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives

the child module down_proj's input
mlp_output Patches [images patches vision_hidden_size] What the MLP adds to the tower's stream

the module's own output
model.vision.layers[i].self_attn PixtralAttentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Patches [images patches vision_hidden_size] What the attention adds to the tower's stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0

Printout

What print(model) shows on mistralai/Ministral-3-3B-Base-2512: the native modules, standard names and every standard value with its layout.

Mistral3ForConditionalGeneration(
  (lm_head): Linear(in_features=3072, out_features=131072, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(131072, 3072, padding_idx=11)
  (layers): ModuleList(
    (0-25): 26 x Ministral3DecoderLayer(
      (self_attn): Ministral3Attention(
        (q_proj): Linear(in_features=3072, out_features=4096, bias=False)
        (k_proj): Linear(in_features=3072, out_features=1024, bias=False)
        (v_proj): Linear(in_features=3072, out_features=1024, bias=False)
        (o_proj): Linear(in_features=4096, out_features=3072, bias=False)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Ministral3MLP(
        (gate_proj): Linear(in_features=3072, out_features=9216, bias=False)
        (up_proj): Linear(in_features=3072, out_features=9216, bias=False)
        (down_proj): Linear(in_features=9216, out_features=3072, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Ministral3RMSNorm((3072,), eps=1e-05)
      (post_attention_layernorm): Ministral3RMSNorm((3072,), eps=1e-05)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): Ministral3RMSNorm((3072,), eps=1e-05)
  (vision): PixtralVisionModel(
    (patch_embed/patch_conv): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14), bias=False)
    (ln_pre): PixtralRMSNorm((1024,), eps=1e-05)
    (patch_positional_embedding): PixtralVisionRotaryEmbedding()
    (layers): ModuleList(
      (0-23): 24 x PixtralAttentionLayer(
        (input_layernorm/attention_norm): PixtralRMSNorm((1024,), eps=1e-05)
        (mlp/feed_forward): PixtralMLP(
          (gate_proj): Linear(in_features=1024, out_features=4096, bias=False)
          (up_proj): Linear(in_features=1024, out_features=4096, bias=False)
          (down_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (act_fn): SiLUActivation()
          (mlp_output) -> Patches [images patches vision_hidden_size]: What the MLP adds to the tower's stream
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives
        )
        (self_attn/attention): PixtralAttention(
          (k_proj): Linear(in_features=1024, out_features=1024, bias=False)
          (v_proj): Linear(in_features=1024, out_features=1024, bias=False)
          (q_proj): Linear(in_features=1024, out_features=1024, bias=False)
          (o_proj): Linear(in_features=1024, out_features=1024, bias=False)
          (attention_output) -> Patches [images patches vision_hidden_size]: What the attention adds to the tower's stream
          (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
          (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
          (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
          (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
          (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
          (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
          (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
        )
        (post_attention_layernorm/ffn_norm): PixtralRMSNorm((1024,), eps=1e-05)
        (layer_input) -> Patches [images patches vision_hidden_size]: The tower's stream entering the block
        (attention_norm_output) -> Patches [images patches vision_hidden_size]: The normed stream entering the attention
        (layer_mid) -> Patches [images patches vision_hidden_size]: The tower's stream after the attention add
        (mlp_norm_output) -> Patches [images patches vision_hidden_size]: The normed stream entering the MLP
        (mlp_input) -> Patches [images patches vision_hidden_size]: What the MLP is called with
        (layer_output) -> Patches [images patches vision_hidden_size]: The tower's stream leaving the block
      )
    )
    (patch_embeddings) -> Patches [images patches vision_hidden_size]: Every image's patch embeddings packed in one row, entering the pre-norm
    (image_token_mask) -> ImageTokenMask [batch seq]: Which positions hold image tokens: input_ids == the config's image_token_id; read-only
    (tower_output) -> Patches [images patches vision_hidden_size]: The last block's stream after the tower's final norm, before any pooling or adapter
    (image_features) -> ImageFeatures [image_tokens hidden_size]: The image features the text model receives at the image tokens, flat over them
  )
  (projector): Mistral3MultiModalProjector(
    (norm): Mistral3RMSNorm((1024,), eps=1e-05)
    (patch_merger): Mistral3PatchMerger(
      (merging_layer): Linear(in_features=4096, out_features=1024, bias=False)
    )
    (linear_1): Linear(in_features=1024, out_features=3072, bias=False)
    (act): GELUActivation()
    (linear_2): Linear(in_features=3072, out_features=3072, bias=False)
  )
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers26
model.hidden_size3072
model.num_heads32
model.num_kv_heads8
model.head_dim128
model.qk_head_dim128
model.vocab_size131072
model.intermediate_size9216
architecturesMistral3ForConditionalGeneration
num_hidden_layers26
hidden_size3072
intermediate_size9216
num_attention_heads32
num_key_value_heads8
head_dim128
vocab_size131072
max_position_embeddings262144
hidden_actsilu
rms_norm_eps1e-05
tie_word_embeddingsTrue
dtypebfloat16
Vision Config

Vision Config

model.vision.num_layers24
model.vision.vision_hidden_size1024
model.vision.num_heads16
model.vision.head_dim64
model.vision.intermediate_size4096
model.vision.patch_size14
model.vision.image_sizevaries

The Notes

03 · what to know before you run

The block, in order

h   = x + self_attn(input_layernorm(x))
out = h + mlp(post_attention_layernorm(h))

Llama's block: two RMSNorms per block, each before its sublayer, none after, and a SwiGLU MLP. attention_output is the attention module's output and mlp_output the MLP's, added to the stream with nothing in between, so the identity holds exactly (float32, pinned checkpoint):

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(x + attn + mlp, out)

attention_queries are the position-scaled queries

After the rotary embedding the attention multiplies each query by 1 + beta * log(1 + floor(position / original_max_position_embeddings)), with beta the rope_parameters' llama_4_scaling_beta (0.1) and original_max_position_embeddings 16384 on every size. attention_queries is read at the attention function's arguments, after the scale. The factor is exactly 1 below position 16384, so on shorter prompts the queries are the rotated ones; it is about 1.069 from 16384 and about 1.22 from 131072. The rotated, unscaled queries are the output of the apply_rotary_pos_emb call in the attention's .source:

attn = model.layers[1].self_attn
with model.trace(prompt):
    rotated = attn.source.apply_rotary_pos_emb_0.output[0].save()
    q = attn.attention_queries.save()            # rotated * the position factor

On the pinned checkpoint rewritten with beta 0.1 and original_max_position_embeddings 4, q equals rotated times the factor at every position, and rotated exactly at positions 0 to 3. The keys are not scaled. The rotary itself is YaRN (rope_type yarn, factor 16, rope_theta 1,000,000 on 3B and 8B and 1,000,000,000 on 14B) over a max_position_embeddings of 262144; its attention_scaling is 1.0 on every size (mscale and mscale_all_dim are both 1), so the cosines and sines are not rescaled.

Grouped-query attention, 4096 wide on every size

Every size has 32 query heads over 8 key/value heads of width 128: query head h reads key/value head h // 4, and attention_keys and attention_values are read before repeat_kv, [batch, 8, seq, 128]. The heads are 4096 wide together, which is wider than the 3B stream (3072), equal to 8B's (4096) and narrower than 14B's (5120): q_proj and o_proj change width on 3B and 14B, and attention_head_outputs flattened is 4096 wide on every size.

Every checkpoint is a vision-language wrapper

Ministral-3-*-2512 checkpoints are model_type mistral3 whose text_config is ministral3, so each loads with task="image-text-to-text", and model.layers are the wrapper's model.language_model.layers. A text-only trace runs on the wrapper as on a text model; the image values are served when the trace is given an image.

Target tokens

The tokenizer is the 131072-token Tekken vocabulary: " Paris" is ĠParis (6993) and "Paris" is Paris (42572), two different tokens (checked on 3B). Pick the one the prompt's spacing produces, and assert it is one token:

ids = model.tokenizer(" Paris", add_special_tokens=False).input_ids
assert len(ids) == 1

Vision notes

· Pixtral · Mistral 3

The Pixtral vision encoder

Two images are one row

The convolution runs on the processor's batch, padded to its largest image, and each image's grid is cropped back to its own size, flattened and concatenated: vision.patch_embeddings is [1, patches, vision_hidden_size], the patches of every image in the invoke, image after image, each in raster order. vision.patch_embed.output is the padded grid, [images, vision_hidden_size, rows, columns], and is not the stream. Every block value is served on the packed row, so the processor's image_sizes split it:

inputs = model.processor(text=prompt, images=[square, wide], return_tensors="pt")
side = model.vision.patch_size
sizes = [h // side * (w // side) for h, w in inputs["image_sizes"].tolist()]
with model.trace(prompt, images=[square, wide]):
    patches = model.vision.layers[1].layer_output.save()
per_image = patches[0].split(sizes)                 # one [patches_j, vision_hidden_size] each

On the pinned Mistral 3 tiny checkpoint a 36 × 36 and a 24 × 36 image at 6-pixel patches are 36 and 24 patches: patch_embed.output is [2, 32, 6, 6] and patch_embeddings [1, 60, 32].

ln_pre sits between the patches and block 0

ln_pre, an RMSNorm, norms the packed row before the first block, so vision.layers[0].input is ln_pre's output and not vision.patch_embeddings. No position embedding is added: positions enter only as the 2D rotary inside each attention, by the patch's row and column within its own image.

The attention runs once over the row, under a block-diagonal mask

Each block calls the attention function once on the whole row. The image boundaries go in as a mask that is zero inside an image's square and the dtype's minimum between images, or, under flash attention, as cu_seqlens. So the interior is whole: under attn_implementation="eager", attention_probabilities is [1, heads, patches, patches], each row summing to one over its own image's patches and exactly zero on the other images'.

with model.trace(prompt, images=[square, wide]):
    pattern = model.vision.layers[0].self_attn.attention_probabilities.save()

a = sizes[0]
pattern[..., :a, a:].abs().max(), pattern[..., a:, :a].abs().max()   # 0.0, 0.0

No final norm, and no image_size

The last block's stream leaves the vision encoder as it is: vision.tower_output equals vision.layers[-1].layer_output, and model.vision has no norm. vision.image_size raises Unavailable: the processor resizes each image to fit its own grid, and the config's image_size is the longest side it resizes to, not a size every image has.

Mistral 3

The projector merges each 2 × 2 block of patches

model.multi_modal_projector (the projector) reads the last block's stream with its leading 1 dropped: model.projector.input is vision.tower_output[0], [patches, vision_hidden_size] (vision_feature_layer is -1). It norms each patch (norm, an RMSNorm), then patch_merger cuts each image's grid into 2 × 2 blocks (spatial_merge_size), flattens each block's four patches into one 4 * vision_hidden_size vector (unfold, channel by channel) and maps it back to vision_hidden_size with merging_layer, a linear without bias; linear_1, GELU and linear_2 take it to the text model's width. So an image of rows × columns patches is (rows // 2) * (columns // 2) image tokens, and vision.image_features is model.projector.output as it is, [image_tokens, hidden_size].

with model.trace(prompt, images=[square, wide]):
    mask = model.vision.image_token_mask.save()
    tower = model.vision.tower_output.save()    # read before the projector
    fed = model.projector.input.save()
    features = model.vision.image_features.save()
    first = model.layers[0].input.save()

torch.equal(fed, tower[0])                      # True
torch.equal(first[mask], features)              # True

Row breaks between the image tokens

The processor writes an image as one [IMG] per image token, row by row, with [IMG_BREAK] after every row but the last and [IMG_END] after the last. Only [IMG] is in vision.image_token_mask; the break and end tokens keep their own embeddings. On the pinned tiny checkpoint an 84 × 84 and a 56 × 112 image at 14-pixel patches are 36 and 32 patches and 9 and 8 [IMG] tokens, with 3 [IMG_BREAK] and 2 [IMG_END]. On every size the vision encoder is 24 blocks of width 1024 with 14-pixel patches, so an image token covers 28 × 28 pixels, and the processor resizes an image's longest side to at most 1540 pixels (longest_edge).

From the family module

04 · nnterp/families/ministral3.py

Ministral 3 (Ministral3ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. The queries are scaled by position (llama_4_scaling_beta) after the rotary embedding and before the interface, so attention_queries are the scaled ones.

The mistral3 wrapper puts a Pixtral tower at model.vision_tower, which is vision (a PixtralVision: every image's patches packed in one row, no final norm), and the projector model.multi_modal_projector (a norm, the patch_merger that merges each 2x2 patch block, two linears), which is projector; vision.image_features is read at the scatter (ImageScatter).