Qwen3-VL-MoE

The block, in order

h   = x + self_attn(input_layernorm(x))        # q_norm, k_norm per head, then M-RoPE
out = h + mlp(post_attention_layernorm(h))     # 8 of 128 experts per token
x   = out + deepstack[k] at the image tokens   # the text model, after blocks 0, 1, 2 only

The block is Qwen3-MoE's, with Llama's names. It adds the attention's output and the mixture's output to the stream and returns a tensor, so the contribution identity is the plain sum; the mixture has no shared expert, and mlp_output is routed_output:

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    routed = model.layers[1].mlp.routed_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(routed, mlp)
torch.testing.assert_close(x + attn + mlp, out)

The third line is DeepStack, done by the text model between the blocks, on an image prompt only.

The mixture: 128 experts, 8 per token, weights renormalized

layers[i].mlp is a mixture on every block (decoder_sparse_step 1, mlp_only_layers empty). The router, gate aliased router, gives one logit per expert; the scoring is a softmax, the 8 largest probabilities are kept, and the router divides them by their sum, so each token's expert_weights sum to one. It always divides: the config's norm_topk_prob (true) is not read.

moe = model.layers[1].mlp
with model.trace(prompt):
    logits = moe.router_logits.save()
    weights = moe.expert_weights.save()

top = logits.softmax(-1).topk(moe.top_k, dim=-1)
torch.testing.assert_close(weights, top.values / top.values.sum(-1, keepdim=True))

The experts are one fused module, experts, each a SwiGLU moe_intermediate_size wide (768 on 30B-A3B, 1536 on 235B-A22B); the config's intermediate_size (6144, 12288) is a dense width no block uses. expert_outputs is each routing slot's weighted output, [batch, seq, 8, hidden_size], and sums to routed_output. The router divides before expert_weights is read, so zeroing one expert's weight leaves the other slots' weights unchanged, and they sum to less than one:

with model.trace(prompt):
    moe.expert_weights = moe.expert_weights.masked_fill(moe.expert_indices == e, 0)
    ablated = model.logits.save()

After blocks 0 to 2, layer_output is not what the next block reads

On an image prompt the text model adds deepstack_output at the image positions between the first blocks (the family module's docstring gives the identity); vision.image_token_mask marks them:

with model.trace(prompt, images=[red]):
    mask = model.vision.image_token_mask.save()
    out = model.layers[0].layer_output.save()
    added = model.layers[0].deepstack_output.save()
    entering = model.layers[1].input.save()

torch.equal(entering[mask], out[mask] + added)   # True
torch.equal(entering[~mask], out[~mask])         # True

A patch from block 0's layer_output into block 1's input drops deepstack_output; a residual-stream SAE or probe at layer_output on blocks 0 to 2 sees the stream before the add. Ablating the image takes vision.image_features and all three deepstack_output (the Vision notes give the numbers). A text prompt makes no DeepStack call, and deepstack_output is not reached.

Queries and keys: normed per head, then a multimodal rotary

q_norm and k_norm are RMSNorms over one head's 128 dimensions, applied before the rotary; attention_queries and attention_keys are read at the attention interface, after both. The rotary is M-RoPE: each token has a time, a height and a width position, and mrope_section [24, 20, 20] gives the 64 rotary frequencies to the three, interleaved. The text model folds the three into one cos and sin before block 0. On text tokens the three positions are equal and count up by one. An image's tokens all take the time position the image starts at; their height and width positions are that start plus their row and column in the merged grid. The text after the image continues from the largest of these plus one, so after an image a token's rotary position is smaller than its index in input_ids. 30B-A3B has 32 query heads over 4 key/value heads, 235B-A22B 64 over 4; the score scale is 128 ** -0.5.

Loading

Every checkpoint is Qwen3VLMoeForConditionalGeneration (model_type qwen3_vl_moe), whose text_config is qwen3_vl_moe_text. With task="image-text-to-text", as this page builds them, the wrapper loads with its processor: the text model at model.language_model, the vision encoder at model.visual as model.vision. Transformers has no text-only class for it. A task="text-generation" load with dispatch=True builds the same wrapper without a processor, and there every vision value and deepstack_output raise Unavailable. attn_implementation="eager" is needed for the six attention interior values on the text blocks; the vision encoder's pattern is Unavailable under every load. expert_outputs needs experts_implementation "grouped_mm" (the default) or "batched_mm".

The prompt

What loads as this family

Qwen3-VL-30B-A3B and 235B-A22B, Instruct and Thinking: 48 and 94 blocks. rope_theta is 5·10⁶ and max_position_embeddings 262144 on each. The dense Qwen3-VL checkpoints, 2B to 32B, are qwen3_vl_text; the text-only Qwen3-MoE checkpoints are qwen3_moe.

The Qwen ViT vision encoder

Every image in one row

The vision encoder runs on [patches, vision_hidden_size], the patches of every image in the invoke concatenated. Its stream values (patch_embeddings, every block's layer_output, attention_output and mlp_output, tower_output) are served [1, patches, vision_hidden_size], a view with a leading images axis of 1, so in-place edits land; vision.layers[i].input and .output stay the native [patches, vision_hidden_size], and the block identity holds across the two by broadcasting. The processor's image_grid_thw ([t, h, w] per image, in patches; an image is two identical frames, so t is 1) splits the row: image j has t * h * w patches.

inputs = model.processor(text=prompt, images=[red, wide], return_tensors="pt")
sizes = inputs["image_grid_thw"].prod(-1).tolist()   # patches per image
with model.trace(prompt, images=[red, wide]):
    patches = model.vision.layers[1].layer_output.save()
per_image = patches[0].split(sizes)                  # one [t*h*w, vision_hidden_size] each

Inside an image the patches are in merge-block order: the spatial_merge_size × spatial_merge_size (2 × 2) block of neighbours the merger folds into one image token is four consecutive rows, blocks in raster order. Index 4 * b to 4 * b + 3 is block b, not a raster row.

The attention runs once per image, so there is no pattern

The attention computes queries, keys and values for the whole row, then calls the attention function once per image (once per window on Qwen2.5-VL's windowed blocks) and concatenates the outputs. attention_queries, attention_keys and attention_values are served whole, before the split, [1, heads, patches, head_dim], the queries and keys after the rotary embedding; attention_head_outputs is the calls' outputs concatenated back, [1, patches, heads, head_dim] (under any implementation but flash, which runs one call over cu_seqlens). attention_scores and attention_probabilities are Unavailable under every implementation, eager included: "no one tensor is the block's pattern". The split points are the attention's cu_seqlens argument, self_attn.inputs[1]["cu_seqlens"], so a pattern per image is a softmax over each slice:

attn = model.vision.layers[1].self_attn
with model.trace(prompt, images=[red, wide]):
    cu = attn.inputs[1]["cu_seqlens"].save()        # read first: the module's input
    q = attn.attention_queries.save()
    k = attn.attention_keys.save()

patterns = []
for a, b in zip(cu[:-1].tolist(), cu[1:].tolist()):
    scores = q[0, :, a:b] @ k[0, :, a:b].transpose(-1, -2) * model.vision.head_dim ** -0.5
    patterns.append(scores.softmax(-1))             # [heads, patches_j, patches_j]

Each pattern times that image's values reproduces attention_head_outputs exactly (checked on Qwen3.5's pinned tiny checkpoint).

The merger is inside the vision encoder

model.projector is the vision encoder's own merger: it norms each patch, concatenates each 2 × 2 block into one 4 * vision_hidden_size vector and runs a two-layer MLP to the text model's width, so image tokens per image are t * h * w / 4. It reads the last block's stream with no norm between (projector.input == tower_output[0], the native [patches, vision_hidden_size]). Because the merger runs inside the vision encoder's forward, vision.tower_output (the encoder's output) is reached after it: read model.projector.input and .output before vision.tower_output in a trace.

Sizes, and no `image_size`

model.vision reads its sizes off the vision config: vision_hidden_size is the encoder's width (embed_dim on Qwen2-VL, whose config's hidden_size is the merger's output width), num_heads, intermediate_size (embed_dim * mlp_ratio on Qwen2-VL), patch_size, and two of its own, spatial_merge_size (2 on every line) and window_size (Qwen2.5-VL's window side in pixels, 112; None on the others). image_size raises Unavailable: the processor cuts each image into its own grid.

The variants

line	block norms, merger norm	MLP	beyond the shared encoder
Qwen2-VL	LayerNorm	`fc1`, QuickGELU, `fc2`	none
Qwen2.5-VL	RMSNorm	SwiGLU, biased	windows of 112 pixels except on `fullatt_block_indexes`
Qwen3-VL, -MoE	LayerNorm	`linear_fc1`, GELU (tanh), `linear_fc2`	learned `pos_embed`; DeepStack taps
Qwen3.5, -MoE	LayerNorm	`linear_fc1`, GELU (tanh), `linear_fc2`	learned `pos_embed`

On Qwen2.5-VL the encoder permutes the patches into windows at entry, so its blocks' values, tower_output and projector.output are in window order, and it restores the merge-block order after the merger: vision.image_features is in scatter order. On Qwen3-VL and Qwen3.5 vision.layers[0].input is patch_embeddings plus the resampled pos_embed. On Qwen2-VL vision.layers[0].input is patch_embeddings; on Qwen2.5-VL it is patch_embeddings permuted into window order; on both, positions enter only through the rotary embedding.

The merger folds four patches into one image token

model.projector is model.visual.merger, the vision encoder's last module: norm (a LayerNorm over vision_hidden_size) on each patch, then each 2 × 2 block of consecutive patches concatenated into one 4 * vision_hidden_size vector, linear_fc1, GELU and linear_fc2 to the text model's width. An image of t * h * w patches (its image_grid_thw row) is t * h * w / 4 image tokens, and vision.image_features is model.projector.output as it is, [image_tokens, hidden_size]. A patch is 16 pixels, so an image token covers 32 × 32. The merger maps 4 × 1152 = 4608 to 2048 on 30B-A3B and to 4096 on 235B-A22B.

with model.trace(prompt, images=[red, wide]):
    mask = model.vision.image_token_mask.save()
    fed = model.projector.input.save()          # [patches, vision_hidden_size]
    features = model.vision.image_features.save()
    first = model.layers[0].input.save()

fed.shape[0] // 4 == mask.sum()                 # True: one image token per 2 x 2 block
torch.equal(first[mask], features)              # True

A learned position embedding follows patch_embed

The vision encoder adds pos_embed, a learned table of 48 × 48 positions resampled to each image's grid, to the patch embedding before block 0, so vision.layers[0].input is not vision.patch_embeddings. Positions also enter every vision block through the 2D rotary embedding.

DeepStack: three more mergers, under their native names

vision_config.deepstack_visual_indexes names three vision blocks, 8, 16 and 24 of 27, on every checkpoint. After each, its own merger, vision.deepstack_merger_list[k], reads that block's output and folds it to image tokens as the main merger does, with its LayerNorm after the 2 × 2 concatenation (over 4 * vision_hidden_size) instead of before it. The mergers keep their native path; their outputs reach the text model as model.layers[k].deepstack_output, added at the image positions after text block k:

with model.trace(prompt, images=[red]):
    tapped = model.vision.deepstack_merger_list[0].output.save()
    added = model.layers[0].deepstack_output.save()

torch.equal(added, tapped)                       # True

vision.image_features is therefore not the only way the image reaches the text model. On the dense Qwen/Qwen3-VL-4B-Instruct, which has the same DeepStack, asked the color of a red square on white, zeroing vision.image_features alone still answers "Red"; zeroing deepstack_output on blocks 0 to 2 as well answers "White".

with model.trace(prompt, images=[red]):
    model.vision.image_features[:] = 0
    for k in range(3):
        model.layers[k].deepstack_output[:] = 0
    ablated = model.logits.save()

Read the vision values in the forward's order

The deepstack mergers run inside the vision encoder, each right after the block it taps, and the merger runs inside it too, before the encoder returns. In one trace, read a tapped block's layer_output, then deepstack_merger_list[k], before any later vision block. Read model.projector.input and .output before vision.tower_output, then vision.image_features. Then read, per text block, layer_output, deepstack_output and the next block's input. A read out of this order raises OutOfOrderError.

Qwen3-VL-MoE

The block

The vision encoder

The API

Values, by host

Printout

Config

Vision Config

Config

Vision Config

The Notes

The block, in order

The mixture: 128 experts, 8 per token, weights renormalized

After blocks 0 to 2, layer_output is not what the next block reads

Queries and keys: normed per head, then a multimodal rotary

Loading

The prompt

What loads as this family

Vision notes

The Qwen ViT vision encoder

Every image in one row

The attention runs once per image, so there is no pattern

The merger is inside the vision encoder

Sizes, and no `image_size`

The variants

Qwen3-VL-MoE

The merger folds four patches into one image token

A learned position embedding follows patch_embed

DeepStack: three more mergers, under their native names

Read the vision values in the forward's order

From the family module

Qwen3-VL-MoE

The block

The vision encoder

The API

Values, by host

Printout

Config

Vision Config

Config

Vision Config

The Notes

The block, in order

The mixture: 128 experts, 8 per token, weights renormalized

After blocks 0 to 2, layer_output is not what the next block reads

Queries and keys: normed per head, then a multimodal rotary

Loading

The prompt

What loads as this family

Vision notes

The Qwen ViT vision encoder

Every image in one row

The attention runs once per image, so there is no pattern

The merger is inside the vision encoder

Sizes, and no image_size

The variants

Qwen3-VL-MoE

The merger folds four patches into one image token

A learned position embedding follows patch_embed

DeepStack: three more mergers, under their native names

Read the vision values in the forward's order

From the family module

Sizes, and no `image_size`