Qwen2-VL

The block, in order

The text block is Qwen2's: a pre-norm attention whose q_proj, k_proj and v_proj add a bias (o_proj has none), then a SwiGLU MLP with no biases, each behind an RMSNorm.

a   = input_layernorm(x)
h   = x + o_proj(attn(rope(q_proj(a)), rope(k_proj(a)), v_proj(a)))
out = h + mlp(post_attention_layernorm(h))     # SwiGLU, no biases

The block returns a tensor and adds both sublayers' outputs to the stream, so the identity is the plain sum:

with model.trace(prompt, images=[image]):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(x + attn + mlp, out)

Positions are three streams

The wrapper passes the text model position_ids of [3, batch, seq]: a temporal, a height and a width position per token. On text the three are equal and count up. Every token of an image takes the image's start position as its temporal position; its height and width positions are that start plus its row and its column in the image's grid of merged tokens. The text after an image resumes one past the image's largest position, so an image h tokens high and w wide advances the positions by max(h, w), not by h * w: after an image, a token's position is less than its index.

rotary = model.model.language_model.rotary_emb
with model.trace(prompt, images=[image]):
    mask = model.vision.image_token_mask.save()
    positions = rotary.inputs[0][1].save()          # [3, batch, seq]: time, height, width

positions[:, 0, mask[0]]                            # the image tokens' three positions

rotary_emb folds the three into one cos/sin by mrope_section, [16, 24, 24] on every checkpoint. Of each head's 64 rotary frequencies, the first 16 turn with the temporal position, the next 24 with the height and the last 24 with the width. In a 128-wide head, dimensions 0 to 15 and 64 to 79 therefore carry time, 16 to 39 and 80 to 103 the row, and 40 to 63 and 104 to 127 the column. The attention applies that pair with rotate_half before the attention interface, so attention_queries and attention_keys are the rotated queries and keys, and on a prompt with no image the rotation is the ordinary rotary one. rope_theta is 10⁶.

Grouped-query attention

2B has 12 query heads over 2 key/value heads, 7B 28 over 4, 72B 64 over 8; head_dim is 128 on all three. An edit to key/value head j on 2B reaches query heads 6j to 6j + 5. The score scale is head_dim ** -0.5; there is no window and no softcap. On the text blocks the six attention interior values need attn_implementation="eager". On the vision encoder's blocks attention_scores and attention_probabilities are unavailable under every implementation.

The image enters at the scatter

The wrapper writes vision.image_features into the token embeddings at the image tokens before block 0, so token_embeddings holds the <|image_pad|> embedding there and layers[0].input the features: they agree at the text positions only. layers[0].input[vision.image_token_mask] == vision.image_features holds exactly. To ablate or patch the image as the text model receives it, edit vision.image_features:

with model.trace(prompt, images=[image]):
    model.vision.image_features[:] = 0
    ablated = model.logits.save()

The readout is the plain projection

model.logits equals model.lm_head.output. On 2B the embedding and the unembedding are one matrix (tie_word_embeddings), so an edit to embed_tokens.weight edits lm_head; 7B and 72B have their own.

The chat template writes a system turn

The tokenizer prepends no BOS (bos_token is None). Given no system message, the Instruct template writes You are a helpful assistant., so a templated prompt opens with a system turn and position 0 is <|im_start|>. The generation configs end on <|im_end|> (151645) or <|endoftext|> (151643).

What loads as this family

Every checkpoint here is Qwen2VLForConditionalGeneration (model_type qwen2_vl) with a text_config of model_type qwen2_vl_text. transformers registers no causal-LM class for it, so every load is the wrapper with its processor, task="image-text-to-text": the text model at model.language_model, the vision encoder at model.visual as model.vision. The text model's names and values are the same whether or not a prompt carries an image.

The Qwen ViT vision encoder

Every image in one row

The vision encoder runs on [patches, vision_hidden_size], the patches of every image in the invoke concatenated. Its stream values (patch_embeddings, every block's layer_output, attention_output and mlp_output, tower_output) are served [1, patches, vision_hidden_size], a view with a leading images axis of 1, so in-place edits land; vision.layers[i].input and .output stay the native [patches, vision_hidden_size], and the block identity holds across the two by broadcasting. The processor's image_grid_thw ([t, h, w] per image, in patches; an image is two identical frames, so t is 1) splits the row: image j has t * h * w patches.

inputs = model.processor(text=prompt, images=[red, wide], return_tensors="pt")
sizes = inputs["image_grid_thw"].prod(-1).tolist()   # patches per image
with model.trace(prompt, images=[red, wide]):
    patches = model.vision.layers[1].layer_output.save()
per_image = patches[0].split(sizes)                  # one [t*h*w, vision_hidden_size] each

Inside an image the patches are in merge-block order: the spatial_merge_size × spatial_merge_size (2 × 2) block of neighbours the merger folds into one image token is four consecutive rows, blocks in raster order. Index 4 * b to 4 * b + 3 is block b, not a raster row.

The attention runs once per image, so there is no pattern

The attention computes queries, keys and values for the whole row, then calls the attention function once per image (once per window on Qwen2.5-VL's windowed blocks) and concatenates the outputs. attention_queries, attention_keys and attention_values are served whole, before the split, [1, heads, patches, head_dim], the queries and keys after the rotary embedding; attention_head_outputs is the calls' outputs concatenated back, [1, patches, heads, head_dim] (under any implementation but flash, which runs one call over cu_seqlens). attention_scores and attention_probabilities are Unavailable under every implementation, eager included: "no one tensor is the block's pattern". The split points are the attention's cu_seqlens argument, self_attn.inputs[1]["cu_seqlens"], so a pattern per image is a softmax over each slice:

attn = model.vision.layers[1].self_attn
with model.trace(prompt, images=[red, wide]):
    cu = attn.inputs[1]["cu_seqlens"].save()        # read first: the module's input
    q = attn.attention_queries.save()
    k = attn.attention_keys.save()

patterns = []
for a, b in zip(cu[:-1].tolist(), cu[1:].tolist()):
    scores = q[0, :, a:b] @ k[0, :, a:b].transpose(-1, -2) * model.vision.head_dim ** -0.5
    patterns.append(scores.softmax(-1))             # [heads, patches_j, patches_j]

Each pattern times that image's values reproduces attention_head_outputs exactly (checked on Qwen3.5's pinned tiny checkpoint).

The merger is inside the vision encoder

model.projector is the vision encoder's own merger: it norms each patch, concatenates each 2 × 2 block into one 4 * vision_hidden_size vector and runs a two-layer MLP to the text model's width, so image tokens per image are t * h * w / 4. It reads the last block's stream with no norm between (projector.input == tower_output[0], the native [patches, vision_hidden_size]). Because the merger runs inside the vision encoder's forward, vision.tower_output (the encoder's output) is reached after it: read model.projector.input and .output before vision.tower_output in a trace.

Sizes, and no `image_size`

model.vision reads its sizes off the vision config: vision_hidden_size is the encoder's width (embed_dim on Qwen2-VL, whose config's hidden_size is the merger's output width), num_heads, intermediate_size (embed_dim * mlp_ratio on Qwen2-VL), patch_size, and two of its own, spatial_merge_size (2 on every line) and window_size (Qwen2.5-VL's window side in pixels, 112; None on the others). image_size raises Unavailable: the processor cuts each image into its own grid.

The variants

line	block norms, merger norm	MLP	beyond the shared encoder
Qwen2-VL	LayerNorm	`fc1`, QuickGELU, `fc2`	none
Qwen2.5-VL	RMSNorm	SwiGLU, biased	windows of 112 pixels except on `fullatt_block_indexes`
Qwen3-VL, -MoE	LayerNorm	`linear_fc1`, GELU (tanh), `linear_fc2`	learned `pos_embed`; DeepStack taps
Qwen3.5, -MoE	LayerNorm	`linear_fc1`, GELU (tanh), `linear_fc2`	learned `pos_embed`

On Qwen2.5-VL the encoder permutes the patches into windows at entry, so its blocks' values, tower_output and projector.output are in window order, and it restores the merge-block order after the merger: vision.image_features is in scatter order. On Qwen3-VL and Qwen3.5 vision.layers[0].input is patch_embeddings plus the resampled pos_embed. On Qwen2-VL vision.layers[0].input is patch_embeddings; on Qwen2.5-VL it is patch_embeddings permuted into window order; on both, positions enter only through the rotary embedding.

The merger folds four patches into one image token

model.projector is model.visual.merger, the vision encoder's last module: ln_q (a LayerNorm over vision_hidden_size) on each patch, then each 2 × 2 block of consecutive patches concatenated into one 4 * vision_hidden_size vector, and mlp: Linear, GELU, Linear to the text model's width. On every checkpoint the vision encoder is 1280 wide, so the merger maps 5120 to 5120 and then to 1536 (2B), 3584 (7B) or 8192 (72B). A patch is 14 pixels, so an image token covers 28 × 28 pixels, and an image of t * h * w patches (its image_grid_thw row) is t * h * w / 4 image tokens: 256 for a 448 × 448 image. vision.image_features is model.projector.output as it is, [image_tokens, hidden_size].

with model.trace(prompt, images=[image]):
    mask = model.vision.image_token_mask.save()
    fed = model.projector.input.save()          # [patches, vision_hidden_size]
    merged = model.projector.output.save()
    features = model.vision.image_features.save()
    first = model.layers[0].input.save()

fed.shape[0] // 4 == mask.sum()                 # True: one image token per 2 x 2 block
torch.equal(merged, features)                   # True
torch.equal(first[mask], features)              # True

The image tokens sit between vision markers

The processor expands each <|image_pad|> (151655) in the prompt to the image's token count, between <|vision_start|> and <|vision_end|>; vision.image_token_mask marks the pads only. The Instruct chat template writes the three for an {"type": "image"} entry. The base checkpoints' template takes one content list, with no roles, and writes the same markers:

content = [{"type": "image"}, {"type": "text", "text": "A photo of"}]
prompt = model.processor.apply_chat_template(content)
# '<|vision_start|><|image_pad|><|vision_end|>A photo of'

Qwen2-VL

The block

The vision encoder

The API

Values, by host

Printout

Config

Vision Config

Config

Vision Config

Config

Vision Config

The Notes

The block, in order

Positions are three streams

Grouped-query attention

The image enters at the scatter

The readout is the plain projection

The chat template writes a system turn

What loads as this family

Vision notes

The Qwen ViT vision encoder

Every image in one row

The attention runs once per image, so there is no pattern

The merger is inside the vision encoder

Sizes, and no `image_size`

The variants

Qwen2-VL

The merger folds four patches into one image token

The image tokens sit between vision markers

From the family module

Qwen2-VL

The block

The vision encoder

The API

Values, by host

Printout

Config

Vision Config

Config

Vision Config

Config

Vision Config

The Notes

The block, in order

Positions are three streams

Grouped-query attention

The image enters at the scatter

The readout is the plain projection

The chat template writes a system turn

What loads as this family

Vision notes

The Qwen ViT vision encoder

Every image in one row

The attention runs once per image, so there is no pattern

The merger is inside the vision encoder

Sizes, and no image_size

The variants

Qwen2-VL

The merger folds four patches into one image token

The image tokens sit between vision markers

From the family module

Sizes, and no `image_size`