vision¶
Vision: an image-text-to-text checkpoint's vision tower, and its blocks.
A multimodal wrapper is a text model plus a vision tower, a projector, and a
step that scatters the projected features into the text stream at the image
tokens. The text model keeps the standard names it has everywhere; the tower
answers to model.vision, its blocks to vision.layers[i], and the last
module before the scatter to model.projector. The family names them in its
RENAME (keys that resolve only on the wrapper's tree) and keys these envoys
on the tower's module types in its ENVOYS.
The tower's blocks are pre-norm attention + MLP blocks on transformers' shared
attention interface (SigLIP, CLIP), so they are Layer, Attention and Mlp
with the stream values re-annotated as Patches: layer_input, layer_output,
attention_output and mlp_output mean what they mean on a text block,
vision.layers[i].layer_input + attention_output + mlp_output == layer_output. The
attention interior is inherited unchanged: attention_probabilities is a
Pattern whose batch axis is the tower's (one row per image) and whose
query/key axes are the image's patches. There is no causal mask (a
patch attends to every patch of its image); a tower that masks does so for
its own layout (Pixtral's block-diagonal mask between packed images, Gemma
4's padded keys). The tower's sizes are on Vision, read off its own config;
the root's sizes stay the text model's.
Where the image meets the text model is the tower's too: vision.image_token_mask
(which positions of the text batch hold an image token, off the root's inputs)
and vision.image_features (what the wrapper scatters into the token
embeddings, flat over those tokens, read at the scatter in the forward of the
module the family keys ImageScatter on), so
layers[0].input[vision.image_token_mask] == vision.image_features.
module-attribute
¶
PER_IMAGE = "the Qwen ViT's attention makes one interface call per image (attention_interface_2; one per window on Qwen2.5-VL's windowed blocks), so no one tensor is the block's pattern; read attention_queries and attention_keys, whole, and split them where the calls do: at the attention's cu_seqlens argument (self_attn.inputs[1]['cu_seqlens']), which per image is the processor's image_grid_thw"
Bases: Standard
The wrapper's inner model (model.model): the module whose forward scatters the image features into the token embeddings.
A family keys it on the wrapper model's type (LlavaModel: ImageScatter);
Vision.image_features reads the scatter_argument of the scatter
operation, the features in inputs_embeds.masked_scatter(image_mask,
image_features), which is what the text model receives at the image
tokens whatever the wrapper did after its projector (LLaVA-NeXT's
unpadding and newline tokens). A wrapper that writes the features in
elsewhere subclasses it with its own operation and argument. The forward
is instrumented at build (sourced), so the scatter is served after the
tower's values, which run inside the same forward. Carries no values of
its own.
A wrapper that scatters in the root's own forward (Llama 4's
Llama4ForConditionalGeneration) has no child to key: its family sets
ROOT_SCATTER to the operation's name, the root is the host
(scatter_host), and StandardizedTransformer instruments its own
forward at build.
Bases: Standard
model.vision: a vision tower's root. Its blocks are layers; its sizes are its own config's.
Values: patch_embeddings, the patch embedding's output one row per patch
(before any position embedding, CLS token or pre-norm the tower adds), and
tower_output, the last block's stream after the tower's final norm
where it has one, before any pooling, CLS dropping or adapter (the tower's
last_hidden_state, unless the tower returns something after those).
The host reads the tower its own way: Gemma 3 pools tower_output,
Llava 1.5 takes vision.layers[-2].layer_output and drops the CLS
token, so what reaches the projector is projector.input, not
necessarily tower_output. Where the image meets the text model:
image_token_mask, read off the model's inputs, and image_features,
read at the scatter (ImageScatter, keyed from the root), so
layers[0].input[image_token_mask] == image_features. Both need an image
to reach the model (no_images).
The sizes (num_layers, vision_hidden_size, num_heads, head_dim,
intermediate_size, patch_size, image_size) are read off the tower's
own config, the plain spellings SigLIP's and CLIP's use; a tower that
spells one its own way overrides the property on a subclass. The tower's
width is a different quantity from the text model's hidden_size and
is named apart, so axis_size on image_features (text width, served
here) walks past the tower to the root; num_heads, head_dim,
num_layers and intermediate_size keep the plain names and shadow
the root's on purpose, because the tower's values want the tower's numbers.
| ATTRIBUTE | DESCRIPTION |
|---|---|
layers |
The tower's blocks, each a
TYPE:
|
patch_embed |
The patch embedding (a convolution on SigLIP and CLIP).
TYPE:
|
norm |
The final norm over the patches, where the tower has one (SigLIP's
TYPE:
|
property
¶
Width of one head: the config's head_dim when it says, else vision_hidden_size // num_heads.
property
¶
Side of the square image the tower is configured for, in pixels; Unavailable on a tower with no fixed resolution (variable_resolution).
Why no image reaches the model this tower belongs to, or None: the one place that decides.
A wrapper whose family names no projector (where the image enters
the text model is unknown), or a load without a processor (a
task="text-generation" load of the wrapper).
Which tower values this checkpoint has, the way StandardizedTransformer.support reports the text model's.
The tower's own values (image_token_mask and image_features
among them) plus every block value over layers ("layer_output",
"self_attn.attention_probabilities", ...): None when available
on every block, else {layer: reason}. With layer, that block's
values alone. Empty on a load no image reaches (no_images): the tower
never runs, so the model's support has no vision rows, and a
read of an image value raises Unavailable saying why.
Bases: VisionAttention
The Qwen ViT's attention: one interface call per image (or window), so the interior is read whole around them.
attention_queries, attention_keys and attention_values are the
whole [1, heads, patches, head_dim] tensors (the queries and keys after
the 2D rotary embedding) before the module splits them per image, and
attention_head_outputs is the per-image outputs concatenated back,
[1, patches, heads, head_dim], under any implementation but flash.
attention_scores and attention_probabilities are Unavailable
(PER_IMAGE): loading eager does not help, since the module splits
under every implementation.
attention_queries(value: Tensor) -> Queries
Every image's queries, [1, heads, patches, head_dim], before the per-image split; assign or edit in place.
attention_keys(value: Tensor) -> Keys
Every image's keys, [1, heads, patches, head_dim], before the per-image split; assign or edit in place.
attention_values(value: Tensor) -> Values
Every image's values, [1, heads, patches, head_dim], before the per-image split; assign or edit in place.
attention_head_outputs(value: Tensor) -> HeadOutputs
Every image's per-head outputs concatenated back, [1, patches, heads, head_dim], before the output projection.
Bases: Vision
The Qwen ViT (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3.5): packed, its merger inside, at model.visual.
The tower runs on [patches, vision_hidden_size], every image's patches
concatenated, so its stream values are Patches with 1 in the images
axis, [1, patches, vision_hidden_size], and its blocks' attention is a
QwenVisionAttention.
The processor cuts each image into patch_size squares (two frames
deep), and lays them out in spatial_merge_size x spatial_merge_size
blocks, each block's patches consecutive, images one after another; the
merger folds each block into one image token. So patches runs over
every image of the invoke, image_grid_thw ([t, h, w] per image, in
patches) splits it, and an image's patches are in merge-block order, not
raster order. Qwen2.5-VL's tower further permutes them into attention
windows at entry: its blocks' values, tower_output and the merger's
output are in window order, and the tower restores the merge-block order
after the merger.
The tower has no final norm, so tower_output is the last block's
stream, the merger's input. It is served at the tower's output, after the
merger has run, so projector.input and projector.output are read
before tower_output.
Sizes, off the vision config: vision_hidden_size is the tower's width
(embed_dim on Qwen2-VL, whose config's hidden_size is the merger's
output width), num_heads, intermediate_size (Qwen2-VL's is
embed_dim * mlp_ratio), patch_size, spatial_merge_size and
window_size (Qwen2.5-VL's, in pixels; None on the others).
image_size is Unavailable: the tower takes any resolution.
Bases: Vision
Pixtral's tower (Mistral 3, Llava-Pixtral): packed, every image's patches in one row.
The convolution runs on the batch padded to its largest image, each image
is cropped to its own grid, and the grids are flattened and concatenated
into [1, all patches, vision_hidden_size] before ln_pre; the blocks
attend under a block-diagonal mask, so no patch sees another image's. So
Patches has a leading 1 here, and patch_embeddings is that packed
row as it enters ln_pre (the convolution's own output, patch_embed.output,
is still the padded grid). The processor's image_sizes split the row
per image: (height // patch_size) * (width // patch_size) patches each,
in order. The attention interior is whole: one interface call over the
packed row, so attention_probabilities is [1, heads, all patches,
all patches], zero between images. image_size is Unavailable: the
config's is the largest side the processor resizes to, not one size every
image has.
patch_embeddings(value: Tensor) -> Patches
Every image's patch embeddings, [1, all patches, vision_hidden_size], image after image, each in raster order.
What enters ln_pre: the convolution's output cropped per image and
concatenated. Assign a tensor of the same shape to replace it.
The id of the token the processor puts where an image's features go: image_token_id (image_token_index on older configs).
variable_resolution(vision: Vision) -> int
image_size on a tower with no fixed resolution: Unavailable, saying where each image's size is.
no_image_tokens(vision: Vision) -> str | None
Why vision.image_token_mask is unavailable, or None.
The module whose forward scatters the image features, with its path from the root, or None.
The root's child the family keys ImageScatter on, or the root itself
(path "") on a wrapper that scatters in its own forward, whose family
names the operation in ROOT_SCATTER (Llama 4).
The scatter's path from the root ("model.source.inputs_embeds_masked_scatter_0") and which argument is the image features.
no_image_features(vision: Vision) -> str | None
Why vision.image_features is unavailable, or None.
image_features is read at the scatter, in the forward of the module the
family keys ImageScatter on; a wrapper whose family keys none there binds
the tower's names but does not serve the value.
image_scatter(vision: Vision) -> str
The path of the scatter's arguments, from the root: "/model.source.inputs_embeds_masked_scatter_0.inputs".
scattered_argument(vision: Vision) -> int | str
Which argument of the scatter is the image features: ImageScatter.scatter_argument.
Why the tower envoy belongs to never runs (Vision.no_images), or None: the gate on every tower value.
A tower's stream as Patches: a packed tower's [patches, vision_hidden_size] gets a leading images axis of 1 (a view).
Patches back in the shape of the module's own tensor current: a packed tower's drops the leading 1.
no_tower_run, then absent: why a tower value at the child name is unavailable.
Why the Qwen ViT's attention_head_outputs is unavailable, or None: flash attention takes the other branch.