qwen3_vl_text¶
Qwen3-VL, text (Qwen3VLTextModel, model_type qwen3_vl_text).
The language model inside Qwen3VLForConditionalGeneration (model_type
qwen3_vl): Qwen3's block (q_norm/k_norm inside the attention,
head_dim the config's) under interleaved multimodal rotary embeddings
(M-RoPE), folded into one cos/sin pair before any block runs and
applied inside the attention after the q/k norms and before the shared
interface, so the base Attention holds. Every checkpoint loads as the
wrapper (task="image-text-to-text"), whose text stack sits at
model.language_model.
DeepStack. The tower taps three of its blocks
(vision_config.deepstack_visual_indexes), merges each tap with its own
deepstack_merger_list[k], and the text model adds merger k's output at
the image positions after text block k, outside the block, in its own
forward (_deepstack_process). So on blocks k < 3 (fewer on a model
with fewer blocks) the stream entering block k + 1 is not block k's
layer_output at the image positions. Layer.deepstack_output serves what
is added there, [image_tokens, hidden_size]:
layers[k+1].input[mask] == layers[k].layer_output[mask] + layers[k].deepstack_output,
the rest of the stream unchanged; on the other blocks it is Unavailable and
layers[k+1].input == layers[k].layer_output.
The wrapper's Qwen ViT model.visual is vision (a QwenVision: packed,
[1, patches, vision_hidden_size], with a learned pos_embed added after
patch_embed); its merger is projector. The deepstack mergers keep
their native path, vision.deepstack_merger_list[k].
module-attribute
¶
RENAME = {'model.language_model.embed_tokens': 'embed_tokens', 'model.language_model.layers': 'layers', 'model.language_model.norm': 'norm', 'model.visual': 'vision', 'model.visual.merger': 'projector', 'blocks': 'layers', 'attn': 'self_attn', 'norm1': 'input_layernorm', 'norm2': 'post_attention_layernorm', 'attn.proj': 'attn.o_proj', 'linear_fc2': 'down_proj'}
module-attribute
¶
ENVOYS = {Qwen3VLModel: ImageScatter, Qwen3VLTextModel: TextModel, Qwen3VLTextDecoderLayer: Layer, Qwen3VLTextAttention: Attention, Qwen3VLTextMLP: Mlp, Qwen3VLVisionModel: QwenVision, Qwen3VLVisionBlock: VisionLayer, Qwen3VLVisionAttention: QwenVisionAttention, Qwen3VLVisionMLP: VisionMlp}
DeepstackEProperty(key: str | Callable[[Envoy], str] | None = None, description: str | None = None, unavailable: str | Callable[[Envoy], str | None] | None = None, select: int | str | Callable[[Envoy], int | str | None] | None = None)
Bases: EProperty
An EProperty at the text model's _deepstack_process call, pinned to this block's occurrence of it.
The call runs once after each of the first blocks, from one line of the
text model's loop, so its location is the same for every block and block
k's is its k-th occurrence. A read or write here asks for that
occurrence explicitly (pinned) rather than the next one the model reaches.
How many blocks receive a deepstack feature: one per tapped tower block, at most the text model's depth.
Why this block has no deepstack_output, or None.