nnterp · encyclopedia qwen3_5_text · nnterp.families.qwen3_5_text

family qwen3_5_text

Qwen3.5Qwen3.6

Llama's pre-norm block with a gated DeltaNet mixer on three blocks in four and, on the fourth, attention whose q_proj also yields a sigmoid gate on the heads' output; every RMSNorm but the mixer's own multiplies by 1 + weight.

HybridGated queryQuery/key normsPartial rotary1 + weight norm gainMultimodal rotaryGrouped-query attentionVision-languagePacked vision encoderVariable resolutionPooled projector

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card. The slider picks a block, each tick's color gives its block's shape, and the diagram redraws for the block picked.

embed_tokenstoken_embeddings
→
layers× 24
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + linear_attn.attention_output + mlp.mlp_output == layer_output

The vision encoder

· Qwen ViT · Qwen3_5VisionModel

The image's path into the text model, then one block of the vision encoder. Hover any part for its nnterp name; click to pin the card.

imagepixel_values
→
patch_embedpatch_embeddings
→
layers× 12 · tower_output
→
projectorthe merger's input: the last block's output
→
image_featuresat the scatter
→
layers[0].inputat image_token_mask
rows
One row, packed: every image's patches, [1, patches, vision_hidden_size], split per image by the processor's image_grid_thw. Each image's patches are in merge-block order (each 2 × 2 block the merger folds is consecutive), and on Qwen2.5-VL in window order inside the vision encoder.
positions
A 2D rotary embedding (row and column of the patch) turns the queries and keys inside each attention. Qwen3-VL and Qwen3.5 also add a learned pos_embed, resampled to each image's grid, after patch_embed.
masking
The attention runs once per image, over that image's patches, so no patch attends to another image's; on Qwen2.5-VL's windowed blocks once per window.
final norm
None after the last block: the merger norms each patch as it reads it, so there is no vision.norm.

identity vision.layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Qwen3_5ForConditionalGenerationlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] Qwen3_5DecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].linear_attn Qwen3_5GatedDeltaNetlayoutwhat it isread at
attention_output Residual [batch seq hidden_size] What the mixer adds to the residual stream

the module's own output
⚠︎state State [batch heads key_dim value_dim] The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')

inside the forward, at the operation that _token_state_op finds on each call
⚠︎states States [batch seq heads key_dim value_dim] The recurrent state after every token of this call; needs route_kernels(family, 'torch')

inside the forward, at the operation that _states finds on each call
attention_queries LinearQK [batch seq heads key_dim] The queries entering the delta rule

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 0
attention_keys LinearQK [batch seq heads key_dim] The keys entering the delta rule

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 1
attention_values LinearV [batch seq heads value_dim] The values entering the delta rule

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 2
decays Gates [batch seq heads] The per-token log decay of the recurrent state

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), argument g
betas Gates [batch seq heads] The per-token write strength into the state

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), argument beta
state_input State | None [batch heads key_dim value_dim] The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), argument initial_state
attention_head_outputs LinearV [batch seq heads value_dim] The per-head outputs before the gated norm and the output projection

inside the forward: the output of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 0
state_output State [batch heads key_dim value_dim] The recurrent state leaving the layer

inside the forward: the output of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 1
model.layers[i].self_attn Qwen3_5Attentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp Qwen3_5MLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output

the vision encoder · Qwen ViT

model.vision Qwen3_5VisionModellayoutwhat it isread at
image_token_mask ImageTokenMask [batch seq] Which positions hold image tokens: input_ids == the config's image_token_id; read-only

the model's own inputs, at the root
patch_embeddings Patches [images patches vision_hidden_size] The patch embedding's output, one row per patch

the child module patch_embed's output
tower_output Patches [images patches vision_hidden_size] The last block's stream after the tower's final norm, before any pooling or adapter

the module's own output
image_features ImageFeatures [image_tokens hidden_size] The image features the text model receives at the image tokens, flat over them

inside model.model's forward: the arguments of inputs_embeds_masked_scatter_0, element 1
model.vision.layers[i] Qwen3_5VisionBlocklayoutwhat it isread at
layer_input Patches [images patches vision_hidden_size] The tower's stream entering the block

the module's own input
attention_norm_output Patches [images patches vision_hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Patches [images patches vision_hidden_size] The tower's stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Patches [images patches vision_hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Patches [images patches vision_hidden_size] What the MLP is called with

the child module mlp's input
layer_output Patches [images patches vision_hidden_size] The tower's stream leaving the block

the module's own output
model.vision.layers[i].self_attn Qwen3_5VisionAttentionlayoutwhat it isread at
attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the output of unsqueeze_0
attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the output of unsqueeze_1
attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the output of unsqueeze_2
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Patches [images patches vision_hidden_size] What the attention adds to the tower's stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of torch_cat_0
model.vision.layers[i].mlp Qwen3_5VisionMLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives

the child module down_proj's input
mlp_output Patches [images patches vision_hidden_size] What the MLP adds to the tower's stream

the module's own output

Printout

What print(model) shows on Qwen/Qwen3.5-0.8B: the native modules, standard names and every standard value with its layout.

Qwen3_5ForConditionalGeneration(
  (lm_head): Linear(in_features=1024, out_features=248320, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(248320, 1024)
  (layers): ModuleList(
    (0-2): 3 x Qwen3_5DecoderLayer(
      (linear_attn): Qwen3_5GatedDeltaNet(
        (conv1d): Conv1d(6144, 6144, kernel_size=(4,), stride=(1,), padding=(3,), groups=6144, bias=False)
        (norm): Qwen3_5RMSNormGated()
        (out_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (in_proj_qkv): Linear(in_features=1024, out_features=6144, bias=False)
        (in_proj_z): Linear(in_features=1024, out_features=2048, bias=False)
        (in_proj_b): Linear(in_features=1024, out_features=16, bias=False)
        (in_proj_a): Linear(in_features=1024, out_features=16, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (3): Qwen3_5DecoderLayer(
      (self_attn): Qwen3_5Attention(
        (q_proj): Linear(in_features=1024, out_features=4096, bias=False)
        (k_proj): Linear(in_features=1024, out_features=512, bias=False)
        (v_proj): Linear(in_features=1024, out_features=512, bias=False)
        (o_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (q_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (4-6): 3 x Qwen3_5DecoderLayer(
      (linear_attn): Qwen3_5GatedDeltaNet(
        (conv1d): Conv1d(6144, 6144, kernel_size=(4,), stride=(1,), padding=(3,), groups=6144, bias=False)
        (norm): Qwen3_5RMSNormGated()
        (out_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (in_proj_qkv): Linear(in_features=1024, out_features=6144, bias=False)
        (in_proj_z): Linear(in_features=1024, out_features=2048, bias=False)
        (in_proj_b): Linear(in_features=1024, out_features=16, bias=False)
        (in_proj_a): Linear(in_features=1024, out_features=16, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (7): Qwen3_5DecoderLayer(
      (self_attn): Qwen3_5Attention(
        (q_proj): Linear(in_features=1024, out_features=4096, bias=False)
        (k_proj): Linear(in_features=1024, out_features=512, bias=False)
        (v_proj): Linear(in_features=1024, out_features=512, bias=False)
        (o_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (q_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (8-10): 3 x Qwen3_5DecoderLayer(
      (linear_attn): Qwen3_5GatedDeltaNet(
        (conv1d): Conv1d(6144, 6144, kernel_size=(4,), stride=(1,), padding=(3,), groups=6144, bias=False)
        (norm): Qwen3_5RMSNormGated()
        (out_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (in_proj_qkv): Linear(in_features=1024, out_features=6144, bias=False)
        (in_proj_z): Linear(in_features=1024, out_features=2048, bias=False)
        (in_proj_b): Linear(in_features=1024, out_features=16, bias=False)
        (in_proj_a): Linear(in_features=1024, out_features=16, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (11): Qwen3_5DecoderLayer(
      (self_attn): Qwen3_5Attention(
        (q_proj): Linear(in_features=1024, out_features=4096, bias=False)
        (k_proj): Linear(in_features=1024, out_features=512, bias=False)
        (v_proj): Linear(in_features=1024, out_features=512, bias=False)
        (o_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (q_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (12-14): 3 x Qwen3_5DecoderLayer(
      (linear_attn): Qwen3_5GatedDeltaNet(
        (conv1d): Conv1d(6144, 6144, kernel_size=(4,), stride=(1,), padding=(3,), groups=6144, bias=False)
        (norm): Qwen3_5RMSNormGated()
        (out_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (in_proj_qkv): Linear(in_features=1024, out_features=6144, bias=False)
        (in_proj_z): Linear(in_features=1024, out_features=2048, bias=False)
        (in_proj_b): Linear(in_features=1024, out_features=16, bias=False)
        (in_proj_a): Linear(in_features=1024, out_features=16, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (15): Qwen3_5DecoderLayer(
      (self_attn): Qwen3_5Attention(
        (q_proj): Linear(in_features=1024, out_features=4096, bias=False)
        (k_proj): Linear(in_features=1024, out_features=512, bias=False)
        (v_proj): Linear(in_features=1024, out_features=512, bias=False)
        (o_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (q_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (16-18): 3 x Qwen3_5DecoderLayer(
      (linear_attn): Qwen3_5GatedDeltaNet(
        (conv1d): Conv1d(6144, 6144, kernel_size=(4,), stride=(1,), padding=(3,), groups=6144, bias=False)
        (norm): Qwen3_5RMSNormGated()
        (out_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (in_proj_qkv): Linear(in_features=1024, out_features=6144, bias=False)
        (in_proj_z): Linear(in_features=1024, out_features=2048, bias=False)
        (in_proj_b): Linear(in_features=1024, out_features=16, bias=False)
        (in_proj_a): Linear(in_features=1024, out_features=16, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (19): Qwen3_5DecoderLayer(
      (self_attn): Qwen3_5Attention(
        (q_proj): Linear(in_features=1024, out_features=4096, bias=False)
        (k_proj): Linear(in_features=1024, out_features=512, bias=False)
        (v_proj): Linear(in_features=1024, out_features=512, bias=False)
        (o_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (q_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (20-22): 3 x Qwen3_5DecoderLayer(
      (linear_attn): Qwen3_5GatedDeltaNet(
        (conv1d): Conv1d(6144, 6144, kernel_size=(4,), stride=(1,), padding=(3,), groups=6144, bias=False)
        (norm): Qwen3_5RMSNormGated()
        (out_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (in_proj_qkv): Linear(in_features=1024, out_features=6144, bias=False)
        (in_proj_z): Linear(in_features=1024, out_features=2048, bias=False)
        (in_proj_b): Linear(in_features=1024, out_features=16, bias=False)
        (in_proj_a): Linear(in_features=1024, out_features=16, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (23): Qwen3_5DecoderLayer(
      (self_attn): Qwen3_5Attention(
        (q_proj): Linear(in_features=1024, out_features=4096, bias=False)
        (k_proj): Linear(in_features=1024, out_features=512, bias=False)
        (v_proj): Linear(in_features=1024, out_features=512, bias=False)
        (o_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (q_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5RMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): Qwen3_5RMSNorm((1024,), eps=1e-06)
  (vision): Qwen3_5VisionModel(
    (patch_embed): Qwen3_5VisionPatchEmbed(
      (proj): Conv3d(3, 768, kernel_size=(2, 16, 16), stride=(2, 16, 16))
    )
    (pos_embed): Embedding(2304, 768)
    (rotary_pos_emb): Qwen3_5VisionRotaryEmbedding()
    (layers/blocks): ModuleList(
      (0-11): 12 x Qwen3_5VisionBlock(
        (input_layernorm/norm1): LayerNorm((768,), eps=1e-06, elementwise_affine=True, bias=True)
        (post_attention_layernorm/norm2): LayerNorm((768,), eps=1e-06, elementwise_affine=True, bias=True)
        (self_attn/attn): Qwen3_5VisionAttention(
          (qkv): Linear(in_features=768, out_features=2304, bias=True)
          (o_proj/proj): Linear(in_features=768, out_features=768, bias=True)
          (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
          (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
          (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
          (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
          (attention_output) -> Patches [images patches vision_hidden_size]: What the attention adds to the tower's stream
          (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
          (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
          (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        )
        (mlp): Qwen3_5VisionMLP(
          (linear_fc1): Linear(in_features=768, out_features=3072, bias=True)
          (down_proj/linear_fc2): Linear(in_features=3072, out_features=768, bias=True)
          (act_fn): GELUTanh()
          (mlp_output) -> Patches [images patches vision_hidden_size]: What the MLP adds to the tower's stream
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives
        )
        (layer_input) -> Patches [images patches vision_hidden_size]: The tower's stream entering the block
        (attention_norm_output) -> Patches [images patches vision_hidden_size]: The normed stream entering the attention
        (layer_mid) -> Patches [images patches vision_hidden_size]: The tower's stream after the attention add
        (mlp_norm_output) -> Patches [images patches vision_hidden_size]: The normed stream entering the MLP
        (mlp_input) -> Patches [images patches vision_hidden_size]: What the MLP is called with
        (layer_output) -> Patches [images patches vision_hidden_size]: The tower's stream leaving the block
      )
    )
    (merger): Qwen3_5VisionPatchMerger(
      (norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True, bias=True)
      (linear_fc1): Linear(in_features=3072, out_features=3072, bias=True)
      (act_fn): GELU(approximate='none')
      (down_proj/linear_fc2): Linear(in_features=3072, out_features=1024, bias=True)
    )
    (image_token_mask) -> ImageTokenMask [batch seq]: Which positions hold image tokens: input_ids == the config's image_token_id; read-only
    (patch_embeddings) -> Patches [images patches vision_hidden_size]: The patch embedding's output, one row per patch
    (tower_output) -> Patches [images patches vision_hidden_size]: The last block's stream after the tower's final norm, before any pooling or adapter
    (image_features) -> ImageFeatures [image_tokens hidden_size]: The image features the text model receives at the image tokens, flat over them
  )
  (projector): Qwen3_5VisionPatchMerger(
    (norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True, bias=True)
    (linear_fc1): Linear(in_features=3072, out_features=3072, bias=True)
    (act_fn): GELU(approximate='none')
    (down_proj/linear_fc2): Linear(in_features=3072, out_features=1024, bias=True)
  )
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers24
model.hidden_size1024
model.num_heads8
model.num_kv_heads2
model.head_dim256
model.qk_head_dim256
model.vocab_size248320
model.intermediate_size3584
architecturesQwen3_5ForConditionalGeneration
num_hidden_layers24
hidden_size1024
intermediate_size3584
num_attention_heads8
num_key_value_heads2
head_dim256
vocab_size248320
max_position_embeddings262144
hidden_actsilu
rms_norm_eps1e-06
tie_word_embeddingsTrue
linear_conv_kernel_dim4
linear_num_key_heads16
linear_num_value_heads16
linear_key_head_dim128
linear_value_head_dim128
full_attention_interval4
attention_biasFalse
Vision Config

Vision Config

model.vision.num_layers12
model.vision.vision_hidden_size768
model.vision.num_heads12
model.vision.head_dim64
model.vision.intermediate_size3072
model.vision.patch_size16
model.vision.image_sizevaries
model.vision.spatial_merge_size2
model.vision.window_sizeNone

The Notes

03 · what to know before you run

The block, in order

h   = x + mixer(input_layernorm(x))          # linear_attn (gated DeltaNet) or self_attn
out = h + mlp(post_attention_layernorm(h))   # SwiGLU, the same class on every block

config.layer_types is full_attention on every fourth block (full_attention_interval 4, blocks 3, 7, 11, ...) and linear_attention on the others: 18 DeltaNet and 6 attention blocks on 0.8B and 2B, 24 and 8 on 4B and 9B, 48 and 16 on 27B. A block has linear_attn or self_attn, never both; the last block is always an attention block. Pick blocks outside the trace:

types = model.config.get_text_config().layer_types
delta = [i for i, t in enumerate(types) if t == "linear_attention"]
full = [i for i, t in enumerate(types) if t == "full_attention"]    # 3, 7, ..., 23 on 0.8B

The contributions are the sublayers' outputs

The block adds the mixer's output and the MLP's output to the stream and returns a tensor, so the identity is the plain sum, exact in float32 on both kinds of block:

with model.trace(prompt):
    x = model.layers[1].input.save()
    mix = model.layers[1].linear_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(x + mix + mlp, out)

On an attention block the middle term is model.layers[3].self_attn.attention_output.

Loading

attn_implementation="eager" is needed only for the six attention interior values on the attention blocks; there is no softcap or window, and on 0.8B in float32 the default sdpa load and the eager one give logits within 2e-5 of each other. The DeltaNet values are read at the call to the delta-rule kernel and need no eager load. They need transformers' pure-torch delta rule, which is what runs when flash-linear-attention is not installed (transformers then warns that it falls back to it). With it installed, every linear_attn value but attention_output is unavailable until nnterp.route_kernels(model.family, "torch"). The same call is what makes state and states exist: it routes a prompt through the token-by-token kernel, which on 0.8B on CPU took about twice as long on a 250-token prompt and moved the logits by 2e-5. states holds one float32 state per token, num_v_heads × 128 × 128: 1 MiB per token per block on 0.8B and 2B, 2 MiB on 4B and 9B, 3 MiB on 27B.

The attention block gates its heads' output

q_proj is 2 × num_heads × head_dim wide (4096 on 0.8B) and holds, for each head in turn, its 256 query dimensions followed by its 256 gate dimensions, so the first half of q_proj.output is not the queries. The heads' output is multiplied by the sigmoid of that gate before o_proj:

q, gate = split per head(q_proj(x))             # [..., num_heads, 256] each
q = rope(q_norm(q)); k = rope(k_norm(k_proj(x))); v = v_proj(x)
out = o_proj(attention(q, k, v).flatten() * sigmoid(gate.flatten()))

attention_queries is read at the attention interface, after the split, the norm and the rotary; attention_head_outputs is read there too, before the gate. The gate is mostly closed: on block 3 its sigmoid averages 0.07 on 0.8B (78% of entries under 0.1) and 0.03 on 9B (96%), so what reaches o_proj is a small fraction of attention_head_outputs, and an edit to a head's output reaches the stream multiplied by its gate. o_proj.input is the gated tensor:

attn = model.layers[3].self_attn
d = model.head_dim
with model.trace(prompt):
    q_and_gate = attn.q_proj.output.save()            # heads * 2 * head_dim wide
query, gate = q_and_gate.unflatten(-1, (-1, 2 * d)).chunk(2, dim=-1)

with model.trace(prompt):
    heads = attn.attention_head_outputs.save()        # before the gate
    gated = attn.o_proj.input.save()                  # after it

torch.testing.assert_close(gated, (heads * gate.sigmoid()).flatten(2))

Queries and keys: normed per head, a quarter of each rotated

q_norm and k_norm are RMSNorms over one head's 256 dimensions with gain 1 + weight, applied before the rotary. The rotary turns only the first 64 dimensions of each query and key head (partial_rotary_factor 0.25, rope_theta 10⁷); the other 192 carry no position, and attention_queries[..., 64:] equals q_norm.output[..., 64:] exactly. The rotary is M-RoPE: mrope_section [11, 11, 10] gives the 32 rotary frequencies to time, height and width, interleaved, which matters on image and video tokens. On text the three positions are equal, and the rotation is the standard rotate_half one. 0.8B and 2B have 8 query heads over 2 key/value heads, 4B and 9B 16 over 4, 27B 24 over 4, so an edit to key/value head j on 0.8B reaches query heads 4j to 4j + 3. The score scale is 256 ** -0.5.

The DeltaNet mixer

q, k, v = split(silu(conv1d(in_proj_qkv(x))))     # causal, width 4
beta = sigmoid(in_proj_b(x))
g = -exp(A_log) * softplus(in_proj_a(x) + dt_bias)
y, state = delta_rule(l2norm(q) / sqrt(128), l2norm(k), v, g, beta)
out = out_proj(norm(y, gate=silu(in_proj_z(x))))  # gain: weight

attention_queries, attention_keys and attention_values are the kernel's arguments: after the convolution and the SiLU, before the kernel's l2 norm and the queries' 1/sqrt(128) scale (on 0.8B's block 0 the queries' norms run from 0.09 to 4.1). A recurrence written on these values without the l2 norm and the scale leaves a 42% error in block 0's final state on 0.8B; with them, its output matches attention_head_outputs to 2e-8. Each head's key and value are 128 wide. 0.8B and 2B have 16 key heads and 16 value heads; 4B and 9B 16 key heads over 32 value heads, 27B over 48. The queries and keys are repeated up to the value heads (repeat_interleave) before the kernel, so there they are served num_v_heads wide, with heads 2j and 2j + 1 (on 4B and 9B) copies of key head j, and an edit to one copy reaches that one value head.

decays is one log decay per head and token, betas the sigmoid write strength, so each token keeps decays.exp() of the state. On 0.8B's block 0 the heads' mean kept fraction runs from 0.05 (a head that forgets within a few tokens) to above 0.999. attention_head_outputs is y, before the gated norm and out_proj. The state after a prompt, state_output, is [batch, num_v_heads, 128, 128].

Position 0 is the text's first token, with no massive activation

The tokenizer sets bos_token to None and prepends nothing; <|endoftext|> is 248044, <|im_start|> 248045, <|im_end|> 248046. Position 0 does not grow into an outlier in the stream: across the prompts tried, its layer_output norm is at most 6 times the median of the other positions on 0.8B (18.6 at most, on block 14) and at most 3.5 times on 9B. Position 0 still draws attention: from query 8 on, the attention blocks put a mean 0.10 to 0.42 of their probability on key 0 on 0.8B and 0.13 to 0.61 on 9B, up to 0.80 for single heads. Block 6, a DeltaNet block, writes a vector at position 0 several times larger than at any other position (10.6 against at most 1.0 on 0.8B, 18 to 30 against at most 6.4 on 9B). Prepending <|endoftext|> is not neutral: on 0.8B it takes ' Paris' after "The Eiffel Tower is in the city of" from 0.592 to 0.019, and ' cold' after "The opposite of hot is" from 0.254 to 0.755.

The chat template, and thinking by size

The template writes no system turn of its own. With add_generation_prompt=True, the 0.8B and 2B templates (Base included) end the prompt at an empty <think>\n\n</think>\n\n unless enable_thinking=True is passed; the 4B, 9B (Base included), 27B and Qwen3.6-27B templates end it at <think>\n unless enable_thinking=False is passed. After <think>\n the next token is 'Thinking' (1.000 on 0.8B and 9B), the start of a reasoning trace, not the answer. Pass the flag either way rather than relying on the default:

text = model.tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
with model.trace(text):
    probs = model.next_token_probs.save()

After the closed think block the answer's first token differs by size: on 0.8B ' Paris' 0.784 and 'Paris' 0.174, on 9B 'Paris' 1.000, so a target should count both forms. 0.8B to 9B ship no generation_config.json, so generate stops only on the config's eos_token_id, <|endoftext|>: a templated reply on 0.8B runs on as ' Paris<|im_end|>\n<|endoftext|>'. 27B and Qwen3.6-27B stop on <|im_end|> or <|endoftext|>.

The readout is the plain projection

model.logits equals model.lm_head.output, and project_on_vocab on the last block's layer_output equals logits exactly. The final norm's gain is 1 + model.norm._module.weight (about 4.3 on average on 0.8B), as for input_layernorm, post_attention_layernorm, q_norm and k_norm; the DeltaNet's own output norm multiplies by its weight. On 0.8B, 2B and 4B the embedding and the unembedding are one matrix, so an edit to embed_tokens.weight edits lm_head; on 9B and 27B they are separate.

What loads as this family

Every checkpoint here is multimodal: its config is Qwen3_5ForConditionalGeneration (model_type qwen3_5) with a text_config of model_type qwen3_5_text, and nnterp picks the family from the text_config. With task="image-text-to-text", as this page builds them, the wrapper loads with its processor: the text model at model.language_model, the vision encoder at model.visual as model.vision. With the default task="text-generation" it builds Qwen3_5ForCausalLM from the model.language_model weights, without the vision encoder, and takes text only; the text model's names and values are the same under both. The multi-token-prediction block (mtp) the checkpoints ship is loaded by neither. The mixture-of-experts releases, Qwen3.5-35B-A3B, 122B-A10B, 397B-A17B and Qwen3.6-35B-A3B, are qwen3_5_moe_text.

Sparse autoencoders

Qwen-Scope's residual-stream SAEs cover Qwen3.5-2B-Base (Qwen/SAE-Res-Qwen3.5-2B-Base-W32K-L0_50 and -L0_100, 32768 features), Qwen3.5-9B-Base (Qwen/SAE-Res-Qwen3.5-9B-Base-W64K-L0_50 and -L0_100, 65536) and the post-trained Qwen3.5-27B (Qwen/SAE-Res-Qwen3.5-27B-W80K-L0_50 and -L0_100, 81920), one per block, TopK with k 50 or 100. Their hook point is resid_post, a forward hook on the decoder block's output, which is model.layers[i].layer_output, on DeltaNet and attention blocks alike. Each layer<i>.sae.pt holds W_enc [d_sae, hidden_size], b_enc, W_dec [hidden_size, d_sae] and b_dec in float32.

Vision notes

· Qwen ViT · Qwen3.5

The Qwen ViT vision encoder

Every image in one row

The vision encoder runs on [patches, vision_hidden_size], the patches of every image in the invoke concatenated. Its stream values (patch_embeddings, every block's layer_output, attention_output and mlp_output, tower_output) are served [1, patches, vision_hidden_size], a view with a leading images axis of 1, so in-place edits land; vision.layers[i].input and .output stay the native [patches, vision_hidden_size], and the block identity holds across the two by broadcasting. The processor's image_grid_thw ([t, h, w] per image, in patches; an image is two identical frames, so t is 1) splits the row: image j has t * h * w patches.

inputs = model.processor(text=prompt, images=[red, wide], return_tensors="pt")
sizes = inputs["image_grid_thw"].prod(-1).tolist()   # patches per image
with model.trace(prompt, images=[red, wide]):
    patches = model.vision.layers[1].layer_output.save()
per_image = patches[0].split(sizes)                  # one [t*h*w, vision_hidden_size] each

Inside an image the patches are in merge-block order: the spatial_merge_size × spatial_merge_size (2 × 2) block of neighbours the merger folds into one image token is four consecutive rows, blocks in raster order. Index 4 * b to 4 * b + 3 is block b, not a raster row.

The attention runs once per image, so there is no pattern

The attention computes queries, keys and values for the whole row, then calls the attention function once per image (once per window on Qwen2.5-VL's windowed blocks) and concatenates the outputs. attention_queries, attention_keys and attention_values are served whole, before the split, [1, heads, patches, head_dim], the queries and keys after the rotary embedding; attention_head_outputs is the calls' outputs concatenated back, [1, patches, heads, head_dim] (under any implementation but flash, which runs one call over cu_seqlens). attention_scores and attention_probabilities are Unavailable under every implementation, eager included: "no one tensor is the block's pattern". The split points are the attention's cu_seqlens argument, self_attn.inputs[1]["cu_seqlens"], so a pattern per image is a softmax over each slice:

attn = model.vision.layers[1].self_attn
with model.trace(prompt, images=[red, wide]):
    cu = attn.inputs[1]["cu_seqlens"].save()        # read first: the module's input
    q = attn.attention_queries.save()
    k = attn.attention_keys.save()

patterns = []
for a, b in zip(cu[:-1].tolist(), cu[1:].tolist()):
    scores = q[0, :, a:b] @ k[0, :, a:b].transpose(-1, -2) * model.vision.head_dim ** -0.5
    patterns.append(scores.softmax(-1))             # [heads, patches_j, patches_j]

Each pattern times that image's values reproduces attention_head_outputs exactly (checked on Qwen3.5's pinned tiny checkpoint).

The merger is inside the vision encoder

model.projector is the vision encoder's own merger: it norms each patch, concatenates each 2 × 2 block into one 4 * vision_hidden_size vector and runs a two-layer MLP to the text model's width, so image tokens per image are t * h * w / 4. It reads the last block's stream with no norm between (projector.input == tower_output[0], the native [patches, vision_hidden_size]). Because the merger runs inside the vision encoder's forward, vision.tower_output (the encoder's output) is reached after it: read model.projector.input and .output before vision.tower_output in a trace.

Sizes, and no image_size

model.vision reads its sizes off the vision config: vision_hidden_size is the encoder's width (embed_dim on Qwen2-VL, whose config's hidden_size is the merger's output width), num_heads, intermediate_size (embed_dim * mlp_ratio on Qwen2-VL), patch_size, and two of its own, spatial_merge_size (2 on every line) and window_size (Qwen2.5-VL's window side in pixels, 112; None on the others). image_size raises Unavailable: the processor cuts each image into its own grid.

The variants

line block norms, merger norm MLP beyond the shared encoder
Qwen2-VL LayerNorm fc1, QuickGELU, fc2 none
Qwen2.5-VL RMSNorm SwiGLU, biased windows of 112 pixels except on fullatt_block_indexes
Qwen3-VL, -MoE LayerNorm linear_fc1, GELU (tanh), linear_fc2 learned pos_embed; DeepStack taps
Qwen3.5, -MoE LayerNorm linear_fc1, GELU (tanh), linear_fc2 learned pos_embed

On Qwen2.5-VL the encoder permutes the patches into windows at entry, so its blocks' values, tower_output and projector.output are in window order, and it restores the merge-block order after the merger: vision.image_features is in scatter order. On Qwen3-VL and Qwen3.5 vision.layers[0].input is patch_embeddings plus the resampled pos_embed. On Qwen2-VL vision.layers[0].input is patch_embeddings; on Qwen2.5-VL it is patch_embeddings permuted into window order; on both, positions enter only through the rotary embedding.

Qwen3.5

The merger folds four patches into one image token

model.projector is model.visual.merger, the vision encoder's last module: norm (a LayerNorm over vision_hidden_size) on each patch, then each 2 × 2 block of consecutive patches concatenated into one 4 * vision_hidden_size vector, linear_fc1, GELU and linear_fc2 to the text model's width. An image of t * h * w patches (its image_grid_thw row) is t * h * w / 4 image tokens, and vision.image_features is model.projector.output as it is, [image_tokens, hidden_size]. On 0.8B the merger maps 4 × 768 = 3072 to 1024; a patch is 16 pixels, so an image token covers 32 × 32 pixels.

with model.trace(prompt, images=[red, wide]):
    mask = model.vision.image_token_mask.save()
    fed = model.projector.input.save()          # [patches, vision_hidden_size]
    features = model.vision.image_features.save()
    first = model.layers[0].input.save()

fed.shape[0] // 4 == mask.sum()                 # True: one image token per 2 x 2 block
torch.equal(first[mask], features)              # True

No DeepStack

vision_config.deepstack_visual_indexes is empty on every checkpoint: the image reaches the text model only through vision.image_features; the text blocks have no deepstack_output. The vision encoder is otherwise Qwen3-VL's, with its learned pos_embed: 12 blocks of width 768 on 0.8B, 24 of width 1024 on 2B and 4B, 27 of width 1152 on 9B and 27B.

From the family module

04 · nnterp/families/qwen3_5_text.py

Qwen3.5 / 3.6 text (Qwen3_5ForCausalLM, model_type qwen3_5_text).

Llama's tree with a hybrid block: three blocks in four carry a gated DeltaNet mixer, linear_attn, the fourth ordinary attention, self_attn (config.layer_types). Each block has one or the other, never both, so on a linear block every self_attn value is reported missing and the linear values live at layers[i].linear_attn (see nnterp.LinearAttention). Every released checkpoint is a qwen3_5 wrapper (Qwen3_5ForConditionalGeneration): task="text-generation" (the default) builds Qwen3_5ForCausalLM out of it, and task="image-text-to-text" the wrapper, whose text stack sits at model.language_model; RENAME carries both spellings.

On the wrapper the Qwen ViT model.visual is vision (a QwenVision: packed, [1, patches, vision_hidden_size], with a learned pos_embed added after patch_embed and no deepstack), its blocks vision.layers; its merger is projector.