nnterp · encyclopedia qwen3_5_moe_text · nnterp.families.qwen3_5_moe_text

family qwen3_5_moe_text

Qwen3.5-MoEQwen3.6-MoE

Qwen3.5's hybrid block, a gated DeltaNet mixer on three blocks in four and output-gated attention on the fourth, with a mixture of experts on every block whose shared expert is scaled by a sigmoid gate per token.

HybridGated queryMixture of expertsQuery/key normsPartial rotary1 + weight norm gainMultimodal rotaryGrouped-query attentionVision-languagePacked vision encoderVariable resolutionPooled projector

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card. The slider picks a block, each tick's color gives its block's shape, and the diagram redraws for the block picked.

embed_tokenstoken_embeddings
→
layers× 40
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + linear_attn.attention_output + mlp.mlp_output == layer_output

The vision encoder

· Qwen ViT · Qwen3_5MoeVisionModel

The image's path into the text model, then one block of the vision encoder. Hover any part for its nnterp name; click to pin the card.

imagepixel_values
→
patch_embedpatch_embeddings
→
layers× 27 · tower_output
→
projectorthe merger's input: the last block's output
→
image_featuresat the scatter
→
layers[0].inputat image_token_mask
rows
One row, packed: every image's patches, [1, patches, vision_hidden_size], split per image by the processor's image_grid_thw. Each image's patches are in merge-block order (each 2 × 2 block the merger folds is consecutive), and on Qwen2.5-VL in window order inside the vision encoder.
positions
A 2D rotary embedding (row and column of the patch) turns the queries and keys inside each attention. Qwen3-VL and Qwen3.5 also add a learned pos_embed, resampled to each image's grid, after patch_embed.
masking
The attention runs once per image, over that image's patches, so no patch attends to another image's; on Qwen2.5-VL's windowed blocks once per window.
final norm
None after the last block: the merger norms each patch as it reads it, so there is no vision.norm.

identity vision.layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Qwen3_5MoeForConditionalGenerationlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] Qwen3_5MoeDecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].linear_attn Qwen3_5MoeGatedDeltaNetlayoutwhat it isread at
attention_output Residual [batch seq hidden_size] What the mixer adds to the residual stream

the module's own output
⚠︎state State [batch heads key_dim value_dim] The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')

inside the forward, at the operation that _token_state_op finds on each call
⚠︎states States [batch seq heads key_dim value_dim] The recurrent state after every token of this call; needs route_kernels(family, 'torch')

inside the forward, at the operation that _states finds on each call
attention_queries LinearQK [batch seq heads key_dim] The queries entering the delta rule

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 0
attention_keys LinearQK [batch seq heads key_dim] The keys entering the delta rule

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 1
attention_values LinearV [batch seq heads value_dim] The values entering the delta rule

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 2
decays Gates [batch seq heads] The per-token log decay of the recurrent state

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), argument g
betas Gates [batch seq heads] The per-token write strength into the state

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), argument beta
state_input State | None [batch heads key_dim value_dim] The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), argument initial_state
attention_head_outputs LinearV [batch seq heads value_dim] The per-head outputs before the gated norm and the output projection

inside the forward: the output of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 0
state_output State [batch heads key_dim value_dim] The recurrent state leaving the layer

inside the forward: the output of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 1
model.layers[i].self_attn Qwen3_5MoeAttentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp Qwen3_5MoeSparseMoeBlocklayoutwhat it isread at
⚠︎mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
⚠︎mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output
router_logits RouterLogits [batch seq num_experts] The router's logits, one per expert, before the scoring

inside router's forward: the output of F_linear_0
expert_weights ExpertWeights [batch seq top_k] The weight each selected expert's output is scaled by

the child module experts's inputs, element 2
expert_indices ExpertIndices [batch seq top_k] The expert each routing slot sends the token to

the child module experts's inputs, element 1
expert_outputs ExpertOutputs [batch seq top_k hidden_size] Each routing slot's weighted expert output

inside experts's forward: the output of weighted_out_view_0 inside experts_forward_1
routed_output Residual [batch seq hidden_size] The routed experts' combined output

the child module experts's output
shared_expert_output Residual [batch seq hidden_size] The shared expert's output

inside the forward: the output of shared_expert_output_1

the vision encoder · Qwen ViT

model.vision Qwen3_5MoeVisionModellayoutwhat it isread at
image_token_mask ImageTokenMask [batch seq] Which positions hold image tokens: input_ids == the config's image_token_id; read-only

the model's own inputs, at the root
patch_embeddings Patches [images patches vision_hidden_size] The patch embedding's output, one row per patch

the child module patch_embed's output
tower_output Patches [images patches vision_hidden_size] The last block's stream after the tower's final norm, before any pooling or adapter

the module's own output
image_features ImageFeatures [image_tokens hidden_size] The image features the text model receives at the image tokens, flat over them

inside model.model's forward: the arguments of inputs_embeds_masked_scatter_0, element 1
model.vision.layers[i] Qwen3_5MoeVisionBlocklayoutwhat it isread at
layer_input Patches [images patches vision_hidden_size] The tower's stream entering the block

the module's own input
attention_norm_output Patches [images patches vision_hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Patches [images patches vision_hidden_size] The tower's stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Patches [images patches vision_hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Patches [images patches vision_hidden_size] What the MLP is called with

the child module mlp's input
layer_output Patches [images patches vision_hidden_size] The tower's stream leaving the block

the module's own output
model.vision.layers[i].self_attn Qwen3_5MoeVisionAttentionlayoutwhat it isread at
attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the output of unsqueeze_0
attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the output of unsqueeze_1
attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the output of unsqueeze_2
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Patches [images patches vision_hidden_size] What the attention adds to the tower's stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of torch_cat_0
model.vision.layers[i].mlp Qwen3_5MoeVisionMLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives

the child module down_proj's input
mlp_output Patches [images patches vision_hidden_size] What the MLP adds to the tower's stream

the module's own output

Printout

What print(model) shows on Qwen/Qwen3.5-35B-A3B-Base: the native modules, standard names and every standard value with its layout.

Qwen3_5MoeForConditionalGeneration(
  (lm_head): Linear(in_features=2048, out_features=248320, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(248320, 2048)
  (layers): ModuleList(
    (0-2): 3 x Qwen3_5MoeDecoderLayer(
      (linear_attn): Qwen3_5MoeGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (norm): Qwen3_5MoeRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (in_proj_qkv): Linear(in_features=2048, out_features=8192, bias=False)
        (in_proj_z): Linear(in_features=2048, out_features=4096, bias=False)
        (in_proj_b): Linear(in_features=2048, out_features=32, bias=False)
        (in_proj_a): Linear(in_features=2048, out_features=32, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (3): Qwen3_5MoeDecoderLayer(
      (self_attn): Qwen3_5MoeAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (4-6): 3 x Qwen3_5MoeDecoderLayer(
      (linear_attn): Qwen3_5MoeGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (norm): Qwen3_5MoeRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (in_proj_qkv): Linear(in_features=2048, out_features=8192, bias=False)
        (in_proj_z): Linear(in_features=2048, out_features=4096, bias=False)
        (in_proj_b): Linear(in_features=2048, out_features=32, bias=False)
        (in_proj_a): Linear(in_features=2048, out_features=32, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (7): Qwen3_5MoeDecoderLayer(
      (self_attn): Qwen3_5MoeAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (8-10): 3 x Qwen3_5MoeDecoderLayer(
      (linear_attn): Qwen3_5MoeGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (norm): Qwen3_5MoeRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (in_proj_qkv): Linear(in_features=2048, out_features=8192, bias=False)
        (in_proj_z): Linear(in_features=2048, out_features=4096, bias=False)
        (in_proj_b): Linear(in_features=2048, out_features=32, bias=False)
        (in_proj_a): Linear(in_features=2048, out_features=32, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (11): Qwen3_5MoeDecoderLayer(
      (self_attn): Qwen3_5MoeAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (12-14): 3 x Qwen3_5MoeDecoderLayer(
      (linear_attn): Qwen3_5MoeGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (norm): Qwen3_5MoeRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (in_proj_qkv): Linear(in_features=2048, out_features=8192, bias=False)
        (in_proj_z): Linear(in_features=2048, out_features=4096, bias=False)
        (in_proj_b): Linear(in_features=2048, out_features=32, bias=False)
        (in_proj_a): Linear(in_features=2048, out_features=32, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (15): Qwen3_5MoeDecoderLayer(
      (self_attn): Qwen3_5MoeAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (16-18): 3 x Qwen3_5MoeDecoderLayer(
      (linear_attn): Qwen3_5MoeGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (norm): Qwen3_5MoeRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (in_proj_qkv): Linear(in_features=2048, out_features=8192, bias=False)
        (in_proj_z): Linear(in_features=2048, out_features=4096, bias=False)
        (in_proj_b): Linear(in_features=2048, out_features=32, bias=False)
        (in_proj_a): Linear(in_features=2048, out_features=32, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (19): Qwen3_5MoeDecoderLayer(
      (self_attn): Qwen3_5MoeAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (20-22): 3 x Qwen3_5MoeDecoderLayer(
      (linear_attn): Qwen3_5MoeGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (norm): Qwen3_5MoeRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (in_proj_qkv): Linear(in_features=2048, out_features=8192, bias=False)
        (in_proj_z): Linear(in_features=2048, out_features=4096, bias=False)
        (in_proj_b): Linear(in_features=2048, out_features=32, bias=False)
        (in_proj_a): Linear(in_features=2048, out_features=32, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (23): Qwen3_5MoeDecoderLayer(
      (self_attn): Qwen3_5MoeAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (24-26): 3 x Qwen3_5MoeDecoderLayer(
      (linear_attn): Qwen3_5MoeGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (norm): Qwen3_5MoeRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (in_proj_qkv): Linear(in_features=2048, out_features=8192, bias=False)
        (in_proj_z): Linear(in_features=2048, out_features=4096, bias=False)
        (in_proj_b): Linear(in_features=2048, out_features=32, bias=False)
        (in_proj_a): Linear(in_features=2048, out_features=32, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (27): Qwen3_5MoeDecoderLayer(
      (self_attn): Qwen3_5MoeAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (28-30): 3 x Qwen3_5MoeDecoderLayer(
      (linear_attn): Qwen3_5MoeGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (norm): Qwen3_5MoeRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (in_proj_qkv): Linear(in_features=2048, out_features=8192, bias=False)
        (in_proj_z): Linear(in_features=2048, out_features=4096, bias=False)
        (in_proj_b): Linear(in_features=2048, out_features=32, bias=False)
        (in_proj_a): Linear(in_features=2048, out_features=32, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (31): Qwen3_5MoeDecoderLayer(
      (self_attn): Qwen3_5MoeAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (32-34): 3 x Qwen3_5MoeDecoderLayer(
      (linear_attn): Qwen3_5MoeGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (norm): Qwen3_5MoeRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (in_proj_qkv): Linear(in_features=2048, out_features=8192, bias=False)
        (in_proj_z): Linear(in_features=2048, out_features=4096, bias=False)
        (in_proj_b): Linear(in_features=2048, out_features=32, bias=False)
        (in_proj_a): Linear(in_features=2048, out_features=32, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (35): Qwen3_5MoeDecoderLayer(
      (self_attn): Qwen3_5MoeAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (36-38): 3 x Qwen3_5MoeDecoderLayer(
      (linear_attn): Qwen3_5MoeGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (norm): Qwen3_5MoeRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (in_proj_qkv): Linear(in_features=2048, out_features=8192, bias=False)
        (in_proj_z): Linear(in_features=2048, out_features=4096, bias=False)
        (in_proj_b): Linear(in_features=2048, out_features=32, bias=False)
        (in_proj_a): Linear(in_features=2048, out_features=32, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (39): Qwen3_5MoeDecoderLayer(
      (self_attn): Qwen3_5MoeAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3_5MoeRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3_5MoeSparseMoeBlock(
        (router/gate): Qwen3_5MoeTopKRouter()
        (experts): Qwen3_5MoeExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3_5MoeMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): Qwen3_5MoeRMSNorm((2048,), eps=1e-06)
  (vision): Qwen3_5MoeVisionModel(
    (patch_embed): Qwen3_5MoeVisionPatchEmbed(
      (proj): Conv3d(3, 1152, kernel_size=(2, 16, 16), stride=(2, 16, 16))
    )
    (pos_embed): Embedding(2304, 1152)
    (rotary_pos_emb): Qwen3_5MoeVisionRotaryEmbedding()
    (layers/blocks): ModuleList(
      (0-26): 27 x Qwen3_5MoeVisionBlock(
        (input_layernorm/norm1): LayerNorm((1152,), eps=1e-06, elementwise_affine=True, bias=True)
        (post_attention_layernorm/norm2): LayerNorm((1152,), eps=1e-06, elementwise_affine=True, bias=True)
        (self_attn/attn): Qwen3_5MoeVisionAttention(
          (qkv): Linear(in_features=1152, out_features=3456, bias=True)
          (o_proj/proj): Linear(in_features=1152, out_features=1152, bias=True)
          (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
          (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
          (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
          (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
          (attention_output) -> Patches [images patches vision_hidden_size]: What the attention adds to the tower's stream
          (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
          (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
          (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        )
        (mlp): Qwen3_5MoeVisionMLP(
          (linear_fc1): Linear(in_features=1152, out_features=4304, bias=True)
          (down_proj/linear_fc2): Linear(in_features=4304, out_features=1152, bias=True)
          (act_fn): GELUTanh()
          (mlp_output) -> Patches [images patches vision_hidden_size]: What the MLP adds to the tower's stream
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives
        )
        (layer_input) -> Patches [images patches vision_hidden_size]: The tower's stream entering the block
        (attention_norm_output) -> Patches [images patches vision_hidden_size]: The normed stream entering the attention
        (layer_mid) -> Patches [images patches vision_hidden_size]: The tower's stream after the attention add
        (mlp_norm_output) -> Patches [images patches vision_hidden_size]: The normed stream entering the MLP
        (mlp_input) -> Patches [images patches vision_hidden_size]: What the MLP is called with
        (layer_output) -> Patches [images patches vision_hidden_size]: The tower's stream leaving the block
      )
    )
    (merger): Qwen3_5MoeVisionPatchMerger(
      (norm): LayerNorm((1152,), eps=1e-06, elementwise_affine=True, bias=True)
      (linear_fc1): Linear(in_features=4608, out_features=4608, bias=True)
      (act_fn): GELU(approximate='none')
      (down_proj/linear_fc2): Linear(in_features=4608, out_features=2048, bias=True)
    )
    (image_token_mask) -> ImageTokenMask [batch seq]: Which positions hold image tokens: input_ids == the config's image_token_id; read-only
    (patch_embeddings) -> Patches [images patches vision_hidden_size]: The patch embedding's output, one row per patch
    (tower_output) -> Patches [images patches vision_hidden_size]: The last block's stream after the tower's final norm, before any pooling or adapter
    (image_features) -> ImageFeatures [image_tokens hidden_size]: The image features the text model receives at the image tokens, flat over them
  )
  (projector): Qwen3_5MoeVisionPatchMerger(
    (norm): LayerNorm((1152,), eps=1e-06, elementwise_affine=True, bias=True)
    (linear_fc1): Linear(in_features=4608, out_features=4608, bias=True)
    (act_fn): GELU(approximate='none')
    (down_proj/linear_fc2): Linear(in_features=4608, out_features=2048, bias=True)
  )
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers40
model.hidden_size2048
model.num_heads16
model.num_kv_heads2
model.head_dim256
model.qk_head_dim256
model.vocab_size248320
architecturesQwen3_5MoeForConditionalGeneration
num_hidden_layers40
hidden_size2048
num_attention_heads16
num_key_value_heads2
head_dim256
vocab_size248320
max_position_embeddings262144
hidden_actsilu
rms_norm_eps1e-06
tie_word_embeddingsFalse
linear_conv_kernel_dim4
moe_intermediate_size512
num_experts256
num_experts_per_tok8
linear_num_key_heads16
linear_num_value_heads32
linear_key_head_dim128
linear_value_head_dim128
full_attention_interval4
attention_biasFalse
shared_expert_intermediate_size512
Vision Config

Vision Config

model.vision.num_layers27
model.vision.vision_hidden_size1152
model.vision.num_heads16
model.vision.head_dim72
model.vision.intermediate_size4304
model.vision.patch_size16
model.vision.image_sizevaries
model.vision.spatial_merge_size2
model.vision.window_sizeNone

The Notes

03 · what to know before you run

The block, in order

h   = x + mixer(input_layernorm(x))          # linear_attn (gated DeltaNet) or self_attn
out = h + mlp(post_attention_layernorm(h))   # routed experts + gate * shared expert

config.layer_types is full_attention on every fourth block (full_attention_interval 4, blocks 3, 7, 11, ...) and linear_attention on the others: 30 DeltaNet and 10 attention blocks on 35B-A3B, 36 and 12 on 122B-A10B, 45 and 15 on 397B-A17B. A block has linear_attn or self_attn, never both; the last block is an attention block. Every block's mlp is the mixture: the block builds one whatever the config's mlp_only_layers says. Pick blocks outside the trace:

types = model.config.get_text_config().layer_types
delta = [i for i, t in enumerate(types) if t == "linear_attention"]
full = [i for i, t in enumerate(types) if t == "full_attention"]    # 3, 7, ..., 39 on 35B-A3B

The contributions are the sublayers' outputs

The block adds the mixer's output and the mixture's output to the stream and returns a tensor, so the identity is the plain sum, exact in float32 on both kinds of block; the mixture's output is its routed sum plus its gated shared expert:

with model.trace(prompt):
    x = model.layers[1].input.save()
    mix = model.layers[1].linear_attn.attention_output.save()
    routed = model.layers[1].mlp.routed_output.save()
    shared = model.layers[1].mlp.shared_expert_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.equal(routed + shared, mlp)               # True
torch.testing.assert_close(x + mix + mlp, out)

On an attention block the middle term is model.layers[3].self_attn.attention_output.

The mixture: a softmax router, renormalized, and a gated shared expert

The router, gate aliased router, gives one logit per expert; the scoring is a softmax, the top num_experts_per_tok probabilities are kept and always divided by their sum (the config has no norm_topk_prob switch), so each token's expert_weights sum to one. 35B-A3B and 122B-A10B route each token to 8 of 256 experts, 397B-A17B to 10 of 512. The experts are one fused module, experts, each a SwiGLU moe_intermediate_size wide (512 on 35B-A3B, 1024 on 122B-A10B and 397B-A17B); expert_outputs is each slot's weighted output, [batch, seq, top_k, hidden_size], and sums to routed_output:

moe = model.layers[1].mlp
with model.trace(prompt):
    logits = moe.router_logits.save()
    weights = moe.expert_weights.save()

top = logits.softmax(-1).topk(moe.top_k, dim=-1)
torch.testing.assert_close(weights, top.values / top.values.sum(-1, keepdim=True))

Beside them every token runs through one shared expert, a SwiGLU as wide as a routed one, whose output the mixture multiplies by sigmoid(shared_expert_gate(x)), one scalar per token. shared_expert_output is that product, what the mixture adds; moe.shared_experts.output is the expert's output before the gate:

with model.trace(prompt):
    x = moe.input.save()
    ungated = moe.shared_experts.output.save()
with model.trace(prompt):
    shared = moe.shared_expert_output.save()

gate = torch.sigmoid(moe._module.shared_expert_gate(x))   # [batch, seq, 1]
torch.equal(gate * ungated, shared)                       # True

Zeroing shared_expert_output removes the shared expert's whole contribution; zeroing one routed expert's weight leaves the other slots' weights as they are, so they sum to less than one:

with model.trace(prompt):
    moe.expert_weights = moe.expert_weights.masked_fill(moe.expert_indices == e, 0)
    ablated = model.logits.save()

The shared expert runs before the router

The mixture computes the shared expert first, then the router and the experts, and multiplies by the gate last. In one trace moe.shared_experts.output comes before router_logits, while shared_expert_output comes after routed_output; router_logits before shared_experts.output raises OutOfOrderError.

Loading

attn_implementation="eager" is needed only for the six attention interior values on the attention blocks; there is no softcap or window. The DeltaNet values are read at the call to the delta-rule kernel and need no eager load. They need transformers' pure-torch delta rule, which is what runs when flash-linear-attention is not installed. With it installed, every linear_attn value but attention_output is unavailable until nnterp.route_kernels(model.family, "torch"). The same call is what makes state and states exist: it routes a prompt through the token-by-token kernel. states holds one float32 state per token, num_v_heads × 128 × 128: 2 MiB per token per block on 35B-A3B, 4 MiB on 122B-A10B and 397B-A17B. expert_outputs needs experts_implementation "grouped_mm" (the default) or "batched_mm".

The attention block gates its heads' output

q_proj is 2 × num_heads × head_dim wide (8192 on 35B-A3B) and holds, for each head in turn, its 256 query dimensions followed by its 256 gate dimensions, so the first half of q_proj.output is not the queries. The heads' output is multiplied by the sigmoid of that gate before o_proj; attention_head_outputs is read before the gate and o_proj.input is the gated tensor:

attn = model.layers[3].self_attn
d = model.head_dim
with model.trace(prompt):
    q_and_gate = attn.q_proj.output.save()            # heads * 2 * head_dim wide
query, gate = q_and_gate.unflatten(-1, (-1, 2 * d)).chunk(2, dim=-1)

with model.trace(prompt):
    heads = attn.attention_head_outputs.save()        # before the gate
    gated = attn.o_proj.input.save()                  # after it

torch.testing.assert_close(gated, (heads * gate.sigmoid()).flatten(2))

q_norm and k_norm are RMSNorms over one head's 256 dimensions with gain 1 + weight, applied before the rotary. The rotary turns only the first 64 dimensions of each query and key head (partial_rotary_factor 0.25, rope_theta 10⁷); attention_queries[..., 64:] equals q_norm.output[..., 64:] exactly. The rotary is M-RoPE: mrope_section [11, 11, 10] gives the 32 rotary frequencies to time, height and width, interleaved, which matters on image tokens; on text the three positions are equal. 35B-A3B has 16 query heads over 2 key/value heads, 122B-A10B and 397B-A17B 32 over 2, so an edit to key/value head j reaches 8 query heads on 35B-A3B and 16 on the others. The score scale is 256 ** -0.5.

The DeltaNet mixer

q, k, v = split(silu(conv1d(in_proj_qkv(x))))     # causal, width 4
beta = sigmoid(in_proj_b(x))
g = -exp(A_log) * softplus(in_proj_a(x) + dt_bias)
y, state = delta_rule(l2norm(q) / sqrt(128), l2norm(k), v, g, beta)
out = out_proj(norm(y, gate=silu(in_proj_z(x))))  # gain: weight

attention_queries, attention_keys and attention_values are the kernel's arguments: after the convolution and the SiLU, before the kernel's l2 norm and the queries' 1/sqrt(128) scale. Each head's key and value are 128 wide. There are 16 key heads, and 32 value heads on 35B-A3B, 64 on 122B-A10B and 397B-A17B; the queries and keys are repeated up to the value heads (repeat_interleave) before the kernel, so they are served num_v_heads wide, heads 2j and 2j + 1 on 35B-A3B being copies of key head j. decays is one log decay per head and token (float32, at most 0), betas the sigmoid write strength, and attention_head_outputs is y, before the gated norm and out_proj. The state after a prompt, state_output, is [batch, num_v_heads, 128, 128].

The prompt and the chat template

The tokenizer sets bos_token to None and prepends nothing; <|endoftext|> is 248044, <|im_start|> 248045, <|im_end|> 248046. With add_generation_prompt=True every checkpoint's template, the Base one included, ends the prompt at <think>\n unless enable_thinking=False is passed; then it ends at an empty <think>\n\n</think>\n\n. Pass the flag either way. The post-trained checkpoints' generation_config.json stops on <|im_end|> or <|endoftext|>; 35B-A3B-Base ships none, so generate stops only on <|endoftext|>.

What loads as this family

Every checkpoint is multimodal: Qwen3_5MoeForConditionalGeneration (model_type qwen3_5_moe) with a text_config of model_type qwen3_5_moe_text, and nnterp picks the family from the text_config. With task="image-text-to-text", as this page builds them, the wrapper loads with its processor: the text model at model.language_model, the vision encoder at model.visual as model.vision. With the default task="text-generation" it builds Qwen3_5MoeForCausalLM from the model.language_model weights, without the vision encoder; the text model's names and values are the same under both. The multi-token-prediction block (mtp) the checkpoints ship is loaded by neither. The dense Qwen3.5 and Qwen3.6 checkpoints are qwen3_5_text; Qwen3-Next is qwen3_next.

Sparse autoencoders

Qwen-Scope's residual-stream SAEs cover Qwen3.5-35B-A3B-Base: Qwen/SAE-Res-Qwen3.5-35B-A3B-Base-W32K-L0_50 (32768 features, k 50) and Qwen/SAE-Res-Qwen3.5-35B-A3B-Base-W128K-L0_100 (131072 features, k 100), TopK, one per block, blocks 0 to 39. Their hook point is resid_post, the decoder block's output, which is model.layers[i].layer_output on DeltaNet and attention blocks alike. Each layer<i>.sae.pt holds W_enc [d_sae, 2048], b_enc, W_dec [2048, d_sae] and b_dec in float32.

Vision notes

· Qwen ViT · Qwen3.5-MoE

The Qwen ViT vision encoder

Every image in one row

The vision encoder runs on [patches, vision_hidden_size], the patches of every image in the invoke concatenated. Its stream values (patch_embeddings, every block's layer_output, attention_output and mlp_output, tower_output) are served [1, patches, vision_hidden_size], a view with a leading images axis of 1, so in-place edits land; vision.layers[i].input and .output stay the native [patches, vision_hidden_size], and the block identity holds across the two by broadcasting. The processor's image_grid_thw ([t, h, w] per image, in patches; an image is two identical frames, so t is 1) splits the row: image j has t * h * w patches.

inputs = model.processor(text=prompt, images=[red, wide], return_tensors="pt")
sizes = inputs["image_grid_thw"].prod(-1).tolist()   # patches per image
with model.trace(prompt, images=[red, wide]):
    patches = model.vision.layers[1].layer_output.save()
per_image = patches[0].split(sizes)                  # one [t*h*w, vision_hidden_size] each

Inside an image the patches are in merge-block order: the spatial_merge_size × spatial_merge_size (2 × 2) block of neighbours the merger folds into one image token is four consecutive rows, blocks in raster order. Index 4 * b to 4 * b + 3 is block b, not a raster row.

The attention runs once per image, so there is no pattern

The attention computes queries, keys and values for the whole row, then calls the attention function once per image (once per window on Qwen2.5-VL's windowed blocks) and concatenates the outputs. attention_queries, attention_keys and attention_values are served whole, before the split, [1, heads, patches, head_dim], the queries and keys after the rotary embedding; attention_head_outputs is the calls' outputs concatenated back, [1, patches, heads, head_dim] (under any implementation but flash, which runs one call over cu_seqlens). attention_scores and attention_probabilities are Unavailable under every implementation, eager included: "no one tensor is the block's pattern". The split points are the attention's cu_seqlens argument, self_attn.inputs[1]["cu_seqlens"], so a pattern per image is a softmax over each slice:

attn = model.vision.layers[1].self_attn
with model.trace(prompt, images=[red, wide]):
    cu = attn.inputs[1]["cu_seqlens"].save()        # read first: the module's input
    q = attn.attention_queries.save()
    k = attn.attention_keys.save()

patterns = []
for a, b in zip(cu[:-1].tolist(), cu[1:].tolist()):
    scores = q[0, :, a:b] @ k[0, :, a:b].transpose(-1, -2) * model.vision.head_dim ** -0.5
    patterns.append(scores.softmax(-1))             # [heads, patches_j, patches_j]

Each pattern times that image's values reproduces attention_head_outputs exactly (checked on Qwen3.5's pinned tiny checkpoint).

The merger is inside the vision encoder

model.projector is the vision encoder's own merger: it norms each patch, concatenates each 2 × 2 block into one 4 * vision_hidden_size vector and runs a two-layer MLP to the text model's width, so image tokens per image are t * h * w / 4. It reads the last block's stream with no norm between (projector.input == tower_output[0], the native [patches, vision_hidden_size]). Because the merger runs inside the vision encoder's forward, vision.tower_output (the encoder's output) is reached after it: read model.projector.input and .output before vision.tower_output in a trace.

Sizes, and no image_size

model.vision reads its sizes off the vision config: vision_hidden_size is the encoder's width (embed_dim on Qwen2-VL, whose config's hidden_size is the merger's output width), num_heads, intermediate_size (embed_dim * mlp_ratio on Qwen2-VL), patch_size, and two of its own, spatial_merge_size (2 on every line) and window_size (Qwen2.5-VL's window side in pixels, 112; None on the others). image_size raises Unavailable: the processor cuts each image into its own grid.

The variants

line block norms, merger norm MLP beyond the shared encoder
Qwen2-VL LayerNorm fc1, QuickGELU, fc2 none
Qwen2.5-VL RMSNorm SwiGLU, biased windows of 112 pixels except on fullatt_block_indexes
Qwen3-VL, -MoE LayerNorm linear_fc1, GELU (tanh), linear_fc2 learned pos_embed; DeepStack taps
Qwen3.5, -MoE LayerNorm linear_fc1, GELU (tanh), linear_fc2 learned pos_embed

On Qwen2.5-VL the encoder permutes the patches into windows at entry, so its blocks' values, tower_output and projector.output are in window order, and it restores the merge-block order after the merger: vision.image_features is in scatter order. On Qwen3-VL and Qwen3.5 vision.layers[0].input is patch_embeddings plus the resampled pos_embed. On Qwen2-VL vision.layers[0].input is patch_embeddings; on Qwen2.5-VL it is patch_embeddings permuted into window order; on both, positions enter only through the rotary embedding.

Qwen3.5-MoE

The merger folds four patches into one image token

model.projector is model.visual.merger, the vision encoder's last module: norm (a LayerNorm over vision_hidden_size) on each patch, then each 2 × 2 block of consecutive patches concatenated into one 4 * vision_hidden_size vector, linear_fc1, GELU and linear_fc2 to the text model's width. An image of t * h * w patches (its image_grid_thw row) is t * h * w / 4 image tokens, and vision.image_features is model.projector.output as it is, [image_tokens, hidden_size]. The merger maps 4 × 1152 = 4608 to 2048 on 35B-A3B, to 3072 on 122B-A10B and to 4096 on 397B-A17B; a patch is 16 pixels, so an image token covers 32 × 32 pixels.

with model.trace(prompt, images=[red, wide]):
    mask = model.vision.image_token_mask.save()
    fed = model.projector.input.save()          # [patches, vision_hidden_size]
    features = model.vision.image_features.save()
    first = model.layers[0].input.save()

fed.shape[0] // 4 == mask.sum()                 # True: one image token per 2 x 2 block
torch.equal(first[mask], features)              # True

No DeepStack

vision_config.deepstack_visual_indexes is empty on every checkpoint: the image reaches the text model only through vision.image_features; the text blocks have no deepstack_output. The vision encoder is the same on every checkpoint: 27 blocks of width 1152, 16 heads, with Qwen3-VL's learned pos_embed. The image token is <|image_pad|> (248056), and vision.image_token_mask marks it.

Read the vision values in the forward's order

The merger runs inside the vision encoder, before it returns. In one trace, read the last vision block's layer_output, then model.projector.input and .output, then vision.tower_output, then vision.image_features, then the text blocks. A read out of this order raises OutOfOrderError.

From the family module

04 · nnterp/families/qwen3_5_moe_text.py

Qwen3.5-MoE / 3.6-MoE text (Qwen3_5MoeForCausalLM, model_type qwen3_5_moe_text).

Llama's tree with a hybrid block: three blocks in four carry a gated DeltaNet mixer, linear_attn, the fourth ordinary attention, self_attn (config.layer_types). Each block has one or the other, never both, so on a linear block every self_attn value is reported missing and the linear values live at layers[i].linear_attn (see nnterp.LinearAttention). The MLP is a mixture of experts. Every released checkpoint is a qwen3_5_moe wrapper: task="text-generation" (the default) builds Qwen3_5MoeForCausalLM out of it, task="image-text-to-text" the wrapper, whose text stack sits at model.language_model; RENAME carries both.

On the wrapper the Qwen ViT model.visual is vision (a QwenVision: packed, [1, patches, vision_hidden_size], with a learned pos_embed added after patch_embed and no deepstack), its blocks vision.layers; its merger is projector.