nnterp · encyclopedia qwen3_next · nnterp.families.qwen3_next

family qwen3_next

Qwen3-NextQwen3-Coder-Next

Llama's pre-norm block with a gated DeltaNet mixer on three blocks in four, output-gated attention on the fourth, and a mixture of 512 experts whose shared expert is scaled by a sigmoid gate per token; the mixer fuses its projections per key head.

HybridGated queryMixture of expertsQuery/key normsPartial rotary1 + weight norm gainGrouped-query attention

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card. The slider picks a block, each tick's color gives its block's shape, and the diagram redraws for the block picked.

embed_tokenstoken_embeddings
→
layers× 48
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + linear_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Qwen3NextForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] Qwen3NextDecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].linear_attn Qwen3NextGatedDeltaNetlayoutwhat it isread at
attention_output Residual [batch seq hidden_size] What the mixer adds to the residual stream

the module's own output
⚠︎state State [batch heads key_dim value_dim] The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')

inside the forward, at the operation that _token_state_op finds on each call
⚠︎states States [batch seq heads key_dim value_dim] The recurrent state after every token of this call; needs route_kernels(family, 'torch')

inside the forward, at the operation that _states finds on each call
attention_queries LinearQK [batch seq heads key_dim] The queries entering the delta rule

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 0
attention_keys LinearQK [batch seq heads key_dim] The keys entering the delta rule

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 1
attention_values LinearV [batch seq heads value_dim] The values entering the delta rule

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 2
decays Gates [batch seq heads] The per-token log decay of the recurrent state

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), argument g
betas Gates [batch seq heads] The per-token write strength into the state

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), argument beta
state_input State | None [batch heads key_dim value_dim] The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)

inside the forward: the arguments of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), argument initial_state
attention_head_outputs LinearV [batch seq heads value_dim] The per-head outputs before the gated norm and the output projection

inside the forward: the output of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 0
state_output State [batch heads key_dim value_dim] The recurrent state leaving the layer

inside the forward: the output of the kernel this call runs (a prompt's chunked one or a decode step's recurrent one), element 1
model.layers[i].self_attn Qwen3NextAttentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp Qwen3NextSparseMoeBlocklayoutwhat it isread at
⚠︎mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
⚠︎mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output
router_logits RouterLogits [batch seq num_experts] The router's logits, one per expert, before the scoring

inside router's forward: the output of F_linear_0
expert_weights ExpertWeights [batch seq top_k] The weight each selected expert's output is scaled by

the child module experts's inputs, element 2
expert_indices ExpertIndices [batch seq top_k] The expert each routing slot sends the token to

the child module experts's inputs, element 1
expert_outputs ExpertOutputs [batch seq top_k hidden_size] Each routing slot's weighted expert output

inside experts's forward: the output of weighted_out_view_0 inside experts_forward_1
routed_output Residual [batch seq hidden_size] The routed experts' combined output

the child module experts's output
shared_expert_output Residual [batch seq hidden_size] The shared expert's output

inside the forward: the output of shared_expert_output_1

Printout

What print(model) shows on Qwen/Qwen3-Next-80B-A3B-Instruct: the native modules, standard names and every standard value with its layout.

Qwen3NextForCausalLM(
  (lm_head): Linear(in_features=2048, out_features=151936, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(151936, 2048)
  (layers): ModuleList(
    (0-2): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (3): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (4-6): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (7): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (8-10): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (11): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (12-14): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (15): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (16-18): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (19): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (20-22): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (23): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (24-26): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (27): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (28-30): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (31): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (32-34): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (35): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (36-38): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (39): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (40-42): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (43): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (44-46): 3 x Qwen3NextDecoderLayer(
      (linear_attn): Qwen3NextGatedDeltaNet(
        (conv1d): Conv1d(8192, 8192, kernel_size=(4,), stride=(1,), padding=(3,), groups=8192, bias=False)
        (in_proj_qkvz): Linear(in_features=2048, out_features=12288, bias=False)
        (in_proj_ba): Linear(in_features=2048, out_features=64, bias=False)
        (norm): Qwen3NextRMSNormGated()
        (out_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (attention_queries) -> LinearQK [batch seq heads key_dim]: The queries entering the delta rule
        (attention_keys) -> LinearQK [batch seq heads key_dim]: The keys entering the delta rule
        (attention_values) -> LinearV [batch seq heads value_dim]: The values entering the delta rule
        (decays) -> Gates [batch seq heads]: The per-token log decay of the recurrent state
        (betas) -> Gates [batch seq heads]: The per-token write strength into the state
        (state_input) -> State | None [batch heads key_dim value_dim]: The recurrent state entering the layer, or None at the start of a prompt (a copy of the cache's buffer)
        (attention_head_outputs) -> LinearV [batch seq heads value_dim]: The per-head outputs before the gated norm and the output projection
        (state_output) -> State [batch heads key_dim value_dim]: The recurrent state leaving the layer
        (attention_output) -> Residual [batch seq hidden_size]: What the mixer adds to the residual stream
        (state) -> State [batch heads key_dim value_dim]: The recurrent state after one token of the prompt; iterate it with tracer.iter; needs route_kernels(family, 'torch')
        (states) -> States [batch seq heads key_dim value_dim]: The recurrent state after every token of this call; needs route_kernels(family, 'torch')
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (47): Qwen3NextDecoderLayer(
      (self_attn): Qwen3NextAttention(
        (q_proj): Linear(in_features=2048, out_features=8192, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (k_norm): Qwen3NextRMSNorm((256,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3NextSparseMoeBlock(
        (router/gate): Qwen3NextTopKRouter()
        (experts): Qwen3NextExperts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_expert): Qwen3NextMLP(
          (gate_proj): Linear(in_features=2048, out_features=512, bias=False)
          (up_proj): Linear(in_features=2048, out_features=512, bias=False)
          (down_proj): Linear(in_features=512, out_features=2048, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (shared_expert_gate): Linear(in_features=2048, out_features=1, bias=False)
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3NextRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): Qwen3NextRMSNorm((2048,), eps=1e-06)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers48
model.hidden_size2048
model.num_heads16
model.num_kv_heads2
model.head_dim256
model.qk_head_dim256
model.vocab_size151936
model.intermediate_size5120
architecturesQwen3NextForCausalLM
num_hidden_layers48
hidden_size2048
intermediate_size5120
num_attention_heads16
num_key_value_heads2
head_dim256
vocab_size151936
max_position_embeddings262144
hidden_actsilu
rms_norm_eps1e-06
tie_word_embeddingsFalse
dtypebfloat16
linear_conv_kernel_dim4
moe_intermediate_size512
num_experts512
num_experts_per_tok10
norm_topk_probTrue
linear_num_key_heads16
linear_num_value_heads32
linear_key_head_dim128
linear_value_head_dim128
attention_biasFalse
use_sliding_windowFalse
decoder_sparse_step1
shared_expert_intermediate_size512

The Notes

03 · what to know before you run

The block, in order

h   = x + mixer(input_layernorm(x))          # linear_attn (gated DeltaNet) or self_attn
out = h + mlp(post_attention_layernorm(h))   # routed experts + gate * shared expert

config.layer_types is full_attention on every fourth block (full_attention_interval 4, blocks 3, 7, ..., 47) and linear_attention on the others: 36 DeltaNet and 12 attention blocks of 48 on every checkpoint. A block has linear_attn or self_attn, never both. Every block's mlp is the mixture (decoder_sparse_step 1, mlp_only_layers empty); the config's intermediate_size (5120) is the width of a dense MLP no block has. Pick blocks outside the trace:

types = model.config.layer_types
delta = [i for i, t in enumerate(types) if t == "linear_attention"]
full = [i for i, t in enumerate(types) if t == "full_attention"]    # 3, 7, ..., 47

The contributions are the sublayers' outputs

The block adds the mixer's output and the mixture's output to the stream and returns a tensor, so the identity is the plain sum, exact in float32 on both kinds of block; the mixture's output is its routed sum plus its gated shared expert:

with model.trace(prompt):
    x = model.layers[1].input.save()
    mix = model.layers[1].linear_attn.attention_output.save()
    routed = model.layers[1].mlp.routed_output.save()
    shared = model.layers[1].mlp.shared_expert_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.equal(routed + shared, mlp)               # True
torch.testing.assert_close(x + mix + mlp, out)

On an attention block the middle term is model.layers[3].self_attn.attention_output.

The mixture: 10 of 512 experts, renormalized, and a gated shared expert

The router, gate aliased router, gives one logit per expert; the scoring is a softmax, the 10 largest probabilities are kept, and norm_topk_prob (true on every checkpoint) divides them by their sum, so each token's expert_weights sum to one. The experts are SwiGLUs 512 wide (moe_intermediate_size); expert_outputs is each slot's weighted output, [batch, seq, 10, hidden_size], and sums to routed_output:

moe = model.layers[1].mlp
with model.trace(prompt):
    logits = moe.router_logits.save()
    weights = moe.expert_weights.save()

top = logits.softmax(-1).topk(moe.top_k, dim=-1)
torch.testing.assert_close(weights, top.values / top.values.sum(-1, keepdim=True))

Beside them every token runs through one shared expert, a SwiGLU 512 wide (shared_expert_intermediate_size), whose output the mixture multiplies by sigmoid(shared_expert_gate(x)), one scalar per token. shared_expert_output is that product, what the mixture adds; moe.shared_experts.output is the expert's output before the gate:

with model.trace(prompt):
    x = moe.input.save()
    ungated = moe.shared_experts.output.save()
with model.trace(prompt):
    shared = moe.shared_expert_output.save()

gate = torch.sigmoid(moe._module.shared_expert_gate(x))   # [batch, seq, 1]
torch.equal(gate * ungated, shared)                       # True

Zeroing one routed expert's weight leaves the other slots' weights as they are, so they sum to less than one:

with model.trace(prompt):
    moe.expert_weights = moe.expert_weights.masked_fill(moe.expert_indices == e, 0)
    ablated = model.logits.save()

The shared expert runs before the router

The mixture computes the shared expert first, then the router and the experts, and multiplies by the gate last. In one trace moe.shared_experts.output comes before router_logits, while shared_expert_output comes after routed_output; router_logits before shared_experts.output raises OutOfOrderError.

Loading

attn_implementation="eager" is needed only for the six attention interior values on the attention blocks; there is no softcap or window. The DeltaNet values are read at the call to the delta-rule kernel and need no eager load. They need transformers' pure-torch delta rule, which is what runs when flash-linear-attention is not installed. With it installed, every linear_attn value but attention_output is unavailable until nnterp.route_kernels(model.family, "torch"). The same call is what makes state and states exist: it routes a prompt through the token-by-token kernel. states holds one float32 state per token, 32 × 128 × 128, 2 MiB per token per block. expert_outputs needs experts_implementation "grouped_mm" (the default) or "batched_mm".

The attention block gates its heads' output

q_proj is 2 × num_heads × head_dim wide (8192) and holds, for each head in turn, its 256 query dimensions followed by its 256 gate dimensions, so the first half of q_proj.output is not the queries. The heads' output is multiplied by the sigmoid of that gate before o_proj; attention_head_outputs is read before the gate and o_proj.input is the gated tensor:

attn = model.layers[3].self_attn
d = model.head_dim
with model.trace(prompt):
    q_and_gate = attn.q_proj.output.save()            # heads * 2 * head_dim wide
query, gate = q_and_gate.unflatten(-1, (-1, 2 * d)).chunk(2, dim=-1)

with model.trace(prompt):
    heads = attn.attention_head_outputs.save()        # before the gate
    gated = attn.o_proj.input.save()                  # after it

torch.testing.assert_close(gated, (heads * gate.sigmoid()).flatten(2))

q_norm and k_norm are RMSNorms over one head's 256 dimensions with gain 1 + weight, applied before the rotary. The rotary (rotate_half) turns only the first 64 dimensions of each query and key head (partial_rotary_factor 0.25; rope_theta 10⁷ on Qwen3-Next, 5·10⁶ on Qwen3-Coder-Next); attention_queries[..., 64:] equals q_norm.output[..., 64:] exactly. There are 16 query heads over 2 key/value heads, so an edit to key/value head j reaches query heads 8j to 8j + 7. The score scale is 256 ** -0.5.

The DeltaNet mixer fuses its projections per key head

q, k, v, z, b, a = regroup(in_proj_qkvz(x), in_proj_ba(x))   # per key head
q, k, v = split(silu(conv1d(cat(q, k, v))))                  # causal, width 4
beta = sigmoid(b)
g = -exp(A_log) * softplus(a + dt_bias)
y, state = delta_rule(l2norm(q) / sqrt(128), l2norm(k), v, g, beta)
out = out_proj(norm(y, gate=silu(z)))                        # gain: weight

in_proj_qkvz is laid out by key head: for each of the 16 key heads, its query (128), its key (128), then the values and output gates of the two value heads it serves (2 × 128 each), 768 columns per key head. in_proj_ba is laid out the same way, the two value heads' b then their a per key head. No slice of the raw outputs is therefore the queries or the betas; the module's own fix_query_key_value_ordering regroups them:

mix = model.layers[0].linear_attn
with model.trace(prompt):
    qkvz = mix.in_proj_qkvz.output.save()
    ba = mix.in_proj_ba.output.save()
with model.trace(prompt):
    betas = mix.betas.save()

q, k, v, z, b, a = mix._module.fix_query_key_value_ordering(qkvz, ba)
torch.testing.assert_close(b.sigmoid(), betas)

attention_queries, attention_keys and attention_values are the kernel's arguments: after the convolution and the SiLU, before the kernel's l2 norm and the queries' 1/sqrt(128) scale. The 16 key heads' queries and keys are repeated up to the 32 value heads (repeat_interleave) before the kernel, so they are served 32 wide, heads 2j and 2j + 1 being copies of key head j. decays is one log decay per head and token (float32, at most 0), betas the sigmoid write strength, and attention_head_outputs is y, before the gated norm and out_proj. The state after a prompt, state_output, is [batch, 32, 128, 128].

The prompt and the chat template

The tokenizer sets bos_token to None and prepends nothing; <|endoftext|> is 151643, <|im_start|> 151644, <|im_end|> 151645. With add_generation_prompt=True the Instruct and Coder templates end the prompt at <|im_start|>assistant\n; the Thinking template adds <think>\n, so the next token starts a reasoning trace, not the answer. generate stops on <|im_end|> or <|endoftext|> (generation_config.json).

What loads as this family

Qwen3-Next-80B-A3B (Instruct and Thinking) and Qwen3-Coder-Next (and its Base), each Qwen3NextForCausalLM, text only, with the same sizes. The multi-token-prediction block (mtp) the checkpoints ship is not loaded. Qwen3.5's mixture-of-experts checkpoints, the same hybrid under a vision-language wrapper with separate DeltaNet projections, are qwen3_5_moe_text.

From the family module

04 · nnterp/families/qwen3_next.py

Qwen3-Next (Qwen3NextForCausalLM).

Llama's tree with a hybrid block: three blocks in four carry a gated DeltaNet mixer, linear_attn, the fourth ordinary attention, self_attn (config.layer_types). Each block has one or the other, never both, so on a linear block every self_attn value is reported missing and the linear values live at layers[i].linear_attn (see nnterp.LinearAttention). The MLP is a mixture of experts (a dense MLP class exists for the shared expert).