nnterp · encyclopedia qwen3_moe · nnterp.families.qwen3_moe

family qwen3_moe

Qwen3-MoE

Qwen3's attention, an RMSNorm on each query and key head, with a mixture of experts on every block in place of the MLP: 8 experts per token, their softmax weights renormalized to sum to one, and no shared expert.

Query/key normsMixture of expertsGrouped-query attention

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card.

embed_tokenstoken_embeddings
→
layers× 48
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Qwen3MoeForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] Qwen3MoeDecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn Qwen3MoeAttentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp Qwen3MoeSparseMoeBlocklayoutwhat it isread at
⚠︎mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
⚠︎mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output
router_logits RouterLogits [batch seq num_experts] The router's logits, one per expert, before the scoring

inside router's forward: the output of F_linear_0
expert_weights ExpertWeights [batch seq top_k] The weight each selected expert's output is scaled by

the child module experts's inputs, element 2
expert_indices ExpertIndices [batch seq top_k] The expert each routing slot sends the token to

the child module experts's inputs, element 1
expert_outputs ExpertOutputs [batch seq top_k hidden_size] Each routing slot's weighted expert output

inside experts's forward: the output of weighted_out_view_0 inside experts_forward_1
routed_output Residual [batch seq hidden_size] The routed experts' combined output

the child module experts's output

Printout

What print(model) shows on Qwen/Qwen3-30B-A3B: the native modules, standard names and every standard value with its layout.

Qwen3MoeForCausalLM(
  (lm_head): Linear(in_features=2048, out_features=151936, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(151936, 2048)
  (layers): ModuleList(
    (0-47): 48 x Qwen3MoeDecoderLayer(
      (self_attn): Qwen3MoeAttention(
        (q_proj): Linear(in_features=2048, out_features=4096, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (q_norm): Qwen3MoeRMSNorm((128,), eps=1e-06)
        (k_norm): Qwen3MoeRMSNorm((128,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3MoeSparseMoeBlock(
        (experts): Qwen3MoeExperts(
          (act_fn): SiLUActivation()
        )
        (router/gate): Qwen3MoeTopKRouter()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3MoeRMSNorm((2048,), eps=1e-06)
      (post_attention_layernorm): Qwen3MoeRMSNorm((2048,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): Qwen3MoeRMSNorm((2048,), eps=1e-06)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers48
model.hidden_size2048
model.num_heads32
model.num_kv_heads4
model.head_dim128
model.qk_head_dim128
model.vocab_size151936
model.intermediate_size6144
architecturesQwen3MoeForCausalLM
num_hidden_layers48
hidden_size2048
intermediate_size6144
num_attention_heads32
num_key_value_heads4
head_dim128
vocab_size151936
max_position_embeddings40960
hidden_actsilu
rms_norm_eps1e-06
tie_word_embeddingsFalse
dtypebfloat16
moe_intermediate_size768
num_experts_per_tok8
norm_topk_probTrue
num_local_experts128
attention_biasFalse
use_sliding_windowFalse
max_window_layers48
decoder_sparse_step1

The Notes

03 · what to know before you run

The block, in order

h   = x + self_attn(input_layernorm(x))          # q_norm, k_norm per head, then rope
out = h + mlp(post_attention_layernorm(h))       # 8 of 128 experts per token

Qwen3's pre-norm block with Llama's names, and a mixture where Qwen3 has its MLP. No norm follows either sublayer: attention_output is o_proj's output and mlp_output the mixture's, which has no shared expert, so mlp_output is routed_output. The identity is the plain sum:

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    routed = model.layers[1].mlp.routed_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(routed, mlp)
torch.testing.assert_close(x + attn + mlp, out)

Every block is a mixture

A block's mlp is a mixture where its index is not in mlp_only_layers and (i + 1) % decoder_sparse_step == 0. Every checkpoint here sets mlp_only_layers to [] and decoder_sparse_step to 1, so all 48 blocks of 30B-A3B (94 of 235B-A22B, 62 of Coder-480B-A35B) are mixtures. The config's intermediate_size (6144 on 30B-A3B, 12288 on 235B-A22B) is the width of a dense MLP no block has; the experts are moe_intermediate_size wide (768, 1536, 2560 on Coder-480B-A35B), and model.layers[i].mlp.intermediate_size reads that.

The router renormalizes its top 8

The router (gate, aliased router) gives one logit per expert. It takes a softmax over all of them in float32, keeps the 8 largest, and, with norm_topk_prob true on every checkpoint, divides them by their sum before casting back to the model's dtype, so a token's expert_weights sum to one (to bfloat16's rounding in a bfloat16 load). expert_indices are the softmax's top 8, slot 0 the largest:

moe = model.layers[1].mlp
with model.trace(prompt):
    logits = moe.router_logits.save()       # [batch, seq, num_experts], before the softmax
    w = moe.expert_weights.save()           # [batch, seq, top_k]
    idx = moe.expert_indices.save()

top = logits.float().softmax(-1).topk(moe.top_k, dim=-1)
torch.testing.assert_close((top.values / top.values.sum(-1, keepdim=True)).to(w.dtype), w)
assert torch.equal(top.indices, idx)

moe.SCORING is "softmax". 30B-A3B and 235B-A22B have 128 experts and Coder-480B-A35B has 160, all with 8 per token.

Ablating an expert

Zeroing expert e's weight where expert_indices == e removes that slot's term; the token's other weights are not renormalized again, so they sum to less than one. mlp_output changes on exactly the tokens that chose e, and an expert no token chose has no effect:

moe, e = model.layers[1].mlp, 3
with model.trace(prompt):
    idx = moe.expert_indices.save()
    clean = moe.mlp_output.save()
with model.trace(prompt):
    moe.expert_weights = moe.expert_weights.masked_fill(moe.expert_indices == e, 0)
    ablated = moe.mlp_output.save()

changed = (ablated != clean).any(-1)                       # [batch, seq]
assert torch.equal(changed, (idx == e).any(-1))            # only the tokens that chose e

The check holds on the pinned tiny checkpoint for every expert, in float32 and in bfloat16.

expert_outputs needs the default experts implementation

The experts are one module, experts, with every expert's weights stacked. expert_outputs, each slot's weighted output, is read inside transformers' grouped_mm (the default) and batched_mm forwards; under experts_implementation="eager" support() reports it missing and names the kwarg. The other mixture values read the same under every implementation. expert_outputs sums over its slot axis to routed_output.

Queries and keys are normed per head

q_norm and k_norm are RMSNorms over one head's 128 dimensions, one gain for every head, applied before the rotary embedding; attention_queries and attention_keys are read after both. An edit to a head's slice of q_proj.output or k_proj.output passes through that norm, which undoes its scale; edit attention_queries or attention_keys to change a head's scores. No projection has a bias (attention_bias false).

head_dim is 128 on every checkpoint, so the attention is wider than the stream: 30B-A3B has 32 query heads over 4 key/value heads (4096 wide against hidden_size 2048), 235B-A22B 64 over 4 (8192 against 4096), Coder-480B-A35B 96 over 8 (12288 against 6144). An edit to key/value head j on 30B-A3B reaches query heads 8j to 8j + 7.

Loading

The attention interior needs attn_implementation="eager". Every checkpoint sets use_sliding_window false, so sliding_window is None and no block has a window; there is no softcap or sink. rope_theta is 10⁶ on 30B-A3B, 30B-A3B-Base and 235B-A22B, 5·10⁶ on 235B-A22B-Instruct-2507 and -Thinking-2507, and 10⁷ on the 30B-A3B 2507 pair and both Coder models.

The chat templates

The tokenizer prepends nothing (bos_token is None). On 30B-A3B and 235B-A22B (and the Base model, which ships the same template) add_generation_prompt=True ends the prompt at <|im_start|>assistant\n, where the model opens a <think> block; enable_thinking=False appends an empty <think>\n\n</think>\n\n. The Thinking-2507 templates end the prompt at <think>\n and ignore enable_thinking. The Instruct-2507 and Coder templates have no thinking block. No template writes a system turn of its own.

text = model.tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
with model.trace(text):
    probs = model.next_token_probs.save()

The readout

model.logits equals model.lm_head.output, and project_on_vocab on the last block's layer_output equals logits. lm_head and embed_tokens are separate weights on every checkpoint here.

From the family module

04 · nnterp/families/qwen3_moe.py

Qwen3-MoE (Qwen3MoeForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. The MLP is a sparse mixture of experts; head_dim is the config's.

A config with mlp_only_layers set gives those blocks a dense Qwen3MoeMLP that no envoy is keyed to: layers[i].mlp is a plain Envoy there and support() reports "no mlp module on this block". No released checkpoint sets it.