nnterp · encyclopedia glm4_moe · nnterp.families.glm4_moe

family glm4_moe

GLM-4.5GLM-4.6GLM-4.7

Llama's pre-norm block with rotary on half of each head and, after the first dense blocks, a mixture whose sigmoid router picks 8 experts with a selection bias, beside one shared expert.

Mixture of expertsDense first blocksBiased q, k, vPartial rotaryGrouped-query attention

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card. The slider picks a block, each tick's color gives its block's shape, and the diagram redraws for the block picked.

embed_tokenstoken_embeddings
→
layers× 46
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Glm4MoeForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] Glm4MoeDecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn Glm4MoeAttentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp Glm4MoeMLP / Glm4MoeMoElayoutwhat it isread at
⚠︎mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
⚠︎mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output
router_logits RouterLogits [batch seq num_experts] The router's logits, one per expert, before the scoring

inside router's forward: the output of F_linear_0
expert_weights ExpertWeights [batch seq top_k] The weight each selected expert's output is scaled by

the child module experts's inputs, element 2
expert_indices ExpertIndices [batch seq top_k] The expert each routing slot sends the token to

the child module experts's inputs, element 1
expert_outputs ExpertOutputs [batch seq top_k hidden_size] Each routing slot's weighted expert output

inside experts's forward: the output of weighted_out_view_0 inside experts_forward_1
routed_output Residual [batch seq hidden_size] The routed experts' combined output

the child module experts's output
shared_expert_output Residual [batch seq hidden_size] The shared expert's output

the child module shared_experts's output

Printout

What print(model) shows on zai-org/GLM-4.5-Air: the native modules, standard names and every standard value with its layout.

Glm4MoeForCausalLM(
  (lm_head): Linear(in_features=4096, out_features=151552, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(151552, 4096, padding_idx=151329)
  (layers): ModuleList(
    (0): Glm4MoeDecoderLayer(
      (self_attn): Glm4MoeAttention(
        (q_proj): Linear(in_features=4096, out_features=12288, bias=True)
        (k_proj): Linear(in_features=4096, out_features=1024, bias=True)
        (v_proj): Linear(in_features=4096, out_features=1024, bias=True)
        (o_proj): Linear(in_features=12288, out_features=4096, bias=False)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Glm4MoeMLP(
        (gate_proj): Linear(in_features=4096, out_features=10944, bias=False)
        (up_proj): Linear(in_features=4096, out_features=10944, bias=False)
        (down_proj): Linear(in_features=10944, out_features=4096, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Glm4MoeRMSNorm((4096,), eps=1e-05)
      (post_attention_layernorm): Glm4MoeRMSNorm((4096,), eps=1e-05)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
    (1-45): 45 x Glm4MoeDecoderLayer(
      (self_attn): Glm4MoeAttention(
        (q_proj): Linear(in_features=4096, out_features=12288, bias=True)
        (k_proj): Linear(in_features=4096, out_features=1024, bias=True)
        (v_proj): Linear(in_features=4096, out_features=1024, bias=True)
        (o_proj): Linear(in_features=12288, out_features=4096, bias=False)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Glm4MoeMoE(
        (experts): Glm4MoeExperts(
          (act_fn): SiLUActivation()
        )
        (router/gate): Glm4MoeTopkRouter()
        (shared_experts): Glm4MoeMLP(
          (gate_proj): Linear(in_features=4096, out_features=1408, bias=False)
          (up_proj): Linear(in_features=4096, out_features=1408, bias=False)
          (down_proj): Linear(in_features=1408, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
          (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
          (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
          (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
        )
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Glm4MoeRMSNorm((4096,), eps=1e-05)
      (post_attention_layernorm): Glm4MoeRMSNorm((4096,), eps=1e-05)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): Glm4MoeRMSNorm((4096,), eps=1e-05)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers46
model.hidden_size4096
model.num_heads96
model.num_kv_heads8
model.head_dim128
model.qk_head_dim128
model.vocab_size151552
model.intermediate_size10944
architecturesGlm4MoeForCausalLM
num_hidden_layers46
hidden_size4096
intermediate_size10944
num_attention_heads96
num_key_value_heads8
head_dim128
vocab_size151552
max_position_embeddings131072
hidden_actsilu
rms_norm_eps1e-05
tie_word_embeddingsFalse
dtypebfloat16
first_k_dense_replace1
moe_intermediate_size1408
n_routed_experts128
num_experts_per_tok8
n_shared_experts1
routed_scaling_factor1.0
norm_topk_probTrue
attention_biasTrue
use_qk_normFalse
n_group1
topk_group1

The Notes

03 · what to know before you run

The block, in order

h   = x + self_attn(input_layernorm(x))        # q_norm, k_norm inside on 4.5, 4.6, 4.7
out = h + mlp(post_attention_layernorm(h))     # dense first, then routed + shared

Llama's pre-norm block and names. first_k_dense_replace blocks come first with a dense SwiGLU mlp (3 on GLM-4.5, 4.6 and 4.7, 1 on Air); every later block's mlp is a mixture. Nothing norms a sublayer's output, so the identity is the plain sum on both kinds of block, exact:

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

assert torch.equal(x + attn + mlp, out)

On the dense blocks support() reports every mixture value missing ("no router_logits value on this block's mlp").

The router: sigmoid scores, a selection bias, renormalized weights

router_logits are the router's logits, computed in float32. The router takes their sigmoid, adds router.e_score_correction_bias (a per-expert buffer) and keeps the 8 highest (n_group and topk_group are 1, so every expert competes). expert_weights are the chosen experts' sigmoids without the bias, renormalized to sum to one and multiplied by routed_scaling_factor, so a token's weights sum to 2.5 on GLM-4.5, 4.6 and 4.7 and to 1.0 on Air. They stay float32 under a bfloat16 load. The bias decides which experts are chosen but not how much each counts; the slots come in no promised order.

moe = model.layers[1].mlp
with model.trace(prompt):
    logits = moe.router_logits.save()       # [batch, seq, num_experts], before the sigmoid
    w = moe.expert_weights.save()           # [batch, seq, top_k]
    idx = moe.expert_indices.save()

s = logits.sigmoid()
chosen = (s + moe.router._module.e_score_correction_bias).topk(moe.top_k, dim=-1).indices
assert torch.equal(chosen.sort(-1).values, idx.sort(-1).values)
expected = s.gather(-1, idx)
expected = expected / expected.sum(-1, keepdim=True) * model.config.routed_scaling_factor
torch.testing.assert_close(expected, w)

The mixture's output is routed plus shared

GLM-4.5, 4.6 and 4.7 route over 160 experts, Air over 128, 8 per token on both, and each mixture has one shared expert (mlp.shared_experts, a dense SwiGLU moe_intermediate_size wide) that every token runs through. The shared expert runs after the routed experts, so read routed_output before shared_expert_output; their sum is mlp_output exactly:

moe = model.layers[1].mlp
with model.trace(prompt):
    routed = moe.routed_output.save()
    shared = moe.shared_expert_output.save()
    out = moe.mlp_output.save()

assert torch.equal(routed + shared, out)

Ablating an expert touches only the tokens routed to it

Zeroing expert e's weight where expert_indices == e removes that slot's term from routed_output and leaves the token's other weights as they were (they are not renormalized again). mlp_output changes on exactly the tokens that chose e, by that slot's expert_outputs; the shared expert is untouched.

moe, e = model.layers[1].mlp, 2
with model.trace(prompt):
    idx = moe.expert_indices.save()
    clean = moe.mlp_output.save()
with model.trace(prompt):
    moe.expert_weights = moe.expert_weights.masked_fill(moe.expert_indices == e, 0)
    ablated = moe.mlp_output.save()

changed = (ablated != clean).any(-1)                 # [batch, seq]
assert torch.equal(changed, (idx == e).any(-1))

Rotary turns half of each head, rotate_half style

The rotary covers the first partial_rotary_factor × head_dim dimensions of each query and key head, 64 of 128 (partial_rotary_factor 0.5), pairing i with i + 32 as Llama's rotate_half does; dimensions 64 to 127 carry no position. The base (rope_theta) is 10⁶. attention_queries and attention_keys are read after the rotary; recomputing them from q_proj.output and k_proj.output with rotate_half pairing matches exactly, and the unturned half equals the projection (after q_norm where there is one). GLM-4-9B and GLM-4-0414 (glm, glm4) pair adjacent dimensions instead.

Query and key norms on GLM-4.5, 4.6 and 4.7, not on Air

Where use_qk_norm is true (GLM-4.5, 4.5-Base, 4.6, 4.7) the attention has q_norm and k_norm, RMSNorms over one head's 128 dimensions with one gain shared by every head, applied to [batch, seq, num_heads, head_dim] before the transpose and the rotary. An edit to one head's slice of q_proj.output is renormalized and loses its size; edit attention_queries or attention_keys, which come after the norm. GLM-4.5-Air and Air-Base have no q_norm or k_norm.

with model.trace(prompt):
    normed = model.layers[1].self_attn.q_norm.output.save()   # [batch, seq, num_heads, head_dim]

Attention

Every checkpoint has 96 query heads over 8 key/value heads (an edit to key/value head j reaches query heads 12j to 12j + 11), 128 wide, with biases on q_proj, k_proj and v_proj (attention_bias) and none on o_proj. head_dim is the config's, so num_heads × head_dim is 12288, wider than hidden_size (4096 on Air, 5120 on GLM-4.5): q_proj widens the stream and o_proj narrows it back, and attention_head_outputs flattened is wider than layer_output. The score scale is 128 ** -0.5; there is no window, sink or softcap, the default sdpa load computes the same function as an eager one, and the six attention interior values need attn_implementation="eager".

The tokenizer and the chat template

The tokenizer prepends nothing: a raw prompt is the text's tokens from position 0. The chat template starts with [gMASK]<sop><|user|>; on GLM-4.7 add_generation_prompt=True ends the prompt at <|assistant|><think>, so the next token begins a reasoning trace.

What loads as this family

The configs that say Glm4MoeForCausalLM are GLM-4.5, GLM-4.5-Air, their Base models, GLM-4.6 and GLM-4.7 (and their FP8 copies). Each stores one multi-token-prediction block after the last (num_nextn_predict_layers 1); transformers skips its weights at load, so model.layers has num_hidden_layers blocks (92, or 46 on Air). GLM-4.7-Flash is glm4_moe_lite (latent attention), and GLM-5 is glm_moe_dsa.

From the family module

04 · nnterp/families/glm4_moe.py

GLM-4.5 / 4.6 and GLM-4.5-Air (Glm4MoeForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, which returns a bare tensor. Attention goes through the shared eager forward; rotary embeddings cover only partial_rotary_factor of each head and q_norm/k_norm run on the heads when use_qk_norm is set, both before the interface, so queries and keys are read as the interface sees them. head_dim is the config's (128 on the real checkpoints, not hidden_size // num_heads).

The first first_k_dense_replace blocks have a dense MLP, the rest a mixture of experts. The mixture returns the routed experts' sum plus a shared expert's output as one tensor, so mlp_output is everything the block adds after attention; the shared expert (mlp.shared_experts) is a dense MLP of the same class.