nnterp · encyclopedia glm4 · nnterp.families.glm4

family glm4

GLM-4-0414GLM-Z1

GLM's block with a second norm after each sublayer, so the stream receives the post-norms' outputs; post_attention_layernorm keeps Llama's meaning, the MLP's input norm.

Sandwich normsBiased q, k, vPartial rotaryInterleaved rotaryGrouped-query attention

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card.

embed_tokenstoken_embeddings
→
layers× 40
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Glm4ForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] Glm4DecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn Glm4Attentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream: the post-self-attention norm's output

the sibling module post_self_attn_layernorm's output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp Glm4MLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream: the post-MLP norm's output

the sibling module post_mlp_layernorm's output

Printout

What print(model) shows on zai-org/GLM-4-9B-0414: the native modules, standard names and every standard value with its layout.

Glm4ForCausalLM(
  (lm_head): Linear(in_features=4096, out_features=151552, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(151552, 4096, padding_idx=151329)
  (layers): ModuleList(
    (0-39): 40 x Glm4DecoderLayer(
      (self_attn): Glm4Attention(
        (q_proj): Linear(in_features=4096, out_features=4096, bias=True)
        (k_proj): Linear(in_features=4096, out_features=256, bias=True)
        (v_proj): Linear(in_features=4096, out_features=256, bias=True)
        (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream: the post-self-attention norm's output
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Glm4MLP(
        (gate_up_proj): Linear(in_features=4096, out_features=27392, bias=False)
        (down_proj): Linear(in_features=13696, out_features=4096, bias=False)
        (act_fn/activation_fn): SiLUActivation()
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream: the post-MLP norm's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
      )
      (input_layernorm): Glm4RMSNorm((4096,), eps=1e-05)
      (post_attention_layernorm): Glm4RMSNorm((4096,), eps=1e-05)
      (post_self_attn_layernorm): Glm4RMSNorm((4096,), eps=1e-05)
      (post_mlp_layernorm): Glm4RMSNorm((4096,), eps=1e-05)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): Glm4RMSNorm((4096,), eps=1e-05)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers40
model.hidden_size4096
model.num_heads32
model.num_kv_heads2
model.head_dim128
model.qk_head_dim128
model.vocab_size151552
model.intermediate_size13696
architecturesGlm4ForCausalLM
num_hidden_layers40
hidden_size4096
intermediate_size13696
num_attention_heads32
num_key_value_heads2
head_dim128
vocab_size151552
max_position_embeddings32768
hidden_actsilu
rms_norm_eps1e-05
tie_word_embeddingsFalse
dtypebfloat16
attention_biasTrue

The Notes

03 · what to know before you run

The block, in order

h   = x + post_self_attn_layernorm(self_attn(input_layernorm(x)))
out = h + post_mlp_layernorm(mlp(post_attention_layernorm(h)))

Four RMSNorms per block, two per sublayer. The names mix two conventions: post_attention_layernorm is the MLP's input norm (Llama's meaning), while the norms after the sublayers are post_self_attn_layernorm and post_mlp_layernorm. So self_attn.input is input_layernorm's output and mlp.input is post_attention_layernorm's.

The contributions are the post-norms' outputs

attention_output is post_self_attn_layernorm's output and mlp_output is post_mlp_layernorm's, the tensors the block adds. self_attn.output[0] is o_proj's output, the tensor entering the post-norm, and mlp.output is down_proj's. The identity is the plain sum of the post-norms' outputs, exact:

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

assert torch.equal(x + attn + mlp, out)

An RMSNorm is scale-invariant up to its eps (1e-5 on every checkpoint here), so scaling a module's output does not scale what the block adds: post_mlp_layernorm(0.5 * y) is post_mlp_layernorm(y). Scale, steer or attribute through attention_output and mlp_output:

with model.trace(prompt):
    model.layers[1].mlp.mlp_output[:] *= 0.5     # halves what the block adds

with model.trace(prompt):
    model.layers[1].mlp.output[:] *= 0.5         # the post-norm rescales it back

Zeroing mlp.output does zero the contribution, since the norm of zero is zero.

Rotary turns adjacent pairs, on half of each head

The attention is GLM's. The rotary embedding turns dimension 2i with 2i + 1, where Llama turns i with i + rot / 2 (rot being the number of rotated dimensions), and pair i turns at Llama's frequency i. It covers only the first partial_rotary_factor × head_dim dimensions of each query and key head: 64 of 128 on every checkpoint here (partial_rotary_factor 0.5), so dimensions 64 to 127 carry no position. The base (rope_theta) is 10,000. attention_queries and attention_keys are read after the rotary; recomputing them from q_proj.output and k_proj.output with adjacent pairs matches exactly. To put the turned part in Llama's order, evens then odds:

hd = model.head_dim
rot = int(hd * model.config.rope_parameters["partial_rotary_factor"])
perm = torch.cat([torch.arange(0, rot, 2), torch.arange(1, rot, 2), torch.arange(rot, hd)])

with model.trace(prompt):
    q = model.layers[1].self_attn.attention_queries.save()

q_half = q[..., perm]

Biases on the 9B models only

attention_bias is true on GLM-4-9B-0414 and GLM-Z1-9B-0414: q_proj, k_proj and v_proj add a bias (o_proj has none), so a zero input to self_attn still gives each value head its slice of v_proj.bias. The 32B models set it false.

The MLP is GLM's fused one: mlp.gate_up_proj.output is the gate (first intermediate_size columns) then the up projection, and mlp.output is down_proj(act(gate) * up).

Attention

The 9B models have 32 query heads over 2 key/value heads (an edit to key/value head j reaches query heads 16j to 16j + 15); GLM-4-32B and GLM-Z1-32B 48 over 2; GLM-Z1-Rumination-32B 48 over 8. head_dim is 128 everywhere, the score scale 128 ** -0.5, and num_heads × head_dim is hidden_size on both sizes. There is no window, sink or softcap; the default sdpa load computes the same function as an eager one, and the six attention interior values need attn_implementation="eager".

The tokenizer prepends nothing; the chat template writes [gMASK]<sop>

A raw prompt is the text's tokens alone, from position 0. The chat template starts with [gMASK]<sop><|user|>, and on GLM-Z1-9B-0414 add_generation_prompt=True ends the prompt at <|assistant|>\n<think>, so the next token begins a reasoning trace rather than the answer. GLM-4-9B-0414's tokenizer names <|user|> as its eos_token; GLM-Z1-9B-0414's names <|endoftext|>.

What loads as this family

The configs that say Glm4ForCausalLM are the 0414 releases: GLM-4-9B, GLM-4-32B and its Base, and the GLM-Z1 reasoning models (9B, 32B, Rumination-32B). max_position_embeddings is 32768, 131072 on Rumination. GLM-4-9B's first release is glm (no post-norms), and GLM-4.5, 4.6 and 4.7 are glm4_moe.

From the family module

04 · nnterp/families/glm4.py

GLM-4 (Glm4ForCausalLM, GLM-4-0414 and GLM-Z1).

Llama's tree with a sandwich block of four norms::

h = x + post_self_attn_layernorm(self_attn(input_layernorm(x)))
out = h + post_mlp_layernorm(mlp(post_attention_layernorm(h)))

What the block adds is each post-norm's output, not the module's, so attention_output points at post_self_attn_layernorm and mlp_output at post_mlp_layernorm. Unlike Gemma-2, post_attention_layernorm here is the pre-MLP norm (the Llama meaning), so the standard aliases need no renaming and mlp.input is its output. The attention runs the shared eager forward with a partial rotary embedding; the MLP is a fused gate_up_proj.