nnterp · encyclopedia glm · nnterp.families.glm

family glm

GLM-4GLM-Edge

Llama's block with a fused gate and up projection and a rotary embedding that turns adjacent pairs of dimensions; on GLM-4-9B the rotary covers only the first half of each head, and the queries, keys and values carry a bias.

Biased q, k, vPartial rotaryInterleaved rotaryGrouped-query attention

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card.

embed_tokenstoken_embeddings
→
layers× 40
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model GlmForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] GlmDecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn GlmAttentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp GlmMLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output

Printout

What print(model) shows on zai-org/glm-4-9b-hf: the native modules, standard names and every standard value with its layout.

GlmForCausalLM(
  (lm_head): Linear(in_features=4096, out_features=151552, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(151552, 4096, padding_idx=151329)
  (layers): ModuleList(
    (0-39): 40 x GlmDecoderLayer(
      (self_attn): GlmAttention(
        (q_proj): Linear(in_features=4096, out_features=4096, bias=True)
        (k_proj): Linear(in_features=4096, out_features=256, bias=True)
        (v_proj): Linear(in_features=4096, out_features=256, bias=True)
        (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): GlmMLP(
        (gate_up_proj): Linear(in_features=4096, out_features=27392, bias=False)
        (down_proj): Linear(in_features=13696, out_features=4096, bias=False)
        (act_fn/activation_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): GlmRMSNorm((4096,), eps=1.5625e-07)
      (post_attention_layernorm): GlmRMSNorm((4096,), eps=1.5625e-07)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): GlmRMSNorm((4096,), eps=1.5625e-07)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers40
model.hidden_size4096
model.num_heads32
model.num_kv_heads2
model.head_dim128
model.qk_head_dim128
model.vocab_size151552
model.intermediate_size13696
architecturesGlmForCausalLM
num_hidden_layers40
hidden_size4096
intermediate_size13696
num_attention_heads32
num_key_value_heads2
head_dim128
vocab_size151552
max_position_embeddings8192
hidden_actsilu
rms_norm_eps1.5625e-07
tie_word_embeddingsFalse
dtypebfloat16
attention_biasTrue

The Notes

03 · what to know before you run

The block, in order

h   = x + self_attn(input_layernorm(x))      # biased q, k, v; paired rotary
out = h + mlp(post_attention_layernorm(h))   # one gate_up_proj

Llama's pre-norm block and Llama's names: post_attention_layernorm is the MLP's input norm. Nothing norms a sublayer's output, so attention_output is o_proj's output, mlp_output is down_proj's, and the identity is the plain sum, exact:

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

assert torch.equal(x + attn + mlp, out)

Rotary turns adjacent pairs, on half of each head on GLM-4-9B

The rotary embedding turns dimension 2i with 2i + 1, where Llama turns i with i + rot / 2 (rot being the number of rotated dimensions). Pair i turns at Llama's frequency i: GLM's apply_rotary_pos_emb repeats each frequency twice in a row, and its rotate_half pairs each even dimension with the odd one after it. The rotation covers the first partial_rotary_factor × head_dim dimensions of each query and key head and passes the rest through unchanged. On GLM-4-9B the factor is 0.5 (the config does not say it; GlmConfig sets it), so dimensions 0 to 63 of each 128-wide head are turned and 64 to 127 carry no position. GLM-Edge's configs set 1.0: the whole head turns. The base (rope_theta) is 10,000 on every checkpoint here.

attention_queries and attention_keys are read after the rotary. Recomputing them from q_proj.output and k_proj.output with the pairs adjacent matches exactly; with Llama's pairing it does not. To compare with a family that rotates Llama's way, or to reuse rotary code written for one, reorder the turned part, evens then odds:

hd = model.head_dim
rot = int(hd * model.config.rope_parameters["partial_rotary_factor"])
perm = torch.cat([torch.arange(0, rot, 2), torch.arange(1, rot, 2), torch.arange(rot, hd)])

with model.trace(prompt):
    q = model.layers[1].self_attn.attention_queries.save()

q_half = q[..., perm]          # rotate_half's order; q[..., rot:] carries no position

Queries, keys and values carry a bias

attention_bias is true on GLM-4-9B: q_proj, k_proj and v_proj add a bias, and o_proj has none. GLM-Edge sets it false. Where the bias is on, a zero input still gives queries, keys and values. Zeroing self_attn.input makes every value head equal to its slice of v_proj.bias; each query head then outputs its key/value head's slice whatever its pattern, and attention_output is o_proj applied to those slices, not zero. To zero-ablate the attention, set attention_output to zero, not its input.

with model.trace(prompt):
    model.layers[1].self_attn.input[:] = 0
    v = model.layers[1].self_attn.attention_values.save()

# every value head is its slice of v_proj.bias

The MLP fuses its gate and up projections

mlp.gate_up_proj is one Linear from hidden_size to 2 × intermediate_size; its output's first half is the gate and the second half the up projection, and mlp.output is down_proj(act(gate) * up). There is no gate_proj or up_proj module to hook: a neuron's pre-activation is a column of the first half.

I = model.config.intermediate_size
with model.trace(prompt):
    gate_up = model.layers[1].mlp.gate_up_proj.output.save()

gate, up = gate_up[..., :I], gate_up[..., I:]

Attention

GLM-4-9B and its 128k chat model have 32 query heads over 2 key/value heads, so an edit to key/value head j reaches query heads 16j to 16j + 15; the 1M-context chat model has 4 key/value heads, GLM-Edge-1.5B 16 over 4 and GLM-Edge-4B 24 over 6. head_dim is 128 throughout and the scores are scaled by 128 ** -0.5. There is no window, sink or softcap, and the default sdpa load computes the same function as an eager one; the six attention interior values need attn_implementation="eager".

The tokenizer prepends [gMASK]<sop> on GLM-4-9B

GLM-4-9B's tokenizers (base and chat) put [gMASK] and <sop> before every encoded string, so model.input_ids starts with them and the prompt's first token is at position 2. The chat template writes the same two tokens itself: tokenize a templated prompt with add_special_tokens=False, or the prefix appears twice. GLM-Edge's tokenizer prepends nothing, and its chat template starts at <|user|>.

The readout is the plain projection

model.logits equals lm_head.output; there is no softcap or scale. rms_norm_eps is 1.5625e-07 on GLM-4-9B and 1e-5 on GLM-Edge.

What loads as this family

The configs that say GlmForCausalLM are GLM-4-9B's transformers-format releases (glm-4-9b-hf, glm-4-9b-chat-hf, glm-4-9b-chat-1m-hf) and the GLM-Edge chat models. The repositories without -hf (zai-org/glm-4-9b, glm-4-9b-chat) are chatglm, loaded through their own remote code, which no family covers. GLM-4-0414 and GLM-Z1 are glm4, and GLM-4.5, 4.6 and 4.7 are glm4_moe.

From the family module

04 · nnterp/families/glm.py

GLM-4 (GlmForCausalLM), the first generation: GLM-4-9B.

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. The rotary embedding covers partial_rotary_factor of each head, before the interface. The MLP fuses its gate and up projections into one gate_up_proj and returns a bare tensor.