glm4¶
GLM-4 (Glm4ForCausalLM, GLM-4-0414 and GLM-Z1).
Llama's tree with a sandwich block of four norms::
h = x + post_self_attn_layernorm(self_attn(input_layernorm(x)))
out = h + post_mlp_layernorm(mlp(post_attention_layernorm(h)))
What the block adds is each post-norm's output, not the module's, so
attention_output points at post_self_attn_layernorm and mlp_output at
post_mlp_layernorm. Unlike Gemma-2, post_attention_layernorm here is the
pre-MLP norm (the Llama meaning), so the standard aliases need no renaming and
mlp.input is its output. The attention runs the shared eager forward with a
partial rotary embedding; the MLP is a fused gate_up_proj.