glm4_moe¶
GLM-4.5 / 4.6 and GLM-4.5-Air (Glm4MoeForCausalLM).
Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm,
self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual
added in the block, which returns a bare tensor. Attention goes through the shared
eager forward; rotary embeddings cover only partial_rotary_factor of each head
and q_norm/k_norm run on the heads when use_qk_norm is set, both before
the interface, so queries and keys are read as the interface sees them. head_dim
is the config's (128 on the real checkpoints, not hidden_size // num_heads).
The first first_k_dense_replace blocks have a dense MLP, the rest a mixture of
experts. The mixture returns the routed experts' sum plus a shared expert's output
as one tensor, so mlp_output is everything the block adds after attention; the
shared expert (mlp.shared_experts) is a dense MLP of the same class.