glm4_moe_lite¶
GLM-4.7-Flash (Glm4MoeLiteForCausalLM).
Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm,
self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual
added in the block, which returns a bare tensor. Attention is DeepSeek-V2's
multi-head latent attention through the shared eager forward: queries and keys are
qk_head_dim wide (qk_nope_head_dim + qk_rope_head_dim), values
v_head_dim, and the interface sees num_heads key/value heads. The config
maps head_dim to qk_rope_head_dim, the rotary part only, so both widths come
from deepseek_v2's size functions.
mlp_layer_types says which blocks are dense (the first, by default) and which
are a mixture of experts. The mixture returns the routed experts' sum plus a shared
expert's output as one tensor, so mlp_output is everything the block adds after
attention; the shared expert (mlp.shared_experts) is a dense MLP of the same class.