glm4_moe¶
GLM-4.5 / 4.6 (GLM-4-MoE) on vLLM (vllm.model_executor.models.glm4_moe).
Llama's names and Llama's fused block: it takes and returns the stream as
(hidden_states, residual). The attention rotates only
partial_rotary_factor of each head and, with use_qk_norm, norms each
head's queries and keys first; both happen before vLLM's attention layer, so
the queries and keys it receives are the ones transformers reads at its
interface. The first first_k_dense_replace blocks have a dense MLP, the
rest a mixture of experts with a shared expert; either way the mlp's
output is what the block adds. The routing is inside vLLM's fused MoE kernel
and has no values here.
module-attribute
¶
module-attribute
¶
ENVOYS = {Glm4MoeDecoderLayer: Layer, Glm4MoeAttention: Attention, Glm4MoE: Mlp, Glm4MoeMLP: Mlp}
Bases: FusedLayer
The decoder block: forward(positions, hidden_states, residual) -> (hidden_states, residual).