glm4_moe_lite¶
GLM-4.7-Flash (GLM-4-MoE-Lite) on vLLM (vllm.model_executor.models.glm4_moe_lite).
DeepSeek-V2's block under GLM's names: Llama's names, a fused block that takes
and returns the stream as (hidden_states, residual), DeepSeek's latent
attention (vLLM's own DeepSeek-V2 attention classes, subclassed) and GLM-4-MoE's
mixture of experts. A block's mlp is the mixture of experts with its shared
expert, or a dense MLP on the first first_k_dense_replace blocks; either
way its output is what the block adds, and the routing is inside vLLM's fused
MoE kernel.
The attention is latent (MLA). By default vLLM runs it in a kernel of its own
(mla_attn), which works on the compressed keys and values and serves
nothing of the per-head queries, keys and values transformers reads; with
VLLM_MLA_DISABLE=1 it runs its ordinary attention layer on padded
tensors. Either way the attention's interior is not mapped here, and its
contribution, which is the module's output on both paths, is.
The two engines' latent norms (q_a_layernorm, kv_a_layernorm) differ:
vLLM's take the config's rms_norm_eps (1e-5 on GLM-4.7-Flash),
transformers' a fixed 1e-6, so the attention's values differ between them by
that much (8e-4 of the queries' scale in the first block of GLM-4.7-Flash).
module-attribute
¶
module-attribute
¶
LATENT = "vLLM runs GLM-4-MoE-Lite's latent attention in its own kernel, on compressed keys and values; not mapped on vLLM"
module-attribute
¶
ENVOYS = {Glm4MoeLiteDecoderLayer: Layer, Glm4MoeLiteMLAAttention: Attention, Glm4MoeLiteAttention: Attention, Glm4MoeLite: Mlp, Glm4MoeLiteMLP: Mlp}
Bases: FusedLayer
The decoder block: forward(positions, hidden_states, residual, llama_4_scaling) -> (hidden_states, residual).