hunyuan_v1_moe¶
Hunyuan MoE V1 (Hunyuan-A13B) on vLLM (vllm.model_executor.models.hunyuan_v1).
vLLM runs Hunyuan's dense and MoE models through one implementation. The
names are Llama's and the block is fused, with something more beside the
two halves: it is called forward(positions, hidden_states, residual,
kv_states) and returns (hidden_states, residual, kv_states), the stream
being the sum of the first two and the third the keys and values its
attention computed, which the next block reads when the checkpoint shares
them across layers (CLA). So the block's stream is read and written on the
first two elements and the third is passed through, and a skipped block hands
on no keys and values (None), so skipping is for checkpoints without
CLA, as are these values: a CLA block's cross-attention, which reuses the
previous block's keys and values, is not mapped.
The attention returns (output, (keys, values)): its contribution is the
first element. It norms its queries and keys per head after the rotary
embedding, as on transformers, and the queries and keys served are the
normed ones the attention layer receives. Every block's mlp is the
mixture of experts, whose output (the shared expert shared_mlp included)
is what the block adds; the routing is inside vLLM's fused MoE kernel and has
no values here.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'shared_mlp': 'shared_experts'}
module-attribute
¶
ENVOYS = {HunYuanDecoderLayer: Layer, HunYuanAttention: Attention, HunYuanSparseMoeBlock: Mlp}
Bases: FusedLayer
The decoder block: forward(positions, hidden_states, residual, kv_states) -> (hidden_states, residual, kv_states).