Skip to content

hunyuan_v1_moe

hunyuan_v1_moe

Hunyuan MoE V1 (Hunyuan-A13B) (HunYuanMoEV1ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. The attention norms its queries and keys per head after the rotary embedding, before the interface. Every block's MLP is a mixture of experts that adds a shared expert (mlp.shared_mlp) to the routed sum and returns one tensor.

RENAME module-attribute 

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'gate': 'router', 'shared_mlp': 'shared_experts'}

ENVOYS module-attribute 

ENVOYS = {HunYuanMoEV1DecoderLayer: Layer, HunYuanMoEV1Attention: Attention, HunYuanMoEV1Moe: Mlp}

Mlp 

Mlp(*args: Any, **kwargs: Any)

Bases: Moe

A mixture of experts plus a shared expert (shared_mlp, run first), returned as one bare tensor; the residual is added in the block.

The router (gate) projects with a child nn.Linear, wg, in float32: its output is the logits.

router_logits 

router_logits(value) -> RouterLogits