hunyuan_v1_dense

hunyuan_v1_dense

Hunyuan dense V1 (HunYuanDenseV1ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. The attention norms its queries and keys per head (query_layernorm, key_layernorm) after the rotary embedding and before the interface, so the queries and keys are read after both.

RENAME module-attribute

RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm'}

ENVOYS module-attribute

ENVOYS = {HunYuanDenseV1DecoderLayer: Layer, HunYuanDenseV1Attention: Attention, HunYuanDenseV1MLP: Mlp}