nnterp · encyclopedia hunyuan_v1_moe · nnterp.families.hunyuan_v1_moe

family hunyuan_v1_moe

Hunyuan-A13B

Hunyuan's block, query and key heads RMS-normed after the rotary embedding, with a mixture on every block: a shared expert that runs first, plus 8 of 64 routed experts whose weights are renormalized to sum to one.

Query/key normsMixture of expertsGrouped-query attention

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card.

embed_tokenstoken_embeddings
→
layers× 32
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model HunYuanMoEV1ForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] HunYuanMoEV1DecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn HunYuanMoEV1Attentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp HunYuanMoEV1Moelayoutwhat it isread at
⚠︎mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
⚠︎mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output
router_logits RouterLogits [batch seq num_experts] The router's logits, one per expert, before the scoring

the child module router.wg's output
expert_weights ExpertWeights [batch seq top_k] The weight each selected expert's output is scaled by

the child module experts's inputs, element 2
expert_indices ExpertIndices [batch seq top_k] The expert each routing slot sends the token to

the child module experts's inputs, element 1
expert_outputs ExpertOutputs [batch seq top_k hidden_size] Each routing slot's weighted expert output

inside experts's forward: the output of weighted_out_view_0 inside experts_forward_1
routed_output Residual [batch seq hidden_size] The routed experts' combined output

the child module experts's output
shared_expert_output Residual [batch seq hidden_size] The shared expert's output

the child module shared_experts's output

Printout

What print(model) shows on tencent/Hunyuan-A13B-Instruct: the native modules, standard names and every standard value with its layout.

HunYuanMoEV1ForCausalLM(
  (lm_head): Linear(in_features=4096, out_features=128167, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(128167, 4096, padding_idx=127961)
  (layers): ModuleList(
    (0-31): 32 x HunYuanMoEV1DecoderLayer(
      (self_attn): HunYuanMoEV1Attention(
        (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
        (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
        (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
        (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        (query_layernorm): HunYuanMoEV1RMSNorm((128,), eps=1e-05)
        (key_layernorm): HunYuanMoEV1RMSNorm((128,), eps=1e-05)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): HunYuanMoEV1Moe(
        (router/gate): HunYuanMoEV1Gate(
          (wg): Linear(in_features=4096, out_features=64, bias=False)
        )
        (experts): HunYuanMoEV1Experts(
          (act_fn): SiLUActivation()
        )
        (shared_experts/shared_mlp): HunYuanMoEV1MLP(
          (gate_proj): Linear(in_features=4096, out_features=3072, bias=False)
          (up_proj): Linear(in_features=4096, out_features=3072, bias=False)
          (down_proj): Linear(in_features=3072, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
        )
        (router_logits) -> RouterLogits [batch seq num_experts]: The router's logits, one per expert, before the scoring
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (expert_weights) -> ExpertWeights [batch seq top_k]: The weight each selected expert's output is scaled by
        (expert_indices) -> ExpertIndices [batch seq top_k]: The expert each routing slot sends the token to
        (expert_outputs) -> ExpertOutputs [batch seq top_k hidden_size]: Each routing slot's weighted expert output
        (routed_output) -> Residual [batch seq hidden_size]: The routed experts' combined output
        (shared_expert_output) -> Residual [batch seq hidden_size]: The shared expert's output
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): HunYuanMoEV1RMSNorm((4096,), eps=1e-05)
      (post_attention_layernorm): HunYuanMoEV1RMSNorm((4096,), eps=1e-05)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): HunYuanMoEV1RMSNorm((4096,), eps=1e-05)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers32
model.hidden_size4096
model.num_heads32
model.num_kv_heads8
model.head_dim128
model.qk_head_dim128
model.vocab_size128167
model.intermediate_size3072
architecturesHunYuanMoEV1ForCausalLM
num_hidden_layers32
hidden_size4096
intermediate_size3072
num_attention_heads32
num_key_value_heads8
head_dim128
vocab_size128167
max_position_embeddings32768
hidden_actsilu
rms_norm_eps1e-05
tie_word_embeddingsTrue
dtypebfloat16
moe_intermediate_size32 entries
num_experts64
routed_scaling_factor1.0
norm_topk_probTrue
attention_biasFalse
use_qk_normTrue

The Notes

03 · what to know before you run

The block, in order

h   = x + self_attn(input_layernorm(x))           # query_layernorm, key_layernorm inside
n   = post_attention_layernorm(h)
out = h + shared_mlp(n) + experts(n, gate(n))     # one shared expert, then 8 of 64 routed

Llama's pre-norm block and Llama's names, post_attention_layernorm included (the mixture's input norm). Every block's mlp is a mixture; none is dense. The block adds the attention's output and the mixture's, so the identity is the plain sum, exact in bfloat16:

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(x + attn + mlp, out)

The shared expert runs first, and mlp_output includes it

The mixture runs shared_mlp (shared_experts in nnterp's names) on every token before the router, then adds the routed sum to it: mlp_output is routed_output + shared_expert_output, exactly. Read shared_expert_output before router_logits in a trace; the other order fails as out of order.

moe = model.layers[1].mlp
with model.trace(prompt):
    shared = moe.shared_expert_output.save()     # first: it runs before the router
    logits = moe.router_logits.save()
    routed = moe.routed_output.save()
    mlp = moe.mlp_output.save()

torch.testing.assert_close(routed + shared, mlp)

The shared expert and each routed expert are SwiGLU MLPs intermediate_size wide, 3072 on A13B. The config also carries moe_intermediate_size (3072 on each block) and num_shared_expert (1 on each block); the module reads neither, and builds one shared expert intermediate_size wide. Zeroing shared_expert_output leaves mlp_output equal to routed_output; the routing does not depend on it.

The router runs in float32 and its 8 weights sum to one

The router gate projects with a child nn.Linear, wg, whose weight is stored in float32 and stays float32 in a bfloat16 load; router_logits is that projection's output, [batch, seq, 64]. The router takes a softmax over all 64 logits in float32, keeps the 8 largest and divides them by their sum, whatever the config's norm_topk_prob says. expert_weights are those renormalized weights, slot 0 the largest, and they and expert_outputs are float32 in a bfloat16 model; routed_output and mlp_output are in the model's dtype.

moe = model.layers[1].mlp
with model.trace(prompt):
    logits = moe.router_logits.save()       # [batch, seq, num_experts], float32
    w = moe.expert_weights.save()           # [batch, seq, top_k], float32
    idx = moe.expert_indices.save()

top = logits.softmax(-1).topk(moe.top_k, dim=-1)
torch.testing.assert_close(top.values / top.values.sum(-1, keepdim=True), w)
assert torch.equal(top.indices, idx)

moe_topk is a list in the config, one entry per block, and each block's router reads its own; on A13B every entry is 8. top_k is the block's value.

Ablating an expert touches only the tokens routed to it

Zeroing expert e's weight where expert_indices == e removes that slot's term and leaves the token's other weights as they were: they are not renormalized again, so the token's routed terms then sum with weights below one, while its shared expert term is unchanged. mlp_output changes on exactly the tokens that chose e:

moe, e = model.layers[1].mlp, 3
with model.trace(prompt):
    idx = moe.expert_indices.save()
    clean = moe.mlp_output.save()
with model.trace(prompt):
    moe.expert_weights = moe.expert_weights.masked_fill(moe.expert_indices == e, 0)
    ablated = moe.mlp_output.save()

changed = (ablated != clean).any(-1)                       # [batch, seq]
assert torch.equal(changed, (idx == e).any(-1))            # only the tokens that chose e

Attention: per-head query and key norms, grouped heads

The attention is Hunyuan's dense one: query_layernorm and key_layernorm are RMSNorms 128 wide, applied to each head's rotated query and key, and attention_queries and attention_keys are read after them, so scale a head's query at attention_queries, not at q_proj.output, where the norm removes it. A13B has 32 query heads over 8 key/value heads, head_dim 128 and a query scale of head_dim ** -0.5; an edit to key/value head j reaches query heads 4j to 4j + 3. The rotary base is rope_theta * alpha ** (head_dim / (head_dim - 2)), 1.116e7 with alpha 1000. The interior needs attn_implementation="eager".

The readout and the tokenizer

logits is lm_head.output, with no cap or scale, and project_on_vocab on the last block's layer_output equals logits. A13B stores no lm_head weight: the head is embed_tokens (tie_word_embeddings), so an edit to embed_tokens.weight is an edit to the unembedding. The tokenizer prepends nothing to a plain prompt; the chat template starts with <|startoftext|>.

What this family module covers

The module serves model_type hunyuan_v1_moe, which is tencent/Hunyuan-A13B-Instruct (80B parameters, 13B active per token, 32 blocks). Hunyuan-A13B-Pretrain names the same architecture class but says model_type hunyuan, which loads only with its own remote code. The dense Hunyuan models are hunyuan_v1_dense.

From the family module

04 · nnterp/families/hunyuan_v1_moe.py

Hunyuan MoE V1 (Hunyuan-A13B) (HunYuanMoEV1ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. The attention norms its queries and keys per head after the rotary embedding, before the interface. Every block's MLP is a mixture of experts that adds a shared expert (mlp.shared_mlp) to the routed sum and returns one tensor.