nnterp · encyclopedia hunyuan_v1_dense · nnterp.families.hunyuan_v1_dense

family hunyuan_v1_dense

Hunyuan

Llama's block with each query and key head RMS-normed after the rotary embedding (query_layernorm, key_layernorm), so a head's query and key lengths are set by the norms' gains.

Query/key normsGrouped-query attention

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card.

embed_tokenstoken_embeddings
→
layers× 24
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model HunYuanDenseV1ForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] HunYuanDenseV1DecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn HunYuanDenseV1Attentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp HunYuanDenseV1MLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output

Printout

What print(model) shows on tencent/Hunyuan-0.5B-Pretrain: the native modules, standard names and every standard value with its layout.

HunYuanDenseV1ForCausalLM(
  (lm_head): Linear(in_features=1024, out_features=120818, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(120818, 1024, padding_idx=120002)
  (layers): ModuleList(
    (0-23): 24 x HunYuanDenseV1DecoderLayer(
      (self_attn): HunYuanDenseV1Attention(
        (q_proj): Linear(in_features=1024, out_features=2048, bias=False)
        (k_proj): Linear(in_features=1024, out_features=1024, bias=False)
        (v_proj): Linear(in_features=1024, out_features=1024, bias=False)
        (o_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (query_layernorm): HunYuanDenseV1RMSNorm((128,), eps=1e-05)
        (key_layernorm): HunYuanDenseV1RMSNorm((128,), eps=1e-05)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): HunYuanDenseV1MLP(
        (gate_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3584, bias=False)
        (down_proj): Linear(in_features=3584, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): HunYuanDenseV1RMSNorm((1024,), eps=1e-05)
      (post_attention_layernorm): HunYuanDenseV1RMSNorm((1024,), eps=1e-05)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): HunYuanDenseV1RMSNorm((1024,), eps=1e-05)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers24
model.hidden_size1024
model.num_heads16
model.num_kv_heads8
model.head_dim128
model.qk_head_dim128
model.vocab_size120818
model.intermediate_size3584
architecturesHunYuanDenseV1ForCausalLM
num_hidden_layers24
hidden_size1024
intermediate_size3584
num_attention_heads16
num_key_value_heads8
head_dim128
vocab_size120818
max_position_embeddings262144
hidden_actsilu
rms_norm_eps1e-05
tie_word_embeddingsTrue
dtypebfloat16
attention_biasFalse
use_qk_normTrue

The Notes

03 · what to know before you run

The block, in order

q, k = rope(q_proj(n), k_proj(n))                 # n = input_layernorm(x)
q, k = query_layernorm(q), key_layernorm(k)       # per head, after the rotary
h    = x + o_proj(attention(q, k, v_proj(n)))
out  = h + mlp(post_attention_layernorm(h))

Llama's pre-norm block and Llama's names, post_attention_layernorm included (the MLP's input norm). Nothing norms a sublayer's output, so attention_output is o_proj's output and mlp_output the MLP's, and the identity is the plain sum, exact in bfloat16:

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(x + attn + mlp, out)

Each query and key head is normed after the rotary embedding

query_layernorm and key_layernorm are RMSNorms head_dim wide (128 on every checkpoint), one gain vector shared by every head, applied to each head's rotated query and key. attention_queries and attention_keys are read after them: attention_queries equals query_layernorm.output exactly. The norm removes the length a head's query or key had before it, so the dot products' scale comes from the two gains (their means are 1.15 to 1.41 across the 24 blocks of 0.5B). On 0.5B, tripling head 0's slice of q_proj.output at block 8 moves the logits by a few thousandths at most, while tripling head 0 at attention_queries sharpens its pattern from about 0.8 nats of entropy to about 0.02. Scale or steer a head's query or key where no norm follows:

attn = model.layers[1].self_attn
with model.trace(prompt):
    attn.attention_queries[:, 0] *= 3        # head 0 attends more sharply
    logits = model.logits.save()

Each head is normed on its own, so an edit to one head's slice of q_proj.output stays in that head.

Grouped-query attention, scaled by head_dim ** -0.5

0.5B has 16 query heads over 8 key/value heads (1.8B 16 over 4, 4B and 7B 32 over 8). attention_keys and attention_values are read before repeat_kv, [batch, 8, seq, 128] on 0.5B, so an edit to key/value head j reaches query heads 2j and 2j + 1. The query scale is head_dim ** -0.5, and attention_scores are the scaled products with the causal mask added:

attn = model.layers[1].self_attn
with model.trace(prompt):
    q = attn.attention_queries.save()
    k = attn.attention_keys.save()
    scores = attn.attention_scores.save()

k = k.repeat_interleave(model.num_heads // model.num_kv_heads, dim=1)
qk = q @ k.transpose(-1, -2) * model.head_dim ** -0.5
causal = torch.ones_like(qk[0, 0], dtype=torch.bool).tril()
torch.testing.assert_close(qk[..., causal], scores[..., causal])

The interior needs attn_implementation="eager". A default load runs sdpa, which computes the same function: its logits differ from eager's by at most 3e-5.

The rotary base is raised by alpha at load

rope_scaling is {"type": "dynamic", "alpha": 1000.0} with rope_theta 10000 on every checkpoint but the two translation models, whose alpha is 100000. The rotary embedding builds its frequencies once, from a base of rope_theta * alpha ** (head_dim / (head_dim - 2)): 1.116e7 at alpha 1000, 1.2e9 at 100000. Code that recomputes the rotation takes that base, not rope_theta:

rotary = model._module.model.rotary_emb
base = 1 / rotary.inv_freq[1] ** (model.head_dim / 2)    # 1.116e7 on 0.5B

Position 0 is a sink, and no BOS token is added

The tokenizer prepends nothing to a plain prompt, so the first real token takes the role of a sink. On 0.5B its residual norm is 78 after blocks 8 and 16, against 0.6 to 1.5 at the other positions, and at block 16 every head puts 0.7 to 0.9 of its attention on it (test prompts). The chat templates start with the BOS token (<｜hy_begin▁of▁sentence｜> on 0.5B to 4B, <|startoftext|> on 7B), which then is position 0. Slice [:, 1:] before averaging activations for steering vectors, mean ablation or probes.

The readout is plain, and the head is the embedding

logits is lm_head.output, with no cap or scale, and project_on_vocab on the last block's layer_output equals logits. lm_head and embed_tokens are one parameter on every checkpoint (tie_word_embeddings), so an edit to embed_tokens.weight is an edit to the unembedding. token_embeddings equals layers[0].input.

What this family module covers

The module serves every checkpoint whose config says model_type hunyuan_v1_dense: Hunyuan 0.5B, 1.8B, 4B and 7B, base and instruct, the earlier 7B-Instruct-0124, and the translation models Hunyuan-MT-7B and MT-Chimera-7B. The 0.5B to 4B checkpoints share a 120818-token vocabulary, the 7B ones about 128K. Hunyuan-A13B is hunyuan_v1_moe; Hunyuan-7B-Pretrain-0124 says model_type hunyuan and loads only with its own remote code.

From the family module

04 · nnterp/families/hunyuan_v1_dense.py

Hunyuan dense V1 (HunYuanDenseV1ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. The attention norms its queries and keys per head (query_layernorm, key_layernorm) after the rotary embedding and before the interface, so the queries and keys are read after both.