HyperCLOVA X

The block, in order

x      = embed_tokens(ids) * embedding_multiplier
h      = x + post_norm1(self_attn(input_layernorm(x))) * residual_multiplier
out    = h + post_norm2(mlp(post_attention_layernorm(h))) * residual_multiplier
logits = lm_head(norm(out)) * logits_scaling

With use_post_norm (Think-14B) a block has four RMSNorms, two per sublayer, and the names are the trap: post_attention_layernorm is the MLP's input norm, as on Llama, and the norms after the sublayers are post_norm1 (after the attention) and post_norm2 (after the MLP). nnterp keeps the native names. Each multiplier is one number for the whole model. On Think-14B they are embedding_multiplier 10, residual_multiplier 1, attention_multiplier 1/128 and logits_scaling 0.125.

The contributions are the post-norms' outputs, times residual_multiplier

attention_output is post_norm1.output * residual_multiplier and mlp_output is post_norm2.output * residual_multiplier: the tensors the block adds. On Think-14B the multiplier is 1.0, so they equal the post-norms' outputs bit for bit. self_attn.output[0] and mlp.output are the tensors entering the post-norms. The identity is the plain sum, exact in float32 and bfloat16:

with model.trace(prompt):
    x = model.layers[1].input.save()
    post = model.layers[1].post_norm1.output.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(attn, post * model.config.residual_multiplier)
torch.testing.assert_close(x + attn + mlp, out)

With post-norms, scale the contribution, not the module. RMSNorm is scale-invariant: post_norm2(0.5 * y) equals post_norm2(y) up to eps: halving mlp.output leaves the stream as it was, and zeroing it works only because the norm of zero is zero. Partial ablations and steering go through mlp_output / attention_output, which the block adds as they are:

with model.trace(prompt):
    model.layers[1].mlp.mlp_output[:] *= 0.5        # halves what the block adds

with model.trace(prompt):
    model.layers[1].mlp.output[:] *= 0.5            # a no-op: post_norm2 rescales it back

A write to a contribution goes through the multiplier

The contributions are computed copies: a write to attention_output or mlp_output reaches the model by dividing the whole edited copy by residual_multiplier and handing it back as the post-norm's output, and the block multiplies it again. With the multiplier at 1.0, as on Think-14B, both steps are exact: an edit at one position leaves every other position bit-identical, in bfloat16 too. With another multiplier the unedited positions go through the round trip as well and move by rounding in bfloat16. Editing the post-norm's output by the vector over the multiplier reaches the same place and touches nothing else, for any multiplier:

m = model.config.residual_multiplier
with model.trace(prompt):
    model.layers[1].mlp.mlp_output[:, -1] += v            # exact when m is 1.0

with model.trace(prompt):
    model.layers[1].post_norm2.output[:, -1] += v / m     # position -1 only, for any m

A read changes nothing: the forward stays bit-identical.

The query scale is attention_multiplier

The attention multiplies q · kᵀ by config.attention_multiplier in place of head_dim ** -0.5. On Think-14B it is 1/128, 1 / head_dim, a factor of 11.3 below 128 ** -0.5; on Think-32B it is 128 ** -0.5. attention_scores are read after the scale and the causal mask, and softmax(attention_scores) reproduces attention_probabilities. Code that recomputes scores takes the multiplier from the config:

attn = model.layers[1].self_attn
with model.trace(prompt):
    q = attn.attention_queries.save()
    k = attn.attention_keys.save()
    scores = attn.attention_scores.save()

k = k.repeat_interleave(model.num_heads // model.num_kv_heads, dim=1)
qk = q @ k.transpose(-1, -2) * model.config.attention_multiplier    # not head_dim ** -0.5
causal = torch.ones_like(qk[0, 0], dtype=torch.bool).tril()
torch.testing.assert_close(qk[..., causal], scores[..., causal])

Think-14B has 48 query heads over 8 key/value heads: attention_keys and attention_values are [batch, 8, seq, 128], read before repeat_kv, and an edit to key/value head j reaches query heads 6j to 6j + 5. The six interior values need attn_implementation="eager".

Logits are the head's output times logits_scaling

model.logits is model.lm_head.output * logits_scaling exactly (0.125 on Think-14B), and project_on_vocab multiplies too, so a logit lens at the last block equals logits. A softmax over lm_head.output, or over lm_head(norm(hidden)) written by hand, is at temperature 1/8: sharper than the model's own distribution. Take probabilities and KLs from logits or project_on_vocab.

with model.trace(prompt):
    resid = model.layers[-1].layer_output.save()
    raw = model.lm_head.output.save()
    logits = model.logits.save()

torch.testing.assert_close(logits, raw * model.config.logits_scaling)
torch.testing.assert_close(model.project_on_vocab(resid), logits)

lm_head and embed_tokens are separate weights.

token_embeddings is the unscaled lookup

The embedding module returns the plain lookup, and the model multiplies it by embedding_multiplier afterwards: layers[0].input is token_embeddings * 10 on Think-14B. Embeddings passed as inputs_embeds are multiplied too. An edit to token_embeddings reaches block 0 multiplied by embedding_multiplier; to set what block 0 reads, edit layers[0].input.

with model.trace(prompt):
    emb = model.token_embeddings.save()
    first = model.layers[0].input.save()

torch.testing.assert_close(first, emb * model.config.embedding_multiplier)

Loading Think-14B

The checkpoint is stored in float32 (torch_dtype in its config), and a default load keeps it, about 59 GB for its 14.7B parameters; pass dtype=torch.bfloat16 to halve it. The tokenizer prepends no BOS token to a plain prompt; the chat template starts with <|im_start|>.

The 32B vision-language checkpoint

HyperCLOVAX-SEED-Think-32B is a vision-language wrapper (hyperclovax_vision_v2, remote code) whose text model is this family. Its text config turns the post-norms off (use_post_norm false), sets embedding_multiplier, residual_multiplier and logits_scaling to 1.0 and attention_multiplier to 128 ** -0.5: under these names it computes Llama's block. With use_post_norm false, post_norm1 and post_norm2 are identities, and attention_output and mlp_output are the modules' outputs times residual_multiplier.

The Qwen ViT vision encoder

Every image in one row

The vision encoder runs on [patches, vision_hidden_size], the patches of every image in the invoke concatenated. Its stream values (patch_embeddings, every block's layer_output, attention_output and mlp_output, tower_output) are served [1, patches, vision_hidden_size], a view with a leading images axis of 1, so in-place edits land; vision.layers[i].input and .output stay the native [patches, vision_hidden_size], and the block identity holds across the two by broadcasting. The processor's image_grid_thw ([t, h, w] per image, in patches; an image is two identical frames, so t is 1) splits the row: image j has t * h * w patches.

inputs = model.processor(text=prompt, images=[red, wide], return_tensors="pt")
sizes = inputs["image_grid_thw"].prod(-1).tolist()   # patches per image
with model.trace(prompt, images=[red, wide]):
    patches = model.vision.layers[1].layer_output.save()
per_image = patches[0].split(sizes)                  # one [t*h*w, vision_hidden_size] each

Inside an image the patches are in merge-block order: the spatial_merge_size × spatial_merge_size (2 × 2) block of neighbours the merger folds into one image token is four consecutive rows, blocks in raster order. Index 4 * b to 4 * b + 3 is block b, not a raster row.

The attention runs once per image, so there is no pattern

The attention computes queries, keys and values for the whole row, then calls the attention function once per image (once per window on Qwen2.5-VL's windowed blocks) and concatenates the outputs. attention_queries, attention_keys and attention_values are served whole, before the split, [1, heads, patches, head_dim], the queries and keys after the rotary embedding; attention_head_outputs is the calls' outputs concatenated back, [1, patches, heads, head_dim] (under any implementation but flash, which runs one call over cu_seqlens). attention_scores and attention_probabilities are Unavailable under every implementation, eager included: "no one tensor is the block's pattern". The split points are the attention's cu_seqlens argument, self_attn.inputs[1]["cu_seqlens"], so a pattern per image is a softmax over each slice:

attn = model.vision.layers[1].self_attn
with model.trace(prompt, images=[red, wide]):
    cu = attn.inputs[1]["cu_seqlens"].save()        # read first: the module's input
    q = attn.attention_queries.save()
    k = attn.attention_keys.save()

patterns = []
for a, b in zip(cu[:-1].tolist(), cu[1:].tolist()):
    scores = q[0, :, a:b] @ k[0, :, a:b].transpose(-1, -2) * model.vision.head_dim ** -0.5
    patterns.append(scores.softmax(-1))             # [heads, patches_j, patches_j]

Each pattern times that image's values reproduces attention_head_outputs exactly (checked on Qwen3.5's pinned tiny checkpoint).

The merger is inside the vision encoder

model.projector is the vision encoder's own merger: it norms each patch, concatenates each 2 × 2 block into one 4 * vision_hidden_size vector and runs a two-layer MLP to the text model's width, so image tokens per image are t * h * w / 4. It reads the last block's stream with no norm between (projector.input == tower_output[0], the native [patches, vision_hidden_size]). Because the merger runs inside the vision encoder's forward, vision.tower_output (the encoder's output) is reached after it: read model.projector.input and .output before vision.tower_output in a trace.

Sizes, and no `image_size`

model.vision reads its sizes off the vision config: vision_hidden_size is the encoder's width (embed_dim on Qwen2-VL, whose config's hidden_size is the merger's output width), num_heads, intermediate_size (embed_dim * mlp_ratio on Qwen2-VL), patch_size, and two of its own, spatial_merge_size (2 on every line) and window_size (Qwen2.5-VL's window side in pixels, 112; None on the others). image_size raises Unavailable: the processor cuts each image into its own grid.

The variants

line	block norms, merger norm	MLP	beyond the shared encoder
Qwen2-VL	LayerNorm	`fc1`, QuickGELU, `fc2`	none
Qwen2.5-VL	RMSNorm	SwiGLU, biased	windows of 112 pixels except on `fullatt_block_indexes`
Qwen3-VL, -MoE	LayerNorm	`linear_fc1`, GELU (tanh), `linear_fc2`	learned `pos_embed`; DeepStack taps
Qwen3.5, -MoE	LayerNorm	`linear_fc1`, GELU (tanh), `linear_fc2`	learned `pos_embed`

On Qwen2.5-VL the encoder permutes the patches into windows at entry, so its blocks' values, tower_output and projector.output are in window order, and it restores the merge-block order after the merger: vision.image_features is in scatter order. On Qwen3-VL and Qwen3.5 vision.layers[0].input is patch_embeddings plus the resampled pos_embed. On Qwen2-VL vision.layers[0].input is patch_embeddings; on Qwen2.5-VL it is patch_embeddings permuted into window order; on both, positions enter only through the rotary embedding.

HyperCLOVA X Vision V2

A linear projector after the merger

The vision encoder is Qwen2.5-VL's at model.vision_model, its merger inside, so the merger is vision.merger, not the projector. model.projector is one nn.Linear with a bias after the encoder: on Think-32B the encoder is 32 blocks of 1280, the merger folds each 2 × 2 block of patches into one 5120-wide row (out_hidden_size), and the projector maps 5120 to the text model's 5120. vision.image_features is model.projector.output, and the scatter holds exactly:

with model.trace(prompt, images=[image]):
    mask = model.vision.image_token_mask.save()
    fed = model.projector.input.save()           # [image_tokens, out_hidden_size]
    projected = model.projector.output.save()
    features = model.vision.image_features.save()
    first = model.layers[0].input.save()

torch.equal(projected, features)                # True
torch.equal(first[mask], features)              # True

The projector reads the merger's output in token order

The encoder runs its blocks in window order and restores the image tokens' order after the merger, so vision.merger.output is in window order and model.projector.input holds the same rows in the image tokens' order. An image of one 112 × 112 window has the two orders the same. Edit the image at model.projector.input or vision.image_features to address a token by its position; an edit at vision.merger.output lands on the window-ordered row. vision.tower_output is served at the encoder's output, after the merger has read it, so a write there does not reach the text model.

Image tokens

The processor expands each image to t * h * w / 4 copies of <|IMAGE_PAD|> (the config's image_token_id, 128060 on Think-32B), one per merged block of its image_grid_thw row, and vision.image_token_mask marks those. Under generate the encoder runs before the prompt's forward, so in a block the encoder's values come before vision.image_token_mask; vision.image_features is read at the scatter as in a trace.

The block

The vision encoder

The API

Values, by host

Printout

Config

Config

Vision Config

The Notes

The block, in order

The contributions are the post-norms' outputs, times residual_multiplier

A write to a contribution goes through the multiplier

The query scale is attention_multiplier

Logits are the head's output times logits_scaling

token_embeddings is the unscaled lookup

Loading Think-14B

The 32B vision-language checkpoint

Vision notes

The Qwen ViT vision encoder

Every image in one row

The attention runs once per image, so there is no pattern

The merger is inside the vision encoder

Sizes, and no `image_size`

The variants

HyperCLOVA X Vision V2

A linear projector after the merger

The projector reads the merger's output in token order

Image tokens

From the family module

HyperCLOVA X

The block

The vision encoder

The API

Values, by host

Printout

Config

Config

Vision Config

The Notes

The block, in order

The contributions are the post-norms' outputs, times residual_multiplier

A write to a contribution goes through the multiplier

The query scale is attention_multiplier

Logits are the head's output times logits_scaling

token_embeddings is the unscaled lookup

Loading Think-14B

The 32B vision-language checkpoint

Vision notes

The Qwen ViT vision encoder

Every image in one row

The attention runs once per image, so there is no pattern

The merger is inside the vision encoder

Sizes, and no image_size

The variants

HyperCLOVA X Vision V2

A linear projector after the merger

The projector reads the merger's output in token order

Image tokens

From the family module

Sizes, and no `image_size`