nnterp · encyclopedia exaone4 · nnterp.families.exaone4

family exaone4

EXAONE 4.0

Llama's tree with the norms moved after the sublayers: attention and MLP read the raw residual stream, what reaches the stream is a post-norm's output, queries and keys are RMS-normed per head, and on 32B three sliding-window blocks alternate with one full-attention block that applies no rotary.

Post-norms onlyQuery/key normsSliding windowBlocks without rotaryGrouped-query attention

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card.

embed_tokenstoken_embeddings
→
layers× 30
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Exaone4ForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] Exaone4DecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
⚠︎attention_norm_output scalar Unavailable: a post-norm block: the attention reads the residual stream itself, layer_input

nowhere: the family does not serve it
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module mlp's input
⚠︎mlp_norm_output scalar Unavailable: a post-norm block: the MLP reads the residual stream itself, layer_mid

nowhere: the family does not serve it
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn Exaone4Attentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream: the post-attention norm's output

the sibling module post_attention_layernorm's output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp Exaone4MLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream: the post-feedforward norm's output

the sibling module post_feedforward_layernorm's output

Printout

What print(model) shows on LGAI-EXAONE/EXAONE-4.0-1.2B: the native modules, standard names and every standard value with its layout.

Exaone4ForCausalLM(
  (lm_head): Linear(in_features=2048, out_features=102400, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(102400, 2048, padding_idx=0)
  (layers): ModuleList(
    (0-29): 30 x Exaone4DecoderLayer(
      (self_attn): Exaone4Attention(
        (q_proj): Linear(in_features=2048, out_features=2048, bias=False)
        (k_proj): Linear(in_features=2048, out_features=512, bias=False)
        (v_proj): Linear(in_features=2048, out_features=512, bias=False)
        (o_proj): Linear(in_features=2048, out_features=2048, bias=False)
        (q_norm): Exaone4RMSNorm((64,), eps=1e-05)
        (k_norm): Exaone4RMSNorm((64,), eps=1e-05)
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream: the post-attention norm's output
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Exaone4MLP(
        (gate_proj): Linear(in_features=2048, out_features=4096, bias=False)
        (up_proj): Linear(in_features=2048, out_features=4096, bias=False)
        (down_proj): Linear(in_features=4096, out_features=2048, bias=False)
        (act_fn): SiLUActivation()
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream: the post-feedforward norm's output
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
      )
      (post_attention_layernorm): Exaone4RMSNorm((2048,), eps=1e-05)
      (post_feedforward_layernorm): Exaone4RMSNorm((2048,), eps=1e-05)
      (attention_norm_output): Unavailable: a post-norm block: the attention reads the residual stream itself, layer_input
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output): Unavailable: a post-norm block: the MLP reads the residual stream itself, layer_mid
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): Exaone4RMSNorm((2048,), eps=1e-05)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers30
model.hidden_size2048
model.num_heads32
model.num_kv_heads8
model.head_dim64
model.qk_head_dim64
model.vocab_size102400
model.intermediate_size4096
architecturesExaone4ForCausalLM
num_hidden_layers30
hidden_size2048
intermediate_size4096
num_attention_heads32
num_key_value_heads8
head_dim64
vocab_size102400
max_position_embeddings65536
hidden_actsilu
rms_norm_eps1e-05
tie_word_embeddingsTrue
dtypebfloat16

The Notes

03 · what to know before you run

The block, in order

h   = x + post_attention_layernorm(self_attn(x))
out = h + post_feedforward_layernorm(mlp(h))

Two RMSNorms on the block, both after a sublayer, none before. post_attention_layernorm has Llama's name and the opposite place: on Llama it is the norm before the MLP, here it follows the attention, and nothing norms the MLP's input. There is no input_layernorm, so code that hooks input_layernorm.output to read what the attention sees has nothing to attach to.

The sublayers read the raw stream

self_attn.input is layers[i].input itself and mlp.input is layers[i].input + attention_output, unnormalized. A probe or dictionary trained on a sublayer's input sees the stream at its own scale.

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn_in = model.layers[1].self_attn.input.save()           # equals x
    attn = model.layers[1].self_attn.attention_output.save()
    mlp_in = model.layers[1].mlp.input.save()                  # equals x + attn

The attention's contribution depends almost only on the direction of its input: q_norm and k_norm remove the scale from the queries and keys, the projections are linear with no bias, and post_attention_layernorm removes it again. Doubling what the attention reads changes attention_output by 1e-2 relative at block 2 of 1.2B, 9e-4 at block 15 and 1e-4 at block 28; the residue is the norms' eps (with every eps set to 0 it is exactly 0). The MLP has no such invariance: doubling what it reads changes mlp_output by 40 to 68% at the same blocks.

The contributions are the post-norms' outputs

attention_output and mlp_output are what the block adds, the outputs of post_attention_layernorm and post_feedforward_layernorm, not of the modules. The identity holds exactly, in float32 and bfloat16 alike:

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(x + attn + mlp, out)

Scale the contribution, not the module, and do not count on the post-norm undoing a scale in the first blocks. The modules' outputs are small at the bottom of 1.2B: the MLP's median RMS per token is 4e-4 at block 0 and 2e-3 at block 2, below √eps (3.2e-3, eps 1e-5), so there the post-norm is close to a fixed multiplier rather than a rescaling. Halving mlp.output at block 0 moves layer_output by 42% relative, as much as halving mlp_output does. At block 2 the two move it by 19% and 25%. From block 15 up, halving mlp.output moves it by under 0.4% (2e-5 at block 28) against 13 to 23% for mlp_output. The attention is the same in kind (10% against 45% at block 0, 0.1% against 17% at block 15). Only mlp_output and attention_output scale what the block adds at every depth:

with model.trace(prompt):
    model.layers[1].mlp.mlp_output[:] *= 0.5     # halves what the block adds

with model.trace(prompt):
    model.layers[1].mlp.output[:] *= 0.5         # its effect depends on the depth

The post-norms' gains set the size of each term in a decomposition. Across 1.2B's 30 blocks their means rise on the attention side from 0.02 at block 0 to between 0.3 and 0.36 in the last blocks, and lie between 0.12 and 0.48 on the MLP side, largest in the last blocks.

Queries and keys are normed per head

q_norm and k_norm are RMSNorms over head_dim (64 on 1.2B, 128 on 32B), applied after the heads are split and before the rotary embedding; attention_queries and attention_keys are read after both. Each head is normed on its own, so an edit to one head's slice of q_proj.output reaches no other head, and scaling the slice reaches nothing: at block 15 of 1.2B, tripling head 0's slice changes head 0's queries by 4e-5 and head 1's by 0, while adding head 1's slice to head 0's changes head 0 by 35%. To weaken or strengthen a head's attention, edit attention_queries or attention_keys, which no norm follows. Zeroing a head's queries makes its scores zero and its pattern uniform over the causal prefix:

with model.trace(prompt):
    model.layers[1].self_attn.attention_queries[:, 0] = 0   # head 0: uniform

Which blocks rotate

The forward applies the rotary when config.sliding_window is null or the block is a sliding one, so the two sizes differ:

  • 1.2B: sliding_window is null and layer_types is full_attention on all 30 blocks; every block is full causal attention with the rotary (llama3 scaling, factor 16, rope_theta 1e6).
  • 32B (and 4.0.1-32B): sliding_window is 4096 and layer_types repeats three sliding_attention blocks and one full_attention (LLLG): blocks 3, 7, ..., 63 are full, 16 of 64. The sliding blocks apply the rotary over a 4096-token window; the full blocks apply none (NoPE) and see the whole prefix.

attention_queries and attention_keys are read after the rotary, so on a 32B full block they equal q_norm.output and k_norm.output and carry no position; on every other block they are rotated. On a full NoPE block the only order is the causal mask.

with model.trace(prompt):
    normed = model.layers[1].self_attn.q_norm.output.save()
    queries = model.layers[1].self_attn.attention_queries.save()   # = normed on NoPE

To tell the kinds apart, read model.layers[i].self_attn._module.is_sliding, not sliding_window: every attention module stores the config's window, 4096 on 32B's full blocks too. On a prompt shorter than 4096 tokens the sliding and full masks coincide, and on 32B the two kinds differ only in the rotary.

Grouped-query attention

32 query heads share 8 key/value heads on 1.2B (four each), 40 share 8 on 32B (five each). attention_keys and attention_values are served before repeat_kv, [batch, 8, seq, head_dim], so an edit to key/value head j reaches query heads 4j to 4j + 3 on 1.2B (an edit to head 1's keys at block 8 changes heads 4 to 7 and no other) and 5j to 5j + 4 on 32B. The query scale is head_dim ** -0.5.

Load with eager for the attention interior only

The attention interior needs attn_implementation="eager". The default sdpa load computes the same function: on 1.2B in float32 the two loads' logits differ by at most 5e-5 (the largest logit is about 25). There is no softcap or sink.

The readout is lm_head after a plain RMSNorm

logits is lm_head.output, with no cap or scale after it, and project_on_vocab is lm_head(norm(h)), equal to logits on the last block's layer_output. The final norm's gain is model.norm._module.weight itself, mean 1.79 on 1.2B, between -0.64 and 2.03. On 1.2B lm_head.weight is embed_tokens.weight, so an edit to the embedding is an edit to the unembedding; 32B keeps them separate. token_embeddings is the unscaled lookup and equals layers[0].input.

Every checkpoint is a chat model, and the tokenizer adds no BOS

The released checkpoints are the post-trained models, with a non-reasoning and a reasoning mode; there is no base checkpoint. The tokenizer prepends nothing (add_bos_token is false; [BOS], id 1, exists and the chat template does not use it either), and eos_token is [|endofturn|] (id 361). The template writes [|user|]\n…[|endofturn|]\n[|assistant|]\n and, with add_generation_prompt, an empty <think>\n\n</think>\n\n block; enable_thinking=True ends on an open <think>\n instead. Every assistant turn already in the conversation is rendered with an empty think block, its reasoning dropped. skip_think=False keeps the reasoning of the last message only, when that message is an assistant turn:

ids = model.tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    add_generation_prompt=True, enable_thinking=True, tokenize=False,
)

Checkpoints and kin

EXAONE-4.0.1-32B is a patch release of 32B with the same config. EXAONE 3.5 and EXAONE Deep are model_type exaone, a pre-norm block (ln_1, ln_2) loaded as remote code, with no query/key norms, and K-EXAONE is exaone_moe: neither is this family. EXAONE 4.5 (exaone4_5) is a vision-language wrapper whose text model is this family's: loaded with task="image-text-to-text", its text model takes this page's names, but its vision encoder and projector keep their native names only: there is no model.vision.

From the family module

04 · nnterp/families/exaone4.py

EXAONE 4.0 (Exaone4ForCausalLM).

Llama's names plus a post-norm block, OLMo-2's layout: every sublayer is normed after, x + post_attention_layernorm(attn(x)) and then + post_feedforward_layernorm(mlp(x)). What the block adds is the post-norm's output, not the module's, so the contributions point at the sibling norms. EXAONE 4.0 has only the post-norms: no input_layernorm exists to alias. The attention runs the shared eager forward with per-head q/k norms. A checkpoint with a sliding_window mixes sliding-window layers, which apply the rotary, and full-attention layers, which apply none (NoPE); both reach the interface the same way, so the values are the same on every block.