nnterp · encyclopedia apertus · nnterp.families.apertus

family apertus

Apertus

Llama's block with its norms named attention_layernorm and feedforward_layernorm, every query and key head RMS-normed before the rotary embedding, and an MLP with no gate whose activation, xIELU, has two parameters learned per block.

Query/key normsLearned activationGrouped-query attention

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card.

embed_tokenstoken_embeddings
→
layers× 32
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model ApertusForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] ApertusDecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn ApertusAttentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp ApertusMLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output

Printout

What print(model) shows on swiss-ai/Apertus-8B-2509: the native modules, standard names and every standard value with its layout.

ApertusForCausalLM(
  (lm_head): Linear(in_features=4096, out_features=131072, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(131072, 4096, padding_idx=3)
  (layers): ModuleList(
    (0-31): 32 x ApertusDecoderLayer(
      (self_attn): ApertusAttention(
        (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
        (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
        (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
        (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        (q_norm): ApertusRMSNorm((128,), eps=1e-05)
        (k_norm): ApertusRMSNorm((128,), eps=1e-05)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): ApertusMLP(
        (up_proj): Linear(in_features=4096, out_features=21504, bias=False)
        (down_proj): Linear(in_features=21504, out_features=4096, bias=False)
        (act_fn): XIELUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm/attention_layernorm): ApertusRMSNorm((4096,), eps=1e-05)
      (post_attention_layernorm/feedforward_layernorm): ApertusRMSNorm((4096,), eps=1e-05)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): ApertusRMSNorm((4096,), eps=1e-05)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers32
model.hidden_size4096
model.num_heads32
model.num_kv_heads8
model.head_dim128
model.qk_head_dim128
model.vocab_size131072
model.intermediate_size21504
architecturesApertusForCausalLM
num_hidden_layers32
hidden_size4096
intermediate_size21504
num_attention_heads32
num_key_value_heads8
vocab_size131072
max_position_embeddings65536
hidden_actxielu
rms_norm_eps1e-05
tie_word_embeddingsFalse
dtypebfloat16
attention_biasFalse

The Notes

03 · what to know before you run

The block, in order

h   = x + self_attn(attention_layernorm(x))
out = h + mlp(feedforward_layernorm(h))
mlp(z) = down_proj(xielu(up_proj(z)))

Llama's two pre-norms under their own names: attention_layernorm feeds the attention and feedforward_layernorm the MLP. nnterp aliases them input_layernorm and post_attention_layernorm, so either name reads the same module. The config carries post_norm: false and qk_norm: true; the modeling code reads neither and builds this block on every checkpoint.

The contributions are the modules' outputs

attention_output is self_attn.output[0] and mlp_output is mlp.output, added to the stream with nothing in between. The identity holds exactly in float32 on the pinned checkpoint:

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(x + attn + mlp, out)

The MLP has no gate, and its activation is learned per block

up_proj widens the normed stream to intermediate_size (21504 on 8B), act_fn applies xIELU, and down_proj narrows it back. xIELU is alpha_p·x² + beta·x for x > 0 and alpha_n·(expm1(min(x, eps)) − x) + beta·x otherwise, with beta 0.5, eps −1e-6 and two scalars learned per block. The parameters mlp.act_fn.alpha_p and mlp.act_fn.alpha_n hold them before a transform: the effective alpha_p is softplus(alpha_p) and the effective alpha_n is beta + softplus(alpha_n). On v1.1-0.5B the effective alpha_p is largest at block 1 (17.75) and smallest at block 12 (0.54), and alpha_n lies between 0.85 and 6.28, so one pre-activation value gives a different activation in each block. act_fn is a module, so mlp.act_fn.input is the pre-activation and mlp.act_fn.output the activation:

mlp = model.layers[1].mlp
with model.trace(prompt):
    pre = mlp.act_fn.input.save()        # equals mlp.up_proj.output
    acts = mlp.act_fn.output.save()      # equals mlp.down_proj.input

Negative pre-activations are not switched off

The negative branch dips below zero and then grows again: as x falls, the output grows like (alpha_n − beta)·|x|. At block 8 of v1.1-0.5B its minimum is −0.14, near x = −0.63; it gives −0.107 at −1, 1.78 at −5 and 10.3 at −20. At that block 93% of the pre-activations are negative, and under 0.4% of the activations lie within 1e-3 of zero. The activations are therefore not sparse: a large negative pre-activation writes its down_proj column with a positive weight, as a large positive one does. Read act_fn.output, not the sign of act_fn.input, to tell whether a neuron fires.

Queries and keys are normed per head, before the rotary

q = rope(q_norm(q_proj(x).view(..., heads, head_dim)))     # one gain, every head
k = rope(k_norm(k_proj(x).view(..., kv_heads, head_dim)))

q_norm and k_norm are RMSNorms over one head's head_dim, one gain vector for every head, applied before the rotary. attention_queries and attention_keys are read after both; at position 0, where the rotation is the identity, they equal q_norm.output and k_norm.output. On v1.1-0.5B the gains reach 81.5 (q_norm) and 87.0 (k_norm) at block 0. Scaling one head's slice of q_proj.output is renormalized away. At block 8 of v1.1-0.5B, doubling head 3's slice moves the logits by 8e-5, doubling head 3 of attention_queries moves them by 1.26, and tripling a key/value head's slice of v_proj.output, which no norm follows, moves them by 23.8.

attn = model.layers[1].self_attn
with model.trace(prompt):
    attn.attention_queries[:, 1] *= 2     # head 1's scores x2

Grouped-query attention

8B has 32 query heads over 8 key/value heads, 70B 64 over 8, both 128 wide. The v1.1 sizes: 16 over 4 at 64 wide (0.5B), 32 over 8 at 64 wide (1.5B), 24 over 8 at 128 wide (4B). attention_keys and attention_values are read before repeat_kv, [batch, kv_heads, seq, head_dim], and an edit to key/value head j reaches query heads 4j to 4j + 3 on 8B.

The default load runs the same attention as eager

The six attention interior values need attn_implementation="eager". There is no softcap, window or sink, and on v1.1-0.5B in float32 the default sdpa load's logits differ from the eager load's by at most 5e-5. The 2509 checkpoints use rope_type llama3 (rope_theta 12,000,000, factor 8 over 8192 original positions, 65536 in all). The v1.1 checkpoints use the plain rotary (rope_theta 500,000, 4096 positions).

Position 0 is <s>, and it is the sink

The tokenizer prepends <s> (id 1). On v1.1-0.5B the layer_output at that position, block 8, has a norm of 19,000 against 140 to 240 at the other positions of a test prompt. A mean over positions is dominated by it unless [:, 1:] is dropped first.

The instruct templates write <s> themselves, so a templated string traced as is starts with two; pass add_special_tokens=False. The 2509 instruct template also writes a system turn with Current date: set to the day it runs, so the same messages tokenize differently from one day to the next:

text = model.tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True)
with model.trace(text, add_special_tokens=False):
    probs = model.next_token_probs.save()

The readout

model.logits is model.lm_head.output: no cap and no scale, and project_on_vocab is lm_head(norm(hidden)). lm_head and embed_tokens are one matrix on v1.1-0.5B and its Instruct, and separate on every other checkpoint listed here. Apertus v1.5 (apertus1p5) is another architecture, a multimodal wrapper, and is not this family.

From the family module

04 · nnterp/families/apertus.py

Apertus (ApertusForCausalLM).

Llama's tree and Llama's block under other norm names: model.{embed_tokens, layers[i].{attention_layernorm, self_attn, feedforward_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. attention_layernorm feeds the attention and feedforward_layernorm the MLP, so they are aliased input_layernorm and post_attention_layernorm. The attention norms its queries and keys per head (q_norm, k_norm) before the rotary embedding. The MLP has no gate: down_proj(xielu(up_proj(x))).