nnterp · encyclopedia qwen2 · nnterp.families.qwen2

family qwen2

Qwen2Qwen2.5

Llama's block with a bias on the query, key and value projections; Qwen's tokenizer prepends no BOS, so position 0 is the text's first token and becomes the attention sink.

Biased q, k, vGrouped-query attention

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card.

embed_tokenstoken_embeddings
→
layers× 24
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Qwen2ForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] Qwen2DecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn Qwen2Attentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp Qwen2MLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output

Printout

What print(model) shows on Qwen/Qwen2.5-0.5B: the native modules, standard names and every standard value with its layout.

Qwen2ForCausalLM(
  (lm_head): Linear(in_features=896, out_features=151936, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(151936, 896)
  (layers): ModuleList(
    (0-23): 24 x Qwen2DecoderLayer(
      (self_attn): Qwen2Attention(
        (q_proj): Linear(in_features=896, out_features=896, bias=True)
        (k_proj): Linear(in_features=896, out_features=128, bias=True)
        (v_proj): Linear(in_features=896, out_features=128, bias=True)
        (o_proj): Linear(in_features=896, out_features=896, bias=False)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen2MLP(
        (gate_proj): Linear(in_features=896, out_features=4864, bias=False)
        (up_proj): Linear(in_features=896, out_features=4864, bias=False)
        (down_proj): Linear(in_features=4864, out_features=896, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen2RMSNorm((896,), eps=1e-06)
      (post_attention_layernorm): Qwen2RMSNorm((896,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): Qwen2RMSNorm((896,), eps=1e-06)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers24
model.hidden_size896
model.num_heads14
model.num_kv_heads2
model.head_dim64
model.qk_head_dim64
model.vocab_size151936
model.intermediate_size4864
architecturesQwen2ForCausalLM
num_hidden_layers24
hidden_size896
intermediate_size4864
num_attention_heads14
num_key_value_heads2
vocab_size151936
max_position_embeddings32768
hidden_actsilu
rms_norm_eps1e-06
tie_word_embeddingsTrue
dtypebfloat16
use_sliding_windowFalse
max_window_layers24

The Notes

03 · what to know before you run

Queries, keys and values carry a bias

q_proj, k_proj and v_proj add a bias; o_proj and the three MLP projections do not. The bias is added before the rotary embedding, so it is rotated with the rest of the query and key:

q, k, v = q_proj(x), k_proj(x), v_proj(x)          # each W·x + b
q, k    = rope(q), rope(k)                         # the bias turns with the position
out     = o_proj(softmax(q·kᵀ / √head_dim) · v)    # o_proj: no bias

The biases are large. On 0.5B the query bias has a larger norm than the input-dependent part W_q·x (median over tokens) on all 24 blocks, and the key bias is larger than W_k·x on 8 of them. On block 0 the two biases have norms 235 and 367, against about 20 and 7 for W_q·x and W_k·x. A query–key analysis from W_q and W_k alone leaves out the larger part of every block's query. Zeroing the attention's input does not silence the attention: the queries, keys and values fall back to their biases, and attention_output is o_proj of the value bias at every position.

with model.trace(prompt):
    model.layers[5].self_attn.input[:] = 0
    k = model.layers[5].self_attn.attention_keys.save()      # rope(k_proj.bias), not zero
    attn = model.layers[5].self_attn.attention_output.save() # not zero either

To remove the attention's effect, ablate attention_output itself.

Grouped-query attention

0.5B has 14 query heads over 2 key/value heads, so attention_keys and attention_values are [batch, 2, seq, 64] and an edit to key/value head j reaches query heads 7j to 7j + 6. The other sizes: 12 over 2 (1.5B), 16 over 2 (3B), 28 over 4 (7B), 40 over 8 (14B, 32B), 64 over 8 (72B); head_dim is 64 on 0.5B and 128 on the rest. The score scale is head_dim ** -0.5. Of the Qwen1.5 checkpoints, which load in this family, 0.5B and 7B have no grouping (num_key_value_heads equals num_attention_heads) and 32B and 110B group 5 and 8 query heads.

The sliding-window keys do nothing

Most configs on this page carry sliding_window and max_window_layers, but every checkpoint has use_sliding_window false, set in its config.json or left at the default. Qwen2Config then sets sliding_window to None and every entry of layer_types to full_attention, so every block attends causally over the whole prompt and model.layers[i].self_attn._module.sliding_window is None. With no window and no softcap, the default sdpa load and the eager one compute the same model: on 0.5B in float32 their logits agree to 6e-5. attn_implementation="eager" is needed only to read the six attention interior values.

Position 0 is the first token, and it becomes the sink

The tokenizers prepend nothing (bos_token is None; the config's bos_token_id, 151643, is <|endoftext|>, which no tokenizer call adds). Position 0 holds the text's first token, and whatever that token is, the model turns it into an attention sink. On 0.5B, position 0's layer_output points the same way on every prompt from block 2 to block 20 (cosine 0.999), almost entirely along coordinate 62; from block 3 to block 20 its norm is 1400 to 1750, against under 65 at every other position, and block 21 dissolves it. From block 3 to block 20, a query puts on average 0.45 to 0.96 of its attention on key 0.

From block 2 to block 20 the first word has no representation of its own: a probe, a patch or a logit lens at position 0 reads the sink. A mean over positions (for mean ablation, a steering vector, a normalization) is dominated by position 0 unless it is dropped. Prepending <|endoftext|> moves the sink onto that token and leaves the first word its own position; on 0.5B the next-token prediction barely moves (' Paris' 0.797 without, 0.807 with).

with model.trace("<|endoftext|>The Eiffel Tower is in the city of"):
    resid = model.layers[10].layer_output.save()

resid[0].norm(dim=-1)        # position 0 about 1400, the others under 25

With no BOS to skip, tokenizer(" Paris").input_ids[0] is the word's own token. The DeepSeek-R1-Distill-Qwen tokenizers name a BOS (<｜begin▁of▁sentence｜>), which their chat template writes; a plain tokenizer call does not add it. get_first_tokens tokenizes with add_special_tokens=False and returns 'Paris' and ' Paris' on Qwen2.5 and on the distills.

The chat template writes a system turn

The base checkpoints ship the same ChatML template as the instruct ones (<|im_start|>, <|im_end|>). Given no system message, the template writes one, and its text depends on the checkpoint:

  • You are a helpful assistant (no full stop) on the Qwen2 base models and Qwen1.5;
  • You are a helpful assistant. on the Qwen2.5 base models, Qwen2-Instruct and Qwen2.5-7B-Instruct-1M;
  • You are Qwen, created by Alibaba Cloud. You are a helpful assistant. on Qwen2.5-Instruct and Qwen2.5-Coder;
  • Please reason step by step, and put your final answer within \boxed{}. on Qwen2.5-Math.

QwQ-32B's template writes no system turn and ends the generation prompt at <think>\n. A templated prompt can therefore open with a system turn you did not write, and its position 0 is <|im_start|>, which takes the sink. The instruct configs end generation on <|im_end|> (151645), the base configs on <|endoftext|> (151643). The DeepSeek-R1-Distill-Qwen tokenizers carry DeepSeek's own template instead.

The readout is the plain projection

model.logits equals model.lm_head.output, and project_on_vocab on the last block's layer_output equals logits exactly. The embedding and the unembedding are one matrix on Qwen2 0.5B and 1.5B and Qwen2.5 0.5B, 1.5B and 3B, so an edit to embed_tokens.weight there edits lm_head; from 7B up they are separate. DeepSeek-R1-Distill-Qwen-1.5B is untied although Qwen2.5-Math-1.5B is tied.

What loads as this family

Every checkpoint whose config says Qwen2ForCausalLM: Qwen2 and Qwen2.5 base and instruct, Qwen2.5-Coder and Qwen2.5-Math, Qwen2.5-7B-Instruct-1M, the dense Qwen1.5 models, QwQ-32B, and the DeepSeek-R1-Distill-Qwen models. rope_theta is 10000 on Qwen2.5-Math and the 1.5B and 7B distills, 10⁷ on the 1M model and 10⁶ on the rest. Qwen2-57B-A14B and Qwen1.5-MoE are qwen2_moe.

Sparse autoencoders

andyrdt/saes-qwen2.5-7b-instruct holds BatchTopK SAEs on Qwen2.5-7B-Instruct, trained on the output of blocks 3, 7, 11, 15, 19, 23 and 27 (resid_post_layer_<i>, io: out), which is model.layers[i].layer_output; four per block, k 32 to 256. SAELens lists them as qwen2.5-7b-instruct-andyrdt. Their training skipped each sample's first 8 tokens and dropped activations above 10× the batch's median norm, so the sink at position 0 is outside what they were trained on.

From the family module

04 · nnterp/families/qwen2.py

Qwen2 / Qwen2.5 (Qwen2ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward.

llava-interleave (a llava wrapper) and LLaVA-OneVision put a SigLIP tower at model.vision_tower, which is vision (its post_layernorm over the patches is vision.norm), and the projector model.multi_modal_projector, which is projector. OneVision feeds the projector the image's crops and then unpads its output and adds newline tokens, so vision.image_features, read at the scatter (ImageScatter), is not the projector's output there. Both set vision_feature_layer -1 with strategy "full": the projector reads the last block's layer_output before post_layernorm, so a vision.tower_output write does not reach the text model.