nnterp · encyclopedia qwen3 · nnterp.families.qwen3

family qwen3

Qwen3

Llama's block with an RMSNorm on each query and key head before the rotary embedding, so a query's or key's size comes from the norm's gain, not from the input's scale; there is no BOS, and position 0 becomes the attention sink.

Query/key normsGrouped-query attention

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card.

embed_tokenstoken_embeddings
→
layers× 28
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Qwen3ForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] Qwen3DecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn Qwen3Attentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp Qwen3MLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output

Printout

What print(model) shows on Qwen/Qwen3-0.6B: the native modules, standard names and every standard value with its layout.

Qwen3ForCausalLM(
  (lm_head): Linear(in_features=1024, out_features=151936, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(151936, 1024)
  (layers): ModuleList(
    (0-27): 28 x Qwen3DecoderLayer(
      (self_attn): Qwen3Attention(
        (q_proj): Linear(in_features=1024, out_features=2048, bias=False)
        (k_proj): Linear(in_features=1024, out_features=1024, bias=False)
        (v_proj): Linear(in_features=1024, out_features=1024, bias=False)
        (o_proj): Linear(in_features=2048, out_features=1024, bias=False)
        (q_norm): Qwen3RMSNorm((128,), eps=1e-06)
        (k_norm): Qwen3RMSNorm((128,), eps=1e-06)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Qwen3MLP(
        (gate_proj): Linear(in_features=1024, out_features=3072, bias=False)
        (up_proj): Linear(in_features=1024, out_features=3072, bias=False)
        (down_proj): Linear(in_features=3072, out_features=1024, bias=False)
        (act_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Qwen3RMSNorm((1024,), eps=1e-06)
      (post_attention_layernorm): Qwen3RMSNorm((1024,), eps=1e-06)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): Qwen3RMSNorm((1024,), eps=1e-06)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers28
model.hidden_size1024
model.num_heads16
model.num_kv_heads8
model.head_dim128
model.qk_head_dim128
model.vocab_size151936
model.intermediate_size3072
architecturesQwen3ForCausalLM
num_hidden_layers28
hidden_size1024
intermediate_size3072
num_attention_heads16
num_key_value_heads8
head_dim128
vocab_size151936
max_position_embeddings40960
hidden_actsilu
rms_norm_eps1e-06
tie_word_embeddingsTrue
dtypebfloat16
attention_biasFalse
use_sliding_windowFalse
max_window_layers28

The Notes

03 · what to know before you run

Queries and keys are normed per head, before the rotary

q = rope(q_norm(q_proj(x).view(..., num_heads, 128)))     # one gain, every head
k = rope(k_norm(k_proj(x).view(..., num_kv_heads, 128)))
v = v_proj(x).view(..., num_kv_heads, 128)                # no norm
out = o_proj(softmax(q·kᵀ / √128) · v)

q_norm and k_norm are RMSNorms over one head's 128 dimensions, applied to every head with the same gain vector. attention_queries and attention_keys are read at the attention interface, after both the norm and the rotary: on 0.6B they equal rope(q_norm(...)) of q_proj.output exactly. q_norm.output is [batch, seq, num_heads, 128], before the transpose and the rotary. Because each head is rescaled to a fixed RMS, the size of a query or key comes from the gains and the head's direction, not from the input's scale: on 0.6B the k_norm gains reach 96.5 on block 0 and the q_norm gains 11.8.

An edit to q_proj.output is renormalized

A head's slice of q_proj.output or k_proj.output passes through its own norm before it is used, so the edit reaches that head only, and the norm undoes its scale. On 0.6B, doubling head 3's slice of q_proj.output moves the logits by under 1e-4, and tripling a key/value head's slice of k_proj.output by about 1e-4. Tripling the same slice of v_proj.output (no norm) moves them by 5 to 15, depending on the head. Zeroing a slice gives a zero query, and that head attends uniformly over the prefix. To change how sharply a head attends, edit attention_queries (or attention_keys), which come after the norm:

d = model.head_dim
with model.trace(prompt):
    model.layers[1].self_attn.q_proj.output[:, :, 3 * d:4 * d] *= 2  # undone

with model.trace(prompt):
    model.layers[1].self_attn.attention_queries[:, 3] *= 2  # head 3's scores x2

On 0.6B the second edit moves the logits by up to 1.8 and changes only head 3's pattern. An additive edit to a slice survives only as a change of the head's direction.

No projection has a bias

attention_bias is false on every checkpoint listed here, so q_proj, k_proj, v_proj and o_proj are bias-free, as are the MLP's three projections. Zeroing self_attn.input gives zero keys, values and queries and an attention_output of exactly zero.

Grouped-query attention, with 128-wide heads on every size

0.6B has 16 query heads over 8 key/value heads: attention_keys and attention_values are [batch, 8, seq, 128], and an edit to key/value head j reaches query heads 2j and 2j + 1. The other sizes: 16 over 8 (1.7B), 32 over 8 (4B, 8B), 40 over 8 (14B), 64 over 8 (32B). The score scale is 128 ** -0.5. Since head_dim is 128 throughout, num_heads × head_dim is not hidden_size on 0.6B (2048 against 1024), 4B (4096 against 2560) or 32B (8192 against 5120): there q_proj widens the stream, o_proj narrows it back, and attention_head_outputs flattened is wider than layer_output.

The default load computes the same model

Every checkpoint here sets use_sliding_window: false, so config.layer_types is full_attention on every block and self_attn._module.sliding_window is None. With no window and no softcap, the default sdpa load and an eager one agree: on 0.6B in float32 their logits differ by at most 3e-5. attn_implementation="eager" is needed only to read the six attention interior values.

Position 0 is the first token, and it becomes the sink

Qwen's tokenizers set bos_token to None and prepend nothing; the config's bos_token_id, 151643, is <|endoftext|>. Whatever token sits at position 0 becomes an attention sink. On 0.6B, from block 2 to block 26, position 0's layer_output has a norm of 5600 to 6600 on every prompt, 98 to 99% of it (squared) on coordinate 35, pointing the same way on every prompt (cosine 1.000); up to block 15 every other position is under 80. The last block shrinks it to 1650 to 1850. From block 3 on, queries put on average 0.63 to 0.93 of their attention on key 0. 8B (from block 6, coordinate 2276) and 4B-Thinking-2507 (from block 6, coordinate 4) do the same.

A probe, a patch or a logit lens at position 0 reads the sink, and a mean over positions is dominated by it unless it is dropped. Prepending <|endoftext|> moves the sink onto that token and gives the first word its own position; on 0.6B ' Paris' goes from 0.885 to 0.976.

with model.trace("<|endoftext|>The Eiffel Tower is in the city of"):
    resid = model.layers[10].layer_output.save()

resid[0].norm(dim=-1)        # position 0 about 6500, the others under 55

The chat template opens a thinking block

The template writes no system turn of its own, so a templated prompt starts with <|im_start|>user and its position 0, <|im_start|>, takes the sink. On the hybrid checkpoints (0.6B to 32B; the Base ones ship a template with the same ending) add_generation_prompt=True ends the prompt at <|im_start|>assistant\n, and the next token is <think> (151667): on 0.6B with probability 1.000. A next-token metric at the end of that prompt measures the start of a reasoning trace, not the answer. enable_thinking=False appends an empty <think>\n\n</think>\n\n, after which the next token starts the answer:

text = model.tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
with model.trace(text):
    probs = model.next_token_probs.save()

The answer at the start of the assistant's turn is 'Paris', with no leading space; get_first_tokens("Paris", model) returns both 'Paris' and ' Paris'. The template also strips the <think> part from assistant turns before the last user message.

Qwen3-4B-Thinking-2507's template ignores enable_thinking and ends the prompt at <think>\n. Closing the block yourself does not skip the reasoning: the next token still opens it ('Hmm', 'Okay'). Qwen3-4B-Instruct-2507 has no thinking block and answers right after <|im_start|>assistant\n. Qwen3Guard-Gen and Qwen3-Reranker ship task templates of their own: Guard wraps the conversation in a safety-classification instruction, Reranker writes a system turn and a query/document prompt, and both close an empty thinking block. Generation stops on <|im_end|> (151645) or <|endoftext|> (151643).

The readout is the plain projection

model.logits equals model.lm_head.output, and project_on_vocab on the last block's layer_output equals logits exactly. On 0.6B, 1.7B and 4B (the 2507 releases included) the embedding and the unembedding are one matrix, so an edit to embed_tokens.weight edits lm_head; on 8B, 14B and 32B they are separate.

What loads as this family

The configs that say Qwen3ForCausalLM include the dense Qwen3 checkpoints from 0.6B to 32B and their Base models (there is no 32B Base), Qwen3-4B-Instruct-2507 and Thinking-2507, Qwen3-4B-SafeRL, Qwen3Guard-Gen, Qwen3-Reranker, and DeepSeek-R1-0528-Qwen3-8B. rope_theta is 10⁶ everywhere but the 4B-2507 pair (5·10⁶, 262144 positions); the DeepSeek model adds YaRN scaling (factor 4). Qwen3-30B-A3B, 235B-A22B and their 2507 releases are qwen3_moe. Qwen3-Next is qwen3_next, and Qwen3.5's text model is qwen3_5_text, a hybrid with gated DeltaNet blocks.

Sparse autoencoders

Qwen-Scope's residual-stream SAEs are trained on Qwen3-1.7B-Base (Qwen/SAE-Res-Qwen3-1.7B-Base-W32K-L0_50 and -L0_100, 32768 features) and Qwen3-8B-Base (Qwen/SAE-Res-Qwen3-8B-Base-W64K-L0_50 and -L0_100, 65536 features), one per block, TopK with k 50 or 100. Their hook point is resid_post, a forward hook on the decoder block's output, which is model.layers[i].layer_output. Each layer<i>.sae.pt holds W_enc [d_sae, hidden_size], b_enc, W_dec [hidden_size, d_sae] and b_dec in float32.

From the family module

04 · nnterp/families/qwen3.py

Qwen3 (Qwen3ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. q_norm / k_norm sit inside the attention; head_dim is not hidden // heads.