SmolLM3

SmolLM3ForCausalLM( (lm_head): Linear(in_features=2048, out_features=128256, bias=False) (generator): Generator( (streamer): Streamer() ) (embed_tokens): Embedding(128256, 2048, padding_idx=128004) (layers): ModuleList( (0-35): 36 x SmolLM3DecoderLayer( (self_attn): SmolLM3Attention( (q_proj): Linear(in_features=2048, out_features=2048, bias=False) (k_proj): Linear(in_features=2048, out_features=512, bias=False) (v_proj): Linear(in_features=2048, out_features=512, bias=False) (o_proj): Linear(in_features=2048, out_features=2048, bias=False) (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection ) (mlp): SmolLM3MLP( (gate_proj): Linear(in_features=2048, out_features=11008, bias=False) (up_proj): Linear(in_features=2048, out_features=11008, bias=False) (down_proj): Linear(in_features=11008, out_features=2048, bias=False) (act_fn): SiLUActivation() (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream ) (input_layernorm): SmolLM3RMSNorm((2048,), eps=1e-06) (post_attention_layernorm): SmolLM3RMSNorm((2048,), eps=1e-06) (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple ) ) (norm): SmolLM3RMSNorm((2048,), eps=1e-06) (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale) (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale) (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only (input_ids) -> Tokens [batch seq]: The token ids the model was called with (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding (input_size): [batch, seq] of the current call; read-only )

The block, in order

h   = x + self_attn(input_layernorm(x))
out = h + mlp(post_attention_layernorm(h))

Llama's block and Llama's names: two RMSNorms, each before its sublayer, none after. attention_output is self_attn.output[0] and mlp_output is mlp.output, and the identity holds exactly, in float32 on the pinned checkpoint and in bfloat16 on SmolLM3-3B (blocks 2 and 3):

with model.trace(prompt):
    x = model.layers[3].input.save()
    attn = model.layers[3].self_attn.attention_output.save()
    mlp = model.layers[3].mlp.mlp_output.save()
    out = model.layers[3].layer_output.save()

torch.equal(x + attn + mlp, out)   # True

Every fourth block has no rotary

config.no_rope_layers holds one flag per block, 1 where the attention applies the rotary and 0 where it does not. On both 3B checkpoints it is [1, 1, 1, 0] repeated, so blocks 3, 7, 11, ..., 35 are NoPE blocks, 9 of 36; SmolLM3-3B-Base leaves the list out of its config and transformers builds the same one from no_rope_layer_interval (4). The pinned checkpoint has two blocks and no_rope_layers [1, 0], so its block 1 is the NoPE one. Read the list before choosing a block:

nope = [i for i, rope in enumerate(model.config.no_rope_layers) if not rope]
# [3, 7, 11, 15, 19, 23, 27, 31, 35] on 3B

layer_types says nothing about this: on 3B every block is full_attention, so the slider's ticks are all one colour and the NoPE blocks are not marked on it.

On a NoPE block the queries and keys are the projections

On a NoPE block attention_queries and attention_keys equal q_proj and k_proj's outputs split into heads, at every position; on a rotary block they agree only at position 0, where the rotation is the identity. Checked on the pinned checkpoint (block 1 against block 0) and on SmolLM3-3B (block 3 against block 2):

attn = model.layers[3].self_attn
with model.trace(prompt):
    q_raw = attn.source.self_q_proj_0.output.save()
    q = attn.attention_queries.save()

b, s, _ = q_raw.shape
torch.equal(q_raw.view(b, s, -1, model.head_dim).transpose(1, 2), q)   # True on a NoPE block

So on a NoPE block a query–key score depends on the two tokens' contents alone: a key patched in from another position scores the same as it did there, and what orders the tokens is the causal mask, plus whatever position information earlier rotary blocks wrote into the stream.

Grouped-query attention

3B has 16 query heads over 4 key/value heads, head_dim 128. attention_keys and attention_values are [batch, 4, seq, 128], and query head h reads key/value head h // 4: an edit to key/value head j reaches query heads 4j to 4j + 3. The rotary blocks use the plain rotary with rope_theta 5000000; the query scale is head_dim ** -0.5.

A sliding window the 3B checkpoints leave off

The config can put a sliding window on some blocks (use_sliding_window, sliding_window); when layer_types is not given, transformers marks the NoPE blocks sliding_attention. Both 3B checkpoints set use_sliding_window to false and sliding_window to null, so every block attends over the whole prefix. The pinned checkpoint turns it on, with a window of 128 on its block 1, its NoPE block, which is why its slider shows a second kind of block.

Load with eager for the interior

A default load runs sdpa, and the six attention interior values report unavailable; attn_implementation="eager" serves them. With no window, softcap or sink on 3B, the two compute the same function.

The readout, the embeddings and the tokenizer

model.logits is model.lm_head.output, and lm_head is tied to embed_tokens (one parameter), so an edit to embed_tokens.weight is an edit to the unembedding. The tokenizer has Llama-3's vocabulary, 128256 ids, and prepends no BOS (bos_token is None): position 0 is the text's first token. " Paris" (12366) and "Paris" (60704) are two whole-word tokens, as on Llama-3.

The block

The API

Values, by host

Printout

Config