nnterp · encyclopedia phi3 · nnterp.families.phi3

family phi3

Phi-3Phi-3.5Phi-4

Llama's block with two fused projections, qkv_proj holding the queries, keys and values as three slices and gate_up_proj holding the gate and up projections as two halves; the 128k checkpoints rescale the rotary by sequence length, and the 4k ones attend over a 2047-token window.

Sliding window

The block

01 · the residual stream

One block of the stack. Hover any part for its nnterp name; click to pin the card.

embed_tokenstoken_embeddings
→
layers× 32
→
normfinal_norm_output
→
lm_headlm_head_output
→
logitsthe output
i = 0

identity layers[i].input + self_attn.attention_output + mlp.mlp_output == layer_output

The API

02 · support()

Values, by host

⚠︎ marks a value this checkpoint has only under some load: hover it for the condition.

model Phi3ForCausalLMlayoutwhat it isread at
logits Logits [batch seq vocab_size] The logits the model returns, after anything it does past lm_head (a softcap, a scale)

the module's own output
token_embeddings Residual [batch seq hidden_size] The embedding module's output; layers[0].input is what enters the first block

the child module embed_tokens's output
final_norm_output Residual [batch seq hidden_size] The residual stream after the final norm, what lm_head reads

the child module norm's output
lm_head_output Logits [batch seq vocab_size] The head's logits before anything the model does past lm_head (a softcap, a scale)

the child module lm_head's output
next_token_probs NextTokenProbs [batch vocab_size] The next-token distribution at the last position; derived, read-only

the module's own output
input_ids Tokens [batch seq] The token ids the model was called with

the module's own input
attention_mask Tokens [batch seq] The attention mask the model was called with; zeros are padding

the module's own input
input_size scalar [batch, seq] of the current call; read-only

the module's own input
model.layers[i] Phi3DecoderLayerlayoutwhat it isread at
layer_input Residual [batch seq hidden_size] The residual stream entering the block

the module's own input
attention_norm_output Residual [batch seq hidden_size] The normed stream entering the attention

the child module input_layernorm's output
layer_mid Residual [batch seq hidden_size] The residual stream after the attention add

the child module post_attention_layernorm's input
mlp_norm_output Residual [batch seq hidden_size] The normed stream entering the MLP

the child module post_attention_layernorm's output
mlp_input Residual [batch seq hidden_size] What the MLP is called with

the child module mlp's input
layer_output Residual [batch seq hidden_size] The residual stream leaving the block, a tensor even when the block returns a tuple

the module's own output
model.layers[i].self_attn Phi3Attentionlayoutwhat it isread at
⚠︎attention_queries Queries [batch num_heads seq qk_head_dim] The queries entering attention

inside the forward: the arguments of attention_interface_1, element 1
⚠︎attention_keys Keys [batch num_kv_heads seq qk_head_dim] The keys entering attention

inside the forward: the arguments of attention_interface_1, element 2
⚠︎attention_values Values [batch num_kv_heads seq head_dim] The values entering attention

inside the forward: the arguments of attention_interface_1, element 3
⚠︎attention_scores Pattern [batch num_heads query key] The attention scores entering the softmax, masked

inside the forward: the input of nn_functional_softmax_0 inside attention_interface_1
attention_premix HeadOutputs [batch seq num_heads head_dim] Every head's output side by side, as the output projection takes it

the child module o_proj's input
attention_output Residual [batch seq hidden_size] What the attention adds to the residual stream

the module's own output
⚠︎attention_probabilities Pattern [batch num_heads query key] The attention pattern the values are mixed with

inside the forward: the output of nn_functional_dropout_0 inside attention_interface_1
⚠︎attention_head_outputs HeadOutputs [batch seq num_heads head_dim] The per-head outputs before the output projection

inside the forward: the output of attention_interface_1, element 0
model.layers[i].mlp Phi3MLPlayoutwhat it isread at
mlp_activation Neurons [batch seq intermediate_size] The activation function's output

the child module act_fn's output
mlp_neurons Neurons [batch seq intermediate_size] What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise

the child module down_proj's input
mlp_output Residual [batch seq hidden_size] What the MLP adds to the residual stream

the module's own output

Printout

What print(model) shows on microsoft/Phi-3-mini-4k-instruct: the native modules, standard names and every standard value with its layout.

Phi3ForCausalLM(
  (lm_head): Linear(in_features=3072, out_features=32064, bias=False)
  (generator): Generator(
    (streamer): Streamer()
  )
  (embed_tokens): Embedding(32064, 3072, padding_idx=32000)
  (layers): ModuleList(
    (0-31): 32 x Phi3DecoderLayer(
      (self_attn): Phi3Attention(
        (o_proj): Linear(in_features=3072, out_features=3072, bias=False)
        (qkv_proj): Linear(in_features=3072, out_features=9216, bias=False)
        (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention
        (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention
        (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention
        (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked
        (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it
        (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream
        (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with
        (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection
      )
      (mlp): Phi3MLP(
        (gate_up_proj): Linear(in_features=3072, out_features=16384, bias=False)
        (down_proj): Linear(in_features=8192, out_features=3072, bias=False)
        (act_fn/activation_fn): SiLUActivation()
        (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output
        (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise
        (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream
      )
      (input_layernorm): Phi3RMSNorm((3072,), eps=1e-05)
      (post_attention_layernorm): Phi3RMSNorm((3072,), eps=1e-05)
      (resid_attn_dropout): Dropout(p=0.0, inplace=False)
      (resid_mlp_dropout): Dropout(p=0.0, inplace=False)
      (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block
      (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention
      (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add
      (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP
      (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with
      (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple
    )
  )
  (norm): Phi3RMSNorm((3072,), eps=1e-05)
  (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale)
  (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block
  (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads
  (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale)
  (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only
  (input_ids) -> Tokens [batch seq]: The token ids the model was called with
  (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding
  (input_size): [batch, seq] of the current call; read-only
)

Config

model.num_layers32
model.hidden_size3072
model.num_heads32
model.num_kv_heads32
model.head_dim96
model.qk_head_dim96
model.vocab_size32064
model.intermediate_size8192
architecturesPhi3ForCausalLM
num_hidden_layers32
hidden_size3072
intermediate_size8192
num_attention_heads32
num_key_value_heads32
vocab_size32064
max_position_embeddings4096
sliding_window2047
hidden_actsilu
rms_norm_eps1e-05
tie_word_embeddingsFalse
dtypebfloat16
attention_biasFalse

The Notes

03 · what to know before you run

The block, in order

h   = x + self_attn(input_layernorm(x))       # one qkv_proj, then o_proj
out = h + mlp(post_attention_layernorm(h))    # one gate_up_proj, then down_proj

Llama's pre-norm block and Llama's names, with RMSNorms and no biases anywhere in the block. attention_output is o_proj's output and mlp_output is down_proj's; the identity is the plain sum, exact in float32 on the pinned checkpoint. resid_attn_dropout and resid_mlp_dropout sit between each sublayer and its add and are inert in eval. The block returns a bare tensor, so model.layers[i].output is layer_output.

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.testing.assert_close(x + attn + mlp, out)

qkv_proj holds the queries, the keys and the values as three slices

self_attn.qkv_proj is one Linear without a bias, from hidden_size to (num_heads + 2 * num_kv_heads) * head_dim. Its output is [Q | K | V] in three contiguous slices, queries first and num_heads * head_dim wide, then keys and values num_kv_heads * head_dim wide each; within a slice the heads follow one another. On Phi-3-mini that is 3072 + 3072 + 3072 columns, on Phi-4-mini 3072 + 1024 + 1024, on Phi-3-medium and phi-4 5120 + 1280 + 1280. No module holds the queries alone, so there is no q_proj to hook; attention_queries, attention_keys and attention_values are the three split into heads, the first two after the rotary. To read a slice before the rotary, split the projection's output in the same trace:

with model.trace(prompt):
    qkv = model.layers[1].self_attn.source.self_qkv_proj_0.output.save()
    v = model.layers[1].self_attn.attention_values.save()

H, KV, D = model.num_heads, model.num_kv_heads, model.head_dim
q_pre, k_pre, v_flat = qkv.split([H * D, KV * D, KV * D], dim=-1)
b, s, _ = qkv.shape
assert torch.equal(v_flat.view(b, s, KV, D).transpose(1, 2), v)

The weight's rows are in the same order: qkv_proj.weight[:H * D] are the queries' rows. An edit to one head belongs at attention_queries, attention_keys or attention_values, which are per head.

gate_up_proj holds the gate and the up projection as two halves

mlp.gate_up_proj is one Linear from hidden_size to 2 * intermediate_size, without a bias. The forward cuts its output with chunk(2): the first half is the gate, the second the up projection, and down_proj reads up * silu(gate). A neuron's activation is down_proj's input:

with model.trace(prompt):
    gu = model.layers[1].mlp.gate_up_proj.output.save()
    act = model.layers[1].mlp.down_proj.input.save()

gate, up = gu.chunk(2, dim=-1)
assert torch.equal(up * torch.nn.functional.silu(gate), act)

Gate neuron j and up neuron j are columns j and intermediate_size + j of the output.

The 128k checkpoints rescale the rotary by sequence length

Phi-3-mini-128k, Phi-3.5-mini and Phi-4-mini set rope_scaling type longrope: two lists of per-frequency factors, short_factor and long_factor, over an original_max_position_embeddings of 4096. The rotary multiplies every cos and sin by sqrt(1 + ln(32) / ln(4096)) = 1.19, where 32 is 131072 / 4096, at every length, so the rotated dimensions of a query or key have 1.19 times the projection's norm. The rotary picks the factors once per forward pass, from the largest position in it: up to position 4095 it uses short_factor, past it long_factor, at every position of that pass. Position 5 of a 4097-token prompt is therefore rotated differently from position 5 of a 4096-token prompt, so attention_queries and attention_keys at the same position move when the prompt crosses 4096 tokens.

In generation, a step past position 4095 rotates its new query and key with long_factor, while the cached keys keep the short_factor rotation of the pass that computed them. The 4k checkpoints and phi-4 have no scaling: plain rotary at rope_theta 10000 (phi-4: 250000; Phi-4-reasoning and -plus: 500000). Phi-3-medium-128k spells the type su, and transformers cannot read its config: it raises a KeyError for a missing original_max_position_embeddings, so the page lists it greyed out.

Rotary covers whole heads, except on Phi-4-mini

Phi-3, Phi-3.5 and phi-4 rotate all of each head (partial_rotary_factor 1.0): 96 dimensions on mini, 128 on medium and phi-4. Phi-4-mini and Phi-4-mini-reasoning set 0.75, so 96 of each 128-dimensional head turn, as rotate_half over those 96, and the last 32 carry no position. The scale is head_dim ** -0.5 on every checkpoint.

The 4k checkpoints attend over a sliding window

Phi-3-mini-4k and Phi-3-medium-4k set sliding_window 2047, on every block: a query sees itself and the 2046 positions before it, and attention_probabilities are exactly zero further back. A prompt shorter than 2048 tokens is not affected. The 128k checkpoints set a window at or past their 131072 positions (262144 on Phi-3-mini-128k, Phi-3.5-mini and Phi-4-mini, 131072 on Phi-3-medium-128k), which never masks anything, and phi-4 and its reasoning checkpoints set none. The window holds under the default sdpa load and under attn_implementation="eager", which the attention interior needs.

Grouped-query attention on medium, phi-4 and Phi-4-mini

Phi-3-mini and Phi-3.5-mini have 32 key/value heads for 32 query heads, so no grouping. Phi-3-medium and phi-4 have 10 for 40, and Phi-4-mini 8 for 24. attention_keys and attention_values are served before repeat_kv, num_kv_heads wide, so an edit to key/value head j reaches query heads 4j to 4j + 3 on medium and phi-4, 3j to 3j + 2 on Phi-4-mini.

Three tokenizers, none of which adds a BOS

  • Phi-3 and Phi-3.5: Llama's SentencePiece vocabulary, 32011 tokens with the chat tokens, in 32064 embedding rows. <s> (id 1) is the bos_token, and the tokenizer does not prepend it. A word after a space splits as ' Paris' → ['▁', '▁Paris']; encode "Paris" (id 3681, ▁Paris) to get one token.
  • phi-4 and its reasoning checkpoints: 100352 tokens in 100352 rows, a GPT-style byte-level BPE.
  • Phi-4-mini: 200029 tokens in 200064 rows, also byte-level BPE.

Position 0 is the prompt's own first token on every checkpoint.

The checkpoints

The sizes, from the configs:

  • Phi-3-mini-4k and -128k, Phi-3.5-mini (3.8B): 32 blocks, hidden 3072, 32 heads × 96, MLP 8192
  • Phi-3-medium-4k and -128k (14B): 40 blocks, hidden 5120, 40 heads × 128 over 10, MLP 17920
  • Phi-4-mini-instruct and -reasoning (3.8B): 32 blocks, hidden 3072, 24 heads × 128 over 8, MLP 8192, tied embeddings
  • phi-4, Phi-4-reasoning, -reasoning-plus (14B): 40 blocks, hidden 5120, 40 heads × 128 over 10, MLP 17920, a 16384-token context on phi-4 and 32768 on the reasoning checkpoints (rope_theta 500000 there)

Phi-3-small (microsoft/Phi-3-small-8k-instruct, -128k-instruct) is another architecture, model_type phi3small, and so is Phi-4-mini-flash-reasoning (phi4flash): neither loads as this family.

From the family module

04 · nnterp/families/phi3.py

Phi-3 / 3.5 / 4 (Phi3ForCausalLM).

Llama's tree and Llama's block: model.{embed_tokens, layers[i].{input_layernorm, self_attn, post_attention_layernorm, mlp}, norm} and lm_head, the residual added in the block, attention through the shared eager forward. Queries, keys and values are indexed out of one fused qkv_proj.