Seed-OSS

SeedOssForCausalLM( (lm_head): Linear(in_features=5120, out_features=155136, bias=False) (generator): Generator( (streamer): Streamer() ) (embed_tokens): Embedding(155136, 5120, padding_idx=1) (layers): ModuleList( (0-63): 64 x SeedOssDecoderLayer( (self_attn): SeedOssAttention( (q_proj): Linear(in_features=5120, out_features=10240, bias=True) (k_proj): Linear(in_features=5120, out_features=1024, bias=True) (v_proj): Linear(in_features=5120, out_features=1024, bias=True) (o_proj): Linear(in_features=10240, out_features=5120, bias=False) (attention_queries) -> Queries [batch num_heads seq qk_head_dim]: The queries entering attention (attention_keys) -> Keys [batch num_kv_heads seq qk_head_dim]: The keys entering attention (attention_values) -> Values [batch num_kv_heads seq head_dim]: The values entering attention (attention_scores) -> Pattern [batch num_heads query key]: The attention scores entering the softmax, masked (attention_premix) -> HeadOutputs [batch seq num_heads head_dim]: Every head's output side by side, as the output projection takes it (attention_output) -> Residual [batch seq hidden_size]: What the attention adds to the residual stream (attention_probabilities) -> Pattern [batch num_heads query key]: The attention pattern the values are mixed with (attention_head_outputs) -> HeadOutputs [batch seq num_heads head_dim]: The per-head outputs before the output projection ) (mlp): SeedOssMLP( (gate_proj): Linear(in_features=5120, out_features=27648, bias=False) (up_proj): Linear(in_features=5120, out_features=27648, bias=False) (down_proj): Linear(in_features=27648, out_features=5120, bias=False) (act_fn): SiLUActivation() (mlp_activation) -> Neurons [batch seq intermediate_size]: The activation function's output (mlp_neurons) -> Neurons [batch seq intermediate_size]: What the down projection receives: act(gate)·up on a gated MLP, the activation itself otherwise (mlp_output) -> Residual [batch seq hidden_size]: What the MLP adds to the residual stream ) (input_layernorm): SeedOssRMSNorm((5120,), eps=1e-06) (post_attention_layernorm): SeedOssRMSNorm((5120,), eps=1e-06) (layer_input) -> Residual [batch seq hidden_size]: The residual stream entering the block (attention_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the attention (layer_mid) -> Residual [batch seq hidden_size]: The residual stream after the attention add (mlp_norm_output) -> Residual [batch seq hidden_size]: The normed stream entering the MLP (mlp_input) -> Residual [batch seq hidden_size]: What the MLP is called with (layer_output) -> Residual [batch seq hidden_size]: The residual stream leaving the block, a tensor even when the block returns a tuple ) ) (norm): SeedOssRMSNorm((5120,), eps=1e-06) (logits) -> Logits [batch seq vocab_size]: The logits the model returns, after anything it does past lm_head (a softcap, a scale) (token_embeddings) -> Residual [batch seq hidden_size]: The embedding module's output; layers[0].input is what enters the first block (final_norm_output) -> Residual [batch seq hidden_size]: The residual stream after the final norm, what lm_head reads (lm_head_output) -> Logits [batch seq vocab_size]: The head's logits before anything the model does past lm_head (a softcap, a scale) (next_token_probs) -> NextTokenProbs [batch vocab_size]: The next-token distribution at the last position; derived, read-only (input_ids) -> Tokens [batch seq]: The token ids the model was called with (attention_mask) -> Tokens [batch seq]: The attention mask the model was called with; zeros are padding (input_size): [batch, seq] of the current call; read-only )

The block, in order

h   = x + self_attn(input_layernorm(x))
out = h + mlp(post_attention_layernorm(h))

Llama's block and Llama's names: two RMSNorms, each before its sublayer, none after. The attention and the MLP each end in a residual dropout inside the module (residual_dropout, 0.1 in every config), after o_proj and after down_proj. A loaded model is in eval mode, where the dropout is the identity.

The contributions are the modules' outputs

attention_output is self_attn.output[0] and mlp_output is mlp.output, added to the stream with nothing in between, and the identity holds exactly in float32 on the pinned checkpoint:

with model.trace(prompt):
    x = model.layers[1].input.save()
    attn = model.layers[1].self_attn.attention_output.save()
    mlp = model.layers[1].mlp.mlp_output.save()
    out = model.layers[1].layer_output.save()

torch.equal(x + attn + mlp, out)   # True

Under model.train() (fine-tuning an adapter through the model) the residual dropout is active. The contributions are read after it, so attention_output differs from o_proj.output, and the identity still holds: on the pinned checkpoint with residual_dropout=0.5, about half of attention_output's entries are zero and x + attn + mlp == out stays exact.

The attention is twice as wide as the stream

The 36B checkpoints have hidden_size 5120 and 80 query heads of head_dim 128, so q_proj maps 5120 to 10240 and o_proj maps 10240 back to 5120. attention_head_outputs is [batch, seq, 80, 128], and the concatenated heads are twice the stream's width: a head's output is not a slice of a stream vector until o_proj has mapped it.

There are 8 key/value heads, so attention_keys and attention_values are [batch, 8, seq, 128] and query head h reads key/value head h // 10: an edit to key/value head j reaches query heads 10j to 10j + 9.

attn = model.layers[1].self_attn
with model.trace(prompt):
    k = attn.attention_keys.save()            # [batch, num_kv_heads, seq, head_dim]
    heads = attn.attention_head_outputs.save()  # [batch, seq, num_heads, head_dim]

Queries, keys and values carry a bias

q_proj, k_proj and v_proj add a bias (attention_bias); o_proj does not (attention_out_bias is false), nor do the three MLP projections (mlp_bias). The bias is added before the rotary embedding, so it is rotated with the rest of the query and key. With the attention's input zeroed, attention_keys is the rotated k_proj.bias: equal to the bias at position 0 and rotated at every other position (checked on the pinned checkpoint after setting its zero bias to ones).

with model.trace(prompt):
    model.layers[1].self_attn.input[:] = 0
    k = model.layers[1].self_attn.attention_keys.save()   # rope(k_proj.bias), not zero

To remove the attention's effect, ablate attention_output itself.

The default load runs the same attention as eager

A default load runs sdpa, and the six attention interior values report unavailable; attn_implementation="eager" serves them. There is no softcap, window or sink, so the two compute the same function. The query scale is head_dim ** -0.5. The rotary is the plain one with rope_theta 10000000, over a max_position_embeddings of 524288.

The readout and the tokenizer

model.logits is model.lm_head.output, so project_on_vocab is lm_head(norm(hidden)), and lm_head has its own weight (tie_word_embeddings is false). The tokenizer prepends no BOS (<seed:bos>, id 0, is not added): position 0 is the text's first token. The tokenizer has 155121 tokens against a vocab_size of 155136, so the last 15 ids get logits but decode to nothing.

Three checkpoints, one shape

Seed-OSS-36B-Base, Seed-OSS-36B-Base-woSyn and Seed-OSS-36B-Instruct have the same config: 64 blocks, the same widths, heads and rotary. -woSyn is the base model pretrained without synthetic instruction data, released for research that needs a base model untouched by it.

The block

The API

Values, by host

Printout

Config

The Notes