gpt_oss¶
GPT-OSS (GptOssForCausalLM).
Llama's tree with a mixture-of-experts MLP that returns (hidden_states,
router_scores), sliding-window layers, and an attention sink: each head
carries a learned logit that joins the softmax as one extra key column and is
dropped afterwards. So the pattern's rows sum to less than one, and the
softmax's input is one key wider than the pattern and shifted by its row max;
the standard attention_scores is therefore read one step earlier, at the
masked scores just before the sink joins them (a binding that exists only
when an attention mask is passed, as it is on every prompt).