olmo_hybrid¶
OLMo-Hybrid (OlmoHybridForCausalLM).
Llama's containers over two block classes, chosen by config.layer_types:
OlmoHybridLinearAttentionDecoderLayer(linear_attention) is pre-norm, as Llama:input_layernorm -> linear_attn(a gated DeltaNet, seennterp.LinearAttention), thenpost_attention_layernorm -> mlp, each added to the stream as the module returns it.OlmoHybridAttentionDecoderLayer(full_attention) is OLMo-3's post-norm sandwich:self_attn -> post_attention_layernormandmlp -> post_feedforward_layernorm, the post-norms' outputs added to the stream; it has noinput_layernorm, so the block input enters the attention.
Each block has self_attn or linear_attn, never both. The attention runs
the shared eager forward (query and key norms before it; no rotary embedding
when the checkpoint sets no rope_theta), so its contribution is OLMo-3's
override at the post-attention norm. Both block classes hold one MLP class,
OlmoHybridMLP, whose contribution is its own output on a linear block and the
post-feedforward norm's output on an attention block, so mlp_output is a
EProperty whose key is chosen by the parent block's layer type.
The DeltaNet calls transformers' torch_chunk_gated_delta_rule /
torch_recurrent_gated_delta_rule as Qwen3-Next does, so the base
LinearAttention holds; with linear_allow_neg_eigval the betas are doubled
before the kernel and lie in (0, 2).
module-attribute
¶
module-attribute
¶
ENVOYS = {OlmoHybridAttentionDecoderLayer: Layer, OlmoHybridLinearAttentionDecoderLayer: Layer, OlmoHybridAttention: Attention, OlmoHybridGatedDeltaNet: LinearAttention, OlmoHybridMLP: Mlp}
Whether this is an attention block: post-norms only, the block input entering the attention.
The key of this MLP's contribution: the post-feedforward norm on an attention block, the MLP's own output on a linear one.