layer¶
Layer: a decoder block, whose residual stream is layer_output.
module-attribute
¶
PARALLEL = 'a parallel block: the attention and the MLP read the same input, so there is no stream between them'
module-attribute
¶
POST_NORM_ATTENTION = 'a post-norm block: the attention reads the residual stream itself, layer_input'
module-attribute
¶
module-attribute
¶
STATE_SPACE = 'a state-space block: one sublayer, the mixer, so there is no stream between sublayers and no MLP'
module-attribute
¶
ONE_SUBLAYER = 'a one-sublayer block: its norm feeds its one sublayer, so there is no stream between sublayers'
Bases: Standard
A decoder block. Its hidden states are layer_output, whatever the block returns.
| ATTRIBUTE | DESCRIPTION |
|---|---|
self_attn |
The softmax attention, an
TYPE:
|
linear_attn |
The recurrent mixer, a
TYPE:
|
mlp |
The feed-forward, an
TYPE:
|
input_layernorm, |
The block's norms under their aliased names, where the family has them (their meaning varies; see the README).
TYPE:
|
Between layer_input and layer_output the block has four more boundaries,
each the input or output of one of its modules: attention_norm_output,
layer_mid, mlp_norm_output and mlp_input. They are defined on the
pre-norm block (x + attn(norm1(x)), then + mlp(norm2(.))) and keyed on
the aliased norms, so on a block built otherwise the family's subclass points
them at its own modules or marks them unavailable with the reason; a block
without the module reports no <module> module on this block.
Skip this block, handing hidden on as its residual stream.
The block does not run; hidden takes the place of its
layer_output, packed the way the block would have returned it (a
tuple family gets (hidden, None): the second element is the
attention weights nothing downstream reads). Call it inside a trace,
before the block runs.
layer_input(value: Tensor) -> Residual
The residual stream entering this block, before any of its norms.
The block's first argument where the block is called with the stream
(every transformers family), and the stream wherever an engine passes
it otherwise (vLLM's fused blocks take it in two halves). With the
contributions it is the identity the values are defined by:
layer_input + attention_output + mlp_output == layer_output.
In-place edits and assignment reach the model.
attention_norm_output(value: Tensor) -> Residual
The attention's pre-norm output: what the attention is called with, [batch, seq, hidden_size].
input_layernorm.output on a pre-norm block. In-place edits and
assignment reach the model.
layer_mid(value: Tensor) -> Residual
The residual stream between the two sublayers, [batch, seq, hidden_size].
What the MLP's pre-norm is called with: layer_input +
attention_output on a pre-norm block, where it is read as
post_attention_layernorm.input. A parallel block has no such
stream, and marks this unavailable. In-place edits and assignment reach
the model.
mlp_norm_output(value: Tensor) -> Residual
The MLP's pre-norm output, [batch, seq, hidden_size]: post_attention_layernorm.output on a pre-norm block.
The same tensor as mlp_input where nothing sits between the norm
and the MLP. In-place edits and assignment reach the model.