layer¶
Layer and FusedLayer: a vLLM decoder block, called with the residual stream or with its two halves.
Bases: Layer
A vLLM block that is called with the residual stream and returns it.
Two things vary between the blocks that work this way, and a family
states them: STREAM, where the stream sits in the block's call (GPT-2's
block takes it alone, most take the positions first), and the base's
returns_tuple, for a block that returns the stream with something
beside it (Exaone4's and Cohere's return (hidden_states, residual)
with the whole stream first, where a FusedLayer returns its two halves).
Skip this block, handing hidden ([1, tokens, hidden_size]) on as its residual stream.
Safe on a request that runs alone. The engine batches whatever is in flight into one step, and a skipped block has to answer for all of it; see nnsight's vLLM guide before skipping on a shared engine.
Bases: Layer
A vLLM block with the residual add fused into the next norm (Llama, Qwen, Gemma, most families).
It is called forward(positions, hidden_states, residual) and returns
(hidden_states, residual): hidden_states is what the last sublayer
produced and residual the stream before it, so the stream is their
sum, entering and leaving. The first block is called with residual
None and the embeddings as hidden_states.
The sum is computed for the read, so it is nobody's buffer. An edit goes
back as a change to hidden_states (the next norm adds the two), and
a read that edits nothing changes nothing: the difference it hands back is
exactly zero.
class-attribute
instance-attribute
¶
attention_norm_output = Layer.attention_norm_output
class-attribute
instance-attribute
¶
layer_mid = summed('post_attention_layernorm.inputs', description='The residual stream after the attention add: hidden_states + residual, [1, tokens, hidden_size]')
Skip this block, handing hidden ([1, tokens, hidden_size]) on as its residual stream.
Safe on a request that runs alone. The engine batches whatever is in flight into one step, and a skipped block has to answer for all of it; see nnsight's vLLM guide before skipping on a shared engine.
One argument of a call served as (args, kwargs), however the caller passed it; None when absent.
inputs with that argument replaced, where the caller passed it.
summed(key: str, hidden: tuple = (0, 'x'), residual: tuple = (1, 'residual'), description: str | None = None) -> EProperty
The stream at a call that takes it in two halves, hidden + residual, [1, tokens, hidden_size].
A fused block hands its stream to the next norm as norm(hidden_states,
residual), and the norm adds the two before it normalizes (in place):
the stream is their sum, computed for the read. key is the call's
inputs, hidden and residual where the halves sit in it, as
(index, name). An edit goes back as a change to the hidden half, and
a read that edits nothing hands the difference back as exactly zero; a
call without the residual (a block's first norm) is its stream alone.