attention¶
Attention: a vLLM attention module, its contribution, what crosses the engine's attention layer, and the pattern.
module-attribute
¶
DECODE_STEP = "recomputed from this step's queries and keys, and on a decode step the earlier tokens' keys are in vLLM's cache, which only its kernel reads; read it on the prefill"
Bases: Attention
A vLLM attention module: its contribution, what goes into and comes out of the engine's attention layer, and the pattern.
The module projects (and, on a rotary family, rotates) its queries, keys
and values and calls vLLM's attention layer on them, self.attn(q, k,
v), which returns the per-head outputs the output projection then
mixes. Those four are the attn child's inputs and output, each
[tokens, num_heads * head_dim], served with the heads split out by the
module's own head_dim. A family whose attention module names that
child, or its head width, another way points the values at it.
They are this step's rows: on a decode step the keys and values are the new token's alone, and the earlier ones are in the engine's cache, which the kernel reads for itself.
The scores and the pattern are computed inside the kernel, which serves
nothing between its inputs and its output, so here they are recomputed
from the queries and keys, with the layer's own scale, softcap and
window. That makes them read-only (the kernel never takes a pattern: edit
the queries or keys to change what a head attends to, or its head outputs
to change what it wrote) and there on the prefill only, where a step
holds every key; a decode step raises Unavailable (on_decode_step).
class-attribute
instance-attribute
¶
attention_scores = DerivedEProperty(scores, description='The scores entering the softmax, [1, num_heads, query, key], recomputed from the queries and keys; prefill only, read-only', unavailable=on_decode_step)
class-attribute
instance-attribute
¶
attention_probabilities = DerivedEProperty(probabilities, description='The attention pattern, [1, num_heads, query, key], recomputed from the queries and keys; prefill only, read-only', unavailable=on_decode_step)
attention_queries(value: Tensor) -> Queries
The queries the attention layer receives, after the projection, any query norm and the rotary embedding.
attention_keys(value: Tensor) -> Keys
This step's keys as the attention layer receives them; num_kv_heads wide under grouped-query attention.
attention_values(value: Tensor) -> Values
This step's values as the attention layer receives them; num_kv_heads wide under grouped-query attention.
attention_head_outputs(value: Tensor) -> HeadOutputs
Each head's output as the attention layer returns it, before the output projection mixes them.
attention_premix(value: Tensor) -> HeadOutputs
The output projection's input split by head, after any gate the attention applies; the head outputs where there is none.
DECODE_STEP when a read made now would be served a decode step of the request, else None.
A request's first visit to its attention layer is the prefill, every
prompt token at once, and each later one a decode step of one token. The
worker knows which visit a read would ask for before anything is read,
so the value refuses there, by name. Outside a trace there is no step to
ask about and the answer is the prefill's: support lists it available.
scores(self: 'Attention') -> Pattern
The scaled, masked scores the kernel's softmax runs on, [1, num_heads, query, key], recomputed.
queries @ keys.T * scale, softcapped where the layer softcaps, plus
the ALiBi bias where the layer has slopes, with the keys a query may not
see (later tokens, and tokens past the layer's sliding window) at
-inf. The scale, the softcap, the slopes and the window are the ones
vLLM's attention layer was built with. The ALiBi bias is each head's
slope times how far behind the query the key is; transformers' families
write the same bias from another origin, which moves a query's scores by
one constant and leaves the pattern alone.