flat

flat

Flat: a [tokens, ...] tensor of one vLLM request, served in nnterp's layout as a private copy.

Flat

Flat(*args: Any, heads: str | None = None, batch: bool | Callable[[Envoy], bool] = False, factor: Callable[[Envoy], float] | None = None, **kwargs: Any)

Bases: EProperty

An EProperty over a [tokens, ...] tensor, served as a private [1, tokens, ...] copy.

Takes what EProperty takes (a path for a key, select for one argument of a call). The decorated function receives the batched copy and returns the value; its annotation is the layout. In-place edits are handed back to the model when the block moves on, and an assignment is checked against the rows this request has before it is swapped in. An output that is a tuple serves its first element unless select says another: a norm vLLM calls with the residual beside the stream returns (normed, residual), and the same norm called alone returns the tensor.

PARAMETER	DESCRIPTION
heads	For a [tokens, num_heads * head_dim] tensor, where the head axis goes: "first" serves [1, num_heads, tokens, head_dim] (the queries, keys and values), "last" [1, tokens, heads, head_dim] (the head outputs). The width of a head is the head_size of the module the key names, vLLM's attention layer, or, for a module that has none (the output projection), the host's head_dim. TYPE: str | None DEFAULT: None
batch	Whether the tensor already carries the batch axis, [1, tokens, ...], as it does inside the transformers model vLLM's transformers backend runs: its rows are then tensor[0], served and handed back the same way. A function of the host for a value whose model decides (the root's token_embeddings). TYPE: bool | Callable[[Envoy], bool] DEFAULT: False
factor	For a value that is the tensor times a scalar of the model (Granite's residual_multiplier), a function of the host giving that scalar. The value is served multiplied and is a computed copy: a write goes back divided by the factor, and a read that edits nothing hands the model's tensor back untouched, so dividing never rounds a clean forward. TYPE: Callable[[Envoy], float] | None DEFAULT: None

heads instance-attribute

heads = heads

batch instance-attribute

batch = batch

factor instance-attribute

factor = factor

__call__

__call__(preprocess: Callable) -> 'Flat'

__set__

__set__(obj: Envoy, value: Tensor) -> None

on_transformers_backend

on_transformers_backend(envoy: Envoy) -> bool

Whether envoy's model runs on vLLM's transformers backend, where every activation carries the batch axis.

batched

batched(rows: Tensor, head_dim: int | None = None, heads_first: bool = False) -> Tensor

[tokens, ...] as a private [1, tokens, ...] copy; with head_dim, the last axis split into heads.

[tokens, num_heads * head_dim] (or [tokens, num_heads, head_dim], where a module keeps its heads apart) becomes [1, tokens, num_heads, head_dim], or [1, num_heads, tokens, head_dim] with heads_first.

unbatched

unbatched(value: Tensor, rows: Tensor, name: str, head_dim: int | None = None, heads_first: bool = False) -> Tensor

What batched served, back as the [tokens, ...] the model holds, refusing another shape.

A written value is spliced into the step the engine is running, among other requests' rows. One of the wrong height would reach the next kernel as it is, where the mismatch can end the engine and every request in it, so it is refused here, in the block that wrote it.