mlp

mlp

Mlp: a vLLM feed-forward module, whose contribution is its output.

ROUTED module-attribute

ROUTED = "routed experts, inside vLLM's fused MoE kernel: there is no single activation or down projection"

GATED module-attribute

GATED = "a gated MLP on vLLM: act(gate) is never a module's output (gate_up_proj feeds act_fn, an act-and-multiply kernel, whose output is already act(gate) * up), so it is not served; mlp_neurons is the product"

Mlp

Mlp(*args: Any, **kwargs: Any)

Bases: Mlp

A vLLM feed-forward module: its contribution is its output.

The neurons are the down projection's input, as on transformers. On a plain MLP (GPT-2's GELU) that is also the activation's output, and it is read there for both: vLLM builds an activation by name once and shares the one module between every block, so act_fn's own output cannot say which block called it. A gated MLP is different on vLLM: its gate and up projections are one gate_up_proj and its act_fn (SiluAndMul, GeluAndMul) computes act(gate) * up in one kernel, so what it returns is already the neurons and act(gate) alone is never a tensor. There mlp_activation is unavailable (GATED) and mlp_neurons is act_fn's output. A mixture of experts has neither (ROUTED).

mlp_activation

mlp_activation(value: Tensor) -> Neurons

act(up_proj(x)) on a plain MLP, [1, tokens, intermediate_size]: read as the down projection's input, which it is.

mlp_neurons

mlp_neurons(value: Tensor) -> Neurons

The neurons' values the down projection sums, [1, tokens, intermediate_size]: act(gate) * up, which is act_fn's output on vLLM's fused gate, or the activation itself on a plain MLP.

mlp_output

mlp_output(value: Tensor) -> Residual

activation_reason

activation_reason(envoy: Envoy) -> str | None

Why envoy has no activation to serve: a mixture, a gate, or no act_fn or down_proj; else None.

neurons_reason

neurons_reason(envoy: Envoy) -> str | None

Why there is no single down projection on envoy: a mixture, or no down_proj; else None.