afmoe¶
AFMoE (Arcee's Trinity) on vLLM (vllm.model_executor.models.afmoe).
Llama's names and a fused block with AFMoE's sandwich norms: it takes and
returns the stream as (hidden_states, residual), and each sublayer is
normed before and after. What the block adds is each post norm's output,
as on transformers: post_attention_layernorm follows the attention and
post_mlp_layernorm the MLP, so the contributions point at them (the
pre-MLP norm fuses the attention's add into the residual, keeps its native
name, and its output is mlp.input). The attention norms its queries and
keys per head, rotates them only on its sliding-window blocks, and gates the
attention layer's output with a sigmoid before o_proj, so the head
outputs are ungated, as on transformers. The first num_dense_layers
blocks have a dense MLP, the others a mixture of experts whose output,
shared expert included, is what the block adds; the routing is inside vLLM's
fused MoE kernel and has no values here. The shared expert is the dense MLP's
class, so it is an Mlp too; its mlp_output is unavailable.
module-attribute
¶
module-attribute
¶
ENVOYS = {AfmoeDecoderLayer: Layer, AfmoeAttention: Attention, AfmoeMLP: Mlp, AfmoeMoE: Mlp}
Bases: FusedLayer
The decoder block: forward(positions, hidden_states, residual) -> (hidden_states, residual).