dbrx¶
DBRX on vLLM (vllm.model_executor.models.dbrx).
Transformers' tree: transformer.{wte, blocks[i].{norm_attn_norm.{norm_1,
attn, norm_2}, ffn}, norm_f} and lm_head, aliased as on transformers.
The block is plain: it is called with the positions and the residual stream
and returns the stream. Its norm_attn_norm norms, runs the attention,
adds the residual and norms again, returning (normed, stream); the block
adds the FFN's output to that stream. So the attention module's output is its
contribution, and so is the FFN's, a mixture of experts whose routing is
inside vLLM's fused MoE kernel and has no values here.
module-attribute
¶
RENAME = {'transformer.wte': 'embed_tokens', 'transformer.blocks': 'layers', 'transformer.norm_f': 'norm', 'norm_attn_norm.attn': 'self_attn', 'norm_attn_norm.norm_1': 'input_layernorm', 'norm_attn_norm.norm_2': 'post_attention_layernorm', 'ffn': 'mlp', 'out_proj': 'o_proj'}