granitemoe¶
GraniteMoE (GraniteMoeForCausalLM, IBM Granite 3.x MoE).
Granite's block and multipliers (granite.py) with a mixture of experts for
the MLP: block_sparse_moe (GraniteMoeMoE, routed experts only) is
mlp. The block adds h * residual_multiplier for each sublayer, so
attention_output and mlp_output are the module's output times the
multiplier, computed copies that divide on assignment and carry an in-place edit
back through a transform, as on Granite. The mixture keeps no config, so its
Mlp reads the multiplier off its parent block (residual_multiplier). embedding_multiplier scales the
embedding module's output before the first block and logits_scaling divides
the head's output (the family's project_on_vocab, Granite's). Every block has
the mixture, so the config's intermediate_size is the experts' width. The
mixture is a Moe: its router returns (indices, weights, logits), and
router_logits is read where the router computes them. routed_output is the
module's output, unscaled: mlp_output == routed_output * residual_multiplier.
module-attribute
¶
RENAME = {'model.embed_tokens': 'embed_tokens', 'model.layers': 'layers', 'model.norm': 'norm', 'block_sparse_moe': 'mlp'}
module-attribute
¶
Bases: Attention
GraniteMoE's attention: Granite's, the block adds its output times residual_multiplier.
Bases: Moe
GraniteMoE's mixture of experts; the block adds its output times residual_multiplier, read off the block.
class-attribute
instance-attribute
¶
residual_multiplier = property(residual_multiplier)
The block's residual_multiplier, off the parent block's module: a mixture's or a mixer's module keeps no config.
The value an edited scaled copy stands for: edited / multiplier, rewrapped; an unedited copy hands raw back untouched.