gpt_neox_japanese¶
GPT-NeoX-Japanese (GPTNeoXJapaneseForCausalLM).
gpt_neox_japanese.{embed_in, layers[i].{input_layernorm, post_attention_layernorm,
attention, mlp}, final_layer_norm, rotary_emb} and embed_out, the head. GPT-NeoX's
names, but a sequential block, not a parallel one: post_attention_layernorm
normalizes the stream after the attention has been added. The block adds each
sublayer through bias_dropout_add(x, bias, residual), residual + dropout(x +
bias), and returns (hidden_states, attn_weights). The attention returns
(attn_output, attn_weights, dense_bias): its output projection has no bias, and
on the last block only a separate dense_bias is handed to the block to add, so
what the attention adds is its output plus that bias: a sum served as
attention_output, with an edit carried back to the module's output minus the
bias. The attention does its own arithmetic in _attn whatever
attn_implementation says (the model has no other): a baddbmm scaled by
1/sqrt(head_dim), the mask added, softmax, attention_dropout, a cast to the
values' dtype, @ v; the interior needs no eager load.
module-attribute
¶
RENAME = {'gpt_neox_japanese.embed_in': 'embed_tokens', 'gpt_neox_japanese.layers': 'layers', 'gpt_neox_japanese.final_layer_norm': 'norm', 'embed_out': 'lm_head', 'attention': 'self_attn', 'dense': 'o_proj', 'act': 'act_fn', 'dense_4h_to_h': 'down_proj'}
module-attribute
¶
ENVOYS = {GPTNeoXJapaneseLayer: Layer, GPTNeoXJapaneseAttention: Attention, GPTNeoXJapaneseMLP: Mlp}
intermediate_size(model: StandardizedTransformer) -> int
The MLP width is hidden_size * intermediate_multiple_size.