mlp¶
Mlp: a feed-forward module, whose contribution is mlp_output.
Bases: Standard
A feed-forward module. Its contribution is mlp_output; its width, intermediate_size, is read off the module.
property
¶
Width of this MLP's hidden layer; on a mixture of experts, one routed expert's.
Read off the module: its experts' intermediate_dim / expert_dim /
intermediate_size / ffn_hidden_size, else its own
intermediate_size / ffn_dim, else the down projection's input
width. A family whose module spells it another way overrides this.
mlp_activation(value: Tensor) -> Neurons
The activation's output, [batch, seq, intermediate_size]: act(gate_proj(x)) on a gated MLP,
act(up_proj(x)) on a plain one. The activation is act_fn, the family aliasing its own
spelling (act, activation_fn, activation, gelu_impl) in RENAME. In-place edits
and assignment reach the model.
mlp_neurons(value: Tensor) -> Neurons
The neurons' values the down projection sums, [batch, seq, intermediate_size].
On a gated MLP act(gate_proj(x)) * up_proj(x); on a plain one (GPT-2) the
same tensor as mlp_activation. The projection is down_proj, the family
aliasing its own spelling (c_proj, dense_4h_to_h, fc2, fc_out,
w2, output_linear) in RENAME. In-place edits and assignment reach the model.