doge¶
Doge (DogeForCausalLM, SmallDoge).
Llama's tree with a gated residual: the block scales the stream it carries,
not what it adds. Each sublayer's output is added to the residual times a learned
per-channel gate (input_residual, then post_attention_residual, both
[hidden_size] parameters of the block)::
h = input_residual * x + self_attn(input_layernorm(x))
out = post_attention_residual * h + mlp(post_attention_layernorm(h))
What each sublayer adds is its module's output, unscaled, so the base holds for
attention_output and mlp_output; the contribution identity is
layer_output == post_attention_residual * (input_residual * input + attention_output) + mlp_output,
which is the plain one only where both gates are one (their initial value). The
gates are layers[i]._module.input_residual / post_attention_residual;
nothing is computed or divided for you. The attention runs the shared eager
forward with q/k norms and a dynamic mask: dt_proj of the values, through
A, gives each key a per-head bias, added to the causal mask (and, past
keep_window_size keys, used to drop all but the top keys), so
attention_scores carry it. The MLP is a gated DogeMLP, or the cross-domain
mixture DogeCDMoE under is_moe, which returns (hidden_states, router_logits).
Doge-20M, -40M and -260M load and run. Of the is_moe checkpoints, Doge-40M-MoE
loads and runs; it needs transformers 5.18 or later, whose block takes the first element
of DogeCDMoE's tuple. Where it runs, mlp_output is the mixture's hidden states
(dense MLP plus routed experts); the six mixture values stay unavailable, since
the product-key routing has no router or experts module to read them at.
module-attribute
¶
module-attribute
¶
CANNOT_RUN = 'DogeCDMoE routes by product keys over embedding experts: no router or experts module to read the mixture values at' if Version(transformers.__version__) >= Version('5.18') else 'transformers < 5.18 cannot run DogeCDMoE: the block passes its tuple output to dropout'