Org
Quirks
Vision encoders
Trinity NanoTrinity MiniTrinity Large
A sandwich block whose attention gates its heads with a sigmoid and applies rotary only on its sliding-window blocks, and whose MLP is dense on the first blocks and a mixture with a shared expert after them, its routed experts picked by sigmoid scores plus a selection bias.
Apertus
Llama's block with its norms named attention_layernorm and feedforward_layernorm, every query and key head RMS-normed before the rotary embedding, and an MLP with no gate whose activation, xIELU, has two parameters learned per block.
Arcee AFM
Llama's block whose MLP has no gate: up_proj, a squared ReLU and down_proj, so a neuron is exactly zero wherever its pre-activation is negative.
Bamba
Llama's pre-norm block whose mixer is a Mamba-2 state-space scan on all but three blocks and, on those three, attention that rotates half of each head; the block returns a tuple.
BitNet b1.58
Llama's tree whose projections run ternary weights on 8-bit inputs, whose sublayers each RMS-norm the tensor entering their output projection (attn_sub_norm, ffn_sub_norm), and whose MLP gates with a squared ReLU.
BLOOMBLOOMZ
Each sublayer adds the residual inside its own module, so the contributions are the tensors before that add; a LayerNorm sits between the embedding and block 0.
CodeGen
GPT-J's parallel block and adjacent-pair rotary, with the queries, values and keys cut from one projection laid out as four groups of [q | v | k].
Command RCommand R+
A parallel block whose one LayerNorm, without a bias, feeds both the attention and the MLP; rotary turns adjacent pairs of dimensions, Command R+ norms queries and keys per head, and the logits are the head's output times logit_scale.
Command R7BCommand A
Command R's parallel block on one bias-free LayerNorm, with three sliding-window blocks to each full one; only the sliding blocks apply rotary, and the logits are the head's output times logit_scale.
DBRX
A pre-norm block whose attention sits with both norms inside norm_attn_norm, queries, keys and values from one clamped Wqkv projection, and a mixture of 16 experts, 4 per token, on every block.
DeepSeek-V2DeepSeek-Coder-V2
Latent attention with queries and keys wider than the values and one rotary key shared by every head, and a softmax-routed mixture with two shared experts on every block but the first.
DeepSeek-V3DeepSeek-R1Moonlight
Latent attention with queries and keys wider than the values, and a mixture with a shared expert, whose router chooses experts by sigmoid scores plus a selection bias and weights them without it.
DeepSeek-V3.2
Latent attention over only the 2048 keys a lightning indexer scores best for each query, and a mixture with a shared expert that chooses experts by sigmoid score plus a selection bias and weights them without the bias.
DeepSeek-V4
Four parallel residual streams, which a learned matrix mixes around each sublayer while the sublayer reads one weighted sum of them and writes into all four; multi-query attention over a sliding window plus compressed keys; and a mixture of experts on every block.
Doge
Llama's pre-norm block whose residual stream is multiplied by a learned per-channel gate before each add, and whose attention adds a learned bias per key, computed from the values, to the scores.
dots.llm1
Llama's pre-norm block with each query and key head RMS-normed, a dense MLP on block 0, and on every block after it a mixture whose experts are chosen by a sigmoid plus a selection bias that the weights leave out, beside a shared expert every token runs through.
ERNIE 4.5
Llama's pre-norm block whose rotary turns adjacent pairs of query and key dimensions, (2i, 2i + 1), and whose grouped query heads are together wider than the residual stream.
ERNIE 4.5 MoE
Llama's pre-norm block with rotary on adjacent pairs of dimensions, a dense MLP on the first blocks and a mixture after them, whose experts are chosen by a softmax plus a correction bias that the weights leave out, renormalized to sum to one.
EXAONE 4.0
Llama's tree with the norms moved after the sublayers: attention and MLP read the raw residual stream, what reaches the stream is a post-norm's output, queries and keys are RMS-normed per head, and on 32B three sliding-window blocks alternate with one full-attention block that applies no rotary.
Falcon
A parallel block on every checkpoint but Falcon-RW: the block adds the attention's output into the MLP's output tensor in place, so mlp_output is a copy taken as the MLP returns.
Falcon-H1
Every block runs a Mamba-2 mixer and attention side by side on one normed input and adds both, each times its own multiplier, then a gated MLP; the embeddings and the logits carry multipliers too.
Falcon-MambaFalcon3-Mamba
Mamba's block, one norm and a selective-scan mixer, with no attention and no MLP; inside the mixer, weightless RMS norms sit on B, C and the low-rank step input before the scan.
FlexOlmo
OLMo-2's post-norm block with a mixture of experts for the MLP: the released checkpoints route every token to every expert, weighted by a softmax over all of them, and the stream receives the post-norm of the weighted sum.
GemmaPaliGemma
Llama's pre-norm block with a GeGLU MLP: the embeddings are multiplied by √hidden_size before block 0, every RMSNorm multiplies by 1 + weight, and the head is the embedding matrix.
Gemma 2PaliGemma 2
Llama's module tree with a sandwich block: every sublayer is normed on the way in and on the way out, so what reaches the residual stream is a post-norm's output, and the logits are softcapped.
Gemma 3
Llama's module tree with a sandwich block, so what reaches the residual stream is a post-norm's output; queries and keys are normed per head before the rotary, and five sliding-window blocks precede each full one.
Gemma 4
A sandwich block with normed queries, keys and values, whose sum a learned per-block scalar multiplies; on E2B and E4B per-layer embeddings add a third term and the last blocks borrow keys and values from earlier ones.
Gemma 4 unified
A sandwich block whose sum is multiplied by a learned per-block scalar, with normed queries, keys and values, and full blocks that take their values from the key projection.
GLM-4GLM-Edge
Llama's block with a fused gate and up projection and a rotary embedding that turns adjacent pairs of dimensions; on GLM-4-9B the rotary covers only the first half of each head, and the queries, keys and values carry a bias.
GLM-4-0414GLM-Z1
GLM's block with a second norm after each sublayer, so the stream receives the post-norms' outputs; post_attention_layernorm keeps Llama's meaning, the MLP's input norm.
GLM-4.5GLM-4.6GLM-4.7
Llama's pre-norm block with rotary on half of each head and, after the first dense blocks, a mixture whose sigmoid router picks 8 experts with a selection bias, beside one shared expert.
GLM-4.7-Flash
Latent attention, whose keys and values are expanded for every head from one latent per token, and after a dense first block a mixture: 4 of 64 experts chosen by a sigmoid plus a selection bias, beside one shared expert.
GLM-5
Latent attention over only the keys an indexer selects for each query, a selection each block returns beside the stream for the next block to reuse, and a sigmoid-routed mixture with a shared expert after three dense blocks.
GPT-2
Llama's sequential block with LayerNorm in place of RMSNorm, a learned position embedding added to the token embeddings before block 0, and queries, keys and values cut from one fused c_attn projection.
StarCoderSantaCoder
GPT-2's sequential block with multi-query attention: one key head and one value head serve every query head, and the queries, keys and values are cut from one fused c_attn projection.
GPT-NeoTinyStories
GPT-2's sequential block and learned position embedding; the attention sits one module down, inside an attn wrapper, computes its own scores without scaling them, and on every other block masks keys outside a local window.
GPT-NeoX
Pythia's block is parallel: attention and MLP each read their own LayerNorm of the block input, and the block adds both at once, so a block's MLP never sees that block's attention.
GPT-NeoX-Japanese
GPT-NeoX's module names on a sequential block, where the last block's attention returns a separate bias that the block adds and attention_output includes.
GPT-OSS
Llama's pre-norm block with a mixture of experts on every block, sliding-window and full-attention blocks in turn, and a learned sink logit per head in the softmax, so a pattern row sums to less than one.
GPT-J
A parallel block with one LayerNorm: ln_1's output is the input of both the attention and the MLP, the block adds both to the stream at once, and rotary turns only the first rotary_dim dimensions of each head, in adjacent pairs.
Granite
Llama's block with four config scalars around it: the embeddings are multiplied before block 0, each sublayer's output is multiplied by residual_multiplier before it is added, the attention scales its scores by attention_multiplier, and the logits are the head's output divided by logits_scaling.
Granite Swash
Granite's block and multipliers with sliding-window attention on most blocks and a learned sink per head that scales each head's output after the softmax: the pattern's rows sum to one, and each head writes only a share of what its pattern mixes.
GraniteMoE
Granite's block with a mixture of experts for its MLP: the embeddings are multiplied before block 0, each sublayer's output is multiplied by residual_multiplier before it is added, and the logits are divided by logits_scaling, so mlp_output is the mixture's output times 0.22.
Granite Swash MoE
Granite Swash's sliding-window attention with its sink in GraniteMoE's scaled block, where a shared expert runs beside the mixture of experts and mlp_output carries the mixture's term alone.
Granite 4.0
Granite's scaled block with a Mamba-2 mixer on most blocks and attention without rotary on a few (rotary attention on every block of the non-H models), and a feed-forward of a shared MLP plus, on the mixture checkpoints, routed experts, added as one scaled sum.
GraniteMoE-Shared
GraniteMoE's block with a dense shared expert beside the mixture: the block adds the sum of the two times residual_multiplier, and mlp_output is that scaled sum, while mlp.output is the mixture's output alone.
Helium
Llama's block, with one key/value head per query head and a rotary embedding that turns adjacent pairs of dimensions (2i, 2i + 1) rather than i with i + head_dim / 2.
Hunyuan
Llama's block with each query and key head RMS-normed after the rotary embedding (query_layernorm, key_layernorm), so a head's query and key lengths are set by the norms' gains.
Hunyuan-A13B
Hunyuan's block, query and key heads RMS-normed after the rotary embedding, with a mixture on every block: a shared expert that runs first, plus 8 of 64 routed experts whose weights are renormalized to sum to one.
HyperCLOVA X
Llama's block with a norm after each sublayer (post_norm1, post_norm2) where use_post_norm is set, and four config multipliers: on the embeddings, on each sublayer's contribution, on the attention scores (in place of head_dim ** -0.5) and on the logits.
JambaJamba 1.5–1.7Jamba2
Llama's pre-norm block with one of two mixers, a Mamba-1 selective scan with RMS norms on its step, B and C on most blocks and attention without rotary on one block in eight or fourteen, and, on every other block of the larger checkpoints, a mixture of experts in place of the MLP.
JetMoE
Llama's pre-norm block in which both sublayers are mixtures of experts: besides the MLP's, a router picks attention experts per token, each with its own query and output projections over shared keys and values.
Kimi K2
DeepSeek-V3's architecture: latent attention with queries and keys wider than the values, and a mixture of 384 experts plus a shared expert on every block but the first.
Kimi-Linear
Llama's pre-norm block with two kinds of mixer, Kimi Delta Attention on three blocks in four and latent attention without rotary on the rest, and a mixture of experts after a dense first block.
Laguna XSLaguna SLaguna M
Llama's pre-norm block with a softplus gate on each head's output before o_proj, and after the first blocks a sigmoid-routed mixture of experts beside a shared expert; on XS and S the blocks differ in query heads.
Llama
The reference block: one RMSNorm before each sublayer and none after, so what reaches the residual stream is the attention's and the MLP's own output, with rotary positions applied to queries and keys inside the attention.
Llama 4
Llama's pre-norm block with a mixture of experts and a shared expert as the MLP (every block on Scout, every other block on Maverick), routed by a sigmoid of the top-1 logit; every fourth block attends over the whole context without rotary, and the others attend within fixed chunks.
Mamba
No attention and no MLP: each block is one norm and a selective-scan mixer with a state per channel, and reading inside the mixer needs mamba_ssm's CUDA kernels routed to transformers' torch functions.
Mamba-2
No attention and no MLP: each block is one norm and a Mamba-2 (SSD) mixer whose heads each keep a state matrix, with one scalar decay per head and token.
MiMo-V2-Flash
Llama's pre-norm block with queries and keys wider than values, sliding-window blocks that have a learned sink and twice the full blocks' key/value heads, and a mixture of experts chosen by sigmoid scores plus a bias.
MiniMax-M2MiniMax-M2.1MiniMax-M2.5MiniMax-M2.7
Llama's pre-norm block with one RMSNorm over all query heads and one over all key heads, and a mixture of experts on every block, chosen by sigmoid scores plus a bias and weighted by the sigmoids without it.
Ministral
Llama's pre-norm block with grouped-query attention, each block attending either over every earlier position or over a sliding window, as config.layer_types says.
Ministral 3
Llama's pre-norm block whose queries are multiplied, after the rotary embedding, by a factor that grows with the position past 16384 tokens; every checkpoint is a vision-language model on the Pixtral encoder.
MistralMistral NemoMistral Small
Llama's pre-norm block with grouped-query attention; on Mistral-7B-v0.1 every block attends over a sliding window of the last 4096 positions, and from Nemo on the attention is narrower than the stream.
Mixtral
Mistral's pre-norm block with a mixture of 8 SwiGLU experts in place of the MLP: each token goes to 2 of them, whose softmax weights are renormalized to sum to one, and no shared expert.
MPT
The block adds the attention to the stream, but the MLP adds the residual inside its own module, so mlp_output is the tensor before that add; no projection or norm has a bias.
Nemotron-4Minitron
Llama's tree with Nemotron-4's block: the norms are LayerNorms with a bias whose gain is stored minus one, the MLP has no gate and squares a ReLU, and rotary positions turn half of each query and key head.
Nemotron-HNemotron Nano 2Nemotron 3
Every block is one RMSNorm and one sublayer: a Mamba-2 mixer on about half the blocks, attention without rotary on a few, and on the rest a squared-ReLU MLP or, on Nemotron 3's mixture checkpoints, a mixture of experts.
OLMoOLMo 0424OLMo 0724
Llama's pre-norm block with LayerNorm in place of RMSNorm, and every norm without weight or bias, so a sublayer reads the stream standardized to mean 0 and variance 1, and the 0424 and 0724 base checkpoints clamp queries, keys and values to ±8 inside the attention.
OLMo 2
Llama's tree with the norms moved after the sublayers: attention and MLP read the raw residual stream, what reaches the stream is a post-norm's output, and the queries and keys are RMS-normed across all heads before the rotary embedding.
OLMo 3OLMo 3.1
OLMo-2's post-norm block: attention and MLP read the raw stream and the stream receives the post-norms' outputs; three blocks in four attend over a 4096-token window with the plain rotary, and the fourth attends over the whole prefix with a YaRN-scaled rotary.
OLMo-Hybrid
Two block classes that norm in opposite places: three blocks in four are pre-norm, with a gated DeltaNet mixer, and every fourth is OLMo-3's post-norm block, whose attention (without rotary on most checkpoints) and MLP read the raw stream and have their outputs normed before the stream receives them.
OLMoE
Llama's pre-norm block with queries and keys RMS-normed across all heads, and a mixture of 64 experts on every block that weights a token's 8 experts by their softmax entries without renormalizing them, so the 8 weights sum to less than one.
OPT
A LayerNorm block with a learned position embedding offset by 2 and no MLP module: fc1, the activation and fc2 sit on the block itself, and fc2.output is what the MLP path adds to the stream.
Persimmon
Llama's sequential block with LayerNorms; the attention cuts queries, keys and values per head from one fused projection and norms each query and key head before the rotary.
Phi-1Phi-1.5Phi-2
A parallel block, in which one LayerNorm feeds both the attention and the MLP and the block adds their outputs at once, with a bias on every attention projection and on lm_head, and rotary on only 32 dimensions of each head.
Phi-3Phi-3.5Phi-4
Llama's block with two fused projections, qkv_proj holding the queries, keys and values as three slices and gate_up_proj holding the gate and up projections as two halves; the 128k checkpoints rescale the rotary by sequence length, and the 4k ones attend over a 2047-token window.
Phi-3.5-MoE
Llama's tree with LayerNorms and biased projections, and on every block a mixture of 16 experts whose router, sparsemixer, picks two experts in turn and gives each a weight of exactly 1 unless another logit is within 2% of that expert's, so a token's two weights sum to up to 2.
Qwen2Qwen2.5
Llama's block with a bias on the query, key and value projections; Qwen's tokenizer prepends no BOS, so position 0 is the text's first token and becomes the attention sink.
Qwen2.5-VL
Qwen2's block, biased q, k and v, with multimodal rotary embeddings: three position streams (time, row, column) folded into one cos/sin before block 0, so an image token's position is its row and column in the image.
Qwen1.5-MoEQwen2-MoE
Qwen2's attention, with biased q, k and v, and a mixture of experts on every block: the router's top-k softmax weights are not renormalized, and a shared expert is added through a sigmoid gate of its own.
Qwen2-VL
Qwen2's block, biased q, k and v, with multimodal rotary embeddings: three position streams (time, row, column) folded into one cos/sin before block 0, so an image token's position is its row and column in the image.
Qwen3
Llama's block with an RMSNorm on each query and key head before the rotary embedding, so a query's or key's size comes from the norm's gain, not from the input's scale; there is no BOS, and position 0 becomes the attention sink.
Qwen3.5-MoEQwen3.6-MoE
Qwen3.5's hybrid block, a gated DeltaNet mixer on three blocks in four and output-gated attention on the fourth, with a mixture of experts on every block whose shared expert is scaled by a sigmoid gate per token.
Qwen3.5Qwen3.6
Llama's pre-norm block with a gated DeltaNet mixer on three blocks in four and, on the fourth, attention whose q_proj also yields a sigmoid gate on the heads' output; every RMSNorm but the mixer's own multiplies by 1 + weight.
Qwen3-MoE
Qwen3's attention, an RMSNorm on each query and key head, with a mixture of experts on every block in place of the MLP: 8 experts per token, their softmax weights renormalized to sum to one, and no shared expert.
Qwen3-NextQwen3-Coder-Next
Llama's pre-norm block with a gated DeltaNet mixer on three blocks in four, output-gated attention on the fourth, and a mixture of 512 experts whose shared expert is scaled by a sigmoid gate per token; the mixer fuses its projections per key head.
Qwen3-VL-MoE
Qwen3's attention, queries and keys RMS-normed per head, under a multimodal rotary embedding, and a mixture of 128 experts on every block; on an image prompt the text model adds vision features at the image positions again after blocks 0, 1 and 2, outside the blocks.
Qwen3-VL
Qwen3's block, queries and keys RMS-normed per head, under a multimodal rotary embedding; on an image prompt the text model adds vision features at the image positions again after blocks 0, 1 and 2, outside the blocks.
Seed-OSS
Llama's block with a bias on the query, key and value projections, and query heads that together are twice as wide as the stream.
SmolLM3
Llama's block in which every fourth attention applies no rotary embedding (NoPE): there queries and keys are the projections as split, and only the causal mask orders the tokens.
Solar Open
Llama's pre-norm block with a mixture of experts on every block: a sigmoid router whose correction bias affects only which experts are chosen, and a shared expert that runs on every token.
StableLM 2StableLM-3B
Llama's block with LayerNorms and rotary on the first quarter of each head; StableLM-2-12B makes it parallel (use_parallel_residual) and norms each query and key head (qk_layernorm).
StarCoder2
Llama's block with LayerNorms, a bias on every projection, a two-matrix GELU MLP (c_fc, c_proj) in place of SwiGLU, and a sliding window of 4096 positions on every block.
VaultGemma
Gemma 2's names on a pre-norm block: each sublayer is normed on the way in only, so what reaches the residual stream is the module's own output, and the MLP's input norm is pre_feedforward_layernorm.
XGLM
OPT's pre-norm block with no MLP module, so the MLP path's term is fc2.output; the embedding is scaled by sqrt(d_model), and sinusoidal positions are added after it.
Youtu-LLM
Llama's dense block with DeepSeek-V2's latent attention: queries and keys wider than values, and every head's key and value expanded from one latent per token.
ZAYA1
A pre-norm block that scales and shifts, per channel, both the sublayer's output and the stream at every add; its queries and keys are mixed over tokens by a causal convolution, and its mixture runs one expert per token, chosen beside a skip class by a router whose state carries from block to block.
kimi_k25
nnterp.families.kimi_k25 · page not written yet