Transformers

A large language model is a deep stack of identical Transformer layers: early layers capture grammar, middle layers grasp semantics, and deep layers handle reasoning and world knowledge.

Adding position-dependent linear bias to attention scores, allowing models to handle sequences longer than their training context window.

Encodes relative position by rotating Q/K vectors in pairs, enabling better generalization to sequence lengths not seen during training.

Each token only attends to a fixed window of nearby tokens instead of the full sequence, reducing cost from O(n²) to O(n·w).

Groups of heads share K/V projections (e.g., 8 groups for 32 heads), balancing quality retention with efficiency — the default in LLaMA 3 and Mistral.

All attention heads share a single set of key/value projections, dramatically reducing KV cache memory and boosting inference speed.

SwiGLU replaces older ReLU in modern transformers (LLaMA, Mistral), providing smoother gradients and measurably better training dynamics.

Attention scales quadratically with sequence length; a 100K-token input requires 10 billion attention pair computations per layer.

Future tokens are masked during training so each position only attends to past tokens, enabling left-to-right autoregressive generation.

BERT uses an encoder (understanding), GPT uses a decoder (generation), T5 uses both — different configurations optimized for different GenAI tasks.

Skip connections add each sub-layer's input to its output, and normalization prevents values from exploding, enabling stable 100+ layer training.

After attention mixes information across tokens, independent feed-forward layers transform each token's representation with nonlinear activation functions.

Multiple attention mechanisms run simultaneously, each learning to capture different relationship types like syntax, semantics, and coreference.

Tokens generate Q, K, V projections; attention scores come from Q·K dot-product similarity, and the output is V weighted by those scores.

Each token computes relevance scores against all other tokens, capturing long-range dependencies in a single parallel computation step.

Published in 2017's "Attention Is All You Need," this architecture replaced recurrent networks and became the foundation of every frontier GenAI model.