Attention Mechanism

Adding position-dependent linear bias to attention scores, allowing models to handle sequences longer than their training context window.

Encodes relative position by rotating Q/K vectors in pairs, enabling better generalization to sequence lengths not seen during training.

Each token only attends to a fixed window of nearby tokens instead of the full sequence, reducing cost from O(n²) to O(n·w).

Groups of heads share K/V projections (e.g., 8 groups for 32 heads), balancing quality retention with efficiency — the default in LLaMA 3 and Mistral.

All attention heads share a single set of key/value projections, dramatically reducing KV cache memory and boosting inference speed.