Distributed Training

Microsoft DeepSpeed (ZeRO stages 1-3) and PyTorch FSDP manage the complexity of sharding parameters, gradients, and optimizer states across clusters.

Layers 1-20 on GPU set A, layers 21-40 on GPU set B — combined with micro-batching to keep all devices busy.

Weight matrices are sharded across devices so each GPU computes a slice of every layer — required for models too large for one device's memory.

Every GPU holds a full copy of the model but processes different mini-batches; gradients are averaged across devices after each step.

Frontier model weights, activations, and optimizer states vastly exceed any single GPU's memory; training requires coordinating thousands of devices.