What is gradient accumulation and how does it simulate larger batch sizes on small GPUs?

Short answer: Gradient accumulation accumulates calculated gradients across $K$ micro-batches during successive forward/backward passes without updating model parameters; after $K$ steps, optimizer.step() updates weights once, simulating an effective batch size $K\times$ larger than hardware VRAM normally fits.

Detail

Training LLMs requires large effective batch sizes (e.g. 128 or 256) for stable gradient estimates.

If a single GPU VRAM limit permits fitting a micro-batch size of only 4 sequences, gradient accumulation bridges the gap:

$\text{Effective Batch Size} = \text{Per-Device Micro-Batch Size} \times \text{Gradient Accumulation Steps} \times \text{Number of GPUs}$

Micro-Batch 1 (Size 4) ──► Forward + Backward ──► Accumulate Gradients (No weight update)
Micro-Batch 2 (Size 4) ──► Forward + Backward ──► Accumulate Gradients (No weight update)
Micro-Batch 3 (Size 4) ──► Forward + Backward ──► Accumulate Gradients (No weight update)
Micro-Batch 4 (Size 4) ──► Forward + Backward ──► Accumulate Gradients
                                                      │
                                                      ▼
                            `optimizer.step()` (Effective Batch Size = 16)

Example

PyTorch implementation pattern for gradient accumulation:

import torch

model = torch.nn.Linear(10, 2)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
accumulation_steps = 4

optimizer.zero_grad()
for step, (inputs, targets) in enumerate(dataset_loader):
    outputs = model(inputs)
    loss = criterion(outputs, targets) / accumulation_steps # Scale loss
    loss.backward() # Accumulate gradients

    if (step + 1) % accumulation_steps == 0:
        optimizer.step() # Apply accumulated gradient updates
        optimizer.zero_grad() # Reset accumulated gradients

Interview tips

• Emphasize dividing the loss by accumulation_steps before loss.backward() so accumulated gradients represent the true mathematical average across the full effective batch.
• Discuss how gradient accumulation trades compute time for VRAM memory.

Related Concepts

• [[What is learning rate scheduling (cosine decay) during LLM fine-tuning?]] (#142): What is learning rate scheduling (cosine decay) during LLM fine-tuning?
• [[What is mixed precision training (FP16 vs BF16) and why is BF16 preferred on modern GPUs?]] (#144): What is mixed precision training (FP16 vs BF16) and why is BF16 preferred on modern GPUs?
• [[How does DeepSpeed ZeRO stage 1, 2, and 3 partition optimizer states, gradients, and parameters?]] (#150): How does DeepSpeed ZeRO stage 1, 2, and 3 partition optimizer states, gradients, and parameters?

⬅ Back to Fine-Tuning and Adaptation · All topics