Skip to content

LLM Fundamentals interview questions

Core architectures, Transformer mechanics, self-attention, context windows, tokenization, KV cache, and sampling algorithms.

What to study

  • Explain scaled dot-product attention and QKV projection math.
  • Trade-offs between autoregressive (decoder-only) vs encoder-decoder models.
  • How KV caching eliminates redundant recomputation during inference.

Questions and answers

  1. What is KV Cache and how does it speed up inference?

    Intermediate · 3 min read

  2. How does Grouped-Query Attention (GQA) differ from Multi-Head Attention (MHA)?

    Intermediate · 2 min read

  3. What is the difference between encoder-only, decoder-only, and encoder-decoder LLMs?

    Beginner · 2 min read

  4. How does Byte-Pair Encoding (BPE) tokenization work?

    Beginner · 2 min read

  5. How do Rotary Position Embeddings (RoPE) and RoPE scaling work?

    Advanced · 2 min read

  6. What is Temperature, Top-p, and Top-k sampling?

    Beginner · 2 min read

  7. What is the difference between greedy decoding and beam search?

    Beginner · 2 min read

  8. What is a context window and how does it limit LLM processing?

    Beginner · 2 min read

  9. How does Multi-Query Attention (MQA) differ from Multi-Head Attention?

    Intermediate · 2 min read

  10. How does FlashAttention optimize memory and speed via tiling?

    Advanced · 2 min read

  11. What is tokenization and why can't LLMs process raw string characters directly?

    Beginner · 2 min read

  12. What is the difference between causal and bidirectional self-attention?

    Beginner · 2 min read

  13. What is a logit and how is it converted to token probabilities via Softmax?

    Beginner · 2 min read

  14. What is an attention mask and why is it needed during batch processing?

    Beginner · 2 min read

  15. What is positional encoding and why do Transformers need it?

    Beginner · 2 min read

  16. How do Mixture of Experts (MoE) architectures scale parameter count efficiently?

    Intermediate · 2 min read

  17. What is linear attention and how does it attempt to solve quadratic complexity?

    Intermediate · 2 min read

  18. How does sliding window attention (SWA) reduce memory usage in long-context models?

    Intermediate · 2 min read

  19. How do state space models (SSMs) like Mamba compare to Transformer self-attention?

    Advanced · 2 min read

  20. How does Differential Attention work to suppress noise and improve long-context focus?

    Advanced · 2 min read

Study LLM Fundamentals with progress tracking