Skip to content

AI System Design interview questions

High-throughput inference, TTFT/TPOT, streaming (SSE), semantic caching, GPU resource planning, and serving infrastructure (vLLM, TGI).

What to study

  • Optimizing Time to First Token (TTFT) and Time Per Output Token (TPOT).
  • PagedAttention, continuous batching, and speculative decoding.
  • Designing semantic caching layers to reduce API expenses and latency.

Questions and answers

  1. How do you optimize Time to First Token (TTFT) vs Time Per Output Token (TPOT)?

    Intermediate · 3 min read

  2. What is Server-Sent Events (SSE) streaming for LLM responses?

    Beginner · 2 min read

  3. How does semantic caching reduce LLM API latency and cost?

    Beginner · 2 min read

  4. How does PagedAttention in vLLM solve memory fragmentation?

    Intermediate · 2 min read

  5. How does speculative decoding speed up LLM inference?

    Advanced · 2 min read

  6. What is continuous batching and how does it improve GPU utilization?

    Beginner · 2 min read

  7. What is the difference between prefill phase and decoding phase?

    Beginner · 2 min read

  8. What is GPU VRAM bandwidth and why is decoding memory-bound?

    Beginner · 2 min read

  9. How do you design a multi-tenant LLM gateway with rate limits?

    Intermediate · 2 min read

  10. How does chunked prefill disaggregate prefill and decoding nodes?

    Advanced · 2 min read

  11. What is load balancing for LLM inference clusters across multi-region GPU pools?

    Beginner · 2 min read

  12. What is prefix caching (prompt caching) and how does it eliminate redundant KV computation?

    Beginner · 2 min read

  13. What is an LLM router and how does it dynamically direct queries based on complexity?

    Beginner · 2 min read

  14. What is connection pooling and keep-alive strategy for high-throughput LLM streaming APIs?

    Beginner · 2 min read

  15. What is graceful degradation in AI services when model providers experience downtime?

    Beginner · 2 min read

  16. How does Chunked Prefill prevent decoding latency spikes during concurrent batch processing?

    Intermediate · 2 min read

  17. How do tensor parallelism (TP) and pipeline parallelism (PP) split large model weights across multi-GPU nodes?

    Intermediate · 2 min read

  18. How does Medusa multi-head decoding accelerate inference without a separate draft model?

    Intermediate · 2 min read

  19. How does vLLM PagedAttention manage Virtual Memory Pages to eliminate KV cache fragmentation?

    Advanced · 2 min read

  20. How to design a global multi-region AI inference architecture with sub-100ms TTFT SLAs?

    Advanced · 2 min read

Study AI System Design with progress tracking