What is context window truncation and how do you handle overflow gracefully?

Short answer: Context window truncation trims incoming prompt text or message history to fit within a model's maximum token limit; graceful overflow strategies preserve high-priority system prompts and the latest user queries while dropping or summarizing middle conversation turns.

Detail

Exceeding model context window bounds causes API exceptions (400 BadRequest: context_length_exceeded).

Raw History (Over Limit):  [System Prompt] [Old Turn 1] [Old Turn 2] ... [Old Turn 50] [Latest Query]
Truncated Safe History:   [System Prompt] ──► [Summarized Past History] ──► [Latest 3 Turns]

Strategic Truncation Strategies

1. System Prompt Preservation: Never truncate or drop position-0 system rules.
2. Sliding Window History: Keep only the most recent $K$ user-assistant turns while dropping older turns.
3. Progressive Summarization: Compress historical turns into a concise summary block inserted between system prompt and active query.

Example

Python sliding-window message truncator:

import tiktoken

def truncate_chat_history(messages: list[dict], max_tokens: int = 4000, model: str = "gpt-4o") -> list[dict]:
    tokenizer = tiktoken.encoding_for_model(model)
    system_msg = [m for m in messages if m["role"] == "system"]
    other_msgs = [m for m in messages if m["role"] != "system"]

    selected_msgs = []
    current_tokens = sum(len(tokenizer.encode(m["content"])) for m in system_msg)

    # Iterate from newest to oldest messages
    for msg in reversed(other_msgs):
        msg_tokens = len(tokenizer.encode(msg["content"]))
        if current_tokens + msg_tokens > max_tokens:
            break
        selected_msgs.insert(0, msg)
        current_tokens += msg_tokens

    return system_msg + selected_msgs

Interview tips

• Emphasize estimating input tokens using fast local tokenizers prior to dispatching API calls.
• Discuss how context truncation interacts with cost governance.

Related Concepts

• [[How does contextual compression reduce context window token usage during RAG?]] (#128): How does contextual compression reduce context window token usage during RAG?
• [[What is prefix caching (prompt caching) and how does it eliminate redundant KV computation?]] (#152): What is prefix caching (prompt caching) and how does it eliminate redundant KV computation?
• [[How does Chunked Prefill prevent decoding latency spikes during concurrent batch processing?]] (#156): How does Chunked Prefill prevent decoding latency spikes during concurrent batch processing?

⬅ Back to Prompt Engineering · All topics