Tokens and Context Windows Explained

1. What Actually Counts as a Token?

Large language models process subword chunks called tokens rather than raw characters or whole words. For standard English prose, 1,000 tokens roughly equals 750 words (around 4 characters per token). However, AI agents rarely exchange plain prose—they exchange structured JSON, source code, URLs, stack traces, and UUIDs, which tokenize much more densely.

Punctuation, indentation whitespace, repeated JSON keys ("customer_id", "timestamp_utc"), and base64 strings can push token density closer to 2.5 to 3.2 characters per token. Consequently, a 10 KB API response often measures 3,000 to 4,000 tokens rather than 2,500.

2. The Five Components Inside Every Agent Context Window

Every model enforces a maximum context window—the hard ceiling covering all input tokens plus the generated output tokens for a single API call. Inside an agent loop, five distinct buckets compete for that space:

1. System Prompt: Behavioral instructions, compliance rules, formatting constraints, and domain policies.
2. Tool & MCP Definitions: Function names, parameter descriptions, enums, and JSON schemas injected into the model's tool registry on every call.
3. Initial User & Retrieval Payload: The user's request, session metadata, and any upfront RAG snippets.
4. Accumulated Step History: Prior assistant tool-call intents plus returned tool outputs from steps 1 through i - 1.
5. Current Step Output & Reasoning: Both the visible tool arguments/prose and any hidden internal thinking tokens generated during the turn.

3. Context Window Sizes vs. Tiered Long-Context Pricing

Modern models span context limits from 128,000 tokens (such as GPT-4o and Groq Llama 3.3 70B) and 200,000 tokens (OpenAI o3, o4-mini, Claude Haiku 4.5) up to 400,000 tokens (GPT-5 series) and ~1,000,000 tokens (GPT-4.1, Claude Sonnet/Opus 4.6 & 5.5, Google Gemini 2.5/3.x/4, DeepSeek V4.1 Flash, Qwen 3.8 Max).

Crucially, fitting inside a 1M context window does not always mean paying the lowest listed per-token rate. Several providers apply prompt-length pricing thresholds:

• Anthropic Claude Haiku 5.5: Standard $0.10/1M input and $0.50/1M output rates apply to prompts up to 100,000 tokens; requests above 100K move to $0.50/1M input and $2.50/1M output.
• Google Gemini 2.5 Pro: Base $1.25/1M input and $10.00/1M output apply up to 200,000 tokens; prompts exceeding 200K move to $2.50/1M input and $15.00/1M output.

In addition to cost jumps, empirical "needle-in-a-haystack" and multi-hop reasoning evaluations consistently show that cramming 150K+ tokens of noisy logs into a prompt degrades tool-selection accuracy and latency. Keeping peak context lean improves both reliability and unit cost.

4. Managing Context Growth: Sliding Windows vs. Checkpointed Summaries

When a coding or research agent approaches 15 to 25 steps, raw append-only conversation history can balloon past 80,000 tokens. Engineers manage that curve using two primary patterns:

• Tool-Result Pruning: Keep tool calls and final conclusions in the thread while replacing older raw outputs (such as Step 2's 4,000-token HTML scrape or file dump) with a compact placeholder note once the model has extracted what it needed. Note that modifying earlier turns invalidates prompt cache prefixes from the edited index onward, so batch pruning at explicit milestone boundaries (for example, every 5 steps) rather than on every single turn.
• Checkpointed State Compaction: Summarize resolved sub-tasks into a deterministic state object and reset the message history cleanly at phase transitions (such as moving from repository discovery into patch generation). Because the static system prompt and tool schemas remain first in the prompt, prefix caching recovers immediately after the reset.

5. Multimodal & Reasoning Token Pitfalls

Agents inspecting screenshots, PDFs, or browser viewports consume multimodal vision tokens in addition to text. A high-resolution full-page screenshot or multi-page PDF rasterized at high detail can easily add 1,000 to 5,000+ input tokens per step—and remains in history on subsequent steps unless explicitly stripped after inspection. Meanwhile, reasoning-enabled models generate internal chain-of-thought tokens that are hidden from final user output yet billed at the output token rate, which is typically 4x to 10x higher than input rates.

6. Estimating Tokens Before Deployment

To populate our AI Agent Cost Calculator accurately before writing code, capture token counts from a 20-trace sample using each provider's native tokenizer endpoint or SDK usage metadata (usage.prompt_tokens, usage.completion_tokens, and usage.prompt_tokens_details.cached_tokens). If you do not yet have live traces, use these practical starting rules:

• Count 150–250 tokens per registered tool schema depending on nesting depth and parameter enums.
• Measure representative API responses in characters and divide by 3.2 for JSON or code, or 4.0 for English prose.
• For reasoning models, add 400 to 2,000 tokens per step to your output token field to capture thinking tokens.
• Verify tokenizer differences before switching providers: BPE tokenizers from OpenAI, Anthropic, Google, and Llama families can vary by 8% to 15% on code and non-English text, so calibrate using real SDK usage headers whenever possible.

Compare how different context sizes behave across 27 verified models using the Research Agent Cost Calculator and Coding Agent Cost Calculator, and review How to Reduce AI Agent Costs for context compaction tactics.