# Integrating 2-Million-Token Context Windows Into Real Web Apps: Patterns That Actually Work Most context window integration failures stem from treating expanded capacity as a license to dump everything into the prompt. Teams that upgraded from 8K to 200K to 2M tokens saw their latency triple and their costs explode because they never implemented context boundaries. The vendors marketed context windows as a feature. Engineers need to treat them as a constraint. The distinction between available tokens and effective tokens determines whether your AI integration becomes a production asset or a runaway cost center. This post covers the patterns that bridge that gap. ## Understanding Context Window Mechanics: What 2M Tokens Actually Means A 2-million-token context window does not mean developers should use 2 million tokens per request. The relationship between context size and response quality is nonlinear. Models trained on smaller contexts show degradation at the extremes. Latency scales poorly past 500K tokens for most architectures. Cost per token remains constant, which means a 2M-token request costs 250× more than an 8K request. The practical ceiling for most applications sits between 200K and 800K tokens. That range provides enough headroom for complex documents while keeping response times under 10 seconds and costs predictable. The architecture decisions that matter happen at the boundaries between what goes into context and what gets excluded. ```mermaid %% alt: Token flow from document corpus through context budget to model inference flowchart TD Corpus["Document Corpus
5M tokens available"] Budget["Token Budget Controller
Max 500K tokens"] Priority["Context Prioritization
Rank by relevance"] Window["Active Context Window
480K tokens used"] Model["LLM Inference
Returns response"] Corpus --> Budget Budget --> Priority Priority --> Window Window --> Model classDef dataStore fill:#1e293b,stroke:#64ffda,color:#e2e8f0 classDef framework fill:#064e3b,stroke:#34d399,color:#6ee7b7 class Corpus,Window dataStore class Budget,Priority,Model framework ``` ![Context window architecture diagram showing token flow](https://emzvxuokuqzdkmyrzfut.supabase.co/storage/v1/object/public/blog/posts/2-million-token-context-windows-real-web-apps/content-1.jpg) The token budget pattern enforces this ceiling. Teams that implement budget controllers see 60% cost reduction and 3× latency improvement compared to naive context stuffing. The pattern works by establishing hard limits before documents enter the context pipeline. ## The Token Budget Pattern: Controlling Cost and Latency The token budget pattern treats context capacity as a finite resource that gets allocated across competing needs. A budget controller sits between document retrieval and prompt construction. It tracks cumulative token count and rejects additions that would exceed the limit. ```typescript interface ContextBudget { maxTokens: number; reservedTokens: number; usedTokens: number; } class ContextBudgetController { private budget: ContextBudget; private tokenCounter: (text: string) => number; constructor(maxTokens: number, reservedForResponse: number) { this.budget = { maxTokens, reservedTokens: reservedForResponse, usedTokens: 0, }; // Use tiktoken or similar for accurate counting this.tokenCounter = (text) => Math.ceil(text.length / 4); } canAdd(text: string): boolean { const tokens = this.tokenCounter(text); const available = this.budget.maxTokens - this.budget.reservedTokens - this.budget.usedTokens; return tokens <= available; } add(text: string): boolean { if (!this.canAdd(text)) return false; this.budget.usedTokens += this.tokenCounter(text); return true; } getRemaining(): number { return this.budget.maxTokens - this.budget.reservedTokens - this.budget.usedTokens; } reset(): void { this.budget.usedTokens = 0; } } ``` The reserved token allocation prevents truncated responses. Models that run out of context mid-generation return incomplete or corrupted output. Reserving 10-20% of the window for response generation eliminates this failure mode. The tradeoff is reduced input capacity, but that constraint forces better context selection upstream. The implementation tracks three numbers: total capacity, reserved capacity, and used capacity. When a document or code file requests admission to context, the controller checks available space. Rejection at this stage is cheaper than discovering the limit during inference. ## Context Window Strategies: RAG vs Full Context vs Hybrid Approaches The choice between retrieval-augmented generation and full-context approaches depends on document stability and query patterns. RAG works when the corpus is large and mostly static. Full context works when the entire working set fits in the window and changes frequently. Hybrid approaches combine both and handle the majority of real production scenarios. ```mermaid %% alt: Comparison of RAG-only approach versus hybrid context approach flowchart LR subgraph RAGOnly["RAG Approach: retrieve on demand"] RQ[Query arrives] RS[Search vector store] RR[Retrieve top K] RC[Build context] RQ --> RS --> RR --> RC end subgraph HybridApproach["Hybrid: base context + dynamic retrieval"] HQ[Query arrives] HBase[Load base context
cached in memory] HSearch[Search for gaps] HMerge[Merge contexts] HQ --> HBase HBase --> HSearch HSearch --> HMerge end classDef framework fill:#064e3b,stroke:#34d399,color:#6ee7b7 classDef dataStore fill:#1e293b,stroke:#64ffda,color:#e2e8f0 class RS,HSearch framework class RR,HBase,HMerge dataStore ``` RAG-only architectures pay a search penalty on every request. The vector store lookup adds 50-200ms of latency. For applications where response time matters more than freshness, this overhead accumulates. Caching search results helps but introduces staleness risk. Full-context approaches load the entire working set into the window at request time. This works for codebases under 400K tokens or document sets that fit comfortably in budget. The advantage is zero search latency and perfect recall. The disadvantage is wasted capacity when queries only need a subset of the context. Hybrid strategies maintain a base context of frequently accessed documents and augment it with retrieval for specific queries. A code review agent might keep the file under review plus its direct dependencies in base context, then retrieve related tests or documentation on demand. This pattern delivers 90% of full-context recall with 40% of the token cost. ## Implementing Dynamic Context Windowing in TypeScript Dynamic context windowing adjusts the active context based on query type and available budget. The implementation tracks context segments separately and swaps them in or out based on relevance scores. This differs from static context construction where the window gets built once per session. ```typescript interface ContextSegment { id: string; content: string; tokens: number; relevanceScore: number; priority: 'high' | 'medium' | 'low'; } class DynamicContextWindow { private segments: Map; private activeSegments: Set; private budget: ContextBudgetController; constructor(maxTokens: number) { this.segments = new Map(); this.activeSegments = new Set(); this.budget = new ContextBudgetController(maxTokens, maxTokens * 0.15); } registerSegment(segment: ContextSegment): void { this.segments.set(segment.id, segment); } buildContext(query: string): string { this.budget.reset(); this.activeSegments.clear(); // Always include high-priority segments first const sorted = Array.from(this.segments.values()) .sort((a, b) => { if (a.priority !== b.priority) { const priorityOrder = { high: 0, medium: 1, low: 2 }; return priorityOrder[a.priority] - priorityOrder[b.priority]; } return b.relevanceScore - a.relevanceScore; }); const contextParts: string[] = []; for (const segment of sorted) { if (this.budget.canAdd(segment.content)) { this.budget.add(segment.content); this.activeSegments.add(segment.id); contextParts.push(segment.content); } } return contextParts.join('\n\n---\n\n'); } updateRelevance(segmentId: string, score: number): void { const segment = this.segments.get(segmentId); if (segment) { segment.relevanceScore = score; } } } ``` The segment registration pattern separates context loading from context selection. Documents get tokenized and scored once during registration. Context building becomes a filter operation over the segment collection. This architecture scales to repositories with thousands of files because most files never enter the active window. Priority levels create a two-tier system. High-priority segments always get included if budget allows. Medium and low segments compete based on relevance scores. For a code review agent, the file under review gets high priority. Test files and documentation get medium priority. Unrelated modules get low priority or exclusion. ![Dynamic context windowing showing segment prioritization](https://emzvxuokuqzdkmyrzfut.supabase.co/storage/v1/object/public/blog/posts/2-million-token-context-windows-real-web-apps/content-2.jpg) ## The Context Prioritization Pattern: What to Include When You Can't Include Everything Context prioritization makes the difference between an agent that seems intelligent and one that hallucinates. The failure mode happens when critical information gets excluded while tangential details consume the budget. Teams need a ranking system that surfaces essential context before optional context. ```mermaid %% alt: Context prioritization flow showing filtering and ranking stages flowchart TD Input["Input Document Set
2000 files"] Filter["Relevance Filter
Remove unrelated"] Rank["Ranking Engine
Score by importance"] Budget["Budget Check
Fit to limit"] Window["Final Context
320 files included"] Input --> Filter Filter --> Rank Rank --> Budget Budget --> Window Exclude["Excluded: 1680 files"] Filter -.->|"fails relevance"| Exclude Budget -.->|"exceeds budget"| Exclude classDef framework fill:#064e3b,stroke:#34d399,color:#6ee7b7 classDef dataStore fill:#1e293b,stroke:#64ffda,color:#e2e8f0 class Filter,Rank,Budget framework class Input,Window,Exclude dataStore style Exclude stroke:#ef4444,fill:#450a0a,color:#fca5a5 ``` The prioritization pipeline operates in three stages. First, relevance filtering removes documents with no connection to the current query. A code review for an authentication module excludes database migration scripts. This stage typically cuts the candidate set by 60-80%. Second, ranking scores the remaining documents. The scoring function combines multiple signals: direct references in the query, import relationships for code, edit recency, and past usage patterns. Documents with higher scores get preferential allocation when budget runs tight. Third, budget allocation walks the ranked list and includes documents until the budget exhausts. Documents at the boundary get truncated rather than excluded entirely. The first 200 lines of a 500-line file often contain enough context for the model to understand structure and purpose. The practical implementation requires domain-specific heuristics. For code review agents, files imported by the changed file score higher than files that import it. For document analysis, sections that contain query keywords score higher than sections that match broader semantic similarity. These rules encode the difference between context that helps and context that distracts. ## Real-World Case Study: Building a Code Review Agent with Context Management A production code review agent demonstrates the patterns in combination. The agent receives a pull request with 8 changed files totaling 3,200 lines. The repository contains 2,400 files and 480K lines of code. The naive approach loads all 2,400 files into context. The result is 12M tokens, which exceeds the window and costs $180 per review. ```mermaid %% alt: Code review agent context building workflow flowchart TD PR["Pull Request
8 files changed"] Parse["Parse Changes
Extract imports"] Base["Build Base Context
Changed files + deps"] Score["Score Repository
Rank by relevance"] Augment["Augment Context
Add high-scoring files"] Review["Generate Review
480K tokens used"] PR --> Parse Parse --> Base Base --> Score Score --> Augment Augment --> Review Cost["Cost: $7.20/review
Latency: 8.2s"] Review --> Cost classDef userAction fill:#1e3a8a,stroke:#60a5fa,color:#e0eaff classDef framework fill:#064e3b,stroke:#34d399,color:#6ee7b7 classDef dataStore fill:#1e293b,stroke:#64ffda,color:#e2e8f0 class PR userAction class Parse,Score,Augment,Review framework class Base,Cost dataStore ``` The optimized approach uses dynamic context windowing. Base context includes the 8 changed files plus their direct dependencies, which averages 32 files and 140K tokens. The agent parses imports to identify the dependency graph, then walks one level deep. This captures 95% of the context needed for accurate review. For complex changes, the agent augments base context with relevant test files and documentation. Relevance scoring uses file path similarity and recent commit history. Test files that cover the changed modules score highest. Documentation that matches function names in the diff scores second. This stage adds another 80-120K tokens. The final context averages 480K tokens per review. Cost drops to $7.20 per review. Latency averages 8.2 seconds. Review quality measured by developer acceptance rate matches the naive approach at 89% but without the cost explosion. The key difference is context discipline enforced by the budget controller and prioritization pipeline. ## Production Patterns: Monitoring, Fallbacks, and Cost Control Context window integration becomes production-ready when it includes monitoring, fallback paths, and cost controls. Teams that ship without these safeguards see incidents when context size spikes unexpectedly or model performance degrades under load. The failure modes are subtle because the API returns 200 status even when output quality suffers. Token usage monitoring tracks actual versus budgeted tokens per request. A 10% variance signals context leakage where documents bypass the budget controller. A 50% variance indicates the prioritization logic broke. Alerting on these thresholds catches regressions before they affect users. Fallback paths handle budget overflow gracefully. When a query requires more context than budget allows, the system can truncate low-priority segments, switch to a summarization pass, or split the query into smaller sub-queries. The choice depends on latency tolerance and accuracy requirements. For interactive applications, truncation with a warning keeps response time predictable. For batch processing, sub-query splitting preserves accuracy at the cost of throughput. Cost controls set hard limits on token spend per user session or time window. A runaway context loop that calls the API in a tight loop can burn thousands of dollars in minutes. Rate limiting at the application layer stops this failure mode. The limit should be 10× normal usage to allow legitimate spikes while blocking obvious errors. That covers the essential patterns for integrating 2-million-token context windows into production web applications. Apply these in your codebase and the difference in cost, latency, and output quality will be immediate.

Posts tagged “LLM integration”

Integrating 2-Million-Token Context Windows Into Real Web Apps: Patterns That Actually Work