Integrating 2-Million-Token Context Windows Into Real Web Apps: Patterns That Actually Work

Most context window integration failures stem from treating expanded capacity as a license to dump everything into the prompt. Teams that upgraded from 8K to 200K to 2M tokens saw their latency triple and their costs explode because they never implemented context boundaries. The vendors marketed context windows as a feature. Engineers need to treat them as a constraint.

The distinction between available tokens and effective tokens determines whether your AI integration becomes a production asset or a runaway cost center. This post covers the patterns that bridge that gap.

Understanding Context Window Mechanics: What 2M Tokens Actually Means

A 2-million-token context window does not mean developers should use 2 million tokens per request. The relationship between context size and response quality is nonlinear. Models trained on smaller contexts show degradation at the extremes. Latency scales poorly past 500K tokens for most architectures. Cost per token remains constant, which means a 2M-token request costs 250× more than an 8K request.

The practical ceiling for most applications sits between 200K and 800K tokens. That range provides enough headroom for complex documents while keeping response times under 10 seconds and costs predictable. The architecture decisions that matter happen at the boundaries between what goes into context and what gets excluded.

%% alt: Token flow from document corpus through context budget to model inference
flowchart TD
    Corpus["Document Corpus<br/>5M tokens available"]
    Budget["Token Budget Controller<br/>Max 500K tokens"]
    Priority["Context Prioritization<br/>Rank by relevance"]
    Window["Active Context Window<br/>480K tokens used"]
    Model["LLM Inference<br/>Returns response"]
    
    Corpus --> Budget
    Budget --> Priority
    Priority --> Window
    Window --> Model
    
    classDef dataStore fill:#1e293b,stroke:#64ffda,color:#e2e8f0
    classDef framework fill:#064e3b,stroke:#34d399,color:#6ee7b7
    
    class Corpus,Window dataStore
    class Budget,Priority,Model framework

The token budget pattern enforces this ceiling. Teams that implement budget controllers see 60% cost reduction and 3× latency improvement compared to naive context stuffing. The pattern works by establishing hard limits before documents enter the context pipeline.

The Token Budget Pattern: Controlling Cost and Latency

The token budget pattern treats context capacity as a finite resource that gets allocated across competing needs. A budget controller sits between document retrieval and prompt construction. It tracks cumulative token count and rejects additions that would exceed the limit.

interface ContextBudget {
  maxTokens: number;
  reservedTokens: number;
  usedTokens: number;
}
 
class ContextBudgetController {
  private budget: ContextBudget;
  private tokenCounter: (text: string) => number;
 
  constructor(maxTokens: number, reservedForResponse: number) {
    this.budget = {
      maxTokens,
      reservedTokens: reservedForResponse,
      usedTokens: 0,
    };
    // Use tiktoken or similar for accurate counting
    this.tokenCounter = (text) => Math.ceil(text.length / 4);
  }
 
  canAdd(text: string): boolean {
    const tokens = this.tokenCounter(text);
    const available = this.budget.maxTokens - 
                     this.budget.reservedTokens - 
                     this.budget.usedTokens;
    return tokens <= available;
  }
 
  add(text: string): boolean {
    if (!this.canAdd(text)) return false;
    this.budget.usedTokens += this.tokenCounter(text);
    return true;
  }
 
  getRemaining(): number {
    return this.budget.maxTokens - 
           this.budget.reservedTokens - 
           this.budget.usedTokens;
  }
 
  reset(): void {
    this.budget.usedTokens = 0;
  }
}

The reserved token allocation prevents truncated responses. Models that run out of context mid-generation return incomplete or corrupted output. Reserving 10-20% of the window for response generation eliminates this failure mode. The tradeoff is reduced input capacity, but that constraint forces better context selection upstream.

The implementation tracks three numbers: total capacity, reserved capacity, and used capacity. When a document or code file requests admission to context, the controller checks available space. Rejection at this stage is cheaper than discovering the limit during inference.

Context Window Strategies: RAG vs Full Context vs Hybrid Approaches

The choice between retrieval-augmented generation and full-context approaches depends on document stability and query patterns. RAG works when the corpus is large and mostly static. Full context works when the entire working set fits in the window and changes frequently. Hybrid approaches combine both and handle the majority of real production scenarios.

%% alt: Comparison of RAG-only approach versus hybrid context approach
flowchart LR
    subgraph RAGOnly["RAG Approach: retrieve on demand"]
        RQ[Query arrives]
        RS[Search vector store]
        RR[Retrieve top K]
        RC[Build context]
        RQ --> RS --> RR --> RC
    end
    
    subgraph HybridApproach["Hybrid: base context + dynamic retrieval"]
        HQ[Query arrives]
        HBase[Load base context<br/>cached in memory]
        HSearch[Search for gaps]
        HMerge[Merge contexts]
        HQ --> HBase
        HBase --> HSearch
        HSearch --> HMerge
    end
    
    classDef framework fill:#064e3b,stroke:#34d399,color:#6ee7b7
    classDef dataStore fill:#1e293b,stroke:#64ffda,color:#e2e8f0
    
    class RS,HSearch framework
    class RR,HBase,HMerge dataStore

RAG-only architectures pay a search penalty on every request. The vector store lookup adds 50-200ms of latency. For applications where response time matters more than freshness, this overhead accumulates. Caching search results helps but introduces staleness risk.

Full-context approaches load the entire working set into the window at request time. This works for codebases under 400K tokens or document sets that fit comfortably in budget. The advantage is zero search latency and perfect recall. The disadvantage is wasted capacity when queries only need a subset of the context.

Hybrid strategies maintain a base context of frequently accessed documents and augment it with retrieval for specific queries. A code review agent might keep the file under review plus its direct dependencies in base context, then retrieve related tests or documentation on demand. This pattern delivers 90% of full-context recall with 40% of the token cost.

Implementing Dynamic Context Windowing in TypeScript

Dynamic context windowing adjusts the active context based on query type and available budget. The implementation tracks context segments separately and swaps them in or out based on relevance scores. This differs from static context construction where the window gets built once per session.

interface ContextSegment {
  id: string;
  content: string;
  tokens: number;
  relevanceScore: number;
  priority: 'high' | 'medium' | 'low';
}
 
class DynamicContextWindow {
  private segments: Map<string, ContextSegment>;
  private activeSegments: Set<string>;
  private budget: ContextBudgetController;
 
  constructor(maxTokens: number) {
    this.segments = new Map();
    this.activeSegments = new Set();
    this.budget = new ContextBudgetController(maxTokens, maxTokens * 0.15);
  }
 
  registerSegment(segment: ContextSegment): void {
    this.segments.set(segment.id, segment);
  }
 
  buildContext(query: string): string {
    this.budget.reset();
    this.activeSegments.clear();
 
    // Always include high-priority segments first
    const sorted = Array.from(this.segments.values())
      .sort((a, b) => {
        if (a.priority !== b.priority) {
          const priorityOrder = { high: 0, medium: 1, low: 2 };
          return priorityOrder[a.priority] - priorityOrder[b.priority];
        }
        return b.relevanceScore - a.relevanceScore;
      });
 
    const contextParts: string[] = [];
 
    for (const segment of sorted) {
      if (this.budget.canAdd(segment.content)) {
        this.budget.add(segment.content);
        this.activeSegments.add(segment.id);
        contextParts.push(segment.content);
      }
    }
 
    return contextParts.join('\n\n---\n\n');
  }
 
  updateRelevance(segmentId: string, score: number): void {
    const segment = this.segments.get(segmentId);
    if (segment) {
      segment.relevanceScore = score;
    }
  }
}

The segment registration pattern separates context loading from context selection. Documents get tokenized and scored once during registration. Context building becomes a filter operation over the segment collection. This architecture scales to repositories with thousands of files because most files never enter the active window.

Priority levels create a two-tier system. High-priority segments always get included if budget allows. Medium and low segments compete based on relevance scores. For a code review agent, the file under review gets high priority. Test files and documentation get medium priority. Unrelated modules get low priority or exclusion.

The Context Prioritization Pattern: What to Include When You Can't Include Everything

Context prioritization makes the difference between an agent that seems intelligent and one that hallucinates. The failure mode happens when critical information gets excluded while tangential details consume the budget. Teams need a ranking system that surfaces essential context before optional context.

%% alt: Context prioritization flow showing filtering and ranking stages
flowchart TD
    Input["Input Document Set<br/>2000 files"]
    Filter["Relevance Filter<br/>Remove unrelated"]
    Rank["Ranking Engine<br/>Score by importance"]
    Budget["Budget Check<br/>Fit to limit"]
    Window["Final Context<br/>320 files included"]
    
    Input --> Filter
    Filter --> Rank
    Rank --> Budget
    Budget --> Window
    
    Exclude["Excluded: 1680 files"]
    Filter -.->|"fails relevance"| Exclude
    Budget -.->|"exceeds budget"| Exclude
    
    classDef framework fill:#064e3b,stroke:#34d399,color:#6ee7b7
    classDef dataStore fill:#1e293b,stroke:#64ffda,color:#e2e8f0
    
    class Filter,Rank,Budget framework
    class Input,Window,Exclude dataStore
    
    style Exclude stroke:#ef4444,fill:#450a0a,color:#fca5a5

The prioritization pipeline operates in three stages. First, relevance filtering removes documents with no connection to the current query. A code review for an authentication module excludes database migration scripts. This stage typically cuts the candidate set by 60-80%.

Second, ranking scores the remaining documents. The scoring function combines multiple signals: direct references in the query, import relationships for code, edit recency, and past usage patterns. Documents with higher scores get preferential allocation when budget runs tight.

Third, budget allocation walks the ranked list and includes documents until the budget exhausts. Documents at the boundary get truncated rather than excluded entirely. The first 200 lines of a 500-line file often contain enough context for the model to understand structure and purpose.

The practical implementation requires domain-specific heuristics. For code review agents, files imported by the changed file score higher than files that import it. For document analysis, sections that contain query keywords score higher than sections that match broader semantic similarity. These rules encode the difference between context that helps and context that distracts.

Real-World Case Study: Building a Code Review Agent with Context Management

A production code review agent demonstrates the patterns in combination. The agent receives a pull request with 8 changed files totaling 3,200 lines. The repository contains 2,400 files and 480K lines of code. The naive approach loads all 2,400 files into context. The result is 12M tokens, which exceeds the window and costs $180 per review.

%% alt: Code review agent context building workflow
flowchart TD
    PR["Pull Request<br/>8 files changed"]
    Parse["Parse Changes<br/>Extract imports"]
    Base["Build Base Context<br/>Changed files + deps"]
    Score["Score Repository<br/>Rank by relevance"]
    Augment["Augment Context<br/>Add high-scoring files"]
    Review["Generate Review<br/>480K tokens used"]
    
    PR --> Parse
    Parse --> Base
    Base --> Score
    Score --> Augment
    Augment --> Review
    
    Cost["Cost: $7.20/review<br/>Latency: 8.2s"]
    Review --> Cost
    
    classDef userAction fill:#1e3a8a,stroke:#60a5fa,color:#e0eaff
    classDef framework fill:#064e3b,stroke:#34d399,color:#6ee7b7
    classDef dataStore fill:#1e293b,stroke:#64ffda,color:#e2e8f0
    
    class PR userAction
    class Parse,Score,Augment,Review framework
    class Base,Cost dataStore

The optimized approach uses dynamic context windowing. Base context includes the 8 changed files plus their direct dependencies, which averages 32 files and 140K tokens. The agent parses imports to identify the dependency graph, then walks one level deep. This captures 95% of the context needed for accurate review.

For complex changes, the agent augments base context with relevant test files and documentation. Relevance scoring uses file path similarity and recent commit history. Test files that cover the changed modules score highest. Documentation that matches function names in the diff scores second. This stage adds another 80-120K tokens.

The final context averages 480K tokens per review. Cost drops to $7.20 per review. Latency averages 8.2 seconds. Review quality measured by developer acceptance rate matches the naive approach at 89% but without the cost explosion. The key difference is context discipline enforced by the budget controller and prioritization pipeline.

Production Patterns: Monitoring, Fallbacks, and Cost Control

Context window integration becomes production-ready when it includes monitoring, fallback paths, and cost controls. Teams that ship without these safeguards see incidents when context size spikes unexpectedly or model performance degrades under load. The failure modes are subtle because the API returns 200 status even when output quality suffers.

Token usage monitoring tracks actual versus budgeted tokens per request. A 10% variance signals context leakage where documents bypass the budget controller. A 50% variance indicates the prioritization logic broke. Alerting on these thresholds catches regressions before they affect users.

Fallback paths handle budget overflow gracefully. When a query requires more context than budget allows, the system can truncate low-priority segments, switch to a summarization pass, or split the query into smaller sub-queries. The choice depends on latency tolerance and accuracy requirements. For interactive applications, truncation with a warning keeps response time predictable. For batch processing, sub-query splitting preserves accuracy at the cost of throughput.

Cost controls set hard limits on token spend per user session or time window. A runaway context loop that calls the API in a tight loop can burn thousands of dollars in minutes. Rate limiting at the application layer stops this failure mode. The limit should be 10× normal usage to allow legitimate spikes while blocking obvious errors.

That covers the essential patterns for integrating 2-million-token context windows into production web applications. Apply these in your codebase and the difference in cost, latency, and output quality will be immediate.