Temporal Context Windows

By Arc Labs Research9 min read

The Problem: Unbounded Memory

Agent memory grows linearly with time. Without limits:

  • Token explosion: Agent stores 1000 messages. Context window = 4K tokens. Agent can no longer fit conversation + memory + reasoning. Costs 5x higher.
  • Retrieval noise: Retrieving "customer sentiment" returns memories from 2 years ago—irrelevant to today's conversation. Wastes tokens on stale context.
  • Cost spiral: Older memories still retrieved (BM25 rank them high if keywords match). Cost per agent per month grows unbounded.

Example: Customer support agent

  • Day 1: 100 memories, retrieve top 5 → 500 tokens
  • Month 1: 3000 memories, retrieve top 5 → 3000 tokens (same 5 from 2x larger corpus)
  • Year 1: 36K memories, retrieve top 5 → 36K tokens (BM25 searches entire history)

Costs grow 360x without temporal boundaries.

Temporal windows bound memory and costs. Three patterns:

  1. Sliding windows: Keep last N hours, drop older
  2. Session boundaries: Reset memory between user sessions
  3. Recency decay: Rank recent memories higher (exponentially decay older)

Pattern 1: Sliding Windows

Idea: Keep only the last N hours of memory. Older memories are archived (or deleted).

When to use:

  • Stateless agent loops (chat bots, customer support) where context resets
  • Real-time systems (trading, monitoring) where yesterday's data is stale
  • Cost is critical; token budgets are tight

How it works:

from datetime import datetime, timedelta
from brain_sdk import Brain

brain = Brain(api_key="...")

async def sliding_window_retrieve(
    agent_id: str,
    query: str,
    window_hours: int = 24,
) -> list:
    """Retrieve memories within a sliding time window"""
    now = datetime.utcnow()
    cutoff = (now - timedelta(hours=window_hours)).isoformat()
    
    results = await brain.retrieve(
        agentId=agent_id,
        query=query,
        filters={
            "timestamp": {"$gte": cutoff}  # Only last N hours
        },
        topK=10,
        mode="hybrid"
    )
    return results

async def cleanup_old_memories(
    agent_id: str,
    window_hours: int = 24,
) -> int:
    """Delete memories older than window"""
    now = datetime.utcnow()
    cutoff = (now - timedelta(hours=window_hours)).isoformat()
    
    # List old memories
    old_memories = await brain.list(
        agentId=agent_id,
        filters={"timestamp": {"$lt": cutoff}}
    )
    
    # Delete each
    deleted = 0
    for memory in old_memories:
        try:
            await brain.delete(agentId=agent_id, memoryId=memory["id"])
            deleted += 1
        except Exception as e:
            print(f"Failed to delete {memory['id']}: {e}")
    
    return deleted

Cleanup schedule:

from apscheduler.schedulers.background import BackgroundScheduler

scheduler = BackgroundScheduler()

# Cleanup old memories every 6 hours
scheduler.add_job(
    cleanup_old_memories,
    "interval",
    hours=6,
    args=["agent_1"],
    kwargs={"window_hours": 24}
)

scheduler.start()

Cost impact:

  • Without window: 36K memories, 36K tokens to search → $0.18/retrieval
  • With 24h window: 50–100 memories, 500 tokens to search → $0.0025/retrieval
  • Savings: 72x reduction

Pattern 2: Session Boundaries

Idea: Reset memory at session boundaries (e.g., end of chat, end of day, end of trading session).

When to use:

  • Conversational agents (each conversation is independent)
  • Daily-reset systems (trading agents reset at market close)
  • Each session has its own context; no cross-session learning

How it works:

import { Brain } from "brain-sdk";
import { v4 as uuidv4 } from "uuid";

const brain = new Brain({ apiKey: process.env.BRAIN_API_KEY });

class ConversationAgent {
  private sessionId: string;
  private agentId: string;

  constructor(agentId: string) {
    this.agentId = agentId;
    this.sessionId = uuidv4();
  }

  async storeInConversation(
    content: string,
    metadata: Record<string, any> = {}
  ) {
    // All memories include sessionId
    await brain.store({
      agentId: this.agentId,
      content: content,
      metadata: {
        sessionId: this.sessionId,
        ...metadata,
      },
    });
  }

  async retrieveInSession(query: string, topK: number = 5) {
    // Retrieve only from current session
    const results = await brain.retrieve({
      agentId: this.agentId,
      query: query,
      filters: {
        sessionId: this.sessionId, // Only this session
      },
      topK: topK,
      mode: "hybrid",
    });
    return results;
  }

  async endSession() {
    // Delete session memories (or archive)
    const sessionMemories = await brain.list({
      agentId: this.agentId,
      filters: { sessionId: this.sessionId },
    });

    for (const memory of sessionMemories) {
      await brain.delete({
        agentId: this.agentId,
        memoryId: memory.id,
      });
    }

    // Reset session
    this.sessionId = uuidv4();
  }
}

// Usage
const agent = new ConversationAgent("support_agent_1");

await agent.storeInConversation(
  "Customer is frustrated about late delivery"
);
await agent.storeInConversation(
  "Offered 20% discount, customer accepted"
);

// Retrieve only within this conversation
const context = await agent.retrieveInSession(
  "customer sentiment",
  topK=10
);

// End session, reset
await agent.endSession();

Cost impact:

  • Without boundaries: Agent retrieves from 36K lifetime memories → 36K tokens
  • With session boundaries: Agent retrieves from <100 session memories → 500 tokens
  • Savings: 72x reduction, plus cleaner agent behavior

Pattern 3: Recency Decay

Idea: Don't delete old memories, but weight recent ones higher. Exponentially decay older memories' relevance.

When to use:

  • Long-term learning is needed (trading strategies learned over months)
  • You want to keep history but focus on recent context
  • Compliance/audit requires keeping all records

How it works:

from datetime import datetime, timedelta
from brain_sdk import Brain
import math

brain = Brain(api_key="...")

async def retrieve_with_recency_decay(
    agent_id: str,
    query: str,
    decay_rate: float = 0.1,  # 10% per day decay
    topK: int = 10,
) -> list:
    """Retrieve memories and re-rank by recency decay"""
    
    # Get all matching memories (unranked)
    all_memories = await brain.list(
        agentId=agent_id,
        limit=1000
    )
    
    # Perform retrieval
    retrieved = await brain.retrieve(
        agentId=agent_id,
        query=query,
        topK=1000,  # Get many, then re-rank
        mode="semantic"
    )
    
    now = datetime.utcnow()
    
    # Apply recency decay
    scored = []
    for memory in retrieved:
        timestamp_str = memory.get("metadata", {}).get("timestamp")
        if not timestamp_str:
            age_days = 0
        else:
            timestamp = datetime.fromisoformat(timestamp_str.replace("Z", "+00:00"))
            age_days = (now - timestamp).days
        
        # Decay: e^(-decay_rate * age_days)
        decay_factor = math.exp(-decay_rate * age_days)
        
        # Original score from semantic retrieval
        original_score = memory.get("score", 1.0)
        
        # New score combines original + decay
        new_score = original_score * decay_factor
        
        scored.append({
            **memory,
            "original_score": original_score,
            "decay_factor": decay_factor,
            "final_score": new_score,
        })
    
    # Sort by final score, return top K
    scored.sort(key=lambda m: m["final_score"], reverse=True)
    return scored[:topK]

Decay schedule:

Day 0 (today):   decay = 1.0   → full weight
Day 1:           decay = 0.90  → 90% weight
Day 7:           decay = 0.49  → 49% weight
Day 30:          decay = 0.05  → 5% weight

A memory from 30 days ago is weighted at 5% of its original retrieval score.

Cost impact:

  • Without decay: All 36K memories scored equally high for old + new queries → 36K tokens
  • With decay: Recent memories ranked higher; fewer old memories in top-10 → 1K tokens (focus on recent)
  • Savings: 36x reduction in retrieval tokens

Comparison: Window Size vs Cost vs Recall Quality

How do different window sizes affect cost and retrieval quality?

Window SizeMemories KeptTokens/RetrievalQualityCost/Month (100 agents)
No window36,00036,000Best (all history)$1,800
30 days7,2007,200Very good$360
7 days1,2001,200Good$60
24 hours150–3001,500Good (recent)$7.50
1 hour20–50500Fair (very recent)$2.50

Trade-off: Larger window = more context but higher cost. Start with 7–24 hour windows; expand if agents need longer memory.

Production Implementation

Combine patterns for optimal behavior:

import { Brain } from "brain-sdk";
import { subHours, subDays } from "date-fns";

const brain = new Brain({ apiKey: process.env.BRAIN_API_KEY });

async function productionRetrieve(
  agentId: string,
  query: string,
  options: {
    sessionId?: string;
    windowHours?: number;
    applyDecay?: boolean;
  } = {}
) {
  const { sessionId, windowHours = 24, applyDecay = true } = options;

  // Build filters
  const filters: Record<string, any> = {};

  // Session boundary
  if (sessionId) {
    filters.sessionId = sessionId;
  }

  // Temporal window
  if (windowHours) {
    const cutoff = subHours(new Date(), windowHours).toISOString();
    filters.timestamp = { $gte: cutoff };
  }

  // Retrieve
  let results = await brain.retrieve({
    agentId,
    query,
    filters,
    topK: 50, // Get more; re-rank if decay
    mode: "hybrid",
  });

  // Apply recency decay if enabled
  if (applyDecay) {
    results = applyRecencyDecay(results);
  }

  return results.slice(0, 10); // Return top 10
}

function applyRecencyDecay(memories: any[]): any[] {
  const now = new Date();
  return memories
    .map((mem) => {
      const timestamp = new Date(mem.metadata?.timestamp || now);
      const ageHours = (now.getTime() - timestamp.getTime()) / (1000 * 60 * 60);
      const decayFactor = Math.exp(-0.1 * ageHours); // 10% per hour decay

      return {
        ...mem,
        score: (mem.score || 1.0) * decayFactor,
      };
    })
    .sort((a, b) => b.score - a.score);
}

Implementation Checklist

  • Choose window pattern: sliding window (stateless), session boundary (conversations), or recency decay (long-term learning)
  • Set window size: 24 hours for stateless, 1 day per session for conversations, 30 days for long-term
  • Add timestamp to all memories (required for temporal filtering)
  • If session boundary: add sessionId to metadata on store()
  • Implement cleanup: cron job to delete/archive old memories (sliding window)
  • If recency decay: implement re-ranking logic in retrieve()
  • Monitor cost: track tokens/retrieval, confirm savings
  • Test: verify agent behavior with and without temporal limits

Next Steps

Related reading

Updates from the lab.

Engineering notes, research drops, occasional product updates. Roughly monthly.