Memory Consolidation Patterns

By Arc Labs Research9 min read

The Problem: Memory Bloat

Agent memory grows faster than retrieval slows, creating bloat:

  • Retrieval noise: Searching "customer sentiment" returns 100+ results. Top 10 have low signal. Wastes tokens scoring irrelevant memories.
  • Storage cost: 50K memories × 0.001/vector=0.001/vector = 50/month per agent. With consolidation: 5K summaries = $5/month.
  • Latency creep: Searching 50K memories is slower than 5K. BM25 and semantic search degrade.

Example: Customer support agent

  • Day 1–30: 100 interactions, 500 memories (5 per interaction)
  • Month 3: 300 interactions, 1,500 memories
  • Month 6: 600 interactions, 3,000 memories

Retrieval "customer sentiment" now searches 3,000 items. Many are duplicates:

  • "Customer angry about shipping" (Day 1)
  • "Customer still angry about shipping" (Day 2)
  • "Customer angry about shipping and billing" (Day 3)

Consolidation: Collapse these 3 into one summary: "Customer upset about shipping delays and billing errors; prefers email contact."

Results:

  • 3,000 memories → 1,000 consolidated summaries (66% reduction)
  • Retrieval latency improves 3x
  • Storage cost drops from 3to3 to 1
  • Retrieval quality improves (less noise, more relevant summaries)

Pattern 1: Summarize Similar Memories

Idea: Group similar memories by topic, summarize the group, replace with one consolidated memory.

When to use:

  • Agent stores detailed logs that are repetitive (trading logs, chat turns)
  • You want to reduce noise while preserving facts
  • Long-term memory is more important than detailed history

How it works:

import json
from datetime import datetime, timedelta
from brain_sdk import Brain
from anthropic import Anthropic

brain = Brain(api_key="...")
claude = Anthropic(api_key="...")

async def consolidate_by_topic(
    agent_id: str,
    topic: str,
    lookback_days: int = 7,
) -> dict:
    """
    Consolidate memories matching a topic.
    
    Example: topic="trading_losses"
    Retrieves all loss memories from the last week, summarizes them into one.
    """
    
    # 1. Retrieve all memories matching topic
    cutoff = (datetime.utcnow() - timedelta(days=lookback_days)).isoformat()
    
    memories = await brain.retrieve(
        agentId=agent_id,
        query=topic,
        filters={
            "timestamp": {"$gte": cutoff}
        },
        topK=100,  # Get many
        mode="semantic"
    )
    
    if len(memories) < 2:
        return {"consolidated": False, "reason": "Less than 2 memories"}
    
    # 2. Extract memory texts
    memory_texts = [m["content"] for m in memories]
    
    # 3. Summarize using Claude
    summary = claude.messages.create(
        model="claude-opus-4-1",
        max_tokens=500,
        messages=[
            {
                "role": "user",
                "content": f"""Consolidate these {len(memory_texts)} memories about {topic} into a single summary.
Focus on patterns, key facts, and trends. Preserve specifics.

Memories:
{json.dumps(memory_texts, indent=2)}

Return a single consolidated memory (2-3 sentences max)."""
            }
        ]
    ).content[0].text
    
    # 4. Store consolidated memory
    consolidated_id = await brain.store(
        agentId=agent_id,
        content=summary,
        metadata={
            "type": "consolidation_summary",
            "topic": topic,
            "consolidated_from": len(memories),
            "lookback_days": lookback_days,
            "timestamp": datetime.utcnow().isoformat(),
        }
    )
    
    # 5. Delete original memories
    for memory in memories:
        try:
            await brain.delete(agentId=agent_id, memoryId=memory["id"])
        except Exception as e:
            print(f"Failed to delete {memory['id']}: {e}")
    
    return {
        "consolidated": True,
        "consolidated_id": consolidated_id,
        "memories_merged": len(memories),
        "summary": summary
    }

# Usage
result = await consolidate_by_topic("trader_1", "trading_losses", lookback_days=7)
print(f"Consolidated {result['memories_merged']} losses into 1 summary")

Example execution:

Input (5 trading loss memories):
1. "Sold AAPL too early, lost 1.2% on recovery"
2. "Held MSFT too long, got stopped at -2.5%"
3. "TSLA bad entry, exited at -1.8%"
4. "Missed NVDA dip, entered at peak, -2.1%"
5. "Tech sector rotation caught me unprepared, -3.2% total"

Output (1 consolidated summary):
"Lost 10.8% across 5 trades in tech sector (AAPL, MSFT, TSLA, NVDA).
Errors: premature exit, overholding, bad entry, missed dips.
Pattern: Sector rotation without position rotation."

Pattern 2: Filter Duplicates

Idea: Detect and remove near-duplicate memories (same content, slightly different wording).

When to use:

  • Agent stores redundant information (chat bots storing every message + summary)
  • You want to deduplicate without manual review
  • Exact deduplication isn't needed (near-duplicate is fine)

How it works:

import { Brain } from "brain-sdk";

const brain = new Brain({ apiKey: process.env.BRAIN_API_KEY });

async function deduplicateMemories(agentId: string): Promise<number> {
  // 1. Get all memories
  const allMemories = await brain.list({
    agentId,
    limit: 10000,
  });

  // 2. Group by embedding similarity (near-duplicates)
  const clusters: Map<number, typeof allMemories> = new Map();
  const processed = new Set<string>();

  for (let i = 0; i < allMemories.length; i++) {
    if (processed.has(allMemories[i].id)) continue;

    const cluster: typeof allMemories = [allMemories[i]];
    processed.add(allMemories[i].id);

    // Find similar memories (cosine similarity)
    for (let j = i + 1; j < allMemories.length; j++) {
      if (processed.has(allMemories[j].id)) continue;

      const similarity = cosineSimilarity(
        allMemories[i].embedding,
        allMemories[j].embedding
      );

      // If similarity > 0.95, it's a duplicate
      if (similarity > 0.95) {
        cluster.push(allMemories[j]);
        processed.add(allMemories[j].id);
      }
    }

    if (cluster.length > 1) {
      clusters.set(i, cluster);
    }
  }

  // 3. For each cluster, keep the best and delete the rest
  let deletedCount = 0;

  for (const cluster of clusters.values()) {
    // Keep the most recent
    cluster.sort(
      (a, b) =>
        new Date(b.metadata?.timestamp || 0).getTime() -
        new Date(a.metadata?.timestamp || 0).getTime()
    );

    const keep = cluster[0];
    const toDelete = cluster.slice(1);

    for (const memory of toDelete) {
      try {
        await brain.delete({ agentId, memoryId: memory.id });
        deletedCount++;
      } catch (e) {
        console.error(`Failed to delete ${memory.id}:`, e);
      }
    }
  }

  return deletedCount;
}

function cosineSimilarity(a: number[], b: number[]): number {
  const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
  return dotProduct / (magnitudeA * magnitudeB);
}

// Usage
const deleted = await deduplicateMemories("agent_1");
console.log(`Deleted ${deleted} duplicate memories`);

Pattern 3: Archive Cold Data

Idea: Move rarely-retrieved memories to cold storage (separate table, S3, etc.). Keep hot data in Brain.

When to use:

  • You must keep long history (compliance, audit)
  • But most queries only need recent data
  • Cold storage is much cheaper than active memory

How it works:

from datetime import datetime, timedelta
from brain_sdk import Brain
import json
import boto3

brain = Brain(api_key="...")
s3_client = boto3.client("s3")
S3_BUCKET = "agent_memory_archive"

async def archive_cold_memories(
    agent_id: str,
    hot_days: int = 30,
    cold_days: int = 365,
) -> dict:
    """
    Move old, rarely-retrieved memories to S3 cold storage.
    Keep recent memories in hot Brain storage.
    
    hot_days: Keep memories newer than this (active in Brain)
    cold_days: Archive memories older than this (to S3)
    """
    
    # 1. Find cold memories (older than cold_days, not accessed recently)
    cutoff_date = (datetime.utcnow() - timedelta(days=cold_days)).isoformat()
    
    cold_memories = await brain.list(
        agentId=agent_id,
        filters={
            "timestamp": {"$lt": cutoff_date},
            "last_accessed": {"$lt": cutoff_date}  # Not accessed in cold_days
        },
        limit=10000
    )
    
    if not cold_memories:
        return {"archived": 0, "reason": "No cold memories"}
    
    # 2. Create archive file
    archive = {
        "agent_id": agent_id,
        "archived_at": datetime.utcnow().isoformat(),
        "count": len(cold_memories),
        "memories": [
            {
                "id": m["id"],
                "content": m["content"],
                "metadata": m.get("metadata", {}),
                "timestamp": m["metadata"].get("timestamp")
            }
            for m in cold_memories
        ]
    }
    
    # 3. Upload to S3
    archive_key = f"archives/{agent_id}/{datetime.utcnow().strftime('%Y-%m-%d')}.json"
    
    s3_client.put_object(
        Bucket=S3_BUCKET,
        Key=archive_key,
        Body=json.dumps(archive),
        ContentType="application/json"
    )
    
    # 4. Delete from Brain
    deleted_count = 0
    for memory in cold_memories:
        try:
            await brain.delete(agentId=agent_id, memoryId=memory["id"])
            deleted_count += 1
        except Exception as e:
            print(f"Failed to delete {memory['id']}: {e}")
    
    return {
        "archived": len(cold_memories),
        "archive_key": archive_key,
        "deleted_from_brain": deleted_count
    }

Retrieval from archive (when needed):

async def retrieve_from_archive(agent_id: str, date: str) -> list:
    """Retrieve archived memories if needed"""
    archive_key = f"archives/{agent_id}/{date}.json"
    
    obj = s3_client.get_object(Bucket=S3_BUCKET, Key=archive_key)
    archive = json.loads(obj["Body"].read())
    
    return archive["memories"]

Consolidation Triggers

When should you consolidate?

TriggerConditionAction
Size-basedMemories > 1,000Consolidate oldest 500 by topic
Time-basedWeekly cron jobConsolidate duplicates, archive >30 days
Quality-basedLow retrieval scoreSummarize 10 low-scoring memories into 1
ManualUser requestRun consolidation on demand

Implementation (size-based trigger):

import { Brain } from "brain-sdk";

const brain = new Brain({ apiKey: process.env.BRAIN_API_KEY });

async function monitorMemorySize(agentId: string) {
  const allMemories = await brain.list({ agentId });
  const size = allMemories.length;

  if (size > 1000) {
    console.log(`${agentId} has ${size} memories. Triggering consolidation...`);
    await consolidateByTopic(agentId, "observed_pattern", lookback_days = 7);
  }
}

// Run every hour
setInterval(() => {
  monitorMemorySize("agent_1");
}, 1000 * 60 * 60);

Consolidation Patterns: Cost Impact

Consolidate 5 memories into 1; how much do you save?

BeforeAfterSavings
5 memories @ 500 bytes each1 summary @ 200 bytes80% storage, 72% retrieval tokens
50K memories, BM25 search10K summaries, BM25 search80% search time, 80% retrieval cost
3,000 vectors @ $0.001 ea600 summaries @ $0.001 ea$2.40/month per agent

At scale (100 agents, 3K memories each):

  • Before: 300K memories = 300/monthstorage+300/month storage + 150/month retrieval = $450/month
  • After: 60K summaries = 60/monthstorage+60/month storage + 30/month retrieval = $90/month
  • Savings: $360/month (80%)

Implementation Checklist

  • Choose consolidation pattern: summarize similar (topic-based), deduplicate (near-exact), or archive (cold)
  • If summarizing: identify topics (trading_losses, customer_complaints, decisions)
  • If deduplicating: threshold for similarity (>0.95 cosine similarity)
  • If archiving: define hot/cold boundaries (30 days hot, >365 days cold)
  • Implement consolidation function with error handling
  • Add consolidation trigger: size-based (>1K), time-based (weekly), or quality-based (low score)
  • Test consolidation on a staging agent before production
  • Monitor: measure token reduction, retrieval latency, cost per agent
  • Document: when consolidation runs, what gets consolidated, where archives live

Next Steps

Related reading

Updates from the lab.

Engineering notes, research drops, occasional product updates. Roughly monthly.