Cost Optimization for Agent Memory

By Arc Labs Research8 min read

The Problem: Memory Costs Grow Unbounded

Agent memory systems can become expensive:

  • Storage bloat: Every agent session, every conversation turn stored forever. 1000 agents × 100 queries/day × 365 days = 36.5M entries. At $0.10/GB/month, that's expensive.
  • Redundant data: Customer ID "12345" stored 500 times across memory entries. Same semantic concept (e.g., "user prefers Nike") stored in 5 different ways.
  • Expensive retrieval: Semantic search costs (embeddings, compute) charged per query. Retrieving 1000 results to filter down to 5 wastes API budget.
  • Old data kept forever: Memory from 2024 still retrieved and charged for, even though it's irrelevant now.
  • Compute overhead: Cloud memory solutions charge for CPU, storage, and API calls. Self-hosting the open-source server (free software) can save 80% on those fees, but you take on running the infra.

Result: Monthly memory bills grow 20–30% month-over-month, eating into agent profitability.

Five optimization strategies cut costs 30–50% while maintaining quality.

Strategy 1: Schema Typing & Structured Fields

The cost: Unstructured free-text memory requires semantic search (expensive embeddings + compute per query).

The fix: Use typed fields for common queries. Metadata filters are 10–100x cheaper than semantic search.

Example Before (Expensive):

// Unstructured memory—every query needs semantic search
await memory.store({
  agent_id: "user-123",
  content: "Customer ordered Nike Air Jordan 1 Low on 2026-05-01, size 11, price $150, black color",
  type: "purchase",
});

// Every retrieval scans embeddings
const results = await memory.retrieve({
  agent_id: "user-123",
  query: "What Nike shoes did customer buy?", // Semantic search: expensive
  topK: 5,
});

Example After (Efficient):

// Structured metadata—queries use filters, not embeddings
await memory.store({
  agent_id: "user-123",
  content: "Purchased Air Jordan 1 Low",
  type: "purchase",
  metadata: {
    productId: "aj1-low-blk",
    brand: "Nike",
    category: "shoes",
    size: 11,
    price: 15000, // In cents
    color: "black",
    timestamp: "2026-05-01T10:30:00Z",
  },
});

// Retrieval uses filters (zero semantic cost)
const results = await memory.retrieve({
  agent_id: "user-123",
  filters: { brand: "Nike", category: "shoes" }, // Metadata filter: cheap
  topK: 5,
});

Cost reduction: 80% cheaper retrieval (metadata filter vs. semantic search).

Strategy 2: Aggressive Eviction & TTL

The cost: Storing irrelevant old memory forever.

The fix: Set time-to-live (TTL) and eviction policies by memory type.

Example:

interface EvictionPolicy {
  type: string;
  ttl_days: number;
  keep_most_recent_n: number; // Keep top N even if expired
  reason: string;
}

const policies: EvictionPolicy[] = [
  {
    type: "browsing_session",
    ttl_days: 30,
    keep_most_recent_n: 5,
    reason: "Sessions older than 30d are not predictive; keep 5 most recent for trend analysis",
  },
  {
    type: "purchase_history",
    ttl_days: null, // Never expire
    keep_most_recent_n: 1000,
    reason: "Purchase history is always relevant",
  },
  {
    type: "preference_fact",
    ttl_days: 180,
    keep_most_recent_n: null,
    reason: "Preferences change every ~6 months; older ones are stale",
  },
  {
    type: "temporary_context",
    ttl_days: 7,
    keep_most_recent_n: 1,
    reason: "Temporary context (API rate limits, current task) is short-lived",
  },
];

// Implementation
async function evict_old_memory(agent_id: string) {
  for (const policy of policies) {
    if (!policy.ttl_days) continue; // No TTL = never evict
    
    const cutoff_date = new Date();
    cutoff_date.setDate(cutoff_date.getDate() - policy.ttl_days);
    
    // Delete old entries, but keep the most recent N
    const deleted = await memory.delete({
      agent_id,
      type: policy.type,
      where: {
        created_at: { $lt: cutoff_date.toISOString() },
        rank: { $gt: policy.keep_most_recent_n }, // Keep top N by rank
      },
    });
    
    console.log(`Evicted ${deleted.count} ${policy.type} entries older than ${cutoff_date.toISOString()}`);
  }
}

Cost reduction: 40–60% storage reduction by removing old, irrelevant data.

Strategy 3: Deduplication & Consolidation

The cost: Same fact stored multiple times (e.g., "user prefers Nike" stored 5 different ways).

The fix: Deduplicate on write. Merge similar entries into one.

Example:

// BAD: Same preference stored 5 times
[
  "Customer prefers Nike",
  "Customer likes Nike shoes",
  "Nike is customer favorite brand",
  "Customer always buys Nike",
  "Prefers Nike over Adidas",
]

// GOOD: Single consolidated entry with metadata
{
  content: "Customer prefers Nike over competitors",
  type: "preference_fact",
  metadata: {
    preference: "brand",
    value: "Nike",
    strength: "very_strong", // Inferred from frequency
    observed_count: 5, // Deduplicated 5 observations
    first_observed: "2026-01-15",
    last_updated: "2026-05-12",
  },
}

// Implementation: Before storing, check if similar fact exists
async function store_deduped(agent_id: string, content: string, type: string, metadata: any) {
  // Check for duplicates with semantic similarity or metadata match
  const existing = await memory.retrieve({
    agent_id,
    filters: { type, metadata: metadata.preference }, // Metadata match first
    topK: 5,
  });
  
  if (existing.length > 0) {
    // Merge with highest-confidence match
    const best_match = existing[0];
    
    // Update metadata: increment observed_count, update last_updated
    await memory.update(best_match.id, {
      metadata: {
        ...best_match.metadata,
        observed_count: (best_match.metadata.observed_count || 1) + 1,
        last_updated: new Date().toISOString(),
      },
    });
    
    return; // Don't store duplicate
  }
  
  // New fact, store it
  await memory.store({
    agent_id,
    content,
    type,
    metadata,
  });
}

Cost reduction: 30–40% storage reduction by consolidating duplicates.

Strategy 4: Retrieval Caps & Caching

The cost: Retrieving 1000 results to use 5 (100 embeddings computed, all charged).

The fix: Cap topK by necessity, cache frequent queries.

Example:

import { LRUCache } from "lru-cache";

const retrievalCache = new LRUCache<string, any>({
  max: 1000, // Cache 1000 queries
  ttl: 1000 * 60 * 5, // 5-minute TTL
});

async function retrieve_with_caching(
  agent_id: string,
  query: string,
  topK: number = 5,
  filters?: any
) {
  // Create cache key
  const cacheKey = `${agent_id}:${JSON.stringify(filters)}:${topK}`;
  
  // Check cache first
  const cached = retrievalCache.get(cacheKey);
  if (cached) {
    return cached;
  }
  
  // Cap topK to reduce compute
  const actualTopK = Math.min(topK, 10); // Never retrieve >10, even if asked
  
  // Retrieve
  const results = await memory.retrieve({
    agent_id,
    query,
    topK: actualTopK,
    filters,
  });
  
  // Cache result
  retrievalCache.set(cacheKey, results);
  
  return results;
}

// Example: Don't retrieve 100 results to filter to 5
// BEFORE (expensive):
// const all_purchases = await memory.retrieve({ agent_id, query: "purchases", topK: 100 });
// const nike = all_purchases.filter(p => p.metadata.brand === "Nike").slice(0, 5);

// AFTER (efficient):
// const nike = await memory.retrieve({
//   agent_id,
//   query: "Nike purchases",
//   topK: 5,
//   filters: { brand: "Nike" }, // Filter on write, not read
// });

Cost reduction: 40–80% retrieval cost reduction via caching and strict topK limits.

Strategy 5: Self-Hosted vs. Cloud Trade-Offs

The cost: Cloud memory solutions charge per API call.

The fix: Evaluate self-hosting the open-source server vs. managed cloud.

DeploymentCost per QueryLatencyOps OverheadData SovereigntyBest For
Self-Hosted (single node)$0.00<\10ms (co-located)Low100%≤1000 agents, strict latency SLA, privacy-critical
Cloud (Brain managed, Pinecone)0.001–0.001–0.01 per query50–200msMinimal (Brain manages)Provider-hosted1000+ agents, scaling budget, dev velocity
Self-Hosted (HA cluster)$0.0001 per query + infra30–100msHigh (ops team manages)Self-hosted10k+ agents, cost-sensitive, custom logic

Decision logic:

// Estimate monthly cost
function estimate_memory_cost(agents: number, queries_per_day: number, deployment: "single_node" | "cloud" | "ha_cluster") {
  const queries_per_month = agents * queries_per_day * 30;
  
  switch (deployment) {
    case "single_node":
      return 0; // Free software, but limited to ~1000 agents per node
    
    case "cloud":
      const cost_per_query = 0.005; // Brain managed: ~$0.005/query
      const monthly_api_cost = queries_per_month * cost_per_query;
      const monthly_storage_cost = agents * 1000 * 0.0001; // $0.0001 per GB/month
      return monthly_api_cost + monthly_storage_cost;
    
    case "ha_cluster":
      const server_cost = 1000; // Modest multi-node cluster + compute: $1000/month
      const self_query_cost = queries_per_month * 0.0001;
      return server_cost + self_query_cost;
  }
}

// Example
const cost_cloud = estimate_memory_cost(5000, 1000, "cloud"); // ~$150/month
const cost_self_hosted = estimate_memory_cost(5000, 1000, "ha_cluster"); // ~$1150/month
// Cloud is cheaper at 5k agents. If you had 50k agents, a self-hosted cluster wins.

Decision flowchart:

Do you have &lt;\1000 agents?
  ├─ YES, low latency critical (&lt;\10ms)?
  │  └─ Self-host a single node (co-located)
  │
  ├─ YES, willing to accept 50–200ms?
  │  └─ Use Cloud (Brain managed, scale later)
  │
  └─ NO, have 1000–10k agents?
     └─ Use Cloud (cost-effective, ops-simple)

Do you have >50k agents AND ops budget?
  └─ Consider a Self-Hosted HA cluster (lower per-query cost)

Cost Comparison Table: Before & After Optimization

ComponentBeforeAfterSavings
Storage (12-month history, 10k agents)250 GB @ 0.10/GB=0.10/GB = 300/month100 GB @ 0.10/GB=0.10/GB = 100/month67%
Retrieval API (1M queries/day)1M × 0.005=0.005 = 5000/month300k queries (caching + caps) × 0.005=0.005 = 1500/month70%
Deduplication overheadNone10 CPU hours/month to dedupe+$10
TTL/Eviction jobNone5 CPU hours/month+$5
Total Monthly$5300/month$1615/month70% savings

Implementation Checklist

  • Schema audit: Review all memory types. Are high-volume types using semantic search? Move to metadata filters where possible.
  • TTL policy design: Define TTL and keep_most_recent_n for each memory type. Test on staging.
  • Deduplication job: Write script to find duplicates (metadata match + semantic similarity). Consolidate and delete.
  • Retrieval optimization: Set strict topK limits (≤10). Add caching layer with 5–15 minute TTL.
  • Deployment evaluation: Estimate cost for current agent count. Would self-hosting save money?
  • Cost monitoring: Add metrics for queries_per_month, storage_gb, api_cost_per_month. Track weekly.
  • A/B test changes: Deploy schema change to 5% of agents, verify hit rate and latency stay >95% of baseline before rolling out.

Cost Monitoring Dashboard

Track these metrics weekly:

Memory Cost Metrics (Weekly)
├─ Total Queries: 500k (↓10% vs. last week)
├─ Storage: 120 GB (↓5% via TTL)
├─ Estimated Monthly Cost: $1,850 (↓$200 since optimization started)
├─ Cost Per Agent Per Month: $0.37 (target: <$0.50)
├─ Hit Rate: 73% (baseline: 72%)
└─ P99 Latency: 92ms (baseline: 95ms, acceptable)

Next Steps

Related reading

Updates from the lab.

Engineering notes, research drops, occasional product updates. Roughly monthly.