Top Mistakes Building Agent Memory

1. Ignoring latency until it's too late

Mistake: Starting with a remote vector DB (Pinecone, Weaviate) and realizing too late that agent loop latency doubled.

Why it happens: Early prototypes feel fast because you're testing with small loops. At scale, 200ms retrieval × 10 loops per decision = 2s slowdown.

Fix:

• Measure latency (P50, P99, P99.9) from day 1
• Co-locate a self-hosted memory server and use a cache-first architecture for tight loops
• Reserve remote DBs for non-latency-critical paths

Example:

// ❌ Wrong: latency adds up
const memory = await remoteVectorDB.query(query); // 200ms
const context = await llm.complete(prompt); // 1000ms
const action = await executeAction(context); // 200ms
// Total: 1.4s per turn

// ✅ Right: cache-first + co-located memory server
const cached = await redis.get(query); // 5ms hit
if (!cached) {
  const retrieved = await brain.retrieve(query); // 30ms
  redis.set(query, retrieved); // Cache
}

2. Poor schema design (too flexible, too rigid)

Mistake: Starting with a single content field and no metadata, or over-indexing every field.

Why it happens: Either "we'll figure it out later" (you won't) or "we might need to filter by anything" (you don't).

Fix:

• Define schema upfront: what fields will you filter by?
• Use JSON metadata for flexible key-value data
• Index only what you query

Example:

// ❌ Wrong: undefined schema
await memory.store({ agentId, content: "something" });

// ✓ Right: explicit schema with metadata
await memory.store({
  agentId: "agent-1",
  namespace: "conversation",
  content: "User asked about pricing",
  type: "user_intent",
  metadata: {
    turn: 5,
    user_id: "u1",
    confidence: 0.95,
  },
  timestamp: new Date(),
});

3. Using semantic retrieval only (no keyword filtering)

Mistake: Embedding everything and relying on semantic similarity. Often returns plausible but wrong results.

Why it happens: Semantic search feels magical at first. You assume it "just works."

Fix:

• Always add keyword filtering (BM25, prefix matching)
• Use hybrid retrieval: semantic score + keyword score + metadata filter
• Validate recall on your actual queries

Example:

// ❌ Wrong: semantic only
const results = await memory.retrieve({
  query: "What did the user ask?",
  topK: 5, // Could return anything vaguely similar
});

// ✓ Right: hybrid
const results = await memory.retrieve({
  query: "What did the user ask?",
  filter: { type: "user_intent", user_id: "u1" },
  topK: 5,
  hybrid: { semantic: 0.6, keyword: 0.4 },
});

4. No eviction policy (unbounded memory growth)

Mistake: Never removing old memories. Costs and retrieval quality degrade over time.

Why it happens: You assume more memory is always better. In practice, old memories become noise.

Fix:

• Set TTLs: short-term (1 hour), mid-term (1 week), long-term (1 month)
• Implement decay: recent memories weight higher
• Monitor memory size and cost per agent

Example:

// ✓ Explicit TTL
const ttl = {
  conversation: 1 * 60 * 60, // 1 hour
  decision: 7 * 24 * 60 * 60, // 1 week
  insight: 30 * 24 * 60 * 60, // 30 days
};

await memory.store({
  content,
  type: "conversation",
  ttl: new Date(Date.now() + ttl.conversation * 1000),
});

5. Storing raw conversations (too verbose)

Mistake: Embedding every message as-is. Results in poor retrieval quality and high costs.

Why it happens: Easiest thing to do. Feels like you're capturing everything.

Fix:

• Summarize conversations: extract intent, action, outcome
• Compress: "User: Hello. Agent: Hi. User: What's the price?" → "User asked about pricing"
• Lazy evaluation: summarize only on retrieval, not on storage

Example:

// ❌ Wrong: raw messages
await memory.store({
  agentId,
  content: "User: What's the price? Agent: Our pricing is...",
});

// ✓ Right: summarized
const summary = await llm.summarize(conversation);
await memory.store({
  agentId,
  content: summary, // "User inquired about pricing for enterprise plan"
  type: "user_intent",
  metadata: { originalLength: conversation.length },
});

6. Embedding drift (changing models without migration)

Mistake: Upgrading your embedding model but not re-embedding old vectors. Queries fail silently.

Why it happens: You test with new embeddings, assume old ones work fine.

Fix:

• Freeze embedding model version in production
• Plan migration path if you upgrade
• Version your embeddings in metadata

Example:

// Store with embedding model version
await memory.store({
  content,
  embedding_model: "text-embedding-3-small:v1",
  embedding_date: new Date(),
});

// Detect drift
const migrationNeeded = memories.filter(
  (m) => m.embedding_model !== CURRENT_MODEL
).length;

7. Cold start latency (no initialization)

Mistake: Initializing embeddings/models on first agent request (adds 500ms–5s latency).

Why it happens: Lazy loading "simplifies" deployment.

Fix:

• Warm up embedding model before agent starts
• Pre-load common queries
• Use managed Cloud to avoid cold starts

Example:

// Warm up before agent starts
async function initializeAgent() {
  const embedding = await embedModel.embed("warmup");
  const memory = new Brain();
  await memory.retrieve({ query: "warmup", topK: 1 });
  // Now ready for traffic
}

await initializeAgent();
startAgent();

8. No monitoring (flying blind)

Mistake: Not measuring retrieval quality, latency, cost. Finding problems in production.

Why it happens: "Monitoring is for later." Later never comes.

Fix:

• Measure latency percentiles (P50, P99, P99.9)
• Track retrieval recall: Is the right memory being retrieved?
• Monitor cost per agent and total
• Set alerts for anomalies

Example:

async function retrieveWithMetrics(query, opts) {
  const start = performance.now();
  const results = await memory.retrieve(query, opts);
  const latency = performance.now() - start;

  analytics.track("memory_retrieve", {
    latency,
    resultCount: results.length,
    query: query.substring(0, 50),
  });

  return results;
}

Bonus: Anti-patterns to avoid

Checklist: Before shipping agent memory

• Latency is < 50ms P99 for agent loop
• Schema is defined and documented
• Hybrid retrieval (semantic + keyword) is implemented
• TTL/eviction policy is set
• Memory is summarized, not raw text
• Embedding model version is pinned
• Cold start is handled (pre-warmed or managed)
• Monitoring is in place (latency, recall, cost)
• You've tested on realistic agent loops (10+ turns)
• You understand cost per agent at your target scale

Learning resources

• Cheat Sheet — Quick reference for schema, retrieval, deployment
• Memory Architectures — System design patterns
• Hybrid Retrieval — Fusion techniques and tuning