Migrating from Pinecone to Brain

By Arc Labs ResearchMay 13, 20268 min read

The Problem: Pinecone vs Brain

Pinecone is a vector database. You send embeddings and metadata, it indexes them, and serves semantic search. It's built for large external corpora (documents, products, articles).

Brain is agent memory—it's designed to live inside agent loops and store agent-specific state (decisions, outcomes, observations, learned patterns).

They solve different problems. But if your Pinecone index contains agent state (not external data), you should migrate to Brain for:

Hybrid retrieval (keyword + semantic) for agent context
Temporal windows ("all decisions in the last 24 hours")
Type filtering ("errors only" or "successful trades only")
Lower latency for real-time agent loops
Tighter cost for small-to-medium indexes

When NOT to Migrate

Keep Pinecone if:

Your data is external (documents, knowledge base, product catalog)
You have >10M vectors and need distributed search
You need Pinecone's enterprise features (RLS, POD types, namespaces for multi-tenancy)
Semantic search alone is enough (no keyword or temporal filtering)

Migrate to Brain if:

Your vectors represent agent state (decisions, observations, learned patterns)
You need <\100ms latency for agent loops
You want to filter by type, time, or metadata
Your index is <\1M vectors per agent (typical for agent memory)

Schema Translation: Pinecone → Brain

Pinecone vectors look like:

{
  "id": "trade_001",
  "values": [0.12, -0.45, 0.88, ...],
  "metadata": {
    "agent_id": "trader_1",
    "content": "Long BTC @ 62K, exit @ 64K, +$2K",
    "timestamp": "2026-05-12T10:30:00Z",
    "strategy": "bull_bias"
  }
}

Brain memories are:

{
  "content": "Long BTC @ 62K, exit @ 64K, +$2K",
  "agentId": "trader_1",
  "metadata": {
    "type": "trade_outcome",
    "strategy": "bull_bias",
    "pnl": 0.032,
    "timestamp": "2026-05-12T10:30:00Z"
  }
}

Key differences:

Vectors are implicit: Brain computes embeddings from content. You don't store raw vectors.
Content is explicit: The human-readable memory is the primary data; embeddings are derived.
Metadata is typed: Add structured fields (type, strategy, pnl) for filtering and retrieval logic.

Step 1: Export Pinecone Index

Export all vectors and metadata from your Pinecone index:

import json
from pinecone import Pinecone

# Initialize Pinecone
pc = Pinecone(api_key="your_api_key")
index = pc.Index("your_index_name")

# Fetch all vectors (paginate if >100k)
def export_pinecone(index, limit=10000):
    """Export all vectors from Pinecone index"""
    exported = []
    
    # Use describe_index_stats to get vector count
    stats = index.describe_index_stats()
    total_vectors = stats.total_vector_count
    
    print(f"Exporting {total_vectors} vectors...")
    
    # Fetch in batches using list_ids or query
    from pinecone import QueryResponse
    
    for offset in range(0, total_vectors, 100):
        results = index.query(
            vector=[0.0] * 1536,  # Dummy query to list vectors
            top_k=100,
            include_metadata=True,
            offset=offset
        )
        
        for match in results.matches:
            exported.append({
                "id": match.id,
                "values": match.values,
                "metadata": match.metadata or {}
            })
    
    return exported

vectors = export_pinecone(index)

# Save to JSON
with open("pinecone_export.json", "w") as f:
    json.dump(vectors, f, indent=2)

print(f"Exported {len(vectors)} vectors")

Step 2: Transform to Brain Schema

Transform Pinecone vectors to Brain memories. This step converts the vector embedding into a human-readable content field:

import json

def transform_to_brain(pinecone_vectors):
    """Transform Pinecone vectors to Brain schema"""
    brain_memories = []
    
    for vec in pinecone_vectors:
        metadata = vec.get("metadata", {})
        
        # Extract or reconstruct content
        content = metadata.get("content", vec.get("id", ""))
        
        # Map Pinecone metadata to Brain typed metadata
        brain_mem = {
            "content": content,
            "agentId": metadata.get("agent_id", "agent_default"),
            "metadata": {
                "type": metadata.get("type", "observation"),
                "source": "pinecone_migration",
                "imported_at": datetime.now().isoformat(),
            }
        }
        
        # Copy over numeric/structured fields
        if "strategy" in metadata:
            brain_mem["metadata"]["strategy"] = metadata["strategy"]
        if "pnl" in metadata:
            brain_mem["metadata"]["pnl"] = metadata["pnl"]
        if "timestamp" in metadata:
            brain_mem["metadata"]["timestamp"] = metadata["timestamp"]
        if "tags" in metadata:
            brain_mem["metadata"]["tags"] = metadata["tags"]
        
        brain_memories.append(brain_mem)
    
    return brain_memories

# Load exported vectors
with open("pinecone_export.json", "r") as f:
    vectors = json.load(f)

# Transform
brain_mems = transform_to_brain(vectors)

# Save transformed schema
with open("brain_import.json", "w") as f:
    json.dump(brain_mems, f, indent=2)

print(f"Transformed {len(brain_mems)} memories")

Step 3: Run Migration Script

This Python script handles the full Pinecone → Brain migration:

import json
import requests
from datetime import datetime
from pinecone import Pinecone

# Pinecone setup
PINECONE_API_KEY = "your_pinecone_api_key"
PINECONE_INDEX = "your_index_name"

# Brain setup
BRAIN_API_KEY = "your_brain_api_key"
BRAIN_BASE_URL = "https://api.brain.cloud"

def export_from_pinecone(index_name: str) -> list:
    """Export all vectors from Pinecone"""
    pc = Pinecone(api_key=PINECONE_API_KEY)
    index = pc.Index(index_name)
    
    exported = []
    stats = index.describe_index_stats()
    total = stats.total_vector_count
    
    print(f"Exporting {total} vectors from Pinecone...")
    
    # Batch export
    for offset in range(0, total, 100):
        results = index.query(
            vector=[0.0] * 1536,
            top_k=100,
            include_metadata=True,
            offset=offset
        )
        
        for match in results.matches:
            exported.append({
                "id": match.id,
                "values": match.values,
                "metadata": match.metadata or {}
            })
        
        if (offset + 100) % 1000 == 0:
            print(f"  Exported {offset + 100}/{total}...")
    
    return exported

def transform_to_brain_schema(pinecone_vectors: list) -> list:
    """Transform to Brain schema"""
    brain_mems = []
    
    for vec in pinecone_vectors:
        metadata = vec.get("metadata", {})
        
        # Reconstruct human-readable content from metadata
        if "content" in metadata:
            content = metadata["content"]
        else:
            # Fallback: summarize metadata
            content = f"{metadata.get('type', 'observation')}: {json.dumps(metadata)}"
        
        brain_mem = {
            "content": content,
            "agentId": metadata.get("agent_id", "agent_default"),
            "metadata": {
                "type": metadata.get("type", "observation"),
                "source": "pinecone_migration",
                "imported_at": datetime.now().isoformat(),
            }
        }
        
        # Preserve structured fields
        for key in ["strategy", "pnl", "timestamp", "tags", "category", "status"]:
            if key in metadata:
                brain_mem["metadata"][key] = metadata[key]
        
        brain_mems.append(brain_mem)
    
    return brain_mems

def import_to_brain(memories: list) -> tuple:
    """Import memories to Brain Cloud"""
    headers = {
        "Authorization": f"Bearer {BRAIN_API_KEY}",
        "Content-Type": "application/json"
    }
    
    imported_count = 0
    failed_count = 0
    failed_memories = []
    
    print(f"Importing {len(memories)} memories to Brain...")
    
    for i, memory in enumerate(memories):
        try:
            response = requests.post(
                f"{BRAIN_BASE_URL}/v1/memories",
                json=memory,
                headers=headers,
                timeout=10
            )
            response.raise_for_status()
            imported_count += 1
        except Exception as e:
            print(f"Failed to import memory: {memory['content'][:50]}... Error: {e}")
            failed_count += 1
            failed_memories.append((memory, str(e)))
        
        if (i + 1) % 100 == 0:
            print(f"  Imported {i + 1}/{len(memories)}...")
    
    return imported_count, failed_count, failed_memories

# Run migration
print("Starting Pinecone → Brain migration...")
print("=" * 50)

# Export from Pinecone
pinecone_vectors = export_from_pinecone(PINECONE_INDEX)
print(f"✓ Exported {len(pinecone_vectors)} vectors from Pinecone")

# Transform schema
brain_memories = transform_to_brain_schema(pinecone_vectors)
print(f"✓ Transformed to Brain schema")

# Import to Brain
imported, failed, failed_list = import_to_brain(brain_memories)
print(f"✓ Import complete: {imported} succeeded, {failed} failed")

# Save failed memories for manual review
if failed_list:
    with open("migration_failures.json", "w") as f:
        json.dump(
            [{"memory": mem, "error": err} for mem, err in failed_list],
            f,
            indent=2
        )
    print(f"⚠ {failed} failures saved to migration_failures.json")

print("=" * 50)
print(f"Migration complete! {imported}/{len(brain_memories)} memories imported successfully.")

Step 4: Update Your Code

Before (Pinecone):

from pinecone import Pinecone

pc = Pinecone(api_key="api_key")
index = pc.Index("agent_memory")

# Store
vector = embedding_model.embed("User decision: bought AAPL")
index.upsert([("decision_001", vector, {"agent_id": "trader_1", "type": "decision"})])

# Retrieve
query_vec = embedding_model.embed("What decisions did I make?")
results = index.query(query_vec, top_k=5, include_metadata=True)

After (Brain):

from brain_sdk import Brain

brain = Brain(api_key="api_key")

# Store (embeddings computed automatically)
brain.store(
    agentId="trader_1",
    content="User decision: bought AAPL",
    metadata={"type": "decision"}
)

# Retrieve (hybrid: keyword + semantic)
results = brain.retrieve(
    agentId="trader_1",
    query="What decisions did I make?",
    topK=5,
    mode="hybrid"
)

Cost Comparison: Pinecone vs Brain

Assume: 500K vectors (agent memories), 10K retrievals/month

Service	Cost per Month	Notes
Pinecone (s1)	$70–$ 200	$0.084/1M vectors/month +$ 0.084/1M queries
Brain Cloud	$50–$ 150	$0.001/vector +$ 0.0005/retrieval
Brain Open Source	$0	Self-host; pay for infra only

Savings: 25–40% reduction + lower operational overhead with Brain.

Migration Checklist

Identify which Pinecone vectors are agent state (not external data)
Export all vectors from Pinecone (use script above)
Review exported metadata; identify type, strategy, timestamp, and other structured fields
Transform vectors to Brain schema (reconstruct content from metadata if needed)
Set up Brain Cloud account or self-hosted instance
Run migration script to import memories
Verify sample: retrieve 10 random memories, confirm content and metadata accuracy
Update agent code: replace Pinecone upsert/query with Brain store/retrieve
Test retrieval quality: compare Brain results to Pinecone for the same queries
Monitor latency: confirm Brain is <\100ms P99 for your agent loop
Decommission Pinecone index after validation period

Why Brain Wins for Agent Memory

Hybrid retrieval: Pinecone searches by vectors only. Brain combines keyword (BM25) and semantic, so "retrieve all decisions from yesterday" works without custom filtering.

Metadata filtering: Brain's metadata is fully queryable. Filter by type="decision", strategy="bull_bias", or timestamp > "2026-05-01".

Temporal windows: Brain natively supports time-based retrieval. "Last 24 hours" is a parameter, not a post-retrieval filter.

Lower cost for agent memory: Pinecone optimizes for massive scale (100M+ vectors). Brain is right-sized for agent loops (100K–5M vectors).

Next Steps

Learn more: See Migrating from Mem0 for another migration pattern
Hybrid retrieval: Check Five Retrievers for a deep dive into Brain's retrieval modes
Build: Read Memory Architectures to design agent memory systems with Brain