Memory Monitoring & Observability

By Arc Labs Research8 min read

The Problem: Blind Memory Systems Fail Silently

Agents with invisible memory systems often degrade unnoticed:

  • Latency creep: Memory retrieval takes 500ms, then 1s, then 3s. Agents time out. You discover it when customers complain.
  • Hit rate collapse: Memory returns stale or irrelevant data. Agents ignore results, behavior becomes stateless again.
  • Junk accumulation: Duplicate, contradictory, or garbage memory entries pile up. Retrieval returns noise instead of signal.
  • Data drift: Older memory entries become outdated but aren't cleaned. Agents prefer stale facts over recent ones.
  • Silent failures: A retrieval hook crashes, but the agent continues with empty context. No alert fires.

Result: Degraded agent quality, customer complaints, and no visibility into why.

Production memory systems need observability: four key metrics, Prometheus-based alerting, structured logging, and dashboards that surface problems before they cascade.

Four Key Metrics

1. P99 Latency (Retrieval Speed)

What it is: The 99th percentile latency of memory retrieval calls (e.g., memory.retrieve()).

Why it matters: Agents are time-bound. A 500ms retrieval timeout means no context, so the agent runs blind.

Threshold: < 200ms for real-time agents, < 1s for batch agents. Alert if P99 exceeds threshold for >2 minutes.

Prometheus metric:

brain_retrieval_duration_ms{percentile="p99", agent_id="...", namespace="..."}

2. Hit Rate (Cache Effectiveness)

What it is: Fraction of retrieval calls that return ≥1 non-empty result.

Why it matters: A hit rate of 30% means 70% of retrievals return empty or unrelated data. Agents lose context.

Threshold: > 70% for stable workloads, > 50% for new/sparse agents. Alert if hit rate drops >20 percentage points.

Prometheus metric:

brain_retrieval_hit_rate{agent_id="...", namespace="...", query_type="semantic|keyword"}

3. Junk Rate (Data Quality)

What it is: Fraction of retrieved results that are marked as "irrelevant" or "stale" by downstream agents/evaluators.

Why it matters: High junk rate = agents ignore memory and run stateless.

Threshold: < 10% junk acceptable, < 5% ideal. Alert if junk rate > 15%.

Prometheus metric:

brain_retrieval_junk_rate{agent_id="...", namespace="...", reason="irrelevant|stale|duplicate"}

4. Drift (Temporal Data Quality)

What it is: Age distribution of retrieved results (e.g., % from last 24h, % from last 7d, % from >30d ago).

Why it matters: If most results are >30 days old, agents are running on stale memory.

Threshold: >60% of results should be <\1 days old for fast-moving domains. Alert if average result age > 14 days.

Prometheus metric:

brain_retrieval_age_days{percentile="p50|p99", agent_id="...", namespace="..."}

Prometheus & Grafana Setup

Client-Side Instrumentation (Python)

from prometheus_client import Histogram, Counter, Gauge
import time
from brain_ai import Brain

# Metrics
retrieval_duration = Histogram(
    'brain_retrieval_duration_ms',
    'Memory retrieval latency in milliseconds',
    ['agent_id', 'namespace', 'query_type'],
    buckets=(50, 100, 200, 500, 1000, 2000),
)

retrieval_hits = Counter(
    'brain_retrieval_hits_total',
    'Successful memory retrievals',
    ['agent_id', 'namespace', 'query_type'],
)

retrieval_misses = Counter(
    'brain_retrieval_misses_total',
    'Failed or empty memory retrievals',
    ['agent_id', 'namespace', 'query_type'],
)

memory_size_bytes = Gauge(
    'brain_memory_size_bytes',
    'Total memory stored for agent',
    ['agent_id', 'namespace'],
)

junk_rate_gauge = Gauge(
    'brain_junk_rate',
    'Fraction of retrieved results marked as irrelevant',
    ['agent_id', 'namespace', 'reason'],
)

result_age_histogram = Histogram(
    'brain_result_age_days',
    'Age of retrieved memory in days',
    ['agent_id', 'namespace'],
    buckets=(0.1, 1, 7, 30, 90),
)

# Memory client
memory = Brain(namespace="myapp")

async def retrieve_with_instrumentation(agent_id: str, query: str, topK: int = 5):
    """Retrieve memory and track metrics."""
    
    start = time.time()
    try:
        results = await memory.retrieve(
            agent_id=agent_id,
            query=query,
            top_k=topK,
        )
        duration_ms = (time.time() - start) * 1000
        
        # Record latency
        retrieval_duration.labels(
            agent_id=agent_id,
            namespace="myapp",
            query_type="semantic",
        ).observe(duration_ms)
        
        # Record hit/miss
        if results:
            retrieval_hits.labels(
                agent_id=agent_id,
                namespace="myapp",
                query_type="semantic",
            ).inc()
            
            # Calculate result age
            import datetime
            now = datetime.datetime.now(datetime.timezone.utc)
            for result in results:
                created = datetime.datetime.fromisoformat(result.get('created_at'))
                age_days = (now - created).days
                result_age_histogram.labels(
                    agent_id=agent_id,
                    namespace="myapp",
                ).observe(age_days)
        else:
            retrieval_misses.labels(
                agent_id=agent_id,
                namespace="myapp",
                query_type="semantic",
            ).inc()
        
        return results
        
    except Exception as e:
        retrieval_misses.labels(
            agent_id=agent_id,
            namespace="myapp",
            query_type="semantic",
        ).inc()
        raise

async def store_with_instrumentation(agent_id: str, content: str, memory_type: str):
    """Store memory and update size gauge."""
    
    await memory.store(
        agent_id=agent_id,
        content=content,
        type=memory_type,
    )
    
    # Update memory size (poll periodically or track incremental)
    size = await get_agent_memory_size(agent_id)  # Custom function
    memory_size_bytes.labels(
        agent_id=agent_id,
        namespace="myapp",
    ).set(size)

async def evaluate_junk_rate(agent_id: str, results: list[dict]):
    """Evaluate retrieved results for relevance."""
    
    junk_count = 0
    for result in results:
        # Call evaluator (LLM, keyword matcher, or manual flag)
        is_relevant = await relevance_evaluator(result, context="...")
        if not is_relevant:
            junk_count += 1
            reason = categorize_junk(result)  # "stale", "duplicate", "irrelevant"
            junk_rate_gauge.labels(
                agent_id=agent_id,
                namespace="myapp",
                reason=reason,
            ).set(junk_count / len(results))
    
    return junk_count / len(results)

Prometheus Configuration (prometheus.yml)

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'brain-agent-1'
    static_configs:
      - targets: ['localhost:8000']  # Your agent's metrics port
        labels:
          agent_id: 'agent-1'
          namespace: 'myapp'

  - job_name: 'brain-agent-2'
    static_configs:
      - targets: ['localhost:8001']
        labels:
          agent_id: 'agent-2'
          namespace: 'myapp'

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']  # Alertmanager

Alert Rules (alerts.yml)

groups:
  - name: brain_alerts
    interval: 30s
    rules:
      # P99 latency spike
      - alert: HighMemoryRetrievalLatency
        expr: |
          histogram_quantile(0.99, brain_retrieval_duration_ms) > 200
        for: 2m
        annotations:
          summary: "Memory retrieval P99 latency > 200ms for {{ $labels.agent_id }}"
          description: "Retrieval is slow. Check memory size, network latency, or Brain service health."

      # Hit rate collapse
      - alert: LowMemoryHitRate
        expr: |
          (brain_retrieval_hits_total / (brain_retrieval_hits_total + brain_retrieval_misses_total)) < 0.5
        for: 5m
        annotations:
          summary: "Memory hit rate < 50% for {{ $labels.agent_id }}"
          description: "Retrievals are mostly empty or unrelated. Check schema, queries, or data freshness."

      # High junk rate
      - alert: HighJunkRate
        expr: |
          brain_junk_rate > 0.15
        for: 5m
        annotations:
          summary: "Memory junk rate > 15% for {{ $labels.agent_id }}"
          description: "Too many irrelevant or stale results returned. Agents are ignoring memory."

      # Data drift (old results)
      - alert: HighResultAge
        expr: |
          histogram_quantile(0.50, brain_result_age_days) > 14
        for: 10m
        annotations:
          summary: "Median retrieved memory age > 14 days for {{ $labels.agent_id }}"
          description: "Agents running on stale memory. Check eviction policy or data ingestion."

Logging Patterns

Structured Logs (JSON)

Log every retrieval with context for later debugging:

{
  "timestamp": "2026-05-13T14:23:45Z",
  "level": "info",
  "event": "memory_retrieve",
  "agent_id": "customer-12345",
  "namespace": "ecommerce",
  "query": "winter jacket preferences",
  "query_type": "semantic",
  "duration_ms": 87,
  "result_count": 3,
  "hit": true,
  "result_ages_days": [0, 1, 5],
  "junk_rate": 0.0,
  "request_id": "req-xyz-789"
}
{
  "timestamp": "2026-05-13T14:24:12Z",
  "level": "warn",
  "event": "memory_retrieve_slow",
  "agent_id": "customer-12345",
  "duration_ms": 1240,
  "threshold_ms": 200,
  "result_count": 0,
  "request_id": "req-abc-456"
}

Log Aggregation (ELK / Loki)

Query logs to debug patterns:

# Find all slow retrievals for an agent
agent_id: "customer-12345" AND event: "memory_retrieve_slow"

# Find retrievals that returned junk
agent_id: "*" AND junk_rate: > 0.10

# Find misses in the last hour
event: "memory_retrieve" AND hit: false AND duration >= -1h

Dashboards: What to Display

Real-Time Health Dashboard

Display these four graphs:

GraphMetricThresholdAction
P99 Latencyhistogram_quantile(0.99, brain_retrieval_duration_ms)Red > 200msCheck memory size, network, service
Hit Ratehits / (hits + misses)Red < 50%Validate queries, check schema matches
Junk Ratejunk / total_retrievedRed > 15%Review data quality, update relevance filter
Median Result Agehistogram_quantile(0.50, brain_result_age_days)Red > 14dReview eviction policy, add fresh data

Per-Agent Breakdown

For each agent, show:

  • Hit rate trend (last 7 days)
  • Latency percentiles (p50, p95, p99)
  • Memory size growth
  • Query types distribution (semantic vs. keyword)

Operational View

  • Alert firing rate (by severity)
  • Mean time to resolution (MTTR) for alerts
  • Agent health score (composite of all 4 metrics)

Implementation Checklist

  • Add Prometheus client library to agent codebase
  • Instrument retrieve() and store() with latency, hit/miss, and result age metrics
  • Deploy Prometheus scraper and configure alerts for P99 > 200ms, hit rate < 50%, junk rate > 15%
  • Set up structured JSON logging with request_id, agent_id, duration, result count, hit flag
  • Create Grafana dashboard with 4-metric health view and per-agent breakdown
  • Route alert notifications to Slack/PagerDuty with runbook links
  • Weekly review: analyze junk rate and drift metrics, adjust eviction/retrieval policies

Next Steps

Related reading

Updates from the lab.

Engineering notes, research drops, occasional product updates. Roughly monthly.