Production Deployment Checklist

The Problem: Untested Deployments Fail at Scale

Deploying memory system changes without a checklist leads to:

Silent schema breaks: New code expects a field that old entries don't have. Queries silently return empty. Hit rate collapses at midnight when batch loads new data.
Missing alerts: Monitoring isn't wired up, so performance degradation goes unseen until users complain.
Ungraceful rollback: You need to revert, but you don't have a baseline or a fast rollback plan. Downtime extends from minutes to hours.
Customer impact asymmetry: New memory gets stored in the new schema. Old queries fail on old data. Mixed workload breaks.

Production memory deployments need three phases: pre-deploy verification, deploy-day safety measures, and post-deploy monitoring for the first 60 minutes.

Phase 1: Pre-Deployment (48 Hours Before)

Schema Validation

Schema compatibility check: Does new code read old memory entries without errors? Test with 10% sample of prod data in staging.
Backward compatibility for reads: Old agents can still query memory stored in the old schema (no breaking renames or type changes).
Migration script (if needed): If schema is incompatible, write migration. Test on staging data first. Estimate runtime on prod volume.
Rollback schema: Does old code read new memory entries? If new schema is incompatible backward, ensure old code won't crash on new data post-rollback.

Example:

// OLD schema: { type: string, content: string }
// NEW schema: { type: string, content: string, source: string }

// ❌ This breaks backward compatibility:
// const source = entry.source.toUpperCase(); // undefined.toUpperCase() crashes

// ✓ This works:
// const source = (entry.source || "unknown").toUpperCase();

Monitoring & Alerting Setup

Prometheus metrics deployed: Latency, hit rate, junk rate, drift metrics scraping from staging.
Alert rules created: P99 latency > 200ms, hit rate < 50%, junk rate > 15%, result age > 14d.
Grafana dashboards live: 4-metric health dashboard + per-agent breakdown accessible.
Log aggregation running: ELK / Loki / CloudWatch collecting structured logs with request_id, agent_id, duration, hit flag.
Slack/PagerDuty notifications tested: Send test alert, verify it reaches on-call.

Load Testing

Staging load test: Replay 10% of prod traffic to staging, run for 1 hour, verify P99 latency < 200ms and hit rate > 70%.
Peak load test: Simulate 2x peak traffic (e.g., 1000 req/s if prod is 500 req/s), verify no degradation.
Memory size estimate: Will memory usage grow? Estimate new memory footprint; confirm it fits within allocated quota.

Rollback Plan

Rollback procedure documented: Can you revert in <\1 minutes? Specify exact steps: git revert, redeploy, monitor metrics.
Baseline metrics captured: Screenshot current P99 latency, hit rate, junk rate for comparison post-deployment.
Incident runbook created: If alerts fire, what do you do? Link runbook in Slack alert template.

Phase 2: Deploy Day (Deployment + 2 Hours After)

Pre-Deployment

Team availability: On-call engineer watching Slack. No deploying on Friday afternoon.
Production backup taken: Database snapshot, memory backup (if applicable).
Change log drafted: What changed? Why? How to rollback? Share in #deployments.

Canary Deployment

Canary size: Roll out to 5–10% of agents/traffic first, monitor for 10 minutes.
Canary metrics monitored: Watch P99 latency, hit rate, error rate in real-time. Alert on any regression.
Canary baseline established: Record P99 latency and hit rate for canary cohort at T+0.

Full Rollout (If Canary Passes)

Full rollout staged: Deploy to 50% of fleet, monitor 10 minutes. Then 100%.
Metrics monitored continuously: Watch all 4 metrics (latency, hit rate, junk, drift) for 60 minutes post-deploy.
Error rate tracked: Any spikes in 5xx errors or timeout errors?

Rollback Triggers

If any of these fire in the first 60 minutes, rollback immediately:

Metric	Threshold	Action
P99 Latency	> 200ms (or 50% higher than baseline)	Rollback
Hit Rate	< 50% (or 20 percentage points lower than baseline)	Rollback
Error Rate	> 1% (or 10x higher than baseline)	Rollback
Memory Query Timeouts	> 1% of requests	Rollback

Fast rollback:

# 1. Identify last stable commit
git log --oneline | head -5

# 2. Revert
git revert <deploy-commit-hash>

# 3. Redeploy
./deploy.sh production

# 4. Verify metrics return to baseline within 5 minutes
# P99 latency < 100ms? Hit rate > 70%? Alert firing?

Phase 3: Post-Deployment (First 24 Hours)

Hour 0–1 (Immediate Post-Deploy)

Metrics steady at baseline: P99 latency back to <\150ms, hit rate > 70%, no junk spike, no result age drift.
No customer complaints in Slack/support: Agents performing normally, no "Why is memory slow?" reports.
All alerts cleared: No firing alerts; no p99 latency or hit rate warnings.

Hours 1–4

Spot-check agent behavior: Run sample queries manually. Do results look correct? Are they fresh?
Monitor junk rate: Run a junk-rate evaluator on 100 random retrievals. Junk rate < 10%?
Data freshness: Check result age distribution. >60% < 7 days old?

Hours 4–24

Aggregated metrics reviewed: Average latency, P95/P99 stable? Hit rate consistent?
No silent regressions: Compare to previous 7-day baseline. Any metric 10% worse?
Logs analyzed: Any recurring error patterns or slow-query outliers?

Post-Deployment Report

Document in #deployments:

**Deploy Summary: Memory Schema v2.1**
- Deployed: 2026-05-13 14:30 UTC
- Rollback: No rollbacks needed
- Canary (5% cohort): 10 min, all metrics green
- Full rollout: Completed 14:45 UTC

**Post-Deploy Metrics (vs. Baseline)**
| Metric | Baseline | 1hr Post | 4hr Post | 24hr Post |
|---|---|---|---|---|
| P99 Latency | 95ms | 92ms | 91ms | 94ms |
| Hit Rate | 73% | 74% | 73% | 72% |
| Junk Rate | 8% | 8% | 7% | 7% |
| Median Result Age | 4d | 4d | 4d | 4d |

**Notes**: No rollbacks. All metrics at or above baseline. Zero customer complaints.
**Signed**: @on-call-engineer

Decision Table: Deployment Priority Levels

Use this table to prioritize what to test and how long to canary:

Change Type	Example	Risk	Canary Time	Rollback Speed	Priority
High	New memory type, schema migration, retrieval algorithm	Can break queries for all agents	30 min, monitor 4 metrics closely	<\1 min if P99 > 300ms	CRITICAL: Full testing required
Medium	New metadata field (backward compatible), improved eviction logic, minor query optimization	Possible impact on subset of agents; unlikely to break all	15 min, monitor P99 latency	<\10 min if hit rate drops	Required: Load test + basic canary
Low	Logging improvements, metric additions, config tweaks, documentation updates	No impact on retrieval behavior	5 min, spot-check one metric	<\1 min, deploy with confidence	Optional: Spot-check only

Deployment Checklist

Pre-Deploy (48 Hours Before)

Schema backward compatibility verified on 10% prod sample
Migration script tested (if needed)
Prometheus alerts created: P99 > 200ms, hit rate < 50%, junk > 15%, age > 14d
Grafana dashboards live and tested
Log aggregation running
Slack alerts tested
Staging load test passed: P99 < 200ms, hit rate > 70%
2x peak load test passed
Rollback plan documented
Baseline metrics captured

Deploy Day

Team available, no Friday afternoon deploys
Database backup taken
Change log drafted and shared
Canary deployed to 5–10%, monitored 10 min
P99 latency and hit rate stable during canary
Full rollout staged (50% → 100%) with 10 min pauses
All 4 metrics monitored for 60 min post-deploy
Rollback triggers defined and watched

Post-Deploy (First 24 Hours)

Hour 0–1: Metrics at baseline, no customer complaints
Hours 1–4: Agent behavior spot-checked, junk rate < 10%
Hours 4–24: Aggregated metrics reviewed, no silent regressions
Post-deploy report filed in #deployments

Next Steps

Monitor: See Memory Monitoring & Observability for alerting rules and dashboard setup
Optimize: Check Cost Optimization for Agent Memory after deployment to review resource usage
Integrate: Learn Choosing Your Memory Schema for schema design best practices