Self-Hosted vs Managed Cloud

By Arc Labs Research7 min read

The Problem: Wrong Deployment Kills Economics or Performance

Brain ships in exactly two forms: the open-source Rust server you self-host, and managed cloud. There is no in-process or on-device library — every client speaks the wire protocol to a running server. The choice, then, is where that server runs and who operates it. Choosing wrong cascades into problems:

  • Single node too small: Scaled a single self-hosted instance to 10,000 agents. One node can't absorb the concurrent write load. Latency spikes to 5s. Agents timeout.
  • Cloud too expensive: Running 100,000 queries/day at 0.005/query=0.005/query = 150/month just for memory. 2x other infrastructure costs.
  • HA cluster too much ops: You hired DevOps to run a multi-node cluster. Kubernetes for redundancy. Scaling headaches.
  • Low latency requirement unmet: Chose cloud (50–200ms), but your agent needs <10ms. Customer experience suffers.

The right deployment model balances latency SLA, cost-per-query, ops overhead, and data sovereignty.

Comparison Table: All Dimensions

DimensionSelf-Hosted (single node)Cloud (Managed)Self-Hosted (HA cluster)
Latency (p99)<10ms (co-located)50–200ms30–100ms
Cost per query$0.000.001–0.001–0.01$0.0001 (+ infra)
Monthly infra cost$0 (on your servers)100–100–5,000 (scales with volume)1,000–1,000–10,000 (fixed + scale)
Concurrent agents≤1,000 per instance1,000–100,000+10,000–1M+
Scaling strategyShard across instancesManaged by providerYour Kubernetes/load balancer
Data sovereignty100% (your hardware)Provider-hosted100% (your hardware)
Compliance (HIPAA/SOC2)Your responsibilityBrain certifiedYour responsibility
Backup/HAYour responsibilityManaged + 99.99% SLAYour responsibility
Schema migrationsYou manageBrain managesYou manage
Ops overheadLow (single instance)Minimal (provider-managed)High (dedicated team)
Time to scale 10xWeeks (re-architect)Hours (provider scales)Hours (spin infra)
Biggest riskOutage = total lossVendor lock-inOperational burden

Decision Flowchart

START: Estimate your agent population and query rate

1. How many agents do you have?
   ├─ <1,000 agents
   │  └─ Go to Q2
   │
   ├─ 1,000–50,000 agents
   │  └─ LIKELY CLOUD (see Q3 for exceptions)
   │
   └─ >50,000 agents
      └─ LIKELY SELF-HOSTED HA CLUSTER or ENTERPRISE CLOUD

2. (For <1,000 agents) How strict is your latency SLA?
   ├─ <10ms (real-time agent, trading, robotics)
   │  └─ SELF-HOST A SINGLE NODE (co-located with your agents)
   │
   ├─ 10–50ms (acceptable latency, good UX)
   │  └─ CLOUD or SINGLE-NODE SELF-HOST (cloud if you want to scale later)
   │
   └─ >50ms (batch, async, not latency-critical)
      └─ CLOUD (lower ops burden)

3. (For 1,000–50,000 agents) Do you have strict data sovereignty or compliance needs?
   ├─ YES (HIPAA, GDPR, China region, on-premise required)
   │  └─ SELF-HOSTED (Brain cannot host for you)
   │
   └─ NO
      └─ CLOUD (Brain managed)

4. (For >50,000 agents) What is your cost sensitivity?
   ├─ Cost is primary concern (>$5,000/month matters)
   │  └─ SELF-HOSTED HA CLUSTER (lower per-query cost)
   │
   └─ Cost secondary, want ops simplicity
      └─ ENTERPRISE CLOUD (negotiate volume pricing with Brain)

Cost Calculator: Which Deployment is Cheapest?

interface DeploymentCosts {
  agents: number;
  queries_per_day_per_agent: number;
  memory_gb_per_agent: number;
}

function estimate_monthly_cost(config: DeploymentCosts) {
  const { agents, queries_per_day_per_agent, memory_gb_per_agent } = config;
  
  const total_queries_per_month = agents * queries_per_day_per_agent * 30;
  const total_storage_gb = agents * memory_gb_per_agent;
  
  // SELF-HOSTED (single node)
  const single_node_server_cost = 200; // Small instance on your infra
  const single_node_total = single_node_server_cost;
  
  // CLOUD (Brain)
  const cloud_query_cost = total_queries_per_month * 0.005; // $0.005 per query
  const cloud_storage_cost = total_storage_gb * 0.10; // $0.10 per GB/month
  const cloud_total = cloud_query_cost + cloud_storage_cost;
  
  // SELF-HOSTED (HA cluster)
  const ha_cluster_server_cost = 2000; // Multi-node Brain cluster + load balancer
  const ha_cluster_query_cost = total_queries_per_month * 0.0001; // $0.0001 per query (amortized compute)
  const ha_cluster_storage_cost = total_storage_gb * 0.05; // $0.05 per GB/month
  const ha_cluster_total = ha_cluster_server_cost + ha_cluster_query_cost + ha_cluster_storage_cost;
  
  return {
    agents,
    total_queries_per_month,
    total_storage_gb,
    single_node: {
      server: single_node_server_cost,
      total: single_node_total,
    },
    cloud: {
      queries: cloud_query_cost,
      storage: cloud_storage_cost,
      total: cloud_total,
    },
    ha_cluster: {
      server: ha_cluster_server_cost,
      queries: ha_cluster_query_cost,
      storage: ha_cluster_storage_cost,
      total: ha_cluster_total,
    },
  };
}

// Example: 5,000 agents, 500 queries/agent/day, 100 MB/agent
const costs = estimate_monthly_cost({
  agents: 5000,
  queries_per_day_per_agent: 500,
  memory_gb_per_agent: 0.1,
});

console.log(costs);
// agents: 5000,
// total_queries_per_month: 75,000,000
// total_storage_gb: 500
//
// single_node: { server: 200, total: 200 } ← INFEASIBLE (latency, scale)
// cloud: { queries: 375,000, storage: 50, total: 375,050 }
// ha_cluster: { server: 2000, queries: 7,500, storage: 25, total: 9,525 }
//
// Winner: self-hosted HA cluster (97% cheaper!)

Cost Comparison Examples

Scenario 1: 100 agents, 100 queries/day/agent, 50 MB/agent

Single node:      $200/month (2 small instances for redundancy)
Cloud:            $1,500/month (1.5M queries × $0.005 + 5 GB storage)
Self-Hosted HA:   $2,200/month (not worth the overhead)
→ CHOOSE: Single-node self-host (simplest, cheapest)

Scenario 2: 5,000 agents, 500 queries/day/agent, 100 MB/agent

Single node:      INFEASIBLE (too many agents)
Cloud:            $375,000/month (75M queries × $0.005 + 500 GB storage)
Self-Hosted HA:   $9,500/month
→ CHOOSE: Self-Hosted HA cluster (40x cheaper than cloud)

Scenario 3: 10,000 agents, 200 queries/day/agent, 50 MB/agent, strict latency SLA (<10ms)

Single node:      INFEASIBLE (scale + latency)
Cloud:            $30,000/month (can meet latency)
Self-Hosted HA:   $7,500/month + ops team ($50k/yr engineer)
→ CHOOSE: Cloud (ops simplicity, latency SLA guaranteed)

Use Case Matrix

Use CaseAgentsLatency SLAData SovereigntyBest ChoiceRationale
Small prototype<100FlexibleNoSingle-node self-hostFree, simple, no scaling yet
Small SaaS (early)100–1,000<100msNoSingle node or CloudSingle node if cheap; Cloud if want to scale later
Trading system10–100<10msNoSingle-node self-hostLatency critical, small fleet
Mid-market SaaS1,000–10,000<100msNoCloudSweet spot for Brain managed
Enterprise SaaS10,000–50,000<100msNoCloudBrain handles scale, you focus on product
High-frequency ops1,000–10,000<20msNoSelf-Hosted HALatency + cost both matter
HIPAA/compliant1,000–100,000<100msYESSelf-HostedOn-premise required
Data-sovereign (China)1,000–10,000<100msYES (local)Self-Hosted + local infraMust be in-region
Cost-sensitive scale>50,000<100msNoSelf-Hosted HAPer-query cost dominates

Implementation Guide: Choosing & Deploying

Decision Checklist

  • Estimate agent count: How many agents will you run in 1 year? 5 years?
  • Define latency SLA: What is your p99 latency requirement? <10ms, <50ms, <200ms?
  • Check data sovereignty: Do you need on-premise deployment, HIPAA compliance, or GDPR compliance?
  • Calculate cost breakeven: For your agent count and query rate, which deployment is cheapest?
  • Ops capacity: Can your team maintain self-hosted infra, or do you prefer managed?
  • Scaling timeline: When do you expect to hit scale limits? Plan migration path.

Self-Hosted (single node)

Run the open-source Brain server co-located with your agents, then point the SDK at it. The server owns storage, indexing, and the embedding model; your client only speaks the wire protocol.

# Run the open-source Brain server (Docker)
docker run -d --name brain \
  --security-opt seccomp=unconfined \
  -p 9090:9090 \
  -e BRAIN__LLM__API_KEY=$OPENAI_API_KEY \
  -v $PWD/brain-data:/data \
  arc-labs/brain-server:latest
import { Brain } from "brain-ai";

// Connect the SDK to your locally-run server
const memory = new Brain({
  serverUrl: "http://localhost:9090",
  namespace: "production",
});

// Retrieval is local, fast (<10ms co-located)
const results = await memory.retrieve({
  agentId: "agent-123",
  query: "preferences",
  topK: 5,
});
// Latency: <10ms

Cloud Deployment (Brain Managed)

import { Brain } from "brain-ai";

const memory = new Brain({
  apiKey: process.env.BRAIN_API_KEY,
  apiUrl: "https://api.brain.arc-labs.ai",
  namespace: "production",
});

// No infrastructure to manage; Brain scales automatically
const results = await memory.retrieve({
  agentId: "agent-123",
  query: "preferences",
  topK: 5,
});
// Latency: 50–150ms (network round-trip)

Self-Hosted (HA cluster)

Scale out by running multiple Brain server instances behind a load balancer, sharded across nodes. Same open-source binary as the single-node case — you own backups, HA, and capacity planning.

# Each node runs the same server binary; shard traffic across them
docker run -d --name brain-node-1 \
  --security-opt seccomp=unconfined \
  -p 9090:9090 \
  -e BRAIN__LLM__API_KEY=$OPENAI_API_KEY \
  -e BRAIN__CLUSTER__SHARDS=16 \
  -v /mnt/brain-1:/data \
  arc-labs/brain-server:latest
# ...repeat per node, front with your load balancer
import { Brain } from "brain-ai";

// Point the SDK at your load balancer in front of the cluster
const memory = new Brain({
  serverUrl: process.env.BRAIN_CLUSTER_URL, // e.g. https://brain.internal
  namespace: "production",
});

// Retrieval goes to your cluster; latency depends on your infra
const results = await memory.retrieve({
  agentId: "agent-123",
  query: "preferences",
  topK: 5,
});
// Latency: 30–100ms (depends on your network + node tuning)

Migration Paths

Self-Hosted → Cloud

When your agent fleet grows beyond 1,000:

  1. Phase 1: Keep a single self-hosted node for latency-critical agents, add Cloud for bulk agents.
  2. Phase 2: Migrate all agents to Cloud, retire the self-hosted node.
  3. Cost: ~$5,000/month at scale, but you stop managing infra.
# At scale, all agents point to cloud API
export BRAIN_API_KEY=your-key
export BRAIN_API_URL=https://api.brain.arc-labs.ai

# No local server, no ops burden

Cloud → Self-Hosted

When Cloud costs exceed $50,000/month:

  1. Phase 1: Negotiate volume discount with Brain.
  2. Phase 2: If discount insufficient, evaluate a self-hosted HA cluster.
  3. Migration: Export data, stand up the Brain server cluster, import, switch the SDK endpoint.
# Export from Cloud
brain-cli export --format json > memory-export.json

# Import to your self-hosted cluster
brain-cli import --server http://localhost:9090 < memory-export.json

# Update client
export BRAIN_CLUSTER_URL=https://brain.internal

Monitoring Deployment Choice

Track these metrics to validate your choice:

Deployment Health Dashboard
├─ Latency P99: <10ms (single node), <200ms (cloud), <100ms (HA cluster)
├─ Hit Rate: >70%
├─ Cost per Query: <$0.00 (single node), $0.001–$0.01 (cloud), <$0.0001 (HA cluster)
├─ Ops Hours/Month: 5h (single node), 0.5h (cloud), 40h (HA cluster)
├─ Uptime: 99%+
└─ Agent Satisfaction: >90%

If any metric drifts, revisit your deployment choice.

Next Steps

Related reading

Updates from the lab.

Engineering notes, research drops, occasional product updates. Roughly monthly.