Lessons from building Brain in Rust

We built Brain's core in Rust. This is a blunt account of why, what worked, and what we'd reconsider.

Why Rust

Three properties drove the choice:

• Predictable latency. Memory retrieval sits on the critical path of every agent turn. Garbage-collected runtimes introduce p99 spikes that compound across a multi-retriever fan-out. Rust's deterministic deallocation gives us flat tail latency.
• Zero-copy serialization. Vectors are large; copying them between layers adds up. Rust's ownership model lets us pass references through the pipeline without paying for serialization at every stage.
• Cross-language SDKs. A Rust core compiles cleanly into both Node (via napi-rs) and Python (via PyO3). One codebase; native bindings everywhere.

What worked well

napi-rs is the right Node binding story in 2026

We tried two paths: classic N-API via node-addon-api and napi-rs. The latter was the right call. Build is dramatically faster, the type ergonomics are real Rust types, and the resulting SDK feels like idiomatic TypeScript without compromise.

#[napi]
pub struct Brain {
  inner: Arc<BrainCore>,
}

#[napi]
impl Brain {
  #[napi]
  pub async fn write(&self, args: WriteArgs) -> Result<WriteResult> {
    self.inner.write(args.into()).await
      .map(Into::into)
      .map_err(napi_err)
  }
}

Async functions on the Rust side become real Promise<T> in TypeScript. No callback bridging. No ceremony.

PyO3 is solid; the GIL is the gotcha

The Python bindings via PyO3 work, but the GIL costs us throughput in batch operations. Our workaround: release the GIL aggressively before any Rust-internal compute, re-acquire only at the boundary back to Python.

#[pyfunction]
fn write_batch(py: Python, items: Vec<PyTurn>) -> PyResult<Vec<PyResult>> {
  let rust_items: Vec<Turn> = items.into_iter().map(Into::into).collect();
  py.allow_threads(|| {
    // Heavy compute happens here without the GIL held
    runtime.block_on(core.write_batch(rust_items))
  })
  .map(|res| res.into_iter().map(Into::into).collect())
  .map_err(into_py_err)
}

rkyv + bytemuck give us zero-copy storage

Brain runs its own storage engine — a memory-mapped vector arena, a write-ahead log with group-commit fsync, and a redb B-tree for metadata — not an external database. Records serialize with rkyv and cast with bytemuck, so hot reads deref straight out of the mmap with no deserialization copy. CRC32C on every WAL record and arena slot catches corruption at read time and fails stop.

Performance characteristics we measured

We don't publish performance claims without measuring them. Here's what our internal benchmarks showed:

Write pipeline throughput: 380 candidates/second on a single 16-core ARM machine (no GPU). The bottleneck is the extraction LLM call, not the Rust pipeline — the pipeline itself can process > 10,000 candidates/second when extraction is mocked out. The practical limit is LLM throughput.

Retrieval latency breakdown (5-retriever fan-out, N=1M memories, p99 measured over 10,000 queries):

The fan-out is fully concurrent: all five retrievers run in a single tokio::join! block. The P99 total is dominated by HNSW (the slowest retriever) plus serialization overhead, not by any sequential execution.

GC pause comparison: we ran an equivalent Python implementation of the same five-retriever fan-out and measured p99 latency at 340ms — 4.4x slower than the Rust path. Most of the gap is GC pauses compounded across the fan-out: each retriever runs in Python's GIL context, and a major GC cycle mid-fan-out inflates tail latency dramatically. Rust's deterministic deallocation eliminates this tail.

Memory footprint: the Brain server process uses ~18MB at startup, growing proportional to the HNSW warm buffer (configured separately, default 64MB). For a typical self-hosted deployment this is negligible; on memory-constrained hosts, the warm buffer can be disabled.

The HNSW implementation decision

We considered four options for approximate nearest-neighbor search:

1. Pure Rust, in-process (hnsw_rs, usearch): maximum control, we maintain the index ourselves
2. pgvector: Postgres extension, mature, SQL-compatible, but couples us to Postgres ops
3. Qdrant: dedicated vector database, fast, but adds a network boundary and a separate service
4. Weaviate/Pinecone: managed services, adds latency and external dependency

We chose the pure-Rust, in-RAM index (hnsw_rs). The reasoning:

Durability is our own, not the index's. The 7-stage write pipeline makes decisions (deduplicate, supersede, persist) that must be atomic. We get that from Brain's own write path: single-writer-per-shard plus WAL-before-acknowledge — nothing returns success until its WAL record is fsynced, and recovery replays the log. That means we don't need a SQL transaction boundary around the vector store, so the vector index can live in RAM purely for speed.

Keeping HNSW in the process removes the network boundary. An external vector DB (pgvector or Qdrant) puts a socket, serialization, and a separate failure domain between the retriever and the index. An in-process hnsw_rs index in each shard is a function call, which is what makes the sub-10ms p50 achievable across a five-retriever fan-out.

Operational simplicity: there is no second service to run, scale, or back up. The whole shard — arena, WAL, redb, HNSW, tantivy — is one process with one data directory.

We scale by sharding (thread-per-core Glommio executors, each owning its own indexes), not by moving the embedding layer to an external vector database. Adding capacity means adding shards, not adopting a new storage system.

Lessons from the N-API boundary

The TypeScript SDK is a thin wrapper over the Rust binary via napi-rs. This boundary has some surprising properties:

Async semantics work cleanly: napi-rs maps Rust async fn to JavaScript Promise<T>. The tokio runtime runs on a dedicated thread pool; JavaScript awaits are genuine non-blocking suspensions. The Node event loop is not blocked during any Rust operation.

// In Rust: async fn that doesn't block Node's event loop
#[napi]
impl Brain {
  #[napi]
  pub async fn search(&self, args: SearchArgs) -> Result<Vec<MemoryResult>> {
    // Runs on tokio threadpool; Node event loop is free
    self.inner.search(args.into()).await
      .map(|results| results.into_iter().map(Into::into).collect())
      .map_err(napi_err)
  }
}

Error propagation requires explicit mapping: Rust errors are typed (BrainError::NotFound, BrainError::Conflict, etc.). JavaScript errors are Error objects with string messages. napi-rs provides a napi::Error type, but you have to write the mapping:

fn napi_err(e: BrainError) -> napi::Error {
  match e {
    BrainError::NotFound(id) => {
      napi::Error::from_reason(format!("Memory not found: {}", id))
    }
    BrainError::Conflict { memory_id, conflict_with } => {
      napi::Error::from_reason(
        format!("Conflict: {} contradicts {}", memory_id, conflict_with)
      )
    }
    BrainError::DatabaseError(db_err) => {
      // Don't leak internal DB state to JavaScript callers
      napi::Error::from_reason("Internal storage error")
    }
    _ => napi::Error::from_reason(e.to_string()),
  }
}

This mapping is where we've caught most of our API design mistakes. When you write the Rust-to-JavaScript error conversion, you're forced to think about what the JavaScript caller can actually do with each error type. BrainError::Conflict should surface enough information for the caller to retry with a merge strategy; BrainError::DatabaseError should not surface internal state.

Type generation is real but incomplete: napi-rs generates TypeScript .d.ts files from the Rust code. The generated types are accurate for primitive arguments, but complex nested structs require hand-written type augmentations. We generate the base types and then maintain a types.ts with the augmentations — a small but real maintenance surface.

Testing the boundary is the hardest part: Rust unit tests can't test the napi boundary (they're pure Rust). The napi binary can't be imported into a Node test runner without the full build. We ended up with three test layers: Rust unit tests (90% of coverage), a small Rust integration test suite that mocks the LLM calls, and a TypeScript integration test suite that tests the JavaScript API surface against a real Rust binary. The TypeScript tests caught the most boundary-specific bugs.

What we'd reconsider

Async story is heavier than it needs to be

The Rust async ecosystem is great in production but heavyweight to learn. Send bounds, lifetime gymnastics inside async fn, and the runtime split (tokio vs async-std vs smol) cost new contributors weeks of setup. If we were starting today with a smaller team, we might prototype in Go and migrate later.

Error type design needs more discipline

Rust's Result<T, E> is wonderful — until you have ten error types and every binding boundary requires explicit conversions. We've consolidated to a single BrainError enum at the public API; internally, thiserror keeps things readable.

Testing the bindings is harder than testing the core

Rust unit tests are great. Testing that the napi-rs binding behaves correctly under all the JavaScript / async / error edge cases is a separate effort. We invested heavily in TypeScript-side integration tests run against the actual binary; that caught more bugs than any amount of Rust unit tests.

Would we pick Rust again? Yes — but knowing the costs. The right rule of thumb: pick Rust when the workload is latency-critical, when cross-language bindings matter, and when the team has at least one Rust-experienced senior. Pick Go or TypeScript for everything else.

What we'd tell teams evaluating Rust for an AI workload

Three questions determine if Rust is the right call:

Is latency-predictability a hard requirement? If you need flat p99 tail latency — because you're on the critical path of agent reasoning, because you're doing multi-retriever fan-out, because a single slow response breaks the user experience — Rust's deterministic deallocation eliminates the GC tail. If latency is loose (batch jobs, background processing) this advantage doesn't apply.

Do you need cross-language SDKs? A Rust core compiles cleanly to Node (napi-rs) and Python (PyO3). One codebase, native bindings, no network boundary. If you need SDKs in three or more languages, Rust amortizes the cross-language investment better than any alternative. If you're TypeScript-only, this advantage doesn't apply.

Does the team have Rust depth? The ramp-up cost for an experienced engineer coming from Go or TypeScript is 3-6 months before they're productive in idiomatic Rust. The borrow checker, lifetime annotations, and async send bounds are genuinely hard. We've seen experienced engineers take 4 weeks just to get comfortable with Arc<Mutex<T>> vs Arc<RwLock<T>> trade-offs in the context of our specific access patterns. Budget for this honestly.

If all three are true: Rust is the right call. If one is false, evaluate carefully. If two are false, pick Go (deterministic performance, fast cross-language bindings via CGo or gRPC, shorter ramp).

redb and typed, zero-copy metadata access

There's no SQL anywhere in Brain. Metadata lives in redb, an embedded B-tree, and records are serialized with rkyv — so instead of a query language verified against a live database, we get typed table access verified by the Rust compiler. It matters particularly for a system where the schema evolves alongside the pipeline.

Here's what it looks like in practice:

// The table's key/value types are part of its definition, so a
// mismatched key or a wrong value type fails to compile — no query
// string to get wrong, no column names to typo.
const MEMORIES: TableDefinition<MemoryKey, &[u8]> = TableDefinition::new("memories");

let read_txn = db.begin_read()?;
let table = read_txn.open_table(MEMORIES)?;

let mut memories = Vec::new();
for entry in table.range(MemoryKey::user_prefix(user_id))? {
    let (_, bytes) = entry?;
    // Zero-copy: deref straight out of the mmap, no deserialization.
    let row = rkyv::access::<ArchivedMemoryRow, _>(bytes.value())?;
    if row.valid_until.is_none() {
        memories.push(row);
    }
    if memories.len() == limit { break; }
}

The table's key and value types are baked into the TableDefinition, so a mismatched key type or a wrong value type is a compile error — there is no query string to get wrong and no column names to typo. Ordering, tombstone (valid_until) filtering, and confidence ranking are plain Rust over the archived struct, which the borrow checker also checks. Because reads deref straight out of the memory-mapped file via rkyv, there's no deserialization copy on the hot path.

The trade-off versus a SQL engine: we write the range scans and filters ourselves instead of leaning on a planner. In exchange we get an embedded store with no separate service, no DATABASE_URL, no migrations tool, and no live database required at build time — the schema is Rust types, checked at cargo build.

Ownership model and concurrent retrieval

The five-retriever fan-out is the critical performance path, and Rust's ownership model makes writing it safely straightforward:

pub async fn retrieve(&self, query: &ParsedQuery) -> Result<Vec<RankedMemory>> {
    let (semantic, bm25, entity_graph, temporal, type_filter) = tokio::join!(
        self.semantic_retriever.retrieve(query),
        self.bm25_retriever.retrieve(query),
        self.entity_graph_retriever.retrieve(query),
        self.temporal_retriever.retrieve(query),
        self.type_filter_retriever.retrieve(query),
    );

    // All five complete; now fuse with RRF
    let candidates = merge_results([semantic?, bm25?, entity_graph?, temporal?, type_filter?]);
    Ok(self.rrf_fuse(candidates))
}

Each retriever holds a cheap handle to the shard's in-RAM indexes — the HNSW graph, the tantivy reader, the redb snapshot — read through ArcSwap so readers never block the single writer. The compiler verifies the ownership discipline: the borrow checker enforces at compile time what would be a runtime concern (and a code-review comment) in Python or Go.

Because the indexes live in the process and reads are lock-free (ArcSwap + epoch reclamation), all five retrievers run concurrently against the same snapshot with no explicit synchronization code and no connection pool to contend on. The result: genuinely parallel retrieval with no locking ceremony on our side.

One subtlety: tokio::join! waits for the slowest of the five. If HNSW takes 22ms p99 and the others take 3-11ms, the total fan-out p99 is the HNSW p99 plus the fusion overhead (≈5ms), not the sum of all five. This is the right model for retrieval — the bottleneck determines the tail, not the accumulation.

Cross-compilation and distribution

Shipping native binaries to three language ecosystems (Node, Python, and direct Rust) without breaking users across three operating systems and two CPU architectures requires discipline.

Our build matrix:

• x86_64-unknown-linux-gnu (most server deployments)
• aarch64-unknown-linux-gnu (AWS Graviton, Apple Silicon containers)
• x86_64-apple-darwin (developer machines)
• aarch64-apple-darwin (Apple Silicon developer machines)
• x86_64-pc-windows-msvc (Windows development environments)

napi-rs has first-class support for this build matrix via GitHub Actions. The workflow: compile each target in its native environment (cross-compilation works but produces larger binaries and has occasional edge cases with dynamic linking). Each target produces a .node file; the npm package ships all of them and selects the right one at install time via a platform-detect shim.

PyO3 follows a similar pattern via maturin, which handles the Python wheel packaging across targets.

The hardest part: libssl dynamic linking (our LLM/HTTP client needs TLS). On Linux, the .so version varies by distribution. Our solution: link statically against openssl (via the openssl crate's vendored feature). Static linking makes the binary self-contained at the cost of a larger file size (~8MB larger). Worth it: no "wrong libssl version" errors in production.

Closing

Rust is the right call for the core of an agent memory system. The investment compounds: faster bindings, predictable latency, fewer 3am pages. The investment is real: longer ramp time, more careful API design, more discipline around error types and async shapes.

The specific wins — flat p99 tail latency, zero-copy vector reads straight out of the mmap arena, thread-safety guarantees enforced by the borrow checker, CRC-checked fail-stop storage integrity — are not things you can retrofit onto a Go or Python codebase later. They emerge from the ownership model at every layer of the architecture. If you're building something where those properties matter, the ramp cost is worth paying. If you're prototyping or working on a batch job, pick the language your team already knows.

The core is open source. Read the code.