Running federation

This is the operational companion to the federation concept page, which explains what federation is. This page covers how to run it: enabling the feature, key material, the mTLS boundary, peer approval, conflict strategies, and the tombstone/erasure behavior. It follows the authoritative spec in docs/architecture/FEDERATION.md.

Federation is not a real-time replication layer — it is an eventually-consistent knowledge sync with explicit conflict resolution. For synchronous high availability, use Postgres streaming replication instead.

Enabling federation

Federation lives in the optional @mnemosyne/federation package and is wired in at the composition root behind a feature flag — the zero-dependency core never imports it. When the flag is off, none of the sync machinery runs.

if (config.features.federation) {
  const { FederationPeerManager } = await import("@mnemosyne/federation");
  await new FederationPeerManager(mnemo, config.federation).start();
}

Peers are listed explicitly in mnemo.config.yaml (static discovery is recommended for production). Each peer entry carries an id, endpoint, the workspaces it may exchange, a direction, and an optional certificate pin:

federation:
  peers:
    - id: "instance:eu.example.com"
      endpoint: "https://mnemo-eu.example.com"
      certPin: "sha256/Abc123..."
      workspaces: ["ws_shared", "ws_products"]
      direction: "bidirectional"
    - id: "instance:dr.example.com"
      endpoint: "https://mnemo-dr.example.com"
      certPin: "sha256/Xyz789..."
      workspaces: ["ws_shared"]
      direction: "push" # this instance pushes; DR only receives

DNS SRV records (_mnemosyne._tcp.<domain>) are also supported for dynamic deployments and are re-resolved every 5 minutes; discovered peers are still subject to certificate pinning before trust is established. mDNS LAN auto-discovery is not implemented — there is no discovery/ code in the package.

The mTLS boundary

Mnemosyne does not terminate TLS in the application. The server runs raw node:http; a trusted TLS-terminating proxy (nginx, a cloud load balancer, or Fly.io’s edge) performs the mutual-TLS handshake and forwards the verified client certificate in the x-mnemo-client-cert header. See docs/architecture/MTLS_BOUNDARY.md for the full rationale.

The application only trusts that header when the request originates from the proxy IP configured in MNEMO_TRUSTED_PROXY_IP. The middleware compares the first X-Forwarded-For entry (the IP the TLS terminator saw on the wire) against that value; if they differ, clientCert is left unset and the request falls through to other auth (API key, Bearer token).

On top of transport mTLS, every federation request also carries a short-lived signed JWT (exp = iat + 300, a unique jti for replay prevention, and a payload_hash binding the token to the request body). The jti is checked against a Redis replay store; a repeat is rejected with 401.

Generating and registering Ed25519 peer keys

Federation supports an optional application-layer signing + sealing layer on top of transport mTLS. It is opt-in via key material: when keys are absent, serve and pull fall back to plaintext (transport security still applies), so existing deployments keep working unchanged.

On this instance (serve side)

# Ed25519 private key (PEM) — signs the served envelope
openssl genpkey -algorithm ed25519 -out fed_priv.pem
export MNEMO_FEDERATION_PRIVATE_KEY="$(cat fed_priv.pem)"
 
# Shared 32-byte symmetric key (base64) — seals item payloads
export MNEMO_FEDERATION_SYMMETRIC_KEY="$(openssl rand -base64 32)"
 
# Public key to share with peers that pull from this instance
openssl pkey -in fed_priv.pem -pubout -out fed_pub.pem

When both env vars are set, GET /memory/federation/items returns a signed, sealed envelope. On bad or short keys, boot logs federation.serve_crypto_init_failed and falls back to plaintext.

On each peer (pull side)

In the local peer registry (PeerConfig), set per peer:

• signingPublicKey — the peer’s Ed25519 public key (PEM), used to verify the signature on pull.
• symmetricKey — the shared 32-byte symmetric key (base64), used to open sealed items.

When both are present for a peer, the pull path enforces verify + open and rejects items that fail either; otherwise it accepts plaintext for back-compat. Keys are never accepted from the peer payload — only from local config.

Approving peers and driving sync

Federation exposes REST endpoints under /v1/memory/federation/*. All of them require the federation scope in the API key’s JWT claims.

Kicking off a sync returns a job handle that you poll to completion:

// POST /v1/memory/federation/sync → 202
{ "jobId": "job_sync_001", "status": "processing" }

// GET /v1/memory/federation/sync/job_sync_001
{
  "jobId": "job_sync_001",
  "status": "completed",
  "peer": "instance:eu.example.com",
  "itemsSynced": 142,
  "conflicts": 2,
  "durationMs": 3210
}

A peer moves through a connection lifecycle — DISCOVERED → HANDSHAKE → ACTIVE, dropping to DEGRADED on timeout and reconnecting with exponential backoff (30s → 2m → 10m → 30m → 1h ceiling). A peer with no successful reconnect for 24 hours is flagged stale and raises an admin alert.

Conflict-resolution strategies

When the same logical item (same origin id) exists on both sides with different content hashes, the incoming version is run through a resolution pipeline before it is written. The pipeline (FEDERATION.md §12) is, in order:

1. Resurrection guard — if the id is in mnemo_deletion_log, reject with RESURRECTION_BLOCKED (a tombstone is idempotently ACKed). This runs first, so a lagging peer can never re-introduce a deleted item.
2. Hash match → no-op, accept (deduplicated).
3. Attribution cap + version guard — every incoming item is clamped to federation_received (trust score 20) regardless of what the sender claims; any local user_stated / verified fact can never be overwritten by a federated item; and items jumping more than 100 versions ahead are rejected.
4. Trust comparison — within the same tier, the higher trust anchor wins when the verdict is confident.
5. Configured strategy — applied when trust is ambiguous.

The strategy is chosen per sync via SyncRequest.conflictResolution:

Consensus weighting comes from the local peer trust config (never self-reported by a peer), decayed by clock skew and item freshness. The implementation lives in packages/federation/src/consensus.ts (resolveConsensus) and trust.ts (the frozen 9-tier TRUST_ANCHORS).

When resolution lands on contested, both versions are preserved: the local one stays active, the incoming one is stored with status = "contested", a conflict_review task is created, and both appear in recall flagged contested: true. A workspace admin then settles it via POST /v1/memory/federation/resolve, which accepts either the legacy { itemA, itemB } LWW form or a { items: FederationClaim[] } weighted consensus form.

Secure envelopes, tombstones, and GDPR erasure

Secure envelope. Sync payloads are wrapped in a signed envelope: an Ed25519 signature over SHA-256(canonical_json(items)), with item bodies sealed under AES-256-GCM. The helpers — buildSecureItemsEnvelope / openSecureItemsEnvelope, signPayload / verifyPayload, sealItem / openItem — are exported from @mnemosyne/federation (secureEnvelope.ts, crypto.ts). Verification failures surface as FederationVerifyError / FederationDecryptError.

Tombstones. Deletes propagate as tombstone sync items (a type suffix of :deleted, or an ERASE_NOTIFY message for GDPR erasure). On receipt, the instance hard-deletes the item, writes a row to mnemo_deletion_log, flushes the Redis hot tier, purges the embedding from the vector store, and — when requiresAck is set — returns a SYNC_ACK confirming the deletion. The deletion-log entry is what powers the resurrection guard above, so an erased item stays erased across the mesh even if a stale peer re-offers it.

Monitoring

Federation emits Prometheus metrics on /metrics (labelled per peer): mnemo_federation_lag_seconds, mnemo_federation_items_synced_total, mnemo_federation_conflicts_total, mnemo_federation_peers_active / _degraded, mnemo_federation_certificate_expiry_days, and mnemo_federation_replay_detected_total. Recommended alerts from the spec:

See also

• Federation protocol — what federation is
• Security & isolation — the mTLS boundary in context
• Trust-anchor consensus — zero-LLM conflict resolution