InternalsConsolidation

Consolidation

Consolidation is the act of merging similar facts into a single canonical record. Done blindly, it destroys useful raw evidence (see Consolidation gate). Mnemosyne applies it carefully.

The pipeline

mnemo.janitor cron (weekly)
  ↓
1. Cluster (cluster.ts)
   - For each workspace, find clusters of facts with cosine ≥ 0.85.
   - Maximum cluster size: 25 facts.
   - Each cluster carries a candidate canonical fact (the most-cited
     or most-pinned member).

2. Gate (the six conditions)
   - Worth < 0.5 for every member?
   - Cluster size ≥ 3?
   - No member recalled in the last 14 days?
   - No member with worth confidence > 0.7?
   - Similarity ≥ 0.85 (already pre-filtered)?
   - No admin override active?

3. Summarise (summarize.ts) — only on gated clusters
   - Generate the consolidated content. Either deterministic merge
     (longest common subsequence) or LLM if a provider is configured.
   - Preserve every member's citations on the canonical fact.

4. Persist
   - Insert the canonical fact (new ID, attributed to
     `consolidation@<timestamp>`).
   - Move members to mnemo_fact_archive (preserve every column).
   - Close their bitemporal windows.
   - Update relations: every edge pointing to a member now points to
     the canonical fact.

5. Validate
   - Replay the last 10 recall queries that would have hit any of the
     merged members.
   - Compare result quality (top-K precision against the pre-merge
     state).
   - If quality drops > 5%, ROLLBACK.

6. Audit
   - Log to mnemo_audit_log: who consolidated, which facts merged,
     the validation result.
   - If rolled back, emit a health alert.

Cross-workspace consolidation

cross-workspace.ts runs a separate consolidation pass for the org-wide view (mnemo_org_fact_view). It only sees facts marked shared:true and runs with stricter conditions (similarity ≥ 0.92, worth confidence ≥ 0.5 required).

Rollback path

Every consolidation produces an archive row. Restoring is a deterministic inverse of the merge:

-- Pseudo-code; the inverse the consolidation worker applies on a failed
-- ReplayValidator check. Not a migration — there is no `migrate down`.
UPDATE mnemo_fact SET valid_to = NULL
  WHERE id IN (SELECT id FROM mnemo_fact_archive
                 WHERE archived_by = 'consolidation@<timestamp>');
DELETE FROM mnemo_fact WHERE attributed_to = 'consolidation@<timestamp>';

When NOT to consolidate

  • Episodes are NEVER consolidated. They’re immutable evidence.
  • Pinned facts are exempt.
  • Facts tagged worth_canonical are exempt.

The gate handles those automatically.