Consolidation
Consolidation is the act of merging similar facts into a single canonical record. Done blindly, it destroys useful raw evidence (see Consolidation gate). Mnemosyne applies it carefully.
The pipeline
mnemo.janitor cron (weekly)
↓
1. Cluster (cluster.ts)
- For each workspace, find clusters of facts with cosine ≥ 0.85.
- Maximum cluster size: 25 facts.
- Each cluster carries a candidate canonical fact (the most-cited
or most-pinned member).
2. Gate (the six conditions)
- Worth < 0.5 for every member?
- Cluster size ≥ 3?
- No member recalled in the last 14 days?
- No member with worth confidence > 0.7?
- Similarity ≥ 0.85 (already pre-filtered)?
- No admin override active?
3. Summarise (summarize.ts) — only on gated clusters
- Generate the consolidated content. Either deterministic merge
(longest common subsequence) or LLM if a provider is configured.
- Preserve every member's citations on the canonical fact.
4. Persist
- Insert the canonical fact (new ID, attributed to
`consolidation@<timestamp>`).
- Move members to mnemo_fact_archive (preserve every column).
- Close their bitemporal windows.
- Update relations: every edge pointing to a member now points to
the canonical fact.
5. Validate
- Replay the last 10 recall queries that would have hit any of the
merged members.
- Compare result quality (top-K precision against the pre-merge
state).
- If quality drops > 5%, ROLLBACK.
6. Audit
- Log to mnemo_audit_log: who consolidated, which facts merged,
the validation result.
- If rolled back, emit a health alert.Cross-workspace consolidation
cross-workspace.ts runs a separate consolidation pass for the
org-wide view (mnemo_org_fact_view). It only sees facts marked
shared:true and runs with stricter conditions (similarity ≥ 0.92,
worth confidence ≥ 0.5 required).
Rollback path
Every consolidation produces an archive row. Restoring is a deterministic inverse of the merge:
-- Pseudo-code; the inverse the consolidation worker applies on a failed
-- ReplayValidator check. Not a migration — there is no `migrate down`.
UPDATE mnemo_fact SET valid_to = NULL
WHERE id IN (SELECT id FROM mnemo_fact_archive
WHERE archived_by = 'consolidation@<timestamp>');
DELETE FROM mnemo_fact WHERE attributed_to = 'consolidation@<timestamp>';When NOT to consolidate
- Episodes are NEVER consolidated. They’re immutable evidence.
- Pinned facts are exempt.
- Facts tagged
worth_canonicalare exempt.
The gate handles those automatically.