Vault Sync

Feature Overview [edit]

spec-memo persists sync state through three cooperating channels: an opt-in private vault-git remote, hybrid HTTP delta sync to a remote SSE daemon, and multi-machine filesystem peer sync (memo sync-vault). memo sync is the explicit operator command; session_end and graceful memo serve shutdown flush automatically.

• vault-git mirrors the vault root (~/.spec-memo/) to a private git remote. Off by default; requires vaultGit.enabled: true. In the default atomic: false (batched) mode, mutations write markdown only and git commit/pull/push runs on flush.
• Hybrid keeps the local vault authoritative and exchanges changesets with a daemon over authenticated HTTP (/api/sync/pull, /api/sync/push). It pulls on bootstrap, debounces pushes after mutations, and syncs explicitly via memo sync.
• Multi-machine memo sync-vault <target> exchanges deltas between two filesystem vault locations, optionally two-way.
• Conflict reconciliation (memo reconcile) cleans conflict sidecars, applies semantic auto-merge, and can drive one sync pass with a chosen source of truth.

Business Rules & Logic [edit]

• Vault-git is a no-op when vaultGit.enabled is omitted/false or mode is remote. initVaultGit is idempotent: it does not re-init an existing repo, sets origin only when remoteUrl is configured, and appends missing .gitignore lines (never overwrites operator entries). Required ignores: memo.sqlite, memo.sqlite-wal, memo.sqlite-shm, .sync/, error.logs, telemetry/, backups/.
• atomic default is false; the unused autoCommit maps to atomic when atomic is absent; non-boolean atomic is treated as false (effective value reported by status/doctor).
• Batched (`atomic: false`): a mutation (upsert, append, forget, gc, import, reset, prompt record/sessionstart, recurrence backfill) creates no commit and runs no pull/push. Flush points are exactly: `memo sync`, a successful MCP/CLI `prompt` `sessionend, and graceful stdio/SSE shutdown. A flush with a clean tree skips the empty commit but still pulls/pushes when remoteUrl is set. Commit message is vault-git flush <ISO> (plus session <sessionId> on session end). Staged paths stay in the allow-list (projects/, config.json, .gitignore), never git add .`.
• Atomic (`atomic: true`): after a listed mutation the process commits (upsert kind:id, etc.) then pulls/pushes fail-open. Rapid atomic mutations single-flight remote sync per vault root (one in-flight plus exactly one trailing). Hybrid debounce (2000 ms) remains independent.
• Remote git pull uses git pull --rebase --autostash origin <branch>; a failed autostash pull aborts the half-rebase and drains every autostash entry by explicit ref (never a bare stash pop). Git child processes use a 30 s timeout (SPEC_MEMO_SYNC_TIMEOUT_MS). Every git/FS/network error logs under subsystem vault-git and marks $SPEC_MEMO_ROOT/.sync/vault-git-state.json { dirty, lastError, lastSyncAt } (credentials redacted). Errors never crash MCP/SSE/CLI.
• Dual mode (mode: hybrid + vaultGit.enabled): syncDual runs hybrid first, then vault-git sequentially in one run, so the git commit captures views that hybrid just rewrote. Channels are independent: either may fail while the other still reports. DualSyncReport.ok is true only when every enabled channel succeeds, hybrid has zero conflicts, and hybrid state is not dirty. flushOnShutdown caps the wait at SPEC_MEMO_SYNC_TIMEOUT_MS or 8000 ms and never throws.
• Hybrid cursors are monotonic per project. Pull writes a cursor from changeset.generatedAt only when records were applied; push writes a cursor from the export snapshot generatedAt (not push-completion wall clock) so records updated during an in-flight push remain visible. A dirty project pushes with since unset, so older offline records survive an empty pull. Push failure sets dirty/lastError without rolling back local writes; a successful push clears only its own project flag.
• Conflict policy (smart-merge default, plus local-wins, remote-wins, sidecar): bodies are compared semantically (line endings and trailing whitespace ignored). Identical bodies auto-merge metadata (max hits/occurrences, latest lastHit/lastSeen, union tags/linkedPaths) without a conflict. Diverging bodies: remote-wins overwrites; local-wins keeps local; sidecar and equal-timestamp smart-merge write one deterministic {slug}.conflict.md; differing timestamps in smart-merge take the newer side. --prefer local flips the push direction to force remote, and --prefer remote flips to local.
• applyChangeset is transactional: it pre-validates every record (schema, path containment, secret rules) before writing, journals original content, rolls back all mutated files plus rebuilds indexes on failure, and on success rebuilds compiled views per project with one retry then skip-and-log, then rebuilds FTS. cleanSidecars removes conflict sidecars whose body matches the base record.
• Issue #55 hardening: sync-context records whose pathPatterns are fully covered by builtin ignore rules (e.g. absolute cross-host paths) are skipped and logged per record instead of aborting the whole changeset; direct local upsert still throws the strict capture-ignore error. View/FTS rebuild failures are retried once then skipped and logged, preserving applied counts.
• memo sync with neither hybrid nor vaultGit.enabled exits non-zero; memo sync is unavailable in remote mode. --dry-run reports hybrid counts and vault-git porcelain without committing, pulling, or pushing.
• Background sync: when sync.autoSyncIntervalMinutes > 0 and mode is hybrid, the daemon runs a non-blocking per-project syncDual cycle that skips when the vault lock is busy, never stalling MCP traffic.

Technical Architecture [edit]

• Config: VaultConfig.vaultGit (enabled, remoteUrl?, atomic?, autoCommit?, branch?) and VaultConfig.sync (conflictStrategy?, cleanSidecars?, autoSyncIntervalMinutes?) in src/types.ts; git state file helpers in src/vault-git-state.ts, redaction in src/vault-git-redact.ts.
• Git engine (src/vault.ts): initVaultGit, commitVaultChange, scheduleVaultGitRemoteSync, flushScheduledVaultGit, parseAutostashRefs, flushVaultGit, syncVault, getGitTimeoutMs, getShutdownFlushMs, resolveVaultGitAtomic, REQUIRED_VAULT_GITIGNORE. flushVaultGit returns VaultGitChannelResult ({ ok, committed, pulled, pushed, message, error?, skipped?, wouldCommit? }).
• Dual orchestrator (src/dual-sync.ts): syncDual({ vaultRoot?, projectId?, all?, dryRun?, trigger, sessionId?, force?, prefer?, strategy?, cleanSidecars? }) and flushOnShutdown.
• Hybrid client (src/hybrid-sync.ts) and state (src/hybrid-state.ts) as documented in Deployment Modes. Daemon routes live in src/server.ts (/api/sync/pull|push, /api/sync).
• Changeset engine (src/sync.ts): exportChangeset, applyChangeset, syncVaults, areBodiesSemanticallyEqual, mergeRecordMetadata, cleanConflictSidecars, cursor/tombstone helpers. memo sync-vault <target> [--two-way] [--dry-run] [--json].
• CLI: memo sync [--all] [--dry-run] [--prefer local|remote] [--strategy ...] [--clean-sidecars] [--force] [--json]; memo reconcile runs cleanConflictSidecars then syncDual with --prefer, --strategy, --clean-sidecars, --dry-run, --all (interactive source-of-truth prompt only on a TTY).
• Provenance: 0018-vault-git.spec.md, 0044-multi-machine-sync.spec.md, 0033-vault-git-hybrid-sync.spec.md (batched/atomic cadence, dual dispatch), 0035-sync-conflict-reconciliation.spec.md (semantic merge, strategies, rollback journal, background worker, sidecar cleanup), 0051-us-55.spec.md (autostash recovery, per-record skip-and-log, view-rebuild skip). See Deployment Modes and Backup & Restore.