echo/ROADMAP-1.0.md

echo-memory — Roadmap to v1.0

Status: draft / in progress. Current shipping version: 0.9.0 (schema 3). Target: 1.0.0 (schema 4) — "memory you can actually trust and retrieve."

This roadmap turns the 10 code-review suggestions into a sequenced plan and tracks the scaffold for each. It is the single index: every v1.0 workstream maps to one or more files under echo-memory.plugin.src/skills/echo-memory/scripts/ (plus tests and CI).

---

1.0 thesis

0.x made the plugin correct and self-describing: status-checked writes, idempotent appends, a routing manifest the linter enforces, an entity index, and one-call capture/recall. 1.0 makes it trustworthy and genuinely recall-driven:

1. You can retrieve a fact even when you phrase the query differently than you stored it. (H1)
2. Nothing is lost when the backend is down. (H2)
3. Two clients writing at once cannot corrupt the index or clobber state. (H3)
4. Every feature is proven by tests that model the real API, gated in CI. (H4)
5. Memory accrues on its own instead of only when the agent remembers to write. (H5)

…on a base that is secure (M1), clean-linking (M2), self-diagnosing (M3), machine-parseable (M4), and maintainable (M5).

---

Non-negotiable constraints (these bound every solution)

These are inherited from the 0.x design and must not regress:

Constraint	Consequence for 1.0
Pure-Python stdlib, no third-party deps	H1 recall = local BM25 + graph fusion, not an embedding API. No numpy, keyring, requests.
Cross-platform (Win/macOS/Linux)	No bash, no platform date; pathlib, os, UTF-8-safe streams.
Plugin is the single source of truth; vault holds data only	Derived artifacts (recall index) live in _agent/index/ (machine-maintained, rebuildable) — never new control docs in the vault.
Fail loud, never silent	New write paths keep status-checking and non-zero exits.
Local state can't depend on the vault being up	H2 queue/cache lives in ECHO_STATE_DIR (default ~/.echo-memory/), not the vault.

---

Build order vs. scaffold order

The user asked to scaffold heaviest → lightest (H1→M5). That is the order the files below are stubbed. The recommended build/merge order is different — safety and proof first, so the heavy features land on a tested, secure base:

Phase A (land first):   M1 secrets  ·  H4 tests+CI
Phase B (correctness):  H3 concurrency  ·  M2 link/slug quality
Phase C (durability):   H2 offline queue + cache
Phase D (intelligence): H1 hybrid recall  ·  H5 reflection capture
Phase E (polish):       M3 doctor  ·  M4 --json  ·  M5 hygiene  →  tag 1.0.0

Scaffolds are independent stubs, so stubbing heaviest-first is safe; only the integration (wiring into echo.py, schema bump, migration) follows Phase order.

---

Workstreams (heaviest → lightest)

Each item: goal · why · scope · files · storage/schema · acceptance · depends-on · lift.

H1 — Hybrid/semantic recall · lift: ●●●●●

• Goal: recall "X" returns relevant notes even when the wording differs, ranked, with scored multi-hop graph expansion.
• Why: Recall is the product. Today it's keyword /search/simple + fixed one-hop (echo_ops.py:88). Differently-phrased memory is unfindable → dead weight.
• Scope: local BM25 index over note bodies (stdlib only); fuse lexical score + graph distance (decay per hop); relevance threshold; configurable hops.
• Files: scripts/echo_recall.py (new) → later replaces echo_ops.recall.
• Storage/schema: _agent/index/recall-index.json (postings + doc stats), machine-maintained, rebuilt by sweep.py. Schema 3→4.
• Acceptance: retrieval precision/recall measured in eval beats keyword-only baseline; recall returns scored, deduped hits + neighbourhood; index rebuildable offline.
• Depends-on: entity index (have), H4 eval harness for the precision/recall numbers.

H2 — Offline durability: write-ahead queue + read cache · lift: ●●●●●

• Goal: No durable write is lost when the API/Obsidian is down; cold-start degrades to last-known context instead of "no memory."
• Why: Today vault-unreachable = silently proceed without memory (SKILL.md:383). 502 (Obsidian not running) is the most common real failure.
• Scope: append-only NDJSON outbox; replay on next reachable session (idempotency makes replay safe); last-known-good read cache for load.
• Files: scripts/echo_queue.py (new); integration hook safe_request() wrapping echo.request.
• Storage/schema: local ECHO_STATE_DIR/outbox.ndjson + ECHO_STATE_DIR/cache/. No vault schema change.
• Acceptance: kill the API mid-session → writes queue; restart → replay lands exactly once (no dupes); load serves cache when offline and says so.
• Depends-on: existing idempotent-append discipline; H4 for fault-injection tests.

H3 — Concurrency correctness · lift: ●●●●○

• Goal: Make the advertised multi-writer (Claude + CoWork) story actually safe.
• Why: Index is full-file load→PUT with no lock (echo_index.py:105, echo_ops.py:230) → concurrent capture silently drops an entity. Lock is manual/advisory only (echo.py:293).
• Scope: auto-lock context manager around capture/scope set/index writes; atomic index update (re-read under lock, merge, save); per-resource locks; crash-safe TTL reclaim (have).
• Files: scripts/echo_concurrency.py (new); refactor echo_index.save + echo_ops.capture to route through it.
• Storage/schema: reuse _agent/locks/. No schema change.
• Acceptance: two concurrent captures both land in the index; no lost entries; lock auto-released on normal + error exit.
• Depends-on: none (pure refactor); H4 to prove it.

H4 — High-fidelity tests + CI + current eval · lift: ●●●●○

• Goal: Prove every 0.9 + 1.0 feature against an API model that reproduces real failure modes; gate on CI.
• Why: Mock PATCH is "naive append" (mock_olrapi.py:138) — can't model heading replace/insert; eval still benchmarks 0.6 vs 0.7 (two versions stale).
• Scope: higher-fidelity mock (real heading/frontmatter PATCH semantics) or containerized real Obsidian Local REST API; refresh eval to measure retrieval precision/recall, link correctness, dup rate; wire test_echo_client.py + check_routing.py + test_features.py + new module tests into GitHub Actions.
• Files: .github/workflows/ci.yml (new); eval/mock_olrapi_hifi.py (new); scripts/test_v1_scaffold.py (interface guard, new); refreshed eval/run_eval.py.
• Acceptance: CI green on every push; eval reports current-version metrics; mock reproduces the 40080 invalid-target and replace/insert paths.
• Depends-on: none — foundational, build in Phase A.

H5 — Automatic session-reflection capture · lift: ●●●●○

• Goal: At session end, propose durable captures extracted from the conversation for one-tap confirm — memory that fills itself.
• Why: Memory only accrues when the agent decides to write. "As useful as possible" means not depending on that judgment firing.
• Scope: model-side extraction emits a JSON proposal set; script dedups against the entity index, previews, and applies on confirm (respects "show before large writes" safety rule).
• Files: scripts/echo_reflect.py (new); /echo-reflect command (later).
• Storage/schema: none new (routes through capture).
• Acceptance: given a transcript, proposals are deduped vs index, previewed, and only applied on explicit confirm; nothing written without go-ahead.
• Depends-on: H1 index/recall for dedup quality; H3 for safe concurrent apply.

M1 — Harden secret handling · lift: ●●○○○ (do now — Phase A)

• Goal: Stop shipping a live bearer token in source, docs, and every .plugin zip.
• Why: Token hardcoded at echo.py:70 and repeated through api-reference.md/README — committed to git, only rotatable by rebuild. Violates the plugin's own "never store secrets" rule.
• Scope: resolve key env → local ECHO_STATE_DIR/credentials (0600) → deprecated baked default with a loud warning; scrub literal from docs to a placeholder; document rotation.
• Files: scripts/echo_secrets.py (new); echo.py KEY= → echo_secrets.resolve_key().
• Acceptance: no live token in tracked source/docs/package; ECHO_KEY still overrides; rotation documented.
• Depends-on: none.

M2 — Tame auto-link false positives & slug collisions · lift: ●●○○○

• Goal: Keep the graph (which recall depends on) clean.
• Why: Auto-link fires on any ≥3-char name/alias word-match (echo_ops.py:236) → a concept "API" or alias "rs" links everywhere; slugify truncates to 40 chars with no collision check (echo_index.py:58) → distinct titles silently share a note.
• Scope: raise alias floor / require multi-token or kind-aware match / stopword guard; collision-aware slug disambiguation in derive_path.
• Files: scripts/echo_quality.py (new); patches to echo_links/echo_index.
• Acceptance: known false-positive cases no longer link; colliding titles get distinct slugs; covered by tests.
• Depends-on: none.

M3 — echo.py doctor + complete load fallback · lift: ●●○○○

• Goal: One-call readiness check; make load self-sufficient.
• Why: No single "is everything OK" check. cmd_load reads the heartbeat but never falls back to listing _agent/sessions/ though SKILL.md:122 documents it (echo.py:395).
• Scope: doctor = Python version + reachability + auth + marker/schema_version + lint summary, green/red; load lists recent sessions when heartbeat missing/stale.
• Files: scripts/echo_doctor.py (new); patch echo.cmd_load.
• Acceptance: doctor prints actionable status and exits non-zero on any red; load orients with no heartbeat.
• Depends-on: none.

M4 — Machine output (--json) + --dry-run · lift: ●○○○○

• Goal: Let the agent act on structured results and preview writes.
• Why: capture/recall/resolve/scope print prose; agent re-parses free text.
• Scope: --json result envelope (action, path, links_added, hits); --dry-run for capture/scope set.
• Files: scripts/echo_output.py (new helpers); thread through echo_ops/echo.py.
• Acceptance: every high-level op emits valid JSON under --json; --dry-run writes nothing.
• Depends-on: none.

M5 — Repo hygiene & manifest completeness · lift: ●○○○○

• Goal: Remove "which tree is canonical?" traps; complete the manifest.
• Why: Two plugin trees (codex plugin/ vs echo-memory.plugin.src/) have already drifted (codex lacks the 0.9 modules); stack of versioned .plugin artifacts; sparse plugin.json.
• Scope: consolidate/mark canonical tree; prune old artifacts; add license/homepage/command registration to plugin.json; refresh version-history + eval references.
• Files: MAINTENANCE.md (new checklist); plugin.json.
• Acceptance: one canonical source tree; complete manifest; docs reference current version.
• Depends-on: none.

---

Schema 3 → 4 migration (owned by H1/H2)

• Add _agent/index/recall-index.json (H1), rebuilt by sweep.py.
• migrate.py: add [3→4] step (create recall-index placeholder; no destructive moves).
• routing.json: add a route for _agent/index/recall-index.json (already covered by the generic ^_agent/index/[^/]+\.json$ route — verify in check_routing.py).
• Bump CURRENT_SCHEMA to 4 in sweep.py/migrate.py; vault_lint.py unaffected.

---

Scaffold status

#	Workstream	Lift	Scaffold file(s)	State
H1	Hybrid recall	●●●●●	scripts/echo_recall.py	✅ WIRED — BM25 + vault crawl + recall-index.json persistence + graph fusion; maintained on capture, rebuilt by sweep; replaces echo_ops.recall; schema 4. Covered by test_features.py.
H2	Offline queue/cache	●●●●●	scripts/echo_queue.py, eval/test_offline_queue.py	✅ WIRED — write verbs (put/post/append/patch/delete) queue on outage and report "queued"; flush (+ flush-on-load) replays idempotently and re-bases to the current endpoint (survives base migration); load degrades to last-known-good cache and flags OFFLINE. Proven end-to-end (down→queue→up→flush→land, idempotent, cache fallback). Follow-up: route capture's internal index/link writes through the queue for full offline-atomic captures.
H3	Concurrency	●●●●○	scripts/echo_concurrency.py	✅ WIRED — vault_lock CM (auto acquire/release, crash-safe, advisory) + atomic_index_update (lock + fresh re-read-merge). capture entity write and echo_recall.update_note both routed through it. Proven by test_features.py (merge-no-clobber + lock-release). Follow-up: deeper resolve-level title-collision detection; true concurrent stress test.
H4	Tests + CI	●●●●○	.github/workflows/ci.yml, eval/mock_olrapi_hifi.py, eval/test_patch_semantics.py, scripts/test_v1_scaffold.py	✅ hi-fi PATCH mock + semantics test + CI matrix (Win/macOS/Linux × 3.10/3.12) live. Remaining: refresh run_eval.py (still 0.6-vs-0.7) → publish current metrics.
H5	Reflection capture	●●●●○	scripts/echo_reflect.py, eval/test_reflect.py, commands/echo-reflect.md	✅ WIRED — validate/classify/preview/apply + echo.py reflect (dry-run unless --apply, reads file or stdin) + /echo-reflect command. Dedups proposals vs the entity index, drops below-confidence, routes inbox, applies via capture (indexed/linked/logged, self-lock-guarded). Proven by test_reflect.py.
M1	Secret handling	●●○○○	scripts/echo_secrets.py	✅ WIRED — resolve_key (env → ~/.echo-memory/credentials → deprecated fallback + loud warning), write-key CLI, docs scrubbed. Live token now only in echo.py DEFAULT_KEY (remove at 1.0; operator env already sets ECHO_KEY).
M2	Link/slug quality	●●○○○	scripts/echo_quality.py	✅ WIRED — is_confident_link gates capture auto-linking (rejects <4-char / common tokens, trusts multi-word names); safe_slug disambiguates a colliding 40-char-truncated slug in the create path. Follow-up: resolve-level title collision (two long titles → same slug → false merge).
M3	Doctor + load fallback	●●○○○	scripts/echo_doctor.py, commands/echo-doctor.md	✅ WIRED — echo.py doctor (Python/reachability/auth/bootstrap/schema/key-source) + /echo-doctor; cmd_load now falls back to a recent-sessions listing when the heartbeat pointer is absent. Verified against the live vault.
M4	--json / --dry-run	●○○○○	scripts/echo_output.py	✅ WIRED — capture --json emits a clean result envelope (helper chatter redirected) and --dry-run previews the create/update plan without writing; resolve already emits JSON. Proven by test_features.py. Follow-up: --json for recall/scope/link.
M5	Hygiene	●○○○○	plugin.json, .gitignore, MAINTENANCE.md	✅ DONE — plugin.json → 1.0.0 + license; .gitignore (.DS_Store, __pycache__, local .echo-memory/ state, eval output). Remaining (operator judgment, not code): consolidate/retire the drifted codex plugin/ tree and prune old .plugin artifacts — see MAINTENANCE.md.

Wiring: scaffolds are not imported by the live echo.py yet — 0.9.0 behavior is unchanged. Integration happens per Phase, each gated by its tests (H4).