From e7d86e14da4b0c91bd17623aa6a974632b8c18ba Mon Sep 17 00:00:00 2001 From: Jason Stedwell Date: Tue, 28 Jul 2026 21:49:53 -0500 Subject: [PATCH] Add post-1.5.1 planning docs: 9 improvement plans + containerized MCP server spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docs/IMPROVEMENT-PLANS.md — nine improvements from the 2026-07-28 full-code review (brief load, offline capture durability, session-end verb, local-first recall index, incremental sweep, lifecycle decay, supersession, stemming + alias expansion, resolve-miss learning), sequenced with TODO-1.6 and ROADMAP-2.0 into one release train (1.6.0 -> 1.6.x -> 2.0 -> 2.1 -> 2.2 -> 2.3). docs/MCP-SERVER-SPEC.md — build spec for the 2.1 flagship: echo-mcp as a Docker container on ALPHA (FastMCP, streamable HTTP, bearer auth, published as echomcp.alwisp.com), talking to Obsidian directly at http://10.2.0.35:27123. 13 v1 tools with result budgets + tool profiles; v1.1 dividends (nightly vault backup, shadow write history/undo, background sweeps, ops alerting, rollup data). Prereqs: session-end verb + return-not-print refactor (Phase 0). Co-Authored-By: Claude Fable 5 --- docs/IMPROVEMENT-PLANS.md | 352 ++++++++++++++++++++++++++++++++++ docs/MCP-SERVER-SPEC.md | 391 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 743 insertions(+) create mode 100644 docs/IMPROVEMENT-PLANS.md create mode 100644 docs/MCP-SERVER-SPEC.md diff --git a/docs/IMPROVEMENT-PLANS.md b/docs/IMPROVEMENT-PLANS.md new file mode 100644 index 0000000..6749d42 --- /dev/null +++ b/docs/IMPROVEMENT-PLANS.md @@ -0,0 +1,352 @@ +# echo-memory — Improvement Plans (post-1.5.1 review) + +> Status: **planned, not started.** Source: full-code review 2026-07-28. +> Nine improvements, each scoped for an independent build session. The tenth item +> from that review — the **MCP server** — has its own detailed spec: +> `docs/MCP-SERVER-SPEC.md`. +> +> Sequencing is at the bottom. Nothing here conflicts with `TODO-1.6.md` or +> `ROADMAP-2.0.md`; release targets assume 1.6 (skills migration) ships first. + +--- + +## 1. Token-budgeted load (`load --brief`) + parallel orientation reads + +**Problem.** The SessionStart hook injects the *full text* of six files into every +session. Live measurement: 15.9KB of hook context on a single cold start, and two of +the six files grow without bound (operator-preferences `## Observations`, +current-context `## Scope History`). Every session pays this tax before any work +happens. Separately, `cmd_load` (echo.py) issues its six GETs serially even though +`read_many()` exists. + +**Design.** +- New `echo.py load --brief` (and make the SessionStart hook use it): + - marker → one line (`schema_version`, bootstrapped yes/no). + - operator-preferences → `## Fact / Pattern` in full; `## Observations` capped at + the last 10 lines with a `(+N older — /echo-load for full)` note. + - current-context → `## Scope` + `scope_updated` + sessions-since (i.e. the + `scope show` output); omit `## Scope History`. + - heartbeat → the pointer line + only the `## Summary`/`## Outcomes` sections of + the pointed-at session log (fetch it, extract those headings, cap ~30 lines). + - today's daily note → `## Agent Log` lines only (or "absent"). + - inbox → count + age of oldest item, not the contents (that's all the reconcile + needs; `/echo-triage --list` has the details). +- Budget guard: after assembly, if the brief output still exceeds + `ECHO_LOAD_BUDGET` chars (default ~8000), truncate lowest-priority sections first + (observations → agent log → session summary) with explicit `(truncated)` markers. +- Full `load` unchanged; `/echo-load` keeps using it. +- Fetch all reads via `read_many()` (the 6 targets + the heartbeat-pointed session + log + the sessions listing fallback) instead of the serial loop at + `echo.py:640`. Cache-put still applies per path. + +**Files.** `echo.py` (cmd_load), `echo_hook_session_start.py`, SKILL.md (document +brief-vs-full), `eval/test_features.py`. + +**Tests.** Mock vault with oversized preferences/history files → assert brief output +under budget, sections present, truncation markers correct; hook emits brief. + +**Size/target.** Small-medium. **1.6.x point release** — highest leverage per line. + +--- + +## 2. Local-first recall index + +**Problem.** `echo_recall.update_note()` does GET-whole-index → add → PUT-whole-index +against the vault, under the global advisory lock, on every capture and every corpus +`put`. The index carries full BM25 postings for the whole corpus, so this round-trip +is O(vault) network per write and is the emerging bottleneck + lock hot-spot as the +vault grows past a few hundred notes. + +**Design.** +- The **live** recall index moves to the local state dir + (`~/.echo-memory/recall-index.json`, honoring `ECHO_STATE_DIR`) keyed by endpoint + (hash of `BASE`) so multiple vaults don't collide. +- `update_note()` becomes a local read-modify-write (file lock via `os.O_EXCL` + sidecar or atomic `os.replace`), **no vault round-trip, no advisory lock**. +- The vault copy (`_agent/index/recall-index.json`) becomes a **snapshot**, written + by `sweep.py` and at session end (piggyback on the heartbeat write / future + `session-end` verb). It exists so a *fresh machine* can seed its local index + without a full rebuild. +- Freshness rule on recall: local index used when present; if the vault snapshot's + embedded `built` stamp is newer than the local one (another client swept), pull it. + Staleness across clients is tolerable — recall degrades gracefully and improvement + #4 (incremental sweep) trues it up cheaply. +- Schema bump: recall-index schema 3 adds `built` (ISO timestamp) + `endpoint_hash`. + Old schema-2 vault copies are still readable as seeds. +- The **entity index stays in the vault** (it is small, and it is the shared routing + authority for concurrent clients) — only the BM25 index moves. + +**Files.** `echo_recall.py` (load/save/update_note/rebuild), `echo_queue.py` +(state-dir helpers shared), `sweep.py`, `echo.py` (cmd_put upkeep path), routing.json +(the vault snapshot path is unchanged), README performance section. + +**Tests.** Capture with vault mock → assert zero recall-index PUTs; sweep → snapshot +written; fresh state dir + existing snapshot → seeded without rebuild; two-vault +(endpoint) separation. + +**Size/target.** Medium. **Pair with #4 in one index-focused minor** (they touch the +same maintenance loop). Fine to land before or after 2.0. + +--- + +## 3. Offline durability for the high-level ops (capture path) + +**Problem.** The low-level verbs queue on outage via `safe_request`, but +`_append_to_existing()` and `ensure_daily_log()` (echo_ops.py) call `echo.request` +directly — so during an outage a raw `append` survives, while a `capture` **update** +and its Agent Log line silently vanish. Inverted from the documented contract +("capture is the default write"). + +**Design.** Queue *semantic* records, not byte-level requests, for the high-level +path: +- New outbox record type `{"op": "capture", "args": {...}}` alongside the existing + raw records. On an unreachable vault, `capture` short-circuits: enqueue the full + argument set (title/kind/body/tags/aliases/sources/date/domain/merge_into) and + report `queued (offline): capture …`. +- `flush()` replays `op:capture` records by calling `echo_ops.capture(...)` against + the *current* index — so routing, the duplicate gate, aliasing, and linking all + re-run with fresh state instead of replaying stale byte writes. A gate stop on + replay (exit 76) keeps the record queued with a `needs_attention` flag and is + surfaced by `load`/`flush` output ("1 queued capture needs --merge-into/--force"). +- Rationale for semantic-over-byte: a byte replay of `_append_to_existing`'s PATCH + could target a heading that moved, and the create-vs-update decision made offline + may be wrong by replay time. Re-running the op is the idempotent, correct unit + (the dated-bullet idempotency key already prevents double-append). +- `ensure_daily_log` stays best-effort but routes its POST/PATCH through + `safe_request` so a standalone agent-log line survives an outage too. +- reflect/triage `--apply` inherit this for free (they call capture). + +**Files.** `echo_ops.py`, `echo_queue.py` (record schema v2 + replay dispatch), +`eval/test_offline_queue.py`. + +**Tests.** Fault-injected mock: capture-update offline → queued; flush replays via +capture; gate-on-replay keeps record + flags; agent-log line queued; no duplicate +bullets on double flush. + +**Size/target.** Small-medium. **1.6.x point release** — it's a correctness gap. + +--- + +## 4. Incremental sweep via content hashes + +**Problem.** Human edits in Obsidian and writes by other clients never touch the +entity/recall indexes; drift is only corrected by a full sweep someone must remember +to run. Full sweep re-reads the whole vault (fast now, but O(vault)). + +**Design.** +- Entity index entries and the recall-index snapshot gain a per-note + `h` — SHA-1 of the note body (entity index schema 2; tolerated-absent for old + entries). +- `sweep.py --fast`: walk the listing (cheap), `read_many` only paths that are + (a) new, (b) missing a stored hash, or (c) whose hash mismatches after fetch — to + avoid fetching everything just to hash it, fast mode fetches only paths whose + **listing is new/gone** plus a rotating shard (e.g. `hash(path) % 7 == weekday`) + of existing notes, so the whole vault is verified over a week of fast sweeps while + each run stays tiny. `--fast --all-shards` forces full verification. +- Deletions: listing walk catches removed files → drop index + recall entries. +- Auto-run: `load` (brief or full) runs `sweep --fast` opportunistically when the + last fast-sweep stamp (state dir) is older than `ECHO_FAST_SWEEP_DAYS` (default 7), + in the background-tolerant sense: bounded by the same read_many concurrency, and + skipped entirely when offline. +- `/echo-health` reports last fast/full sweep ages. + +**Files.** `sweep.py`, `echo_index.py` (schema 2 + hash field), `echo_recall.py` +(snapshot hash reuse), `echo.py` (load hook-in), `vault_lint.py` (report), migrate +note (index schema is machine-owned; no vault migration needed). + +**Tests.** Mock: edit a note out-of-band → fast sweep in that shard re-indexes it; +delete → entry dropped; stamp gating; full sweep unchanged. + +**Size/target.** Medium. **Ship with #2** (same schema-bump train). + +--- + +## 5. Memory lifecycle — enforce forgetting + +**Problem.** The vault only accretes. Working memory is "time-boxed" by convention +only; Observations trimming is manual prose; stale-active detection only reports. +Recall precision degrades as dead weight accumulates. + +**Design.** +- `capture --kind working` stamps `expires: ` + (default 14; `--ttl ` overrides). Template updated to carry the field. +- `vault_lint.py` new checks: `expired-working` (past `expires`), + `stale-observations` (Observations > 30 bullets) — advisory, like the rest. +- `sweep.py --decay` (dry-run by default, `--apply` to write; **always + preview-first**, honoring the operating contract): + - expired working notes → `status: archived` + move body under a dated + `## Archived` marker (never delete); + - `projects/active/` untouched > 30 days → *propose* `on-hold` (folder move + + status, the usual paired change) — apply only the ones confirmed; + - Observations > 30 → trim oldest into a dated `_agent/memory/episodic/` + overflow note (`observations-archive-YYYY.md` append), keeping the last 30 + in place. Nothing is lost, the hot file stays small (compounds with #1). +- `/echo-health` output nudges: "run `sweep --decay` to review N aging items". + +**Files.** `echo_ops.py` (`--ttl`/expires stamp), `echo_index.py` +(KIND_REQUIRED_FM unchanged; working gains optional expires), `vault_lint.py`, +`sweep.py`, scaffold working-memory template, SKILL.md (decay section), +routing-map note. + +**Tests.** Mock: expired note proposed+archived on apply; observations trim keeps +last 30 and appends overflow; stale-active proposal list; dry-run writes nothing. + +**Size/target.** Medium. Feature minor (1.7.x). + +--- + +## 6. Contradiction / supersession handling + +**Problem.** A new fact that contradicts a stored one lands beside it; both surface +in recall with no signal about which is current. Biggest remaining trust gap in the +write path. + +**Design.** +- **Write side:** `capture --supersedes ""`: + - if it names an entity slug → the old entity gets `status: superseded` + + a `superseded_by:` frontmatter path; new note carries `supersedes:` back-link + in `## Related`. Recall's STATUS_FACTOR gains `superseded: 0.5`. + - if it quotes a Fact/Pattern or Observations line in operator-preferences → + the line is struck (`~~old~~ (superseded YYYY-MM-DD)`) and the new line + appended, one PATCH each, in the same invocation. +- **Detect side (deterministic assist, model decides):** `echo_reflect.classify` + gains a `conflict scan` — for each proposal, run its distinctive tokens + (`_sig_tokens`) against the Fact/Pattern + recent Observations lines; overlapping + lines are attached as `_conflicts: [...]` and the preview row renders + `update-or-conflict?` with the matched line(s). The model (which has the + conversation) decides supersede-vs-append; the script only guarantees the + question gets asked before the write. +- Lint: `superseded_by` pointing at a missing note → violation. + +**Files.** `echo_ops.py`, `echo_reflect.py`, `echo_recall.py` (STATUS_FACTOR), +`echo.py` (flag plumbing), `vault_lint.py`, SKILL.md + operating-contract +(supersession is additive: strike, never delete). + +**Tests.** Mock: supersede entity → status/backlinks both sides; supersede line → +strike + append idempotent; reflect preview flags a planted conflict; recall ranks +superseded below active twin. + +**Size/target.** Medium. Feature minor (1.7.x/1.8) — after #5 (shares the +status-vocabulary touchpoints). + +--- + +## 7. `session-end` bundle verb + +**Problem.** Ending a session correctly is 4+ invocations (session-log PUT, Agent +Log append, heartbeat PUT, optional scope set, optional reflect apply). Cost per +session, and a partial failure leaves broken orientation state (log written, +heartbeat stale). + +**Design.** +- `echo.py session-end [--apply]`, bundle: + ```json + { + "slug": "echo-mcp-spec", + "log_body": "", + "agent_log_line": "- 2026-07-28: wrote MCP spec [[...]]", + "scope": "optional new scope text", + "reflect": [ { ...PROPOSAL_SCHEMA... } ] + } + ``` +- Dry-run previews the whole plan (paths + reflect preview). `--apply`, in order, + under **one** lock acquisition: PUT `_agent/sessions/--.md` + (HHMM from `ECHO_NOW` env or wall clock, validated against the canonical regex) → + agent-log line (ensure_daily_log) → reflect proposals via capture → scope set if + given → heartbeat PUT **last** (it's the commit marker: a crash before it leaves + the previous pointer intact, never a dangling one). +- Reports per-step ok/queued/gated; offline → every step queues (rides on #3). +- Stop hook reason text updated to name the one command; `/echo-reflect` docs point + session logging at it. **Prerequisite for the MCP `echo_log_session` tool** — + build this verb first so CLI and MCP share one implementation. + +**Files.** new `echo_session.py`, `echo.py` (subcommand), `echo_hook_stop.py` +(REASON text), SKILL.md (Session Logging section), commands/echo-reflect.md. + +**Tests.** Mock: full bundle lands all five artifacts; heartbeat-last ordering +(inject failure mid-way → heartbeat untouched); dry-run writes nothing; offline +queues all steps. + +**Size/target.** Small-medium. **1.6.x point release** (and before the MCP build). + +--- + +## 8. Recall robustness — stemming + alias query expansion + +**Problem.** BM25 is purely lexical: "deploy"/"deployment"/"deployed" are distinct +terms; a paraphrased query misses. The 8-query gold set can't see this class of miss. + +**Design.** +- **Stemming:** a light suffix-stripper (Porter-lite: s/es/ed/ing/ly/tion/ment/ness + families, ~40 lines, pure stdlib) applied in `tokenize()` at **both** index and + query time. Index schema bump (stemmed postings) → auto-rebuild on next + recall/sweep, exactly like the 1→2 transition. +- **Alias query expansion:** at recall time, run the query against + `fuzzy_candidates()` (top 2, score ≥ 0.5); fold the matched entities' title + + alias tokens into the BM25 query as **down-weighted expansion terms** (×0.5 idf + contribution) so "the ECHO plugin" also scores `echo-memory`-alias vocabulary. + Exact-match seeding stays as-is; expansion never *creates* hits ranked above + genuine lexical matches without corpus support. +- **Eval:** grow the gold set with ≥6 paraphrase queries (morphological variants + + alias phrasings); publish before/after recall@5/MRR in the README table. + +**Files.** `echo_recall.py` (tokenize, INDEX_SCHEMA, score, recall), new +`echo_stem.py` (pure function, unit-testable), `eval/run_eval.py` + gold set. + +**Tests.** Stemmer unit table (word → stem, incl. non-cases); paraphrase gold +queries pass; schema-2 index discarded and rebuilt cleanly. + +**Size/target.** Small-medium. Ride the **same index-schema train as #2/#4**. + +--- + +## 9. Resolve-miss telemetry → alias suggestions + +**Problem.** Every resolve that misses but ends in a confirmed candidate +(`--merge-into`, or the operator picking a candidate) is a free mention→entity +training pair, currently discarded. + +**Design.** +- `echo_index.resolve()` miss + `fuzzy_candidates` non-empty → append one NDJSON + record to the local state dir (`~/.echo-memory/resolve-misses.ndjson`): + `{mention, candidates:[slugs], ts?}` (no wall clock needed — session date via + ECHO_TODAY). `capture --merge-into ` appends the *confirmation* record + `{mention, resolved: slug}` (this already learns the alias immediately — that + path stays; the log covers the cases that never reach --merge-into). +- `sweep.py` (and `/echo-health`) read the log: any `{mention → slug}` pair + confirmed ≥2 times, or any mention that repeatedly fuzzy-matched a single + candidate ≥3 times without confirmation, is **proposed** as an alias + (`sweep --apply` writes it into the note's `aliases:` frontmatter + index, the + existing fold-back path). Log entries consumed on apply; file capped (rotate at + ~500 lines). +- Privacy note: mentions can contain operator phrasing — the log lives in the + state dir (never the vault), same trust level as the offline queue. + +**Files.** `echo_index.py` (log hooks), `echo_ops.py` (confirmation hook), +`sweep.py` (suggest/apply), `vault_lint.py` or sweep report (surface count), +SKILL.md one-liner. + +**Tests.** Miss → record written; confirm ×2 → sweep proposes; apply folds alias + +consumes log; cap/rotation. + +**Size/target.** Small. Feature minor, pairs naturally with #8. + +--- + +## Sequencing (unified with TODO-1.6 and ROADMAP-2.0) + +TODO-1.6 folds in as the opening train — the skills migration restructures the +files everything below touches, so it goes first. ROADMAP-2.0 stays a **pure +packaging major** (its own stated thesis) and is *not* folded into any feature +train; it slots between the quick wins and the MCP server so the MCP work lands +on the final packaging baseline (single command format, Gitea releases, one +manifest-verification pass). + +| Release | Items | Rationale | +|---|---|---| +| **1.6.0** | TODO-1.6: skills-format migration (+ per-skill `allowed-tools`, `disable-model-invocation` on write-heavy entries, `$ECHO` block dedupe) · lint blind-spot fix (retired-dir/leaf-README) · plugin.json `homepage` | Restructures the command/SKILL files later work edits; allowed-tools removes prompt friction for every verb added below. Vault link repointing (TODO-1.6 §3) is vault data, not plugin code — any session. | +| **1.6.x quick wins** | #1 brief load · #3 offline capture · #7 session-end (+ return-not-print refactor, MCP spec Phase 0, if it fits) | Small, correctness/efficiency, no schema changes. #7 unblocks the MCP tool. Users get these without a reinstall — don't gate them behind 2.0. | +| **2.0** | ROADMAP-2.0 unchanged: delete legacy `commands/` · artifact policy (Gitea releases, gitignore, tags) · Codex note · README layout | Packaging only, no feature riders — a bad feature must never force reverting a packaging break. | +| **2.1** | MCP server (containerized, `echomcp.alwisp.com`) — `docs/MCP-SERVER-SPEC.md` | Needs #7 + Phase 0. Lands after 2.0 so its docs target one command format and the Gitea release channel exists. Deployed via CI + PORT, not the plugin manifest. | +| **2.2 index train** | #2 local-first recall (CLI-path half) · #4 incremental sweep · #8 stemming+expansion | One schema-bump train; all touch the same index/sweep loop. The container already keeps its index warm in memory (spec §7), so #2 here covers the CLI fallback path; #4's fast sweep also becomes the container's background job. | +| **2.3 memory train** | #5 lifecycle/decay · #6 supersession · #9 resolve-miss learning | Capability tier; #6 after #5 (status vocabulary). | diff --git a/docs/MCP-SERVER-SPEC.md b/docs/MCP-SERVER-SPEC.md new file mode 100644 index 0000000..79a6a41 --- /dev/null +++ b/docs/MCP-SERVER-SPEC.md @@ -0,0 +1,391 @@ +# echo-mcp — MCP Server Build Spec (containerized) + +> Status: **spec for a future build session** (written 2026-07-28 against v1.5.1; +> revised same day: **remote container architecture**, operator decision — heavy +> lifting belongs in a deployed container, not on any one machine). +> Companion plan for the other nine review items: `docs/IMPROVEMENT-PLANS.md`. +> +> **Prerequisites before starting this build:** +> 1. The `session-end` verb (IMPROVEMENT-PLANS #7) — the MCP tool wraps it. +> 2. Phase 0 below (the return-not-print refactor) — lands in the plugin tree +> first and is independently shippable. + +--- + +## 1. Why an MCP server + +Every ECHO operation today rides through Bash: resolve `$ECHO` (with the CoWork +path-fallback snippet copy-pasted everywhere), quote a Python invocation, run +it, re-parse stdout. Costs: + +- **Tokens** — each memory op carries bash scaffolding + output re-parsing; a + capture is ~3× the tokens of a typed tool call. +- **Fragility** — quoting hazards, the `${CLAUDE_PLUGIN_ROOT}` sandbox mismatch + (the whole 1.4.2 release), Windows `python3`-vs-`python` dispatch. +- **Reach** — surfaces without a Python-capable shell can't use ECHO at all. +- **Efficiency** — every CLI invocation is a cold Python start that re-reads the + entity index from the vault over HTTPS. + +What the server does **not** replace: SKILL.md remains the procedure authority +(when to load, reconcile, search-first, third-person, etc.). The server replaces +the *mechanics* — tools are the verbs, the skill is the discipline. + +## 2. Architecture: remote container (decided) + +A standalone Docker container on the Unraid box (ALPHA), published behind +Cloudflare + NPM as **`echomcp.alwisp.com`**, speaking **MCP streamable HTTP +(stateless JSON)** with bearer-token auth. Chosen over the earlier local-stdio +draft for four reasons: + +1. **The backend is already remote.** Clients round-trip to + `echoapi.alwisp.com` today; a server co-located with the vault host turns + every vault op into a LAN hop and collapses N client→vault calls into one + client→server call. +2. **Any device, no per-machine anything.** Desktop, CoWork sandbox, claude.ai + (custom connector), mobile — same URL + token. The `${CLAUDE_PLUGIN_ROOT}` + CoWork registration risk from the stdio draft disappears entirely. +3. **Credentials consolidate server-side.** The vault key lives in the + container env (PORT secret store); the image is credential-free; clients + hold only an MCP bearer token. The per-user baked-key builds (1.4.x) become + unnecessary for any surface that has MCP. Rotation = redeploy. +4. **Room to grow.** The plugin's pure-stdlib constraint exists because its + scripts run in arbitrary sandboxes. The container is ours: real SDK, + SQLite/in-memory indexes that stay warm, background jobs, embeddings later — + all without touching the plugin. + +**What stays client-side (unchanged and shipped):** the skill + CLI path — it +is the fallback when the server or network is down, the offline queue + read +cache keep their per-machine role there, and the SessionStart/Stop **hooks stay +CLI-based** (hooks run shell commands regardless of MCP). + +**Cost acknowledged:** a second service to run and an exposed auth surface. +Mitigations are existing infra: CF + NPM cert, long random bearer token, Kuma +monitor, PORT deploy/rollback. Server down ⇒ exactly today's behavior. + +## 3. Decisions (made — don't relitigate in the build session) + +| Decision | Choice | Why | +|---|---|---| +| Language | **Python** | Imports the existing `echo_*.py` modules in-process — the entire ops layer is reused, not wrapped. | +| SDK | **Official `mcp` Python SDK (FastMCP)** | The stdlib-only rule doesn't bind inside our own image. Hand-rolling the protocol (stdio draft) is no longer justified. | +| Transport | **Streamable HTTP, stateless JSON** | Remote, multi-client, simplest to scale; SSE/sessions explicitly avoided. | +| Auth | **Static bearer token** (`ECHO_MCP_TOKEN`), constant-time compare, 401 otherwise | Single-operator service; OAuth is overkill. Token from the PORT secret store. | +| Vault credentials | Container env: `ECHO_BASE`, `ECHO_KEY`, `ECHO_OWNER` (SECRET: refs in the PORT template) | `echo_config.py` already resolves env-first — zero code change. Image ships no secrets. | +| Server name / repo home | `echo-mcp`; lives in this repo under `mcp-server/` (own Dockerfile), deployed from `git.alwisp.com/jason/echo` CI | One canonical tree — the server vendors `skills/echo-memory/scripts/` into the image at build time; no second hand-maintained copy (the Codex-drift lesson). | +| Tool prefix | `echo_` | Namespace safety next to other servers. | +| Result shape | `structuredContent` (the `echo_output.envelope` dict) + a 1–3 line human text block | Envelope shape already exists. | +| Statefulness | Stateless protocol; warm in-process caches | See §7. | +| Version target | **2.1** (after the 1.6.x train and the 2.0 packaging major) | Needs #7 + Phase 0; docs then target the single post-2.0 command format. | +| Local stdio variant | **Dropped for v1** (documented, not built) | The CLI/skill fallback covers the no-server case; two transports = two test matrices for little gain. Phase 0 keeps the door open — the `*_op` core is transport-agnostic. | +| Result budgets | Every read tool takes an explicit size budget (`budget_chars` / `max_chars`) and truncates with a marker + "call X for more" | The single biggest token lever: one right-sized answer instead of follow-up fetches. | +| Tool profiles | `ECHO_MCP_TOOLS=core\|full` env: `core` exposes only load/recall/capture/triage_inbox/log_session/health | Tool schemas cost context on every surface that lists them; the claude.ai connector doesn't need the escape hatches. Desktop registers `full`. | + +## 4. Phase 0 — the return-not-print refactor (prerequisite, lands in the plugin) + +Unchanged from the stdio draft, and still first: the ops layer prints results +(even `--json` mode prints from inside `echo_ops.capture`) and returns exit +codes; the server needs return values. + +1. Each high-level op gets a **core function returning an envelope dict**, + raising `echo.EchoError` on failure; CLI entry points become thin printing + wrappers: + + | Module | New core fn | CLI wrapper keeps | + |---|---|---| + | `echo_ops` | `capture_op`, `resolve_op`, `link_op` | `capture/resolve/link` | + | `echo_recall` | `recall_op(query, limit)` | `recall` | + | `echo_triage` | `list_op()` · `route_op(proposals, apply)` | `list_inbox/apply` | + | `echo_reflect` | `apply_op(proposals, apply)` | `apply` | + | `echo.py` | `load_op(brief)` · `scope_show_op/scope_set_op` · `doctor_op` | `cmd_*` | + | `echo_session` (new, #7) | `session_end_op(bundle, apply)` | subcommand | + +2. Special results are **data, not exit codes**: duplicate gate ⇒ + `{ok: false, action: "duplicate-gate", candidates}` (CLI maps to exit 76); + offline queueing ⇒ `{ok: true, queued: true}`. +3. Helper chatter (`ok: PUT …`) moves behind a `notify(msg)` callback + defaulting to stderr; delete the `redirect_stdout` hack in `capture`. +4. Tests: existing suites pass unchanged; new unit tests hit `*_op` directly + against the mock and assert envelopes. + +Shippable on its own as a 1.6.x refactor with zero behavior change. + +## 5. Server application (`mcp-server/app.py`) + +- **FastMCP** app, streamable HTTP, stateless. One `@mcp.tool` per tool in §6, + each a thin adapter: validate → call the `*_op` core → wrap. +- **Auth middleware**: require `Authorization: Bearer ` on every + request (constant-time compare); 401 with no detail otherwise. Additionally + bind the app to the container interface only; exposure policy lives at + NPM/Cloudflare. +- **Result wrapper**: + + ```python + def tool_result(env: dict, summary: str) -> ...: + # content: [{type: "text", text: summary}] (1–3 lines, human) + # structuredContent: env (echo_output envelope) + # isError: not env.get("ok", True) — except the deliberate non-errors below + ``` + +- **Error map** (tool-level results, never protocol errors): + + | Condition | Result | + |---|---| + | Vault unreachable on a **read** | `isError: true` — "vault unreachable (Obsidian/REST plugin likely down); proceed without memory, writes will queue server-side". | + | Vault unreachable on a **write** | **Not an error**: `{ok: true, queued: true}` — the server-side outbox (§7) replays when the vault returns. | + | 404 | `isError: true`, `{code: "not-found", path}`. | + | Duplicate gate | **Not an error**: `{ok: false, action: "duplicate-gate", candidates}` + text naming the two resolutions (`merge_into` / `force`) — the model must decide, not blind-retry. | + | Lock held | `{ok: false, action: "lock-held", holder}`. | + | Misconfigured deployment (no `ECHO_BASE`/`ECHO_KEY`) | `isError: true` — "server deployment is missing vault credentials — operator: check the PORT template env". Startup also fails loudly (see §8 healthcheck). | + | Anything else | `isError: true`, first line only; traceback to the container log. | + +- **Validation**: FastMCP/pydantic handles types/enums/required; add + `model_config = ConfigDict(extra="forbid")` so unknown params (model typos) + fail with a field-naming message. + +## 6. Tool surface (13 tools) + +Same surface as the stdio draft **minus `echo_configure`** — with a remote +server there is no per-machine config to import; credentials are +deployment-side. Descriptions are written for the model: what it does, when to +use it, what it returns, ≤3 sentences. + +### 6.1 Orientation & read + +- **`echo_load`** — cold-start orientation. `brief: bool = true` + (IMPROVEMENT-PLANS #1 digest; `false` = full six-file dump). Returns + `{sections: {marker, preferences, scope, last_session, today, inbox_count, + inbox_oldest_days}, queued_flushed, offline}`. Also flushes the server-side + outbox. `readOnlyHint: false` (flush), `openWorldHint: true`. +- **`echo_recall`** — `query: str`, `limit: int = 6 (1–20)`, + `include_linked: bool = true`, `budget_chars: int = 4000 (500–20000)`. Returns + the `recall --json` shape (`primary` / `linked` with + path/score/type/updated/status/excerpt) **packed to the budget**: the server + allocates excerpt space by score, so the model gets the most relevant content + in one right-sized answer instead of follow-up fetches. Description: truncated + hits are marked — call `echo_get_note` for full content. + `readOnlyHint: true`, `idempotentHint: true`. +- **`echo_resolve`** — `mention: str` → match `{slug, path, kind, title, + aliases}` or `{match: false, suggest_slug, candidates}`. "Call before creating + any note by hand; `echo_capture` does this automatically." `readOnlyHint: true`. +- **`echo_get_note`** — `path: str` (vault-relative; reject `..`/leading `/`; + warn-not-block on paths outside `routing.json`); `section: str = null` + (a heading name — return only that section, e.g. `Status`); + `max_chars: int = 8000`. Returns `{path, content, frontmatter, truncated?}`. + `readOnlyHint: true`. +- **`echo_get_scope`** — `{scope, scope_updated, sessions_since}`; description + tells the model to confirm scope with the operator when `sessions_since ≥ 3`. + `readOnlyHint: true`. +- **`echo_set_scope`** — `scope: str`; atomic switch (history + replace + + stamp). `idempotentHint: true`. +- **`echo_health`** — `deep: bool = false`. Shallow = doctor checks (vault + reachability, auth, marker/schema, outbox depth). Deep = full `vault_lint` + → `{violations: [{check, path, detail}]}`. `readOnlyHint: true`. + +### 6.2 Write + +- **`echo_capture`** — the default write (route + frontmatter + index + + auto-link + agent-log in one call). Params: `title` (req); `kind: + enum[person, company, concept, reference, meeting, project, area, semantic, + episodic, working, skill, decision]` (omit ⇒ inbox line); `body: str = ""` + (markdown, inline); `tags/aliases/sources: str[]`; `status: str`; + `date: YYYY-MM-DD` (meeting/decision); `domain: enum[business, personal, + learning, systems] = business` (area); `merge_into: str` (slug); + `force: bool = false`; `dry_run: bool = false`. Returns the capture envelope + (`action: created|updated|inbox|duplicate-gate`, `path`, `links_added`, + `near_duplicates?`, `candidates?`). Description spells out the gate contract + (never blind-retry; `merge_into` or confirmed `force`). `destructiveHint: + false`, `idempotentHint: true`. +- **`echo_link`** — `a`, `b` (paths **or resolvable names** — server resolves + via the index). Returns `{a, b, a_changed, b_changed}`. `idempotentHint: true`. +- **`echo_append_note`** — `path`, `line`; whole-line idempotent append. +- **`echo_patch_note`** — `path`, `operation: enum[append, prepend, replace]`, + `target_type: enum[heading, frontmatter, block]`, `target`, `content`. + Description carries the hard-won rules: heading targets are the full + `::`-delimited path from the H1; on a 400 invalid-target the server fetches + the document map and returns the actual heading list in the error — the worst + silent-loss failure becomes self-correcting. `destructiveHint: true`. + (Deliberately **no `echo_put_note` / `echo_delete_note`** in v1 — whole-file + overwrite and deletion stay behind the CLI + operator explicitness; session + logs go through `echo_log_session`.) +- **`echo_triage_inbox`** — `proposals: object[] = []` (PROPOSAL_SCHEMA + + optional `line`), `apply: bool = false`. Empty ⇒ structured listing; with + proposals ⇒ preview, then route + processing-log audit on `apply: true`. + "List → propose → preview → apply only after the operator confirms." +- **`echo_reflect`** — `proposals: object[]` (req), `apply: bool = false`. Same + preview/apply contract as the CLI; description restates: never apply without + the operator's go-ahead, never invent memories. +- **`echo_log_session`** — the session-end bundle (wraps `session_end_op`, #7): + `slug`, `log_body`, `agent_log_line`, `scope?: str`, `reflect?: object[]`, + `apply: bool = false`. Per-step results; heartbeat written last as the commit + marker. + +### 6.3 v1.1 tools (specced now, built after v1 ships) + +- **`echo_rollup_data`** — `period: enum[week, month]`, `date: YYYY-MM-DD`. + Assembles the rollup *digest data* in one call: open threads across + `projects/active/` (each note's `## Status` + `updated:`), inbox items aging + past 7 days, and the period's `## Scope History` entries. The model writes + the prose; the server does the N fetches. Same division of labor as reflect. + `readOnlyHint: true`. +- **`echo_note_history`** — `path: str` → `{versions: [{id, ts, bytes, + summary_line}]}` from the server's shadow write history (§7). `readOnlyHint: + true`. +- **`echo_restore_note`** — `path: str`, `version_id: str`, + `confirm: bool = false` (preview diff unless confirmed). The undo for a bad + `replace` PATCH or merge PUT. Description: operator confirmation required + before calling with `confirm: true`. `destructiveHint: true`. + +Deliberately absent from v1 (documented in the server README): sweep, +bootstrap, migrate, lock/unlock (vault-wide maintenance stays operator-initiated +via slash commands — routine sweeps run as background jobs instead), raw +search/ls/map (recall/resolve/get_note cover reads; add only if transcripts +show the need), and reflect *extraction* (only the model has the conversation — +that division of labor is permanent, not a v1 scope cut). + +## 7. Server internals — where the container earns its keep + +### 7.1 v1 + +- **Warm entity index**: loaded once, invalidated on any index-writing tool and + on a short TTL (`ECHO_MCP_INDEX_TTL`, default 60s) to pick up other clients' + writes. Ends the per-invocation index re-read entirely. +- **Recall index in memory (+ SQLite file at `/data`)**: the BM25 index lives + in process memory, persisted to the container volume — recall answers in + milliseconds with **zero** vault round-trips. This *supersedes the + server-side half of IMPROVEMENT-PLANS #2*; #2's local-state-dir design still + applies to the CLI fallback path. +- **Note read cache**: short-TTL (30–60s) body cache for `get_note` and + recall's neighbourhood expansion — repeated same-session reads of the same + files stop hitting the vault at all. +- **Per-path write serialization**: an internal per-path mutex serializes + conflicting MCP-path writes — since all server-mediated writes flow through + one process, the advisory-lock race window disappears for them. The server + still takes the vault advisory lock around index updates to coordinate with + CLI clients; idempotent appends stay as the last line of defense. +- **Server-side outbox**: the existing `echo_queue` pointed at `/data` + (`ECHO_STATE_DIR=/data`) — writes queue when the vault is down and flush on + recovery/load. One queue at the always-on host instead of per-laptop. +- **Concurrency**: FastMCP serves requests concurrently; guard the in-process + caches with a plain `threading.Lock`; write integrity per the bullet above. +- **Logging**: one line per call to stdout (container log): tool, ms, + ok/queued/gated/error. Never log bodies or keys. + +### 7.2 v1.1 container dividends (backlog, in priority order) + +- **Nightly vault backup** — the vault currently has **no disaster-recovery + story**; if the Obsidian host dies, memory is gone. A timer job walks the + vault via `read_many`, writes a dated zip to `/data/backups/` (rotate 30), + and reports the last-backup age in `/health`. Zero tokens; arguably the + highest-value item in this spec. +- **Shadow write history + undo** — before any PUT / PATCH-replace the server + passes through, snapshot the prior body to `/data/history/` (content- + addressed, per-path ring of ~20 versions). Backs the `echo_note_history` / + `echo_restore_note` tools (§6.3). The vault's additive philosophy finally + gets an undo for its residual risk: a bad `replace` or merge PUT. +- **Background maintenance jobs** — a timer thread running `sweep --fast` + (IMPROVEMENT-PLANS #4) hourly and the decay pass (#5) proposals weekly; both + publish findings to `/health` and the vault-health note rather than + auto-fixing. +- **Ops alerting** — outbox stuck > N hours, vault unreachable > 1h, new lint + violations after a background sweep ⇒ notify via the existing Unraid + notification plumbing (Kuma already watches `/health` for liveness). + Problems surface without anyone polling. +- **Embeddings (explicitly deferred)**: the container is where an embedding + recall tier would land later (model API or local), as an image upgrade — + no plugin change. Noted so nobody bolts it into the plugin. + +## 8. Container & deployment + +- **Image**: `python:3.12-slim` (not alpine — no musl surprises), `pip install + mcp` pinned, copy `skills/echo-memory/scripts/` + `mcp-server/` in. + **Dockerfile must be legacy-format** — the git.alwisp.com CI runner has no + BuildKit: no `# syntax=` line, no `RUN --mount`. +- **Healthcheck**: probe `http://127.0.0.1:/health` — **127.0.0.1, not + localhost** (the ::1-vs-IPv4 lesson from cpas/memer/breedr). `/health` + (unauthenticated, no vault data): `{ok, vault_reachable, outbox_depth, + index_age_s}` — also the Kuma target. +- **Env** (via the PORT template, secrets as `SECRET:` refs): `ECHO_BASE`, + `ECHO_KEY`, `ECHO_OWNER`, `ECHO_MCP_TOKEN`, `ECHO_STATE_DIR=/data`, + `ECHO_MCP_TOOLS=full` (or `core` — see §3 tool profiles), optional + `ECHO_WORKERS/ECHO_TIMEOUT/ECHO_MCP_INDEX_TTL`. Volume: `/data` + (outbox + recall index + backups + shadow history). +- **Startup validation**: fail fast (non-zero exit) if `ECHO_BASE`/`ECHO_KEY`/ + `ECHO_MCP_TOKEN` are missing — a misdeployed container should crash-loop + visibly, not serve errors. +- **CI/deploy**: Gitea workflow on push-to-main (tags/releases do **not** + autobuild on this host) builds the image and the PORT autodeploy webhook + rolls it out; `deploy.unraid.yml` manifest in `mcp-server/`. Publish + `echomcp.alwisp.com` via the usual CF + NPM flow; add the Kuma monitor. +- **Vault adjacency (confirmed: Obsidian runs on ALPHA)**: the Local REST + API's non-encrypted HTTP binding is enabled (2026-07-28) — + **`ECHO_BASE=http://10.2.0.35:27123`** — so the container talks to Obsidian + directly, never through the public `echoapi.alwisp.com` hairpin. All vault + chatter behind a tool call stays host/LAN-local; vault ops stop depending on + Cloudflare/DNS/NPM health entirely (the public chain is only needed for the + `echomcp` ingress). Why HTTP not HTTPS here: the REST API's HTTPS port + (27124) uses a self-signed cert, which `echo.py`'s default-verifying + `HTTPSConnection` rejects — cleartext on the internal network beats adding an + insecure-TLS flag to the client. **Exposure check (do once):** 27123 carries + the vault bearer token in cleartext, so it must not be reachable from + outside — confirm no router/firewall forward for 27123; the only public + doors stay HTTPS 443 via NPM (`echoapi` for the CLI fallback path, which + remains unchanged, and `echomcp` for this server). + +## 9. Client registration & fallback + +- **Claude Code / CoWork**: register as a remote MCP server — + `https://echomcp.alwisp.com/mcp` with the `Authorization: Bearer` header + (project or user scope). No plugin-manifest coupling; the plugin does **not** + register the server. +- **claude.ai**: custom connector with the same URL + token — ECHO memory from + the browser/phone, a surface the plugin could never reach. +- **SKILL.md** gains: *"When the `echo_*` MCP tools are available, prefer them + for every operation they cover; the `$ECHO` CLI recipes are the fallback for + hosts without the connector or when the server is unreachable."* Procedures + unchanged. +- **Fallback matrix**: server up ⇒ tools. Server down, machine has the plugin ⇒ + CLI path exactly as today (including its own offline queue). Both down ⇒ + today's "vault unreachable, proceed without memory". + +## 10. Multi-user note (Andy / Bryan / Gretchen / Arsalan) + +One container serves **one vault**. The image is credential-free, so additional +users are additional deployments of the same image with their own env (their +vault endpoint/key + their own `ECHO_MCP_TOKEN`), e.g. `echomcp-bryan` on +another port/subdomain. Cheap, isolated, no code change. A single multi-tenant +server (token → vault map) is deliberately out of scope — that's CHORUS's +territory. + +## 11. Testing & eval + +1. **Protocol/integration tests** (`eval/test_mcp_server.py`): run the FastMCP + app in-process (or via `httpx` test client) against `mock_olrapi`; assert + initialize/tools-list schemas, auth (no token ⇒ 401; bad token ⇒ 401; + `/health` open), envelope purity on every tool. +2. **Tool behavior**: per tool — happy path, validation failure (unknown param + rejected with field name), vault-down (writes ⇒ `queued: true`, reads ⇒ + actionable error), duplicate-gate flow (gate → `merge_into` retry succeeds). +3. **Parity**: `capture` via MCP and via CLI against twin mock vaults produce + byte-identical notes/index entries (guards Phase 0). +4. **Container smoke** (CI): build image, start with mock env, `/health` green, + one authed `tools/call` round-trip; healthcheck probes 127.0.0.1. +5. **MCP Inspector** manual pass against the deployed container. +6. **Eval set** (mcp-builder Phase 4): 10 read-only Q&A pairs against a seeded + mock vault exercising recall→get_note→resolve chains; `eval/mcp_eval.xml`. + +## 12. Build-session plan of record + +| Phase | Deliverable | Est. size | +|---|---|---| +| 0 | Return-not-print refactor in the plugin tree (`*_op` cores); suites green | ~⅓ the session | +| 1 | `mcp-server/`: FastMCP app, auth middleware, result wrapper, error map, `/health` | ~200 lines | +| 2 | Read tools (load, recall, resolve, get_note, scope×2, health) | wiring | +| 3 | Write tools (capture, link, append/patch, triage, reflect, log_session) | wiring + descriptions | +| 4 | Warm caches + `/data` outbox + startup validation | small | +| 5 | Dockerfile (legacy syntax) + CI workflow + PORT manifest + deploy + CF/NPM publish + Kuma | infra pass | +| 6 | Tests (§11) + Inspector pass + eval XML; SKILL.md/README/CHANGELOG | ~⅓ the session | + +Definition of done: Inspector lists 13 tools against `echomcp.alwisp.com`; a +live session (desktop **and** CoWork) performs a full memory day — load → +recall → capture → gate → merge_into → triage → log_session — without one Bash +call; parity test green; container smoke test in CI; Kuma monitor green.