Add post-1.5.1 planning docs: 9 improvement plans + containerized MCP server spec

docs/IMPROVEMENT-PLANS.md — nine improvements from the 2026-07-28 full-code
review (brief load, offline capture durability, session-end verb, local-first
recall index, incremental sweep, lifecycle decay, supersession, stemming +
alias expansion, resolve-miss learning), sequenced with TODO-1.6 and
ROADMAP-2.0 into one release train (1.6.0 -> 1.6.x -> 2.0 -> 2.1 -> 2.2 -> 2.3).

docs/MCP-SERVER-SPEC.md — build spec for the 2.1 flagship: echo-mcp as a
Docker container on ALPHA (FastMCP, streamable HTTP, bearer auth, published
as echomcp.alwisp.com), talking to Obsidian directly at http://10.2.0.35:27123.
13 v1 tools with result budgets + tool profiles; v1.1 dividends (nightly vault
backup, shadow write history/undo, background sweeps, ops alerting, rollup
data). Prereqs: session-end verb + return-not-print refactor (Phase 0).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/IMPROVEMENT-PLANS.md

@@ -0,0 +1,352 @@
+# echo-memory — Improvement Plans (post-1.5.1 review)
+
+> Status: **planned, not started.** Source: full-code review 2026-07-28.
+> Nine improvements, each scoped for an independent build session. The tenth item
+> from that review — the **MCP server** — has its own detailed spec:
+> `docs/MCP-SERVER-SPEC.md`.
+>
+> Sequencing is at the bottom. Nothing here conflicts with `TODO-1.6.md` or
+> `ROADMAP-2.0.md`; release targets assume 1.6 (skills migration) ships first.
+
+---
+
+## 1. Token-budgeted load (`load --brief`) + parallel orientation reads
+
+**Problem.** The SessionStart hook injects the *full text* of six files into every
+session. Live measurement: 15.9KB of hook context on a single cold start, and two of
+the six files grow without bound (operator-preferences `## Observations`,
+current-context `## Scope History`). Every session pays this tax before any work
+happens. Separately, `cmd_load` (echo.py) issues its six GETs serially even though
+`read_many()` exists.
+
+**Design.**
+- New `echo.py load --brief` (and make the SessionStart hook use it):
+  - marker → one line (`schema_version`, bootstrapped yes/no).
+  - operator-preferences → `## Fact / Pattern` in full; `## Observations` capped at
+    the last 10 lines with a `(+N older — /echo-load for full)` note.
+  - current-context → `## Scope` + `scope_updated` + sessions-since (i.e. the
+    `scope show` output); omit `## Scope History`.
+  - heartbeat → the pointer line + only the `## Summary`/`## Outcomes` sections of
+    the pointed-at session log (fetch it, extract those headings, cap ~30 lines).
+  - today's daily note → `## Agent Log` lines only (or "absent").
+  - inbox → count + age of oldest item, not the contents (that's all the reconcile
+    needs; `/echo-triage --list` has the details).
+- Budget guard: after assembly, if the brief output still exceeds
+  `ECHO_LOAD_BUDGET` chars (default ~8000), truncate lowest-priority sections first
+  (observations → agent log → session summary) with explicit `(truncated)` markers.
+- Full `load` unchanged; `/echo-load` keeps using it.
+- Fetch all reads via `read_many()` (the 6 targets + the heartbeat-pointed session
+  log + the sessions listing fallback) instead of the serial loop at
+  `echo.py:640`. Cache-put still applies per path.
+
+**Files.** `echo.py` (cmd_load), `echo_hook_session_start.py`, SKILL.md (document
+brief-vs-full), `eval/test_features.py`.
+
+**Tests.** Mock vault with oversized preferences/history files → assert brief output
+under budget, sections present, truncation markers correct; hook emits brief.
+
+**Size/target.** Small-medium. **1.6.x point release** — highest leverage per line.
+
+---
+
+## 2. Local-first recall index
+
+**Problem.** `echo_recall.update_note()` does GET-whole-index → add → PUT-whole-index
+against the vault, under the global advisory lock, on every capture and every corpus
+`put`. The index carries full BM25 postings for the whole corpus, so this round-trip
+is O(vault) network per write and is the emerging bottleneck + lock hot-spot as the
+vault grows past a few hundred notes.
+
+**Design.**
+- The **live** recall index moves to the local state dir
+  (`~/.echo-memory/recall-index.json`, honoring `ECHO_STATE_DIR`) keyed by endpoint
+  (hash of `BASE`) so multiple vaults don't collide.
+- `update_note()` becomes a local read-modify-write (file lock via `os.O_EXCL`
+  sidecar or atomic `os.replace`), **no vault round-trip, no advisory lock**.
+- The vault copy (`_agent/index/recall-index.json`) becomes a **snapshot**, written
+  by `sweep.py` and at session end (piggyback on the heartbeat write / future
+  `session-end` verb). It exists so a *fresh machine* can seed its local index
+  without a full rebuild.
+- Freshness rule on recall: local index used when present; if the vault snapshot's
+  embedded `built` stamp is newer than the local one (another client swept), pull it.
+  Staleness across clients is tolerable — recall degrades gracefully and improvement
+  #4 (incremental sweep) trues it up cheaply.
+- Schema bump: recall-index schema 3 adds `built` (ISO timestamp) + `endpoint_hash`.
+  Old schema-2 vault copies are still readable as seeds.
+- The **entity index stays in the vault** (it is small, and it is the shared routing
+  authority for concurrent clients) — only the BM25 index moves.
+
+**Files.** `echo_recall.py` (load/save/update_note/rebuild), `echo_queue.py`
+(state-dir helpers shared), `sweep.py`, `echo.py` (cmd_put upkeep path), routing.json
+(the vault snapshot path is unchanged), README performance section.
+
+**Tests.** Capture with vault mock → assert zero recall-index PUTs; sweep → snapshot
+written; fresh state dir + existing snapshot → seeded without rebuild; two-vault
+(endpoint) separation.
+
+**Size/target.** Medium. **Pair with #4 in one index-focused minor** (they touch the
+same maintenance loop). Fine to land before or after 2.0.
+
+---
+
+## 3. Offline durability for the high-level ops (capture path)
+
+**Problem.** The low-level verbs queue on outage via `safe_request`, but
+`_append_to_existing()` and `ensure_daily_log()` (echo_ops.py) call `echo.request`
+directly — so during an outage a raw `append` survives, while a `capture` **update**
+and its Agent Log line silently vanish. Inverted from the documented contract
+("capture is the default write").
+
+**Design.** Queue *semantic* records, not byte-level requests, for the high-level
+path:
+- New outbox record type `{"op": "capture", "args": {...}}` alongside the existing
+  raw records. On an unreachable vault, `capture` short-circuits: enqueue the full
+  argument set (title/kind/body/tags/aliases/sources/date/domain/merge_into) and
+  report `queued (offline): capture …`.
+- `flush()` replays `op:capture` records by calling `echo_ops.capture(...)` against
+  the *current* index — so routing, the duplicate gate, aliasing, and linking all
+  re-run with fresh state instead of replaying stale byte writes. A gate stop on
+  replay (exit 76) keeps the record queued with a `needs_attention` flag and is
+  surfaced by `load`/`flush` output ("1 queued capture needs --merge-into/--force").
+- Rationale for semantic-over-byte: a byte replay of `_append_to_existing`'s PATCH
+  could target a heading that moved, and the create-vs-update decision made offline
+  may be wrong by replay time. Re-running the op is the idempotent, correct unit
+  (the dated-bullet idempotency key already prevents double-append).
+- `ensure_daily_log` stays best-effort but routes its POST/PATCH through
+  `safe_request` so a standalone agent-log line survives an outage too.
+- reflect/triage `--apply` inherit this for free (they call capture).
+
+**Files.** `echo_ops.py`, `echo_queue.py` (record schema v2 + replay dispatch),
+`eval/test_offline_queue.py`.
+
+**Tests.** Fault-injected mock: capture-update offline → queued; flush replays via
+capture; gate-on-replay keeps record + flags; agent-log line queued; no duplicate
+bullets on double flush.
+
+**Size/target.** Small-medium. **1.6.x point release** — it's a correctness gap.
+
+---
+
+## 4. Incremental sweep via content hashes
+
+**Problem.** Human edits in Obsidian and writes by other clients never touch the
+entity/recall indexes; drift is only corrected by a full sweep someone must remember
+to run. Full sweep re-reads the whole vault (fast now, but O(vault)).
+
+**Design.**
+- Entity index entries and the recall-index snapshot gain a per-note
+  `h` — SHA-1 of the note body (entity index schema 2; tolerated-absent for old
+  entries).
+- `sweep.py --fast`: walk the listing (cheap), `read_many` only paths that are
+  (a) new, (b) missing a stored hash, or (c) whose hash mismatches after fetch — to
+  avoid fetching everything just to hash it, fast mode fetches only paths whose
+  **listing is new/gone** plus a rotating shard (e.g. `hash(path) % 7 == weekday`)
+  of existing notes, so the whole vault is verified over a week of fast sweeps while
+  each run stays tiny. `--fast --all-shards` forces full verification.
+- Deletions: listing walk catches removed files → drop index + recall entries.
+- Auto-run: `load` (brief or full) runs `sweep --fast` opportunistically when the
+  last fast-sweep stamp (state dir) is older than `ECHO_FAST_SWEEP_DAYS` (default 7),
+  in the background-tolerant sense: bounded by the same read_many concurrency, and
+  skipped entirely when offline.
+- `/echo-health` reports last fast/full sweep ages.
+
+**Files.** `sweep.py`, `echo_index.py` (schema 2 + hash field), `echo_recall.py`
+(snapshot hash reuse), `echo.py` (load hook-in), `vault_lint.py` (report), migrate
+note (index schema is machine-owned; no vault migration needed).
+
+**Tests.** Mock: edit a note out-of-band → fast sweep in that shard re-indexes it;
+delete → entry dropped; stamp gating; full sweep unchanged.
+
+**Size/target.** Medium. **Ship with #2** (same schema-bump train).
+
+---
+
+## 5. Memory lifecycle — enforce forgetting
+
+**Problem.** The vault only accretes. Working memory is "time-boxed" by convention
+only; Observations trimming is manual prose; stale-active detection only reports.
+Recall precision degrades as dead weight accumulates.
+
+**Design.**
+- `capture --kind working` stamps `expires: <today + ECHO_WORKING_TTL_DAYS>`
+  (default 14; `--ttl <days>` overrides). Template updated to carry the field.
+- `vault_lint.py` new checks: `expired-working` (past `expires`),
+  `stale-observations` (Observations > 30 bullets) — advisory, like the rest.
+- `sweep.py --decay` (dry-run by default, `--apply` to write; **always
+  preview-first**, honoring the operating contract):
+  - expired working notes → `status: archived` + move body under a dated
+    `## Archived` marker (never delete);
+  - `projects/active/` untouched > 30 days → *propose* `on-hold` (folder move +
+    status, the usual paired change) — apply only the ones confirmed;
+  - Observations > 30 → trim oldest into a dated `_agent/memory/episodic/`
+    overflow note (`observations-archive-YYYY.md` append), keeping the last 30
+    in place. Nothing is lost, the hot file stays small (compounds with #1).
+- `/echo-health` output nudges: "run `sweep --decay` to review N aging items".
+
+**Files.** `echo_ops.py` (`--ttl`/expires stamp), `echo_index.py`
+(KIND_REQUIRED_FM unchanged; working gains optional expires), `vault_lint.py`,
+`sweep.py`, scaffold working-memory template, SKILL.md (decay section),
+routing-map note.
+
+**Tests.** Mock: expired note proposed+archived on apply; observations trim keeps
+last 30 and appends overflow; stale-active proposal list; dry-run writes nothing.
+
+**Size/target.** Medium. Feature minor (1.7.x).
+
+---
+
+## 6. Contradiction / supersession handling
+
+**Problem.** A new fact that contradicts a stored one lands beside it; both surface
+in recall with no signal about which is current. Biggest remaining trust gap in the
+write path.
+
+**Design.**
+- **Write side:** `capture --supersedes "<slug-or-quoted-line>"`:
+  - if it names an entity slug → the old entity gets `status: superseded` +
+    a `superseded_by:` frontmatter path; new note carries `supersedes:` back-link
+    in `## Related`. Recall's STATUS_FACTOR gains `superseded: 0.5`.
+  - if it quotes a Fact/Pattern or Observations line in operator-preferences →
+    the line is struck (`~~old~~ (superseded YYYY-MM-DD)`) and the new line
+    appended, one PATCH each, in the same invocation.
+- **Detect side (deterministic assist, model decides):** `echo_reflect.classify`
+  gains a `conflict scan` — for each proposal, run its distinctive tokens
+  (`_sig_tokens`) against the Fact/Pattern + recent Observations lines; overlapping
+  lines are attached as `_conflicts: [...]` and the preview row renders
+  `update-or-conflict?` with the matched line(s). The model (which has the
+  conversation) decides supersede-vs-append; the script only guarantees the
+  question gets asked before the write.
+- Lint: `superseded_by` pointing at a missing note → violation.
+
+**Files.** `echo_ops.py`, `echo_reflect.py`, `echo_recall.py` (STATUS_FACTOR),
+`echo.py` (flag plumbing), `vault_lint.py`, SKILL.md + operating-contract
+(supersession is additive: strike, never delete).
+
+**Tests.** Mock: supersede entity → status/backlinks both sides; supersede line →
+strike + append idempotent; reflect preview flags a planted conflict; recall ranks
+superseded below active twin.
+
+**Size/target.** Medium. Feature minor (1.7.x/1.8) — after #5 (shares the
+status-vocabulary touchpoints).
+
+---
+
+## 7. `session-end` bundle verb
+
+**Problem.** Ending a session correctly is 4+ invocations (session-log PUT, Agent
+Log append, heartbeat PUT, optional scope set, optional reflect apply). Cost per
+session, and a partial failure leaves broken orientation state (log written,
+heartbeat stale).
+
+**Design.**
+- `echo.py session-end <bundle.json> [--apply]`, bundle:
+  ```json
+  {
+    "slug": "echo-mcp-spec",
+    "log_body": "<full session-log markdown, frontmatter included>",
+    "agent_log_line": "- 2026-07-28: wrote MCP spec [[...]]",
+    "scope": "optional new scope text",
+    "reflect": [ { ...PROPOSAL_SCHEMA... } ]
+  }
+  ```
+- Dry-run previews the whole plan (paths + reflect preview). `--apply`, in order,
+  under **one** lock acquisition: PUT `_agent/sessions/<date>-<HHMM>-<slug>.md`
+  (HHMM from `ECHO_NOW` env or wall clock, validated against the canonical regex) →
+  agent-log line (ensure_daily_log) → reflect proposals via capture → scope set if
+  given → heartbeat PUT **last** (it's the commit marker: a crash before it leaves
+  the previous pointer intact, never a dangling one).
+- Reports per-step ok/queued/gated; offline → every step queues (rides on #3).
+- Stop hook reason text updated to name the one command; `/echo-reflect` docs point
+  session logging at it. **Prerequisite for the MCP `echo_log_session` tool** —
+  build this verb first so CLI and MCP share one implementation.
+
+**Files.** new `echo_session.py`, `echo.py` (subcommand), `echo_hook_stop.py`
+(REASON text), SKILL.md (Session Logging section), commands/echo-reflect.md.
+
+**Tests.** Mock: full bundle lands all five artifacts; heartbeat-last ordering
+(inject failure mid-way → heartbeat untouched); dry-run writes nothing; offline
+queues all steps.
+
+**Size/target.** Small-medium. **1.6.x point release** (and before the MCP build).
+
+---
+
+## 8. Recall robustness — stemming + alias query expansion
+
+**Problem.** BM25 is purely lexical: "deploy"/"deployment"/"deployed" are distinct
+terms; a paraphrased query misses. The 8-query gold set can't see this class of miss.
+
+**Design.**
+- **Stemming:** a light suffix-stripper (Porter-lite: s/es/ed/ing/ly/tion/ment/ness
+  families, ~40 lines, pure stdlib) applied in `tokenize()` at **both** index and
+  query time. Index schema bump (stemmed postings) → auto-rebuild on next
+  recall/sweep, exactly like the 1→2 transition.
+- **Alias query expansion:** at recall time, run the query against
+  `fuzzy_candidates()` (top 2, score ≥ 0.5); fold the matched entities' title +
+  alias tokens into the BM25 query as **down-weighted expansion terms** (×0.5 idf
+  contribution) so "the ECHO plugin" also scores `echo-memory`-alias vocabulary.
+  Exact-match seeding stays as-is; expansion never *creates* hits ranked above
+  genuine lexical matches without corpus support.
+- **Eval:** grow the gold set with ≥6 paraphrase queries (morphological variants +
+  alias phrasings); publish before/after recall@5/MRR in the README table.
+
+**Files.** `echo_recall.py` (tokenize, INDEX_SCHEMA, score, recall), new
+`echo_stem.py` (pure function, unit-testable), `eval/run_eval.py` + gold set.
+
+**Tests.** Stemmer unit table (word → stem, incl. non-cases); paraphrase gold
+queries pass; schema-2 index discarded and rebuilt cleanly.
+
+**Size/target.** Small-medium. Ride the **same index-schema train as #2/#4**.
+
+---
+
+## 9. Resolve-miss telemetry → alias suggestions
+
+**Problem.** Every resolve that misses but ends in a confirmed candidate
+(`--merge-into`, or the operator picking a candidate) is a free mention→entity
+training pair, currently discarded.
+
+**Design.**
+- `echo_index.resolve()` miss + `fuzzy_candidates` non-empty → append one NDJSON
+  record to the local state dir (`~/.echo-memory/resolve-misses.ndjson`):
+  `{mention, candidates:[slugs], ts?}` (no wall clock needed — session date via
+  ECHO_TODAY). `capture --merge-into <slug>` appends the *confirmation* record
+  `{mention, resolved: slug}` (this already learns the alias immediately — that
+  path stays; the log covers the cases that never reach --merge-into).
+- `sweep.py` (and `/echo-health`) read the log: any `{mention → slug}` pair
+  confirmed ≥2 times, or any mention that repeatedly fuzzy-matched a single
+  candidate ≥3 times without confirmation, is **proposed** as an alias
+  (`sweep --apply` writes it into the note's `aliases:` frontmatter + index, the
+  existing fold-back path). Log entries consumed on apply; file capped (rotate at
+  ~500 lines).
+- Privacy note: mentions can contain operator phrasing — the log lives in the
+  state dir (never the vault), same trust level as the offline queue.
+
+**Files.** `echo_index.py` (log hooks), `echo_ops.py` (confirmation hook),
+`sweep.py` (suggest/apply), `vault_lint.py` or sweep report (surface count),
+SKILL.md one-liner.
+
+**Tests.** Miss → record written; confirm ×2 → sweep proposes; apply folds alias +
+consumes log; cap/rotation.
+
+**Size/target.** Small. Feature minor, pairs naturally with #8.
+
+---
+
+## Sequencing (unified with TODO-1.6 and ROADMAP-2.0)
+
+TODO-1.6 folds in as the opening train — the skills migration restructures the
+files everything below touches, so it goes first. ROADMAP-2.0 stays a **pure
+packaging major** (its own stated thesis) and is *not* folded into any feature
+train; it slots between the quick wins and the MCP server so the MCP work lands
+on the final packaging baseline (single command format, Gitea releases, one
+manifest-verification pass).
+
+| Release | Items | Rationale |
+|---|---|---|
+| **1.6.0** | TODO-1.6: skills-format migration (+ per-skill `allowed-tools`, `disable-model-invocation` on write-heavy entries, `$ECHO` block dedupe) · lint blind-spot fix (retired-dir/leaf-README) · plugin.json `homepage` | Restructures the command/SKILL files later work edits; allowed-tools removes prompt friction for every verb added below. Vault link repointing (TODO-1.6 §3) is vault data, not plugin code — any session. |
+| **1.6.x quick wins** | #1 brief load · #3 offline capture · #7 session-end (+ return-not-print refactor, MCP spec Phase 0, if it fits) | Small, correctness/efficiency, no schema changes. #7 unblocks the MCP tool. Users get these without a reinstall — don't gate them behind 2.0. |
+| **2.0** | ROADMAP-2.0 unchanged: delete legacy `commands/` · artifact policy (Gitea releases, gitignore, tags) · Codex note · README layout | Packaging only, no feature riders — a bad feature must never force reverting a packaging break. |
+| **2.1** | MCP server (containerized, `echomcp.alwisp.com`) — `docs/MCP-SERVER-SPEC.md` | Needs #7 + Phase 0. Lands after 2.0 so its docs target one command format and the Gitea release channel exists. Deployed via CI + PORT, not the plugin manifest. |
+| **2.2 index train** | #2 local-first recall (CLI-path half) · #4 incremental sweep · #8 stemming+expansion | One schema-bump train; all touch the same index/sweep loop. The container already keeps its index warm in memory (spec §7), so #2 here covers the CLI fallback path; #4's fast sweep also becomes the container's background job. |
+| **2.3 memory train** | #5 lifecycle/decay · #6 supersession · #9 resolve-miss learning | Capability tier; #6 after #5 (status vocabulary). |

docs/MCP-SERVER-SPEC.md

@@ -0,0 +1,391 @@
+# echo-mcp — MCP Server Build Spec (containerized)
+
+> Status: **spec for a future build session** (written 2026-07-28 against v1.5.1;
+> revised same day: **remote container architecture**, operator decision — heavy
+> lifting belongs in a deployed container, not on any one machine).
+> Companion plan for the other nine review items: `docs/IMPROVEMENT-PLANS.md`.
+>
+> **Prerequisites before starting this build:**
+> 1. The `session-end` verb (IMPROVEMENT-PLANS #7) — the MCP tool wraps it.
+> 2. Phase 0 below (the return-not-print refactor) — lands in the plugin tree
+>    first and is independently shippable.
+
+---
+
+## 1. Why an MCP server
+
+Every ECHO operation today rides through Bash: resolve `$ECHO` (with the CoWork
+path-fallback snippet copy-pasted everywhere), quote a Python invocation, run
+it, re-parse stdout. Costs:
+
+- **Tokens** — each memory op carries bash scaffolding + output re-parsing; a
+  capture is ~3× the tokens of a typed tool call.
+- **Fragility** — quoting hazards, the `${CLAUDE_PLUGIN_ROOT}` sandbox mismatch
+  (the whole 1.4.2 release), Windows `python3`-vs-`python` dispatch.
+- **Reach** — surfaces without a Python-capable shell can't use ECHO at all.
+- **Efficiency** — every CLI invocation is a cold Python start that re-reads the
+  entity index from the vault over HTTPS.
+
+What the server does **not** replace: SKILL.md remains the procedure authority
+(when to load, reconcile, search-first, third-person, etc.). The server replaces
+the *mechanics* — tools are the verbs, the skill is the discipline.
+
+## 2. Architecture: remote container (decided)
+
+A standalone Docker container on the Unraid box (ALPHA), published behind
+Cloudflare + NPM as **`echomcp.alwisp.com`**, speaking **MCP streamable HTTP
+(stateless JSON)** with bearer-token auth. Chosen over the earlier local-stdio
+draft for four reasons:
+
+1. **The backend is already remote.** Clients round-trip to
+   `echoapi.alwisp.com` today; a server co-located with the vault host turns
+   every vault op into a LAN hop and collapses N client→vault calls into one
+   client→server call.
+2. **Any device, no per-machine anything.** Desktop, CoWork sandbox, claude.ai
+   (custom connector), mobile — same URL + token. The `${CLAUDE_PLUGIN_ROOT}`
+   CoWork registration risk from the stdio draft disappears entirely.
+3. **Credentials consolidate server-side.** The vault key lives in the
+   container env (PORT secret store); the image is credential-free; clients
+   hold only an MCP bearer token. The per-user baked-key builds (1.4.x) become
+   unnecessary for any surface that has MCP. Rotation = redeploy.
+4. **Room to grow.** The plugin's pure-stdlib constraint exists because its
+   scripts run in arbitrary sandboxes. The container is ours: real SDK,
+   SQLite/in-memory indexes that stay warm, background jobs, embeddings later —
+   all without touching the plugin.
+
+**What stays client-side (unchanged and shipped):** the skill + CLI path — it
+is the fallback when the server or network is down, the offline queue + read
+cache keep their per-machine role there, and the SessionStart/Stop **hooks stay
+CLI-based** (hooks run shell commands regardless of MCP).
+
+**Cost acknowledged:** a second service to run and an exposed auth surface.
+Mitigations are existing infra: CF + NPM cert, long random bearer token, Kuma
+monitor, PORT deploy/rollback. Server down ⇒ exactly today's behavior.
+
+## 3. Decisions (made — don't relitigate in the build session)
+
+| Decision | Choice | Why |
+|---|---|---|
+| Language | **Python** | Imports the existing `echo_*.py` modules in-process — the entire ops layer is reused, not wrapped. |
+| SDK | **Official `mcp` Python SDK (FastMCP)** | The stdlib-only rule doesn't bind inside our own image. Hand-rolling the protocol (stdio draft) is no longer justified. |
+| Transport | **Streamable HTTP, stateless JSON** | Remote, multi-client, simplest to scale; SSE/sessions explicitly avoided. |
+| Auth | **Static bearer token** (`ECHO_MCP_TOKEN`), constant-time compare, 401 otherwise | Single-operator service; OAuth is overkill. Token from the PORT secret store. |
+| Vault credentials | Container env: `ECHO_BASE`, `ECHO_KEY`, `ECHO_OWNER` (SECRET: refs in the PORT template) | `echo_config.py` already resolves env-first — zero code change. Image ships no secrets. |
+| Server name / repo home | `echo-mcp`; lives in this repo under `mcp-server/` (own Dockerfile), deployed from `git.alwisp.com/jason/echo` CI | One canonical tree — the server vendors `skills/echo-memory/scripts/` into the image at build time; no second hand-maintained copy (the Codex-drift lesson). |
+| Tool prefix | `echo_` | Namespace safety next to other servers. |
+| Result shape | `structuredContent` (the `echo_output.envelope` dict) + a 1–3 line human text block | Envelope shape already exists. |
+| Statefulness | Stateless protocol; warm in-process caches | See §7. |
+| Version target | **2.1** (after the 1.6.x train and the 2.0 packaging major) | Needs #7 + Phase 0; docs then target the single post-2.0 command format. |
+| Local stdio variant | **Dropped for v1** (documented, not built) | The CLI/skill fallback covers the no-server case; two transports = two test matrices for little gain. Phase 0 keeps the door open — the `*_op` core is transport-agnostic. |
+| Result budgets | Every read tool takes an explicit size budget (`budget_chars` / `max_chars`) and truncates with a marker + "call X for more" | The single biggest token lever: one right-sized answer instead of follow-up fetches. |
+| Tool profiles | `ECHO_MCP_TOOLS=core\|full` env: `core` exposes only load/recall/capture/triage_inbox/log_session/health | Tool schemas cost context on every surface that lists them; the claude.ai connector doesn't need the escape hatches. Desktop registers `full`. |
+
+## 4. Phase 0 — the return-not-print refactor (prerequisite, lands in the plugin)
+
+Unchanged from the stdio draft, and still first: the ops layer prints results
+(even `--json` mode prints from inside `echo_ops.capture`) and returns exit
+codes; the server needs return values.
+
+1. Each high-level op gets a **core function returning an envelope dict**,
+   raising `echo.EchoError` on failure; CLI entry points become thin printing
+   wrappers:
+
+   | Module | New core fn | CLI wrapper keeps |
+   |---|---|---|
+   | `echo_ops` | `capture_op`, `resolve_op`, `link_op` | `capture/resolve/link` |
+   | `echo_recall` | `recall_op(query, limit)` | `recall` |
+   | `echo_triage` | `list_op()` · `route_op(proposals, apply)` | `list_inbox/apply` |
+   | `echo_reflect` | `apply_op(proposals, apply)` | `apply` |
+   | `echo.py` | `load_op(brief)` · `scope_show_op/scope_set_op` · `doctor_op` | `cmd_*` |
+   | `echo_session` (new, #7) | `session_end_op(bundle, apply)` | subcommand |
+
+2. Special results are **data, not exit codes**: duplicate gate ⇒
+   `{ok: false, action: "duplicate-gate", candidates}` (CLI maps to exit 76);
+   offline queueing ⇒ `{ok: true, queued: true}`.
+3. Helper chatter (`ok: PUT …`) moves behind a `notify(msg)` callback
+   defaulting to stderr; delete the `redirect_stdout` hack in `capture`.
+4. Tests: existing suites pass unchanged; new unit tests hit `*_op` directly
+   against the mock and assert envelopes.
+
+Shippable on its own as a 1.6.x refactor with zero behavior change.
+
+## 5. Server application (`mcp-server/app.py`)
+
+- **FastMCP** app, streamable HTTP, stateless. One `@mcp.tool` per tool in §6,
+  each a thin adapter: validate → call the `*_op` core → wrap.
+- **Auth middleware**: require `Authorization: Bearer <ECHO_MCP_TOKEN>` on every
+  request (constant-time compare); 401 with no detail otherwise. Additionally
+  bind the app to the container interface only; exposure policy lives at
+  NPM/Cloudflare.
+- **Result wrapper**:
+
+  ```python
+  def tool_result(env: dict, summary: str) -> ...:
+      # content: [{type: "text", text: summary}]  (1–3 lines, human)
+      # structuredContent: env                     (echo_output envelope)
+      # isError: not env.get("ok", True) — except the deliberate non-errors below
+  ```
+
+- **Error map** (tool-level results, never protocol errors):
+
+  | Condition | Result |
+  |---|---|
+  | Vault unreachable on a **read** | `isError: true` — "vault unreachable (Obsidian/REST plugin likely down); proceed without memory, writes will queue server-side". |
+  | Vault unreachable on a **write** | **Not an error**: `{ok: true, queued: true}` — the server-side outbox (§7) replays when the vault returns. |
+  | 404 | `isError: true`, `{code: "not-found", path}`. |
+  | Duplicate gate | **Not an error**: `{ok: false, action: "duplicate-gate", candidates}` + text naming the two resolutions (`merge_into` / `force`) — the model must decide, not blind-retry. |
+  | Lock held | `{ok: false, action: "lock-held", holder}`. |
+  | Misconfigured deployment (no `ECHO_BASE`/`ECHO_KEY`) | `isError: true` — "server deployment is missing vault credentials — operator: check the PORT template env". Startup also fails loudly (see §8 healthcheck). |
+  | Anything else | `isError: true`, first line only; traceback to the container log. |
+
+- **Validation**: FastMCP/pydantic handles types/enums/required; add
+  `model_config = ConfigDict(extra="forbid")` so unknown params (model typos)
+  fail with a field-naming message.
+
+## 6. Tool surface (13 tools)
+
+Same surface as the stdio draft **minus `echo_configure`** — with a remote
+server there is no per-machine config to import; credentials are
+deployment-side. Descriptions are written for the model: what it does, when to
+use it, what it returns, ≤3 sentences.
+
+### 6.1 Orientation & read
+
+- **`echo_load`** — cold-start orientation. `brief: bool = true`
+  (IMPROVEMENT-PLANS #1 digest; `false` = full six-file dump). Returns
+  `{sections: {marker, preferences, scope, last_session, today, inbox_count,
+  inbox_oldest_days}, queued_flushed, offline}`. Also flushes the server-side
+  outbox. `readOnlyHint: false` (flush), `openWorldHint: true`.
+- **`echo_recall`** — `query: str`, `limit: int = 6 (1–20)`,
+  `include_linked: bool = true`, `budget_chars: int = 4000 (500–20000)`. Returns
+  the `recall --json` shape (`primary` / `linked` with
+  path/score/type/updated/status/excerpt) **packed to the budget**: the server
+  allocates excerpt space by score, so the model gets the most relevant content
+  in one right-sized answer instead of follow-up fetches. Description: truncated
+  hits are marked — call `echo_get_note` for full content.
+  `readOnlyHint: true`, `idempotentHint: true`.
+- **`echo_resolve`** — `mention: str` → match `{slug, path, kind, title,
+  aliases}` or `{match: false, suggest_slug, candidates}`. "Call before creating
+  any note by hand; `echo_capture` does this automatically." `readOnlyHint: true`.
+- **`echo_get_note`** — `path: str` (vault-relative; reject `..`/leading `/`;
+  warn-not-block on paths outside `routing.json`); `section: str = null`
+  (a heading name — return only that section, e.g. `Status`);
+  `max_chars: int = 8000`. Returns `{path, content, frontmatter, truncated?}`.
+  `readOnlyHint: true`.
+- **`echo_get_scope`** — `{scope, scope_updated, sessions_since}`; description
+  tells the model to confirm scope with the operator when `sessions_since ≥ 3`.
+  `readOnlyHint: true`.
+- **`echo_set_scope`** — `scope: str`; atomic switch (history + replace +
+  stamp). `idempotentHint: true`.
+- **`echo_health`** — `deep: bool = false`. Shallow = doctor checks (vault
+  reachability, auth, marker/schema, outbox depth). Deep = full `vault_lint`
+  → `{violations: [{check, path, detail}]}`. `readOnlyHint: true`.
+
+### 6.2 Write
+
+- **`echo_capture`** — the default write (route + frontmatter + index +
+  auto-link + agent-log in one call). Params: `title` (req); `kind:
+  enum[person, company, concept, reference, meeting, project, area, semantic,
+  episodic, working, skill, decision]` (omit ⇒ inbox line); `body: str = ""`
+  (markdown, inline); `tags/aliases/sources: str[]`; `status: str`;
+  `date: YYYY-MM-DD` (meeting/decision); `domain: enum[business, personal,
+  learning, systems] = business` (area); `merge_into: str` (slug);
+  `force: bool = false`; `dry_run: bool = false`. Returns the capture envelope
+  (`action: created|updated|inbox|duplicate-gate`, `path`, `links_added`,
+  `near_duplicates?`, `candidates?`). Description spells out the gate contract
+  (never blind-retry; `merge_into` or confirmed `force`). `destructiveHint:
+  false`, `idempotentHint: true`.
+- **`echo_link`** — `a`, `b` (paths **or resolvable names** — server resolves
+  via the index). Returns `{a, b, a_changed, b_changed}`. `idempotentHint: true`.
+- **`echo_append_note`** — `path`, `line`; whole-line idempotent append.
+- **`echo_patch_note`** — `path`, `operation: enum[append, prepend, replace]`,
+  `target_type: enum[heading, frontmatter, block]`, `target`, `content`.
+  Description carries the hard-won rules: heading targets are the full
+  `::`-delimited path from the H1; on a 400 invalid-target the server fetches
+  the document map and returns the actual heading list in the error — the worst
+  silent-loss failure becomes self-correcting. `destructiveHint: true`.
+  (Deliberately **no `echo_put_note` / `echo_delete_note`** in v1 — whole-file
+  overwrite and deletion stay behind the CLI + operator explicitness; session
+  logs go through `echo_log_session`.)
+- **`echo_triage_inbox`** — `proposals: object[] = []` (PROPOSAL_SCHEMA +
+  optional `line`), `apply: bool = false`. Empty ⇒ structured listing; with
+  proposals ⇒ preview, then route + processing-log audit on `apply: true`.
+  "List → propose → preview → apply only after the operator confirms."
+- **`echo_reflect`** — `proposals: object[]` (req), `apply: bool = false`. Same
+  preview/apply contract as the CLI; description restates: never apply without
+  the operator's go-ahead, never invent memories.
+- **`echo_log_session`** — the session-end bundle (wraps `session_end_op`, #7):
+  `slug`, `log_body`, `agent_log_line`, `scope?: str`, `reflect?: object[]`,
+  `apply: bool = false`. Per-step results; heartbeat written last as the commit
+  marker.
+
+### 6.3 v1.1 tools (specced now, built after v1 ships)
+
+- **`echo_rollup_data`** — `period: enum[week, month]`, `date: YYYY-MM-DD`.
+  Assembles the rollup *digest data* in one call: open threads across
+  `projects/active/` (each note's `## Status` + `updated:`), inbox items aging
+  past 7 days, and the period's `## Scope History` entries. The model writes
+  the prose; the server does the N fetches. Same division of labor as reflect.
+  `readOnlyHint: true`.
+- **`echo_note_history`** — `path: str` → `{versions: [{id, ts, bytes,
+  summary_line}]}` from the server's shadow write history (§7). `readOnlyHint:
+  true`.
+- **`echo_restore_note`** — `path: str`, `version_id: str`,
+  `confirm: bool = false` (preview diff unless confirmed). The undo for a bad
+  `replace` PATCH or merge PUT. Description: operator confirmation required
+  before calling with `confirm: true`. `destructiveHint: true`.
+
+Deliberately absent from v1 (documented in the server README): sweep,
+bootstrap, migrate, lock/unlock (vault-wide maintenance stays operator-initiated
+via slash commands — routine sweeps run as background jobs instead), raw
+search/ls/map (recall/resolve/get_note cover reads; add only if transcripts
+show the need), and reflect *extraction* (only the model has the conversation —
+that division of labor is permanent, not a v1 scope cut).
+
+## 7. Server internals — where the container earns its keep
+
+### 7.1 v1
+
+- **Warm entity index**: loaded once, invalidated on any index-writing tool and
+  on a short TTL (`ECHO_MCP_INDEX_TTL`, default 60s) to pick up other clients'
+  writes. Ends the per-invocation index re-read entirely.
+- **Recall index in memory (+ SQLite file at `/data`)**: the BM25 index lives
+  in process memory, persisted to the container volume — recall answers in
+  milliseconds with **zero** vault round-trips. This *supersedes the
+  server-side half of IMPROVEMENT-PLANS #2*; #2's local-state-dir design still
+  applies to the CLI fallback path.
+- **Note read cache**: short-TTL (30–60s) body cache for `get_note` and
+  recall's neighbourhood expansion — repeated same-session reads of the same
+  files stop hitting the vault at all.
+- **Per-path write serialization**: an internal per-path mutex serializes
+  conflicting MCP-path writes — since all server-mediated writes flow through
+  one process, the advisory-lock race window disappears for them. The server
+  still takes the vault advisory lock around index updates to coordinate with
+  CLI clients; idempotent appends stay as the last line of defense.
+- **Server-side outbox**: the existing `echo_queue` pointed at `/data`
+  (`ECHO_STATE_DIR=/data`) — writes queue when the vault is down and flush on
+  recovery/load. One queue at the always-on host instead of per-laptop.
+- **Concurrency**: FastMCP serves requests concurrently; guard the in-process
+  caches with a plain `threading.Lock`; write integrity per the bullet above.
+- **Logging**: one line per call to stdout (container log): tool, ms,
+  ok/queued/gated/error. Never log bodies or keys.
+
+### 7.2 v1.1 container dividends (backlog, in priority order)
+
+- **Nightly vault backup** — the vault currently has **no disaster-recovery
+  story**; if the Obsidian host dies, memory is gone. A timer job walks the
+  vault via `read_many`, writes a dated zip to `/data/backups/` (rotate 30),
+  and reports the last-backup age in `/health`. Zero tokens; arguably the
+  highest-value item in this spec.
+- **Shadow write history + undo** — before any PUT / PATCH-replace the server
+  passes through, snapshot the prior body to `/data/history/` (content-
+  addressed, per-path ring of ~20 versions). Backs the `echo_note_history` /
+  `echo_restore_note` tools (§6.3). The vault's additive philosophy finally
+  gets an undo for its residual risk: a bad `replace` or merge PUT.
+- **Background maintenance jobs** — a timer thread running `sweep --fast`
+  (IMPROVEMENT-PLANS #4) hourly and the decay pass (#5) proposals weekly; both
+  publish findings to `/health` and the vault-health note rather than
+  auto-fixing.
+- **Ops alerting** — outbox stuck > N hours, vault unreachable > 1h, new lint
+  violations after a background sweep ⇒ notify via the existing Unraid
+  notification plumbing (Kuma already watches `/health` for liveness).
+  Problems surface without anyone polling.
+- **Embeddings (explicitly deferred)**: the container is where an embedding
+  recall tier would land later (model API or local), as an image upgrade —
+  no plugin change. Noted so nobody bolts it into the plugin.
+
+## 8. Container & deployment
+
+- **Image**: `python:3.12-slim` (not alpine — no musl surprises), `pip install
+  mcp` pinned, copy `skills/echo-memory/scripts/` + `mcp-server/` in.
+  **Dockerfile must be legacy-format** — the git.alwisp.com CI runner has no
+  BuildKit: no `# syntax=` line, no `RUN --mount`.
+- **Healthcheck**: probe `http://127.0.0.1:<port>/health` — **127.0.0.1, not
+  localhost** (the ::1-vs-IPv4 lesson from cpas/memer/breedr). `/health`
+  (unauthenticated, no vault data): `{ok, vault_reachable, outbox_depth,
+  index_age_s}` — also the Kuma target.
+- **Env** (via the PORT template, secrets as `SECRET:` refs): `ECHO_BASE`,
+  `ECHO_KEY`, `ECHO_OWNER`, `ECHO_MCP_TOKEN`, `ECHO_STATE_DIR=/data`,
+  `ECHO_MCP_TOOLS=full` (or `core` — see §3 tool profiles), optional
+  `ECHO_WORKERS/ECHO_TIMEOUT/ECHO_MCP_INDEX_TTL`. Volume: `/data`
+  (outbox + recall index + backups + shadow history).
+- **Startup validation**: fail fast (non-zero exit) if `ECHO_BASE`/`ECHO_KEY`/
+  `ECHO_MCP_TOKEN` are missing — a misdeployed container should crash-loop
+  visibly, not serve errors.
+- **CI/deploy**: Gitea workflow on push-to-main (tags/releases do **not**
+  autobuild on this host) builds the image and the PORT autodeploy webhook
+  rolls it out; `deploy.unraid.yml` manifest in `mcp-server/`. Publish
+  `echomcp.alwisp.com` via the usual CF + NPM flow; add the Kuma monitor.
+- **Vault adjacency (confirmed: Obsidian runs on ALPHA)**: the Local REST
+  API's non-encrypted HTTP binding is enabled (2026-07-28) —
+  **`ECHO_BASE=http://10.2.0.35:27123`** — so the container talks to Obsidian
+  directly, never through the public `echoapi.alwisp.com` hairpin. All vault
+  chatter behind a tool call stays host/LAN-local; vault ops stop depending on
+  Cloudflare/DNS/NPM health entirely (the public chain is only needed for the
+  `echomcp` ingress). Why HTTP not HTTPS here: the REST API's HTTPS port
+  (27124) uses a self-signed cert, which `echo.py`'s default-verifying
+  `HTTPSConnection` rejects — cleartext on the internal network beats adding an
+  insecure-TLS flag to the client. **Exposure check (do once):** 27123 carries
+  the vault bearer token in cleartext, so it must not be reachable from
+  outside — confirm no router/firewall forward for 27123; the only public
+  doors stay HTTPS 443 via NPM (`echoapi` for the CLI fallback path, which
+  remains unchanged, and `echomcp` for this server).
+
+## 9. Client registration & fallback
+
+- **Claude Code / CoWork**: register as a remote MCP server —
+  `https://echomcp.alwisp.com/mcp` with the `Authorization: Bearer` header
+  (project or user scope). No plugin-manifest coupling; the plugin does **not**
+  register the server.
+- **claude.ai**: custom connector with the same URL + token — ECHO memory from
+  the browser/phone, a surface the plugin could never reach.
+- **SKILL.md** gains: *"When the `echo_*` MCP tools are available, prefer them
+  for every operation they cover; the `$ECHO` CLI recipes are the fallback for
+  hosts without the connector or when the server is unreachable."* Procedures
+  unchanged.
+- **Fallback matrix**: server up ⇒ tools. Server down, machine has the plugin ⇒
+  CLI path exactly as today (including its own offline queue). Both down ⇒
+  today's "vault unreachable, proceed without memory".
+
+## 10. Multi-user note (Andy / Bryan / Gretchen / Arsalan)
+
+One container serves **one vault**. The image is credential-free, so additional
+users are additional deployments of the same image with their own env (their
+vault endpoint/key + their own `ECHO_MCP_TOKEN`), e.g. `echomcp-bryan` on
+another port/subdomain. Cheap, isolated, no code change. A single multi-tenant
+server (token → vault map) is deliberately out of scope — that's CHORUS's
+territory.
+
+## 11. Testing & eval
+
+1. **Protocol/integration tests** (`eval/test_mcp_server.py`): run the FastMCP
+   app in-process (or via `httpx` test client) against `mock_olrapi`; assert
+   initialize/tools-list schemas, auth (no token ⇒ 401; bad token ⇒ 401;
+   `/health` open), envelope purity on every tool.
+2. **Tool behavior**: per tool — happy path, validation failure (unknown param
+   rejected with field name), vault-down (writes ⇒ `queued: true`, reads ⇒
+   actionable error), duplicate-gate flow (gate → `merge_into` retry succeeds).
+3. **Parity**: `capture` via MCP and via CLI against twin mock vaults produce
+   byte-identical notes/index entries (guards Phase 0).
+4. **Container smoke** (CI): build image, start with mock env, `/health` green,
+   one authed `tools/call` round-trip; healthcheck probes 127.0.0.1.
+5. **MCP Inspector** manual pass against the deployed container.
+6. **Eval set** (mcp-builder Phase 4): 10 read-only Q&A pairs against a seeded
+   mock vault exercising recall→get_note→resolve chains; `eval/mcp_eval.xml`.
+
+## 12. Build-session plan of record
+
+| Phase | Deliverable | Est. size |
+|---|---|---|
+| 0 | Return-not-print refactor in the plugin tree (`*_op` cores); suites green | ~⅓ the session |
+| 1 | `mcp-server/`: FastMCP app, auth middleware, result wrapper, error map, `/health` | ~200 lines |
+| 2 | Read tools (load, recall, resolve, get_note, scope×2, health) | wiring |
+| 3 | Write tools (capture, link, append/patch, triage, reflect, log_session) | wiring + descriptions |
+| 4 | Warm caches + `/data` outbox + startup validation | small |
+| 5 | Dockerfile (legacy syntax) + CI workflow + PORT manifest + deploy + CF/NPM publish + Kuma | infra pass |
+| 6 | Tests (§11) + Inspector pass + eval XML; SKILL.md/README/CHANGELOG | ~⅓ the session |
+
+Definition of done: Inspector lists 13 tools against `echomcp.alwisp.com`; a
+live session (desktop **and** CoWork) performs a full memory day — load →
+recall → capture → gate → merge_into → triage → log_session — without one Bash
+call; parity test green; container smoke test in CI; Kuma monitor green.