echo/docs/MCP-SERVER-SPEC.md

echo-mcp — MCP Server Build Spec (containerized)

Status: BUILT — shipped as 2.2.0, 2026-07-28 (mcp-server/app.py, 14 tools — the count below saying 13 undercounted the append/patch pair; e2e suite eval/test_mcp_server.py). This document remains the design record; §7.2 is the live v1.1 backlog. Companion plan: docs/IMPROVEMENT-PLANS.md.

Prerequisites before starting this build:

1. The session-end verb (IMPROVEMENT-PLANS #7) — the MCP tool wraps it.
2. Phase 0 below (the return-not-print refactor) — lands in the plugin tree first and is independently shippable.

---

1. Why an MCP server

Every ECHO operation today rides through Bash: resolve $ECHO (with the CoWork path-fallback snippet copy-pasted everywhere), quote a Python invocation, run it, re-parse stdout. Costs:

• Tokens — each memory op carries bash scaffolding + output re-parsing; a capture is ~3× the tokens of a typed tool call.
• Fragility — quoting hazards, the ${CLAUDE_PLUGIN_ROOT} sandbox mismatch (the whole 1.4.2 release), Windows python3-vs-python dispatch.
• Reach — surfaces without a Python-capable shell can't use ECHO at all.
• Efficiency — every CLI invocation is a cold Python start that re-reads the entity index from the vault over HTTPS.

What the server does not replace: SKILL.md remains the procedure authority (when to load, reconcile, search-first, third-person, etc.). The server replaces the mechanics — tools are the verbs, the skill is the discipline.

2. Architecture: remote container (decided)

A standalone Docker container on the Unraid box (ALPHA), published behind Cloudflare + NPM as echomcp.alwisp.com, speaking MCP streamable HTTP (stateless JSON) with bearer-token auth. Chosen over the earlier local-stdio draft for four reasons:

1. The backend is already remote. Clients round-trip to echoapi.alwisp.com today; a server co-located with the vault host turns every vault op into a LAN hop and collapses N client→vault calls into one client→server call.
2. Any device, no per-machine anything. Desktop, CoWork sandbox, claude.ai (custom connector), mobile — same URL + token. The ${CLAUDE_PLUGIN_ROOT} CoWork registration risk from the stdio draft disappears entirely.
3. Credentials consolidate server-side. The vault key lives in the container env (PORT secret store); the image is credential-free; clients hold only an MCP bearer token. The per-user baked-key builds (1.4.x) become unnecessary for any surface that has MCP. Rotation = redeploy.
4. Room to grow. The plugin's pure-stdlib constraint exists because its scripts run in arbitrary sandboxes. The container is ours: real SDK, SQLite/in-memory indexes that stay warm, background jobs, embeddings later — all without touching the plugin.

What stays client-side (unchanged and shipped): the skill + CLI path — it is the fallback when the server or network is down, the offline queue + read cache keep their per-machine role there, and the SessionStart/Stop hooks stay CLI-based (hooks run shell commands regardless of MCP).

Cost acknowledged: a second service to run and an exposed auth surface. Mitigations are existing infra: CF + NPM cert, long random bearer token, Kuma monitor, PORT deploy/rollback. Server down ⇒ exactly today's behavior.

3. Decisions (made — don't relitigate in the build session)

Decision	Choice	Why
Language	Python	Imports the existing echo_*.py modules in-process — the entire ops layer is reused, not wrapped.
SDK	Official mcp Python SDK (FastMCP)	The stdlib-only rule doesn't bind inside our own image. Hand-rolling the protocol (stdio draft) is no longer justified.
Transport	Streamable HTTP, stateless JSON	Remote, multi-client, simplest to scale; SSE/sessions explicitly avoided.
Auth	Static bearer token (ECHO_MCP_TOKEN), constant-time compare, 401 otherwise	Single-operator service; OAuth is overkill. Token from the PORT secret store.
Vault credentials	Container env: ECHO_BASE, ECHO_KEY, ECHO_OWNER (SECRET: refs in the PORT template)	echo_config.py already resolves env-first — zero code change. Image ships no secrets.
Server name / repo home	echo-mcp; lives in this repo under mcp-server/ (own Dockerfile), deployed from git.alwisp.com/jason/echo CI	One canonical tree — the server vendors skills/echo-memory/scripts/ into the image at build time; no second hand-maintained copy (the Codex-drift lesson).
Tool prefix	echo_	Namespace safety next to other servers.
Result shape	structuredContent (the echo_output.envelope dict) + a 1–3 line human text block	Envelope shape already exists.
Statefulness	Stateless protocol; warm in-process caches	See §7.
Version target	2.2 — ALL prerequisites shipped 2026-07-28 (session-end in 2.1.0; Phase 0 in 2.1.1, contract-tested by eval/test_ops_api.py)	The build session starts directly at §5 (the server app).
Local stdio variant	Dropped for v1 (documented, not built)	The CLI/skill fallback covers the no-server case; two transports = two test matrices for little gain. Phase 0 keeps the door open — the *_op core is transport-agnostic.
Result budgets	Every read tool takes an explicit size budget (budget_chars / max_chars) and truncates with a marker + "call X for more"	The single biggest token lever: one right-sized answer instead of follow-up fetches.
Tool profiles	ECHO_MCP_TOOLS=core|full env: core exposes only load/recall/capture/triage_inbox/log_session/health	Tool schemas cost context on every surface that lists them; the claude.ai connector doesn't need the escape hatches. Desktop registers full.

4. Phase 0 — the return-not-print refactor (prerequisite, lands in the plugin)

Unchanged from the stdio draft, and still first: the ops layer prints results (even --json mode prints from inside echo_ops.capture) and returns exit codes; the server needs return values.

1. Each high-level op gets a core function returning an envelope dict, raising echo.EchoError on failure; CLI entry points become thin printing wrappers:

   Module	New core fn	CLI wrapper keeps
   echo_ops	capture_op, resolve_op, link_op	capture/resolve/link
   echo_recall	recall_op(query, limit)	recall
   echo_triage	list_op() · route_op(proposals, apply)	list_inbox/apply
   echo_reflect	apply_op(proposals, apply)	apply
   echo.py	load_op(brief) · scope_show_op/scope_set_op · doctor_op	cmd_*
   echo_session (new, #7)	session_end_op(bundle, apply)	subcommand

2. Special results are data, not exit codes: duplicate gate ⇒ {ok: false, action: "duplicate-gate", candidates} (CLI maps to exit 76); offline queueing ⇒ {ok: true, queued: true}.

3. Helper chatter (ok: PUT …) moves behind a notify(msg) callback defaulting to stderr; delete the redirect_stdout hack in capture.

4. Tests: existing suites pass unchanged; new unit tests hit *_op directly against the mock and assert envelopes.

Shippable on its own as a 1.6.x refactor with zero behavior change.

5. Server application (mcp-server/app.py)

• FastMCP app, streamable HTTP, stateless. One @mcp.tool per tool in §6, each a thin adapter: validate → call the *_op core → wrap.

• Auth middleware: require Authorization: Bearer <ECHO_MCP_TOKEN> on every request (constant-time compare); 401 with no detail otherwise. Additionally bind the app to the container interface only; exposure policy lives at NPM/Cloudflare.

• Result wrapper:

  def tool_result(env: dict, summary: str) -> ...:
      # content: [{type: "text", text: summary}]  (1–3 lines, human)
      # structuredContent: env                     (echo_output envelope)
      # isError: not env.get("ok", True) — except the deliberate non-errors below
  

• Error map (tool-level results, never protocol errors):

  Condition	Result
  Vault unreachable on a read	isError: true — "vault unreachable (Obsidian/REST plugin likely down); proceed without memory, writes will queue server-side".
  Vault unreachable on a write	Not an error: {ok: true, queued: true} — the server-side outbox (§7) replays when the vault returns.
  404	isError: true, {code: "not-found", path}.
  Duplicate gate	Not an error: {ok: false, action: "duplicate-gate", candidates} + text naming the two resolutions (merge_into / force) — the model must decide, not blind-retry.
  Lock held	{ok: false, action: "lock-held", holder}.
  Misconfigured deployment (no ECHO_BASE/ECHO_KEY)	isError: true — "server deployment is missing vault credentials — operator: check the PORT template env". Startup also fails loudly (see §8 healthcheck).
  Anything else	isError: true, first line only; traceback to the container log.

• Validation: FastMCP/pydantic handles types/enums/required; add model_config = ConfigDict(extra="forbid") so unknown params (model typos) fail with a field-naming message.

6. Tool surface (13 tools)

Same surface as the stdio draft minus echo_configure — with a remote server there is no per-machine config to import; credentials are deployment-side. Descriptions are written for the model: what it does, when to use it, what it returns, ≤3 sentences.

6.1 Orientation & read

• echo_load — cold-start orientation. brief: bool = true (IMPROVEMENT-PLANS #1 digest; false = full six-file dump). Returns {sections: {marker, preferences, scope, last_session, today, inbox_count, inbox_oldest_days}, queued_flushed, offline}. Also flushes the server-side outbox. readOnlyHint: false (flush), openWorldHint: true.
• echo_recall — query: str, limit: int = 6 (1–20), include_linked: bool = true, budget_chars: int = 4000 (500–20000). Returns the recall --json shape (primary / linked with path/score/type/updated/status/excerpt) packed to the budget: the server allocates excerpt space by score, so the model gets the most relevant content in one right-sized answer instead of follow-up fetches. Description: truncated hits are marked — call echo_get_note for full content. readOnlyHint: true, idempotentHint: true.
• echo_resolve — mention: str → match {slug, path, kind, title, aliases} or {match: false, suggest_slug, candidates}. "Call before creating any note by hand; echo_capture does this automatically." readOnlyHint: true.
• echo_get_note — path: str (vault-relative; reject ../leading /; warn-not-block on paths outside routing.json); section: str = null (a heading name — return only that section, e.g. Status); max_chars: int = 8000. Returns {path, content, frontmatter, truncated?}. readOnlyHint: true.
• echo_get_scope — {scope, scope_updated, sessions_since}; description tells the model to confirm scope with the operator when sessions_since ≥ 3. readOnlyHint: true.
• echo_set_scope — scope: str; atomic switch (history + replace + stamp). idempotentHint: true.
• echo_health — deep: bool = false. Shallow = doctor checks (vault reachability, auth, marker/schema, outbox depth). Deep = full vault_lint → {violations: [{check, path, detail}]}. readOnlyHint: true.

6.2 Write

• echo_capture — the default write (route + frontmatter + index + auto-link + agent-log in one call). Params: title (req); kind: enum[person, company, concept, reference, meeting, project, area, semantic, episodic, working, skill, decision] (omit ⇒ inbox line); body: str = "" (markdown, inline); tags/aliases/sources: str[]; status: str; date: YYYY-MM-DD (meeting/decision); domain: enum[business, personal, learning, systems] = business (area); merge_into: str (slug); force: bool = false; dry_run: bool = false. Returns the capture envelope (action: created|updated|inbox|duplicate-gate, path, links_added, near_duplicates?, candidates?). Description spells out the gate contract (never blind-retry; merge_into or confirmed force). destructiveHint: false, idempotentHint: true.
• echo_link — a, b (paths or resolvable names — server resolves via the index). Returns {a, b, a_changed, b_changed}. idempotentHint: true.
• echo_append_note — path, line; whole-line idempotent append.
• echo_patch_note — path, operation: enum[append, prepend, replace], target_type: enum[heading, frontmatter, block], target, content. Description carries the hard-won rules: heading targets are the full ::-delimited path from the H1; on a 400 invalid-target the server fetches the document map and returns the actual heading list in the error — the worst silent-loss failure becomes self-correcting. destructiveHint: true. (Deliberately no echo_put_note / echo_delete_note in v1 — whole-file overwrite and deletion stay behind the CLI + operator explicitness; session logs go through echo_log_session.)
• echo_triage_inbox — proposals: object[] = [] (PROPOSAL_SCHEMA + optional line), apply: bool = false. Empty ⇒ structured listing; with proposals ⇒ preview, then route + processing-log audit on apply: true. "List → propose → preview → apply only after the operator confirms."
• echo_reflect — proposals: object[] (req), apply: bool = false. Same preview/apply contract as the CLI; description restates: never apply without the operator's go-ahead, never invent memories.
• echo_log_session — the session-end bundle (wraps session_end_op, #7): slug, log_body, agent_log_line, scope?: str, reflect?: object[], apply: bool = false. Per-step results; heartbeat written last as the commit marker.

6.3 v1.1 tools (specced now, built after v1 ships)

• echo_rollup_data — period: enum[week, month], date: YYYY-MM-DD. Assembles the rollup digest data in one call: open threads across projects/active/ (each note's ## Status + updated:), inbox items aging past 7 days, and the period's ## Scope History entries. The model writes the prose; the server does the N fetches. Same division of labor as reflect. readOnlyHint: true.
• echo_note_history — path: str → {versions: [{id, ts, bytes, summary_line}]} from the server's shadow write history (§7). readOnlyHint: true.
• echo_restore_note — path: str, version_id: str, confirm: bool = false (preview diff unless confirmed). The undo for a bad replace PATCH or merge PUT. Description: operator confirmation required before calling with confirm: true. destructiveHint: true.

Deliberately absent from v1 (documented in the server README): sweep, bootstrap, migrate, lock/unlock (vault-wide maintenance stays operator-initiated via slash commands — routine sweeps run as background jobs instead), raw search/ls/map (recall/resolve/get_note cover reads; add only if transcripts show the need), and reflect extraction (only the model has the conversation — that division of labor is permanent, not a v1 scope cut).

7. Server internals — where the container earns its keep

7.1 v1

• Warm entity index: loaded once, invalidated on any index-writing tool and on a short TTL (ECHO_MCP_INDEX_TTL, default 60s) to pick up other clients' writes. Ends the per-invocation index re-read entirely.
• Recall index in memory (+ SQLite file at /data): the BM25 index lives in process memory, persisted to the container volume — recall answers in milliseconds with zero vault round-trips. This supersedes the server-side half of IMPROVEMENT-PLANS #2; #2's local-state-dir design still applies to the CLI fallback path.
• Note read cache: short-TTL (30–60s) body cache for get_note and recall's neighbourhood expansion — repeated same-session reads of the same files stop hitting the vault at all.
• Per-path write serialization: an internal per-path mutex serializes conflicting MCP-path writes — since all server-mediated writes flow through one process, the advisory-lock race window disappears for them. The server still takes the vault advisory lock around index updates to coordinate with CLI clients; idempotent appends stay as the last line of defense.
• Server-side outbox: the existing echo_queue pointed at /data (ECHO_STATE_DIR=/data) — writes queue when the vault is down and flush on recovery/load. One queue at the always-on host instead of per-laptop.
• Concurrency: FastMCP serves requests concurrently; guard the in-process caches with a plain threading.Lock; write integrity per the bullet above.
• Logging: one line per call to stdout (container log): tool, ms, ok/queued/gated/error. Never log bodies or keys.

7.2 v1.1 container dividends (backlog, in priority order)

• Nightly vault backup — the vault currently has no disaster-recovery story; if the Obsidian host dies, memory is gone. A timer job walks the vault via read_many, writes a dated zip to /data/backups/ (rotate 30), and reports the last-backup age in /health. Zero tokens; arguably the highest-value item in this spec.
• Shadow write history + undo — before any PUT / PATCH-replace the server passes through, snapshot the prior body to /data/history/ (content- addressed, per-path ring of ~20 versions). Backs the echo_note_history / echo_restore_note tools (§6.3). The vault's additive philosophy finally gets an undo for its residual risk: a bad replace or merge PUT.
• Background maintenance jobs — a timer thread running sweep --fast (IMPROVEMENT-PLANS #4) hourly and the decay pass (#5) proposals weekly; both publish findings to /health and the vault-health note rather than auto-fixing.
• Ops alerting — outbox stuck > N hours, vault unreachable > 1h, new lint violations after a background sweep ⇒ notify via the existing Unraid notification plumbing (Kuma already watches /health for liveness). Problems surface without anyone polling.
• Embeddings (explicitly deferred): the container is where an embedding recall tier would land later (model API or local), as an image upgrade — no plugin change. Noted so nobody bolts it into the plugin.

8. Container & deployment

• Image: python:3.12-slim (not alpine — no musl surprises), pip install mcp pinned, copy skills/echo-memory/scripts/ + mcp-server/ in. Dockerfile must be legacy-format — the git.alwisp.com CI runner has no BuildKit: no # syntax= line, no RUN --mount.
• Healthcheck: probe http://127.0.0.1:<port>/health — 127.0.0.1, not localhost (the ::1-vs-IPv4 lesson from cpas/memer/breedr). /health (unauthenticated, no vault data): {ok, vault_reachable, outbox_depth, index_age_s} — also the Kuma target.
• Env (via the PORT template, secrets as SECRET: refs): ECHO_BASE, ECHO_KEY, ECHO_OWNER, ECHO_MCP_TOKEN, ECHO_STATE_DIR=/data, ECHO_MCP_TOOLS=full (or core — see §3 tool profiles), optional ECHO_WORKERS/ECHO_TIMEOUT/ECHO_MCP_INDEX_TTL. Volume: /data (outbox + recall index + backups + shadow history).
• Startup validation: fail fast (non-zero exit) if ECHO_BASE/ECHO_KEY/ ECHO_MCP_TOKEN are missing — a misdeployed container should crash-loop visibly, not serve errors.
• CI/deploy: Gitea workflow on push-to-main (tags/releases do not autobuild on this host) builds the image and the PORT autodeploy webhook rolls it out; deploy.unraid.yml manifest in mcp-server/. Publish echomcp.alwisp.com via the usual CF + NPM flow; add the Kuma monitor.
• Vault adjacency (confirmed: Obsidian runs on ALPHA): the Local REST API's non-encrypted HTTP binding is enabled (2026-07-28) — ECHO_BASE=http://10.2.0.35:27123 — so the container talks to Obsidian directly, never through the public echoapi.alwisp.com hairpin. All vault chatter behind a tool call stays host/LAN-local; vault ops stop depending on Cloudflare/DNS/NPM health entirely (the public chain is only needed for the echomcp ingress). Why HTTP not HTTPS here: the REST API's HTTPS port (27124) uses a self-signed cert, which echo.py's default-verifying HTTPSConnection rejects — cleartext on the internal network beats adding an insecure-TLS flag to the client. Exposure check (do once): 27123 carries the vault bearer token in cleartext, so it must not be reachable from outside — confirm no router/firewall forward for 27123; the only public doors stay HTTPS 443 via NPM (echoapi for the CLI fallback path, which remains unchanged, and echomcp for this server).

9. Client registration & fallback

• Claude Code / CoWork: register as a remote MCP server — https://echomcp.alwisp.com/mcp with the Authorization: Bearer header (project or user scope). No plugin-manifest coupling; the plugin does not register the server.
• claude.ai: custom connector with the same URL + token — ECHO memory from the browser/phone, a surface the plugin could never reach.
• SKILL.md gains: "When the echo_* MCP tools are available, prefer them for every operation they cover; the $ECHO CLI recipes are the fallback for hosts without the connector or when the server is unreachable." Procedures unchanged.
• Fallback matrix: server up ⇒ tools. Server down, machine has the plugin ⇒ CLI path exactly as today (including its own offline queue). Both down ⇒ today's "vault unreachable, proceed without memory".

10. Multi-user note (Andy / Bryan / Gretchen / Arsalan)

One container serves one vault. The image is credential-free, so additional users are additional deployments of the same image with their own env (their vault endpoint/key + their own ECHO_MCP_TOKEN), e.g. echomcp-bryan on another port/subdomain. Cheap, isolated, no code change. A single multi-tenant server (token → vault map) is deliberately out of scope — that's CHORUS's territory.

11. Testing & eval

1. Protocol/integration tests (eval/test_mcp_server.py): run the FastMCP app in-process (or via httpx test client) against mock_olrapi; assert initialize/tools-list schemas, auth (no token ⇒ 401; bad token ⇒ 401; /health open), envelope purity on every tool.
2. Tool behavior: per tool — happy path, validation failure (unknown param rejected with field name), vault-down (writes ⇒ queued: true, reads ⇒ actionable error), duplicate-gate flow (gate → merge_into retry succeeds).
3. Parity: capture via MCP and via CLI against twin mock vaults produce byte-identical notes/index entries (guards Phase 0).
4. Container smoke (CI): build image, start with mock env, /health green, one authed tools/call round-trip; healthcheck probes 127.0.0.1.
5. MCP Inspector manual pass against the deployed container.
6. Eval set (mcp-builder Phase 4): 10 read-only Q&A pairs against a seeded mock vault exercising recall→get_note→resolve chains; eval/mcp_eval.xml.

12. Build-session plan of record

Phase	Deliverable	Est. size
0	Return-not-print refactor in the plugin tree (*_op cores); suites green	~⅓ the session
1	mcp-server/: FastMCP app, auth middleware, result wrapper, error map, /health	~200 lines
2	Read tools (load, recall, resolve, get_note, scope×2, health)	wiring
3	Write tools (capture, link, append/patch, triage, reflect, log_session)	wiring + descriptions
4	Warm caches + /data outbox + startup validation	small
5	Dockerfile (legacy syntax) + CI workflow + PORT manifest + deploy + CF/NPM publish + Kuma	infra pass
6	Tests (§11) + Inspector pass + eval XML; SKILL.md/README/CHANGELOG	~⅓ the session

Definition of done: Inspector lists 13 tools against echomcp.alwisp.com; a live session (desktop and CoWork) performs a full memory day — load → recall → capture → gate → merge_into → triage → log_session — without one Bash call; parity test green; container smoke test in CI; Kuma monitor green.