echo/docs/MCP-SERVER-SPEC.md

# echo-mcp — MCP Server Build Spec (containerized)

> Status: **BUILT — shipped as 2.2.0, 2026-07-28** (`mcp-server/app.py`, 14 tools —
> the count below saying 13 undercounted the append/patch pair; e2e suite
> `eval/test_mcp_server.py`). This document remains the design record; §7.2 is the
> live v1.1 backlog. Companion plan: `docs/IMPROVEMENT-PLANS.md`.
>
> **Prerequisites before starting this build:**
> 1. The `session-end` verb (IMPROVEMENT-PLANS #7) — the MCP tool wraps it.
> 2. Phase 0 below (the return-not-print refactor) — lands in the plugin tree
>    first and is independently shippable.

---

## 1. Why an MCP server

Every ECHO operation today rides through Bash: resolve `$ECHO` (with the CoWork
path-fallback snippet copy-pasted everywhere), quote a Python invocation, run
it, re-parse stdout. Costs:

- **Tokens** — each memory op carries bash scaffolding + output re-parsing; a
  capture is ~3× the tokens of a typed tool call.
- **Fragility** — quoting hazards, the `${CLAUDE_PLUGIN_ROOT}` sandbox mismatch
  (the whole 1.4.2 release), Windows `python3`-vs-`python` dispatch.
- **Reach** — surfaces without a Python-capable shell can't use ECHO at all.
- **Efficiency** — every CLI invocation is a cold Python start that re-reads the
  entity index from the vault over HTTPS.

What the server does **not** replace: SKILL.md remains the procedure authority
(when to load, reconcile, search-first, third-person, etc.). The server replaces
the *mechanics* — tools are the verbs, the skill is the discipline.

## 2. Architecture: remote container (decided)

A standalone Docker container on the Unraid box (ALPHA), published behind
Cloudflare + NPM as **`echomcp.alwisp.com`**, speaking **MCP streamable HTTP
(stateless JSON)** with bearer-token auth. Chosen over the earlier local-stdio
draft for four reasons:

1. **The backend is already remote.** Clients round-trip to
   `echoapi.alwisp.com` today; a server co-located with the vault host turns
   every vault op into a LAN hop and collapses N client→vault calls into one
   client→server call.
2. **Any device, no per-machine anything.** Desktop, CoWork sandbox, claude.ai
   (custom connector), mobile — same URL + token. The `${CLAUDE_PLUGIN_ROOT}`
   CoWork registration risk from the stdio draft disappears entirely.
3. **Credentials consolidate server-side.** The vault key lives in the
   container env (PORT secret store); the image is credential-free; clients
   hold only an MCP bearer token. The per-user baked-key builds (1.4.x) become
   unnecessary for any surface that has MCP. Rotation = redeploy.
4. **Room to grow.** The plugin's pure-stdlib constraint exists because its
   scripts run in arbitrary sandboxes. The container is ours: real SDK,
   SQLite/in-memory indexes that stay warm, background jobs, embeddings later —
   all without touching the plugin.

**What stays client-side (unchanged and shipped):** the skill + CLI path — it
is the fallback when the server or network is down, the offline queue + read
cache keep their per-machine role there, and the SessionStart/Stop **hooks stay
CLI-based** (hooks run shell commands regardless of MCP).

**Cost acknowledged:** a second service to run and an exposed auth surface.
Mitigations are existing infra: CF + NPM cert, long random bearer token, Kuma
monitor, PORT deploy/rollback. Server down ⇒ exactly today's behavior.

## 3. Decisions (made — don't relitigate in the build session)

| Decision | Choice | Why |
|---|---|---|
| Language | **Python** | Imports the existing `echo_*.py` modules in-process — the entire ops layer is reused, not wrapped. |
| SDK | **Official `mcp` Python SDK (FastMCP)** | The stdlib-only rule doesn't bind inside our own image. Hand-rolling the protocol (stdio draft) is no longer justified. |
| Transport | **Streamable HTTP, stateless JSON** | Remote, multi-client, simplest to scale; SSE/sessions explicitly avoided. |
| Auth | **Static bearer token** (`ECHO_MCP_TOKEN`), constant-time compare, 401 otherwise | Single-operator service; OAuth is overkill. Token from the PORT secret store. |
| Vault credentials | Container env: `ECHO_BASE`, `ECHO_KEY`, `ECHO_OWNER` (SECRET: refs in the PORT template) | `echo_config.py` already resolves env-first — zero code change. Image ships no secrets. |
| Server name / repo home | `echo-mcp`; lives in this repo under `mcp-server/` (own Dockerfile), deployed from `git.alwisp.com/jason/echo` CI | One canonical tree — the server vendors `skills/echo-memory/scripts/` into the image at build time; no second hand-maintained copy (the Codex-drift lesson). |
| Tool prefix | `echo_` | Namespace safety next to other servers. |
| Result shape | `structuredContent` (the `echo_output.envelope` dict) + a 1–3 line human text block | Envelope shape already exists. |
| Statefulness | Stateless protocol; warm in-process caches | See §7. |
| Version target | **2.2** — ALL prerequisites shipped 2026-07-28 (`session-end` in 2.1.0; **Phase 0 in 2.1.1**, contract-tested by `eval/test_ops_api.py`) | The build session starts directly at §5 (the server app). |
| Local stdio variant | **Dropped for v1** (documented, not built) | The CLI/skill fallback covers the no-server case; two transports = two test matrices for little gain. Phase 0 keeps the door open — the `*_op` core is transport-agnostic. |
| Result budgets | Every read tool takes an explicit size budget (`budget_chars` / `max_chars`) and truncates with a marker + "call X for more" | The single biggest token lever: one right-sized answer instead of follow-up fetches. |
| Tool profiles | `ECHO_MCP_TOOLS=core\|full` env: `core` exposes only load/recall/capture/triage_inbox/log_session/health | Tool schemas cost context on every surface that lists them; the claude.ai connector doesn't need the escape hatches. Desktop registers `full`. |

## 4. Phase 0 — the return-not-print refactor (prerequisite, lands in the plugin)

Unchanged from the stdio draft, and still first: the ops layer prints results
(even `--json` mode prints from inside `echo_ops.capture`) and returns exit
codes; the server needs return values.

1. Each high-level op gets a **core function returning an envelope dict**,
   raising `echo.EchoError` on failure; CLI entry points become thin printing
   wrappers:

   | Module | New core fn | CLI wrapper keeps |
   |---|---|---|
   | `echo_ops` | `capture_op`, `resolve_op`, `link_op` | `capture/resolve/link` |
   | `echo_recall` | `recall_op(query, limit)` | `recall` |
   | `echo_triage` | `list_op()` · `route_op(proposals, apply)` | `list_inbox/apply` |
   | `echo_reflect` | `apply_op(proposals, apply)` | `apply` |
   | `echo.py` | `load_op(brief)` · `scope_show_op/scope_set_op` · `doctor_op` | `cmd_*` |
   | `echo_session` (new, #7) | `session_end_op(bundle, apply)` | subcommand |

2. Special results are **data, not exit codes**: duplicate gate ⇒
   `{ok: false, action: "duplicate-gate", candidates}` (CLI maps to exit 76);
   offline queueing ⇒ `{ok: true, queued: true}`.
3. Helper chatter (`ok: PUT …`) moves behind a `notify(msg)` callback
   defaulting to stderr; delete the `redirect_stdout` hack in `capture`.
4. Tests: existing suites pass unchanged; new unit tests hit `*_op` directly
   against the mock and assert envelopes.

Shippable on its own as a 1.6.x refactor with zero behavior change.

## 5. Server application (`mcp-server/app.py`)

- **FastMCP** app, streamable HTTP, stateless. One `@mcp.tool` per tool in §6,
  each a thin adapter: validate → call the `*_op` core → wrap.
- **Auth middleware**: require `Authorization: Bearer <ECHO_MCP_TOKEN>` on every
  request (constant-time compare); 401 with no detail otherwise. Additionally
  bind the app to the container interface only; exposure policy lives at
  NPM/Cloudflare.
- **Result wrapper**:

  ```python
  def tool_result(env: dict, summary: str) -> ...:
      # content: [{type: "text", text: summary}]  (1–3 lines, human)
      # structuredContent: env                     (echo_output envelope)
      # isError: not env.get("ok", True) — except the deliberate non-errors below
  ```

- **Error map** (tool-level results, never protocol errors):

  | Condition | Result |
  |---|---|
  | Vault unreachable on a **read** | `isError: true` — "vault unreachable (Obsidian/REST plugin likely down); proceed without memory, writes will queue server-side". |
  | Vault unreachable on a **write** | **Not an error**: `{ok: true, queued: true}` — the server-side outbox (§7) replays when the vault returns. |
  | 404 | `isError: true`, `{code: "not-found", path}`. |
  | Duplicate gate | **Not an error**: `{ok: false, action: "duplicate-gate", candidates}` + text naming the two resolutions (`merge_into` / `force`) — the model must decide, not blind-retry. |
  | Lock held | `{ok: false, action: "lock-held", holder}`. |
  | Misconfigured deployment (no `ECHO_BASE`/`ECHO_KEY`) | `isError: true` — "server deployment is missing vault credentials — operator: check the PORT template env". Startup also fails loudly (see §8 healthcheck). |
  | Anything else | `isError: true`, first line only; traceback to the container log. |

- **Validation**: FastMCP/pydantic handles types/enums/required; add
  `model_config = ConfigDict(extra="forbid")` so unknown params (model typos)
  fail with a field-naming message.

## 6. Tool surface (13 tools)

Same surface as the stdio draft **minus `echo_configure`** — with a remote
server there is no per-machine config to import; credentials are
deployment-side. Descriptions are written for the model: what it does, when to
use it, what it returns, ≤3 sentences.

### 6.1 Orientation & read

- **`echo_load`** — cold-start orientation. `brief: bool = true`
  (IMPROVEMENT-PLANS #1 digest; `false` = full six-file dump). Returns
  `{sections: {marker, preferences, scope, last_session, today, inbox_count,
  inbox_oldest_days}, queued_flushed, offline}`. Also flushes the server-side
  outbox. `readOnlyHint: false` (flush), `openWorldHint: true`.
- **`echo_recall`** — `query: str`, `limit: int = 6 (1–20)`,
  `include_linked: bool = true`, `budget_chars: int = 4000 (500–20000)`. Returns
  the `recall --json` shape (`primary` / `linked` with
  path/score/type/updated/status/excerpt) **packed to the budget**: the server
  allocates excerpt space by score, so the model gets the most relevant content
  in one right-sized answer instead of follow-up fetches. Description: truncated
  hits are marked — call `echo_get_note` for full content.
  `readOnlyHint: true`, `idempotentHint: true`.
- **`echo_resolve`** — `mention: str` → match `{slug, path, kind, title,
  aliases}` or `{match: false, suggest_slug, candidates}`. "Call before creating
  any note by hand; `echo_capture` does this automatically." `readOnlyHint: true`.
- **`echo_get_note`** — `path: str` (vault-relative; reject `..`/leading `/`;
  warn-not-block on paths outside `routing.json`); `section: str = null`
  (a heading name — return only that section, e.g. `Status`);
  `max_chars: int = 8000`. Returns `{path, content, frontmatter, truncated?}`.
  `readOnlyHint: true`.
- **`echo_get_scope`** — `{scope, scope_updated, sessions_since}`; description
  tells the model to confirm scope with the operator when `sessions_since ≥ 3`.
  `readOnlyHint: true`.
- **`echo_set_scope`** — `scope: str`; atomic switch (history + replace +
  stamp). `idempotentHint: true`.
- **`echo_health`** — `deep: bool = false`. Shallow = doctor checks (vault
  reachability, auth, marker/schema, outbox depth). Deep = full `vault_lint`
  → `{violations: [{check, path, detail}]}`. `readOnlyHint: true`.

### 6.2 Write

- **`echo_capture`** — the default write (route + frontmatter + index +
  auto-link + agent-log in one call). Params: `title` (req); `kind:
  enum[person, company, concept, reference, meeting, project, area, semantic,
  episodic, working, skill, decision]` (omit ⇒ inbox line); `body: str = ""`
  (markdown, inline); `tags/aliases/sources: str[]`; `status: str`;
  `date: YYYY-MM-DD` (meeting/decision); `domain: enum[business, personal,
  learning, systems] = business` (area); `merge_into: str` (slug);
  `force: bool = false`; `dry_run: bool = false`. Returns the capture envelope
  (`action: created|updated|inbox|duplicate-gate`, `path`, `links_added`,
  `near_duplicates?`, `candidates?`). Description spells out the gate contract
  (never blind-retry; `merge_into` or confirmed `force`). `destructiveHint:
  false`, `idempotentHint: true`.
- **`echo_link`** — `a`, `b` (paths **or resolvable names** — server resolves
  via the index). Returns `{a, b, a_changed, b_changed}`. `idempotentHint: true`.
- **`echo_append_note`** — `path`, `line`; whole-line idempotent append.
- **`echo_patch_note`** — `path`, `operation: enum[append, prepend, replace]`,
  `target_type: enum[heading, frontmatter, block]`, `target`, `content`.
  Description carries the hard-won rules: heading targets are the full
  `::`-delimited path from the H1; on a 400 invalid-target the server fetches
  the document map and returns the actual heading list in the error — the worst
  silent-loss failure becomes self-correcting. `destructiveHint: true`.
  (Deliberately **no `echo_put_note` / `echo_delete_note`** in v1 — whole-file
  overwrite and deletion stay behind the CLI + operator explicitness; session
  logs go through `echo_log_session`.)
- **`echo_triage_inbox`** — `proposals: object[] = []` (PROPOSAL_SCHEMA +
  optional `line`), `apply: bool = false`. Empty ⇒ structured listing; with
  proposals ⇒ preview, then route + processing-log audit on `apply: true`.
  "List → propose → preview → apply only after the operator confirms."
- **`echo_reflect`** — `proposals: object[]` (req), `apply: bool = false`. Same
  preview/apply contract as the CLI; description restates: never apply without
  the operator's go-ahead, never invent memories.
- **`echo_log_session`** — the session-end bundle (wraps `session_end_op`, #7):
  `slug`, `log_body`, `agent_log_line`, `scope?: str`, `reflect?: object[]`,
  `apply: bool = false`. Per-step results; heartbeat written last as the commit
  marker.

### 6.3 v1.1 tools (specced now, built after v1 ships)

- **`echo_rollup_data`** — `period: enum[week, month]`, `date: YYYY-MM-DD`.
  Assembles the rollup *digest data* in one call: open threads across
  `projects/active/` (each note's `## Status` + `updated:`), inbox items aging
  past 7 days, and the period's `## Scope History` entries. The model writes
  the prose; the server does the N fetches. Same division of labor as reflect.
  `readOnlyHint: true`.
- **`echo_note_history`** — `path: str` → `{versions: [{id, ts, bytes,
  summary_line}]}` from the server's shadow write history (§7). `readOnlyHint:
  true`.
- **`echo_restore_note`** — `path: str`, `version_id: str`,
  `confirm: bool = false` (preview diff unless confirmed). The undo for a bad
  `replace` PATCH or merge PUT. Description: operator confirmation required
  before calling with `confirm: true`. `destructiveHint: true`.

Deliberately absent from v1 (documented in the server README): sweep,
bootstrap, migrate, lock/unlock (vault-wide maintenance stays operator-initiated
via slash commands — routine sweeps run as background jobs instead), raw
search/ls/map (recall/resolve/get_note cover reads; add only if transcripts
show the need), and reflect *extraction* (only the model has the conversation —
that division of labor is permanent, not a v1 scope cut).

## 7. Server internals — where the container earns its keep

### 7.1 v1

- **Warm entity index**: loaded once, invalidated on any index-writing tool and
  on a short TTL (`ECHO_MCP_INDEX_TTL`, default 60s) to pick up other clients'
  writes. Ends the per-invocation index re-read entirely.
- **Recall index in memory (+ SQLite file at `/data`)**: the BM25 index lives
  in process memory, persisted to the container volume — recall answers in
  milliseconds with **zero** vault round-trips. This *supersedes the
  server-side half of IMPROVEMENT-PLANS #2*; #2's local-state-dir design still
  applies to the CLI fallback path.
- **Note read cache**: short-TTL (30–60s) body cache for `get_note` and
  recall's neighbourhood expansion — repeated same-session reads of the same
  files stop hitting the vault at all.
- **Per-path write serialization**: an internal per-path mutex serializes
  conflicting MCP-path writes — since all server-mediated writes flow through
  one process, the advisory-lock race window disappears for them. The server
  still takes the vault advisory lock around index updates to coordinate with
  CLI clients; idempotent appends stay as the last line of defense.
- **Server-side outbox**: the existing `echo_queue` pointed at `/data`
  (`ECHO_STATE_DIR=/data`) — writes queue when the vault is down and flush on
  recovery/load. One queue at the always-on host instead of per-laptop.
- **Concurrency**: FastMCP serves requests concurrently; guard the in-process
  caches with a plain `threading.Lock`; write integrity per the bullet above.
- **Logging**: one line per call to stdout (container log): tool, ms,
  ok/queued/gated/error. Never log bodies or keys.

### 7.2 v1.1 container dividends (backlog, in priority order)

- **Nightly vault backup** — the vault currently has **no disaster-recovery
  story**; if the Obsidian host dies, memory is gone. A timer job walks the
  vault via `read_many`, writes a dated zip to `/data/backups/` (rotate 30),
  and reports the last-backup age in `/health`. Zero tokens; arguably the
  highest-value item in this spec.
- **Shadow write history + undo** — before any PUT / PATCH-replace the server
  passes through, snapshot the prior body to `/data/history/` (content-
  addressed, per-path ring of ~20 versions). Backs the `echo_note_history` /
  `echo_restore_note` tools (§6.3). The vault's additive philosophy finally
  gets an undo for its residual risk: a bad `replace` or merge PUT.
- **Background maintenance jobs** — a timer thread running `sweep --fast`
  (IMPROVEMENT-PLANS #4) hourly and the decay pass (#5) proposals weekly; both
  publish findings to `/health` and the vault-health note rather than
  auto-fixing.
- **Ops alerting** — outbox stuck > N hours, vault unreachable > 1h, new lint
  violations after a background sweep ⇒ notify via the existing Unraid
  notification plumbing (Kuma already watches `/health` for liveness).
  Problems surface without anyone polling.
- **Embeddings (explicitly deferred)**: the container is where an embedding
  recall tier would land later (model API or local), as an image upgrade —
  no plugin change. Noted so nobody bolts it into the plugin.

## 8. Container & deployment

- **Image**: `python:3.12-slim` (not alpine — no musl surprises), `pip install
  mcp` pinned, copy `skills/echo-memory/scripts/` + `mcp-server/` in.
  **Dockerfile must be legacy-format** — the git.alwisp.com CI runner has no
  BuildKit: no `# syntax=` line, no `RUN --mount`.
- **Healthcheck**: probe `http://127.0.0.1:<port>/health` — **127.0.0.1, not
  localhost** (the ::1-vs-IPv4 lesson from cpas/memer/breedr). `/health`
  (unauthenticated, no vault data): `{ok, vault_reachable, outbox_depth,
  index_age_s}` — also the Kuma target.
- **Env** (via the PORT template, secrets as `SECRET:` refs): `ECHO_BASE`,
  `ECHO_KEY`, `ECHO_OWNER`, `ECHO_MCP_TOKEN`, `ECHO_STATE_DIR=/data`,
  `ECHO_MCP_TOOLS=full` (or `core` — see §3 tool profiles), optional
  `ECHO_WORKERS/ECHO_TIMEOUT/ECHO_MCP_INDEX_TTL`. Volume: `/data`
  (outbox + recall index + backups + shadow history).
- **Startup validation**: fail fast (non-zero exit) if `ECHO_BASE`/`ECHO_KEY`/
  `ECHO_MCP_TOKEN` are missing — a misdeployed container should crash-loop
  visibly, not serve errors.
- **CI/deploy**: Gitea workflow on push-to-main (tags/releases do **not**
  autobuild on this host) builds the image and the PORT autodeploy webhook
  rolls it out; `deploy.unraid.yml` manifest in `mcp-server/`. Publish
  `echomcp.alwisp.com` via the usual CF + NPM flow; add the Kuma monitor.
- **Vault adjacency (confirmed: Obsidian runs on ALPHA)**: the Local REST
  API's non-encrypted HTTP binding is enabled (2026-07-28) —
  **`ECHO_BASE=http://10.2.0.35:27123`** — so the container talks to Obsidian
  directly, never through the public `echoapi.alwisp.com` hairpin. All vault
  chatter behind a tool call stays host/LAN-local; vault ops stop depending on
  Cloudflare/DNS/NPM health entirely (the public chain is only needed for the
  `echomcp` ingress). Why HTTP not HTTPS here: the REST API's HTTPS port
  (27124) uses a self-signed cert, which `echo.py`'s default-verifying
  `HTTPSConnection` rejects — cleartext on the internal network beats adding an
  insecure-TLS flag to the client. **Exposure check (do once):** 27123 carries
  the vault bearer token in cleartext, so it must not be reachable from
  outside — confirm no router/firewall forward for 27123; the only public
  doors stay HTTPS 443 via NPM (`echoapi` for the CLI fallback path, which
  remains unchanged, and `echomcp` for this server).

## 9. Client registration & fallback

- **Claude Code / CoWork**: register as a remote MCP server —
  `https://echomcp.alwisp.com/mcp` with the `Authorization: Bearer` header
  (project or user scope). No plugin-manifest coupling; the plugin does **not**
  register the server.
- **claude.ai**: custom connector with the same URL + token — ECHO memory from
  the browser/phone, a surface the plugin could never reach.
- **SKILL.md** gains: *"When the `echo_*` MCP tools are available, prefer them
  for every operation they cover; the `$ECHO` CLI recipes are the fallback for
  hosts without the connector or when the server is unreachable."* Procedures
  unchanged.
- **Fallback matrix**: server up ⇒ tools. Server down, machine has the plugin ⇒
  CLI path exactly as today (including its own offline queue). Both down ⇒
  today's "vault unreachable, proceed without memory".

## 10. Multi-user note (Andy / Bryan / Gretchen / Arsalan)

One container serves **one vault**. The image is credential-free, so additional
users are additional deployments of the same image with their own env (their
vault endpoint/key + their own `ECHO_MCP_TOKEN`), e.g. `echomcp-bryan` on
another port/subdomain. Cheap, isolated, no code change. A single multi-tenant
server (token → vault map) is deliberately out of scope — that's CHORUS's
territory.

## 11. Testing & eval

1. **Protocol/integration tests** (`eval/test_mcp_server.py`): run the FastMCP
   app in-process (or via `httpx` test client) against `mock_olrapi`; assert
   initialize/tools-list schemas, auth (no token ⇒ 401; bad token ⇒ 401;
   `/health` open), envelope purity on every tool.
2. **Tool behavior**: per tool — happy path, validation failure (unknown param
   rejected with field name), vault-down (writes ⇒ `queued: true`, reads ⇒
   actionable error), duplicate-gate flow (gate → `merge_into` retry succeeds).
3. **Parity**: `capture` via MCP and via CLI against twin mock vaults produce
   byte-identical notes/index entries (guards Phase 0).
4. **Container smoke** (CI): build image, start with mock env, `/health` green,
   one authed `tools/call` round-trip; healthcheck probes 127.0.0.1.
5. **MCP Inspector** manual pass against the deployed container.
6. **Eval set** (mcp-builder Phase 4): 10 read-only Q&A pairs against a seeded
   mock vault exercising recall→get_note→resolve chains; `eval/mcp_eval.xml`.

## 12. Build-session plan of record

| Phase | Deliverable | Est. size |
|---|---|---|
| 0 | Return-not-print refactor in the plugin tree (`*_op` cores); suites green | ~⅓ the session |
| 1 | `mcp-server/`: FastMCP app, auth middleware, result wrapper, error map, `/health` | ~200 lines |
| 2 | Read tools (load, recall, resolve, get_note, scope×2, health) | wiring |
| 3 | Write tools (capture, link, append/patch, triage, reflect, log_session) | wiring + descriptions |
| 4 | Warm caches + `/data` outbox + startup validation | small |
| 5 | Dockerfile (legacy syntax) + CI workflow + PORT manifest + deploy + CF/NPM publish + Kuma | infra pass |
| 6 | Tests (§11) + Inspector pass + eval XML; SKILL.md/README/CHANGELOG | ~⅓ the session |

Definition of done: Inspector lists 13 tools against `echomcp.alwisp.com`; a
live session (desktop **and** CoWork) performs a full memory day — load →
recall → capture → gate → merge_into → triage → log_session — without one Bash
call; parity test green; container smoke test in CI; Kuma monitor green.