Plans: 2.3.0 index train shipped

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

.dockerignore

@@ -0,0 +1,11 @@
+# Only mcp-server/ + the plugin scripts reach the image; keep the context lean.
+.git
+*.plugin
+dist/
+CODEX/
+docs/
+eval/
+echo-icon*
+*.pdf
+*.html
+__pycache__/

.gitea/workflows/docker-build.yml

@@ -0,0 +1,50 @@
+name: Build and Push Docker Image
+
+on:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+jobs:
+  build:
+    # Runs on the forgerunner host: bundled Docker CLI + mounted /var/run/docker.sock.
+    runs-on: host
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Log in to Gitea Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: registry.alwisp.com
+          username: ${{ secrets.REGISTRY_USER }}
+          password: ${{ secrets.REGISTRY_TOKEN }}
+
+      - name: Build and Push
+        run: |
+          IMAGE="registry.alwisp.com/${{ gitea.repository }}"
+          GIT_SHA="$(git rev-parse --short HEAD)"
+          COMMIT_COUNT="$(git rev-list --count HEAD)"
+          docker build \
+            --label org.alwisp.git-sha="${{ gitea.sha }}" \
+            --label org.alwisp.version="v2.${COMMIT_COUNT}" \
+            --label org.alwisp.repo="${{ gitea.repository }}" \
+            -t "${IMAGE}:latest" .
+          docker push "${IMAGE}:latest"
+
+      # Dangling-only prune: removes untagged leftovers from previous builds. Never
+      # removes the tagged :latest or any image referenced by a running container.
+      - name: Prune dangling images on host
+        if: always()
+        run: docker image prune -f 2>/dev/null || true
+
+      - name: Trigger PORT redeploy
+        if: success()
+        run: |
+          # Repo is 'echo' but the container is 'echo-mcp' — target it explicitly.
+          curl -fsS -X POST https://port.alwisp.com/hooks/gitea \
+            -H "X-Deploy-Token: ${{ secrets.WEBHOOK_SECRET }}" \
+            -H "Content-Type: application/json" \
+            -d '{"container":"echo-mcp"}' || echo "PORT redeploy trigger failed (non-fatal)"

CHANGELOG.md

@@ -1,5 +1,91 @@
 # Changelog
 
+## 2.3.0
+
+**The index train** (IMPROVEMENT-PLANS #2/#4/#8) — one schema-bump release:
+recall-index schema 3, entity-index schema 2. Old recall indexes are discarded
+and rebuilt automatically (sub-second); no vault migration.
+
+### Changed — the recall index is LOCAL-FIRST
+
+The live BM25 index moves out of the vault into the machine state dir
+(`recall-index-<endpoint-hash>.json` under `ECHO_STATE_DIR`). Capture's index
+upkeep — previously GET-whole-index → PUT-whole-index against the vault under
+the advisory lock, an O(vault) network round trip per write — is now a
+zero-network atomic file write with no lock at all. The vault copy at
+`_agent/index/recall-index.json` becomes a **snapshot**: written by
+`sweep --apply` and best-effort at `session-end`, read only to seed a fresh
+machine or to catch up after another client swept (checked at most every
+`ECHO_RECALL_SYNC_HOURS`, default 24). The read path never PUTs the vault.
+
+### Added — incremental sweep (`sweep --fast`) + content hashes
+
+Entity-index entries and recall-index doc meta now carry a 16-char content
+hash. `sweep --fast` walks the listing (cheap) and fetches only what is new,
+gone, missing a hash, or in **today's rotating verification shard**
+(`hash(path) % 7 == weekday` — the whole vault gets verified across a week of
+fast sweeps while each run stays tiny; `--all-shards` forces everything).
+Index-only and machine-owned, so no `--apply` gate; deletions are dropped from
+both indexes. **`load` auto-runs it** when this machine's last sweep is older
+than `ECHO_FAST_SWEEP_DAYS` (default 7) — edits made directly in Obsidian or by
+other clients now reach the indexes without anyone remembering maintenance.
+`doctor` reports index freshness.
+
+### Added — stemming + alias query expansion
+
+`echo_stem.py`: a conservative Porter-lite stemmer applied by `tokenize()` at
+BOTH index and query time, so "deployed"/"deployment"/"deploys" meet at
+"deploy" and "penalties" finds "penalty" (guards against over-stemming: "sing"
+is not "s"; the -al/d~s irregulars are deliberately not conflated). At recall
+time, a query that fuzzy-matches an entity (score ≥ 0.5) folds that entity's
+title/alias vocabulary into the BM25 query at **half weight** — "turbokappa
+tuning" finds the Kappa Engine note even though the alias appears in no body.
+Expansion can only boost documents that actually contain the terms.
+
+### Eval
+
+Gold set grows 8 → **14 queries** (6 paraphrases the pre-2.3 lexical index
+missed). v2.3.0: recall@5 / MRR **1.00 / 1.00**, 3/3 session-journal queries;
+keyword baseline 0.75 / 0.79 and 0/3. Tests: +3 unit (stemmer families,
+over-stemming guards, hash), +10 end-to-end (local-first no-snapshot-write,
+local recallability, stemmed recall, alias expansion, fast-sweep pickup/
+deletion/hash backfill). All suites — including the offline-queue, ops-api,
+and MCP server suites — green; test harnesses now isolate `ECHO_STATE_DIR`.
+
+## 2.2.0
+
+### Added — echo-mcp: the containerized MCP server
+
+ECHO's operations are now reachable as **typed MCP tools** from any surface with
+an MCP connector (Claude Code, CoWork, claude.ai) — no Python-capable shell, no
+`$ECHO` path resolution, no output re-parsing. New `mcp-server/` + `Dockerfile`
+in this repo; deployed on ALPHA as the `echo-mcp` container behind
+`https://echomcp.alwisp.com` (streamable HTTP, stateless JSON, bearer auth,
+open `/health` for Docker/Kuma).
+
+- **14 tools** wrapping the 2.1.1 `*_op` cores: `echo_load`, `echo_recall`
+  (score-packed `budget_chars`), `echo_resolve`, `echo_get_note`
+  (`section`/`max_chars`, traversal-guarded), `echo_get_scope`/`echo_set_scope`,
+  `echo_health` (`deep` runs the linter), `echo_capture` (gate as data —
+  `merge_into`/`force` are parameters), `echo_link` (paths OR resolvable names),
+  `echo_append_note`/`echo_patch_note` (invalid-target errors include the note's
+  actual headings), `echo_triage_inbox` (list/preview/apply in one tool),
+  `echo_reflect`, `echo_log_session` (the session-end bundle, heartbeat-last).
+- **Tool profiles**: `ECHO_MCP_TOOLS=core` exposes only the six daily drivers.
+- **Vault adjacency**: the container talks to the Obsidian REST API's HTTP
+  binding on the same box (`ECHO_BASE=http://10.2.0.35:27123`) — vault ops stop
+  depending on Cloudflare/NPM/DNS; the public chain only fronts the `echomcp`
+  ingress. Server-side offline queue at `/data`.
+- **All MCP writes serialize** through the server process; the vault advisory
+  lock still coordinates with CLI clients.
+- SKILL.md: prefer the `echo_*` tools when present; CLI recipes are the fallback.
+- New suite `eval/test_mcp_server.py` (skips without the SDK): health/auth/
+  initialize/tools-list + the capture→gate→merge→recall→log_session flow over
+  real streamable HTTP.
+
+The plugin itself is unchanged apart from the SKILL.md note — the container
+vendors `skills/echo-memory/scripts/` at build time (one canonical tree).
+
 ## 2.1.1
 
 ### Changed — Phase 0 of the MCP build: every high-level op returns an envelope

Dockerfile

@@ -0,0 +1,26 @@
+# echo-mcp — containerized MCP server over the ECHO vault (docs/MCP-SERVER-SPEC.md).
+# LEGACY Dockerfile format on purpose: the git.alwisp.com CI runner has no BuildKit
+# (no `# syntax=` line, no RUN --mount).
+FROM python:3.12-slim
+
+WORKDIR /app
+
+COPY mcp-server/requirements.txt /app/requirements.txt
+RUN pip install --no-cache-dir -r /app/requirements.txt
+
+# The canonical plugin scripts ARE the server's ops layer — vendored at build time,
+# never a second source tree.
+COPY echo-memory.plugin.src/skills/echo-memory/scripts /app/scripts
+COPY mcp-server/app.py /app/app.py
+
+ENV ECHO_STATE_DIR=/data \
+    ECHO_MCP_PORT=8765 \
+    PYTHONUNBUFFERED=1
+VOLUME /data
+EXPOSE 8765
+
+# Probe 127.0.0.1, NOT localhost (::1-vs-IPv4 lesson from cpas/memer/breedr).
+HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
+  CMD python3 -c "import urllib.request;urllib.request.urlopen('http://127.0.0.1:8765/health', timeout=4)" || exit 1
+
+CMD ["python3", "/app/app.py"]

README.md

@@ -1,4 +1,4 @@
-# echo-memory — v2.1.1
+# echo-memory — v2.3.0
 
 Persistent memory for Claude / CoWork sessions via the **ECHO** Obsidian vault, driven over the [Obsidian Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api). The skill makes direct REST calls through a bundled validated client (`scripts/echo.py`). The whole toolchain is **pure Python** (stdlib only), so it runs identically on Windows, macOS, and Linux — no bash, no platform-specific `date`.
 
@@ -403,14 +403,14 @@ If the API returns a connection error, timeout, or `502` (usually Obsidian / the
 
 ---
 
-## Eval metrics (v1.5.1, 2026-07-03)
+## Eval metrics (v2.3.0, 2026-07-29)
 
-From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate.
+From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate. The gold set grew to **14 queries in 2.3** — six are paraphrases (morphological variants like "penalties in the SLA" vs a note saying "penalty clause") that a purely lexical index misses; the stemmed index answers all of them.
 
-| Metric | v1.5.1 | pre-1.5 baseline |
+| Metric | v2.3.0 | pre-1.5 baseline |
 |---|---|---|
-| Retrieval recall@5 / MRR (8-query gold set) | **1.00 / 1.00** | 0.75 / 0.75 |
-| Queries answerable only from sessions/journal | **2/2** | 0/2 (not in corpus) |
+| Retrieval recall@5 / MRR (14-query gold set incl. 6 paraphrases) | **1.00 / 1.00** | 0.75 / 0.79 |
+| Queries answerable only from sessions/journal | **3/3** | 0/3 (not in corpus) |
 | Freshness: live note outranks stale archived twin | **yes** | no |
 | Duplicate notes created (3 renamed-entity captures) | **0** (gate blocks) | 3 |
 | Legitimate captures wrongly blocked | **0** | 0 |
@@ -421,6 +421,8 @@ From the credential-free harness (`eval/run_eval.py` against the deterministic m
 
 | Version | Highlights |
 |---------|-----------|
+| **2.3.0** | **The index train (recall-index schema 3, entity-index schema 2).** (1) **Local-first recall index** — the live BM25 index moves to the machine state dir (keyed by endpoint); capture's index upkeep becomes a zero-network atomic file write instead of GET/PUT-whole-index under the vault lock; the vault copy is now a snapshot (written by sweep + session-end) that seeds fresh machines. (2) **Incremental sweep** — entity entries carry a content hash; `sweep --fast` fetches only new/gone/changed notes plus a rotating weekday shard (whole vault verified across a week); auto-runs at load when >`ECHO_FAST_SWEEP_DAYS` (7) — Obsidian-side edits now reach the indexes without manual maintenance. (3) **Stemming + alias expansion** — `echo_stem` (Porter-lite, conservative) applied at index+query time, and a query matching an entity's alias folds its title/alias vocabulary in at half weight. Eval gold set grows to 14 queries (6 paraphrases): recall@5 / MRR **1.00 / 1.00**. |
+| **2.2.0** | **echo-mcp — the containerized MCP server.** ECHO as 14 typed MCP tools from any connector-capable surface (Claude Code, CoWork, claude.ai): `mcp-server/` + Dockerfile in-repo, deployed on ALPHA as `echo-mcp` behind `https://echomcp.alwisp.com` (streamable HTTP, stateless JSON, bearer auth, open `/health`). Talks to the Obsidian REST API directly on the LAN (`10.2.0.35:27123`) — one client round trip per tool call, all vault chatter host-local. Duplicate gate & offline queueing surface as data; `ECHO_MCP_TOOLS=core` trims the surface to six tools; writes serialize server-side. New `eval/test_mcp_server.py` e2e suite. Full spec: `docs/MCP-SERVER-SPEC.md`. |
 | **2.1.1** | **MCP Phase 0 — return-not-print.** Every high-level op gains a core `*_op` function returning an envelope dict with a stdout-purity guarantee (helper chatter → stderr); CLI verbs become thin wrappers with identical output and exit codes. Duplicate gate and offline queueing become data (`action: duplicate-gate` / `queued: true`). New `eval/test_ops_api.py` contract suite. This is the seam the 2.2 containerized MCP server wraps. |
 | **2.1.0** | **Quick-wins train: cheaper sessions, durable capture, one-call session end.** (1) **`load --brief`** — a token-budgeted cold-start digest (Fact/Pattern in full, last ~10 Observations, scope+freshness, last session's key sections, Agent-Log lines, inbox *count*; `ECHO_LOAD_BUDGET` default ~8000 chars) now injected by the SessionStart hook, replacing the full six-file dump that grew unboundedly; all `load` reads are fetched in parallel. (2) **Offline capture durability** — `capture` on an unreachable vault queues the *whole operation* as one semantic record; `flush` replays it **through capture** so routing/gate/aliasing re-run against the current index; a gate stop on replay is kept + flagged, never landed blind; `ensure_daily_log`/update-path writes ride the queue too. (3) **`session-end`** — one call (one lock) writes session log → Agent-Log line → reflect proposals → optional scope switch → **heartbeat last as the commit marker**; dry-run by default; `ECHO_NOW` pins the HHMM. Also fixes the `__main__` twin-module trap so helper-module `EchoError`s exit with their intended codes. +19 end-to-end checks. |
 | **2.0.0** | **Packaging & structure major — memory behavior unchanged.** Skills-only: the legacy `commands/` directory is deleted (breaking for pre-skills clients; the 1.6.0-verified skills are the sole entry points). Artifact policy: the repo tracks only the `echo-memory.plugin` pointer; the 15 historical versioned zips leave the tree and versioned builds ship as **Gitea releases** (one per `v<version>` tag) from `v2.0.0` on; `.gitignore` blocks `*.plugin` except the pointer. README gains the "packaging for other agent runtimes" port note (build-target rule, never a second source tree). |

deploy.unraid.yml

@@ -0,0 +1,28 @@
+# echo-mcp — PORT deploy manifest (ALPHA / br0). See docs/MCP-SERVER-SPEC.md §8.
+# Container name deliberately differs from the repo (echo): the repo ships the
+# plugin AND this server; only the server is a container.
+name: echo-mcp
+image: registry.alwisp.com/jason/echo:latest
+network: br0
+ip: auto
+ports: []            # br0 static IP — app serves :8765 directly
+volumes:
+  - /mnt/user/appdata/echo-mcp:/data     # outbox + read cache (+ backups/history, v1.1)
+env:
+  # Vault adjacency: the Obsidian Local REST API's HTTP binding on the same box —
+  # never the public echoapi.alwisp.com hairpin (spec §8; cleartext stays on the LAN).
+  ECHO_BASE: http://10.2.0.35:27123
+  ECHO_OWNER: Jason Stedwell
+  ECHO_STATE_DIR: /data
+  ECHO_MCP_PORT: "8765"
+  ECHO_MCP_TOOLS: full
+  # Secrets from the PORT secret store — values never appear in this file or chat.
+  ECHO_KEY: SECRET:echo-vault-key
+  ECHO_MCP_TOKEN: SECRET:echo-mcp-token
+health_check_path: /health
+health_check_port: 8765
+webui: https://echomcp.alwisp.com
+proxy:
+  forward_port: 8765
+  forward_scheme: http
+  websockets: true

docs/IMPROVEMENT-PLANS.md

@@ -349,5 +349,5 @@ manifest-verification pass).
 | **2.0.0 — SHIPPED 2026-07-28** | ROADMAP-2.0: commands/ deleted · release-based artifacts · gitignore · Codex note | Packaging-only major; skills-only install re-verified. |
 | **2.1.0 quick wins — SHIPPED 2026-07-28** | #1 brief load · #3 offline capture · #7 session-end | All three landed with +19 e2e checks; see CHANGELOG 2.1.0. The MCP spec's Phase 0 (return-not-print) did NOT ride along — it's the first phase of the MCP build session. |
 | **2.2** | MCP server (containerized, `echomcp.alwisp.com`) — `docs/MCP-SERVER-SPEC.md` | Needs Phase 0 (return-not-print refactor, first step of the build session). #7 session-end shipped, so `echo_log_session` wraps a real verb. Deployed via CI + PORT, not the plugin manifest. |
-| **2.3 index train** | #2 local-first recall (CLI-path half) · #4 incremental sweep · #8 stemming+expansion | One schema-bump train; all touch the same index/sweep loop. The container already keeps its index warm in memory (spec §7), so #2 here covers the CLI fallback path; #4's fast sweep also becomes the container's background job. |
-| **2.4 memory train** | #5 lifecycle/decay · #6 supersession · #9 resolve-miss learning | Capability tier; #6 after #5 (status vocabulary). |
+| **2.3.0 index train — SHIPPED 2026-07-29** | #2 local-first recall · #4 incremental sweep (`sweep --fast`, auto at load) · #8 stemming+alias expansion | Recall schema 3 + entity schema 2 in one bump; eval 14-query gold set at recall@5/MRR 1.00/1.00. echo-mcp auto-redeployed with it. |
+| **2.4 memory train** | #5 lifecycle/decay · #6 supersession · #9 resolve-miss learning | Capability tier; #6 after #5 (status vocabulary). Remaining alongside: the 2.2.x dividends (nightly vault backup FIRST, claude.ai OAuth shim, shadow history, background sweeps, alerting). |

docs/MCP-SERVER-SPEC.md

@@ -1,9 +1,9 @@
 # echo-mcp — MCP Server Build Spec (containerized)
 
-> Status: **spec for a future build session** (written 2026-07-28 against v1.5.1;
-> revised same day: **remote container architecture**, operator decision — heavy
-> lifting belongs in a deployed container, not on any one machine).
-> Companion plan for the other nine review items: `docs/IMPROVEMENT-PLANS.md`.
+> Status: **BUILT — shipped as 2.2.0, 2026-07-28** (`mcp-server/app.py`, 14 tools —
+> the count below saying 13 undercounted the append/patch pair; e2e suite
+> `eval/test_mcp_server.py`). This document remains the design record; §7.2 is the
+> live v1.1 backlog. Companion plan: `docs/IMPROVEMENT-PLANS.md`.
 >
 > **Prerequisites before starting this build:**
 > 1. The `session-end` verb (IMPROVEMENT-PLANS #7) — the MCP tool wraps it.
@@ -337,8 +337,13 @@ that division of labor is permanent, not a v1 scope cut).
   `https://echomcp.alwisp.com/mcp` with the `Authorization: Bearer` header
   (project or user scope). No plugin-manifest coupling; the plugin does **not**
   register the server.
-- **claude.ai**: custom connector with the same URL + token — ECHO memory from
-  the browser/phone, a surface the plugin could never reach.
+- **claude.ai**: **blocked in v1 (found at 2.2.0 deploy)** — claude.ai custom
+  connectors run the MCP OAuth flow (Dynamic Client Registration against the
+  server) and offer no custom-header option, so a bearer-only server cannot
+  connect. Fix = a minimal single-user OAuth layer on echo-mcp (SDK auth
+  interface: DCR + one-time operator approval + token issuance) — **v2.2.x
+  backlog, alongside the nightly vault backup**. Token-in-URL workarounds are
+  rejected (credentials leak into logs).
 - **SKILL.md** gains: *"When the `echo_*` MCP tools are available, prefer them
   for every operation they cover; the `$ECHO` CLI recipes are the fallback for
   hosts without the connector or when the server is unreachable."* Procedures

echo-memory.plugin.src/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "echo-memory",
-  "version": "2.1.1",
+  "version": "2.3.0",
   "homepage": "https://git.alwisp.com/jason/echo",
   "repository": "https://git.alwisp.com/jason/echo",
   "description": "Persistent memory via the ECHO Obsidian vault over the Obsidian Local REST API. Cross-platform Python client: one-call capture/resolve/recall/link/triage over an entity index, hybrid BM25 + graph recall spanning entities + sessions/journal (recency/status-aware), a pre-write duplicate gate, complete-frontmatter capture, session hooks that self-fire load/reflect, offline write-ahead queue, lock-guarded concurrency, linter-enforced routing, and /echo-* commands.",

echo-memory.plugin.src/skills/echo-memory/SKILL.md

@@ -52,6 +52,19 @@ Executable logic ships under `scripts/` — **pure Python**, so the whole toolch
 - `scripts/check_routing.py` — verifies the routing docs stay in sync with `routing.json` (dev/CI; offline)
 - `scripts/bootstrap.py` / `scripts/migrate.py` — deterministic vault setup/repair and schema migration
 
+## MCP tools first (when the echo-mcp connector is available)
+
+When this session has the **`echo_*` MCP tools** (the deployed echo-mcp server —
+`echo_load`, `echo_recall`, `echo_resolve`, `echo_get_note`, `echo_get_scope`/`echo_set_scope`,
+`echo_health`, `echo_capture`, `echo_link`, `echo_append_note`/`echo_patch_note`,
+`echo_triage_inbox`, `echo_reflect`, `echo_log_session`), **prefer them over the CLI
+for every operation they cover** — typed calls, structured results, no path
+resolution or quoting. The procedures in this skill (reconcile at load, search-first,
+scope discipline, third person, preview-before-apply) are unchanged and apply to both
+surfaces. A `duplicate-gate` tool result is the same contract as capture exit 76:
+`merge_into` or confirmed `force`, never a blind retry. The CLI recipes below are the
+fallback for hosts without the connector or when the server is unreachable.
+
 ## Bundled Tooling (prefer over raw curl)
 
 All paths below are under `${CLAUDE_PLUGIN_ROOT}/skills/echo-memory/`. **Invoke with `python3`** (on Windows where that isn't on PATH, use `python` or `py -3`).
@@ -345,6 +358,8 @@ Run it after `migrate.py` (which handles structural/schema changes) — or any t
 
 The pass is cheap and pays for itself by catching drift before it requires a reorg. Write the findings as a digest; act on them only with the operator's go-ahead.
 
+**Incremental maintenance (2.3).** `sweep.py --fast` is the cheap, index-only mode: it fetches just what's new, gone, or changed (content hashes) plus a rotating daily verification shard, so edits made directly in Obsidian or by other clients reach the entity + recall indexes without a full pass. It runs **automatically at load** when this machine's last sweep is older than `ECHO_FAST_SWEEP_DAYS` (default 7) — you rarely need to invoke it. The recall index itself is **local-first**: capture maintains a machine-local copy with zero vault round-trips; the vault's `recall-index.json` is a snapshot (sweep/session-end) used to seed fresh machines.
+
 **Performance.** Full-vault scripts (`sweep.py`, `vault_lint.py`, recall rebuild) are connection-pooled (keep-alive) and read the whole vault concurrently via `echo.read_many`, so they finish in about a second on a few-hundred-note vault instead of timing out on hundreds of serial TLS handshakes. Tune with `ECHO_WORKERS` (default 8, bulk-read concurrency) and `ECHO_TIMEOUT` (default 30s, per-request). For a vault large enough that even the concurrent pass nears the tool timeout, run the script with the harness's background-execution option rather than blocking the turn.
 
 ## Daily Note — Agent Log

echo-memory.plugin.src/skills/echo-memory/scripts/echo.py

@@ -790,6 +790,24 @@ def _load_gather(targets):
     return results, texts, listing_files, offline, marker_missing, heartbeat_absent
 
 
+def _auto_fast_sweep(offline: bool) -> str | None:
+    """Opportunistic index maintenance at load (2.3): when this machine's last fast
+    sweep is older than ECHO_FAST_SWEEP_DAYS (default 7), run `sweep --fast` so the
+    local recall + entity indexes true up against edits made by Obsidian or other
+    clients. Best-effort; skipped offline; never raises."""
+    if offline:
+        return None
+    try:
+        import sweep as sweep_mod
+        days = float(os.environ.get("ECHO_FAST_SWEEP_DAYS", "7"))
+        age = sweep_mod.fast_sweep_age_days()
+        if age is not None and age < days:
+            return None
+        return sweep_mod.fast()
+    except Exception:  # noqa: BLE001 — maintenance must never block a load
+        return None
+
+
 def _fetch_pointed_session(texts: dict) -> None:
     """Follow the heartbeat pointer and stash the pointed session log for the digest."""
     hb = texts.get("heartbeat")
@@ -818,9 +836,10 @@ def load_op(brief: bool = True) -> dict:
     targets = LOAD_TARGETS + [("today", f"journal/daily/{today()}.md"),
                               ("inbox", "inbox/captures/inbox.md")]
     _, texts, listing_files, offline, marker_missing, _ = _load_gather(targets)
+    swept = _auto_fast_sweep(offline)
     data = {"ok": True, "action": "load", "offline": offline,
             "marker_missing": marker_missing, "synced": synced,
-            "needs_attention": flagged,
+            "needs_attention": flagged, "fast_sweep": swept,
             "sections": {k: v for k, v in texts.items() if not k.startswith("_")},
             "recent_sessions": sorted((f for f in listing_files if f.endswith(".md")),
                                       reverse=True)[:5]}
@@ -863,6 +882,10 @@ def cmd_load(brief: bool = False) -> int:
     results, texts, listing_files, offline, marker_missing, heartbeat_absent = \
         _load_gather(targets)
 
+    swept = _auto_fast_sweep(offline)
+    if swept:
+        print(f"({swept})\n")
+
     if brief:
         _fetch_pointed_session(texts)
         print(_render_brief(texts, listing_files, offline))

echo-memory.plugin.src/skills/echo-memory/scripts/echo_doctor.py

@@ -74,6 +74,18 @@ def run_op() -> dict:
             else:
                 line(status < 400, "marker fetch", f"HTTP {status}")
 
+    # Index freshness (2.3) — informational, never red: how stale this machine's
+    # local indexes are (the fast sweep trues them up at load past 7 days).
+    if fatal is None:
+        try:
+            import sweep as sweep_mod
+            age = sweep_mod.fast_sweep_age_days()
+            line(True, "index freshness",
+                 f"last local sweep {age:.1f}d ago" if age is not None
+                 else "no local sweep yet — runs automatically at the next load")
+        except Exception:  # noqa: BLE001
+            pass
+
     reds = sum(1 for c in checks if not c["ok"])
     return {"ok": reds == 0 and fatal is None, "action": "doctor",
             "endpoint": cfg["endpoint"], "fatal": fatal, "reds": reds, "checks": checks}

echo-memory.plugin.src/skills/echo-memory/scripts/echo_index.py

@@ -168,7 +168,10 @@ def kind_for_path(path: str) -> str | None:
 
 
 def empty_index() -> dict:
-    return {"schema": 1, "updated": echo.today(), "entities": {}}
+    # Schema 2 (2.3): entries may carry `h`, a short content hash of the note body —
+    # the incremental sweep's change detector. Tolerated-absent (schema-1 entries
+    # simply get fetched and hashed on their first fast sweep).
+    return {"schema": 2, "updated": echo.today(), "entities": {}}
 
 
 def load() -> dict:
@@ -287,13 +290,20 @@ def gate_candidates(index: dict, mention: str, kind: str | None = None,
     return out
 
 
+def content_hash(text: str) -> str:
+    import hashlib
+    return hashlib.sha1(str(text).encode("utf-8", "replace")).hexdigest()[:16]
+
+
 def upsert(index: dict, slug: str, path: str, kind: str, title: str | None = None,
-           aliases=None) -> dict:
+           aliases=None, h: str | None = None) -> dict:
     ents = index.setdefault("entities", {})
     e = ents.get(slug, {})
     e["path"] = path
     e["kind"] = kind
     e["title"] = title or e.get("title") or slug
+    if h:
+        e["h"] = h
     # Keep any prior + explicit aliases, and ALWAYS fold in the safe title variants so a
     # hyphen/space/case form of the name resolves even if no one passed --aliases. Drop any
     # alias that just equals the slug (resolve already checks the slug).

echo-memory.plugin.src/skills/echo-memory/scripts/echo_ops.py

@@ -333,15 +333,11 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status
             echo.cmd_put(path, echo.temp_file(note.encode()))
             action = "created"
 
-        # H3: the entity-index write is the race the review flagged — a bare load->save lets
-        # two concurrent captures clobber each other's entry. Route it through a lock-guarded,
-        # fresh-re-read transaction. Returns the fresh index (used by auto-link below).
-        import echo_concurrency
-        index = echo_concurrency.atomic_index_update(
-            lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases))
-
-        # auto-link: any other known entity CONFIDENTLY named in the body (M2: is_confident_link
-        # rejects short/common tokens so a 3-char alias or a word like "API" can't link everywhere).
+        # auto-link FIRST (it mutates the note's ## Related section), so the content
+        # hash stored below reflects the note's final state. Uses the pre-update index
+        # — it only needs to find OTHER entities named in the body. (M2: is_confident_link
+        # rejects short/common tokens so a 3-char alias or a word like "API" can't link
+        # everywhere.)
         linked = 0
         for s2, e2 in index.get("entities", {}).items():
             if e2.get("path") in (None, path):
@@ -351,11 +347,24 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status
                 ca, cb = links.link_bidirectional(path, e2["path"])
                 linked += int(ca or cb)
 
-        # Keep the BM25 recall index current for this note (best-effort; lock-guarded inside
-        # update_note, so it can't clobber a concurrent writer either).
+        # The note's final content: hashed into the entity index (the incremental
+        # sweep's change detector, 2.3) and fed to the local recall index.
+        final_text = links.get_text(path) or ""
+        note_h = idx_mod.content_hash(final_text) if final_text else None
+
+        # H3: the entity-index write is the race the review flagged — a bare load->save lets
+        # two concurrent captures clobber each other's entry. Route it through a lock-guarded,
+        # fresh-re-read transaction.
+        import echo_concurrency
+        index = echo_concurrency.atomic_index_update(
+            lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases,
+                                       h=note_h))
+
+        # Keep the local BM25 recall index current for this note (best-effort; a
+        # zero-network atomic file write since 2.3).
         try:
             import echo_recall
-            echo_recall.update_note(path, links.get_text(path) or "")
+            echo_recall.update_note(path, final_text)
         except Exception as exc:  # never let recall-index upkeep fail a capture
             print(f"echo_ops: recall-index update skipped ({exc})", file=sys.stderr)
 

echo-memory.plugin.src/skills/echo-memory/scripts/echo_recall.py

@@ -52,9 +52,17 @@ sys.path.insert(0, str(Path(__file__).resolve().parent))
 import echo                      # noqa: E402
 import echo_index as idx_mod     # noqa: E402
 import echo_links as links       # noqa: E402
+import echo_stem                 # noqa: E402
 
 RECALL_INDEX_PATH = "_agent/index/recall-index.json"
-INDEX_SCHEMA = 2                 # bumped: schema 2 adds per-doc meta (weight/updated/status)
+# Schema 3 (2.3, the "index train"): postings are STEMMED (echo_stem), per-doc meta
+# gains a content hash `h` (incremental sweep), and the index carries a `built`
+# stamp + endpoint hash. The LIVE index is LOCAL-FIRST — it lives in the state dir
+# and is updated with zero vault round-trips; the vault copy at RECALL_INDEX_PATH
+# is a snapshot written by sweep/session-end, used to seed fresh machines. Older
+# schemas are discarded and rebuilt (sub-second since the 1.1 network work).
+INDEX_SCHEMA = 3
+SYNC_HOURS = float(os.environ.get("ECHO_RECALL_SYNC_HOURS", "24"))
 
 # Tuning knobs. Defaults are standard BM25 (k1/b) + a 0.6 hop decay over <=2 hops.
 K1 = 1.5
@@ -131,8 +139,11 @@ _SKIP_BASENAMES = {"README.md"}
 
 
 def tokenize(text: str) -> list[str]:
-    """Lowercase word tokens, stopworded. Frontmatter should be stripped by the caller."""
-    return [t for t in _WORD.findall(text.lower()) if t not in _STOP and len(t) > 1]
+    """Lowercase word tokens, stopworded, STEMMED (2.3) — applied identically at
+    index and query time, so 'deployed'/'deployment' meet at 'deploy'.
+    Frontmatter should be stripped by the caller."""
+    return [echo_stem.stem(t) for t in _WORD.findall(text.lower())
+            if t not in _STOP and len(t) > 1]
 
 
 def strip_frontmatter(text: str) -> str:
@@ -152,9 +163,10 @@ class Bm25Index:
         self.df: Counter[str] = Counter()              # term -> #docs containing it
         self.postings: dict[str, dict[str, int]] = {}  # term -> {doc_path: tf}
         self.length: dict[str, int] = {}               # doc_path -> token count
-        self.meta: dict[str, dict] = {}                # doc_path -> {w, u, s} priors
+        self.meta: dict[str, dict] = {}                # doc_path -> {w, u, s, h} priors
         self.n_docs = 0
         self.avg_len = 0.0
+        self.built = ""                                # ISO stamp of last full (re)build
 
     def _recompute(self) -> None:
         self.n_docs = len(self.length)
@@ -165,12 +177,19 @@ class Bm25Index:
             self.remove(path)
         toks = tokenize(strip_frontmatter(raw_text))
         self.length[path] = len(toks)
-        self.meta[path] = doc_meta(path, raw_text)
+        meta = doc_meta(path, raw_text)
+        # Content hash: the incremental sweep's change detector (2.3).
+        import hashlib
+        meta["h"] = hashlib.sha1(raw_text.encode("utf-8", "replace")).hexdigest()[:16]
+        self.meta[path] = meta
         for term, tf in Counter(toks).items():
             self.postings.setdefault(term, {})[path] = tf
             self.df[term] += 1
         self._recompute()
 
+    def content_hash(self, path: str) -> str | None:
+        return (self.meta.get(path) or {}).get("h")
+
     def remove(self, path: str) -> None:
         if path not in self.length:
             return
@@ -192,11 +211,17 @@ class Bm25Index:
                 * freshness(m.get("u"), today_s)
                 * STATUS_FACTOR.get(m.get("s", ""), 1.0))
 
-    def score(self, query: str, limit: int = 10,
-              today_s: str | None = None) -> list[tuple[str, float]]:
+    def score(self, query: str, limit: int = 10, today_s: str | None = None,
+              extra_terms: dict[str, float] | None = None) -> list[tuple[str, float]]:
+        """BM25 x priors. `extra_terms` (term -> weight) are alias-expansion terms
+        (2.3): they contribute at reduced weight and can only boost documents that
+        actually contain them — expansion never invents corpus-free hits."""
         today_s = today_s or echo.today()
+        weighted: dict[str, float] = {t: 1.0 for t in set(tokenize(query))}
+        for t, w in (extra_terms or {}).items():
+            weighted.setdefault(t, w)
         scores: Counter[str] = Counter()
-        for term in set(tokenize(query)):
+        for term, weight in weighted.items():
             posting = self.postings.get(term)
             if not posting:
                 continue
@@ -204,26 +229,29 @@ class Bm25Index:
             for path, tf in posting.items():
                 dl = self.length.get(path, 0)
                 denom = tf + K1 * (1 - B + B * (dl / self.avg_len if self.avg_len else 1))
-                scores[path] += idf * (tf * (K1 + 1) / denom if denom else 0)
+                scores[path] += weight * idf * (tf * (K1 + 1) / denom if denom else 0)
         fused = {p: s * self._doc_factor(p, today_s) for p, s in scores.items()}
         ranked = sorted(fused.items(), key=lambda kv: -kv[1])[:limit]
         return [(p, s) for p, s in ranked if s > MIN_SCORE]
 
     # ---- serialization (compact; rebuildable, so the format can change freely) ----
     def to_json(self) -> dict:
-        return {"schema": INDEX_SCHEMA, "n_docs": self.n_docs, "avg_len": self.avg_len,
-                "length": self.length, "postings": self.postings, "meta": self.meta}
+        return {"schema": INDEX_SCHEMA, "built": self.built, "n_docs": self.n_docs,
+                "avg_len": self.avg_len, "length": self.length,
+                "postings": self.postings, "meta": self.meta}
 
     @classmethod
     def from_json(cls, d: dict) -> "Bm25Index":
         if d.get("schema") != INDEX_SCHEMA:
             return cls()  # older/newer format -> empty; recall's cold path rebuilds
+            # (a pre-stemming index MUST NOT be reused: its postings are unstemmed)
         ix = cls()
         ix.length = d.get("length", {})
         ix.postings = d.get("postings", {})
         ix.meta = d.get("meta", {})
         ix.n_docs = d.get("n_docs", len(ix.length))
         ix.avg_len = d.get("avg_len", 0.0)
+        ix.built = d.get("built", "")
         ix.df = Counter({term: len(posting) for term, posting in ix.postings.items()})
         return ix
 
@@ -271,27 +299,84 @@ def _indexable(path: str) -> bool:
 
 
 # ------------------------------------------------------------------- persistence
-def load_index() -> Bm25Index:
-    status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH))
-    if status == 404:
-        return Bm25Index()
-    echo.check(status, body, "recall-index load")
+# LOCAL-FIRST (2.3). The live index is a machine-local file — updating it on capture
+# is a zero-network atomic file write instead of GET-whole-index -> PUT-whole-index
+# under the vault lock (the old per-write O(vault) round trip). The vault copy is a
+# SNAPSHOT: written by sweep (and best-effort at session end), read only to seed a
+# fresh machine or to catch up after another client swept (checked at most every
+# ECHO_RECALL_SYNC_HOURS). Staleness across clients is tolerable — recall degrades
+# gracefully and the fast sweep trues it up.
+
+def _local_path() -> Path:
+    import hashlib
+    import echo_queue
+    h = hashlib.sha1((echo.BASE or "").encode()).hexdigest()[:12]
+    return echo_queue.state_dir() / f"recall-index-{h}.json"
+
+
+def _load_local() -> Bm25Index | None:
+    p = _local_path()
+    if not p.exists():
+        return None
     try:
+        return Bm25Index.from_json(json.loads(p.read_text(encoding="utf-8")))
+    except (json.JSONDecodeError, KeyError, TypeError, OSError):
+        return None
+
+
+def save_local(ix: Bm25Index) -> None:
+    p = _local_path()
+    p.parent.mkdir(parents=True, exist_ok=True)
+    tmp = p.with_suffix(".json.tmp")
+    tmp.write_text(json.dumps(ix.to_json(), ensure_ascii=False), encoding="utf-8")
+    os.replace(tmp, p)  # atomic: a crash mid-write can't corrupt the live index
+
+
+def _fetch_snapshot() -> Bm25Index | None:
+    try:
+        status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH))
+        if status != 200:
+            return None
         return Bm25Index.from_json(json.loads(body))
-    except (json.JSONDecodeError, KeyError, TypeError):
-        return Bm25Index()
+    except Exception:  # noqa: BLE001 — the snapshot is an optimization, never required
+        return None
 
 
-def save_index(ix: Bm25Index) -> None:
+def save_snapshot(ix: Bm25Index) -> None:
+    """PUT the vault snapshot — sweep + session-end only, never the per-capture path."""
     body = json.dumps(ix.to_json(), ensure_ascii=False).encode("utf-8")
     status, b = echo.request("PUT", echo.vault_url(RECALL_INDEX_PATH), data=body,
                              headers={"Content-Type": "application/json"})
-    echo.check(status, b, "recall-index save")
+    echo.check(status, b, "recall-index snapshot save")
 
 
-def rebuild(prefetched: dict | None = None) -> Bm25Index:
-    """Full rebuild from every indexable entity note. Called by sweep.py and as the
-    cold-cache path inside recall(). Saves and returns the index.
+def load_index() -> Bm25Index:
+    """The live index: local file first; the vault snapshot only seeds a fresh
+    machine or replaces a local copy that a newer sweep elsewhere has superseded
+    (checked when the local build stamp is older than SYNC_HOURS)."""
+    local = _load_local()
+    if local and local.n_docs:
+        try:
+            age_ok = local.built and (
+                _dt.datetime.now(_dt.timezone.utc)
+                - _dt.datetime.strptime(local.built, "%Y-%m-%dT%H:%M:%SZ").replace(
+                    tzinfo=_dt.timezone.utc)
+            ).total_seconds() < SYNC_HOURS * 3600
+        except ValueError:
+            age_ok = False
+        if age_ok:
+            return local
+    snap = _fetch_snapshot()
+    if snap and snap.n_docs and (not local or (snap.built or "") > (local.built or "")):
+        save_local(snap)
+        return snap
+    return local or Bm25Index()
+
+
+def rebuild(prefetched: dict | None = None, snapshot: bool = False) -> Bm25Index:
+    """Full rebuild from every indexable note. Saves the LOCAL index always; writes
+    the vault snapshot only when `snapshot=True` (sweep) — the read path never
+    surprises the vault with a PUT.
 
     `prefetched` (a {path: text} map, e.g. from echo.read_many) lets sweep.py reuse the
     content it already pulled instead of walking and re-fetching the whole vault again."""
@@ -299,27 +384,30 @@ def rebuild(prefetched: dict | None = None) -> Bm25Index:
     if prefetched is not None:
         items = ((p, t) for p, t in prefetched.items() if _indexable(p))
     else:
-        items = ((p, _get(p)) for p in _walk() if _indexable(p))
+        items = echo.read_many([p for p in _walk() if _indexable(p)]).items()
     for path, text in items:
         if text is not None:
             ix.add(path, text)   # add() strips frontmatter + extracts doc meta itself
-    save_index(ix)
+    ix.built = echo.now_iso()
+    save_local(ix)
+    if snapshot:
+        save_snapshot(ix)
     return ix
 
 
 def update_note(path: str, text: str) -> None:
-    """Best-effort incremental upkeep for a single note (mirrors how entities.json is
-    maintained on capture). Never raises into the caller. The load->save is wrapped in the
-    advisory lock with a fresh re-read (H3), so concurrent captures can't clobber the
-    recall index either."""
+    """Best-effort incremental upkeep for a single note — a LOCAL atomic file write,
+    no vault round-trip, no advisory lock (2.3). Maintains an existing local index
+    only: with no local index yet, the next recall's cold path (or a sweep) builds
+    one, and a one-note index must never shadow the snapshot. Never raises."""
     try:
         if not _indexable(path):
-            return  # only entity notes are part of the recall corpus
-        import echo_concurrency
-        with echo_concurrency.vault_lock():
-            ix = load_index()                      # fresh read inside the lock
-            ix.add(path, text)                     # raw text: add() strips + extracts meta
-            save_index(ix)
+            return  # only corpus notes are part of the recall index
+        ix = _load_local()
+        if ix is None or not ix.n_docs:
+            return
+        ix.add(path, text)                     # raw text: add() strips + extracts meta
+        save_local(ix)
     except Exception as exc:  # noqa: BLE001 — upkeep must never fail a capture
         print(f"echo_recall: index update skipped ({exc})", file=sys.stderr)
 
@@ -405,13 +493,27 @@ def recall_op(query, limit: int = 8) -> dict:
     index = idx_mod.load()
     nmap = idx_mod.name_map(index)
 
+    # --- alias query expansion (2.3): a query phrased by an entity's alias or a
+    # partial name folds that entity's title/alias vocabulary into the BM25 query at
+    # half weight — "the ECHO plugin" also scores `echo-memory` terms. Expansion can
+    # only boost documents that actually contain the terms.
+    extra_terms: dict[str, float] = {}
+    q_toks = set(tokenize(q))
+    for _slug, e, sc in idx_mod.fuzzy_candidates(index, q)[:2]:
+        if sc < 0.5:
+            continue
+        for name in (e.get("title", ""), _slug, *e.get("aliases", [])):
+            for tok in tokenize(str(name).replace("-", " ")):
+                if tok not in q_toks:
+                    extra_terms[tok] = 0.5
+
     # --- lexical layer (BM25); rebuild the cache on a cold miss --------------
     lexical: list[tuple[str, float]] = []
     try:
         ix = load_index()
         if ix.n_docs == 0:
             ix = rebuild()
-        lexical = ix.score(q, limit=limit, today_s=today_s)
+        lexical = ix.score(q, limit=limit, today_s=today_s, extra_terms=extra_terms)
     except echo.EchoError as exc:
         print(f"recall: BM25 index unavailable ({exc}); falling back to server search",
               file=sys.stderr)

echo-memory.plugin.src/skills/echo-memory/scripts/echo_session.py

@@ -135,6 +135,18 @@ def session_end_op(bundle: dict, apply: bool = False) -> dict:
         echo.cmd_put("_agent/heartbeat/last-session.md",
                      echo.temp_file(f"{path} @ {echo.now_iso()}\n".encode("utf-8")))
         steps["heartbeat"] = "ok"
+        # Index maintenance (2.3), NOT part of the bundle contract: push this
+        # machine's local recall index to the vault snapshot so other machines
+        # can seed/catch up. Best-effort — a failure never taints the bundle.
+        try:
+            import echo_recall
+            rix = echo_recall._load_local()
+            if rix and rix.n_docs:
+                rix.built = rix.built or echo.now_iso()
+                echo_recall.save_snapshot(rix)
+                steps["recall_snapshot"] = "ok"
+        except Exception:  # noqa: BLE001
+            steps["recall_snapshot"] = "skipped"
 
     data["steps"] = steps
     return echo_output.envelope("session-end", data)

echo-memory.plugin.src/skills/echo-memory/scripts/echo_stem.py

@@ -0,0 +1,78 @@
+#!/usr/bin/env python3
+"""echo_stem.py — a deliberately light, dependency-free English stemmer.  [2.3]
+
+BM25 is purely lexical: "deploy", "deployed", and "deployment" are three unrelated
+terms to it, so a paraphrased recall query misses notes stored with a different
+inflection. This Porter-LITE stemmer conflates the common inflection families while
+staying conservative — when in doubt it leaves the token alone, because an
+over-stemmed index silently merges unrelated words (far worse than a missed match).
+
+Applied by echo_recall.tokenize() at BOTH index and query time (recall-index
+schema 3; older indexes are discarded and rebuilt). Pure function, unit-tested in
+test_echo_client.py.
+
+Design notes:
+  * plural / -ed / -ing families first (with the classic restore rules: doubled
+    consonant undoubling, at/bl/iz +e, cvc +e), then one derivational suffix pass,
+    then y->i and final-e normalization so "navigate"/"navigation"/"navigating"
+    all land on "navigat".
+  * every rule guards a minimum remaining stem length — "sing" is not "s".
+"""
+from __future__ import annotations
+
+import re
+
+_VOWEL = re.compile(r"[aeiouy]")
+_DOUBLE = re.compile(r"([^aeiouylsz])\1$")
+_CVC = re.compile(r"[^aeiou][aeiouy][^aeiouwxy]$")
+
+# One derivational pass, most-specific first. Replacements keep families together:
+# navigational/navigation -> navigat · saturation/saturated -> saturat ·
+# deployment/deployed -> deploy · usefulness/useful -> use... (guarded by length).
+_SUFFIXES = (
+    ("ization", "ize"), ("ational", "ate"), ("fulness", "ful"), ("ousness", "ous"),
+    ("iveness", "ive"), ("tional", "t"), ("biliti", "ble"), ("ements", ""),
+    ("ment", ""), ("ness", ""), ("tion", "t"), ("sion", "s"), ("ance", ""),
+    ("ence", ""), ("able", ""), ("ible", ""), ("ally", "al"), ("ously", "ous"),
+    ("ively", "ive"), ("ly", ""),
+)
+
+
+def stem(t: str) -> str:
+    """Stem one lowercase token. Never raises; returns the token unchanged when no
+    rule applies safely."""
+    if len(t) <= 3 or not t.isascii():
+        return t
+    # --- plurals -------------------------------------------------------------
+    if t.endswith("sses"):
+        t = t[:-2]
+    elif t.endswith("ies") and len(t) > 4:
+        t = t[:-3] + "i"
+    elif t.endswith("s") and not t.endswith(("ss", "us", "is")):
+        t = t[:-1]
+    # --- -ed / -ing (with restore rules) --------------------------------------
+    for suf in ("ingly", "edly", "ing", "ed"):
+        if t.endswith(suf):
+            base = t[: -len(suf)]
+            if len(base) >= 3 and _VOWEL.search(base):
+                t = base
+                if t.endswith(("at", "bl", "iz")):
+                    t += "e"
+                elif _DOUBLE.search(t):
+                    t = t[:-1]
+                elif len(t) == 3 and _CVC.search(t):
+                    t += "e"
+            break
+    # --- one derivational suffix pass -----------------------------------------
+    for suf, rep in _SUFFIXES:
+        if t.endswith(suf):
+            base = t[: -len(suf)]
+            if len(base) >= 3 and len(base + rep) >= 3:
+                t = base + rep
+            break
+    # --- normalize: y->i (so penalty/penalties agree), then drop a final e -----
+    if len(t) > 3 and t.endswith("y") and t[-2] not in "aeiou":
+        t = t[:-1] + "i"
+    if len(t) > 4 and t.endswith("e"):
+        t = t[:-1]
+    return t

echo-memory.plugin.src/skills/echo-memory/scripts/sweep.py

@@ -18,9 +18,19 @@ backfill what older vaults don't have yet:
   5. stamp the marker `schema_version` to the current schema (4).
 
 Dry-run by default; pass --apply to write. READ-ONLY without --apply.
+
+**--fast (2.3)** is the incremental mode: it walks the listing (cheap), then fetches
+only what's new, gone, missing a content hash, or in today's rotating verification
+shard (hash(path) % 7 == weekday — the whole vault gets verified over a week of
+fast sweeps while each run stays tiny; --all-shards forces everything). It updates
+the LOCAL recall index and the entity index (both machine-owned), so it needs no
+--apply gate and writes no notes. `load` auto-runs it when the last fast sweep is
+older than ECHO_FAST_SWEEP_DAYS (default 7). This is what keeps the indexes honest
+against edits made directly in Obsidian or by other clients.
+
 Cross-platform: pure Python via echo.py.
 
-Usage:  sweep.py [--apply]
+Usage:  sweep.py [--apply] | sweep.py --fast [--all-shards]
 """
 
 from __future__ import annotations
@@ -102,10 +112,128 @@ def walk(prefix: str = ""):
         yield from walk(f"{prefix}{d}/")
 
 
+def _stamp_path() -> Path:
+    import echo_queue
+    return echo_queue.state_dir() / "fast-sweep.stamp"
+
+
+def _stamp_fast_sweep() -> None:
+    try:
+        p = _stamp_path()
+        p.parent.mkdir(parents=True, exist_ok=True)
+        p.write_text(echo.now_iso() + "\n", encoding="utf-8")
+    except OSError:
+        pass
+
+
+def fast_sweep_age_days() -> float | None:
+    """Days since the last fast/full sweep on THIS machine, or None if never."""
+    import datetime as dt
+    try:
+        stamp = _stamp_path().read_text(encoding="utf-8").strip()
+        then = dt.datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=dt.timezone.utc)
+        return (dt.datetime.now(dt.timezone.utc) - then).total_seconds() / 86400
+    except (OSError, ValueError):
+        return None
+
+
+def fast(all_shards: bool = False) -> str:
+    """Incremental index maintenance (2.3): true up the LOCAL recall index and the
+    entity index against the vault, fetching only new / gone / hash-missing notes
+    plus today's rotating verification shard. Index-only — writes no notes; the
+    indexes are machine-owned, so no --apply gate. Returns a one-line summary."""
+    import datetime as dt
+    import echo_recall
+
+    if get("_agent/echo-vault.md") is None:
+        return "fast-sweep skipped: vault not bootstrapped"
+
+    listing = [p for p in walk()
+               if p.endswith(".md") and p.rsplit("/", 1)[-1] not in SKIP_BASENAMES
+               and not TEMPLATE_RE.search(p)]
+    lset = set(listing)
+
+    rix = echo_recall._load_local()
+    if rix is None or not rix.n_docs:
+        # Nothing to maintain incrementally — build the local index outright
+        # (sub-second on a few-hundred-note vault; no vault snapshot write).
+        rix = echo_recall.rebuild()
+        _stamp_fast_sweep()
+        return f"fast-sweep: no local index — built one ({rix.n_docs} docs)"
+
+    index = idx_mod.load()
+    ents = index.get("entities", {})
+    removed = 0
+    for slug in [s for s, e in list(ents.items()) if e.get("path") not in lset]:
+        del ents[slug]
+        removed += 1
+    for p in [p for p in list(rix.length) if p not in lset]:
+        rix.remove(p)
+        removed += 1
+
+    try:
+        weekday = dt.date.fromisoformat(echo.today()).weekday()
+    except ValueError:
+        weekday = dt.date.today().weekday()
+
+    def in_shard(p: str) -> bool:
+        return int(idx_mod.content_hash(p), 16) % 7 == weekday
+
+    cands: set[str] = set()
+    for p in listing:
+        if not echo_recall._indexable(p):
+            continue
+        if p not in rix.length:            # new note (any client, any tool)
+            cands.add(p)
+            continue
+        kind = kind_for(p)
+        if kind:
+            slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3])
+            if not (ents.get(slug) or {}).get("h"):   # pre-2.3 entry: hash it once
+                cands.add(p)
+                continue
+        if all_shards or in_shard(p):      # rotating verification shard
+            cands.add(p)
+
+    texts = echo.read_many(sorted(cands))
+    changed = 0
+    ents_dirty = removed > 0
+    for p, t in texts.items():
+        if t is None:
+            continue
+        h = idx_mod.content_hash(t)
+        if rix.content_hash(p) != h:
+            rix.add(p, t)
+            changed += 1
+        kind = kind_for(p)
+        if kind:
+            slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3])
+            prev = ents.get(slug) or {}
+            if prev.get("h") != h or prev.get("path") != p:
+                title = links.first_h1(t) or p.rsplit("/", 1)[-1][:-3]
+                aliases = list(prev.get("aliases", [])) + idx_mod.aliases_in_frontmatter(t)
+                idx_mod.upsert(index, slug, p, kind, title, aliases, h=h)
+                ents_dirty = True
+
+    echo_recall.save_local(rix)
+    if ents_dirty:
+        idx_mod.save(index)
+    _stamp_fast_sweep()
+    return (f"fast-sweep: {len(cands)} note(s) checked, {changed} reindexed, "
+            f"{removed} removed" + (" (all shards)" if all_shards else ""))
+
+
 def main(argv: list[str] | None = None) -> int:
     ap = argparse.ArgumentParser(description="Bring an ECHO vault up to the current feature spec")
     ap.add_argument("--apply", action="store_true")
+    ap.add_argument("--fast", action="store_true",
+                    help="incremental index maintenance (local recall + entity index only)")
+    ap.add_argument("--all-shards", action="store_true",
+                    help="with --fast: verify every indexed note, not just today's shard")
     args = ap.parse_args(argv)
+    if args.fast:
+        print(fast(all_shards=args.all_shards))
+        return 0
     apply = args.apply
     tag = "APPLY" if apply else "PLAN "
 
@@ -151,7 +279,8 @@ def main(argv: list[str] | None = None) -> int:
         # aliases (e.g. mentions learned by capture) are preserved; upsert adds title variants.
         prev_aliases = prev.get("aliases", []) if prev else []
         aliases = list(prev_aliases) + idx_mod.aliases_in_frontmatter(text)
-        idx_mod.upsert(rebuilt, slug, path, kind, title, aliases)
+        idx_mod.upsert(rebuilt, slug, path, kind, title, aliases,
+                       h=idx_mod.content_hash(text))
         if not prev:
             added += 1
         elif prev.get("path") != path or prev.get("title") != title:
@@ -163,8 +292,9 @@ def main(argv: list[str] | None = None) -> int:
 
     # ---- 2b. (re)build the recall (BM25) index -------------------------------
     if apply:
-        rix = echo_recall.rebuild(prefetched=texts)
-        print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25)")
+        rix = echo_recall.rebuild(prefetched=texts, snapshot=True)
+        _stamp_fast_sweep()
+        print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25; local + vault snapshot)")
     else:
         n_idx = sum(1 for p in all_files if echo_recall._indexable(p))
         print(f"sweep: {tag} recall index — would rebuild BM25 over {n_idx} corpus notes "

echo-memory.plugin.src/skills/echo-memory/scripts/test_echo_client.py

@@ -215,6 +215,42 @@ def test_links_create_related_section_when_absent() -> None:
     assert changed and "## Related" in new and "[[resources/concepts/beta]]" in new
 
 
+def test_stemmer_conflates_inflection_families() -> None:
+    import echo_stem
+    families = [
+        ["deploy", "deploys", "deployed", "deploying", "deployment"],
+        ["navigate", "navigation", "navigating", "navigational"],
+        ["renew", "renews", "renewing"],
+        ["prefer", "prefers", "preferring", "preferred"],
+        ["penalty", "penalties"],
+        ["saturate", "saturation", "saturated"],
+        ["frequency", "frequencies"],
+        ["expand", "expanded", "expanding"],
+        ["meet", "meeting", "meetings"],
+        ["service", "services"],
+    ]
+    for fam in families:
+        stems = {echo_stem.stem(w) for w in fam}
+        assert len(stems) == 1, f"{fam} -> {stems}"
+
+
+def test_stemmer_leaves_short_and_risky_tokens_alone() -> None:
+    import echo_stem
+    # over-stemming merges unrelated words — worse than a missed match
+    for w in ("sing", "thing", "was", "is", "bus", "this", "echo", "vault", "mpm"):
+        assert echo_stem.stem(w) == w, f"{w} mangled to {echo_stem.stem(w)}"
+    # documented non-conflations (the -al / d~s irregulars full Porter also skips)
+    assert echo_stem.stem("renewal") != echo_stem.stem("renew")
+    assert echo_stem.stem("expansion") != echo_stem.stem("expand")
+
+
+def test_content_hash_stable_and_short() -> None:
+    import echo_index as ix
+    a = ix.content_hash("hello world")
+    assert a == ix.content_hash("hello world") and len(a) == 16
+    assert a != ix.content_hash("hello world!")
+
+
 def _run_all() -> int:
     tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
     failed = 0

eval/results/latest.json

@@ -3,17 +3,17 @@
   "date": "2026-07-03",
   "retrieval": {
     "current (hybrid+priors, full corpus)": {
-      "precision_at_5": 0.25,
+      "precision_at_5": 0.243,
       "recall_at_5": 1.0,
       "mrr": 1.0,
-      "session_journal_queries_answered": "2/2",
+      "session_journal_queries_answered": "3/3",
       "freshness_top1_correct": true
     },
     "baseline (keyword, entities only)": {
-      "precision_at_5": 0.2,
+      "precision_at_5": 0.186,
       "recall_at_5": 0.75,
-      "mrr": 0.75,
-      "session_journal_queries_answered": "0/2",
+      "mrr": 0.786,
+      "session_journal_queries_answered": "0/3",
       "freshness_top1_correct": false
     }
   },

eval/run_eval.py

@@ -135,6 +135,16 @@ QUERIES = [
     ("contract renewal Q3", {"resources/companies/vantage-systems.md"}, False),
     ("renegotiation kickoff", {"journal/daily/2026-06-20.md"}, True),
     ("graph neighbourhood expansion", {"resources/concepts/graph-expansion.md"}, False),
+    # ---- paraphrase set (2.3): morphological variants + rephrasings the pre-2.3
+    # purely-lexical index missed — the stemming gain is measured, not assumed.
+    ("renewing enterprise contracts", {"resources/companies/vantage-systems.md"}, False),
+    ("navigational cleanup", {"projects/active/website-redesign.md",
+                              "projects/archived/old-website.md"}, False),
+    ("preferring uv for tooling", {"_agent/memory/semantic/tooling-preferences.md"}, False),
+    ("penalties in the SLA",
+     {"_agent/sessions/2026-06-20-1000-vantage-contract-review.md"}, True),
+    ("saturated term frequencies", {"resources/concepts/bm25-ranking.md"}, False),
+    ("expanded linked neighbourhoods", {"resources/concepts/graph-expansion.md"}, False),
 ]
 FRESHNESS_QUERY = "marketing site navigation"
 FRESHNESS_TOP1 = "projects/active/website-redesign.md"

eval/test_features.py

@@ -27,7 +27,9 @@ def check(name, cond, detail=""):
 
 class Harness:
     def __init__(self, base):
+        import tempfile
         self.base = base
+        self.state_dir = tempfile.mkdtemp()
 
     def http(self, method, url, body=None, headers=None):
         data = body.encode() if isinstance(body, str) else body
@@ -52,8 +54,10 @@ class Harness:
         return None if body == "<<MISSING>>" else body
 
     def echo(self, *args):
+        # ECHO_STATE_DIR isolation matters since 2.3: the recall index is LOCAL-first,
+        # and without this the suite would write into the real ~/.echo-memory.
         env = dict(os.environ, ECHO_BASE=self.base, ECHO_KEY=KEY, ECHO_VERIFY="1",
-                   ECHO_TODAY="2026-06-21")
+                   ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=self.state_dir)
         return subprocess.run([sys.executable, str(args[0]), *args[1:]],
                               capture_output=True, text=True, env=env)
 
@@ -230,8 +234,9 @@ def main():
         check("v1.5 recall finds a session log by body term",
               "_agent/sessions/2026-06-15-1200-mpm-review.md" in r.stdout, r.stdout)
         rix2 = h.ground("_agent/index/recall-index.json") or ""
-        check("v1.5 recall index carries doc meta (schema 2)",
-              '"schema": 2' in rix2 or '"schema":2' in rix2, rix2[:200])
+        check("v2.3 recall snapshot is schema 3 with build stamp + content hashes",
+              ('"schema": 3' in rix2 or '"schema":3' in rix2)
+              and '"built"' in rix2 and '"h"' in rix2, rix2[:200])
 
         # 14. recency-aware ranking: same term, fresher note wins.
         h.seed("resources/concepts/old-idea.md",
@@ -402,6 +407,59 @@ def main():
               r.returncode == 2 and sess_path in (h.ground("_agent/heartbeat/last-session.md") or ""),
               r.stderr)
 
+        # ---------------- v2.3 index train --------------------------------------
+        # 20. local-first recall index: capture updates the LOCAL index with ZERO
+        # vault snapshot writes — the vault copy only changes on sweep/session-end.
+        snap_before = h.ground("_agent/index/recall-index.json") or ""
+        r = h.echo(ECHO, "capture", "Quixotic Widget", "--kind", "concept", "-")
+        r = subprocess.run([sys.executable, str(ECHO), "capture", "Quixotic Widget2", "--kind",
+                            "concept", "-"], input="A gadget for quixotry deployment.\n",
+                           capture_output=True, text=True,
+                           env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY,
+                                    ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=h.state_dir))
+        snap_after = h.ground("_agent/index/recall-index.json") or ""
+        check("v2.3 capture does NOT write the vault snapshot", snap_before == snap_after)
+        local_files = [p for p in Path(h.state_dir).glob("recall-index-*.json")]
+        check("v2.3 capture maintains the LOCAL index", len(local_files) == 1,
+              str(local_files))
+        r = h.echo(ECHO, "recall", "quixotry")
+        check("v2.3 locally-indexed capture is recallable",
+              "resources/concepts/quixotic-widget2.md" in r.stdout, r.stdout[:400])
+
+        # 21. stemming: a morphologically different query still hits (deployment ~ deploying).
+        r = h.echo(ECHO, "recall", "deploying quixotry")
+        check("v2.3 stemmed recall bridges inflections",
+              "resources/concepts/quixotic-widget2.md" in r.stdout, r.stdout[:400])
+
+        # 22. alias query expansion: an alias-phrased query surfaces the entity even
+        # though the alias never appears in any note body.
+        r = subprocess.run([sys.executable, str(ECHO), "capture", "Kappa Engine", "--kind",
+                            "concept", "-", "--aliases", "turbokappa"],
+                           input="Compression tuning notes for the kappa engine core.\n",
+                           capture_output=True, text=True,
+                           env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY,
+                                    ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=h.state_dir))
+        r = h.echo(ECHO, "recall", "turbokappa compression")
+        check("v2.3 alias expansion surfaces the aliased entity",
+              "resources/concepts/kappa-engine.md" in r.stdout, r.stdout[:400])
+
+        # 23. incremental fast sweep: an out-of-band edit (seeded directly, as if by
+        # Obsidian) and a deletion are trued up without a full rebuild.
+        h.seed("resources/concepts/quixotic-widget.md",
+               "---\ntype: concept\nstatus: active\ncreated: 2026-06-21\nupdated: 2026-06-21\n"
+               "tags: [concept]\n---\n# Quixotic Widget\n\nNow mentions zanzibar explicitly.\n")
+        h.http("DELETE", f"{base}/vault/resources/concepts/quixotic-widget2.md")
+        r = h.echo(SWEEP, "--fast", "--all-shards")
+        check("v2.3 fast sweep reports its work",
+              "fast-sweep:" in r.stdout and "reindexed" in r.stdout, r.stdout + r.stderr)
+        r = h.echo(ECHO, "recall", "zanzibar")
+        check("v2.3 fast sweep picked up the out-of-band edit",
+              "resources/concepts/quixotic-widget.md" in r.stdout, r.stdout[:400])
+        idx5 = h.ground("_agent/index/entities.json") or ""
+        check("v2.3 fast sweep drops deleted notes from the entity index",
+              "quixotic-widget2" not in idx5)
+        check("v2.3 entity index carries content hashes", '"h"' in idx5, idx5[:300])
+
         print(f"\n{len(failures)} failure(s)" if failures else "\nall feature tests passed")
         return 1 if failures else 0
     finally:

eval/test_mcp_server.py

@@ -0,0 +1,171 @@
+#!/usr/bin/env python3
+"""test_mcp_server.py — echo-mcp (2.2) end-to-end against the mock vault.
+
+Spawns mcp-server/app.py + mock_olrapi and drives the real streamable-HTTP MCP
+protocol: health (open), auth (401), initialize, tools/list, and the memory-day
+tools/call flow (capture -> duplicate gate as DATA -> merge_into -> recall ->
+get_note -> log_session with heartbeat commit).
+
+Requires the `mcp` package in the interpreter that runs the SERVER. Set
+ECHO_MCP_PYTHON to a venv python that has it; otherwise the current interpreter
+is tried and the suite SKIPS (exit 0) when the SDK is absent — the other suites
+don't depend on it.
+
+Run:  python test_mcp_server.py [--port 8862] [--mcp-port 8767]
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+import tempfile
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+APP = HERE.parent / "mcp-server" / "app.py"
+KEY = "test-key-not-a-real-secret"
+TOKEN = "local-test-token"
+
+failures = []
+
+
+def check(name, cond, detail=""):
+    print(f"{'ok  ' if cond else 'FAIL'} {name}" + (f"  -- {detail}" if not cond else ""))
+    if not cond:
+        failures.append(name)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--port", type=int, default=8862)
+    ap.add_argument("--mcp-port", type=int, default=8767)
+    a = ap.parse_args()
+    mock_base = f"http://127.0.0.1:{a.port}"
+    mcp_url = f"http://127.0.0.1:{a.mcp_port}/mcp"
+
+    server_py = os.environ.get("ECHO_MCP_PYTHON") or sys.executable
+    probe = subprocess.run([server_py, "-c", "import mcp"], capture_output=True)
+    if probe.returncode != 0:
+        print("SKIP: `mcp` SDK not installed for the server interpreter "
+              "(set ECHO_MCP_PYTHON to a venv python that has it)")
+        return 0
+
+    def http(method, url, body=None, headers=None, timeout=30):
+        data = body.encode() if isinstance(body, str) else body
+        req = urllib.request.Request(url, data=data, method=method, headers=headers or {})
+        try:
+            with urllib.request.urlopen(req, timeout=timeout) as r:
+                return r.status, r.read().decode("utf-8", "replace")
+        except urllib.error.HTTPError as e:
+            return e.code, e.read().decode("utf-8", "replace")
+        except Exception as e:  # noqa: BLE001
+            return 0, str(e)
+
+    _id = [0]
+
+    def rpc(method, params):
+        _id[0] += 1
+        st, body = http("POST", mcp_url, json.dumps(
+            {"jsonrpc": "2.0", "id": _id[0], "method": method, "params": params}),
+            headers={"Content-Type": "application/json",
+                     "Accept": "application/json, text/event-stream",
+                     "Authorization": f"Bearer {TOKEN}"})
+        return st, (json.loads(body) if body.strip().startswith("{") else {})
+
+    def call(name, args):
+        st, d = rpc("tools/call", {"name": name, "arguments": args})
+        res = d.get("result") or {}
+        payload = {}
+        if res.get("content"):
+            try:
+                payload = json.loads(res["content"][0]["text"])
+            except Exception:  # noqa: BLE001
+                payload = {}
+        return res.get("isError", False), payload
+
+    mock = subprocess.Popen([sys.executable, str(HERE / "mock_olrapi.py"), "--port", str(a.port)],
+                            stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+    srv = None
+    try:
+        for _ in range(50):
+            try:
+                urllib.request.urlopen(f"{mock_base}/__debug__reset", data=b"", timeout=1)
+                break
+            except Exception:
+                time.sleep(0.1)
+        http("PUT", f"{mock_base}/vault/_agent/echo-vault.md",
+             "---\nschema_version: 4\n---\n# marker\n",
+             headers={"Authorization": f"Bearer {KEY}"})
+
+        env = dict(os.environ, ECHO_BASE=mock_base, ECHO_KEY=KEY, ECHO_MCP_TOKEN=TOKEN,
+                   ECHO_MCP_PORT=str(a.mcp_port), ECHO_STATE_DIR=tempfile.mkdtemp(),
+                   ECHO_TODAY="2026-07-28")
+        srv = subprocess.Popen([server_py, str(APP)], env=env,
+                               stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+        for _ in range(60):
+            st, _b = http("GET", f"http://127.0.0.1:{a.mcp_port}/health", timeout=2)
+            if st == 200:
+                break
+            time.sleep(0.5)
+
+        st, body = http("GET", f"http://127.0.0.1:{a.mcp_port}/health")
+        try:
+            hb = json.loads(body)
+        except Exception:  # noqa: BLE001
+            hb = {}
+        check("health is open and green", st == 200 and hb.get("ok") is True
+              and hb.get("vault_reachable") is True, body[:200])
+        st, _b = http("POST", mcp_url, "{}", headers={"Content-Type": "application/json"})
+        check("MCP endpoint requires the bearer token (401)", st == 401, str(st))
+
+        st, d = rpc("initialize", {"protocolVersion": "2025-06-18", "capabilities": {},
+                                   "clientInfo": {"name": "eval", "version": "0"}})
+        check("initialize answers with serverInfo",
+              d.get("result", {}).get("serverInfo", {}).get("name") == "echo-mcp", json.dumps(d)[:200])
+        st, d = rpc("tools/list", {})
+        tools = [t["name"] for t in d.get("result", {}).get("tools", [])]
+        check("tools/list exposes the full profile (14 tools)", len(tools) == 14, str(tools))
+
+        e, b = call("echo_capture", {"title": "Vera Lumen", "kind": "person",
+                                     "body": "CTO at Fluxcorp, met at the summit."})
+        check("capture create over MCP", not e and b.get("action") == "created"
+              and b.get("path") == "resources/people/vera-lumen.md", json.dumps(b))
+        e, b = call("echo_capture", {"title": "Vera Lumen Jr", "kind": "person"})
+        check("duplicate gate is data, not isError", not e
+              and b.get("action") == "duplicate-gate" and b.get("candidates"), json.dumps(b))
+        e, b = call("echo_capture", {"title": "Vera Lumen Jr", "kind": "person",
+                                     "merge_into": "vera-lumen", "body": "Follow-up."})
+        check("merge_into resolves the gate as an update", not e and b.get("action") == "updated",
+              json.dumps(b))
+        e, b = call("echo_recall", {"query": "fluxcorp", "budget_chars": 900})
+        check("recall over MCP finds by body term", not e and any(
+            h["path"] == "resources/people/vera-lumen.md" for h in b.get("primary", [])),
+            json.dumps(b)[:300])
+        e, b = call("echo_get_note", {"path": "resources/people/vera-lumen.md"})
+        check("get_note returns frontmatter + content", not e
+              and b.get("frontmatter", {}).get("type") == "person", json.dumps(b)[:200])
+        e, b = call("echo_get_note", {"path": "../etc/passwd"})
+        check("get_note rejects path traversal", not b.get("ok"))
+        e, b = call("echo_log_session", {"slug": "mcp-e2e", "hhmm": "2345", "apply": True,
+                                         "log_body": "---\ntype: session-log\n---\n# S\n\n## Goal\ne2e\n"})
+        check("log_session commits with the heartbeat", not e
+              and b.get("steps", {}).get("heartbeat") == "ok", json.dumps(b))
+        e, b = call("echo_capture", {"title": "Bad Kind", "kind": "wizard"})
+        check("unknown kind rejected with the valid list", not b.get("ok")
+              and "wizard" in b.get("error", ""), json.dumps(b))
+
+        print(f"\n{len(failures)} failure(s)" if failures else "\nall mcp-server tests passed")
+        return 1 if failures else 0
+    finally:
+        if srv:
+            srv.terminate()
+        mock.terminate()
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

eval/test_patch_semantics.py

@@ -59,7 +59,9 @@ def main():
                            stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
 
     def echo(*args):
-        env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="0")
+        import tempfile
+        env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="0",
+                   ECHO_STATE_DIR=globals().setdefault("_STATE_DIR", tempfile.mkdtemp()))
         return subprocess.run([sys.executable, str(ECHO), *args], capture_output=True, text=True, env=env)
 
     def ground(path):

eval/test_reflect.py

@@ -51,7 +51,9 @@ def main():
             return getattr(e, "code", 0), ""
 
     def echo(*args, stdin=None):
-        env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1", ECHO_TODAY="2026-06-22")
+        import tempfile
+        env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1",
+                   ECHO_TODAY="2026-06-22", ECHO_STATE_DIR=tempfile.mkdtemp())
         return subprocess.run([sys.executable, str(ECHO), *args], input=stdin,
                               capture_output=True, text=True, env=env)
 

mcp-server/app.py

@@ -0,0 +1,417 @@
+#!/usr/bin/env python3
+"""echo-mcp — the containerized MCP server over the ECHO ops layer.  [2.2]
+
+Streamable-HTTP (stateless JSON) MCP server exposing the plugin's `*_op` cores
+(Phase 0, shipped 2.1.1) as typed tools. Runs next to the Obsidian REST API on
+the same box, so every vault round-trip behind a tool call is LAN-local.
+
+Spec: docs/MCP-SERVER-SPEC.md. Highlights implemented here:
+  * bearer-token auth on everything except /health (constant-time compare);
+  * tool results = the op envelopes, returned as structured JSON;
+  * duplicate gate / offline queueing are DATA (never protocol errors), so the
+    model deliberately chooses merge_into/force instead of blind-retrying;
+  * tool profiles: ECHO_MCP_TOOLS=core exposes only the six daily drivers
+    (schema tokens cost context on every surface that lists them);
+  * result budgets: echo_recall(budget_chars), echo_get_note(section/max_chars);
+  * per-write serialization (one process mediates all MCP writes, so a plain
+    lock removes the advisory-lock race window for server-mediated writes);
+  * startup validation: fail fast (crash-loop visibly) on missing env.
+
+Deliberately NOT here in v1 (spec §7.2 backlog): entity-index TTL cache — the
+atomic_index_update correctness fix depends on a FRESH re-read under the lock,
+so caching idx_mod.load would reintroduce the clobber race it closed. LAN
+adjacency + the warm keep-alive pool make the read ~ms anyway.
+
+Env: ECHO_BASE, ECHO_KEY, ECHO_MCP_TOKEN (required) · ECHO_OWNER ·
+     ECHO_MCP_PORT=8765 · ECHO_MCP_TOOLS=full|core · ECHO_STATE_DIR=/data
+"""
+from __future__ import annotations
+
+import contextlib
+import hmac
+import json
+import os
+import re
+import subprocess
+import sys
+import threading
+from pathlib import Path
+
+# --- locate the plugin scripts (image layout first, repo layout for local dev) ---
+_HERE = Path(__file__).resolve().parent
+for _cand in (_HERE / "scripts",
+              _HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts"):
+    if _cand.is_dir():
+        SCRIPTS = _cand
+        break
+else:  # pragma: no cover
+    sys.exit("echo-mcp: cannot locate the ECHO scripts directory")
+sys.path.insert(0, str(SCRIPTS))
+
+# --- startup validation: a misdeployed container must crash-loop visibly ---------
+_missing = [k for k in ("ECHO_BASE", "ECHO_KEY", "ECHO_MCP_TOKEN") if not os.environ.get(k)]
+if _missing:
+    sys.exit(f"echo-mcp: missing required env: {', '.join(_missing)} "
+             "— check the PORT template / secret refs")
+
+# Modules aliased *_mod: several tool functions below share a module's name
+# (echo_recall, echo_reflect, ...) and would shadow it at module scope otherwise.
+import echo                        # noqa: E402
+import echo_doctor as doctor_mod   # noqa: E402
+import echo_index as idx_mod       # noqa: E402
+import echo_ops as ops_mod         # noqa: E402
+import echo_queue as queue_mod     # noqa: E402
+import echo_recall as recall_mod   # noqa: E402
+import echo_reflect as reflect_mod # noqa: E402
+import echo_session as session_mod # noqa: E402
+import echo_triage as triage_mod   # noqa: E402
+
+from mcp.server.fastmcp import FastMCP  # noqa: E402
+from starlette.requests import Request  # noqa: E402
+from starlette.responses import JSONResponse, Response  # noqa: E402
+from starlette.middleware.base import BaseHTTPMiddleware  # noqa: E402
+
+PORT = int(os.environ.get("ECHO_MCP_PORT", "8765"))
+TOKEN = os.environ["ECHO_MCP_TOKEN"]
+PROFILE = os.environ.get("ECHO_MCP_TOOLS", "full").strip().lower()
+KINDS = sorted(idx_mod.KIND_FOLDER)
+
+mcp = FastMCP(
+    "echo-mcp",
+    instructions=(
+        "Persistent memory over the operator's ECHO Obsidian vault. Use echo_load at "
+        "the start of substantive sessions, echo_recall/echo_resolve to read, "
+        "echo_capture as the default write, and echo_log_session to end a session. "
+        "A duplicate-gate result is a decision point, not an error — merge_into or "
+        "(confirmed-distinct) force. Write about the operator in third person."),
+    host="0.0.0.0",
+    port=PORT,
+    stateless_http=True,
+    json_response=True,
+)
+
+WRITE_LOCK = threading.Lock()   # all MCP-path writes serialize through this process
+
+
+def _guard(write: bool, fn, *args, **kw) -> dict:
+    """Run an op core with stdout redirected to stderr (belt-and-braces stream
+    purity) and EchoError mapped to an actionable error envelope, not a protocol
+    error. Write ops serialize on WRITE_LOCK."""
+    lock = WRITE_LOCK if write else contextlib.nullcontext()
+    try:
+        with lock, contextlib.redirect_stdout(sys.stderr):
+            return fn(*args, **kw)
+    except RuntimeError as exc:  # EchoError (either module instance) subclasses it
+        code = getattr(exc, "code", 1)
+        err = {"ok": False, "error": str(exc), "code": code}
+        if code == 78:
+            err["hint"] = ("server deployment is missing vault credentials — operator: "
+                           "check the PORT template env (SECRET: refs)")
+        elif code == 44:
+            err["code_name"] = "not-found"
+        elif "unreachable" in str(exc):
+            err["hint"] = ("vault unreachable (Obsidian/REST plugin likely down) — "
+                           "proceed without memory; writes queue durably")
+        return err
+
+
+_SAFE_PATH = re.compile(r"^[^/][^\0]*$")
+
+
+def _check_path(path: str) -> str | None:
+    if not path or path.startswith("/") or ".." in path.split("/") or not _SAFE_PATH.match(path):
+        return "invalid path: must be vault-relative, no leading '/' and no '..'"
+    return None
+
+
+def _frontmatter(text: str) -> dict:
+    out: dict = {}
+    if text.startswith("---"):
+        end = text.find("\n---", 3)
+        for ln in text[3:end if end != -1 else len(text)].splitlines():
+            m = re.match(r"^([A-Za-z_][\w-]*):\s*(.*)$", ln)
+            if m:
+                out[m.group(1)] = m.group(2).strip().strip('"').strip("'")
+    return out
+
+
+# ------------------------------------------------------------------ tools -------
+CORE_TOOLS = {"echo_load", "echo_recall", "echo_capture", "echo_triage_inbox",
+              "echo_log_session", "echo_health"}
+
+
+def tool(fn):
+    """Register `fn` as an MCP tool unless the core profile excludes it."""
+    if PROFILE == "core" and fn.__name__ not in CORE_TOOLS:
+        return fn
+    return mcp.tool()(fn)
+
+
+@tool
+def echo_load(brief: bool = True) -> dict:
+    """Cold-start memory orientation — call at the start of a substantive session.
+    brief=true (default) returns a token-budgeted digest plus structured sections;
+    brief=false returns the six raw sections. Also flushes writes queued offline."""
+    return _guard(True, echo.load_op, brief)
+
+
+@tool
+def echo_recall(query: str, limit: int = 6, budget_chars: int = 4000,
+                include_linked: bool = True) -> dict:
+    """Search memory: ranked matches for a topic/person/project PLUS their linked
+    neighbourhood. Excerpts are packed into budget_chars by score — call
+    echo_get_note for any hit's full content. Freshness/status-aware ranking."""
+    limit = max(1, min(int(limit), 20))
+    budget = max(500, min(int(budget_chars), 20000))
+    env = _guard(False, recall_mod.recall_op, query, limit)
+    if not env.get("ok"):
+        return env
+    if not include_linked:
+        env["linked"] = []
+    used = 0
+    for hit in (env.get("primary") or []) + (env.get("linked") or []):
+        ex = hit.get("excerpt") or ""
+        room = max(0, budget - used)
+        if len(ex) > room:
+            hit["excerpt"] = ex[:room] + ("… (truncated — echo_get_note for full)" if room else "")
+            hit["truncated"] = True
+        used += len(hit.get("excerpt") or "")
+    return env
+
+
+@tool
+def echo_resolve(mention: str) -> dict:
+    """Resolve a name/mention to its canonical vault note (alias-aware), or get
+    did-you-mean candidates. Call before creating any note by hand; echo_capture
+    does this automatically."""
+    return _guard(False, ops_mod.resolve_op, mention)
+
+
+@tool
+def echo_get_note(path: str, section: str = "", max_chars: int = 8000) -> dict:
+    """Read one vault note (vault-relative path). section='Status' returns only that
+    ## section. Content over max_chars is truncated with a marker."""
+    bad = _check_path(path)
+    if bad:
+        return {"ok": False, "error": bad}
+
+    def _read() -> dict:
+        status, body = echo.request("GET", echo.vault_url(path))
+        if status == 404:
+            return {"ok": False, "code_name": "not-found", "path": path,
+                    "error": f"{path}: not found"}
+        echo.check(status, body, f"get {path}")
+        text = body.decode(errors="replace")
+        fm = _frontmatter(text)
+        if section:
+            text = echo.extract_heading(text, section)
+            if not text:
+                return {"ok": False, "path": path,
+                        "error": f"no '## {section}' section in {path}"}
+        truncated = len(text) > max_chars
+        return {"ok": True, "path": path, "frontmatter": fm,
+                "content": text[:max_chars] + ("\n… (truncated)" if truncated else ""),
+                "truncated": truncated}
+    return _guard(False, _read)
+
+
+@tool
+def echo_get_scope() -> dict:
+    """The operator's active scope + freshness. If sessions_since >= 3, treat the
+    recorded scope as suspect and confirm with the operator before working."""
+    return _guard(False, echo.scope_show_op)
+
+
+@tool
+def echo_set_scope(scope: str) -> dict:
+    """Switch the active scope atomically (archives the prior scope to Scope History
+    and stamps freshness). Use when the session's work diverges from the recorded
+    scope."""
+    return _guard(True, echo.scope_set_op, scope)
+
+
+@tool
+def echo_health(deep: bool = False) -> dict:
+    """ECHO readiness: config, vault reachability, auth, bootstrap/schema, and the
+    offline-queue depth. deep=true also runs the full vault-invariant linter."""
+    env = _guard(False, doctor_mod.run_op)
+    try:
+        env["outbox_depth"] = len(queue_mod.pending())
+        env["needs_attention"] = len(queue_mod.needs_attention())
+    except Exception:  # noqa: BLE001
+        pass
+    if deep and env.get("ok"):
+        r = subprocess.run([sys.executable, str(SCRIPTS / "vault_lint.py")],
+                           capture_output=True, text=True,
+                           env=dict(os.environ), timeout=120)
+        env["lint_exit"] = r.returncode
+        env["lint_report"] = (r.stdout or r.stderr)[-6000:]
+    return env
+
+
+@tool
+def echo_capture(title: str, kind: str = "", body: str = "", tags: list[str] = [],
+                 aliases: list[str] = [], sources: list[str] = [], status: str = "",
+                 date: str = "", domain: str = "business", merge_into: str = "",
+                 force: bool = False, dry_run: bool = False) -> dict:
+    """The default memory write: routes by kind (person, company, concept, reference,
+    meeting, project, area, semantic, episodic, working, skill, decision), stamps
+    complete frontmatter, indexes, auto-links, and writes the Agent-Log line — one
+    call. Omit kind for an inbox capture. If the result is action='duplicate-gate',
+    do NOT retry blindly: call again with merge_into=<candidate slug> to update the
+    existing entity, or force=true only after confirming they are genuinely distinct.
+    Offline, the whole capture queues durably (queued=true)."""
+    if kind and kind not in KINDS:
+        return {"ok": False, "error": f"unknown kind '{kind}' — one of: {', '.join(KINDS)}"}
+    return _guard(True, ops_mod.capture_op, kind or None, title,
+                  body_text=body or "", status_v=status, aliases=aliases,
+                  sources=sources, tags=tags, date=date or None, domain=domain,
+                  inbox=not kind, dry_run=dry_run, force=force,
+                  merge_into=merge_into or None)
+
+
+@tool
+def echo_link(a: str, b: str) -> dict:
+    """Add reciprocal '## Related' links between two notes. Accepts vault paths or
+    resolvable entity names (aliases work)."""
+    def _to_path(x: str) -> str | None:
+        if "/" in x:
+            return x if x.endswith(".md") else x + ".md"
+        nmap = idx_mod.name_map(idx_mod.load())
+        return nmap.get(idx_mod.slugify(x))
+    def _do() -> dict:
+        pa, pb = _to_path(a), _to_path(b)
+        if not pa or not pb:
+            missing = a if not pa else b
+            return {"ok": False, "error": f"'{missing}' resolves to no note — "
+                                          "pass a vault path or a known entity name"}
+        return ops_mod.link_op(pa, pb)
+    return _guard(True, _do)
+
+
+@tool
+def echo_append_note(path: str, line: str) -> dict:
+    """Append one line to a note, idempotently (skipped if the exact line already
+    exists). For inbox lines, Agent-Log entries, Observations bullets."""
+    bad = _check_path(path)
+    if bad:
+        return {"ok": False, "error": bad}
+    def _do() -> dict:
+        rc = echo.cmd_append(path, line)
+        return {"ok": rc == 0, "action": "append", "path": path}
+    return _guard(True, _do)
+
+
+@tool
+def echo_patch_note(path: str, operation: str, target_type: str, target: str,
+                    content: str) -> dict:
+    """Targeted edit: append/prepend/replace under a heading, frontmatter field, or
+    block. HEADING TARGETS must be the full '::'-delimited path from the top-level
+    heading (e.g. 'Operator Preferences::Fact / Pattern') — on an invalid target the
+    error includes the note's actual headings. replace overwrites the section."""
+    bad = _check_path(path)
+    if bad:
+        return {"ok": False, "error": bad}
+    def _do() -> dict:
+        rc = echo.cmd_patch(path, operation, target_type, target,
+                            echo.temp_file(content.encode("utf-8")))
+        return {"ok": rc == 0, "action": f"patch:{operation}", "path": path,
+                "target": target}
+    env = _guard(True, _do)
+    if not env.get("ok") and "HTTP 400" in str(env.get("error", "")) and target_type == "heading":
+        st, body = echo.request("GET", echo.vault_url(path),
+                                headers={"Accept": "application/vnd.olrapi.document-map+json"})
+        if st == 200:
+            try:
+                headings = [h.get("heading") or h for h in
+                            json.loads(body).get("headings", [])][:40]
+                env["available_headings"] = headings
+                env["hint"] = "use one of available_headings verbatim as the Target"
+            except Exception:  # noqa: BLE001
+                pass
+    return env
+
+
+@tool
+def echo_triage_inbox(proposals: list[dict] = [], apply: bool = False) -> dict:
+    """Inbox triage. No proposals => structured listing of captures (line, date, text,
+    age_days). With proposals (reflect schema + optional 'line'): apply=false previews
+    the routing; apply=true routes via capture and writes the processing-log audit.
+    List first, propose, preview, then apply only after the operator confirms."""
+    if not proposals:
+        return _guard(False, triage_mod.list_op)
+    return _guard(True, triage_mod.route_op, proposals, apply)
+
+
+@tool
+def echo_reflect(proposals: list[dict], apply: bool = False) -> dict:
+    """Session-reflection proposals: validate -> classify against the entity index ->
+    preview (apply=false) -> apply (routes each via capture). Never apply without the
+    operator's go-ahead; never invent memories to have something to save."""
+    return _guard(True, reflect_mod.apply_op, proposals, apply)
+
+
+@tool
+def echo_log_session(slug: str, log_body: str, agent_log_line: str = "",
+                     scope: str = "", reflect: list[dict] = [],
+                     apply: bool = False, hhmm: str = "") -> dict:
+    """End a substantive session in ONE call: session log -> Agent-Log line -> reflect
+    proposals -> optional scope switch -> heartbeat LAST (the commit marker).
+    apply=false previews the plan. Pass hhmm (local time, e.g. '1430') so the log
+    filename sorts truthfully."""
+    bundle = {"slug": slug, "log_body": log_body}
+    if agent_log_line:
+        bundle["agent_log_line"] = agent_log_line
+    if scope:
+        bundle["scope"] = scope
+    if reflect:
+        bundle["reflect"] = reflect
+    def _do() -> dict:
+        prev = os.environ.get("ECHO_NOW")
+        try:
+            if hhmm:
+                os.environ["ECHO_NOW"] = hhmm
+            return session_mod.session_end_op(bundle, apply=apply)
+        finally:
+            if hhmm:
+                if prev is None:
+                    os.environ.pop("ECHO_NOW", None)
+                else:
+                    os.environ["ECHO_NOW"] = prev
+    return _guard(True, _do)
+
+
+# ------------------------------------------------------------- health + auth ----
+@mcp.custom_route("/health", methods=["GET"])
+async def health(_request: Request) -> JSONResponse:
+    """Unauthenticated liveness for Docker HEALTHCHECK + Kuma. No vault data."""
+    st, _ = echo.request("GET", echo.vault_url("_agent/echo-vault.md"))
+    try:
+        outbox = len(queue_mod.pending())
+    except Exception:  # noqa: BLE001
+        outbox = -1
+    return JSONResponse({"ok": True, "vault_reachable": st != 0,
+                         "outbox_depth": outbox, "profile": PROFILE})
+
+
+class BearerAuth(BaseHTTPMiddleware):
+    async def dispatch(self, request, call_next):
+        if request.url.path == "/health":
+            return await call_next(request)
+        auth = request.headers.get("authorization", "")
+        if not hmac.compare_digest(auth, f"Bearer {TOKEN}"):
+            return Response(status_code=401)
+        return await call_next(request)
+
+
+def main() -> None:
+    import uvicorn
+    app = mcp.streamable_http_app()
+    app.add_middleware(BearerAuth)
+    print(f"echo-mcp: serving on :{PORT} (profile={PROFILE}, vault={echo.BASE})",
+          file=sys.stderr)
+    uvicorn.run(app, host="0.0.0.0", port=PORT, log_level="info")
+
+
+if __name__ == "__main__":
+    main()

mcp-server/requirements.txt

@@ -0,0 +1,4 @@
+# echo-mcp server deps. The PLUGIN stays pure-stdlib; the dependency budget for the
+# container is deliberately tiny: the official MCP SDK (brings starlette/uvicorn/
+# pydantic) and nothing else.
+mcp>=1.9,<2