12 Commits

Author SHA1 Message Date
Jason Stedwell b4e923c2e7 2.3.0 — the index train: local-first recall index, incremental sweep, stemming + alias expansion
Build and Push Docker Image / build (push) Successful in 14s
One schema-bump release (recall-index schema 3, entity-index schema 2; old
recall indexes rebuild automatically, no vault migration):

- Local-first recall index: the live BM25 index lives in the machine state dir
  (keyed by endpoint hash); capture's upkeep is a zero-network atomic file
  write, no advisory lock — replacing the O(vault) GET/PUT-whole-index round
  trip per write. The vault copy is a snapshot (sweep + session-end) used to
  seed fresh machines / catch up after another client swept
  (ECHO_RECALL_SYNC_HOURS, default 24). The read path never PUTs the vault.
- Incremental sweep: entity + recall meta carry 16-char content hashes;
  `sweep --fast` fetches only new/gone/hash-missing notes plus a rotating
  weekday shard (--all-shards forces everything); deletions drop from both
  indexes; index-only so no --apply gate. `load` auto-runs it past
  ECHO_FAST_SWEEP_DAYS (7); doctor reports index freshness. Obsidian-side
  edits now reach the indexes without manual maintenance.
- Stemming + alias expansion: echo_stem.py (conservative Porter-lite, unit-
  tested families + over-stemming guards) applied at index AND query time;
  a query fuzzy-matching an entity folds its title/alias vocabulary into the
  BM25 query at half weight — expansion can only boost docs containing the
  terms. capture hashes the note's FINAL content (auto-link reordered first).

Eval gold set 8 -> 14 queries (6 paraphrases): recall@5/MRR 1.00/1.00 vs
keyword baseline 0.75/0.79. +3 unit tests, +10 e2e; test harnesses now isolate
ECHO_STATE_DIR. All seven suites green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 00:08:48 -05:00
Jason Stedwell 882d21dd96 Spec §9: claude.ai custom connectors require OAuth (DCR) — bearer-only blocks them
Build and Push Docker Image / build (push) Successful in 7s
Found at 2.2.0 deploy: the Connect flow attempts Dynamic Client Registration and
offers no custom-header option. Minimal single-user OAuth layer queued as 2.2.x
backlog next to the nightly vault backup. Claude Code / desktop / CoWork header
auth unaffected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 23:32:44 -05:00
Jason Stedwell 227e26c8db 2.2.0 — echo-mcp: the containerized MCP server
Build and Push Docker Image / build (push) Successful in 23s
ECHO as 14 typed MCP tools (streamable HTTP, stateless JSON, bearer auth, open
/health) wrapping the 2.1.1 *_op cores in-process:

- mcp-server/app.py: FastMCP app; duplicate gate + offline queueing surface as
  DATA (merge_into/force are parameters, never blind retries); recall packs
  excerpts into budget_chars by score; get_note is traversal-guarded with
  section/max_chars; patch_note enriches invalid-target errors with the note's
  actual headings; log_session wraps the session-end bundle (heartbeat-last);
  ECHO_MCP_TOOLS=core exposes only the six daily drivers; all MCP writes
  serialize in-process; startup fails fast on missing env.
- Dockerfile (legacy format — no BuildKit on the CI runner), python:3.12-slim,
  healthcheck probes 127.0.0.1; .dockerignore keeps the context lean.
- .gitea/workflows/docker-build.yml: standard image build; PORT redeploy
  trigger targets the echo-mcp container explicitly (repo name is echo).
- deploy.unraid.yml: br0/auto-IP, /data volume, vault adjacency
  (ECHO_BASE=http://10.2.0.35:27123), secrets as SECRET: refs.
- SKILL.md: prefer the echo_* tools when the connector is present; CLI recipes
  stay as the fallback. Spec header marked BUILT (tool count corrected to 14).
- eval/test_mcp_server.py: e2e over real streamable HTTP (health/auth/
  initialize/tools-list + capture->gate->merge->recall->log_session); skips
  cleanly when the SDK is absent. All seven suites green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 23:01:47 -05:00
Jason Stedwell e7792003a7 2.1.1 — MCP Phase 0: every high-level op returns an envelope (return-not-print)
Internal refactor per docs/MCP-SERVER-SPEC.md §4; CLI output and exit codes
unchanged (wrappers reproduce them from the envelopes):

- echo_ops: capture_op/resolve_op/link_op — duplicate gate and offline queueing
  are data (action: duplicate-gate + candidates; queued: true); the --json
  stdout-swallow hack is deleted (helper chatter -> stderr inside the cores);
  capture_op takes body_text= directly (the MCP path, no temp file).
- echo_recall.recall_op — one structured result backs prose and --json.
- echo_triage.list_op/route_op, echo_reflect.apply_op,
  echo_session.session_end_op — call capture_op internally, count gate outcomes
  from envelopes.
- echo.scope_show_op/scope_set_op/load_op (+ shared _load_gather), and
  echo_doctor.run_op (checks as data).

New eval/test_ops_api.py asserts the contract for every core: envelope shape
AND stdout purity (a stray print would corrupt an MCP response stream). All six
suites green. The 2.2 server build now starts directly at the app layer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 22:51:33 -05:00
Jason Stedwell 446cd9a0ff 2.1.0 — quick-wins train: brief load, offline-durable capture, session-end verb
Three independently useful changes (IMPROVEMENT-PLANS #1/#3/#7), no schema changes:

- load --brief: token-budgeted cold-start digest (facts full, last-10 observations,
  scope+freshness, last session's key sections, agent-log lines, inbox count;
  ECHO_LOAD_BUDGET ~8000 chars, lowest-priority-first trimming). SessionStart hook
  injects the brief form; /echo-load keeps the full dump. All load reads (+ the
  sessions listing, which also feeds the heartbeat fallback) fetched in parallel.
- Offline capture durability: a fully-offline capture queues the WHOLE op as one
  semantic record; flush replays it through capture so routing/gate/aliasing re-run
  against the current index; a gate stop on replay is kept + flagged
  (needs_attention, surfaced by flush and load), never landed blind; update-path
  and ensure_daily_log writes ride safe_request; same-args captures dedupe.
- session-end: one call, one lock — session log -> agent-log line -> reflect
  proposals (gate-aware) -> optional scope set -> heartbeat LAST as the commit
  marker; dry-run default; ECHO_NOW pins HHMM; validation runs before any write.
  Stop hook nudge names the command and recognizes it as reflected.
- Fix: main() now catches the __main__ twin-module EchoError (via RuntimeError +
  .code) so helper-module errors exit with their intended codes, not tracebacks.

+19 e2e checks; all suites green. Plans renumbered: MCP container is 2.2.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 22:34:42 -05:00
Jason Stedwell 1deef7299e ROADMAP-2.0: skills-only install re-verified on desktop + CoWork — roadmap complete
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 22:20:47 -05:00
Jason Stedwell 5aace9e430 2.0.0 — packaging & structure major: skills-only plugin, release-based artifacts
Memory behavior unchanged; the major signals a distribution/install break.

- Legacy commands/ directory DELETED (skills-only; the 1.6.0 skills were verified
  on desktop + CoWork and already took precedence). Pre-skills clients install
  1.6.0 from the release history.
- Artifact policy: the repo tracks only the echo-memory.plugin pointer; the 15
  historical versioned zips leave the tree (git history retains them); versioned
  builds ship as Gitea release assets, one release per v<version> tag, starting
  v2.0.0. .gitignore: *.plugin except the pointer.
- README: post-2.0 repository layout; "packaging for other agent runtimes" note
  (ports are build targets from the canonical tree, never a second source tree).
- ROADMAP-2.0 items ticked; manifest -> 2.0.0.

All suites green (25/25 unit + 4 e2e suites); artifact rebuilt skills-only
(79 entries, no commands/).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 22:12:26 -05:00
Jason Stedwell ab1e514f78 TODO-1.6: install verification passed on desktop + CoWork — commands/ deletion unblocked for 2.0
All four checks pass on the baked 1.6.0 artifact (skills register + precedence,
allowed-tools, operator-only sweep/triage, CoWork $ECHO fallback). Caveat noted:
allowed-tools globs match literal script names, not the "$ECHO" variable form —
re-confirm on the Claude Code CLI if that surface becomes routine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 22:08:07 -05:00
Jason Stedwell fb407d606c Add the 1.6.0 lint regression test (missed in the release commit staging)
Seeds archive/notes/README.md and asserts vault_lint flags retired-path —
fails against the pre-1.6.0 linter, passes with the retired-first check.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 21:59:39 -05:00
Jason Stedwell 55afdce70c 1.6.0 — skills-format migration, lint retired-path fix, plugin.json completeness
Skills migration: each of the eight slash commands gains a skills/<name>/SKILL.md
twin (same invocation name, byte-identical body). Per-skill allowed-tools
pre-authorize the resolved python3/python/py -3 invocations + the CoWork ls
fallback probe; disable-model-invocation: true makes /echo-sweep and /echo-triage
operator-only. Docs-confirmed: a same-name skill takes precedence over the legacy
command, so commands/ coexists until its scheduled 2.0 deletion. The $ECHO
path-resolution dedupe is resolved as not-supported (skill dirs are
self-contained; no cross-skill snippet sharing).

Lint fix: vault_lint.py checks retired patterns BEFORE routes, so a seed README
inside a retired tree (archive/...) flags retired-path instead of being absolved
by the permissive leaf-readme route. New end-to-end regression test seeds
archive/notes/README.md (fails pre-fix, passes now).

plugin.json: version 1.6.0, homepage + repository -> git.alwisp.com/jason/echo.

All suites green: 25/25 client unit tests (incl. routing-sync guard), feature,
reflect, offline-queue, PATCH-semantics. Artifact rebuilt (87 entries, both
formats); pointer refreshed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 21:58:57 -05:00
Jason Stedwell e7d86e14da Add post-1.5.1 planning docs: 9 improvement plans + containerized MCP server spec
docs/IMPROVEMENT-PLANS.md — nine improvements from the 2026-07-28 full-code
review (brief load, offline capture durability, session-end verb, local-first
recall index, incremental sweep, lifecycle decay, supersession, stemming +
alias expansion, resolve-miss learning), sequenced with TODO-1.6 and
ROADMAP-2.0 into one release train (1.6.0 -> 1.6.x -> 2.0 -> 2.1 -> 2.2 -> 2.3).

docs/MCP-SERVER-SPEC.md — build spec for the 2.1 flagship: echo-mcp as a
Docker container on ALPHA (FastMCP, streamable HTTP, bearer auth, published
as echomcp.alwisp.com), talking to Obsidian directly at http://10.2.0.35:27123.
13 v1 tools with result budgets + tool profiles; v1.1 dividends (nightly vault
backup, shadow write history/undo, background sweeps, ops alerting, rollup
data). Prereqs: session-end verb + return-not-print refactor (Phase 0).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 21:49:53 -05:00
Jason Stedwell d27db4ae34 Add docs/USAGE.md — operator usage guide for returning users (0.7 -> 1.5.1)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 15:56:28 -05:00
63 changed files with 3739 additions and 461 deletions
+11
View File
@@ -0,0 +1,11 @@
# Only mcp-server/ + the plugin scripts reach the image; keep the context lean.
.git
*.plugin
dist/
CODEX/
docs/
eval/
echo-icon*
*.pdf
*.html
__pycache__/
+50
View File
@@ -0,0 +1,50 @@
name: Build and Push Docker Image
on:
push:
branches: [main]
workflow_dispatch:
jobs:
build:
# Runs on the forgerunner host: bundled Docker CLI + mounted /var/run/docker.sock.
runs-on: host
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Log in to Gitea Container Registry
uses: docker/login-action@v3
with:
registry: registry.alwisp.com
username: ${{ secrets.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_TOKEN }}
- name: Build and Push
run: |
IMAGE="registry.alwisp.com/${{ gitea.repository }}"
GIT_SHA="$(git rev-parse --short HEAD)"
COMMIT_COUNT="$(git rev-list --count HEAD)"
docker build \
--label org.alwisp.git-sha="${{ gitea.sha }}" \
--label org.alwisp.version="v2.${COMMIT_COUNT}" \
--label org.alwisp.repo="${{ gitea.repository }}" \
-t "${IMAGE}:latest" .
docker push "${IMAGE}:latest"
# Dangling-only prune: removes untagged leftovers from previous builds. Never
# removes the tagged :latest or any image referenced by a running container.
- name: Prune dangling images on host
if: always()
run: docker image prune -f 2>/dev/null || true
- name: Trigger PORT redeploy
if: success()
run: |
# Repo is 'echo' but the container is 'echo-mcp' — target it explicitly.
curl -fsS -X POST https://port.alwisp.com/hooks/gitea \
-H "X-Deploy-Token: ${{ secrets.WEBHOOK_SECRET }}" \
-H "Content-Type: application/json" \
-d '{"container":"echo-mcp"}' || echo "PORT redeploy trigger failed (non-fatal)"
+7 -5
View File
@@ -20,8 +20,10 @@ eval/results/*.json
/echo-memory.config.json
/dist/
# Per-user BAKED artifacts (build.py --bake-key, named echo-memory-<ver>-<label>.plugin)
# carry a live vault bearer token — NEVER commit them. They belong in dist/ (above);
# this also guards any that get built at the repo root. The version-only pointer
# (echo-memory.plugin) and version-only artifacts (echo-memory-<ver>.plugin) stay tracked.
echo-memory-*-*.plugin
# Build artifacts (2.0 policy): the repo tracks ONLY the echo-memory.plugin pointer.
# Versioned builds (echo-memory-<ver>.plugin) are published as Gitea release assets
# on git.alwisp.com/jason/echo, one release per tag — not committed. Per-user BAKED
# artifacts (echo-memory-<ver>-<label>.plugin) carry a live vault bearer token and
# must NEVER be committed anywhere (they belong in dist/, also ignored above).
*.plugin
!echo-memory.plugin
+241
View File
@@ -1,5 +1,246 @@
# Changelog
## 2.3.0
**The index train** (IMPROVEMENT-PLANS #2/#4/#8) — one schema-bump release:
recall-index schema 3, entity-index schema 2. Old recall indexes are discarded
and rebuilt automatically (sub-second); no vault migration.
### Changed — the recall index is LOCAL-FIRST
The live BM25 index moves out of the vault into the machine state dir
(`recall-index-<endpoint-hash>.json` under `ECHO_STATE_DIR`). Capture's index
upkeep — previously GET-whole-index → PUT-whole-index against the vault under
the advisory lock, an O(vault) network round trip per write — is now a
zero-network atomic file write with no lock at all. The vault copy at
`_agent/index/recall-index.json` becomes a **snapshot**: written by
`sweep --apply` and best-effort at `session-end`, read only to seed a fresh
machine or to catch up after another client swept (checked at most every
`ECHO_RECALL_SYNC_HOURS`, default 24). The read path never PUTs the vault.
### Added — incremental sweep (`sweep --fast`) + content hashes
Entity-index entries and recall-index doc meta now carry a 16-char content
hash. `sweep --fast` walks the listing (cheap) and fetches only what is new,
gone, missing a hash, or in **today's rotating verification shard**
(`hash(path) % 7 == weekday` — the whole vault gets verified across a week of
fast sweeps while each run stays tiny; `--all-shards` forces everything).
Index-only and machine-owned, so no `--apply` gate; deletions are dropped from
both indexes. **`load` auto-runs it** when this machine's last sweep is older
than `ECHO_FAST_SWEEP_DAYS` (default 7) — edits made directly in Obsidian or by
other clients now reach the indexes without anyone remembering maintenance.
`doctor` reports index freshness.
### Added — stemming + alias query expansion
`echo_stem.py`: a conservative Porter-lite stemmer applied by `tokenize()` at
BOTH index and query time, so "deployed"/"deployment"/"deploys" meet at
"deploy" and "penalties" finds "penalty" (guards against over-stemming: "sing"
is not "s"; the -al/d~s irregulars are deliberately not conflated). At recall
time, a query that fuzzy-matches an entity (score ≥ 0.5) folds that entity's
title/alias vocabulary into the BM25 query at **half weight** — "turbokappa
tuning" finds the Kappa Engine note even though the alias appears in no body.
Expansion can only boost documents that actually contain the terms.
### Eval
Gold set grows 8 → **14 queries** (6 paraphrases the pre-2.3 lexical index
missed). v2.3.0: recall@5 / MRR **1.00 / 1.00**, 3/3 session-journal queries;
keyword baseline 0.75 / 0.79 and 0/3. Tests: +3 unit (stemmer families,
over-stemming guards, hash), +10 end-to-end (local-first no-snapshot-write,
local recallability, stemmed recall, alias expansion, fast-sweep pickup/
deletion/hash backfill). All suites — including the offline-queue, ops-api,
and MCP server suites — green; test harnesses now isolate `ECHO_STATE_DIR`.
## 2.2.0
### Added — echo-mcp: the containerized MCP server
ECHO's operations are now reachable as **typed MCP tools** from any surface with
an MCP connector (Claude Code, CoWork, claude.ai) — no Python-capable shell, no
`$ECHO` path resolution, no output re-parsing. New `mcp-server/` + `Dockerfile`
in this repo; deployed on ALPHA as the `echo-mcp` container behind
`https://echomcp.alwisp.com` (streamable HTTP, stateless JSON, bearer auth,
open `/health` for Docker/Kuma).
- **14 tools** wrapping the 2.1.1 `*_op` cores: `echo_load`, `echo_recall`
(score-packed `budget_chars`), `echo_resolve`, `echo_get_note`
(`section`/`max_chars`, traversal-guarded), `echo_get_scope`/`echo_set_scope`,
`echo_health` (`deep` runs the linter), `echo_capture` (gate as data —
`merge_into`/`force` are parameters), `echo_link` (paths OR resolvable names),
`echo_append_note`/`echo_patch_note` (invalid-target errors include the note's
actual headings), `echo_triage_inbox` (list/preview/apply in one tool),
`echo_reflect`, `echo_log_session` (the session-end bundle, heartbeat-last).
- **Tool profiles**: `ECHO_MCP_TOOLS=core` exposes only the six daily drivers.
- **Vault adjacency**: the container talks to the Obsidian REST API's HTTP
binding on the same box (`ECHO_BASE=http://10.2.0.35:27123`) — vault ops stop
depending on Cloudflare/NPM/DNS; the public chain only fronts the `echomcp`
ingress. Server-side offline queue at `/data`.
- **All MCP writes serialize** through the server process; the vault advisory
lock still coordinates with CLI clients.
- SKILL.md: prefer the `echo_*` tools when present; CLI recipes are the fallback.
- New suite `eval/test_mcp_server.py` (skips without the SDK): health/auth/
initialize/tools-list + the capture→gate→merge→recall→log_session flow over
real streamable HTTP.
The plugin itself is unchanged apart from the SKILL.md note — the container
vendors `skills/echo-memory/scripts/` at build time (one canonical tree).
## 2.1.1
### Changed — Phase 0 of the MCP build: every high-level op returns an envelope
Internal refactor (MCP-SERVER-SPEC §4), no behavior change intended: each
high-level operation now has a core `*_op` function that **returns an envelope
dict and never prints to stdout** — the transport-agnostic seam the upcoming MCP
server (2.2) wraps. The CLI verbs are thin wrappers that print exactly what they
printed before and map envelope outcomes to the same exit codes.
- `echo_ops`: `capture_op` / `resolve_op` / `link_op` — the duplicate gate and
offline queueing are now **data** (`action: duplicate-gate` with candidates;
`queued: true`), mapped to exit 76 / "queued" text only in the wrapper. The
`--json` stdout-swallowing hack is gone: helper chatter from the low-level
verbs routes to **stderr** wholesale inside the cores. `capture_op` accepts
`body_text=` directly (no temp file needed — the MCP path).
- `echo_recall.recall_op` — one structured result backs both the prose and
`--json` renderings (the duplicated prose path was deleted).
- `echo_triage.list_op` / `route_op`, `echo_reflect.apply_op`,
`echo_session.session_end_op` — same pattern; reflect/triage/session now call
`capture_op` internally and count gate outcomes from envelopes.
- `echo.scope_show_op` / `scope_set_op` / `load_op` (sections + brief digest as
data), `echo_doctor.run_op` (checks as a list).
New suite `eval/test_ops_api.py` asserts the contract for every core: envelope
shape AND stdout purity (a stray print would corrupt an MCP response stream).
All existing suites pass unchanged.
## 2.1.0
The "quick-wins train" from the 2026-07-28 review (`docs/IMPROVEMENT-PLANS.md` #1,
#3, #7). Three independently useful changes; no schema or layout changes.
### Added — `load --brief`: a token-budgeted cold-start digest
The SessionStart hook injected the full text of six files every session (measured
15.9KB and growing without bound — Observations and Scope History never shrink).
`load --brief` renders a digest instead: Fact / Pattern rules in full, the last ~10
Observations (with a `+N older` note), scope + `scope_updated` + sessions-since,
the heartbeat-pointed session log's key sections (Goal / Decisions Made / Open
Threads / Suggested Next Step), today's Agent Log lines, and the inbox as a
**count + oldest age** — everything the load-time reconcile needs. Over
`ECHO_LOAD_BUDGET` (default ~8000 chars), sections trim lowest-priority-first
(observations → agent log → session summary) with explicit `(truncated)` markers.
The hook now injects the brief form; `/echo-load` and the manual procedure keep the
full dump. All `load` reads (plus the sessions listing, which also feeds the
heartbeat-absent fallback) are fetched **in parallel** over the pooled connections.
### Fixed — capture is now offline-durable (the write path's inversion bug)
Low-level verbs queued on an outage, but the *default* write (`capture`) died: the
update path's POST/PATCH and `ensure_daily_log` bypassed the queue entirely, and an
unreachable index aborted the whole op. Now: (1) a fully-offline `capture` queues
the **whole operation as one semantic record** (title/kind/body/tags/…, plus the
capture-time date); `flush` replays it **through `capture`** so create-vs-update,
the duplicate gate, and aliasing re-run against the index as it is at replay time —
byte-level replay would freeze a decision made blind. (2) A duplicate-gate stop on
replay keeps the record, flags it `needs_attention` (surfaced by `flush` and
`load`), and does NOT block later records. (3) Mid-capture outages: the update
path and `ensure_daily_log` writes ride `safe_request` (accepted edge: a queued
agent-log PATCH 400-drops loudly if the daily note never existed — the trace line,
not the memory). Same-args offline captures dedupe by idem-key.
### Added — `session-end`: the one-call session close
`echo.py session-end <bundle.json> [--apply]` performs the whole end-of-session
sequence under ONE lock acquisition, in order: session log PUT → Agent Log line →
reflect proposals (via `capture`, gate-aware) → optional `scope set` → **heartbeat
LAST as the commit marker** (a part-way failure leaves the previous pointer intact,
never dangling). Dry-run by default previews the full plan including the reflect
classification. Filename HHMM from `$ECHO_NOW` (else local clock), validated
against the canonical session-log regex **before any write**. The Stop hook's nudge
now names the command, and recognizes a `session-end` invocation as
"already reflected".
### Fixed — helper-module errors exited 2 instead of their intended codes
When `echo.py` runs as `__main__`, helper modules' `import echo` creates a twin
module whose `EchoError` is a different class object, so `main()`'s handler missed
it and the raw traceback leaked. The handler now catches `RuntimeError` (both
classes subclass it) and honors `.code`.
Tests: +19 end-to-end checks (brief-load digest/budget, offline-capture queue /
replay-through-capture / gate-on-replay / idem-key dedup, session-end dry-run /
apply / ordering / validation-abort). All suites green.
## 2.0.0
**Packaging & structure major — nothing about memory behavior changes.** The major
version signals "reinstall, re-clone expectations": how the plugin is distributed
and installed is what breaks.
### Removed — the legacy `commands/` directory (BREAKING for pre-skills clients)
The plugin is **skills-only**. The eight `/echo-*` entry points live exclusively at
`skills/<name>/SKILL.md` (introduced 1.6.0, verified on desktop + CoWork
2026-07-28, where they already took precedence). A Claude Code old enough to lack
skills support loses the slash commands — install 1.6.0 from the release history
instead.
### Changed — artifact policy (BREAKING for anyone pulling zips from the repo)
- The repo tracks **only** `echo-memory.plugin` (the current installable). The 15
historical `echo-memory-<version>.plugin` zips are deleted from the tree (git
history retains them).
- Versioned builds are published as **Gitea releases** on
`git.alwisp.com/jason/echo` — one release per `v<version>` tag, artifact
attached, starting with `v2.0.0`.
- `.gitignore` now ignores `*.plugin` except the pointer.
### Added — "packaging for other agent runtimes" README note
The retired Codex tree stays retired; the README now documents what a port needs
(manifest shape, skill entry point, script invocation + hooks) and the rule that a
port must be a **build target** from the canonical source tree, never a second
hand-maintained copy.
## 1.6.0
### Changed — the eight slash commands migrated to the skills format
Each `commands/<name>.md` now has a `skills/<name>/SKILL.md` twin with the same
invocation name and a byte-identical body — confirmed against the current docs
that a same-name skill **takes precedence** over the legacy command, so both
formats coexist safely in this release (deleting `commands/` is the 2.0
packaging break). Done for the features, not the deprecation notice:
- **`allowed-tools` per skill** — the resolved `python3/python/py -3` script
invocations (plus the CoWork `ls /sessions/*` fallback probe) are
pre-authorized, so `/echo-load`, `/echo-health`, `/echo-recall` etc. stop
prompting.
- **`disable-model-invocation: true`** on the write-heavy entry points
(`/echo-sweep`, `/echo-triage`) — only the operator triggers them; the
read-side skills stay model-invocable.
- `argument-hint` / `$ARGUMENTS` carry over unchanged (same semantics in
SKILL.md bodies).
- The `$ECHO` path-resolution block stays per-skill: the skills format has no
cross-skill snippet sharing (each skill directory is self-contained per the
official docs), and the block is already the 2-line minimum.
### Fixed — lint blind spot: retired trees masked by the leaf-README route
`vault_lint.py` checked retired patterns only for paths matching **no** route,
so the permissive `leaf-readme` route (`^(.+/)?README\.md$`) absolved any seed
README inside a retired tree — `archive/` and `_agent/outputs/` survived the
1.5.0 live cleanup unflagged. Retired patterns are now checked **first**: a
path in a retired tree flags `retired-path` regardless of what else matches.
New end-to-end test seeds `archive/notes/README.md` and asserts the flag
(fails against the pre-fix linter, passes now).
### Added — plugin.json completeness
`homepage` + `repository``https://git.alwisp.com/jason/echo`; manifest →
1.6.0.
## 1.5.1
### Fixed — duplicate gate over-firing on vault-common tokens and cross-kind names
+26
View File
@@ -0,0 +1,26 @@
# echo-mcp — containerized MCP server over the ECHO vault (docs/MCP-SERVER-SPEC.md).
# LEGACY Dockerfile format on purpose: the git.alwisp.com CI runner has no BuildKit
# (no `# syntax=` line, no RUN --mount).
FROM python:3.12-slim
WORKDIR /app
COPY mcp-server/requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir -r /app/requirements.txt
# The canonical plugin scripts ARE the server's ops layer — vendored at build time,
# never a second source tree.
COPY echo-memory.plugin.src/skills/echo-memory/scripts /app/scripts
COPY mcp-server/app.py /app/app.py
ENV ECHO_STATE_DIR=/data \
ECHO_MCP_PORT=8765 \
PYTHONUNBUFFERED=1
VOLUME /data
EXPOSE 8765
# Probe 127.0.0.1, NOT localhost (::1-vs-IPv4 lesson from cpas/memer/breedr).
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python3 -c "import urllib.request;urllib.request.urlopen('http://127.0.0.1:8765/health', timeout=4)" || exit 1
CMD ["python3", "/app/app.py"]
+19 -9
View File
@@ -1,4 +1,4 @@
# echo-memory — v1.5.1
# echo-memory — v2.3.0
Persistent memory for Claude / CoWork sessions via the **ECHO** Obsidian vault, driven over the [Obsidian Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api). The skill makes direct REST calls through a bundled validated client (`scripts/echo.py`). The whole toolchain is **pure Python** (stdlib only), so it runs identically on Windows, macOS, and Linux — no bash, no platform-specific `date`.
@@ -67,7 +67,8 @@ echo-v.05/
├── docs/history/ ← historical inputs (e.g. the 1.5.0 frontmatter field report)
├── build.py ← deterministic .plugin builder (--bake-key/--strip-key/--label/--outdir)
├── echo-memory.plugin ← built, installable plugin (zip artifact, rebuilt on version bump)
├── echo-memory-<version>.plugin ← versioned build artifacts (history; moves to Gitea releases in 2.0)
│ — the ONLY tracked artifact; versioned builds are Gitea release
│ assets (one release per v<version> tag), not committed (2.0 policy)
├── dist/ ← per-user baked artifacts (secret-bearing) — GITIGNORED, never committed
├── eval/ ← credential-free eval + test harness; not bundled
│ ├── mock_olrapi.py ← deterministic mock of the REST API + fault injection
@@ -80,8 +81,9 @@ echo-v.05/
├── .claude-plugin/plugin.json ← manifest (name, version, description)
├── README.md ← plugin-level README
├── hooks/hooks.json ← session hooks: SessionStart auto-load · Stop reflection nudge
├── commands/ ← slash commands (legacy format; skills-format migration = TODO-1.6):
echo-load|save|recall|triage|health|sweep|reflect|doctor
├── skills/echo-{load,save,recall,triage,health,sweep,reflect,doctor}/
← the eight slash commands as skills (skills-only since 2.0):
│ per-skill allowed-tools; sweep/triage are operator-only
└── skills/echo-memory/
├── SKILL.md ← operating procedure (authoritative)
├── references/
@@ -120,6 +122,8 @@ echo-v.05/
└── templates/ ← 8 canonical note templates
```
**Packaging for other agent runtimes** (from the retired Codex tree, deleted 2026-07-03): if a port to another runtime (Codex, etc.) is ever wanted, implement it as a **build target** generated from the canonical `echo-memory.plugin.src/` tree (`build.py --target <runtime>` or similar) — never as a second hand-maintained source tree; that's how the original Codex copy drifted. A port needs three adaptations: the manifest shape (`.claude-plugin/plugin.json` → the runtime's equivalent), the skill entry point (`skills/*/SKILL.md` frontmatter + body conventions), and the script-invocation convention (`${CLAUDE_PLUGIN_ROOT}` resolution + the session-hook wiring in `hooks/hooks.json`). The Python toolchain itself is runtime-agnostic (pure stdlib, env/config-driven).
**Division of responsibility:** `SKILL.md` owns day-to-day *procedure* (loading order, search-first, triage, scope switching, PATCH rules) and points at the bundled tooling. `references/operating-contract.md` owns the durable, client-independent *principles, safety rules, and concurrency model*. `scripts/routing.json` is the machine-readable source of truth for routing; `references/routing-map.md` is its human-readable authority. The other references are the canonical layout, API, and bootstrap specs.
---
@@ -399,14 +403,14 @@ If the API returns a connection error, timeout, or `502` (usually Obsidian / the
---
## Eval metrics (v1.5.1, 2026-07-03)
## Eval metrics (v2.3.0, 2026-07-29)
From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate.
From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate. The gold set grew to **14 queries in 2.3** — six are paraphrases (morphological variants like "penalties in the SLA" vs a note saying "penalty clause") that a purely lexical index misses; the stemmed index answers all of them.
| Metric | v1.5.1 | pre-1.5 baseline |
| Metric | v2.3.0 | pre-1.5 baseline |
|---|---|---|
| Retrieval recall@5 / MRR (8-query gold set) | **1.00 / 1.00** | 0.75 / 0.75 |
| Queries answerable only from sessions/journal | **2/2** | 0/2 (not in corpus) |
| Retrieval recall@5 / MRR (14-query gold set incl. 6 paraphrases) | **1.00 / 1.00** | 0.75 / 0.79 |
| Queries answerable only from sessions/journal | **3/3** | 0/3 (not in corpus) |
| Freshness: live note outranks stale archived twin | **yes** | no |
| Duplicate notes created (3 renamed-entity captures) | **0** (gate blocks) | 3 |
| Legitimate captures wrongly blocked | **0** | 0 |
@@ -417,6 +421,12 @@ From the credential-free harness (`eval/run_eval.py` against the deterministic m
| Version | Highlights |
|---------|-----------|
| **2.3.0** | **The index train (recall-index schema 3, entity-index schema 2).** (1) **Local-first recall index** — the live BM25 index moves to the machine state dir (keyed by endpoint); capture's index upkeep becomes a zero-network atomic file write instead of GET/PUT-whole-index under the vault lock; the vault copy is now a snapshot (written by sweep + session-end) that seeds fresh machines. (2) **Incremental sweep** — entity entries carry a content hash; `sweep --fast` fetches only new/gone/changed notes plus a rotating weekday shard (whole vault verified across a week); auto-runs at load when >`ECHO_FAST_SWEEP_DAYS` (7) — Obsidian-side edits now reach the indexes without manual maintenance. (3) **Stemming + alias expansion**`echo_stem` (Porter-lite, conservative) applied at index+query time, and a query matching an entity's alias folds its title/alias vocabulary in at half weight. Eval gold set grows to 14 queries (6 paraphrases): recall@5 / MRR **1.00 / 1.00**. |
| **2.2.0** | **echo-mcp — the containerized MCP server.** ECHO as 14 typed MCP tools from any connector-capable surface (Claude Code, CoWork, claude.ai): `mcp-server/` + Dockerfile in-repo, deployed on ALPHA as `echo-mcp` behind `https://echomcp.alwisp.com` (streamable HTTP, stateless JSON, bearer auth, open `/health`). Talks to the Obsidian REST API directly on the LAN (`10.2.0.35:27123`) — one client round trip per tool call, all vault chatter host-local. Duplicate gate & offline queueing surface as data; `ECHO_MCP_TOOLS=core` trims the surface to six tools; writes serialize server-side. New `eval/test_mcp_server.py` e2e suite. Full spec: `docs/MCP-SERVER-SPEC.md`. |
| **2.1.1** | **MCP Phase 0 — return-not-print.** Every high-level op gains a core `*_op` function returning an envelope dict with a stdout-purity guarantee (helper chatter → stderr); CLI verbs become thin wrappers with identical output and exit codes. Duplicate gate and offline queueing become data (`action: duplicate-gate` / `queued: true`). New `eval/test_ops_api.py` contract suite. This is the seam the 2.2 containerized MCP server wraps. |
| **2.1.0** | **Quick-wins train: cheaper sessions, durable capture, one-call session end.** (1) **`load --brief`** — a token-budgeted cold-start digest (Fact/Pattern in full, last ~10 Observations, scope+freshness, last session's key sections, Agent-Log lines, inbox *count*; `ECHO_LOAD_BUDGET` default ~8000 chars) now injected by the SessionStart hook, replacing the full six-file dump that grew unboundedly; all `load` reads are fetched in parallel. (2) **Offline capture durability**`capture` on an unreachable vault queues the *whole operation* as one semantic record; `flush` replays it **through capture** so routing/gate/aliasing re-run against the current index; a gate stop on replay is kept + flagged, never landed blind; `ensure_daily_log`/update-path writes ride the queue too. (3) **`session-end`** — one call (one lock) writes session log → Agent-Log line → reflect proposals → optional scope switch → **heartbeat last as the commit marker**; dry-run by default; `ECHO_NOW` pins the HHMM. Also fixes the `__main__` twin-module trap so helper-module `EchoError`s exit with their intended codes. +19 end-to-end checks. |
| **2.0.0** | **Packaging & structure major — memory behavior unchanged.** Skills-only: the legacy `commands/` directory is deleted (breaking for pre-skills clients; the 1.6.0-verified skills are the sole entry points). Artifact policy: the repo tracks only the `echo-memory.plugin` pointer; the 15 historical versioned zips leave the tree and versioned builds ship as **Gitea releases** (one per `v<version>` tag) from `v2.0.0` on; `.gitignore` blocks `*.plugin` except the pointer. README gains the "packaging for other agent runtimes" port note (build-target rule, never a second source tree). |
| **1.6.0** | **Skills-format migration + lint blind-spot fix.** The eight slash commands gain `skills/<name>/SKILL.md` twins (same names, byte-identical bodies; a same-name skill takes precedence, so `commands/` coexists until its 2.0 deletion): per-skill `allowed-tools` end the permission prompts, `disable-model-invocation: true` makes `/echo-sweep` + `/echo-triage` operator-only. `vault_lint` checks retired patterns **before** routes, so a seed README inside a retired tree (`archive/…`) is flagged instead of being absolved by the leaf-README route (+ regression test). `plugin.json` gains `homepage`/`repository`. |
| **1.5.1** | **Duplicate-gate precision.** Live use caught the gate blocking a decision note because it shared the single vault-common token "echo" with an archived project (min-normalized scoring gives any single-token entity a 1.0 match). New `gate_candidates()` blocks only on **same-kind** candidates (cross-kind name collisions — a decision titled after its project — warn instead), and a **lone shared token blocks only when unique to that entity** (vault df == 1); multi-token overlaps still block. Advisory warnings (`fuzzy_candidates`) unchanged; exit-76/`--merge-into`/`--force` unchanged. +5 tests; verified against the live vault both ways (false positive creates cleanly, true lookalike still gates). |
| **1.5.0** | **Six-improvement pass: retention, routing, and self-firing memory.** (1) **Capture-update keeps the whole body** — the update path previously appended only the first line of a multi-line body (silent data loss); now the full body rides under the dated bullet. (2) **Frontmatter completeness** — capture stamps a kind-default `status` and kind-seeded `tags` (`--tags` enriches); `fm` is create-or-replace (missing keys are surgically inserted instead of 400-failing); `vault_lint` gains a kind-scoped `incomplete-frontmatter` check; `sweep --apply` backfills (fixes the 18/71-notes field-audit drift, one data source: `KIND_STATUS`/`KIND_REQUIRED_FM`). (3) **Pre-write duplicate gate** — a strong fuzzy candidate stops `capture` (exit 76 + candidates) before the duplicate exists; `--merge-into <slug>` updates the canonical entity, `--force` overrides. (4) **Recall corpus + ranking** — sessions and journal notes join the BM25 corpus (down-weighted); scores fuse freshness (half-life on `updated:`) and status (`active` boosted, `archived` demoted); hits show `updated:`/`status:`; `recall --json`; recall-index schema 2 (auto-rebuilds). (5) **Session hooks** — SessionStart auto-loads memory into context, Stop nudges reflection once per substantive session; fail-safe, CoWork-fallback-aware. (6) **One-tap triage**`echo.py triage` lists the inbox structured, then classifies → previews → routes accepted items via `capture` with automatic processing-log audit lines; `--json` also lands on `link`/`scope show`. +22 mock end-to-end tests; all suites green. |
| **1.4.2** | **CoWork sandbox path resilience.** In a remote CoWork sandbox `${CLAUDE_PLUGIN_ROOT}` can point at a host path the sandbox can't reach (`/var/folders/…`) while the plugin is mounted under `…/mnt/.remote-plugins/…`, so script invocations failed until the agent found the real path by hand. SKILL.md and all eight slash commands now resolve the scripts dir with a fallback — prefer `${CLAUDE_PLUGIN_ROOT}`, else locate the mounted copy under `/sessions/*/mnt/.remote-plugins/*/…` — reused via `$ECHO`/`$LINT`/`$SWEEP`/`$SDIR`. On a normal host the primary path always wins (no behaviour change). `echo-health`/`echo-sweep` `allowed-tools` broadened to match the resolved invocation. (The scripts never hardcoded a path — root cause is the harness env var — but the plugin now self-heals.) |
+16 -14
View File
@@ -20,12 +20,12 @@ The repo root tracks every historical build (`echo-memory-0.6.0.plugin` …
`echo-memory-1.5.1.plugin`) plus the `echo-memory.plugin` pointer. 2.0 changes the
policy — **breaking for anyone who pulls artifacts straight from the repo**:
- [ ] Track only `echo-memory.plugin` (the current installable); delete the versioned
zips from the tree.
- [ ] Publish versioned builds as **Gitea releases** on `git.alwisp.com/jason/echo`
instead (one release per tag, artifact attached).
- [ ] `.gitignore`: `*.plugin` except the pointer.
- [ ] Tag releases going forward (`v2.0.0`, …) so the release page is the version
- [x] Track only `echo-memory.plugin` (the current installable); delete the versioned
zips from the tree**done 2.0.0** (15 zips removed; history keeps them).
- [x] Publish versioned builds as **Gitea releases** on `git.alwisp.com/jason/echo`
instead (one release per tag, artifact attached)**v2.0.0 onward**.
- [x] `.gitignore`: `*.plugin` except the pointer**done 2.0.0**.
- [x] Tag releases going forward (`v2.0.0`, …) so the release page is the version
history for artifacts, and the README table stays the narrative history.
## 2. Complete the skills-format migration (finishes the 1.6 work)
@@ -33,10 +33,12 @@ policy — **breaking for anyone who pulls artifacts straight from the repo**:
1.6 migrates the eight slash commands to `skills/*/SKILL.md` with both formats
coexisting (see `TODO-1.6.md`). 2.0 finishes it:
- [ ] **Delete the legacy `commands/` directory**the packaging break that partly
motivates the major bump.
- [ ] Re-verify desktop + CoWork installs of the skills-only artifact.
- [ ] Update SKILL.md / README / command docs that reference `commands/`.
- [x] **Delete the legacy `commands/` directory****done 2.0.0** (1.6.0's install
verification on both surfaces cleared the gate; skills had precedence anyway).
- [x] Re-verify desktop + CoWork installs of the skills-only artifact — **verified
2026-07-28** on the baked 2.0.0 build: installs cleanly, all eight skills
register and work with no legacy `commands/` present.
- [x] Update SKILL.md / README / command docs that reference `commands/`**done 2.0.0**.
## 3. Codex packaging (from MAINTENANCE Canonical source tree)
@@ -47,9 +49,9 @@ regenerated after 2.0 if needed). What remains for 2.0:
- [ ] If Codex support returns: implement it as a **build target** generated from the
canonical `echo-memory.plugin.src/` tree (`build.py --target codex` or similar) —
never as a second hand-maintained source tree (that's how the drift happened).
- [ ] Otherwise: a short "packaging for other agent runtimes" note in the README
- [x] Otherwise: a short "packaging for other agent runtimes" note in the README
documenting what a Codex/other-runtime port needs (manifest shape, skill entry
point, script invocation).
point, script invocation)**done 2.0.0** (README after Repository layout).
## 4. Eval refresh — publish current-version metrics (from MAINTENANCE Docs freshness; ROADMAP-1.0 H4 leftover) — ✅ DONE
@@ -67,8 +69,8 @@ the README. Re-run it per release and refresh the README table.
## 5. Repo hygiene odds & ends
- [x] Retire `ROADMAP-1.0.md` (fully shipped; in git history) — **done 2026-07-03**.
- [ ] Root README "Repository layout" section updated for the post-2.0 tree
(no versioned zips, no codex tree, skills-only plugin).
- [x] Root README "Repository layout" section updated for the post-2.0 tree
(no versioned zips, no codex tree, skills-only plugin)**done 2.0.0**.
- [x] `echo-improvements-prompt.md` (the 1.5.0 field report) moved to `docs/history/`
**done 2026-07-03**.
+30 -14
View File
@@ -14,22 +14,36 @@ the official docs (code.claude.com/docs/en/skills.md, plugins.md):
invocation name: `/echo-save` works identically. No deprecation date on `commands/`;
this is future-proofing, not a fire.
- **Do it for the features, not the notice:**
- [ ] `allowed-tools` per skill — pre-authorize the resolved `python3 "$ECHO" …`
invocations so `/echo-load`, `/echo-health`, `/echo-recall` stop prompting
(1.4.2 hand-broadened two commands; the skill format does this properly).
- [ ] `disable-model-invocation: true` on the write-heavy entry points
(`/echo-sweep`, `/echo-triage`) so only the operator triggers them; leave the
read-side ones model-invocable.
- [ ] De-duplicate the CoWork `$ECHO` path-resolution block currently copy-pasted
into all eight command bodies — each skill folder can carry a shared snippet.
- [ ] `argument-hint` carries over as-is; `$ARGUMENTS` substitution unchanged.
- [ ] Verify a migrated build installs cleanly on desktop **and** in a CoWork session
before deleting `commands/` (deleting the legacy dir is a 2.0 item
see `ROADMAP-2.0.md`; in 1.6 both may coexist).
- [x] `allowed-tools` per skill — **done 1.6.0** (all eight skills pre-authorize
the resolved `python3`/`python`/`py -3` invocations + the CoWork `ls` probe).
- [x] `disable-model-invocation: true` on the write-heavy entry points
(`/echo-sweep`, `/echo-triage`) — **done 1.6.0**; read-side stays
model-invocable.
- [x] De-duplicate the CoWork `$ECHO` path-resolution block — **resolved 1.6.0 as
not-supported**: per the official skills docs, skill directories are
self-contained (no cross-skill snippet sharing), so the 2-line block stays
per-skill. Docs confirmed same-name skill takes precedence over the legacy
command, so coexistence is safe.
- [x] `argument-hint` carries over as-is; `$ARGUMENTS` substitution unchanged
**done 1.6.0** (save/recall/reflect).
- [x] Verify a migrated build installs cleanly on desktop **and** in a CoWork session
before deleting `commands/`**verified 2026-07-28** on the baked 1.6.0-jason
artifact, all four checks pass: (1) all eight skills register and take
precedence over the coexisting legacy commands; (2) `allowed-tools` present on
load/health/recall; (3) `disable-model-invocation` confirmed on exactly
sweep+triage; (4) CoWork `$ECHO` fallback resolves via
`/sessions/*/mnt/.remote-plugins/*` and executes. **2.0's `commands/` deletion
is unblocked.** *Known caveat, accepted (CLI is not a normal operating surface
here): the `allowed-tools` globs match the literal script names
(`*echo.py*`/`*vault_lint.py*`) while the bodies invoke via the resolved
`"$ECHO"` variable — if the Claude Code CLI matches permissions before
variable expansion, those skills could still prompt there. Re-confirm on the
CLI if that surface ever matters.*
## 2. plugin.json completeness (from the old MAINTENANCE checklist)
- [ ] Add `homepage` / repository URL — now decided: `https://git.alwisp.com/jason/echo`.
- [x] Add `homepage` / repository URL — **done 1.6.0**: both `homepage` and
`repository` set to `https://git.alwisp.com/jason/echo`.
## 3. Vault follow-ups (noticed during the 1.5.0 dead-link cleanup)
@@ -46,7 +60,9 @@ they were unwrapped per instruction but repointing them would restore real graph
*(add items here as the new plugin gets real use)*
- [ ] **Lint blind spot: retired dirs masked by the leaf-README route** (found 2026-07-03).
- [x] **Lint blind spot: retired dirs masked by the leaf-README route** (found
2026-07-03; **fixed 1.6.0** — retired patterns now checked before routes, with
the `archive/notes/README.md` regression test).
`archive/` and `_agent/outputs/` are retired/unrouted pre-0.6 leftovers that survived
in the live vault holding only their seed READMEs — and the permissive `leaf-readme`
route (`^(.+/)?README\.md$`) matches first, so `vault_lint` never flagged them.
+1 -1
View File
@@ -2,7 +2,7 @@
"""build.py — package the echo-memory plugin source into a .plugin artifact.
Zips the CONTENTS of echo-memory.plugin.src/ at the archive root (the layout the plugin
loader expects: .claude-plugin/plugin.json, commands/, skills/ all at top level), excluding
loader expects: .claude-plugin/plugin.json, hooks/, skills/ all at top level), excluding
dev cruft. The version is read from the manifest, so the output is named automatically.
Usage:
+28
View File
@@ -0,0 +1,28 @@
# echo-mcp — PORT deploy manifest (ALPHA / br0). See docs/MCP-SERVER-SPEC.md §8.
# Container name deliberately differs from the repo (echo): the repo ships the
# plugin AND this server; only the server is a container.
name: echo-mcp
image: registry.alwisp.com/jason/echo:latest
network: br0
ip: auto
ports: [] # br0 static IP — app serves :8765 directly
volumes:
- /mnt/user/appdata/echo-mcp:/data # outbox + read cache (+ backups/history, v1.1)
env:
# Vault adjacency: the Obsidian Local REST API's HTTP binding on the same box —
# never the public echoapi.alwisp.com hairpin (spec §8; cleartext stays on the LAN).
ECHO_BASE: http://10.2.0.35:27123
ECHO_OWNER: Jason Stedwell
ECHO_STATE_DIR: /data
ECHO_MCP_PORT: "8765"
ECHO_MCP_TOOLS: full
# Secrets from the PORT secret store — values never appear in this file or chat.
ECHO_KEY: SECRET:echo-vault-key
ECHO_MCP_TOKEN: SECRET:echo-mcp-token
health_check_path: /health
health_check_port: 8765
webui: https://echomcp.alwisp.com
proxy:
forward_port: 8765
forward_scheme: http
websockets: true
+353
View File
@@ -0,0 +1,353 @@
# echo-memory — Improvement Plans (post-1.5.1 review)
> Status: **planned, not started.** Source: full-code review 2026-07-28.
> Nine improvements, each scoped for an independent build session. The tenth item
> from that review — the **MCP server** — has its own detailed spec:
> `docs/MCP-SERVER-SPEC.md`.
>
> Sequencing is at the bottom. Nothing here conflicts with `TODO-1.6.md` or
> `ROADMAP-2.0.md`; release targets assume 1.6 (skills migration) ships first.
---
## 1. Token-budgeted load (`load --brief`) + parallel orientation reads
**Problem.** The SessionStart hook injects the *full text* of six files into every
session. Live measurement: 15.9KB of hook context on a single cold start, and two of
the six files grow without bound (operator-preferences `## Observations`,
current-context `## Scope History`). Every session pays this tax before any work
happens. Separately, `cmd_load` (echo.py) issues its six GETs serially even though
`read_many()` exists.
**Design.**
- New `echo.py load --brief` (and make the SessionStart hook use it):
- marker → one line (`schema_version`, bootstrapped yes/no).
- operator-preferences → `## Fact / Pattern` in full; `## Observations` capped at
the last 10 lines with a `(+N older — /echo-load for full)` note.
- current-context → `## Scope` + `scope_updated` + sessions-since (i.e. the
`scope show` output); omit `## Scope History`.
- heartbeat → the pointer line + only the `## Summary`/`## Outcomes` sections of
the pointed-at session log (fetch it, extract those headings, cap ~30 lines).
- today's daily note → `## Agent Log` lines only (or "absent").
- inbox → count + age of oldest item, not the contents (that's all the reconcile
needs; `/echo-triage --list` has the details).
- Budget guard: after assembly, if the brief output still exceeds
`ECHO_LOAD_BUDGET` chars (default ~8000), truncate lowest-priority sections first
(observations → agent log → session summary) with explicit `(truncated)` markers.
- Full `load` unchanged; `/echo-load` keeps using it.
- Fetch all reads via `read_many()` (the 6 targets + the heartbeat-pointed session
log + the sessions listing fallback) instead of the serial loop at
`echo.py:640`. Cache-put still applies per path.
**Files.** `echo.py` (cmd_load), `echo_hook_session_start.py`, SKILL.md (document
brief-vs-full), `eval/test_features.py`.
**Tests.** Mock vault with oversized preferences/history files → assert brief output
under budget, sections present, truncation markers correct; hook emits brief.
**Size/target.** Small-medium. **1.6.x point release** — highest leverage per line.
---
## 2. Local-first recall index
**Problem.** `echo_recall.update_note()` does GET-whole-index → add → PUT-whole-index
against the vault, under the global advisory lock, on every capture and every corpus
`put`. The index carries full BM25 postings for the whole corpus, so this round-trip
is O(vault) network per write and is the emerging bottleneck + lock hot-spot as the
vault grows past a few hundred notes.
**Design.**
- The **live** recall index moves to the local state dir
(`~/.echo-memory/recall-index.json`, honoring `ECHO_STATE_DIR`) keyed by endpoint
(hash of `BASE`) so multiple vaults don't collide.
- `update_note()` becomes a local read-modify-write (file lock via `os.O_EXCL`
sidecar or atomic `os.replace`), **no vault round-trip, no advisory lock**.
- The vault copy (`_agent/index/recall-index.json`) becomes a **snapshot**, written
by `sweep.py` and at session end (piggyback on the heartbeat write / future
`session-end` verb). It exists so a *fresh machine* can seed its local index
without a full rebuild.
- Freshness rule on recall: local index used when present; if the vault snapshot's
embedded `built` stamp is newer than the local one (another client swept), pull it.
Staleness across clients is tolerable — recall degrades gracefully and improvement
#4 (incremental sweep) trues it up cheaply.
- Schema bump: recall-index schema 3 adds `built` (ISO timestamp) + `endpoint_hash`.
Old schema-2 vault copies are still readable as seeds.
- The **entity index stays in the vault** (it is small, and it is the shared routing
authority for concurrent clients) — only the BM25 index moves.
**Files.** `echo_recall.py` (load/save/update_note/rebuild), `echo_queue.py`
(state-dir helpers shared), `sweep.py`, `echo.py` (cmd_put upkeep path), routing.json
(the vault snapshot path is unchanged), README performance section.
**Tests.** Capture with vault mock → assert zero recall-index PUTs; sweep → snapshot
written; fresh state dir + existing snapshot → seeded without rebuild; two-vault
(endpoint) separation.
**Size/target.** Medium. **Pair with #4 in one index-focused minor** (they touch the
same maintenance loop). Fine to land before or after 2.0.
---
## 3. Offline durability for the high-level ops (capture path)
**Problem.** The low-level verbs queue on outage via `safe_request`, but
`_append_to_existing()` and `ensure_daily_log()` (echo_ops.py) call `echo.request`
directly — so during an outage a raw `append` survives, while a `capture` **update**
and its Agent Log line silently vanish. Inverted from the documented contract
("capture is the default write").
**Design.** Queue *semantic* records, not byte-level requests, for the high-level
path:
- New outbox record type `{"op": "capture", "args": {...}}` alongside the existing
raw records. On an unreachable vault, `capture` short-circuits: enqueue the full
argument set (title/kind/body/tags/aliases/sources/date/domain/merge_into) and
report `queued (offline): capture …`.
- `flush()` replays `op:capture` records by calling `echo_ops.capture(...)` against
the *current* index — so routing, the duplicate gate, aliasing, and linking all
re-run with fresh state instead of replaying stale byte writes. A gate stop on
replay (exit 76) keeps the record queued with a `needs_attention` flag and is
surfaced by `load`/`flush` output ("1 queued capture needs --merge-into/--force").
- Rationale for semantic-over-byte: a byte replay of `_append_to_existing`'s PATCH
could target a heading that moved, and the create-vs-update decision made offline
may be wrong by replay time. Re-running the op is the idempotent, correct unit
(the dated-bullet idempotency key already prevents double-append).
- `ensure_daily_log` stays best-effort but routes its POST/PATCH through
`safe_request` so a standalone agent-log line survives an outage too.
- reflect/triage `--apply` inherit this for free (they call capture).
**Files.** `echo_ops.py`, `echo_queue.py` (record schema v2 + replay dispatch),
`eval/test_offline_queue.py`.
**Tests.** Fault-injected mock: capture-update offline → queued; flush replays via
capture; gate-on-replay keeps record + flags; agent-log line queued; no duplicate
bullets on double flush.
**Size/target.** Small-medium. **1.6.x point release** — it's a correctness gap.
---
## 4. Incremental sweep via content hashes
**Problem.** Human edits in Obsidian and writes by other clients never touch the
entity/recall indexes; drift is only corrected by a full sweep someone must remember
to run. Full sweep re-reads the whole vault (fast now, but O(vault)).
**Design.**
- Entity index entries and the recall-index snapshot gain a per-note
`h` — SHA-1 of the note body (entity index schema 2; tolerated-absent for old
entries).
- `sweep.py --fast`: walk the listing (cheap), `read_many` only paths that are
(a) new, (b) missing a stored hash, or (c) whose hash mismatches after fetch — to
avoid fetching everything just to hash it, fast mode fetches only paths whose
**listing is new/gone** plus a rotating shard (e.g. `hash(path) % 7 == weekday`)
of existing notes, so the whole vault is verified over a week of fast sweeps while
each run stays tiny. `--fast --all-shards` forces full verification.
- Deletions: listing walk catches removed files → drop index + recall entries.
- Auto-run: `load` (brief or full) runs `sweep --fast` opportunistically when the
last fast-sweep stamp (state dir) is older than `ECHO_FAST_SWEEP_DAYS` (default 7),
in the background-tolerant sense: bounded by the same read_many concurrency, and
skipped entirely when offline.
- `/echo-health` reports last fast/full sweep ages.
**Files.** `sweep.py`, `echo_index.py` (schema 2 + hash field), `echo_recall.py`
(snapshot hash reuse), `echo.py` (load hook-in), `vault_lint.py` (report), migrate
note (index schema is machine-owned; no vault migration needed).
**Tests.** Mock: edit a note out-of-band → fast sweep in that shard re-indexes it;
delete → entry dropped; stamp gating; full sweep unchanged.
**Size/target.** Medium. **Ship with #2** (same schema-bump train).
---
## 5. Memory lifecycle — enforce forgetting
**Problem.** The vault only accretes. Working memory is "time-boxed" by convention
only; Observations trimming is manual prose; stale-active detection only reports.
Recall precision degrades as dead weight accumulates.
**Design.**
- `capture --kind working` stamps `expires: <today + ECHO_WORKING_TTL_DAYS>`
(default 14; `--ttl <days>` overrides). Template updated to carry the field.
- `vault_lint.py` new checks: `expired-working` (past `expires`),
`stale-observations` (Observations > 30 bullets) — advisory, like the rest.
- `sweep.py --decay` (dry-run by default, `--apply` to write; **always
preview-first**, honoring the operating contract):
- expired working notes → `status: archived` + move body under a dated
`## Archived` marker (never delete);
- `projects/active/` untouched > 30 days → *propose* `on-hold` (folder move +
status, the usual paired change) — apply only the ones confirmed;
- Observations > 30 → trim oldest into a dated `_agent/memory/episodic/`
overflow note (`observations-archive-YYYY.md` append), keeping the last 30
in place. Nothing is lost, the hot file stays small (compounds with #1).
- `/echo-health` output nudges: "run `sweep --decay` to review N aging items".
**Files.** `echo_ops.py` (`--ttl`/expires stamp), `echo_index.py`
(KIND_REQUIRED_FM unchanged; working gains optional expires), `vault_lint.py`,
`sweep.py`, scaffold working-memory template, SKILL.md (decay section),
routing-map note.
**Tests.** Mock: expired note proposed+archived on apply; observations trim keeps
last 30 and appends overflow; stale-active proposal list; dry-run writes nothing.
**Size/target.** Medium. Feature minor (1.7.x).
---
## 6. Contradiction / supersession handling
**Problem.** A new fact that contradicts a stored one lands beside it; both surface
in recall with no signal about which is current. Biggest remaining trust gap in the
write path.
**Design.**
- **Write side:** `capture --supersedes "<slug-or-quoted-line>"`:
- if it names an entity slug → the old entity gets `status: superseded` +
a `superseded_by:` frontmatter path; new note carries `supersedes:` back-link
in `## Related`. Recall's STATUS_FACTOR gains `superseded: 0.5`.
- if it quotes a Fact/Pattern or Observations line in operator-preferences →
the line is struck (`~~old~~ (superseded YYYY-MM-DD)`) and the new line
appended, one PATCH each, in the same invocation.
- **Detect side (deterministic assist, model decides):** `echo_reflect.classify`
gains a `conflict scan` — for each proposal, run its distinctive tokens
(`_sig_tokens`) against the Fact/Pattern + recent Observations lines; overlapping
lines are attached as `_conflicts: [...]` and the preview row renders
`update-or-conflict?` with the matched line(s). The model (which has the
conversation) decides supersede-vs-append; the script only guarantees the
question gets asked before the write.
- Lint: `superseded_by` pointing at a missing note → violation.
**Files.** `echo_ops.py`, `echo_reflect.py`, `echo_recall.py` (STATUS_FACTOR),
`echo.py` (flag plumbing), `vault_lint.py`, SKILL.md + operating-contract
(supersession is additive: strike, never delete).
**Tests.** Mock: supersede entity → status/backlinks both sides; supersede line →
strike + append idempotent; reflect preview flags a planted conflict; recall ranks
superseded below active twin.
**Size/target.** Medium. Feature minor (1.7.x/1.8) — after #5 (shares the
status-vocabulary touchpoints).
---
## 7. `session-end` bundle verb
**Problem.** Ending a session correctly is 4+ invocations (session-log PUT, Agent
Log append, heartbeat PUT, optional scope set, optional reflect apply). Cost per
session, and a partial failure leaves broken orientation state (log written,
heartbeat stale).
**Design.**
- `echo.py session-end <bundle.json> [--apply]`, bundle:
```json
{
"slug": "echo-mcp-spec",
"log_body": "<full session-log markdown, frontmatter included>",
"agent_log_line": "- 2026-07-28: wrote MCP spec [[...]]",
"scope": "optional new scope text",
"reflect": [ { ...PROPOSAL_SCHEMA... } ]
}
```
- Dry-run previews the whole plan (paths + reflect preview). `--apply`, in order,
under **one** lock acquisition: PUT `_agent/sessions/<date>-<HHMM>-<slug>.md`
(HHMM from `ECHO_NOW` env or wall clock, validated against the canonical regex) →
agent-log line (ensure_daily_log) → reflect proposals via capture → scope set if
given → heartbeat PUT **last** (it's the commit marker: a crash before it leaves
the previous pointer intact, never a dangling one).
- Reports per-step ok/queued/gated; offline → every step queues (rides on #3).
- Stop hook reason text updated to name the one command; `/echo-reflect` docs point
session logging at it. **Prerequisite for the MCP `echo_log_session` tool** —
build this verb first so CLI and MCP share one implementation.
**Files.** new `echo_session.py`, `echo.py` (subcommand), `echo_hook_stop.py`
(REASON text), SKILL.md (Session Logging section), commands/echo-reflect.md.
**Tests.** Mock: full bundle lands all five artifacts; heartbeat-last ordering
(inject failure mid-way → heartbeat untouched); dry-run writes nothing; offline
queues all steps.
**Size/target.** Small-medium. **1.6.x point release** (and before the MCP build).
---
## 8. Recall robustness — stemming + alias query expansion
**Problem.** BM25 is purely lexical: "deploy"/"deployment"/"deployed" are distinct
terms; a paraphrased query misses. The 8-query gold set can't see this class of miss.
**Design.**
- **Stemming:** a light suffix-stripper (Porter-lite: s/es/ed/ing/ly/tion/ment/ness
families, ~40 lines, pure stdlib) applied in `tokenize()` at **both** index and
query time. Index schema bump (stemmed postings) → auto-rebuild on next
recall/sweep, exactly like the 1→2 transition.
- **Alias query expansion:** at recall time, run the query against
`fuzzy_candidates()` (top 2, score ≥ 0.5); fold the matched entities' title +
alias tokens into the BM25 query as **down-weighted expansion terms** (×0.5 idf
contribution) so "the ECHO plugin" also scores `echo-memory`-alias vocabulary.
Exact-match seeding stays as-is; expansion never *creates* hits ranked above
genuine lexical matches without corpus support.
- **Eval:** grow the gold set with ≥6 paraphrase queries (morphological variants +
alias phrasings); publish before/after recall@5/MRR in the README table.
**Files.** `echo_recall.py` (tokenize, INDEX_SCHEMA, score, recall), new
`echo_stem.py` (pure function, unit-testable), `eval/run_eval.py` + gold set.
**Tests.** Stemmer unit table (word → stem, incl. non-cases); paraphrase gold
queries pass; schema-2 index discarded and rebuilt cleanly.
**Size/target.** Small-medium. Ride the **same index-schema train as #2/#4**.
---
## 9. Resolve-miss telemetry → alias suggestions
**Problem.** Every resolve that misses but ends in a confirmed candidate
(`--merge-into`, or the operator picking a candidate) is a free mention→entity
training pair, currently discarded.
**Design.**
- `echo_index.resolve()` miss + `fuzzy_candidates` non-empty → append one NDJSON
record to the local state dir (`~/.echo-memory/resolve-misses.ndjson`):
`{mention, candidates:[slugs], ts?}` (no wall clock needed — session date via
ECHO_TODAY). `capture --merge-into <slug>` appends the *confirmation* record
`{mention, resolved: slug}` (this already learns the alias immediately — that
path stays; the log covers the cases that never reach --merge-into).
- `sweep.py` (and `/echo-health`) read the log: any `{mention → slug}` pair
confirmed ≥2 times, or any mention that repeatedly fuzzy-matched a single
candidate ≥3 times without confirmation, is **proposed** as an alias
(`sweep --apply` writes it into the note's `aliases:` frontmatter + index, the
existing fold-back path). Log entries consumed on apply; file capped (rotate at
~500 lines).
- Privacy note: mentions can contain operator phrasing — the log lives in the
state dir (never the vault), same trust level as the offline queue.
**Files.** `echo_index.py` (log hooks), `echo_ops.py` (confirmation hook),
`sweep.py` (suggest/apply), `vault_lint.py` or sweep report (surface count),
SKILL.md one-liner.
**Tests.** Miss → record written; confirm ×2 → sweep proposes; apply folds alias +
consumes log; cap/rotation.
**Size/target.** Small. Feature minor, pairs naturally with #8.
---
## Sequencing (unified with TODO-1.6 and ROADMAP-2.0)
TODO-1.6 folds in as the opening train — the skills migration restructures the
files everything below touches, so it goes first. ROADMAP-2.0 stays a **pure
packaging major** (its own stated thesis) and is *not* folded into any feature
train; it slots between the quick wins and the MCP server so the MCP work lands
on the final packaging baseline (single command format, Gitea releases, one
manifest-verification pass).
| Release | Items | Rationale |
|---|---|---|
| **1.6.0** | TODO-1.6: skills-format migration (+ per-skill `allowed-tools`, `disable-model-invocation` on write-heavy entries, `$ECHO` block dedupe) · lint blind-spot fix (retired-dir/leaf-README) · plugin.json `homepage` | Restructures the command/SKILL files later work edits; allowed-tools removes prompt friction for every verb added below. Vault link repointing (TODO-1.6 §3) is vault data, not plugin code — any session. |
| **1.6.0 — SHIPPED 2026-07-28** | Skills migration + lint fix + homepage | Verified on desktop + CoWork. |
| **2.0.0 — SHIPPED 2026-07-28** | ROADMAP-2.0: commands/ deleted · release-based artifacts · gitignore · Codex note | Packaging-only major; skills-only install re-verified. |
| **2.1.0 quick wins — SHIPPED 2026-07-28** | #1 brief load · #3 offline capture · #7 session-end | All three landed with +19 e2e checks; see CHANGELOG 2.1.0. The MCP spec's Phase 0 (return-not-print) did NOT ride along — it's the first phase of the MCP build session. |
| **2.2** | MCP server (containerized, `echomcp.alwisp.com`) — `docs/MCP-SERVER-SPEC.md` | Needs Phase 0 (return-not-print refactor, first step of the build session). #7 session-end shipped, so `echo_log_session` wraps a real verb. Deployed via CI + PORT, not the plugin manifest. |
| **2.3 index train** | #2 local-first recall (CLI-path half) · #4 incremental sweep · #8 stemming+expansion | One schema-bump train; all touch the same index/sweep loop. The container already keeps its index warm in memory (spec §7), so #2 here covers the CLI fallback path; #4's fast sweep also becomes the container's background job. |
| **2.4 memory train** | #5 lifecycle/decay · #6 supersession · #9 resolve-miss learning | Capability tier; #6 after #5 (status vocabulary). |
+396
View File
@@ -0,0 +1,396 @@
# echo-mcp — MCP Server Build Spec (containerized)
> Status: **BUILT — shipped as 2.2.0, 2026-07-28** (`mcp-server/app.py`, 14 tools —
> the count below saying 13 undercounted the append/patch pair; e2e suite
> `eval/test_mcp_server.py`). This document remains the design record; §7.2 is the
> live v1.1 backlog. Companion plan: `docs/IMPROVEMENT-PLANS.md`.
>
> **Prerequisites before starting this build:**
> 1. The `session-end` verb (IMPROVEMENT-PLANS #7) — the MCP tool wraps it.
> 2. Phase 0 below (the return-not-print refactor) — lands in the plugin tree
> first and is independently shippable.
---
## 1. Why an MCP server
Every ECHO operation today rides through Bash: resolve `$ECHO` (with the CoWork
path-fallback snippet copy-pasted everywhere), quote a Python invocation, run
it, re-parse stdout. Costs:
- **Tokens** — each memory op carries bash scaffolding + output re-parsing; a
capture is ~3× the tokens of a typed tool call.
- **Fragility** — quoting hazards, the `${CLAUDE_PLUGIN_ROOT}` sandbox mismatch
(the whole 1.4.2 release), Windows `python3`-vs-`python` dispatch.
- **Reach** — surfaces without a Python-capable shell can't use ECHO at all.
- **Efficiency** — every CLI invocation is a cold Python start that re-reads the
entity index from the vault over HTTPS.
What the server does **not** replace: SKILL.md remains the procedure authority
(when to load, reconcile, search-first, third-person, etc.). The server replaces
the *mechanics* — tools are the verbs, the skill is the discipline.
## 2. Architecture: remote container (decided)
A standalone Docker container on the Unraid box (ALPHA), published behind
Cloudflare + NPM as **`echomcp.alwisp.com`**, speaking **MCP streamable HTTP
(stateless JSON)** with bearer-token auth. Chosen over the earlier local-stdio
draft for four reasons:
1. **The backend is already remote.** Clients round-trip to
`echoapi.alwisp.com` today; a server co-located with the vault host turns
every vault op into a LAN hop and collapses N client→vault calls into one
client→server call.
2. **Any device, no per-machine anything.** Desktop, CoWork sandbox, claude.ai
(custom connector), mobile — same URL + token. The `${CLAUDE_PLUGIN_ROOT}`
CoWork registration risk from the stdio draft disappears entirely.
3. **Credentials consolidate server-side.** The vault key lives in the
container env (PORT secret store); the image is credential-free; clients
hold only an MCP bearer token. The per-user baked-key builds (1.4.x) become
unnecessary for any surface that has MCP. Rotation = redeploy.
4. **Room to grow.** The plugin's pure-stdlib constraint exists because its
scripts run in arbitrary sandboxes. The container is ours: real SDK,
SQLite/in-memory indexes that stay warm, background jobs, embeddings later —
all without touching the plugin.
**What stays client-side (unchanged and shipped):** the skill + CLI path — it
is the fallback when the server or network is down, the offline queue + read
cache keep their per-machine role there, and the SessionStart/Stop **hooks stay
CLI-based** (hooks run shell commands regardless of MCP).
**Cost acknowledged:** a second service to run and an exposed auth surface.
Mitigations are existing infra: CF + NPM cert, long random bearer token, Kuma
monitor, PORT deploy/rollback. Server down ⇒ exactly today's behavior.
## 3. Decisions (made — don't relitigate in the build session)
| Decision | Choice | Why |
|---|---|---|
| Language | **Python** | Imports the existing `echo_*.py` modules in-process — the entire ops layer is reused, not wrapped. |
| SDK | **Official `mcp` Python SDK (FastMCP)** | The stdlib-only rule doesn't bind inside our own image. Hand-rolling the protocol (stdio draft) is no longer justified. |
| Transport | **Streamable HTTP, stateless JSON** | Remote, multi-client, simplest to scale; SSE/sessions explicitly avoided. |
| Auth | **Static bearer token** (`ECHO_MCP_TOKEN`), constant-time compare, 401 otherwise | Single-operator service; OAuth is overkill. Token from the PORT secret store. |
| Vault credentials | Container env: `ECHO_BASE`, `ECHO_KEY`, `ECHO_OWNER` (SECRET: refs in the PORT template) | `echo_config.py` already resolves env-first — zero code change. Image ships no secrets. |
| Server name / repo home | `echo-mcp`; lives in this repo under `mcp-server/` (own Dockerfile), deployed from `git.alwisp.com/jason/echo` CI | One canonical tree — the server vendors `skills/echo-memory/scripts/` into the image at build time; no second hand-maintained copy (the Codex-drift lesson). |
| Tool prefix | `echo_` | Namespace safety next to other servers. |
| Result shape | `structuredContent` (the `echo_output.envelope` dict) + a 13 line human text block | Envelope shape already exists. |
| Statefulness | Stateless protocol; warm in-process caches | See §7. |
| Version target | **2.2** — ALL prerequisites shipped 2026-07-28 (`session-end` in 2.1.0; **Phase 0 in 2.1.1**, contract-tested by `eval/test_ops_api.py`) | The build session starts directly at §5 (the server app). |
| Local stdio variant | **Dropped for v1** (documented, not built) | The CLI/skill fallback covers the no-server case; two transports = two test matrices for little gain. Phase 0 keeps the door open — the `*_op` core is transport-agnostic. |
| Result budgets | Every read tool takes an explicit size budget (`budget_chars` / `max_chars`) and truncates with a marker + "call X for more" | The single biggest token lever: one right-sized answer instead of follow-up fetches. |
| Tool profiles | `ECHO_MCP_TOOLS=core\|full` env: `core` exposes only load/recall/capture/triage_inbox/log_session/health | Tool schemas cost context on every surface that lists them; the claude.ai connector doesn't need the escape hatches. Desktop registers `full`. |
## 4. Phase 0 — the return-not-print refactor (prerequisite, lands in the plugin)
Unchanged from the stdio draft, and still first: the ops layer prints results
(even `--json` mode prints from inside `echo_ops.capture`) and returns exit
codes; the server needs return values.
1. Each high-level op gets a **core function returning an envelope dict**,
raising `echo.EchoError` on failure; CLI entry points become thin printing
wrappers:
| Module | New core fn | CLI wrapper keeps |
|---|---|---|
| `echo_ops` | `capture_op`, `resolve_op`, `link_op` | `capture/resolve/link` |
| `echo_recall` | `recall_op(query, limit)` | `recall` |
| `echo_triage` | `list_op()` · `route_op(proposals, apply)` | `list_inbox/apply` |
| `echo_reflect` | `apply_op(proposals, apply)` | `apply` |
| `echo.py` | `load_op(brief)` · `scope_show_op/scope_set_op` · `doctor_op` | `cmd_*` |
| `echo_session` (new, #7) | `session_end_op(bundle, apply)` | subcommand |
2. Special results are **data, not exit codes**: duplicate gate ⇒
`{ok: false, action: "duplicate-gate", candidates}` (CLI maps to exit 76);
offline queueing ⇒ `{ok: true, queued: true}`.
3. Helper chatter (`ok: PUT …`) moves behind a `notify(msg)` callback
defaulting to stderr; delete the `redirect_stdout` hack in `capture`.
4. Tests: existing suites pass unchanged; new unit tests hit `*_op` directly
against the mock and assert envelopes.
Shippable on its own as a 1.6.x refactor with zero behavior change.
## 5. Server application (`mcp-server/app.py`)
- **FastMCP** app, streamable HTTP, stateless. One `@mcp.tool` per tool in §6,
each a thin adapter: validate → call the `*_op` core → wrap.
- **Auth middleware**: require `Authorization: Bearer <ECHO_MCP_TOKEN>` on every
request (constant-time compare); 401 with no detail otherwise. Additionally
bind the app to the container interface only; exposure policy lives at
NPM/Cloudflare.
- **Result wrapper**:
```python
def tool_result(env: dict, summary: str) -> ...:
# content: [{type: "text", text: summary}] (13 lines, human)
# structuredContent: env (echo_output envelope)
# isError: not env.get("ok", True) — except the deliberate non-errors below
```
- **Error map** (tool-level results, never protocol errors):
| Condition | Result |
|---|---|
| Vault unreachable on a **read** | `isError: true` — "vault unreachable (Obsidian/REST plugin likely down); proceed without memory, writes will queue server-side". |
| Vault unreachable on a **write** | **Not an error**: `{ok: true, queued: true}` — the server-side outbox (§7) replays when the vault returns. |
| 404 | `isError: true`, `{code: "not-found", path}`. |
| Duplicate gate | **Not an error**: `{ok: false, action: "duplicate-gate", candidates}` + text naming the two resolutions (`merge_into` / `force`) — the model must decide, not blind-retry. |
| Lock held | `{ok: false, action: "lock-held", holder}`. |
| Misconfigured deployment (no `ECHO_BASE`/`ECHO_KEY`) | `isError: true` — "server deployment is missing vault credentials — operator: check the PORT template env". Startup also fails loudly (see §8 healthcheck). |
| Anything else | `isError: true`, first line only; traceback to the container log. |
- **Validation**: FastMCP/pydantic handles types/enums/required; add
`model_config = ConfigDict(extra="forbid")` so unknown params (model typos)
fail with a field-naming message.
## 6. Tool surface (13 tools)
Same surface as the stdio draft **minus `echo_configure`** — with a remote
server there is no per-machine config to import; credentials are
deployment-side. Descriptions are written for the model: what it does, when to
use it, what it returns, ≤3 sentences.
### 6.1 Orientation & read
- **`echo_load`** — cold-start orientation. `brief: bool = true`
(IMPROVEMENT-PLANS #1 digest; `false` = full six-file dump). Returns
`{sections: {marker, preferences, scope, last_session, today, inbox_count,
inbox_oldest_days}, queued_flushed, offline}`. Also flushes the server-side
outbox. `readOnlyHint: false` (flush), `openWorldHint: true`.
- **`echo_recall`** — `query: str`, `limit: int = 6 (120)`,
`include_linked: bool = true`, `budget_chars: int = 4000 (50020000)`. Returns
the `recall --json` shape (`primary` / `linked` with
path/score/type/updated/status/excerpt) **packed to the budget**: the server
allocates excerpt space by score, so the model gets the most relevant content
in one right-sized answer instead of follow-up fetches. Description: truncated
hits are marked — call `echo_get_note` for full content.
`readOnlyHint: true`, `idempotentHint: true`.
- **`echo_resolve`** — `mention: str` → match `{slug, path, kind, title,
aliases}` or `{match: false, suggest_slug, candidates}`. "Call before creating
any note by hand; `echo_capture` does this automatically." `readOnlyHint: true`.
- **`echo_get_note`** — `path: str` (vault-relative; reject `..`/leading `/`;
warn-not-block on paths outside `routing.json`); `section: str = null`
(a heading name — return only that section, e.g. `Status`);
`max_chars: int = 8000`. Returns `{path, content, frontmatter, truncated?}`.
`readOnlyHint: true`.
- **`echo_get_scope`** — `{scope, scope_updated, sessions_since}`; description
tells the model to confirm scope with the operator when `sessions_since ≥ 3`.
`readOnlyHint: true`.
- **`echo_set_scope`** — `scope: str`; atomic switch (history + replace +
stamp). `idempotentHint: true`.
- **`echo_health`** — `deep: bool = false`. Shallow = doctor checks (vault
reachability, auth, marker/schema, outbox depth). Deep = full `vault_lint`
→ `{violations: [{check, path, detail}]}`. `readOnlyHint: true`.
### 6.2 Write
- **`echo_capture`** — the default write (route + frontmatter + index +
auto-link + agent-log in one call). Params: `title` (req); `kind:
enum[person, company, concept, reference, meeting, project, area, semantic,
episodic, working, skill, decision]` (omit ⇒ inbox line); `body: str = ""`
(markdown, inline); `tags/aliases/sources: str[]`; `status: str`;
`date: YYYY-MM-DD` (meeting/decision); `domain: enum[business, personal,
learning, systems] = business` (area); `merge_into: str` (slug);
`force: bool = false`; `dry_run: bool = false`. Returns the capture envelope
(`action: created|updated|inbox|duplicate-gate`, `path`, `links_added`,
`near_duplicates?`, `candidates?`). Description spells out the gate contract
(never blind-retry; `merge_into` or confirmed `force`). `destructiveHint:
false`, `idempotentHint: true`.
- **`echo_link`** — `a`, `b` (paths **or resolvable names** — server resolves
via the index). Returns `{a, b, a_changed, b_changed}`. `idempotentHint: true`.
- **`echo_append_note`** — `path`, `line`; whole-line idempotent append.
- **`echo_patch_note`** — `path`, `operation: enum[append, prepend, replace]`,
`target_type: enum[heading, frontmatter, block]`, `target`, `content`.
Description carries the hard-won rules: heading targets are the full
`::`-delimited path from the H1; on a 400 invalid-target the server fetches
the document map and returns the actual heading list in the error — the worst
silent-loss failure becomes self-correcting. `destructiveHint: true`.
(Deliberately **no `echo_put_note` / `echo_delete_note`** in v1 — whole-file
overwrite and deletion stay behind the CLI + operator explicitness; session
logs go through `echo_log_session`.)
- **`echo_triage_inbox`** — `proposals: object[] = []` (PROPOSAL_SCHEMA +
optional `line`), `apply: bool = false`. Empty ⇒ structured listing; with
proposals ⇒ preview, then route + processing-log audit on `apply: true`.
"List → propose → preview → apply only after the operator confirms."
- **`echo_reflect`** — `proposals: object[]` (req), `apply: bool = false`. Same
preview/apply contract as the CLI; description restates: never apply without
the operator's go-ahead, never invent memories.
- **`echo_log_session`** — the session-end bundle (wraps `session_end_op`, #7):
`slug`, `log_body`, `agent_log_line`, `scope?: str`, `reflect?: object[]`,
`apply: bool = false`. Per-step results; heartbeat written last as the commit
marker.
### 6.3 v1.1 tools (specced now, built after v1 ships)
- **`echo_rollup_data`** — `period: enum[week, month]`, `date: YYYY-MM-DD`.
Assembles the rollup *digest data* in one call: open threads across
`projects/active/` (each note's `## Status` + `updated:`), inbox items aging
past 7 days, and the period's `## Scope History` entries. The model writes
the prose; the server does the N fetches. Same division of labor as reflect.
`readOnlyHint: true`.
- **`echo_note_history`** — `path: str` → `{versions: [{id, ts, bytes,
summary_line}]}` from the server's shadow write history (§7). `readOnlyHint:
true`.
- **`echo_restore_note`** — `path: str`, `version_id: str`,
`confirm: bool = false` (preview diff unless confirmed). The undo for a bad
`replace` PATCH or merge PUT. Description: operator confirmation required
before calling with `confirm: true`. `destructiveHint: true`.
Deliberately absent from v1 (documented in the server README): sweep,
bootstrap, migrate, lock/unlock (vault-wide maintenance stays operator-initiated
via slash commands — routine sweeps run as background jobs instead), raw
search/ls/map (recall/resolve/get_note cover reads; add only if transcripts
show the need), and reflect *extraction* (only the model has the conversation —
that division of labor is permanent, not a v1 scope cut).
## 7. Server internals — where the container earns its keep
### 7.1 v1
- **Warm entity index**: loaded once, invalidated on any index-writing tool and
on a short TTL (`ECHO_MCP_INDEX_TTL`, default 60s) to pick up other clients'
writes. Ends the per-invocation index re-read entirely.
- **Recall index in memory (+ SQLite file at `/data`)**: the BM25 index lives
in process memory, persisted to the container volume — recall answers in
milliseconds with **zero** vault round-trips. This *supersedes the
server-side half of IMPROVEMENT-PLANS #2*; #2's local-state-dir design still
applies to the CLI fallback path.
- **Note read cache**: short-TTL (3060s) body cache for `get_note` and
recall's neighbourhood expansion — repeated same-session reads of the same
files stop hitting the vault at all.
- **Per-path write serialization**: an internal per-path mutex serializes
conflicting MCP-path writes — since all server-mediated writes flow through
one process, the advisory-lock race window disappears for them. The server
still takes the vault advisory lock around index updates to coordinate with
CLI clients; idempotent appends stay as the last line of defense.
- **Server-side outbox**: the existing `echo_queue` pointed at `/data`
(`ECHO_STATE_DIR=/data`) — writes queue when the vault is down and flush on
recovery/load. One queue at the always-on host instead of per-laptop.
- **Concurrency**: FastMCP serves requests concurrently; guard the in-process
caches with a plain `threading.Lock`; write integrity per the bullet above.
- **Logging**: one line per call to stdout (container log): tool, ms,
ok/queued/gated/error. Never log bodies or keys.
### 7.2 v1.1 container dividends (backlog, in priority order)
- **Nightly vault backup** — the vault currently has **no disaster-recovery
story**; if the Obsidian host dies, memory is gone. A timer job walks the
vault via `read_many`, writes a dated zip to `/data/backups/` (rotate 30),
and reports the last-backup age in `/health`. Zero tokens; arguably the
highest-value item in this spec.
- **Shadow write history + undo** — before any PUT / PATCH-replace the server
passes through, snapshot the prior body to `/data/history/` (content-
addressed, per-path ring of ~20 versions). Backs the `echo_note_history` /
`echo_restore_note` tools (§6.3). The vault's additive philosophy finally
gets an undo for its residual risk: a bad `replace` or merge PUT.
- **Background maintenance jobs** — a timer thread running `sweep --fast`
(IMPROVEMENT-PLANS #4) hourly and the decay pass (#5) proposals weekly; both
publish findings to `/health` and the vault-health note rather than
auto-fixing.
- **Ops alerting** — outbox stuck > N hours, vault unreachable > 1h, new lint
violations after a background sweep ⇒ notify via the existing Unraid
notification plumbing (Kuma already watches `/health` for liveness).
Problems surface without anyone polling.
- **Embeddings (explicitly deferred)**: the container is where an embedding
recall tier would land later (model API or local), as an image upgrade —
no plugin change. Noted so nobody bolts it into the plugin.
## 8. Container & deployment
- **Image**: `python:3.12-slim` (not alpine — no musl surprises), `pip install
mcp` pinned, copy `skills/echo-memory/scripts/` + `mcp-server/` in.
**Dockerfile must be legacy-format** — the git.alwisp.com CI runner has no
BuildKit: no `# syntax=` line, no `RUN --mount`.
- **Healthcheck**: probe `http://127.0.0.1:<port>/health` — **127.0.0.1, not
localhost** (the ::1-vs-IPv4 lesson from cpas/memer/breedr). `/health`
(unauthenticated, no vault data): `{ok, vault_reachable, outbox_depth,
index_age_s}` — also the Kuma target.
- **Env** (via the PORT template, secrets as `SECRET:` refs): `ECHO_BASE`,
`ECHO_KEY`, `ECHO_OWNER`, `ECHO_MCP_TOKEN`, `ECHO_STATE_DIR=/data`,
`ECHO_MCP_TOOLS=full` (or `core` — see §3 tool profiles), optional
`ECHO_WORKERS/ECHO_TIMEOUT/ECHO_MCP_INDEX_TTL`. Volume: `/data`
(outbox + recall index + backups + shadow history).
- **Startup validation**: fail fast (non-zero exit) if `ECHO_BASE`/`ECHO_KEY`/
`ECHO_MCP_TOKEN` are missing — a misdeployed container should crash-loop
visibly, not serve errors.
- **CI/deploy**: Gitea workflow on push-to-main (tags/releases do **not**
autobuild on this host) builds the image and the PORT autodeploy webhook
rolls it out; `deploy.unraid.yml` manifest in `mcp-server/`. Publish
`echomcp.alwisp.com` via the usual CF + NPM flow; add the Kuma monitor.
- **Vault adjacency (confirmed: Obsidian runs on ALPHA)**: the Local REST
API's non-encrypted HTTP binding is enabled (2026-07-28) —
**`ECHO_BASE=http://10.2.0.35:27123`** — so the container talks to Obsidian
directly, never through the public `echoapi.alwisp.com` hairpin. All vault
chatter behind a tool call stays host/LAN-local; vault ops stop depending on
Cloudflare/DNS/NPM health entirely (the public chain is only needed for the
`echomcp` ingress). Why HTTP not HTTPS here: the REST API's HTTPS port
(27124) uses a self-signed cert, which `echo.py`'s default-verifying
`HTTPSConnection` rejects — cleartext on the internal network beats adding an
insecure-TLS flag to the client. **Exposure check (do once):** 27123 carries
the vault bearer token in cleartext, so it must not be reachable from
outside — confirm no router/firewall forward for 27123; the only public
doors stay HTTPS 443 via NPM (`echoapi` for the CLI fallback path, which
remains unchanged, and `echomcp` for this server).
## 9. Client registration & fallback
- **Claude Code / CoWork**: register as a remote MCP server —
`https://echomcp.alwisp.com/mcp` with the `Authorization: Bearer` header
(project or user scope). No plugin-manifest coupling; the plugin does **not**
register the server.
- **claude.ai**: **blocked in v1 (found at 2.2.0 deploy)** — claude.ai custom
connectors run the MCP OAuth flow (Dynamic Client Registration against the
server) and offer no custom-header option, so a bearer-only server cannot
connect. Fix = a minimal single-user OAuth layer on echo-mcp (SDK auth
interface: DCR + one-time operator approval + token issuance) — **v2.2.x
backlog, alongside the nightly vault backup**. Token-in-URL workarounds are
rejected (credentials leak into logs).
- **SKILL.md** gains: *"When the `echo_*` MCP tools are available, prefer them
for every operation they cover; the `$ECHO` CLI recipes are the fallback for
hosts without the connector or when the server is unreachable."* Procedures
unchanged.
- **Fallback matrix**: server up ⇒ tools. Server down, machine has the plugin ⇒
CLI path exactly as today (including its own offline queue). Both down ⇒
today's "vault unreachable, proceed without memory".
## 10. Multi-user note (Andy / Bryan / Gretchen / Arsalan)
One container serves **one vault**. The image is credential-free, so additional
users are additional deployments of the same image with their own env (their
vault endpoint/key + their own `ECHO_MCP_TOKEN`), e.g. `echomcp-bryan` on
another port/subdomain. Cheap, isolated, no code change. A single multi-tenant
server (token → vault map) is deliberately out of scope — that's CHORUS's
territory.
## 11. Testing & eval
1. **Protocol/integration tests** (`eval/test_mcp_server.py`): run the FastMCP
app in-process (or via `httpx` test client) against `mock_olrapi`; assert
initialize/tools-list schemas, auth (no token ⇒ 401; bad token ⇒ 401;
`/health` open), envelope purity on every tool.
2. **Tool behavior**: per tool — happy path, validation failure (unknown param
rejected with field name), vault-down (writes ⇒ `queued: true`, reads ⇒
actionable error), duplicate-gate flow (gate → `merge_into` retry succeeds).
3. **Parity**: `capture` via MCP and via CLI against twin mock vaults produce
byte-identical notes/index entries (guards Phase 0).
4. **Container smoke** (CI): build image, start with mock env, `/health` green,
one authed `tools/call` round-trip; healthcheck probes 127.0.0.1.
5. **MCP Inspector** manual pass against the deployed container.
6. **Eval set** (mcp-builder Phase 4): 10 read-only Q&A pairs against a seeded
mock vault exercising recall→get_note→resolve chains; `eval/mcp_eval.xml`.
## 12. Build-session plan of record
| Phase | Deliverable | Est. size |
|---|---|---|
| 0 | Return-not-print refactor in the plugin tree (`*_op` cores); suites green | ~⅓ the session |
| 1 | `mcp-server/`: FastMCP app, auth middleware, result wrapper, error map, `/health` | ~200 lines |
| 2 | Read tools (load, recall, resolve, get_note, scope×2, health) | wiring |
| 3 | Write tools (capture, link, append/patch, triage, reflect, log_session) | wiring + descriptions |
| 4 | Warm caches + `/data` outbox + startup validation | small |
| 5 | Dockerfile (legacy syntax) + CI workflow + PORT manifest + deploy + CF/NPM publish + Kuma | infra pass |
| 6 | Tests (§11) + Inspector pass + eval XML; SKILL.md/README/CHANGELOG | ~⅓ the session |
Definition of done: Inspector lists 13 tools against `echomcp.alwisp.com`; a
live session (desktop **and** CoWork) performs a full memory day — load →
recall → capture → gate → merge_into → triage → log_session — without one Bash
call; parity test green; container smoke test in CI; Kuma monitor green.
+99
View File
@@ -0,0 +1,99 @@
# ECHO Memory — Usage Guide
*Written for Bryan (and anyone else coming back to ECHO after a break). Current as of v1.5.1, July 2026.*
ECHO gives Claude persistent memory across sessions. Everything durable — people, companies, projects, decisions, preferences, session history — lives as markdown notes in an Obsidian vault, read and written over the Obsidian Local REST API. The plugin ships the whole toolchain (pure Python, stdlib only — works the same on Windows/macOS/Linux) plus the operating procedure Claude follows.
The short version of how to use it: **you mostly don't have to do anything.** Since v1.5 the system loads itself at session start and nudges itself to save at session end. The slash commands below are for when you want to drive it explicitly.
---
## One-time setup
If Jason handed you a **per-user baked plugin** (`echo-memory-1.x.x-bryan.plugin`), just install it. Your vault credentials are baked into the artifact — no config file, no key pasting, works on desktop and in every CoWork session. This is the normal path.
If you have the **generic plugin** instead, it will report `NOT CONFIGURED` on first use and ask for your config file. Install it once with:
```bash
python3 echo.py config import <your-config.json>
```
That writes `~/.claude/echo-memory/config.json` (owner, endpoint, API key). It survives plugin updates — one time per machine.
**Verify either way:** run `/echo-doctor`. Green across the board (Python, vault reachable, auth, bootstrapped, key source) means you're live.
---
## Day-to-day usage
### What happens automatically (v1.5+)
- **Session start** — a hook runs the cold-start load and injects your memory into context: your profile/preferences, current scope, the last session log, today's journal note, and inbox depth. You don't have to ask Claude to "load memory" anymore.
- **Session end** — if the session was substantive and nothing was saved, a hook nudges (once) to run `/echo-reflect` and write the session log. Reflection always previews before writing — nothing lands without your OK.
- **If the vault is unreachable**, writes queue to a local outbox and replay automatically next time it's up; reads fall back to a cached last-known-good. An outage degrades gracefully instead of erroring.
### The slash commands
| Command | When to use it |
|---|---|
| `/echo-load` | Manually load memory context. Rarely needed now — the session-start hook does this — but useful mid-session after a lot of vault writes, or if the hook didn't fire. |
| `/echo-save <thing>` | Save something right now: "remember the Henagar tower password is in the vault", "log that we decided X". Routes it to the right note automatically. |
| `/echo-recall <topic>` | Ask what memory knows about a topic. Returns ranked matches **plus their linked neighbourhood** — a project comes back with its people, decisions, and recent session mentions attached. |
| `/echo-reflect` | End-of-session sweep: extracts durable facts, decisions, and new entities from the conversation, shows you the proposed writes, applies on confirm. The main way memory grows. |
| `/echo-triage` | Empty the inbox: classifies each quick capture, previews destinations, routes accepted items with an audit trail. Run when load reports the inbox is getting deep. |
| `/echo-doctor` | Readiness check — Python, vault reachability, auth, bootstrap/schema version, where the key came from. First stop when anything seems off. |
| `/echo-health` | Full vault lint: routing violations, broken wikilinks, orphan notes, stale scope, incomplete frontmatter, index drift. Run occasionally or when things feel inconsistent. |
| `/echo-sweep` | Bring the vault up to the current plugin spec after an upgrade — rebuilds the entity index, backfills frontmatter, fixes one-way links. Dry-run by default; **run this once right after updating from an old version.** |
### A typical session
1. Start a conversation — memory loads itself. Claude knows who you are, what's active, and what happened last session.
2. Work normally. Say "save that" / "remember this" whenever something durable comes up (or `/echo-save` explicitly). Not sure where something belongs? Save it anyway — it lands in the inbox and `/echo-triage` sorts it later.
3. At the end, accept the reflection nudge (or run `/echo-reflect` yourself). Confirm the preview. Done — next session picks up from here.
---
## What the memory system can do (current capabilities)
**Smart, deduplicating capture.** One `capture` call routes content to its canonical home via an entity index, stamps complete frontmatter (status, tags, timestamps), cross-links every entity it mentions, and logs the write. Names resolve through aliases and fuzzy matching, so "echo memory" finds the `echo` project instead of spawning a duplicate note. If a new title strongly resembles an existing **same-kind** entity, a pre-write duplicate gate *stops* the write and shows you the candidates — you choose merge or create. Duplicates get blocked before they exist, not cleaned up after.
**Real retrieval, not keyword grep.** Recall fuses BM25 full-text search with graph expansion along note links, over entities *and* session logs *and* journal entries. Ranking is freshness- and status-aware: recently updated and `active` notes float up, `archived` sinks. You get a topic's connected neighbourhood, not one isolated file.
**Structured memory model.** Working (transient) / episodic (what happened, when) / semantic (durable facts and preferences) memory layers, plus a current-context scope, per-session logs, an append-only journal with rollups, and PARA-style projects/areas/resources/decisions. A machine-readable routing manifest defines what may be written where, and the linter enforces it.
**Reflection.** `/echo-reflect` turns a conversation into memory: extract → classify → dedup against the index → confidence-filter → preview → apply. Low-confidence items go to the inbox instead of polluting the graph.
**Durability and safety.** Every write is HTTP-status-checked and read-back-verified (a failed write fails loudly, never silently). Appends are idempotent — retries can't double-write. Offline writes queue and replay. A cooperative advisory lock keeps Claude Code and CoWork from trampling each other on the shared vault.
**Self-maintaining.** The vault bootstraps itself from an empty Obsidian vault, migrates its own schema on version bumps, and `/echo-sweep` + `/echo-health` keep the graph honest (index rebuilds, link symmetry, frontmatter backfill, orphan/broken-link detection).
**Fast.** Connection pooling + concurrent reads mean full-vault operations that used to time out in the sandbox now finish in under a second.
---
## What's new since 0.7 (your last consistent version)
You left off right after the toolchain got its executable spine (`echo.py`, routing manifest, linter). Since then, in rough order of what you'll actually notice:
- **You don't route anything by hand anymore.** 0.7 was "pick the path, write with the right verb." Now `capture`/`recall`/`resolve`/`link` do routing, frontmatter, indexing, and cross-linking in one call — and in practice you just talk to Claude and it uses them.
- **Memory loads and saves itself** (v1.5 hooks). No more remembering to run the loading procedure or write the session log.
- **Recall is actually good now** (v0.91.5): entity index + hybrid BM25/graph search + freshness/status ranking, spanning sessions and journal too.
- **Duplicates get blocked at write time** (v1.5 gate) instead of accumulating.
- **Offline resilience** (v1.0): vault down ≠ data lost; writes queue and replay.
- **Everything is pure Python** (v0.8): no bash, works identically on Windows.
- **Credentials moved out of the plugin** (v1.31.4): generic builds prompt once per machine; your baked build carries them invisibly. Treat a baked `.plugin` file like a password — it contains your vault key. Don't share or commit it.
**After installing the new version, run `/echo-doctor`, then `/echo-sweep` once** to bring your vault up to the current schema (it dry-runs first and shows the plan).
---
## Troubleshooting
| Symptom | Fix |
|---|---|
| `NOT CONFIGURED` banner / doctor shows red config | Generic build with no config on this machine — `echo.py config import <file>`, or ask Jason for your baked build. |
| Vault unreachable | Check that Obsidian + the Local REST API plugin are running on the backend and the endpoint is reachable. Meanwhile writes queue safely and replay on reconnect. |
| Save blocked with "duplicate gate" (exit 76) | Not an error — it found a likely-existing entity. Review the candidates: merge into the existing note, or force-create if it's genuinely distinct. |
| Memory feels stale or inconsistent | `/echo-health` to see what's off, `/echo-sweep` to repair index/links/frontmatter. |
| Inbox keeps getting mentioned at load | `/echo-triage` — one pass empties it. |
| Anything else weird | `/echo-doctor` first, then ask Jason. |
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -1,6 +1,8 @@
{
"name": "echo-memory",
"version": "1.5.1",
"version": "2.3.0",
"homepage": "https://git.alwisp.com/jason/echo",
"repository": "https://git.alwisp.com/jason/echo",
"description": "Persistent memory via the ECHO Obsidian vault over the Obsidian Local REST API. Cross-platform Python client: one-call capture/resolve/recall/link/triage over an entity index, hybrid BM25 + graph recall spanning entities + sessions/journal (recency/status-aware), a pre-write duplicate gate, complete-frontmatter capture, session hooks that self-fire load/reflect, offline write-ahead queue, lock-guarded concurrency, linter-enforced routing, and /echo-* commands.",
"author": {
"name": "Jason Stedwell"
@@ -1,5 +1,7 @@
---
name: echo-doctor
description: Check ECHO readiness — Python, vault reachability, auth, bootstrap/schema, and key source
allowed-tools: Bash(python3 *echo.py*), Bash(python *echo.py*), Bash(py -3 *echo.py*), Bash(ls /sessions/*)
---
Use the **echo-memory** skill to run a one-call readiness check before relying on memory.
@@ -1,6 +1,7 @@
---
name: echo-health
description: Run the ECHO vault-health linter and summarize any invariant violations
allowed-tools: Bash(python3 *vault_lint.py*), Bash(python *vault_lint.py*), Bash(ls /sessions/*), Bash(dirname *)
allowed-tools: Bash(python3 *vault_lint.py*), Bash(python *vault_lint.py*), Bash(py -3 *vault_lint.py*), Bash(ls /sessions/*), Bash(dirname *)
---
Run the bundled, read-only vault linter and report findings. Pass the conversation's current date (`ECHO_TODAY`) so stale/aging math uses the same clock the agent writes with:
@@ -1,5 +1,7 @@
---
name: echo-load
description: Load ECHO memory — cold-start context read (profile, scope, latest session, today, inbox)
allowed-tools: Bash(python3 *echo.py*), Bash(python *echo.py*), Bash(py -3 *echo.py*), Bash(ls /sessions/*)
---
Use the **echo-memory** skill to load memory now. Run the cold-start **Loading procedure** from `SKILL.md`: the six orientation reads (marker, operator-preferences, current-context, heartbeat, today's daily note, inbox) in **one call**, then do the load-time **reconcile** (inbox-depth + scope-drift) and surface it in a single line.
@@ -52,6 +52,19 @@ Executable logic ships under `scripts/` — **pure Python**, so the whole toolch
- `scripts/check_routing.py` — verifies the routing docs stay in sync with `routing.json` (dev/CI; offline)
- `scripts/bootstrap.py` / `scripts/migrate.py` — deterministic vault setup/repair and schema migration
## MCP tools first (when the echo-mcp connector is available)
When this session has the **`echo_*` MCP tools** (the deployed echo-mcp server —
`echo_load`, `echo_recall`, `echo_resolve`, `echo_get_note`, `echo_get_scope`/`echo_set_scope`,
`echo_health`, `echo_capture`, `echo_link`, `echo_append_note`/`echo_patch_note`,
`echo_triage_inbox`, `echo_reflect`, `echo_log_session`), **prefer them over the CLI
for every operation they cover** — typed calls, structured results, no path
resolution or quoting. The procedures in this skill (reconcile at load, search-first,
scope discipline, third person, preview-before-apply) are unchanged and apply to both
surfaces. A `duplicate-gate` tool result is the same contract as capture exit 76:
`merge_into` or confirmed `force`, never a blind retry. The CLI recipes below are the
fallback for hosts without the connector or when the server is unreachable.
## Bundled Tooling (prefer over raw curl)
All paths below are under `${CLAUDE_PLUGIN_ROOT}/skills/echo-memory/`. **Invoke with `python3`** (on Windows where that isn't on PATH, use `python` or `py -3`).
@@ -140,7 +153,7 @@ Load at the start of any substantive conversation — anything beyond a single q
### Loading procedure
**Run `python3 "$ECHO" load`** — one call that performs the six orientation reads below and prints each labeled section (404s on today's note / inbox show as absent, not errors; an absent marker is flagged). This replaces issuing six separate GETs by hand. The table documents what `load` reads and why:
**Run `python3 "$ECHO" load`** — one call that performs the six orientation reads below (fetched in parallel) and prints each labeled section (404s on today's note / inbox show as absent, not errors; an absent marker is flagged). This replaces issuing six separate GETs by hand. **`load --brief`** renders a token-budgeted digest instead — Fact/Pattern rules in full, the last ~10 Observations, scope + freshness, the last session's key sections, today's Agent Log lines, and the inbox as a *count* (`ECHO_LOAD_BUDGET`, default ~8000 chars, trims lowest-priority sections first). The SessionStart hook injects the brief form; `/echo-load` and this procedure use the full form. The table documents what `load` reads and why:
| # | GET | Notes |
|---|-----|-------|
@@ -345,6 +358,8 @@ Run it after `migrate.py` (which handles structural/schema changes) — or any t
The pass is cheap and pays for itself by catching drift before it requires a reorg. Write the findings as a digest; act on them only with the operator's go-ahead.
**Incremental maintenance (2.3).** `sweep.py --fast` is the cheap, index-only mode: it fetches just what's new, gone, or changed (content hashes) plus a rotating daily verification shard, so edits made directly in Obsidian or by other clients reach the entity + recall indexes without a full pass. It runs **automatically at load** when this machine's last sweep is older than `ECHO_FAST_SWEEP_DAYS` (default 7) — you rarely need to invoke it. The recall index itself is **local-first**: capture maintains a machine-local copy with zero vault round-trips; the vault's `recall-index.json` is a snapshot (sweep/session-end) used to seed fresh machines.
**Performance.** Full-vault scripts (`sweep.py`, `vault_lint.py`, recall rebuild) are connection-pooled (keep-alive) and read the whole vault concurrently via `echo.read_many`, so they finish in about a second on a few-hundred-note vault instead of timing out on hundreds of serial TLS handshakes. Tune with `ECHO_WORKERS` (default 8, bulk-read concurrency) and `ECHO_TIMEOUT` (default 30s, per-request). For a vault large enough that even the concurrent pass nears the tool timeout, run the script with the harness's background-execution option rather than blocking the turn.
## Daily Note — Agent Log
@@ -405,25 +420,40 @@ At the end of substantive conversations (ones that produced decisions, artifacts
**Filename format is canonical: `_agent/sessions/YYYY-MM-DD-HHMM-<slug>.md`.** Always include the four-digit local-time HHMM component — it makes filenames lexically sort in true chronological order, which is what Step 3 of loading relies on. Older session logs without the HHMM part exist; leave them alone, but every new one must use the full form.
Write the body (see `references/session-log-template.md`) to a file with the Write tool, then:
**Preferred — one call (`session-end`).** Write a bundle JSON with the Write tool and run it; it performs the whole session-end sequence under one lock, in order, with the **heartbeat written last as the commit marker** (a failure part-way leaves the previous pointer intact):
```json
{ "slug": "<kebab-slug>",
"log_body": "<full session-log markdown, frontmatter included — see references/session-log-template.md>",
"agent_log_line": "- <currentDate>: <one-line summary>",
"scope": "<new scope text — omit to leave unchanged>",
"reflect": [ { "title": "…", "kind": "…", "body": "…", "confidence": 0.9 } ] }
```
```bash
python3 "$ECHO" session-end bundle.json # dry-run: previews the full plan
ECHO_NOW=<HHMM> python3 "$ECHO" session-end bundle.json --apply
```
`--apply` writes: session log → Agent Log line → reflect proposals (via `capture`, gate-aware) → optional scope switch → heartbeat. Set `ECHO_NOW` to the conversation's local HHMM so the filename sorts truthfully. Reflect proposals still follow the reflect contract — preview with the dry-run and apply only with the operator's go-ahead. Offline, every step queues durably.
**Manual fallback** (only if `session-end` is unavailable) — the same sequence by hand:
```bash
python3 "$ECHO" put _agent/sessions/<currentDate>-HHMM-<slug>.md <bodyfile>
```
Then add a one-line entry to today's daily note via the **Daily Note — Agent Log** procedure above.
Finally, update the heartbeat pointer so the next session can orient in one read (this closes the loop with loading-procedure Step 4 — a pointer nobody writes is a pointer nobody can read). It is a single line, `<session-log-path> @ <ISO-timestamp>`; write it to a file and PUT it:
Then add a one-line entry to today's daily note via the **Daily Note — Agent Log** procedure above, and finally update the heartbeat pointer so the next session can orient in one read (a pointer nobody writes is a pointer nobody can read) — a single line, `<session-log-path> @ <ISO-timestamp>`:
```bash
python3 "$ECHO" put _agent/heartbeat/last-session.md <bodyfile>
```
`last-session.md` is overwritten (PUT) each session end — never appended, so it can't grow or duplicate.
`last-session.md` is overwritten (PUT) each session end — never appended, so it can't grow or duplicate. Keep the heartbeat write **last** in the manual sequence too.
## Vault Unreachable
If the API returns a connection error, timeout, or `502`, tell the operator once that the memory vault is unreachable (a `502` usually means Obsidian/the REST plugin is not running on the backend), then proceed without memory. Do not retry repeatedly.
If the API returns a connection error, timeout, or `502`, tell the operator once that the memory vault is unreachable (a `502` usually means Obsidian/the REST plugin is not running on the backend), then proceed — reads degrade to the last-known-good cache at load, and **writes queue durably**: the low-level verbs queue the request, and `capture` queues the whole operation as one semantic record that replays *through capture* (routing + duplicate gate re-run against the index at replay time) on the next reachable session. A queued capture that stops at the duplicate gate on replay is kept and flagged (`flush` lists it) rather than landed blind. Do not retry repeatedly.
## Style Rules
@@ -549,13 +549,12 @@ def extract_heading(markdown: str, heading: str) -> str:
return "\n".join(out).strip()
def cmd_scope(subcommand: str, text: str | None = None, as_json: bool = False) -> int:
def scope_show_op() -> dict:
"""Core scope-show (Phase 0): active scope + freshness + sessions-since, as data."""
path = "_agent/context/current-context.md"
status, body = request("GET", vault_url(path))
check(status, body, f"scope {subcommand}")
check(status, body, "scope show")
current = body.decode(errors="replace")
if subcommand == "show":
scope_text = extract_heading(current, "Scope")
scope_updated = ""
for line in current.splitlines():
@@ -572,23 +571,22 @@ def cmd_scope(subcommand: str, text: str | None = None, as_json: bool = False) -
sessions_since = sum(1 for f in files if f.endswith(".md") and f[:10] > scope_updated)
except json.JSONDecodeError:
pass
if as_json:
print(json.dumps({"ok": True, "action": "scope-show", "scope": scope_text,
"scope_updated": scope_updated or None,
"sessions_since": sessions_since}, ensure_ascii=False))
return 0
print("-- Active scope --")
print(scope_text)
print(f"scope_updated: {scope_updated or '<missing — drift cannot be detected; run scope set or repair>'}")
if sessions_since is not None:
print(f"sessions logged since: {sessions_since}")
return 0
return {"ok": True, "action": "scope-show", "scope": scope_text,
"scope_updated": scope_updated or None, "sessions_since": sessions_since}
if subcommand != "set":
raise EchoError("scope: use 'show' or 'set \"<text>\"'", 2)
def scope_set_op(text: str) -> dict:
"""Core scope-set (Phase 0): atomic switch (history + replace + stamp). Chatter
from the underlying PATCHes routes to stderr; returns the switch envelope."""
import contextlib
if not text:
raise EchoError("scope set needs the new scope text", 2)
path = "_agent/context/current-context.md"
status, body = request("GET", vault_url(path))
check(status, body, "scope set")
current = body.decode(errors="replace")
prior = extract_heading(current, "Scope").replace("\n", " ").strip()[:140] or "(prior scope)"
with contextlib.redirect_stdout(sys.stderr):
cmd_patch(path, "prepend", "heading", "Current Context::Scope History",
temp_file(f"- {today()}: {prior}\n".encode()))
cmd_patch(path, "replace", "heading", "Current Context::Scope",
@@ -601,13 +599,261 @@ def cmd_scope(subcommand: str, text: str | None = None, as_json: bool = False) -
"scope set: body switched, but scope_updated frontmatter is missing "
f"(run bootstrap.py to add it) [{exc}]"
)
print(f"ok: scope switched (prior archived to Scope History; scope_updated={today()})")
return {"ok": True, "action": "scope-set", "scope": text, "prior": prior,
"scope_updated": today()}
def cmd_scope(subcommand: str, text: str | None = None, as_json: bool = False) -> int:
if subcommand == "show":
env = scope_show_op()
if as_json:
print(json.dumps(env, ensure_ascii=False))
return 0
print("-- Active scope --")
print(env["scope"])
print(f"scope_updated: {env['scope_updated'] or '<missing — drift cannot be detected; run scope set or repair>'}")
if env["sessions_since"] is not None:
print(f"sessions logged since: {env['sessions_since']}")
return 0
if subcommand != "set":
raise EchoError("scope: use 'show' or 'set \"<text>\"'", 2)
env = scope_set_op(text or "")
print(f"ok: scope switched (prior archived to Scope History; scope_updated={env['scope_updated']})")
return 0
def cmd_load() -> int:
"""Cold-start orientation: the canonical 6 reads in one call. 404s on today's
daily note and the inbox are normal (printed as absent, not errors)."""
def _fetch_statuses(paths: list[str]) -> dict[str, tuple[int, bytes]]:
"""Concurrently GET vault paths keeping (status, body) per path — unlike read_many,
the 404-vs-offline distinction survives, which cmd_load's cache logic needs."""
if not paths:
return {}
workers = min(MAX_WORKERS, len(paths))
with ThreadPoolExecutor(max_workers=workers) as pool:
return dict(zip(paths, pool.map(lambda p: request("GET", vault_url(p)), paths)))
def _bullet_lines(text: str) -> list[str]:
return [ln for ln in text.splitlines() if ln.lstrip().startswith("- ")]
def _fm_line(text: str, field: str) -> str:
for ln in text.splitlines():
if ln.startswith(f"{field}:"):
return ln.split(":", 1)[1].strip().strip('"').strip("'")
return ""
def _render_brief(texts: dict[str, str | None], listing_files: list[str],
offline: bool) -> str:
"""The token-budgeted cold-start digest (2.1.0). Selection over compression: Fact /
Pattern rules in full, recent Observations only, scope + freshness (not the whole
history), the last session's key sections, today's Agent Log lines, inbox COUNT.
Sections are trimmed lowest-priority-first when over ECHO_LOAD_BUDGET."""
budget = int(os.environ.get("ECHO_LOAD_BUDGET", "8000"))
S: list[tuple[str, str]] = [] # (section-name, text) in display order
marker = texts.get("marker")
if marker is None:
S.append(("marker", "marker: _agent/echo-vault.md ABSENT — vault not bootstrapped "
"(run bootstrap.py before relying on memory)"))
else:
ver = _fm_line(marker, "schema_version") or "?"
S.append(("marker", f"marker: _agent/echo-vault.md — bootstrapped, schema_version {ver}"))
ctx = texts.get("context")
if ctx:
scope = extract_heading(ctx, "Scope") or "(empty)"
upd = _fm_line(ctx, "scope_updated") or "unknown"
since = sum(1 for f in listing_files if f.endswith(".md") and f[:10] > upd) \
if upd != "unknown" else None
head = f"scope (updated {upd}"
if since is not None:
head += f", {since} session(s) logged since"
S.append(("scope", head + "):\n" + scope))
else:
S.append(("scope", "scope: current-context.md absent"))
prefs = texts.get("preferences")
if prefs:
fact = extract_heading(prefs, "Fact / Pattern")
if fact:
S.append(("facts", "preferences — Fact / Pattern:\n" + fact))
obs = _bullet_lines(extract_heading(prefs, "Observations"))
if obs:
shown = obs[-10:]
head = f"preferences — Observations (last {len(shown)} of {len(obs)}):"
S.append(("observations", head + "\n" + "\n".join(shown)))
hb = texts.get("heartbeat")
if hb and hb.strip():
pointer = hb.strip().splitlines()[0]
sess_path = pointer.split(" @ ", 1)[0].strip()
part = [f"last session: {pointer}"]
log_text = texts.get("_pointed_session")
if log_text:
for h in ("Goal", "Decisions Made", "Open Threads", "Suggested Next Step"):
sec = extract_heading(log_text, h)
if sec and sec.strip("() \n"):
part.append(f" {h}: " + " / ".join(ln.strip() for ln in sec.splitlines()
if ln.strip())[:400])
elif sess_path:
part.append(" (session log not readable — see the path above)")
S.append(("session", "\n".join(part)))
elif listing_files:
recent = sorted((f for f in listing_files if f.endswith(".md")), reverse=True)[:3]
S.append(("session", "last session: heartbeat absent — recent logs:\n"
+ "\n".join(f" _agent/sessions/{f}" for f in recent)))
today_note = texts.get("today")
if today_note:
log_lines = _bullet_lines(extract_heading(today_note, "Agent Log"))
S.append(("agent-log", "today's Agent Log:\n" + ("\n".join(log_lines) or " (empty)")))
else:
S.append(("agent-log", "today's Agent Log: (no daily note yet)"))
inbox = texts.get("inbox")
if inbox:
items = re.findall(r"(?m)^\s*-\s*(\d{4}-\d{2}-\d{2})", inbox)
if items:
try:
oldest = (dt.date.fromisoformat(today()) - dt.date.fromisoformat(min(items))).days
S.append(("inbox", f"inbox: {len(items)} capture(s), oldest {oldest}d — "
"offer triage if any are older than ~7 days"))
except ValueError:
S.append(("inbox", f"inbox: {len(items)} capture(s)"))
else:
S.append(("inbox", "inbox: empty"))
else:
S.append(("inbox", "inbox: empty"))
def total() -> int:
return sum(len(t) + 2 for _, t in S)
# Over budget -> trim lowest-priority sections first, with explicit markers.
for name, keep in (("observations", 4), ("agent-log", 4), ("session", 9)):
if total() <= budget:
break
for i, (n, t) in enumerate(S):
if n == name:
lines = t.splitlines()
if len(lines) > keep:
S[i] = (n, "\n".join(lines[:keep]) + "\n (truncated — /echo-load for full)")
out = f"ECHO load (brief) — {today()}\n\n" + "\n\n".join(t for _, t in S)
if len(out) > budget:
out = out[:budget] + "\n(truncated — /echo-load for full)"
if offline:
out += ("\n\nNOTE: vault unreachable — context above is last-known-good cache; "
"writes this session will be queued and synced when the vault returns.")
return out
LOAD_TARGETS = [
("marker", "_agent/echo-vault.md"),
("preferences", "_agent/memory/semantic/operator-preferences.md"),
("context", "_agent/context/current-context.md"),
("heartbeat", "_agent/heartbeat/last-session.md"),
]
def _load_gather(targets):
"""Fetch the orientation reads (+ the sessions listing) in parallel and resolve
each to text via status/cache. Shared by cmd_load and load_op. Returns
(results, texts, listing_files, offline, marker_missing, heartbeat_absent)."""
import echo_queue
listing_path = "_agent/sessions/"
results = _fetch_statuses([p for _, p in targets] + [listing_path])
marker_missing = heartbeat_absent = offline = False
texts: dict[str, str | None] = {}
for label, path in targets:
status, body = results[path]
if status == 200:
echo_queue.cache_put(path, body) # refresh last-known-good for offline fallback
texts[label] = body.decode(errors="replace")
elif status == 0: # vault unreachable -> degrade to last-known-good cache
offline = True
cached = echo_queue.cache_get(path)
texts[label] = cached.decode(errors="replace") if cached is not None else None
else:
texts[label] = None
if status == 404:
if label == "marker":
marker_missing = True
if label == "heartbeat":
heartbeat_absent = True
lst_status, lst_body = results[listing_path]
listing_files: list[str] = []
if lst_status == 200:
try:
listing_files = list(json.loads(lst_body).get("files", []))
except json.JSONDecodeError:
pass
return results, texts, listing_files, offline, marker_missing, heartbeat_absent
def _auto_fast_sweep(offline: bool) -> str | None:
"""Opportunistic index maintenance at load (2.3): when this machine's last fast
sweep is older than ECHO_FAST_SWEEP_DAYS (default 7), run `sweep --fast` so the
local recall + entity indexes true up against edits made by Obsidian or other
clients. Best-effort; skipped offline; never raises."""
if offline:
return None
try:
import sweep as sweep_mod
days = float(os.environ.get("ECHO_FAST_SWEEP_DAYS", "7"))
age = sweep_mod.fast_sweep_age_days()
if age is not None and age < days:
return None
return sweep_mod.fast()
except Exception: # noqa: BLE001 — maintenance must never block a load
return None
def _fetch_pointed_session(texts: dict) -> None:
"""Follow the heartbeat pointer and stash the pointed session log for the digest."""
hb = texts.get("heartbeat")
if hb and hb.strip():
sess_path = hb.strip().splitlines()[0].split(" @ ", 1)[0].strip()
if sess_path:
st2, b2 = request("GET", vault_url(sess_path))
if st2 == 200:
texts["_pointed_session"] = b2.decode(errors="replace")
def load_op(brief: bool = True) -> dict:
"""Core load (Phase 0): the orientation reads as data — sections keyed by label,
plus offline/bootstrap flags, queue counts, recent sessions, and (with brief) the
rendered digest. Raises EchoError(78) when the machine is not configured."""
if not echo_config.is_configured(_CFG):
raise EchoError("NOT CONFIGURED — no usable ECHO key file on this machine "
f"(expected at {echo_config.config_path()})", 78)
import echo_queue
synced = flagged = 0
try:
synced = echo_queue.flush()
flagged = len(echo_queue.needs_attention())
except Exception: # noqa: BLE001 — queue upkeep must never block a load
pass
targets = LOAD_TARGETS + [("today", f"journal/daily/{today()}.md"),
("inbox", "inbox/captures/inbox.md")]
_, texts, listing_files, offline, marker_missing, _ = _load_gather(targets)
swept = _auto_fast_sweep(offline)
data = {"ok": True, "action": "load", "offline": offline,
"marker_missing": marker_missing, "synced": synced,
"needs_attention": flagged, "fast_sweep": swept,
"sections": {k: v for k, v in texts.items() if not k.startswith("_")},
"recent_sessions": sorted((f for f in listing_files if f.endswith(".md")),
reverse=True)[:5]}
if brief:
_fetch_pointed_session(texts)
data["brief"] = _render_brief(texts, listing_files, offline)
return data
def cmd_load(brief: bool = False) -> int:
"""Cold-start orientation: the canonical 6 reads in one call (fetched in parallel).
404s on today's daily note and the inbox are normal (printed as absent, not errors).
`brief` renders the token-budgeted digest the SessionStart hook injects; the default
full mode (and /echo-load) prints the raw sections unchanged."""
if not echo_config.is_configured(_CFG):
print("echo: NOT CONFIGURED — this machine has no usable ECHO key file yet.")
print(f"Expected at: {echo_config.config_path()}")
@@ -617,49 +863,51 @@ def cmd_load() -> int:
print(" python3 echo.py config set --owner \"\" --endpoint \"https://…\" --key \"\"")
print("Then re-run load. (Memory is unavailable until configured.)")
return 78 # distinct: configuration required
targets = [
("marker", "_agent/echo-vault.md"),
("preferences", "_agent/memory/semantic/operator-preferences.md"),
("context", "_agent/context/current-context.md"),
("heartbeat", "_agent/heartbeat/last-session.md"),
("today", f"journal/daily/{today()}.md"),
("inbox", "inbox/captures/inbox.md"),
]
targets = LOAD_TARGETS + [("today", f"journal/daily/{today()}.md"),
("inbox", "inbox/captures/inbox.md")]
import echo_queue
# H2: sync any writes queued during a prior outage, best-effort and quiet on empty.
try:
synced = echo_queue.flush()
if synced:
print(f"(synced {synced} write(s) queued during a prior offline session)\n")
flagged = echo_queue.needs_attention()
if flagged:
print(f"NOTE: {len(flagged)} queued write(s) need attention (e.g. a capture "
"stopped at the duplicate gate on replay) — resolve with capture "
"--merge-into/--force; `echo.py flush` re-lists them.\n")
except Exception as exc: # noqa: BLE001 — never let queue upkeep block a load
print(f"(queue flush skipped: {exc})\n", file=sys.stderr)
marker_missing = False
heartbeat_absent = False
offline = False
results, texts, listing_files, offline, marker_missing, heartbeat_absent = \
_load_gather(targets)
swept = _auto_fast_sweep(offline)
if swept:
print(f"({swept})\n")
if brief:
_fetch_pointed_session(texts)
print(_render_brief(texts, listing_files, offline))
return 0
# ---- full mode: raw sections, output format unchanged ----
for label, path in targets:
status, body = request("GET", vault_url(path))
status, body = results[path]
if status == 200:
echo_queue.cache_put(path, body) # refresh last-known-good for offline fallback
print(f"===== {label}: {path} (HTTP 200) =====")
text = body.decode(errors="replace")
text = texts[label] or ""
sys.stdout.write(text)
if not text.endswith("\n"):
sys.stdout.write("\n")
elif status == 404:
print(f"===== {label}: {path} (HTTP 404) =====")
print("(absent — fine)")
if label == "marker":
marker_missing = True
if label == "heartbeat":
heartbeat_absent = True
elif status == 0: # vault unreachable -> degrade to last-known-good cache
offline = True
cached = echo_queue.cache_get(path)
if cached is not None:
elif status == 0:
if texts[label] is not None:
print(f"===== {label}: {path} (OFFLINE — serving stale cache) =====")
sys.stdout.write(cached.decode(errors="replace"))
if not cached.endswith(b"\n"):
sys.stdout.write(texts[label])
if not texts[label].endswith("\n"):
sys.stdout.write("\n")
else:
print(f"===== {label}: {path} (OFFLINE — no cache) =====")
@@ -669,15 +917,9 @@ def cmd_load() -> int:
print(f"(error HTTP {status}: {body.decode(errors='replace')[:200]})")
print()
# M3: heartbeat pointer missing/stale -> fall back to the recent sessions listing
# (matches the documented loading procedure) so orientation works without the pointer.
if heartbeat_absent and not offline:
st, body = request("GET", vault_url("_agent/sessions/"))
if st == 200:
try:
files = sorted((f for f in json.loads(body).get("files", []) if f.endswith(".md")),
reverse=True)
except json.JSONDecodeError:
files = []
# (already fetched in the batch) so orientation works without the pointer.
if heartbeat_absent and not offline and listing_files:
files = sorted((f for f in listing_files if f.endswith(".md")), reverse=True)
if files:
print("===== recent sessions (heartbeat absent — fallback) =====")
for f in files[:5]:
@@ -732,7 +974,8 @@ def cmd_config(args) -> int:
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Validated cross-platform ECHO vault client")
sub = parser.add_subparsers(dest="cmd", required=True)
sub.add_parser("load")
p = sub.add_parser("load")
p.add_argument("--brief", action="store_true") # token-budgeted digest (hook default)
sub.add_parser("flush") # H2: replay writes queued during a prior outage
sub.add_parser("doctor") # M3: one-call readiness check
sub.add_parser("get").add_argument("path")
@@ -761,6 +1004,9 @@ def build_parser() -> argparse.ArgumentParser:
p = sub.add_parser("reflect") # H5: apply a JSON proposal set (dry-run unless --apply)
p.add_argument("file", nargs="?")
p.add_argument("--apply", action="store_true")
p = sub.add_parser("session-end") # 2.1.0: one-call session end (log+agent-log+reflect+scope+heartbeat)
p.add_argument("file", nargs="?") # bundle JSON; - or stdin
p.add_argument("--apply", action="store_true")
p = sub.add_parser("triage") # one-tap inbox triage (reflect pipeline + audit log)
p.add_argument("file", nargs="?") # proposals JSON; omit to list
p.add_argument("--list", action="store_true", dest="list_inbox")
@@ -796,21 +1042,27 @@ def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.cmd == "load":
return cmd_load()
return cmd_load(brief=args.brief)
if args.cmd == "doctor":
import echo_doctor
return echo_doctor.run()
if args.cmd == "flush":
import echo_queue
n = echo_queue.flush()
remaining = len(echo_queue.pending())
remaining = echo_queue.pending()
flagged = [r for r in remaining if r.get("needs_attention")]
if n:
print(f"ok: flushed {n} queued write(s)"
+ (f"; {remaining} still queued (vault unreachable)" if remaining else ""))
+ (f"; {len(remaining)} still queued" if remaining else ""))
elif remaining:
print(f"ok: {remaining} write(s) still queued (vault unreachable)")
print(f"ok: {len(remaining)} write(s) still queued")
else:
print("ok: queue empty")
for r in flagged:
what = (f"capture '{(r.get('args') or {}).get('title')}'" if r.get("op") == "capture"
else f"{r.get('method')} {r.get('url')}")
print(f" needs attention ({r['needs_attention']}): {what} — resolve "
"(e.g. capture --merge-into <slug> / --force), it will not auto-land")
return 0
if args.cmd == "config":
return cmd_config(args)
@@ -852,6 +1104,14 @@ def main(argv: list[str] | None = None) -> int:
if not isinstance(proposals, list):
raise EchoError("reflect: proposals must be a JSON array of objects", 2)
return echo_reflect.apply(proposals, confirm=args.apply)
if args.cmd == "session-end":
import echo_session
raw = read_body(args.file).decode("utf-8", errors="replace")
try:
bundle = json.loads(raw)
except json.JSONDecodeError as exc:
raise EchoError(f"session-end: bundle must be a JSON object ({exc})", 2)
return echo_session.session_end(bundle, apply=args.apply)
if args.cmd == "triage":
import echo_triage
if args.list_inbox or not args.file:
@@ -880,9 +1140,12 @@ def main(argv: list[str] | None = None) -> int:
date=args.date, domain=args.domain, inbox=args.inbox, no_log=args.no_log,
as_json=args.json, dry_run=args.dry_run,
force=args.force, merge_into=args.merge_into)
except EchoError as exc:
except RuntimeError as exc:
# Catches EchoError from THIS module and from helper modules' own `import echo`
# twin (when echo.py runs as __main__ the two class objects differ, but both
# subclass RuntimeError and carry .code).
print(f"echo.py: {exc}", file=sys.stderr)
return exc.code
return getattr(exc, "code", 1)
return 2
@@ -22,17 +22,18 @@ import echo # noqa: E402
MIN_PY = (3, 9)
def run() -> int:
reds = 0
def run_op() -> dict:
"""Core doctor (Phase 0): the readiness checks as data — {checks: [{ok, label,
detail}], endpoint, fatal, ok}. `fatal` names an early-exit condition (not
configured / unreachable) after which later checks were skipped."""
checks: list[dict] = []
def line(ok: bool, label: str, detail: str = "") -> None:
nonlocal reds
reds += 0 if ok else 1
print(f" [{'OK ' if ok else 'RED'}] {label}" + (f"{detail}" if detail else ""))
checks.append({"ok": ok, "label": label, "detail": detail})
import echo_config
cfg = echo_config.load()
print(f"echo doctor — endpoint {cfg['endpoint'] or '(not configured)'}")
fatal = None
# 1. interpreter
line(sys.version_info >= MIN_PY, f"python >= {MIN_PY[0]}.{MIN_PY[1]}",
@@ -53,18 +54,16 @@ def run() -> int:
line(bool(cfg["owner"]), "vault owner set",
cfg["owner"] if cfg["owner"] else "optional, but recommended for third-person writes")
if not echo_config.is_configured(cfg):
print("\ndoctor: not configured — ask the operator for their key file and install it "
"(`echo.py config import <file>`, or `config set --owner … --endpoint … --key …`), "
"then re-run.")
return 1
fatal = "not-configured"
else:
# 3. reachability + auth + marker, in one GET of the bootstrap marker
status, body = echo.request("GET", echo.vault_url("_agent/echo-vault.md"))
if status == 0:
line(False, "vault reachable", body.decode(errors="replace")[:120])
print("\ndoctor: endpoint unreachable — is Obsidian + the Local REST API running?")
return 1
line(status not in (401, 403), "auth accepted", f"HTTP {status}" if status in (401, 403) else "")
fatal = "unreachable"
else:
line(status not in (401, 403), "auth accepted",
f"HTTP {status}" if status in (401, 403) else "")
if status == 404:
line(False, "vault bootstrapped", "marker absent — run bootstrap.py")
elif status == 200:
@@ -75,11 +74,42 @@ def run() -> int:
else:
line(status < 400, "marker fetch", f"HTTP {status}")
# 4. invariants pointer.
print(" [i] invariants — run `/echo-health` (vault_lint.py) for the full check")
# Index freshness (2.3) — informational, never red: how stale this machine's
# local indexes are (the fast sweep trues them up at load past 7 days).
if fatal is None:
try:
import sweep as sweep_mod
age = sweep_mod.fast_sweep_age_days()
line(True, "index freshness",
f"last local sweep {age:.1f}d ago" if age is not None
else "no local sweep yet — runs automatically at the next load")
except Exception: # noqa: BLE001
pass
print(f"\ndoctor: {'all green' if reds == 0 else f'{reds} issue(s) — see RED above'}")
return 1 if reds else 0
reds = sum(1 for c in checks if not c["ok"])
return {"ok": reds == 0 and fatal is None, "action": "doctor",
"endpoint": cfg["endpoint"], "fatal": fatal, "reds": reds, "checks": checks}
def run() -> int:
env = run_op()
print(f"echo doctor — endpoint {env['endpoint'] or '(not configured)'}")
for c in env["checks"]:
print(f" [{'OK ' if c['ok'] else 'RED'}] {c['label']}"
+ (f"{c['detail']}" if c["detail"] else ""))
if env["fatal"] == "not-configured":
print("\ndoctor: not configured — ask the operator for their key file and install it "
"(`echo.py config import <file>`, or `config set --owner … --endpoint … --key …`), "
"then re-run.")
return 1
if env["fatal"] == "unreachable":
print("\ndoctor: endpoint unreachable — is Obsidian + the Local REST API running?")
return 1
# invariants pointer.
print(" [i] invariants — run `/echo-health` (vault_lint.py) for the full check")
summary = "all green" if env["reds"] == 0 else f"{env['reds']} issue(s) — see RED above"
print(f"\ndoctor: {summary}")
return 1 if env["reds"] else 0
if __name__ == "__main__":
@@ -42,7 +42,7 @@ def main() -> int:
try:
import echo
with contextlib.redirect_stdout(buf):
rc = echo.cmd_load()
rc = echo.cmd_load(brief=True) # token-budgeted digest; /echo-load stays full
text = buf.getvalue()
if rc == 78:
context = ("[echo-memory] ECHO is NOT CONFIGURED on this machine. Before "
@@ -30,9 +30,11 @@ MIN_TURNS = int(os.environ.get("ECHO_REFLECT_MIN_TURNS", "5"))
REASON = (
"[echo-memory] Session-end reflection has not run yet. If this session produced "
"durable facts, decisions, commitments, or artifacts worth remembering: run the "
"/echo-reflect flow (extract -> preview -> apply only with the operator's confirm) "
"and write the session log + heartbeat per the echo-memory skill. If nothing "
"durable emerged, simply finish — do not invent memories. "
"/echo-reflect flow (extract -> preview -> apply only with the operator's confirm), "
"then finish with ONE call — `echo.py session-end <bundle.json> --apply` — which "
"writes the session log, Agent Log line, reflect proposals, optional scope switch, "
"and the heartbeat (last, as the commit marker) together. If nothing durable "
"emerged, simply finish — do not invent memories. "
"(This reminder fires once per session.)"
)
@@ -64,6 +66,7 @@ def already_reflected(raw: str) -> bool:
"""Transcript evidence that reflection or session logging already happened."""
return ("/echo-reflect" in raw
or re.search(r'echo\.py"?\s+reflect\b', raw) is not None
or re.search(r'echo\.py"?\s+session-end\b', raw) is not None
or "put _agent/sessions/" in raw
or "put _agent/heartbeat/last-session.md" in raw)
@@ -168,7 +168,10 @@ def kind_for_path(path: str) -> str | None:
def empty_index() -> dict:
return {"schema": 1, "updated": echo.today(), "entities": {}}
# Schema 2 (2.3): entries may carry `h`, a short content hash of the note body —
# the incremental sweep's change detector. Tolerated-absent (schema-1 entries
# simply get fetched and hashed on their first fast sweep).
return {"schema": 2, "updated": echo.today(), "entities": {}}
def load() -> dict:
@@ -287,13 +290,20 @@ def gate_candidates(index: dict, mention: str, kind: str | None = None,
return out
def content_hash(text: str) -> str:
import hashlib
return hashlib.sha1(str(text).encode("utf-8", "replace")).hexdigest()[:16]
def upsert(index: dict, slug: str, path: str, kind: str, title: str | None = None,
aliases=None) -> dict:
aliases=None, h: str | None = None) -> dict:
ents = index.setdefault("entities", {})
e = ents.get(slug, {})
e["path"] = path
e["kind"] = kind
e["title"] = title or e.get("title") or slug
if h:
e["h"] = h
# Keep any prior + explicit aliases, and ALWAYS fold in the safe title variants so a
# hyphen/space/case form of the name resolves even if no one passed --aliases. Drop any
# alias that just equals the slug (resolve already checks the slug).
@@ -37,12 +37,13 @@ LOG_HEADING = {
# ---------------------------------------------------------------- resolve -----
def resolve(mention: str) -> int:
def resolve_op(mention: str) -> dict:
"""Core resolve: mention -> match/candidates dict. No printing (Phase 0 — the same
return value backs the CLI wrapper and the MCP `echo_resolve` tool)."""
index = idx_mod.load()
slug, e = idx_mod.resolve(index, mention)
if e:
print(json.dumps({"match": True, "slug": slug, **e}, ensure_ascii=False, indent=2))
return 0
return {"match": True, "slug": slug, **e}
# No exact match — surface fuzzy candidates so a shortened/expanded name (e.g. "echo
# memory" for the project "echo") reveals the existing note instead of looking absent.
cands = idx_mod.fuzzy_candidates(index, mention)
@@ -54,21 +55,30 @@ def resolve(mention: str) -> int:
"these before creating a new note; reuse the right path or `echo.py link`.")
else:
out["note"] = "no index entry — derive a path with the right --kind and create it"
print(json.dumps(out, ensure_ascii=False, indent=2))
return out
def resolve(mention: str) -> int:
print(json.dumps(resolve_op(mention), ensure_ascii=False, indent=2))
return 0
# ------------------------------------------------------------------- link -----
def link(a_path: str, b_path: str, as_json: bool = False) -> int:
a_changed, b_changed = links.link_bidirectional(a_path, b_path)
if as_json:
def link_op(a_path: str, b_path: str) -> dict:
"""Core link: reciprocal `## Related` links, returns the change envelope."""
import echo_output
env = echo_output.envelope("link", {"a": a_path, "b": b_path,
a_changed, b_changed = links.link_bidirectional(a_path, b_path)
return echo_output.envelope("link", {"a": a_path, "b": b_path,
"a_changed": a_changed, "b_changed": b_changed})
def link(a_path: str, b_path: str, as_json: bool = False) -> int:
env = link_op(a_path, b_path)
if as_json:
print(json.dumps(env, ensure_ascii=False))
return 0
print(f"ok: linked {a_path} <-> {b_path} "
f"(added: {'A' if a_changed else '-'}{'B' if b_changed else '-'})")
print(f"ok: linked {env['a']} <-> {env['b']} "
f"(added: {'A' if env['a_changed'] else '-'}{'B' if env['b_changed'] else '-'})")
return 0
@@ -83,8 +93,13 @@ def recall(query, limit: int = 8, as_json: bool = False) -> int:
# ----------------------------------------------------------- agent log -------
def ensure_daily_log(line: str) -> None:
"""Resilient: ensure today's daily note + its `## Agent Log` heading exist, then
idempotently append the line. Best-effort — never raises into the caller."""
idempotently append the line. Best-effort — never raises into the caller. Writes go
through the offline queue (2.1.0), so a mid-session outage queues the line instead
of silently dropping it. Offline edge accepted: if the daily note still doesn't
exist at replay time, the queued PATCH 400-drops with a loud warning — the agent-log
line is a convenience trace, not the memory itself."""
try:
import echo_queue
date = echo.today()
path = f"journal/daily/{date}.md"
status, body = echo.request("GET", echo.vault_url(path))
@@ -92,18 +107,19 @@ def ensure_daily_log(line: str) -> None:
ts, tb = echo.request("GET", echo.vault_url("journal/templates/daily-note-template.md"))
tmpl = tb.decode(errors="replace") if ts == 200 else f"# {date}\n\n## Agent Log\n\n## Related\n"
tmpl = tmpl.replace("{{date:YYYY-MM-DD}}", date).replace("{{DATE}}", date)
echo.request("PUT", echo.vault_url(path), data=tmpl.encode(),
echo_queue.safe_request("PUT", echo.vault_url(path), data=tmpl.encode(),
headers={"Content-Type": "text/markdown"})
text = tmpl
else:
text = body.decode(errors="replace")
if not re.search(r"(?m)^## Agent Log\s*$", text):
echo.request("POST", echo.vault_url(path), data=b"\n\n## Agent Log\n",
text = body.decode(errors="replace") if status == 200 else ""
if status == 200 and not re.search(r"(?m)^## Agent Log\s*$", text):
echo_queue.safe_request("POST", echo.vault_url(path), data=b"\n\n## Agent Log\n",
headers={"Content-Type": "text/markdown"})
status, body = echo.request("GET", echo.vault_url(path))
if status == 200 and line in body.decode(errors="replace"):
st2, body2 = echo.request("GET", echo.vault_url(path))
if st2 == 200 and line in body2.decode(errors="replace"):
return
echo.request("PATCH", echo.vault_url(path),
echo_queue.safe_request(
"PATCH", echo.vault_url(path),
data=echo.normalize_patch_body((line + "\n").encode(), "append", "heading"),
headers={"Operation": "append", "Target-Type": "heading",
"Target": f"{date}::Agent Log", "Content-Type": "text/markdown"})
@@ -148,81 +164,106 @@ def _dated_block(today_s: str, body_text: str) -> tuple[str, str]:
def _append_to_existing(path: str, kind: str, today_s: str, body_text: str) -> None:
# Writes route through the offline queue (2.1.0) so a mid-capture outage queues the
# update instead of silently losing it. (A fully-offline capture never reaches here —
# the short-circuit in capture() queues the whole op as one semantic record.)
import echo_queue
text = links.get_text(path) or ""
h1 = links.first_h1(text) or path.rsplit("/", 1)[-1][:-3]
heading = LOG_HEADING.get(kind, "Notes")
if not re.search(rf"(?m)^##\s+{re.escape(heading)}\s*$", text):
echo.request("POST", echo.vault_url(path), data=f"\n\n## {heading}\n".encode(),
echo_queue.safe_request("POST", echo.vault_url(path), data=f"\n\n## {heading}\n".encode(),
headers={"Content-Type": "text/markdown"})
bullet, block = _dated_block(today_s, body_text)
status, body = echo.request("GET", echo.vault_url(path))
if status == 200 and bullet in body.decode(errors="replace"):
return
echo.request("PATCH", echo.vault_url(path),
echo_queue.safe_request(
"PATCH", echo.vault_url(path),
data=echo.normalize_patch_body((block + "\n").encode(), "append", "heading"),
headers={"Operation": "append", "Target-Type": "heading",
"Target": f"{h1}::{heading}", "Content-Type": "text/markdown"})
echo.cmd_fm(path, "updated", json.dumps(today_s))
def capture(kind: str | None, title: str, file_arg: str | None, status_v: str = "",
def capture_op(kind: str | None, title: str, file_arg: str | None = None, status_v: str = "",
aliases=None, sources=None, tags=None, date: str | None = None,
domain: str = "business", inbox: bool = False, no_log: bool = False,
as_json: bool = False, dry_run: bool = False, force: bool = False,
merge_into: str | None = None) -> int:
dry_run: bool = False, force: bool = False,
merge_into: str | None = None, body_text: str | None = None) -> dict:
"""Core capture (Phase 0): route + frontmatter + index + auto-link + agent-log,
returning an envelope dict — never printing to stdout (helper chatter from the
low-level verbs is redirected to stderr). Special outcomes are DATA, not exit
codes: `duplicate-gate` (ok=false, candidates), `queued:capture` (queued=true),
`dry-run:*`. Raises echo.EchoError on hard failure. `body_text` may be passed
directly (MCP path); otherwise it is read from file_arg/stdin."""
import contextlib
import io
import echo_output
import echo_quality
real_stdout = sys.stdout
def quiet():
# M4: in --json mode, swallow the helper "ok:" chatter so stdout is clean JSON.
return contextlib.redirect_stdout(io.StringIO()) if as_json else contextlib.nullcontext()
def done(action: str, path: str, links: int = 0, ok: bool = True, dry: bool = False,
near=None) -> int:
if as_json:
act = f"dry-run:{action}" if dry else action
data = {"kind": kind, "path": path, "title": title, "links_added": links}
if near:
data["near_duplicates"] = near
env = echo_output.envelope(act, data, ok=ok)
print(json.dumps(env, ensure_ascii=False), file=real_stdout)
elif dry:
print(f"would {action} {kind or '-'} -> {path}")
# non-dry human output is the helper "ok:" lines + the summary printed by the caller.
return 0 if ok else 1
def chatter():
# Low-level verbs (cmd_put/cmd_append/cmd_fm) print progress lines; keep
# stdout clean for the envelope by routing them to stderr wholesale.
return contextlib.redirect_stdout(sys.stderr)
if body_text is None:
body_text = echo.read_body(file_arg).decode("utf-8", errors="replace")
today_s = echo.today()
aliases = [a.strip() for a in (aliases or []) if a.strip()]
sources = [s.strip() for s in (sources or []) if s.strip()]
tags = [t.strip() for t in (tags or []) if t.strip()]
def env_for(action: str, path: str, links: int = 0, ok: bool = True, dry: bool = False,
near=None, **extra) -> dict:
act = f"dry-run:{action}" if dry else action
data = {"kind": kind, "path": path, "title": title, "links_added": links}
if near:
data["near_duplicates"] = near
data.update(extra)
return echo_output.envelope(act, data, ok=ok)
# Unknown home -> defer to the inbox (a single idempotent capture line).
if inbox or not kind:
if dry_run:
return done("inbox", "inbox/captures/inbox.md", dry=True)
return env_for("inbox", "inbox/captures/inbox.md", dry=True)
line = f"- {today_s}: {title}"
if body_text.strip():
line += f"{body_text.strip().splitlines()[0]}"
with quiet():
with chatter():
rc = echo.cmd_append("inbox/captures/inbox.md", line)
return done("inbox", "inbox/captures/inbox.md", ok=rc == 0)
return env_for("inbox", "inbox/captures/inbox.md", ok=rc == 0)
slug = idx_mod.slugify(title)
# Offline short-circuit (2.1.0). The routed path needs the entity index and several
# round-trips; when the vault is unreachable, queue the WHOLE capture as one semantic
# record — flush replays it back through this function against the index as it is at
# replay time, so routing, the duplicate gate, and aliasing re-run with fresh state.
# (Byte-level request replay would freeze a create-vs-update decision made blind.)
try:
index = idx_mod.load()
except echo.EchoError as exc:
if "unreachable" not in str(exc):
raise
if dry_run:
return echo_output.envelope("dry-run:queued:capture",
{"kind": kind, "title": title, "queued": True})
import echo_queue
echo_queue.enqueue_capture({
"kind": kind, "title": title, "body_text": body_text, "status_v": status_v,
"aliases": aliases, "sources": sources, "tags": tags, "date": date,
"domain": domain, "inbox": False, "no_log": no_log,
"force": force, "merge_into": merge_into, "today": today_s,
})
return echo_output.envelope("queued:capture",
{"kind": kind, "title": title, "queued": True})
match_slug, existing = idx_mod.resolve(index, title)
# --merge-into: the operator has already identified the canonical entity (e.g. after
# a duplicate-gate stop) — route this capture as an UPDATE to it, whatever the title.
if merge_into and not existing:
match_slug, existing = idx_mod.resolve(index, merge_into)
if not existing:
print(f"echo_ops: --merge-into '{merge_into}' matches no entity in the index "
f"(try `resolve` first)", file=sys.stderr)
return 2
raise echo.EchoError(f"--merge-into '{merge_into}' matches no entity in the "
"index (try `resolve` first)", 2)
existing_reachable = bool(existing and echo.request("GET", echo.vault_url(existing["path"]))[0] == 200)
# Pre-write duplicate gate: a strong fuzzy candidate means this title is very likely
@@ -237,40 +278,26 @@ def capture(kind: str | None, title: str, file_arg: str | None, status_v: str =
for s, c, sc in idx_mod.gate_candidates(index, title, kind=kind,
threshold=DUP_GATE)]
if gate_hits:
if as_json:
env = echo_output.envelope("duplicate-gate", {
return echo_output.envelope("duplicate-gate", {
"kind": kind, "title": title, "candidates": gate_hits,
"note": "likely duplicate — re-run with --merge-into <slug> to update the "
"existing entity, or --force to create anyway"}, ok=False)
print(json.dumps(env, ensure_ascii=False), file=real_stdout)
else:
print(f"STOP: '{title}' likely already exists — not creating a duplicate.",
file=real_stdout)
for c in gate_hits:
print(f" candidate: {c['slug']} -> {c['path']} (score {c['score']})",
file=real_stdout)
print(" re-run with --merge-into <slug> to update the existing entity, "
"or --force to create anyway.", file=real_stdout)
return 76 # distinct exit: duplicate gate (cf. 75 lock, 78 config)
if dry_run:
if existing_reachable:
return done("update", existing["path"], dry=True)
return env_for("update", existing["path"], dry=True)
s2 = echo_quality.safe_slug(set(index.get("entities", {}).keys()), slug)
near = [c.get("path") for _, c, _ in idx_mod.fuzzy_candidates(index, title)][:3]
plan = done("create", idx_mod.derive_path(kind, s2, date=date, domain=domain),
dry=True, near=near)
if (not as_json and not force
and idx_mod.gate_candidates(index, title, kind=kind, threshold=DUP_GATE)):
# (in --json mode the near_duplicates field carries this; keep stdout clean)
print("note: a real run would STOP at the duplicate gate — use --merge-into "
"or --force.", file=real_stdout)
return plan
would_gate = (not force
and bool(idx_mod.gate_candidates(index, title, kind=kind,
threshold=DUP_GATE)))
return env_for("create", idx_mod.derive_path(kind, s2, date=date, domain=domain),
dry=True, near=near, would_gate=would_gate)
near_dupes: list[str] = []
index_title = title # title to record in the index entry
index_aliases = list(aliases)
with quiet():
with chatter():
if existing_reachable:
# Reuse the matched entity's CANONICAL slug — never re-slug from the mention, or
# two index slugs end up pointing at one note. Keep its title; learn the mention
@@ -306,15 +333,11 @@ def capture(kind: str | None, title: str, file_arg: str | None, status_v: str =
echo.cmd_put(path, echo.temp_file(note.encode()))
action = "created"
# H3: the entity-index write is the race the review flagged — a bare load->save lets
# two concurrent captures clobber each other's entry. Route it through a lock-guarded,
# fresh-re-read transaction. Returns the fresh index (used by auto-link below).
import echo_concurrency
index = echo_concurrency.atomic_index_update(
lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases))
# auto-link: any other known entity CONFIDENTLY named in the body (M2: is_confident_link
# rejects short/common tokens so a 3-char alias or a word like "API" can't link everywhere).
# auto-link FIRST (it mutates the note's ## Related section), so the content
# hash stored below reflects the note's final state. Uses the pre-update index
# — it only needs to find OTHER entities named in the body. (M2: is_confident_link
# rejects short/common tokens so a 3-char alias or a word like "API" can't link
# everywhere.)
linked = 0
for s2, e2 in index.get("entities", {}).items():
if e2.get("path") in (None, path):
@@ -324,24 +347,85 @@ def capture(kind: str | None, title: str, file_arg: str | None, status_v: str =
ca, cb = links.link_bidirectional(path, e2["path"])
linked += int(ca or cb)
# Keep the BM25 recall index current for this note (best-effort; lock-guarded inside
# update_note, so it can't clobber a concurrent writer either).
# The note's final content: hashed into the entity index (the incremental
# sweep's change detector, 2.3) and fed to the local recall index.
final_text = links.get_text(path) or ""
note_h = idx_mod.content_hash(final_text) if final_text else None
# H3: the entity-index write is the race the review flagged — a bare load->save lets
# two concurrent captures clobber each other's entry. Route it through a lock-guarded,
# fresh-re-read transaction.
import echo_concurrency
index = echo_concurrency.atomic_index_update(
lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases,
h=note_h))
# Keep the local BM25 recall index current for this note (best-effort; a
# zero-network atomic file write since 2.3).
try:
import echo_recall
echo_recall.update_note(path, links.get_text(path) or "")
echo_recall.update_note(path, final_text)
except Exception as exc: # never let recall-index upkeep fail a capture
print(f"echo_ops: recall-index update skipped ({exc})", file=sys.stderr)
if not no_log:
ensure_daily_log(f"- {today_s}: {action} {kind} [[{links.link_token(path)}]]")
return env_for(action, path, links=linked, near=near_dupes)
def capture(kind: str | None, title: str, file_arg: str | None, status_v: str = "",
aliases=None, sources=None, tags=None, date: str | None = None,
domain: str = "business", inbox: bool = False, no_log: bool = False,
as_json: bool = False, dry_run: bool = False, force: bool = False,
merge_into: str | None = None) -> int:
"""CLI wrapper around capture_op: prints exactly what pre-Phase-0 capture printed
(human prose or the --json envelope) and maps envelope outcomes to exit codes
(76 duplicate-gate, 2 usage-class errors, 0 otherwise)."""
try:
env = capture_op(kind, title, file_arg, status_v=status_v, aliases=aliases,
sources=sources, tags=tags, date=date, domain=domain,
inbox=inbox, no_log=no_log, dry_run=dry_run, force=force,
merge_into=merge_into)
except echo.EchoError as exc:
print(f"echo_ops: {exc}", file=sys.stderr)
return getattr(exc, "code", 1)
action = env.get("action", "")
if as_json:
done(action, path, links=linked, near=near_dupes)
else:
print(f"ok: {action} {kind} -> {path}" + (f"; auto-linked {linked}" if linked else ""))
if near_dupes:
kind_word = "entity" if len(near_dupes) == 1 else "entities"
print(f"WARNING: similar existing {kind_word} ({', '.join(near_dupes)}) — if this is "
f"the same thing, merge or `echo.py link` instead of keeping a duplicate.",
file=real_stdout)
print(json.dumps(env, ensure_ascii=False))
if action == "duplicate-gate":
return 76
return 0 if env.get("ok") else 1
if action == "duplicate-gate":
print(f"STOP: '{title}' likely already exists — not creating a duplicate.")
for c in env.get("candidates", []):
print(f" candidate: {c['slug']} -> {c['path']} (score {c['score']})")
print(" re-run with --merge-into <slug> to update the existing entity, "
"or --force to create anyway.")
return 76 # distinct exit: duplicate gate (cf. 75 lock, 78 config)
if action == "queued:capture":
print(f"queued (offline): capture {kind} '{title}' — will replay through "
"capture on the next reachable session")
return 0
if action == "dry-run:queued:capture":
print("offline: vault unreachable — a real run would queue this capture "
"for replay on the next reachable session.")
return 0
if action.startswith("dry-run:"):
print(f"would {action.split(':', 1)[1]} {kind or '-'} -> {env.get('path')}")
if env.get("would_gate"):
print("note: a real run would STOP at the duplicate gate — use --merge-into "
"or --force.")
return 0
if action == "inbox":
return 0 if env.get("ok") else 1
print(f"ok: {action} {kind} -> {env.get('path')}"
+ (f"; auto-linked {env['links_added']}" if env.get("links_added") else ""))
near = env.get("near_duplicates")
if near:
kind_word = "entity" if len(near) == 1 else "entities"
print(f"WARNING: similar existing {kind_word} ({', '.join(near)}) — if this is "
f"the same thing, merge or `echo.py link` instead of keeping a duplicate.")
return 0 if env.get("ok") else 1
@@ -81,6 +81,23 @@ def enqueue(method: str, url: str, data: bytes | None, headers: dict, idem_key:
fh.write(json.dumps(rec, ensure_ascii=False) + "\n")
def enqueue_capture(args: dict) -> None:
"""Queue a whole capture as ONE semantic record (2.1.0). Byte-level replay is wrong
for the routed capture path: the create-vs-update decision, duplicate gate, and
aliasing must re-run against the index AS IT IS AT REPLAY TIME, so flush re-invokes
echo_ops.capture with the recorded arguments instead of replaying stale requests.
`args` carries body_text inline (never a temp-file path) plus `today` (the capture-
time ECHO_TODAY) so replayed frontmatter/log dates reflect when it was said."""
key = "capture:" + hashlib.sha1(
json.dumps(args, sort_keys=True, ensure_ascii=False).encode("utf-8")).hexdigest()
if any(rec.get("idem_key") == key for rec in pending()):
return
_ensure_dirs()
rec = {"op": "capture", "args": args, "idem_key": key}
with outbox_path().open("a", encoding="utf-8") as fh:
fh.write(json.dumps(rec, ensure_ascii=False) + "\n")
def pending() -> list[dict]:
p = outbox_path()
if not p.exists():
@@ -88,6 +105,12 @@ def pending() -> list[dict]:
return [json.loads(ln) for ln in p.read_text(encoding="utf-8").splitlines() if ln.strip()]
def needs_attention() -> list[dict]:
"""Queued records that replay could not land automatically (e.g. a duplicate-gate
stop) — surfaced by `flush` and `load` so the operator resolves them deliberately."""
return [rec for rec in pending() if rec.get("needs_attention")]
def _rewrite(records: list[dict]) -> None:
p = outbox_path()
if not records:
@@ -108,9 +131,57 @@ def _rebase(url: str) -> str:
return echo.BASE + parts.path + (("?" + parts.query) if parts.query else "")
def _replay_capture(rec: dict) -> str:
"""Replay a queued semantic capture through echo_ops.capture against the CURRENT
index. Returns 'ok', 'retry' (still offline), or 'gated' (duplicate gate / needs
the operator — the record is kept and flagged, later records still replay)."""
# Cheap reachability probe FIRST: if the vault is still down, echo_ops.capture's
# own offline short-circuit would try to re-enqueue this very record — probe and
# bail as 'retry' instead of recursing into the queue.
st, _ = echo.request("GET", echo.vault_url("_agent/echo-vault.md"))
if st == 0:
return "retry"
import echo_ops
a = dict(rec.get("args") or {})
body_text = a.pop("body_text", "") or ""
day = a.pop("today", None)
bodyfile = echo.temp_file(body_text.encode("utf-8")) if body_text else None
prev = os.environ.get("ECHO_TODAY")
try:
if day:
os.environ["ECHO_TODAY"] = day # replayed dates reflect when it was said
rc = echo_ops.capture(
a.get("kind"), a.get("title") or "(untitled)", bodyfile,
status_v=a.get("status_v", ""), aliases=a.get("aliases") or [],
sources=a.get("sources") or [], tags=a.get("tags") or [],
date=a.get("date"), domain=a.get("domain", "business"),
inbox=bool(a.get("inbox")), no_log=bool(a.get("no_log")),
force=bool(a.get("force")), merge_into=a.get("merge_into"))
except Exception as exc: # noqa: BLE001 — a broken record must not wedge the queue
print(f"echo_queue: queued capture '{a.get('title')}' failed on replay ({exc}) "
"— kept and flagged", file=sys.stderr)
return "gated"
finally:
if day:
if prev is None:
os.environ.pop("ECHO_TODAY", None)
else:
os.environ["ECHO_TODAY"] = prev
if rc == 0:
return "ok"
if rc == 76:
print(f"echo_queue: queued capture '{a.get('title')}' stopped at the duplicate "
"gate on replay — resolve with capture --merge-into <slug> or --force, "
"then remove it from the outbox", file=sys.stderr)
return "gated"
def _replay(rec: dict) -> str:
"""Re-issue one queued write idempotently. Returns 'ok' (landed/already-present),
'drop' (permanent 4xx — can never succeed), or 'retry' (still offline / 5xx)."""
'drop' (permanent 4xx — can never succeed), 'gated' (kept + flagged for the
operator), or 'retry' (still offline / 5xx)."""
if rec.get("op") == "capture":
return _replay_capture(rec)
method = rec["method"]
url = _rebase(rec["url"])
data = base64.b64decode(rec["body_b64"]) if rec.get("body_b64") else None
@@ -137,21 +208,30 @@ def _replay(rec: dict) -> str:
def flush() -> int:
"""Replay queued writes in order; stop at the first still-unreachable one (keeping it
and the rest). Returns the number actually replayed. Safe to call when empty."""
and everything after). A 'gated' record (duplicate gate / replay error) is kept and
flagged but does NOT block later records — they are independent writes. Returns the
number actually replayed. Safe to call when empty."""
items = pending()
if not items:
return 0
replayed = 0
remaining: list[dict] = []
for i, rec in enumerate(items):
stopped = False
for rec in items:
if stopped:
remaining.append(rec)
continue
result = _replay(rec)
if result == "ok":
replayed += 1
elif result == "drop":
continue # warned in _replay; remove from queue
elif result == "gated":
rec["needs_attention"] = rec.get("needs_attention") or "replay-gated"
remaining.append(rec)
else: # retry -> still offline; keep this and everything after, in order
remaining = items[i:]
break
remaining.append(rec)
stopped = True
_rewrite(remaining)
return replayed
@@ -52,9 +52,17 @@ sys.path.insert(0, str(Path(__file__).resolve().parent))
import echo # noqa: E402
import echo_index as idx_mod # noqa: E402
import echo_links as links # noqa: E402
import echo_stem # noqa: E402
RECALL_INDEX_PATH = "_agent/index/recall-index.json"
INDEX_SCHEMA = 2 # bumped: schema 2 adds per-doc meta (weight/updated/status)
# Schema 3 (2.3, the "index train"): postings are STEMMED (echo_stem), per-doc meta
# gains a content hash `h` (incremental sweep), and the index carries a `built`
# stamp + endpoint hash. The LIVE index is LOCAL-FIRST — it lives in the state dir
# and is updated with zero vault round-trips; the vault copy at RECALL_INDEX_PATH
# is a snapshot written by sweep/session-end, used to seed fresh machines. Older
# schemas are discarded and rebuilt (sub-second since the 1.1 network work).
INDEX_SCHEMA = 3
SYNC_HOURS = float(os.environ.get("ECHO_RECALL_SYNC_HOURS", "24"))
# Tuning knobs. Defaults are standard BM25 (k1/b) + a 0.6 hop decay over <=2 hops.
K1 = 1.5
@@ -131,8 +139,11 @@ _SKIP_BASENAMES = {"README.md"}
def tokenize(text: str) -> list[str]:
"""Lowercase word tokens, stopworded. Frontmatter should be stripped by the caller."""
return [t for t in _WORD.findall(text.lower()) if t not in _STOP and len(t) > 1]
"""Lowercase word tokens, stopworded, STEMMED (2.3) — applied identically at
index and query time, so 'deployed'/'deployment' meet at 'deploy'.
Frontmatter should be stripped by the caller."""
return [echo_stem.stem(t) for t in _WORD.findall(text.lower())
if t not in _STOP and len(t) > 1]
def strip_frontmatter(text: str) -> str:
@@ -152,9 +163,10 @@ class Bm25Index:
self.df: Counter[str] = Counter() # term -> #docs containing it
self.postings: dict[str, dict[str, int]] = {} # term -> {doc_path: tf}
self.length: dict[str, int] = {} # doc_path -> token count
self.meta: dict[str, dict] = {} # doc_path -> {w, u, s} priors
self.meta: dict[str, dict] = {} # doc_path -> {w, u, s, h} priors
self.n_docs = 0
self.avg_len = 0.0
self.built = "" # ISO stamp of last full (re)build
def _recompute(self) -> None:
self.n_docs = len(self.length)
@@ -165,12 +177,19 @@ class Bm25Index:
self.remove(path)
toks = tokenize(strip_frontmatter(raw_text))
self.length[path] = len(toks)
self.meta[path] = doc_meta(path, raw_text)
meta = doc_meta(path, raw_text)
# Content hash: the incremental sweep's change detector (2.3).
import hashlib
meta["h"] = hashlib.sha1(raw_text.encode("utf-8", "replace")).hexdigest()[:16]
self.meta[path] = meta
for term, tf in Counter(toks).items():
self.postings.setdefault(term, {})[path] = tf
self.df[term] += 1
self._recompute()
def content_hash(self, path: str) -> str | None:
return (self.meta.get(path) or {}).get("h")
def remove(self, path: str) -> None:
if path not in self.length:
return
@@ -192,11 +211,17 @@ class Bm25Index:
* freshness(m.get("u"), today_s)
* STATUS_FACTOR.get(m.get("s", ""), 1.0))
def score(self, query: str, limit: int = 10,
today_s: str | None = None) -> list[tuple[str, float]]:
def score(self, query: str, limit: int = 10, today_s: str | None = None,
extra_terms: dict[str, float] | None = None) -> list[tuple[str, float]]:
"""BM25 x priors. `extra_terms` (term -> weight) are alias-expansion terms
(2.3): they contribute at reduced weight and can only boost documents that
actually contain them — expansion never invents corpus-free hits."""
today_s = today_s or echo.today()
weighted: dict[str, float] = {t: 1.0 for t in set(tokenize(query))}
for t, w in (extra_terms or {}).items():
weighted.setdefault(t, w)
scores: Counter[str] = Counter()
for term in set(tokenize(query)):
for term, weight in weighted.items():
posting = self.postings.get(term)
if not posting:
continue
@@ -204,26 +229,29 @@ class Bm25Index:
for path, tf in posting.items():
dl = self.length.get(path, 0)
denom = tf + K1 * (1 - B + B * (dl / self.avg_len if self.avg_len else 1))
scores[path] += idf * (tf * (K1 + 1) / denom if denom else 0)
scores[path] += weight * idf * (tf * (K1 + 1) / denom if denom else 0)
fused = {p: s * self._doc_factor(p, today_s) for p, s in scores.items()}
ranked = sorted(fused.items(), key=lambda kv: -kv[1])[:limit]
return [(p, s) for p, s in ranked if s > MIN_SCORE]
# ---- serialization (compact; rebuildable, so the format can change freely) ----
def to_json(self) -> dict:
return {"schema": INDEX_SCHEMA, "n_docs": self.n_docs, "avg_len": self.avg_len,
"length": self.length, "postings": self.postings, "meta": self.meta}
return {"schema": INDEX_SCHEMA, "built": self.built, "n_docs": self.n_docs,
"avg_len": self.avg_len, "length": self.length,
"postings": self.postings, "meta": self.meta}
@classmethod
def from_json(cls, d: dict) -> "Bm25Index":
if d.get("schema") != INDEX_SCHEMA:
return cls() # older/newer format -> empty; recall's cold path rebuilds
# (a pre-stemming index MUST NOT be reused: its postings are unstemmed)
ix = cls()
ix.length = d.get("length", {})
ix.postings = d.get("postings", {})
ix.meta = d.get("meta", {})
ix.n_docs = d.get("n_docs", len(ix.length))
ix.avg_len = d.get("avg_len", 0.0)
ix.built = d.get("built", "")
ix.df = Counter({term: len(posting) for term, posting in ix.postings.items()})
return ix
@@ -271,27 +299,84 @@ def _indexable(path: str) -> bool:
# ------------------------------------------------------------------- persistence
def load_index() -> Bm25Index:
status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH))
if status == 404:
return Bm25Index()
echo.check(status, body, "recall-index load")
# LOCAL-FIRST (2.3). The live index is a machine-local file — updating it on capture
# is a zero-network atomic file write instead of GET-whole-index -> PUT-whole-index
# under the vault lock (the old per-write O(vault) round trip). The vault copy is a
# SNAPSHOT: written by sweep (and best-effort at session end), read only to seed a
# fresh machine or to catch up after another client swept (checked at most every
# ECHO_RECALL_SYNC_HOURS). Staleness across clients is tolerable — recall degrades
# gracefully and the fast sweep trues it up.
def _local_path() -> Path:
import hashlib
import echo_queue
h = hashlib.sha1((echo.BASE or "").encode()).hexdigest()[:12]
return echo_queue.state_dir() / f"recall-index-{h}.json"
def _load_local() -> Bm25Index | None:
p = _local_path()
if not p.exists():
return None
try:
return Bm25Index.from_json(json.loads(p.read_text(encoding="utf-8")))
except (json.JSONDecodeError, KeyError, TypeError, OSError):
return None
def save_local(ix: Bm25Index) -> None:
p = _local_path()
p.parent.mkdir(parents=True, exist_ok=True)
tmp = p.with_suffix(".json.tmp")
tmp.write_text(json.dumps(ix.to_json(), ensure_ascii=False), encoding="utf-8")
os.replace(tmp, p) # atomic: a crash mid-write can't corrupt the live index
def _fetch_snapshot() -> Bm25Index | None:
try:
status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH))
if status != 200:
return None
return Bm25Index.from_json(json.loads(body))
except (json.JSONDecodeError, KeyError, TypeError):
return Bm25Index()
except Exception: # noqa: BLE001 — the snapshot is an optimization, never required
return None
def save_index(ix: Bm25Index) -> None:
def save_snapshot(ix: Bm25Index) -> None:
"""PUT the vault snapshot — sweep + session-end only, never the per-capture path."""
body = json.dumps(ix.to_json(), ensure_ascii=False).encode("utf-8")
status, b = echo.request("PUT", echo.vault_url(RECALL_INDEX_PATH), data=body,
headers={"Content-Type": "application/json"})
echo.check(status, b, "recall-index save")
echo.check(status, b, "recall-index snapshot save")
def rebuild(prefetched: dict | None = None) -> Bm25Index:
"""Full rebuild from every indexable entity note. Called by sweep.py and as the
cold-cache path inside recall(). Saves and returns the index.
def load_index() -> Bm25Index:
"""The live index: local file first; the vault snapshot only seeds a fresh
machine or replaces a local copy that a newer sweep elsewhere has superseded
(checked when the local build stamp is older than SYNC_HOURS)."""
local = _load_local()
if local and local.n_docs:
try:
age_ok = local.built and (
_dt.datetime.now(_dt.timezone.utc)
- _dt.datetime.strptime(local.built, "%Y-%m-%dT%H:%M:%SZ").replace(
tzinfo=_dt.timezone.utc)
).total_seconds() < SYNC_HOURS * 3600
except ValueError:
age_ok = False
if age_ok:
return local
snap = _fetch_snapshot()
if snap and snap.n_docs and (not local or (snap.built or "") > (local.built or "")):
save_local(snap)
return snap
return local or Bm25Index()
def rebuild(prefetched: dict | None = None, snapshot: bool = False) -> Bm25Index:
"""Full rebuild from every indexable note. Saves the LOCAL index always; writes
the vault snapshot only when `snapshot=True` (sweep) — the read path never
surprises the vault with a PUT.
`prefetched` (a {path: text} map, e.g. from echo.read_many) lets sweep.py reuse the
content it already pulled instead of walking and re-fetching the whole vault again."""
@@ -299,27 +384,30 @@ def rebuild(prefetched: dict | None = None) -> Bm25Index:
if prefetched is not None:
items = ((p, t) for p, t in prefetched.items() if _indexable(p))
else:
items = ((p, _get(p)) for p in _walk() if _indexable(p))
items = echo.read_many([p for p in _walk() if _indexable(p)]).items()
for path, text in items:
if text is not None:
ix.add(path, text) # add() strips frontmatter + extracts doc meta itself
save_index(ix)
ix.built = echo.now_iso()
save_local(ix)
if snapshot:
save_snapshot(ix)
return ix
def update_note(path: str, text: str) -> None:
"""Best-effort incremental upkeep for a single note (mirrors how entities.json is
maintained on capture). Never raises into the caller. The load->save is wrapped in the
advisory lock with a fresh re-read (H3), so concurrent captures can't clobber the
recall index either."""
"""Best-effort incremental upkeep for a single note — a LOCAL atomic file write,
no vault round-trip, no advisory lock (2.3). Maintains an existing local index
only: with no local index yet, the next recall's cold path (or a sweep) builds
one, and a one-note index must never shadow the snapshot. Never raises."""
try:
if not _indexable(path):
return # only entity notes are part of the recall corpus
import echo_concurrency
with echo_concurrency.vault_lock():
ix = load_index() # fresh read inside the lock
return # only corpus notes are part of the recall index
ix = _load_local()
if ix is None or not ix.n_docs:
return
ix.add(path, text) # raw text: add() strips + extracts meta
save_index(ix)
save_local(ix)
except Exception as exc: # noqa: BLE001 — upkeep must never fail a capture
print(f"echo_recall: index update skipped ({exc})", file=sys.stderr)
@@ -396,44 +484,36 @@ def _doc_summary(path: str, text: str) -> dict:
return out
def _brief(path: str, score: float | None = None, via: str | None = None) -> None:
text = links.get_text(path)
if text is None:
return
info = _doc_summary(path, text)
head = f"\n### {path}"
if score is not None:
head += f" (score {score:.2f})"
if via:
head += f" (via {via})"
print(head)
stamps = []
if info.get("type"):
stamps.append(f"type: {info['type']}")
if info.get("updated"):
stamps.append(f"updated: {info['updated']}")
if info.get("status"):
stamps.append(f"status: {info['status']}")
if stamps:
print("_" + " · ".join(stamps) + "_") # staleness is visible, not guessed
if info.get("excerpt"):
print(info["excerpt"])
# ------------------------------------------------------------------- entrypoint
def recall(query, limit: int = 8, as_json: bool = False) -> int:
def recall_op(query, limit: int = 8) -> dict:
"""Core recall (Phase 0): hybrid BM25 + graph, returning the structured envelope
({query, primary, linked}) that backs both the CLI renderings and the MCP tool."""
q = " ".join(query) if isinstance(query, list) else query
today_s = echo.today()
index = idx_mod.load()
nmap = idx_mod.name_map(index)
# --- alias query expansion (2.3): a query phrased by an entity's alias or a
# partial name folds that entity's title/alias vocabulary into the BM25 query at
# half weight — "the ECHO plugin" also scores `echo-memory` terms. Expansion can
# only boost documents that actually contain the terms.
extra_terms: dict[str, float] = {}
q_toks = set(tokenize(q))
for _slug, e, sc in idx_mod.fuzzy_candidates(index, q)[:2]:
if sc < 0.5:
continue
for name in (e.get("title", ""), _slug, *e.get("aliases", [])):
for tok in tokenize(str(name).replace("-", " ")):
if tok not in q_toks:
extra_terms[tok] = 0.5
# --- lexical layer (BM25); rebuild the cache on a cold miss --------------
lexical: list[tuple[str, float]] = []
try:
ix = load_index()
if ix.n_docs == 0:
ix = rebuild()
lexical = ix.score(q, limit=limit, today_s=today_s)
lexical = ix.score(q, limit=limit, today_s=today_s, extra_terms=extra_terms)
except echo.EchoError as exc:
print(f"recall: BM25 index unavailable ({exc}); falling back to server search",
file=sys.stderr)
@@ -464,7 +544,6 @@ def recall(query, limit: int = 8, as_json: bool = False) -> int:
# --- graph layer ----------------------------------------------------------
neighbours = expand_graph(hits, nmap, base, max_hops=MAX_HOPS)
if as_json:
import echo_output
texts = echo.read_many(hits + [p for p, _ in neighbours[: 2 * limit]])
primary = []
@@ -479,16 +558,35 @@ def recall(query, limit: int = 8, as_json: bool = False) -> int:
continue
linked.append({**_doc_summary(p, texts[p]),
"score": round(sc, 3), "via": via})
env = echo_output.envelope("recall", {"query": q, "primary": primary,
return echo_output.envelope("recall", {"query": q, "primary": primary,
"linked": linked})
def _print_hit(info: dict) -> None:
"""Prose rendering of one recall hit — same fields the envelope carries."""
head = f"\n### {info['path']}"
if info.get("score") is not None:
head += f" (score {info['score']:.2f})"
if info.get("via"):
head += f" (via {info['via']})"
print(head)
stamps = [f"{k}: {info[k]}" for k in ("type", "updated", "status") if info.get(k)]
if stamps:
print("_" + " · ".join(stamps) + "_") # staleness is visible, not guessed
if info.get("excerpt"):
print(info["excerpt"])
def recall(query, limit: int = 8, as_json: bool = False) -> int:
env = recall_op(query, limit=limit)
if as_json:
print(json.dumps(env, ensure_ascii=False))
return 0
print(f"== recall: {q} ==")
print(f"\n# Primary hits ({len(hits)})")
for p in hits:
_brief(p, score=base.get(p))
print(f"\n# Linked context ({len(neighbours)})")
for p, (sc, via) in neighbours[: 2 * limit]:
_brief(p, score=sc, via=via)
print(f"== recall: {env['query']} ==")
print(f"\n# Primary hits ({len(env['primary'])})")
for info in env["primary"]:
_print_hit(info)
print(f"\n# Linked context ({len(env['linked'])})")
for info in env["linked"]:
_print_hit(info)
return 0
@@ -99,43 +99,69 @@ def preview(proposals: list[dict]) -> str:
return "\n".join(rows)
def apply(proposals: list[dict], confirm: bool = False) -> int:
"""Validate -> classify -> preview; with confirm=True, apply each via echo_ops.capture
(which routes/indexes/links/logs, each self-lock-guarded). Without confirm it is a
dry-run that writes nothing — the preview IS the confirmation step."""
def apply_op(proposals: list[dict], confirm: bool = False) -> dict:
"""Core reflect (Phase 0): validate -> classify -> (apply via capture_op). Returns
an envelope — rows carry the classified preview; with confirm, `results` carries
each proposal's capture outcome. Never prints to stdout."""
import echo_output
valid, errors = validate(proposals)
for e in errors:
print(f"skip: {e}", file=sys.stderr)
if not valid:
print("reflect: no valid proposals to apply.")
return 0
rows: list[dict] = []
counts: dict[str, int] = {}
if valid:
classify(valid)
counts = {a: sum(1 for p in valid if p.get("_action") == a) for a in ("create", "update", "inbox", "error")}
print(f"reflect: {len(valid)} proposal(s) — "
f"{counts['create']} new, {counts['update']} update, {counts['inbox']} inbox"
+ (f", {counts['error']} error" if counts["error"] else ""))
print(preview(valid))
if not confirm:
print("\nreflect: dry-run — re-run with --apply to write these to memory.")
return 0
counts = {a: sum(1 for p in valid if p.get("_action") == a)
for a in ("create", "update", "inbox", "error")}
rows = [{"action": p.get("_action"), "kind": p.get("kind"),
"title": p["title"], "path": p.get("_path")} for p in valid]
data = {"proposals": len(proposals or []), "valid": len(valid), "errors": errors,
"counts": counts, "rows": rows, "dry_run": not confirm,
"applied": 0, "gated": 0, "results": []}
if not valid or not confirm:
return echo_output.envelope("reflect", data)
import echo_ops # lazy: the apply path pulls in the capture/index/link stack
applied = gated = 0
results = []
for p in valid:
if p.get("_action") == "error":
continue
bodyfile = echo.temp_file((p.get("body") or "").encode()) if p.get("body") else None
rc = echo_ops.capture(
p.get("kind"), p["title"], bodyfile,
env = echo_ops.capture_op(
p.get("kind"), p["title"], body_text=p.get("body") or "",
aliases=p.get("aliases") or [], sources=p.get("sources") or [],
tags=p.get("tags") or [],
date=p.get("date"), domain=p.get("domain", "business"),
inbox=bool(p.get("inbox")) or not p.get("kind"))
applied += 1 if rc == 0 else 0
gated += 1 if rc == 76 else 0
print(f"reflect: applied {applied}/{len(valid)} proposal(s)."
act = env.get("action", "")
if act == "duplicate-gate":
gated += 1
elif env.get("ok"):
applied += 1
results.append({"title": p["title"], "action": act,
"path": env.get("path"), "ok": bool(env.get("ok"))})
data.update(applied=applied, gated=gated, results=results)
return echo_output.envelope("reflect", data)
def apply(proposals: list[dict], confirm: bool = False) -> int:
"""CLI wrapper: same preview/summary text as pre-Phase-0. Without confirm it is a
dry-run that writes nothing — the preview IS the confirmation step."""
env = apply_op(proposals, confirm=confirm)
for e in env["errors"]:
print(f"skip: {e}", file=sys.stderr)
if not env["valid"]:
print("reflect: no valid proposals to apply.")
return 0
counts = env["counts"]
print(f"reflect: {env['valid']} proposal(s) — "
f"{counts['create']} new, {counts['update']} update, {counts['inbox']} inbox"
+ (f", {counts['error']} error" if counts["error"] else ""))
print("\n".join(f" {r['action'] or '?':7} | {('-' if r['action'] == 'inbox' else r['kind'] or '?'):9} "
f"| {r['title']} -> {r['path'] or '?'}" for r in env["rows"]))
if env["dry_run"]:
print("\nreflect: dry-run — re-run with --apply to write these to memory.")
return 0
gated = env["gated"]
print(f"reflect: applied {env['applied']}/{env['valid']} proposal(s)."
+ (f" {gated} stopped at the duplicate gate — re-propose with the existing "
f"entity's title (or capture --merge-into <slug>)." if gated else ""))
return 0
@@ -0,0 +1,177 @@
#!/usr/bin/env python3
"""echo_session.py — the session-end bundle: one call ends a session correctly. [2.1.0]
Ending a substantive session used to take four-plus separate invocations (session-log
PUT, Agent Log append, heartbeat PUT, optional scope set, optional reflect apply) —
a cost per session, and a partial failure left broken orientation state (log written
but heartbeat stale). This verb does all of it in one call, under ONE advisory-lock
acquisition, in a fixed order with the heartbeat written LAST as the commit marker:
a failure part-way leaves the previous pointer intact, never a dangling one.
Bundle shape (JSON object):
{
"slug": "echo-mcp-spec", # required, kebab-case
"log_body": "<full session-log markdown, frontmatter included>", # required
"agent_log_line": "- 2026-07-28: ...", # optional; derived when omitted
"scope": "new scope text", # optional -> scope set
"reflect": [ { ...PROPOSAL_SCHEMA... } ] # optional -> routed via capture
}
Dry-run by default (previews the whole plan, reflect included); --apply writes.
Times: the filename HHMM comes from $ECHO_NOW (HHMM) else the local clock; the date
from ECHO_TODAY via echo.today(). Offline: every step rides the offline queue, so a
session end during an outage queues durably instead of failing.
CLI: echo.py session-end <bundle.json> [--apply]
"""
from __future__ import annotations
import datetime as dt
import json
import os
import re
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import echo # noqa: E402
SESSION_RE = re.compile(r"^\d{4}-\d{2}-\d{2}-\d{4}-[a-z0-9-]+\.md$")
SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9-]*$")
def _hhmm() -> str:
v = os.environ.get("ECHO_NOW", "").strip()
if v:
if not re.match(r"^\d{4}$", v):
raise echo.EchoError(f"session-end: $ECHO_NOW must be HHMM, got '{v}'", 2)
return v
return dt.datetime.now().strftime("%H%M")
def validate(bundle: dict) -> tuple[str, str]:
"""Return (session_path, agent_log_line) or raise EchoError(2). Validation runs
BEFORE any write — a bad bundle writes nothing at all."""
if not isinstance(bundle, dict):
raise echo.EchoError("session-end: bundle must be a JSON object", 2)
slug = str(bundle.get("slug", "")).strip()
if not SLUG_RE.match(slug):
raise echo.EchoError(f"session-end: slug must be kebab-case, got '{slug}'", 2)
if not str(bundle.get("log_body", "")).strip():
raise echo.EchoError("session-end: log_body is required (the full session-log markdown)", 2)
reflect = bundle.get("reflect")
if reflect is not None and not isinstance(reflect, list):
raise echo.EchoError("session-end: reflect must be a JSON array of proposals", 2)
fname = f"{echo.today()}-{_hhmm()}-{slug}.md"
if not SESSION_RE.match(fname):
raise echo.EchoError(f"session-end: derived filename '{fname}' violates the "
"canonical YYYY-MM-DD-HHMM-<slug>.md form", 2)
path = f"_agent/sessions/{fname}"
line = str(bundle.get("agent_log_line") or "").strip() \
or f"- {echo.today()}: session logged [[{path[:-3]}]]"
return path, line
def session_end_op(bundle: dict, apply: bool = False) -> dict:
"""Core session-end (Phase 0): validate -> plan -> (apply). Returns an envelope
(plan rows + per-step results); never prints to stdout — helper chatter from the
underlying verbs routes to stderr. Raises echo.EchoError(2) on a bad bundle,
BEFORE any write."""
import contextlib
import echo_output
import echo_reflect
path, line = validate(bundle)
scope = str(bundle.get("scope") or "").strip()
proposals = bundle.get("reflect") or []
valid, errors = echo_reflect.validate(proposals)
rows: list[dict] = []
if valid:
echo_reflect.classify(valid)
rows = [{"action": p.get("_action"), "kind": p.get("kind"),
"title": p["title"], "path": p.get("_path")} for p in valid]
data = {"path": path, "agent_log_line": line, "scope": scope or None,
"reflect_rows": rows, "reflect_errors": errors,
"dry_run": not apply, "steps": {}}
if not apply:
return echo_output.envelope("session-end", data)
import echo_concurrency
import echo_ops
steps: dict[str, str] = {}
with echo_concurrency.vault_lock(), contextlib.redirect_stdout(sys.stderr):
# 1. The session log itself. A hard failure here aborts the whole bundle —
# nothing after it (including the heartbeat) runs, so orientation state
# can never point at a log that was never written.
echo.cmd_put(path, echo.temp_file(str(bundle["log_body"]).encode("utf-8")))
steps["session_log"] = "ok"
# 2. Agent-log line (best-effort by contract; queues itself when offline).
echo_ops.ensure_daily_log(line)
steps["agent_log"] = "ok"
# 3. Reflect proposals through the normal capture path (gate-aware).
if valid:
gated = 0
for p in valid:
if p.get("_action") == "error":
continue
env = echo_ops.capture_op(
p.get("kind"), p["title"], body_text=p.get("body") or "",
aliases=p.get("aliases") or [], sources=p.get("sources") or [],
tags=p.get("tags") or [], date=p.get("date"),
domain=p.get("domain", "business"),
inbox=bool(p.get("inbox")) or not p.get("kind"))
gated += 1 if env.get("action") == "duplicate-gate" else 0
steps["reflect"] = f"{len(valid) - gated}/{len(valid)} applied" \
+ (f", {gated} gated (re-propose with --merge-into)" if gated else "")
else:
steps["reflect"] = "skipped (none)"
# 4. Scope switch (optional).
if scope:
echo.cmd_scope("set", scope)
steps["scope"] = "ok"
else:
steps["scope"] = "unchanged"
# 5. Heartbeat LAST — the commit marker for the whole bundle.
echo.cmd_put("_agent/heartbeat/last-session.md",
echo.temp_file(f"{path} @ {echo.now_iso()}\n".encode("utf-8")))
steps["heartbeat"] = "ok"
# Index maintenance (2.3), NOT part of the bundle contract: push this
# machine's local recall index to the vault snapshot so other machines
# can seed/catch up. Best-effort — a failure never taints the bundle.
try:
import echo_recall
rix = echo_recall._load_local()
if rix and rix.n_docs:
rix.built = rix.built or echo.now_iso()
echo_recall.save_snapshot(rix)
steps["recall_snapshot"] = "ok"
except Exception: # noqa: BLE001
steps["recall_snapshot"] = "skipped"
data["steps"] = steps
return echo_output.envelope("session-end", data)
def session_end(bundle: dict, apply: bool = False) -> int:
"""CLI wrapper: same plan/summary text as before; envelope logic in session_end_op."""
env = session_end_op(bundle, apply=apply)
for e in env["reflect_errors"]:
print(f"skip: {e}", file=sys.stderr)
print(f"session-end plan ({'APPLY' if apply else 'dry-run'}):")
print(f" 1. session log -> {env['path']}")
print(f" 2. agent-log line: {env['agent_log_line']}")
rows = env["reflect_rows"]
if rows:
print(f" 3. reflect: {len(rows)} proposal(s)")
print("\n".join(f" {r['action'] or '?':7} | {('-' if r['action'] == 'inbox' else r['kind'] or '?'):9} "
f"| {r['title']} -> {r['path'] or '?'}" for r in rows))
else:
print(" 3. reflect: (none)")
print(f" 4. scope set: {env['scope']!r}" if env["scope"] else " 4. scope: (unchanged)")
print(f" 5. heartbeat -> {env['path']} @ <now> (written LAST — the commit marker)")
if env["dry_run"]:
print("\nsession-end: dry-run — re-run with --apply to write.")
return 0
print("\nsession-end: done — "
+ "; ".join(f"{k}: {v}" for k, v in env["steps"].items()))
return 0
@@ -0,0 +1,78 @@
#!/usr/bin/env python3
"""echo_stem.py — a deliberately light, dependency-free English stemmer. [2.3]
BM25 is purely lexical: "deploy", "deployed", and "deployment" are three unrelated
terms to it, so a paraphrased recall query misses notes stored with a different
inflection. This Porter-LITE stemmer conflates the common inflection families while
staying conservative — when in doubt it leaves the token alone, because an
over-stemmed index silently merges unrelated words (far worse than a missed match).
Applied by echo_recall.tokenize() at BOTH index and query time (recall-index
schema 3; older indexes are discarded and rebuilt). Pure function, unit-tested in
test_echo_client.py.
Design notes:
* plural / -ed / -ing families first (with the classic restore rules: doubled
consonant undoubling, at/bl/iz +e, cvc +e), then one derivational suffix pass,
then y->i and final-e normalization so "navigate"/"navigation"/"navigating"
all land on "navigat".
* every rule guards a minimum remaining stem length — "sing" is not "s".
"""
from __future__ import annotations
import re
_VOWEL = re.compile(r"[aeiouy]")
_DOUBLE = re.compile(r"([^aeiouylsz])\1$")
_CVC = re.compile(r"[^aeiou][aeiouy][^aeiouwxy]$")
# One derivational pass, most-specific first. Replacements keep families together:
# navigational/navigation -> navigat · saturation/saturated -> saturat ·
# deployment/deployed -> deploy · usefulness/useful -> use... (guarded by length).
_SUFFIXES = (
("ization", "ize"), ("ational", "ate"), ("fulness", "ful"), ("ousness", "ous"),
("iveness", "ive"), ("tional", "t"), ("biliti", "ble"), ("ements", ""),
("ment", ""), ("ness", ""), ("tion", "t"), ("sion", "s"), ("ance", ""),
("ence", ""), ("able", ""), ("ible", ""), ("ally", "al"), ("ously", "ous"),
("ively", "ive"), ("ly", ""),
)
def stem(t: str) -> str:
"""Stem one lowercase token. Never raises; returns the token unchanged when no
rule applies safely."""
if len(t) <= 3 or not t.isascii():
return t
# --- plurals -------------------------------------------------------------
if t.endswith("sses"):
t = t[:-2]
elif t.endswith("ies") and len(t) > 4:
t = t[:-3] + "i"
elif t.endswith("s") and not t.endswith(("ss", "us", "is")):
t = t[:-1]
# --- -ed / -ing (with restore rules) --------------------------------------
for suf in ("ingly", "edly", "ing", "ed"):
if t.endswith(suf):
base = t[: -len(suf)]
if len(base) >= 3 and _VOWEL.search(base):
t = base
if t.endswith(("at", "bl", "iz")):
t += "e"
elif _DOUBLE.search(t):
t = t[:-1]
elif len(t) == 3 and _CVC.search(t):
t += "e"
break
# --- one derivational suffix pass -----------------------------------------
for suf, rep in _SUFFIXES:
if t.endswith(suf):
base = t[: -len(suf)]
if len(base) >= 3 and len(base + rep) >= 3:
t = base + rep
break
# --- normalize: y->i (so penalty/penalties agree), then drop a final e -----
if len(t) > 3 and t.endswith("y") and t[-2] not in "aeiou":
t = t[:-1] + "i"
if len(t) > 4 and t.endswith("e"):
t = t[:-1]
return t
@@ -55,19 +55,25 @@ def parse_inbox(text: str) -> list[dict]:
return items
def list_inbox(as_json: bool = False) -> int:
def list_op() -> dict:
"""Core inbox listing (Phase 0): structured captures envelope, no printing."""
import echo_output
status, body = echo.request("GET", echo.vault_url(INBOX_PATH))
if status == 404:
items = []
else:
echo.check(status, body, f"triage list {INBOX_PATH}")
items = parse_inbox(body.decode(errors="replace"))
if as_json:
import echo_output
env = echo_output.envelope("triage-list", {"path": INBOX_PATH, "items": items,
return echo_output.envelope("triage-list", {"path": INBOX_PATH, "items": items,
"count": len(items)})
def list_inbox(as_json: bool = False) -> int:
env = list_op()
if as_json:
print(json.dumps(env, ensure_ascii=False))
return 0
items = env["items"]
if not items:
print("triage: inbox is empty — nothing to route.")
return 0
@@ -80,53 +86,76 @@ def list_inbox(as_json: bool = False) -> int:
return 0
def apply(proposals: list[dict], confirm: bool = False) -> int:
"""Validate -> classify -> preview; with confirm, route each via capture AND write
the processing-log audit line. Mirrors echo_reflect.apply's contract exactly."""
def route_op(proposals: list[dict], confirm: bool = False) -> dict:
"""Core triage routing (Phase 0): reflect pipeline + processing-log audit lines.
Returns an envelope (rows = classified preview; results = per-item outcomes);
never prints to stdout (audit-append chatter routes to stderr)."""
import contextlib
import echo_output
import echo_reflect
valid, errors = echo_reflect.validate(proposals)
for e in errors:
print(f"skip: {e}", file=sys.stderr)
if not valid:
print("triage: no valid proposals to route.")
return 0
rows: list[dict] = []
counts: dict[str, int] = {}
if valid:
echo_reflect.classify(valid)
counts = {a: sum(1 for p in valid if p.get("_action") == a)
for a in ("create", "update", "inbox", "error")}
print(f"triage: {len(valid)} proposal(s) — "
f"{counts['create']} new, {counts['update']} update, {counts['inbox']} stay-in-inbox"
+ (f", {counts['error']} error" if counts["error"] else ""))
print(echo_reflect.preview(valid))
if not confirm:
print("\ntriage: dry-run — re-run with --apply to route these and log the moves.")
return 0
rows = [{"action": p.get("_action"), "kind": p.get("kind"),
"title": p["title"], "path": p.get("_path")} for p in valid]
data = {"proposals": len(proposals or []), "valid": len(valid), "errors": errors,
"counts": counts, "rows": rows, "dry_run": not confirm,
"routed": 0, "gated": 0, "log": f"{LOG_DIR}/{echo.today()}.md", "results": []}
if not valid or not confirm:
return echo_output.envelope("triage", data)
import echo_ops
today_s = echo.today()
applied = gated = 0
routed = gated = 0
results = []
for p in valid:
if p.get("_action") == "error":
continue
if p.get("_action") == "inbox":
if p.get("_action") in ("error", "inbox"):
continue # routing an inbox line back to the inbox is a no-op, not a move
bodyfile = echo.temp_file((p.get("body") or "").encode()) if p.get("body") else None
rc = echo_ops.capture(
p.get("kind"), p["title"], bodyfile,
env = echo_ops.capture_op(
p.get("kind"), p["title"], body_text=p.get("body") or "",
aliases=p.get("aliases") or [], sources=p.get("sources") or [],
tags=p.get("tags") or [],
date=p.get("date"), domain=p.get("domain", "business"))
if rc == 76:
act = env.get("action", "")
results.append({"title": p["title"], "action": act,
"path": env.get("path"), "ok": bool(env.get("ok"))})
if act == "duplicate-gate":
gated += 1
continue
if rc != 0:
if not env.get("ok"):
continue
applied += 1
routed += 1
original = (p.get("line") or p["title"]).strip()
with contextlib.redirect_stdout(sys.stderr):
echo.cmd_append(f"{LOG_DIR}/{today_s}.md",
f"- {original}{p.get('_path', '?')}")
print(f"triage: routed {applied}/{len(valid)} item(s); audit in {LOG_DIR}/{today_s}.md. "
data.update(routed=routed, gated=gated, results=results)
return echo_output.envelope("triage", data)
def apply(proposals: list[dict], confirm: bool = False) -> int:
"""CLI wrapper: same preview/summary text as pre-Phase-0. Mirrors reflect's contract."""
env = route_op(proposals, confirm=confirm)
for e in env["errors"]:
print(f"skip: {e}", file=sys.stderr)
if not env["valid"]:
print("triage: no valid proposals to route.")
return 0
counts = env["counts"]
print(f"triage: {env['valid']} proposal(s) — "
f"{counts['create']} new, {counts['update']} update, {counts['inbox']} stay-in-inbox"
+ (f", {counts['error']} error" if counts["error"] else ""))
print("\n".join(f" {r['action'] or '?':7} | {('-' if r['action'] == 'inbox' else r['kind'] or '?'):9} "
f"| {r['title']} -> {r['path'] or '?'}" for r in env["rows"]))
if env["dry_run"]:
print("\ntriage: dry-run — re-run with --apply to route these and log the moves.")
return 0
gated = env["gated"]
print(f"triage: routed {env['routed']}/{env['valid']} item(s); audit in {env['log']}. "
"Originals kept in the inbox (deletion is explicit-only)."
+ (f" {gated} stopped at the duplicate gate — re-propose with the existing "
f"entity's title or use capture --merge-into." if gated else ""))
@@ -18,9 +18,19 @@ backfill what older vaults don't have yet:
5. stamp the marker `schema_version` to the current schema (4).
Dry-run by default; pass --apply to write. READ-ONLY without --apply.
**--fast (2.3)** is the incremental mode: it walks the listing (cheap), then fetches
only what's new, gone, missing a content hash, or in today's rotating verification
shard (hash(path) % 7 == weekday — the whole vault gets verified over a week of
fast sweeps while each run stays tiny; --all-shards forces everything). It updates
the LOCAL recall index and the entity index (both machine-owned), so it needs no
--apply gate and writes no notes. `load` auto-runs it when the last fast sweep is
older than ECHO_FAST_SWEEP_DAYS (default 7). This is what keeps the indexes honest
against edits made directly in Obsidian or by other clients.
Cross-platform: pure Python via echo.py.
Usage: sweep.py [--apply]
Usage: sweep.py [--apply] | sweep.py --fast [--all-shards]
"""
from __future__ import annotations
@@ -102,10 +112,128 @@ def walk(prefix: str = ""):
yield from walk(f"{prefix}{d}/")
def _stamp_path() -> Path:
import echo_queue
return echo_queue.state_dir() / "fast-sweep.stamp"
def _stamp_fast_sweep() -> None:
try:
p = _stamp_path()
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(echo.now_iso() + "\n", encoding="utf-8")
except OSError:
pass
def fast_sweep_age_days() -> float | None:
"""Days since the last fast/full sweep on THIS machine, or None if never."""
import datetime as dt
try:
stamp = _stamp_path().read_text(encoding="utf-8").strip()
then = dt.datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=dt.timezone.utc)
return (dt.datetime.now(dt.timezone.utc) - then).total_seconds() / 86400
except (OSError, ValueError):
return None
def fast(all_shards: bool = False) -> str:
"""Incremental index maintenance (2.3): true up the LOCAL recall index and the
entity index against the vault, fetching only new / gone / hash-missing notes
plus today's rotating verification shard. Index-only — writes no notes; the
indexes are machine-owned, so no --apply gate. Returns a one-line summary."""
import datetime as dt
import echo_recall
if get("_agent/echo-vault.md") is None:
return "fast-sweep skipped: vault not bootstrapped"
listing = [p for p in walk()
if p.endswith(".md") and p.rsplit("/", 1)[-1] not in SKIP_BASENAMES
and not TEMPLATE_RE.search(p)]
lset = set(listing)
rix = echo_recall._load_local()
if rix is None or not rix.n_docs:
# Nothing to maintain incrementally — build the local index outright
# (sub-second on a few-hundred-note vault; no vault snapshot write).
rix = echo_recall.rebuild()
_stamp_fast_sweep()
return f"fast-sweep: no local index — built one ({rix.n_docs} docs)"
index = idx_mod.load()
ents = index.get("entities", {})
removed = 0
for slug in [s for s, e in list(ents.items()) if e.get("path") not in lset]:
del ents[slug]
removed += 1
for p in [p for p in list(rix.length) if p not in lset]:
rix.remove(p)
removed += 1
try:
weekday = dt.date.fromisoformat(echo.today()).weekday()
except ValueError:
weekday = dt.date.today().weekday()
def in_shard(p: str) -> bool:
return int(idx_mod.content_hash(p), 16) % 7 == weekday
cands: set[str] = set()
for p in listing:
if not echo_recall._indexable(p):
continue
if p not in rix.length: # new note (any client, any tool)
cands.add(p)
continue
kind = kind_for(p)
if kind:
slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3])
if not (ents.get(slug) or {}).get("h"): # pre-2.3 entry: hash it once
cands.add(p)
continue
if all_shards or in_shard(p): # rotating verification shard
cands.add(p)
texts = echo.read_many(sorted(cands))
changed = 0
ents_dirty = removed > 0
for p, t in texts.items():
if t is None:
continue
h = idx_mod.content_hash(t)
if rix.content_hash(p) != h:
rix.add(p, t)
changed += 1
kind = kind_for(p)
if kind:
slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3])
prev = ents.get(slug) or {}
if prev.get("h") != h or prev.get("path") != p:
title = links.first_h1(t) or p.rsplit("/", 1)[-1][:-3]
aliases = list(prev.get("aliases", [])) + idx_mod.aliases_in_frontmatter(t)
idx_mod.upsert(index, slug, p, kind, title, aliases, h=h)
ents_dirty = True
echo_recall.save_local(rix)
if ents_dirty:
idx_mod.save(index)
_stamp_fast_sweep()
return (f"fast-sweep: {len(cands)} note(s) checked, {changed} reindexed, "
f"{removed} removed" + (" (all shards)" if all_shards else ""))
def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(description="Bring an ECHO vault up to the current feature spec")
ap.add_argument("--apply", action="store_true")
ap.add_argument("--fast", action="store_true",
help="incremental index maintenance (local recall + entity index only)")
ap.add_argument("--all-shards", action="store_true",
help="with --fast: verify every indexed note, not just today's shard")
args = ap.parse_args(argv)
if args.fast:
print(fast(all_shards=args.all_shards))
return 0
apply = args.apply
tag = "APPLY" if apply else "PLAN "
@@ -151,7 +279,8 @@ def main(argv: list[str] | None = None) -> int:
# aliases (e.g. mentions learned by capture) are preserved; upsert adds title variants.
prev_aliases = prev.get("aliases", []) if prev else []
aliases = list(prev_aliases) + idx_mod.aliases_in_frontmatter(text)
idx_mod.upsert(rebuilt, slug, path, kind, title, aliases)
idx_mod.upsert(rebuilt, slug, path, kind, title, aliases,
h=idx_mod.content_hash(text))
if not prev:
added += 1
elif prev.get("path") != path or prev.get("title") != title:
@@ -163,8 +292,9 @@ def main(argv: list[str] | None = None) -> int:
# ---- 2b. (re)build the recall (BM25) index -------------------------------
if apply:
rix = echo_recall.rebuild(prefetched=texts)
print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25)")
rix = echo_recall.rebuild(prefetched=texts, snapshot=True)
_stamp_fast_sweep()
print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25; local + vault snapshot)")
else:
n_idx = sum(1 for p in all_files if echo_recall._indexable(p))
print(f"sweep: {tag} recall index — would rebuild BM25 over {n_idx} corpus notes "
@@ -215,6 +215,42 @@ def test_links_create_related_section_when_absent() -> None:
assert changed and "## Related" in new and "[[resources/concepts/beta]]" in new
def test_stemmer_conflates_inflection_families() -> None:
import echo_stem
families = [
["deploy", "deploys", "deployed", "deploying", "deployment"],
["navigate", "navigation", "navigating", "navigational"],
["renew", "renews", "renewing"],
["prefer", "prefers", "preferring", "preferred"],
["penalty", "penalties"],
["saturate", "saturation", "saturated"],
["frequency", "frequencies"],
["expand", "expanded", "expanding"],
["meet", "meeting", "meetings"],
["service", "services"],
]
for fam in families:
stems = {echo_stem.stem(w) for w in fam}
assert len(stems) == 1, f"{fam} -> {stems}"
def test_stemmer_leaves_short_and_risky_tokens_alone() -> None:
import echo_stem
# over-stemming merges unrelated words — worse than a missed match
for w in ("sing", "thing", "was", "is", "bus", "this", "echo", "vault", "mpm"):
assert echo_stem.stem(w) == w, f"{w} mangled to {echo_stem.stem(w)}"
# documented non-conflations (the -al / d~s irregulars full Porter also skips)
assert echo_stem.stem("renewal") != echo_stem.stem("renew")
assert echo_stem.stem("expansion") != echo_stem.stem("expand")
def test_content_hash_stable_and_short() -> None:
import echo_index as ix
a = ix.content_hash("hello world")
assert a == ix.content_hash("hello world") and len(a) == 16
assert a != ix.content_hash("hello world!")
def _run_all() -> int:
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
failed = 0
@@ -165,13 +165,15 @@ def main() -> int:
# this shared cache instead of issuing a fresh GET per file per section.
texts = echo.read_many(md_files)
# Path membership + retired-path detection
# Path membership + retired-path detection. Retired patterns are checked FIRST:
# a file inside a retired tree is flagged even when a permissive route (the
# leaf-README rule) would otherwise match it — previously `archive/x/README.md`
# was silently absolved by `leaf-readme` and retired dirs survived unflagged.
for path in all_files:
if routes and not any(rx.match(path) for _, rx in routes):
replacement = next((repl for rx, repl in retired if rx.match(path)), None)
if replacement is not None:
flag("retired-path", f"{path}: retired location — should be {replacement}")
else:
elif routes and not any(rx.match(path) for _, rx in routes):
flag("unknown-path", f"{path}: matches no route in routing.json")
# Per-note frontmatter checks
@@ -1,6 +1,8 @@
---
name: echo-recall
description: Recall a topic from ECHO memory — matching notes plus their linked neighbourhood
argument-hint: "[topic, person, or project]"
allowed-tools: Bash(python3 *echo.py*), Bash(python *echo.py*), Bash(py -3 *echo.py*), Bash(ls /sessions/*)
---
Use the **echo-memory** skill to recall context for:
@@ -1,6 +1,8 @@
---
name: echo-reflect
description: Reflect on this session — extract durable memories and propose them for one-tap capture
argument-hint: "[optional focus, e.g. 'just decisions']"
allowed-tools: Bash(python3 *echo.py*), Bash(python *echo.py*), Bash(py -3 *echo.py*), Bash(ls /sessions/*)
---
Use the **echo-memory** skill to run **session reflection** (H5). Scan this conversation for
@@ -28,4 +30,9 @@ python3 "$ECHO" reflect proposals.json --apply # write — routes each via ca
`reflect` dedups every proposal against the entity index (so it shows create-vs-update),
skips below-confidence items, and routes `inbox` proposals to the capture inbox. Each
applied item goes through the normal `capture` path — canonical frontmatter, auto-linking,
and the recall index — and the writes are lock-guarded. $ARGUMENTS
and the recall index — and the writes are lock-guarded.
**Finishing the session?** Prefer the one-call bundle — `python3 "$ECHO" session-end
bundle.json --apply` — which applies the reflect proposals AND writes the session log,
Agent Log line, optional scope switch, and the heartbeat (last, as the commit marker)
together. See the echo-memory skill's **Session Logging** section for the bundle shape. $ARGUMENTS
@@ -1,6 +1,8 @@
---
name: echo-save
description: Save to ECHO memory — route content to its canonical home (search-first, idempotent)
argument-hint: "[what to remember]"
allowed-tools: Bash(python3 *echo.py*), Bash(python *echo.py*), Bash(py -3 *echo.py*), Bash(ls /sessions/*)
---
Use the **echo-memory** skill to persist this to the ECHO vault:
@@ -1,6 +1,8 @@
---
name: echo-sweep
description: Bring the ECHO vault up to the current feature spec (entity index + cross-links)
allowed-tools: Bash(python3 *sweep.py*), Bash(python *sweep.py*), Bash(ls /sessions/*), Bash(dirname *)
allowed-tools: Bash(python3 *sweep.py*), Bash(python *sweep.py*), Bash(py -3 *sweep.py*), Bash(ls /sessions/*), Bash(dirname *)
disable-model-invocation: true
---
Use the **echo-memory** skill to bring the vault up to the current spec. Run the sweep **dry-run first**, show the operator the plan, and only `--apply` on their go-ahead:
@@ -1,5 +1,8 @@
---
name: echo-triage
description: Triage the ECHO inbox — route aging captures to their canonical homes and log the moves
allowed-tools: Bash(python3 *echo.py*), Bash(python *echo.py*), Bash(py -3 *echo.py*), Bash(ls /sessions/*)
disable-model-invocation: true
---
Use the **echo-memory** skill to run **one-tap Inbox Triage** via the `triage` verb (it reuses the reflect pipeline: classify against the entity index → preview → apply via `capture`, and writes the audit log for you).
+5 -5
View File
@@ -3,17 +3,17 @@
"date": "2026-07-03",
"retrieval": {
"current (hybrid+priors, full corpus)": {
"precision_at_5": 0.25,
"precision_at_5": 0.243,
"recall_at_5": 1.0,
"mrr": 1.0,
"session_journal_queries_answered": "2/2",
"session_journal_queries_answered": "3/3",
"freshness_top1_correct": true
},
"baseline (keyword, entities only)": {
"precision_at_5": 0.2,
"precision_at_5": 0.186,
"recall_at_5": 0.75,
"mrr": 0.75,
"session_journal_queries_answered": "0/2",
"mrr": 0.786,
"session_journal_queries_answered": "0/3",
"freshness_top1_correct": false
}
},
+10
View File
@@ -135,6 +135,16 @@ QUERIES = [
("contract renewal Q3", {"resources/companies/vantage-systems.md"}, False),
("renegotiation kickoff", {"journal/daily/2026-06-20.md"}, True),
("graph neighbourhood expansion", {"resources/concepts/graph-expansion.md"}, False),
# ---- paraphrase set (2.3): morphological variants + rephrasings the pre-2.3
# purely-lexical index missed — the stemming gain is measured, not assumed.
("renewing enterprise contracts", {"resources/companies/vantage-systems.md"}, False),
("navigational cleanup", {"projects/active/website-redesign.md",
"projects/archived/old-website.md"}, False),
("preferring uv for tooling", {"_agent/memory/semantic/tooling-preferences.md"}, False),
("penalties in the SLA",
{"_agent/sessions/2026-06-20-1000-vantage-contract-review.md"}, True),
("saturated term frequencies", {"resources/concepts/bm25-ranking.md"}, False),
("expanded linked neighbourhoods", {"resources/concepts/graph-expansion.md"}, False),
]
FRESHNESS_QUERY = "marketing site navigation"
FRESHNESS_TOP1 = "projects/active/website-redesign.md"
+126 -3
View File
@@ -27,7 +27,9 @@ def check(name, cond, detail=""):
class Harness:
def __init__(self, base):
import tempfile
self.base = base
self.state_dir = tempfile.mkdtemp()
def http(self, method, url, body=None, headers=None):
data = body.encode() if isinstance(body, str) else body
@@ -52,8 +54,10 @@ class Harness:
return None if body == "<<MISSING>>" else body
def echo(self, *args):
# ECHO_STATE_DIR isolation matters since 2.3: the recall index is LOCAL-first,
# and without this the suite would write into the real ~/.echo-memory.
env = dict(os.environ, ECHO_BASE=self.base, ECHO_KEY=KEY, ECHO_VERIFY="1",
ECHO_TODAY="2026-06-21")
ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=self.state_dir)
return subprocess.run([sys.executable, str(args[0]), *args[1:]],
capture_output=True, text=True, env=env)
@@ -230,8 +234,9 @@ def main():
check("v1.5 recall finds a session log by body term",
"_agent/sessions/2026-06-15-1200-mpm-review.md" in r.stdout, r.stdout)
rix2 = h.ground("_agent/index/recall-index.json") or ""
check("v1.5 recall index carries doc meta (schema 2)",
'"schema": 2' in rix2 or '"schema":2' in rix2, rix2[:200])
check("v2.3 recall snapshot is schema 3 with build stamp + content hashes",
('"schema": 3' in rix2 or '"schema":3' in rix2)
and '"built"' in rix2 and '"h"' in rix2, rix2[:200])
# 14. recency-aware ranking: same term, fresher note wins.
h.seed("resources/concepts/old-idea.md",
@@ -298,6 +303,14 @@ def main():
check("v1.5 sweep backfills the kind tag",
"tags: [company]" in drift or '<fm:tags=["company"]>' in drift, drift)
# 16b. lint blind spot (1.6.0): a seed README inside a retired tree must be
# flagged retired-path — previously the permissive leaf-readme route matched
# first and absolved it, so retired dirs holding only a README went unflagged.
h.seed("archive/notes/README.md", "This folder holds archived notes.\n")
r = h.echo(LINT)
check("v1.6 lint flags a README inside a retired tree",
"archive/notes/README.md: retired location" in r.stdout, r.stdout[-800:])
# 17. hooks: stop-hook nudges once on a substantive session, then stays quiet.
import tempfile
tdir = Path(tempfile.mkdtemp())
@@ -337,6 +350,116 @@ def main():
check("v1.5 session-start hook stays quiet on resume",
r.returncode == 0 and not r.stdout.strip(), r.stdout)
# 18. (2.1.0) brief load: digest form, budget respected, key facts present.
h.seed("_agent/memory/semantic/operator-preferences.md",
"---\ntype: semantic-memory\ncreated: 2026-06-01\n---\n# Operator Preferences\n\n"
"## Fact / Pattern\n- prefers concise output\n\n## Observations\n"
+ "\n".join(f"- 2026-06-{i:02d}: observation {i}" for i in range(1, 15)) + "\n")
h.seed("inbox/captures/inbox.md", "- 2026-06-01: old capture\n- 2026-06-20: new capture\n")
r = h.echo(ECHO, "load", "--brief")
check("v2.1 brief load renders the digest header",
"ECHO load (brief)" in r.stdout, r.stdout[:200])
check("v2.1 brief load keeps Fact / Pattern",
"prefers concise output" in r.stdout, r.stdout[:800])
check("v2.1 brief load trims observations to the last 10",
"last 10 of 14" in r.stdout and "observation 14" in r.stdout
and "observation 2" not in r.stdout.replace("observation 2026", ""), r.stdout[-1200:])
check("v2.1 brief load reports inbox as a count",
"inbox: 2 capture(s), oldest 20d" in r.stdout, r.stdout[-400:])
check("v2.1 brief load includes the marker line",
"echo-vault.md" in r.stdout and "marker" in r.stdout, r.stdout[:300])
# 19. (2.1.0) session-end: dry-run writes nothing; apply lands all steps with
# the heartbeat LAST; a bad bundle aborts before any write.
se_env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1",
ECHO_TODAY="2026-06-21", ECHO_NOW="2359")
bundle = {"slug": "quickwins-ship",
"log_body": "---\ntype: session-log\nstatus: complete\ncreated: 2026-06-21\n---\n"
"# Session Log\n\n## Goal\nShip the quick wins.\n",
"agent_log_line": "- 2026-06-21: shipped the quick wins",
"reflect": [{"title": "Quark Preference", "kind": "semantic",
"body": "The operator prefers quark.", "confidence": 0.9}]}
bfile = HERE / "_session_bundle.json"
bfile.write_text(json.dumps(bundle), encoding="utf-8")
sess_path = "_agent/sessions/2026-06-21-2359-quickwins-ship.md"
r = subprocess.run([sys.executable, str(ECHO), "session-end", str(bfile)],
capture_output=True, text=True, env=se_env)
check("v2.1 session-end dry-run previews the plan",
"dry-run" in r.stdout and sess_path in r.stdout, r.stdout)
check("v2.1 session-end dry-run writes nothing", h.ground(sess_path) is None)
r = subprocess.run([sys.executable, str(ECHO), "session-end", str(bfile), "--apply"],
capture_output=True, text=True, env=se_env)
check("v2.1 session-end apply lands the session log",
"Ship the quick wins" in (h.ground(sess_path) or ""), r.stdout + r.stderr)
check("v2.1 session-end apply routes the reflect proposal",
h.ground("_agent/memory/semantic/quark-preference.md") is not None)
check("v2.1 session-end apply writes the heartbeat last (commit marker)",
sess_path in (h.ground("_agent/heartbeat/last-session.md") or ""))
daily = h.ground("journal/daily/2026-06-21.md") or ""
check("v2.1 session-end apply appends the agent-log line",
"shipped the quick wins" in daily, daily[-300:])
bad = dict(bundle, slug="Bad_Slug!")
bfile.write_text(json.dumps(bad), encoding="utf-8")
r = subprocess.run([sys.executable, str(ECHO), "session-end", str(bfile), "--apply"],
capture_output=True, text=True, env=se_env)
bfile.unlink()
check("v2.1 session-end rejects a bad slug before any write",
r.returncode == 2 and sess_path in (h.ground("_agent/heartbeat/last-session.md") or ""),
r.stderr)
# ---------------- v2.3 index train --------------------------------------
# 20. local-first recall index: capture updates the LOCAL index with ZERO
# vault snapshot writes — the vault copy only changes on sweep/session-end.
snap_before = h.ground("_agent/index/recall-index.json") or ""
r = h.echo(ECHO, "capture", "Quixotic Widget", "--kind", "concept", "-")
r = subprocess.run([sys.executable, str(ECHO), "capture", "Quixotic Widget2", "--kind",
"concept", "-"], input="A gadget for quixotry deployment.\n",
capture_output=True, text=True,
env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY,
ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=h.state_dir))
snap_after = h.ground("_agent/index/recall-index.json") or ""
check("v2.3 capture does NOT write the vault snapshot", snap_before == snap_after)
local_files = [p for p in Path(h.state_dir).glob("recall-index-*.json")]
check("v2.3 capture maintains the LOCAL index", len(local_files) == 1,
str(local_files))
r = h.echo(ECHO, "recall", "quixotry")
check("v2.3 locally-indexed capture is recallable",
"resources/concepts/quixotic-widget2.md" in r.stdout, r.stdout[:400])
# 21. stemming: a morphologically different query still hits (deployment ~ deploying).
r = h.echo(ECHO, "recall", "deploying quixotry")
check("v2.3 stemmed recall bridges inflections",
"resources/concepts/quixotic-widget2.md" in r.stdout, r.stdout[:400])
# 22. alias query expansion: an alias-phrased query surfaces the entity even
# though the alias never appears in any note body.
r = subprocess.run([sys.executable, str(ECHO), "capture", "Kappa Engine", "--kind",
"concept", "-", "--aliases", "turbokappa"],
input="Compression tuning notes for the kappa engine core.\n",
capture_output=True, text=True,
env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY,
ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=h.state_dir))
r = h.echo(ECHO, "recall", "turbokappa compression")
check("v2.3 alias expansion surfaces the aliased entity",
"resources/concepts/kappa-engine.md" in r.stdout, r.stdout[:400])
# 23. incremental fast sweep: an out-of-band edit (seeded directly, as if by
# Obsidian) and a deletion are trued up without a full rebuild.
h.seed("resources/concepts/quixotic-widget.md",
"---\ntype: concept\nstatus: active\ncreated: 2026-06-21\nupdated: 2026-06-21\n"
"tags: [concept]\n---\n# Quixotic Widget\n\nNow mentions zanzibar explicitly.\n")
h.http("DELETE", f"{base}/vault/resources/concepts/quixotic-widget2.md")
r = h.echo(SWEEP, "--fast", "--all-shards")
check("v2.3 fast sweep reports its work",
"fast-sweep:" in r.stdout and "reindexed" in r.stdout, r.stdout + r.stderr)
r = h.echo(ECHO, "recall", "zanzibar")
check("v2.3 fast sweep picked up the out-of-band edit",
"resources/concepts/quixotic-widget.md" in r.stdout, r.stdout[:400])
idx5 = h.ground("_agent/index/entities.json") or ""
check("v2.3 fast sweep drops deleted notes from the entity index",
"quixotic-widget2" not in idx5)
check("v2.3 entity index carries content hashes", '"h"' in idx5, idx5[:300])
print(f"\n{len(failures)} failure(s)" if failures else "\nall feature tests passed")
return 1 if failures else 0
finally:
+171
View File
@@ -0,0 +1,171 @@
#!/usr/bin/env python3
"""test_mcp_server.py — echo-mcp (2.2) end-to-end against the mock vault.
Spawns mcp-server/app.py + mock_olrapi and drives the real streamable-HTTP MCP
protocol: health (open), auth (401), initialize, tools/list, and the memory-day
tools/call flow (capture -> duplicate gate as DATA -> merge_into -> recall ->
get_note -> log_session with heartbeat commit).
Requires the `mcp` package in the interpreter that runs the SERVER. Set
ECHO_MCP_PYTHON to a venv python that has it; otherwise the current interpreter
is tried and the suite SKIPS (exit 0) when the SDK is absent the other suites
don't depend on it.
Run: python test_mcp_server.py [--port 8862] [--mcp-port 8767]
"""
from __future__ import annotations
import argparse
import json
import os
import subprocess
import sys
import tempfile
import time
import urllib.error
import urllib.request
from pathlib import Path
HERE = Path(__file__).resolve().parent
APP = HERE.parent / "mcp-server" / "app.py"
KEY = "test-key-not-a-real-secret"
TOKEN = "local-test-token"
failures = []
def check(name, cond, detail=""):
print(f"{'ok ' if cond else 'FAIL'} {name}" + (f" -- {detail}" if not cond else ""))
if not cond:
failures.append(name)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--port", type=int, default=8862)
ap.add_argument("--mcp-port", type=int, default=8767)
a = ap.parse_args()
mock_base = f"http://127.0.0.1:{a.port}"
mcp_url = f"http://127.0.0.1:{a.mcp_port}/mcp"
server_py = os.environ.get("ECHO_MCP_PYTHON") or sys.executable
probe = subprocess.run([server_py, "-c", "import mcp"], capture_output=True)
if probe.returncode != 0:
print("SKIP: `mcp` SDK not installed for the server interpreter "
"(set ECHO_MCP_PYTHON to a venv python that has it)")
return 0
def http(method, url, body=None, headers=None, timeout=30):
data = body.encode() if isinstance(body, str) else body
req = urllib.request.Request(url, data=data, method=method, headers=headers or {})
try:
with urllib.request.urlopen(req, timeout=timeout) as r:
return r.status, r.read().decode("utf-8", "replace")
except urllib.error.HTTPError as e:
return e.code, e.read().decode("utf-8", "replace")
except Exception as e: # noqa: BLE001
return 0, str(e)
_id = [0]
def rpc(method, params):
_id[0] += 1
st, body = http("POST", mcp_url, json.dumps(
{"jsonrpc": "2.0", "id": _id[0], "method": method, "params": params}),
headers={"Content-Type": "application/json",
"Accept": "application/json, text/event-stream",
"Authorization": f"Bearer {TOKEN}"})
return st, (json.loads(body) if body.strip().startswith("{") else {})
def call(name, args):
st, d = rpc("tools/call", {"name": name, "arguments": args})
res = d.get("result") or {}
payload = {}
if res.get("content"):
try:
payload = json.loads(res["content"][0]["text"])
except Exception: # noqa: BLE001
payload = {}
return res.get("isError", False), payload
mock = subprocess.Popen([sys.executable, str(HERE / "mock_olrapi.py"), "--port", str(a.port)],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
srv = None
try:
for _ in range(50):
try:
urllib.request.urlopen(f"{mock_base}/__debug__reset", data=b"", timeout=1)
break
except Exception:
time.sleep(0.1)
http("PUT", f"{mock_base}/vault/_agent/echo-vault.md",
"---\nschema_version: 4\n---\n# marker\n",
headers={"Authorization": f"Bearer {KEY}"})
env = dict(os.environ, ECHO_BASE=mock_base, ECHO_KEY=KEY, ECHO_MCP_TOKEN=TOKEN,
ECHO_MCP_PORT=str(a.mcp_port), ECHO_STATE_DIR=tempfile.mkdtemp(),
ECHO_TODAY="2026-07-28")
srv = subprocess.Popen([server_py, str(APP)], env=env,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
for _ in range(60):
st, _b = http("GET", f"http://127.0.0.1:{a.mcp_port}/health", timeout=2)
if st == 200:
break
time.sleep(0.5)
st, body = http("GET", f"http://127.0.0.1:{a.mcp_port}/health")
try:
hb = json.loads(body)
except Exception: # noqa: BLE001
hb = {}
check("health is open and green", st == 200 and hb.get("ok") is True
and hb.get("vault_reachable") is True, body[:200])
st, _b = http("POST", mcp_url, "{}", headers={"Content-Type": "application/json"})
check("MCP endpoint requires the bearer token (401)", st == 401, str(st))
st, d = rpc("initialize", {"protocolVersion": "2025-06-18", "capabilities": {},
"clientInfo": {"name": "eval", "version": "0"}})
check("initialize answers with serverInfo",
d.get("result", {}).get("serverInfo", {}).get("name") == "echo-mcp", json.dumps(d)[:200])
st, d = rpc("tools/list", {})
tools = [t["name"] for t in d.get("result", {}).get("tools", [])]
check("tools/list exposes the full profile (14 tools)", len(tools) == 14, str(tools))
e, b = call("echo_capture", {"title": "Vera Lumen", "kind": "person",
"body": "CTO at Fluxcorp, met at the summit."})
check("capture create over MCP", not e and b.get("action") == "created"
and b.get("path") == "resources/people/vera-lumen.md", json.dumps(b))
e, b = call("echo_capture", {"title": "Vera Lumen Jr", "kind": "person"})
check("duplicate gate is data, not isError", not e
and b.get("action") == "duplicate-gate" and b.get("candidates"), json.dumps(b))
e, b = call("echo_capture", {"title": "Vera Lumen Jr", "kind": "person",
"merge_into": "vera-lumen", "body": "Follow-up."})
check("merge_into resolves the gate as an update", not e and b.get("action") == "updated",
json.dumps(b))
e, b = call("echo_recall", {"query": "fluxcorp", "budget_chars": 900})
check("recall over MCP finds by body term", not e and any(
h["path"] == "resources/people/vera-lumen.md" for h in b.get("primary", [])),
json.dumps(b)[:300])
e, b = call("echo_get_note", {"path": "resources/people/vera-lumen.md"})
check("get_note returns frontmatter + content", not e
and b.get("frontmatter", {}).get("type") == "person", json.dumps(b)[:200])
e, b = call("echo_get_note", {"path": "../etc/passwd"})
check("get_note rejects path traversal", not b.get("ok"))
e, b = call("echo_log_session", {"slug": "mcp-e2e", "hhmm": "2345", "apply": True,
"log_body": "---\ntype: session-log\n---\n# S\n\n## Goal\ne2e\n"})
check("log_session commits with the heartbeat", not e
and b.get("steps", {}).get("heartbeat") == "ok", json.dumps(b))
e, b = call("echo_capture", {"title": "Bad Kind", "kind": "wizard"})
check("unknown kind rejected with the valid list", not b.get("ok")
and "wizard" in b.get("error", ""), json.dumps(b))
print(f"\n{len(failures)} failure(s)" if failures else "\nall mcp-server tests passed")
return 1 if failures else 0
finally:
if srv:
srv.terminate()
mock.terminate()
if __name__ == "__main__":
raise SystemExit(main())
+34
View File
@@ -115,6 +115,40 @@ def main():
check("offline load flags OFFLINE", "OFFLINE" in r.stdout, r.stdout)
check("offline load serves cached marker", "EVALMARK-marker" in r.stdout, r.stdout)
# --- Phase 5 (2.1.0): vault DOWN -> capture queues a SEMANTIC record ------
r = echo(DEAD, "capture", "Offline Person", "--kind", "person")
check("offline capture exits 0 (queued)", r.returncode == 0, r.stdout + r.stderr)
check("offline capture reports queued", "queued (offline): capture" in r.stdout, r.stdout)
outbox = Path(state) / "outbox.ndjson"
check("capture queued as an op record",
outbox.exists() and '"op": "capture"' in outbox.read_text(encoding="utf-8"))
# same capture again while offline -> deduped by idem_key, not double-queued
echo(DEAD, "capture", "Offline Person", "--kind", "person")
recs = [ln for ln in outbox.read_text(encoding="utf-8").splitlines() if '"op": "capture"' in ln]
check("offline capture is idempotent in the queue", len(recs) == 1, str(len(recs)))
# --- Phase 6 (2.1.0): vault UP -> flush replays capture THROUGH capture ---
srv = start_mock()
r = echo(mock_base, "flush")
check("flush replays the queued capture", "flushed" in r.stdout, r.stdout + r.stderr)
note = ground("resources/people/offline-person.md")
check("replayed capture created the routed note",
note is not None and "type: person" in note, str(note)[:200])
check("replayed capture used the capture-time date",
note is not None and "created: 2026-06-22" in note, str(note)[:200])
# --- Phase 7 (2.1.0): duplicate gate ON REPLAY keeps + flags the record ---
echo(mock_base, "capture", "Zebulon Quargle", "--kind", "person") # the existing entity
r = echo(DEAD, "capture", "Zebulon Quargle Junior", "--kind", "person")
check("lookalike capture queues offline", "queued (offline)" in r.stdout, r.stdout)
r = echo(mock_base, "flush")
check("gated replay is flagged, not landed",
"needs attention" in r.stdout and "replay-gated" in r.stdout, r.stdout + r.stderr)
check("gated replay did NOT create the duplicate",
ground("resources/people/zebulon-quargle-junior.md") is None)
check("gated record survives in the outbox",
'"needs_attention"' in (outbox.read_text(encoding="utf-8") if outbox.exists() else ""))
print(f"\n{len(failures)} failure(s)" if failures else "\nall offline-queue tests passed")
return 1 if failures else 0
finally:
+182
View File
@@ -0,0 +1,182 @@
#!/usr/bin/env python3
"""test_ops_api.py — Phase 0 (return-not-print) contract: every high-level op has a
core `*_op` function that RETURNS an envelope dict and prints nothing to stdout.
This is the seam the MCP server wraps (docs/MCP-SERVER-SPEC.md §4): the CLI wrappers
are tested by the other suites; here we call the cores in-process against the mock and
assert (a) the envelope shapes and (b) stdout purity a stray print() would corrupt
an MCP stdio/HTTP response stream.
Run: python test_ops_api.py [--port 8850]
"""
from __future__ import annotations
import argparse
import contextlib
import io
import json
import os
import subprocess
import sys
import tempfile
import time
import urllib.request
from pathlib import Path
HERE = Path(__file__).resolve().parent
SCRIPTS = HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts"
KEY = "test-key-not-a-real-secret"
ap = argparse.ArgumentParser()
ap.add_argument("--port", type=int, default=8850)
a = ap.parse_args()
BASE = f"http://127.0.0.1:{a.port}"
# env must be set BEFORE importing echo (it resolves config at import time)
os.environ.update(ECHO_BASE=BASE, ECHO_KEY=KEY, ECHO_TODAY="2026-06-21",
ECHO_NOW="2300", ECHO_STATE_DIR=tempfile.mkdtemp(), ECHO_VERIFY="0")
sys.path.insert(0, str(SCRIPTS))
failures = []
def check(name, cond, detail=""):
print(f"{'ok ' if cond else 'FAIL'} {name}" + (f" -- {detail}" if not cond else ""))
if not cond:
failures.append(name)
def http(method, url, body=None):
data = body.encode() if isinstance(body, str) else body
req = urllib.request.Request(url, data=data, method=method,
headers={"Authorization": f"Bearer {KEY}"})
try:
with urllib.request.urlopen(req, timeout=10) as r:
return r.status, r.read().decode("utf-8", "replace")
except Exception as e: # noqa: BLE001
return getattr(e, "code", 0), ""
def pure(fn, *args, **kw):
"""Call fn capturing stdout; return (result, captured_stdout)."""
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
out = fn(*args, **kw)
return out, buf.getvalue()
def main():
srv = subprocess.Popen([sys.executable, str(HERE / "mock_olrapi.py"), "--port", str(a.port)],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
try:
for _ in range(50):
try:
urllib.request.urlopen(f"{BASE}/__debug__reset", data=b"", timeout=1)
break
except Exception:
time.sleep(0.1)
http("PUT", f"{BASE}/vault/_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n")
http("PUT", f"{BASE}/vault/_agent/context/current-context.md",
"---\ntype: context-bundle\nscope_updated: \"2026-06-20\"\ncreated: 2026-06-01\n---\n"
"# Current Context\n\n## Scope\ntesting phase 0\n\n## Scope History\n")
import echo
import echo_doctor
import echo_ops
import echo_recall
import echo_reflect
import echo_session
import echo_triage
# capture_op: create / duplicate-gate / dry-run — envelopes, stdout pure
env, out = pure(echo_ops.capture_op, "person", "Zara Quix", body_text="Met at the expo.")
check("capture_op create envelope", env.get("ok") is True and env.get("action") == "created"
and env.get("path") == "resources/people/zara-quix.md", json.dumps(env))
check("capture_op prints nothing to stdout", out == "", out[:200])
env, out = pure(echo_ops.capture_op, "person", "Zara Quix Junior", body_text="")
check("capture_op duplicate-gate is data, not an exit code",
env.get("ok") is False and env.get("action") == "duplicate-gate"
and env.get("candidates"), json.dumps(env))
check("capture_op gate prints nothing", out == "", out[:200])
env, out = pure(echo_ops.capture_op, "company", "Plan Co", body_text="", dry_run=True)
check("capture_op dry-run envelope", env.get("action") == "dry-run:create"
and env.get("path") == "resources/companies/plan-co.md", json.dumps(env))
# resolve_op / link_op
env, out = pure(echo_ops.resolve_op, "zara quix")
check("resolve_op returns the match dict", env.get("match") is True
and env.get("path") == "resources/people/zara-quix.md", json.dumps(env))
pure(echo_ops.capture_op, "concept", "Gizmo", body_text="")
env, out = pure(echo_ops.link_op, "resources/people/zara-quix.md", "resources/concepts/gizmo.md")
check("link_op envelope", env.get("ok") is True and env.get("a_changed") in (True, False),
json.dumps(env))
check("link_op prints nothing", out == "", out[:200])
# recall_op
env, out = pure(echo_recall.recall_op, "expo")
check("recall_op envelope shape", env.get("action") == "recall"
and isinstance(env.get("primary"), list) and isinstance(env.get("linked"), list),
json.dumps(env)[:200])
check("recall_op prints nothing", out == "", out[:200])
# triage list_op / route_op (dry-run)
http("PUT", f"{BASE}/vault/inbox/captures/inbox.md", "- 2026-06-10: try the quorlab tool\n")
env, out = pure(echo_triage.list_op)
check("triage list_op envelope", env.get("count") == 1
and env["items"][0]["age_days"] == 11, json.dumps(env))
props = [{"title": "Quorlab Tool", "kind": "reference", "confidence": 0.9,
"line": "- 2026-06-10: try the quorlab tool"}]
env, out = pure(echo_triage.route_op, props, False)
check("triage route_op dry-run rows", env.get("dry_run") is True
and env["rows"][0]["action"] == "create", json.dumps(env))
check("triage route_op prints nothing", out == "", out[:200])
# reflect apply_op (applied path, using capture_op internally)
env, out = pure(echo_reflect.apply_op,
[{"title": "Blorp Pattern", "kind": "semantic",
"body": "The operator prefers blorp.", "confidence": 0.9}], True)
check("reflect apply_op applies via capture_op", env.get("applied") == 1
and env["results"][0]["action"] == "created", json.dumps(env))
check("reflect apply_op prints nothing", out == "", out[:200])
# scope ops
env, out = pure(echo.scope_show_op)
check("scope_show_op returns scope + freshness", env.get("scope") == "testing phase 0"
and env.get("scope_updated") == "2026-06-20", json.dumps(env))
env, out = pure(echo.scope_set_op, "phase zero refactor")
check("scope_set_op envelope", env.get("action") == "scope-set"
and env.get("scope_updated") == "2026-06-21", json.dumps(env))
check("scope_set_op prints nothing", out == "", out[:200])
# load_op (brief)
env, out = pure(echo.load_op, True)
check("load_op returns sections + brief", "marker" in env.get("sections", {})
and "ECHO load (brief)" in env.get("brief", ""), json.dumps(env)[:200])
check("load_op prints nothing", out == "", out[:200])
# doctor run_op
env, out = pure(echo_doctor.run_op)
check("doctor run_op checks list", isinstance(env.get("checks"), list)
and env.get("fatal") is None and env.get("ok") is True, json.dumps(env))
check("doctor run_op prints nothing", out == "", out[:200])
# session_end_op: dry-run envelope; bad bundle raises EchoError(2) pre-write
bundle = {"slug": "phase-zero", "log_body": "---\ntype: session-log\n---\n# S\n\n## Goal\nx\n"}
env, out = pure(echo_session.session_end_op, bundle, False)
check("session_end_op dry-run envelope", env.get("dry_run") is True
and env.get("path") == "_agent/sessions/2026-06-21-2300-phase-zero.md", json.dumps(env))
check("session_end_op prints nothing", out == "", out[:200])
try:
echo_session.session_end_op({"slug": "Bad Slug!", "log_body": "x"}, True)
check("session_end_op rejects a bad slug", False, "no exception raised")
except RuntimeError as exc:
check("session_end_op rejects a bad slug", getattr(exc, "code", 0) == 2, str(exc))
print(f"\n{len(failures)} failure(s)" if failures else "\nall ops-api tests passed")
return 1 if failures else 0
finally:
srv.terminate()
if __name__ == "__main__":
raise SystemExit(main())
+3 -1
View File
@@ -59,7 +59,9 @@ def main():
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
def echo(*args):
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="0")
import tempfile
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="0",
ECHO_STATE_DIR=globals().setdefault("_STATE_DIR", tempfile.mkdtemp()))
return subprocess.run([sys.executable, str(ECHO), *args], capture_output=True, text=True, env=env)
def ground(path):
+3 -1
View File
@@ -51,7 +51,9 @@ def main():
return getattr(e, "code", 0), ""
def echo(*args, stdin=None):
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1", ECHO_TODAY="2026-06-22")
import tempfile
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1",
ECHO_TODAY="2026-06-22", ECHO_STATE_DIR=tempfile.mkdtemp())
return subprocess.run([sys.executable, str(ECHO), *args], input=stdin,
capture_output=True, text=True, env=env)
+417
View File
@@ -0,0 +1,417 @@
#!/usr/bin/env python3
"""echo-mcp — the containerized MCP server over the ECHO ops layer. [2.2]
Streamable-HTTP (stateless JSON) MCP server exposing the plugin's `*_op` cores
(Phase 0, shipped 2.1.1) as typed tools. Runs next to the Obsidian REST API on
the same box, so every vault round-trip behind a tool call is LAN-local.
Spec: docs/MCP-SERVER-SPEC.md. Highlights implemented here:
* bearer-token auth on everything except /health (constant-time compare);
* tool results = the op envelopes, returned as structured JSON;
* duplicate gate / offline queueing are DATA (never protocol errors), so the
model deliberately chooses merge_into/force instead of blind-retrying;
* tool profiles: ECHO_MCP_TOOLS=core exposes only the six daily drivers
(schema tokens cost context on every surface that lists them);
* result budgets: echo_recall(budget_chars), echo_get_note(section/max_chars);
* per-write serialization (one process mediates all MCP writes, so a plain
lock removes the advisory-lock race window for server-mediated writes);
* startup validation: fail fast (crash-loop visibly) on missing env.
Deliberately NOT here in v1 (spec §7.2 backlog): entity-index TTL cache the
atomic_index_update correctness fix depends on a FRESH re-read under the lock,
so caching idx_mod.load would reintroduce the clobber race it closed. LAN
adjacency + the warm keep-alive pool make the read ~ms anyway.
Env: ECHO_BASE, ECHO_KEY, ECHO_MCP_TOKEN (required) · ECHO_OWNER ·
ECHO_MCP_PORT=8765 · ECHO_MCP_TOOLS=full|core · ECHO_STATE_DIR=/data
"""
from __future__ import annotations
import contextlib
import hmac
import json
import os
import re
import subprocess
import sys
import threading
from pathlib import Path
# --- locate the plugin scripts (image layout first, repo layout for local dev) ---
_HERE = Path(__file__).resolve().parent
for _cand in (_HERE / "scripts",
_HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts"):
if _cand.is_dir():
SCRIPTS = _cand
break
else: # pragma: no cover
sys.exit("echo-mcp: cannot locate the ECHO scripts directory")
sys.path.insert(0, str(SCRIPTS))
# --- startup validation: a misdeployed container must crash-loop visibly ---------
_missing = [k for k in ("ECHO_BASE", "ECHO_KEY", "ECHO_MCP_TOKEN") if not os.environ.get(k)]
if _missing:
sys.exit(f"echo-mcp: missing required env: {', '.join(_missing)} "
"— check the PORT template / secret refs")
# Modules aliased *_mod: several tool functions below share a module's name
# (echo_recall, echo_reflect, ...) and would shadow it at module scope otherwise.
import echo # noqa: E402
import echo_doctor as doctor_mod # noqa: E402
import echo_index as idx_mod # noqa: E402
import echo_ops as ops_mod # noqa: E402
import echo_queue as queue_mod # noqa: E402
import echo_recall as recall_mod # noqa: E402
import echo_reflect as reflect_mod # noqa: E402
import echo_session as session_mod # noqa: E402
import echo_triage as triage_mod # noqa: E402
from mcp.server.fastmcp import FastMCP # noqa: E402
from starlette.requests import Request # noqa: E402
from starlette.responses import JSONResponse, Response # noqa: E402
from starlette.middleware.base import BaseHTTPMiddleware # noqa: E402
PORT = int(os.environ.get("ECHO_MCP_PORT", "8765"))
TOKEN = os.environ["ECHO_MCP_TOKEN"]
PROFILE = os.environ.get("ECHO_MCP_TOOLS", "full").strip().lower()
KINDS = sorted(idx_mod.KIND_FOLDER)
mcp = FastMCP(
"echo-mcp",
instructions=(
"Persistent memory over the operator's ECHO Obsidian vault. Use echo_load at "
"the start of substantive sessions, echo_recall/echo_resolve to read, "
"echo_capture as the default write, and echo_log_session to end a session. "
"A duplicate-gate result is a decision point, not an error — merge_into or "
"(confirmed-distinct) force. Write about the operator in third person."),
host="0.0.0.0",
port=PORT,
stateless_http=True,
json_response=True,
)
WRITE_LOCK = threading.Lock() # all MCP-path writes serialize through this process
def _guard(write: bool, fn, *args, **kw) -> dict:
"""Run an op core with stdout redirected to stderr (belt-and-braces stream
purity) and EchoError mapped to an actionable error envelope, not a protocol
error. Write ops serialize on WRITE_LOCK."""
lock = WRITE_LOCK if write else contextlib.nullcontext()
try:
with lock, contextlib.redirect_stdout(sys.stderr):
return fn(*args, **kw)
except RuntimeError as exc: # EchoError (either module instance) subclasses it
code = getattr(exc, "code", 1)
err = {"ok": False, "error": str(exc), "code": code}
if code == 78:
err["hint"] = ("server deployment is missing vault credentials — operator: "
"check the PORT template env (SECRET: refs)")
elif code == 44:
err["code_name"] = "not-found"
elif "unreachable" in str(exc):
err["hint"] = ("vault unreachable (Obsidian/REST plugin likely down) — "
"proceed without memory; writes queue durably")
return err
_SAFE_PATH = re.compile(r"^[^/][^\0]*$")
def _check_path(path: str) -> str | None:
if not path or path.startswith("/") or ".." in path.split("/") or not _SAFE_PATH.match(path):
return "invalid path: must be vault-relative, no leading '/' and no '..'"
return None
def _frontmatter(text: str) -> dict:
out: dict = {}
if text.startswith("---"):
end = text.find("\n---", 3)
for ln in text[3:end if end != -1 else len(text)].splitlines():
m = re.match(r"^([A-Za-z_][\w-]*):\s*(.*)$", ln)
if m:
out[m.group(1)] = m.group(2).strip().strip('"').strip("'")
return out
# ------------------------------------------------------------------ tools -------
CORE_TOOLS = {"echo_load", "echo_recall", "echo_capture", "echo_triage_inbox",
"echo_log_session", "echo_health"}
def tool(fn):
"""Register `fn` as an MCP tool unless the core profile excludes it."""
if PROFILE == "core" and fn.__name__ not in CORE_TOOLS:
return fn
return mcp.tool()(fn)
@tool
def echo_load(brief: bool = True) -> dict:
"""Cold-start memory orientation — call at the start of a substantive session.
brief=true (default) returns a token-budgeted digest plus structured sections;
brief=false returns the six raw sections. Also flushes writes queued offline."""
return _guard(True, echo.load_op, brief)
@tool
def echo_recall(query: str, limit: int = 6, budget_chars: int = 4000,
include_linked: bool = True) -> dict:
"""Search memory: ranked matches for a topic/person/project PLUS their linked
neighbourhood. Excerpts are packed into budget_chars by score call
echo_get_note for any hit's full content. Freshness/status-aware ranking."""
limit = max(1, min(int(limit), 20))
budget = max(500, min(int(budget_chars), 20000))
env = _guard(False, recall_mod.recall_op, query, limit)
if not env.get("ok"):
return env
if not include_linked:
env["linked"] = []
used = 0
for hit in (env.get("primary") or []) + (env.get("linked") or []):
ex = hit.get("excerpt") or ""
room = max(0, budget - used)
if len(ex) > room:
hit["excerpt"] = ex[:room] + ("… (truncated — echo_get_note for full)" if room else "")
hit["truncated"] = True
used += len(hit.get("excerpt") or "")
return env
@tool
def echo_resolve(mention: str) -> dict:
"""Resolve a name/mention to its canonical vault note (alias-aware), or get
did-you-mean candidates. Call before creating any note by hand; echo_capture
does this automatically."""
return _guard(False, ops_mod.resolve_op, mention)
@tool
def echo_get_note(path: str, section: str = "", max_chars: int = 8000) -> dict:
"""Read one vault note (vault-relative path). section='Status' returns only that
## section. Content over max_chars is truncated with a marker."""
bad = _check_path(path)
if bad:
return {"ok": False, "error": bad}
def _read() -> dict:
status, body = echo.request("GET", echo.vault_url(path))
if status == 404:
return {"ok": False, "code_name": "not-found", "path": path,
"error": f"{path}: not found"}
echo.check(status, body, f"get {path}")
text = body.decode(errors="replace")
fm = _frontmatter(text)
if section:
text = echo.extract_heading(text, section)
if not text:
return {"ok": False, "path": path,
"error": f"no '## {section}' section in {path}"}
truncated = len(text) > max_chars
return {"ok": True, "path": path, "frontmatter": fm,
"content": text[:max_chars] + ("\n… (truncated)" if truncated else ""),
"truncated": truncated}
return _guard(False, _read)
@tool
def echo_get_scope() -> dict:
"""The operator's active scope + freshness. If sessions_since >= 3, treat the
recorded scope as suspect and confirm with the operator before working."""
return _guard(False, echo.scope_show_op)
@tool
def echo_set_scope(scope: str) -> dict:
"""Switch the active scope atomically (archives the prior scope to Scope History
and stamps freshness). Use when the session's work diverges from the recorded
scope."""
return _guard(True, echo.scope_set_op, scope)
@tool
def echo_health(deep: bool = False) -> dict:
"""ECHO readiness: config, vault reachability, auth, bootstrap/schema, and the
offline-queue depth. deep=true also runs the full vault-invariant linter."""
env = _guard(False, doctor_mod.run_op)
try:
env["outbox_depth"] = len(queue_mod.pending())
env["needs_attention"] = len(queue_mod.needs_attention())
except Exception: # noqa: BLE001
pass
if deep and env.get("ok"):
r = subprocess.run([sys.executable, str(SCRIPTS / "vault_lint.py")],
capture_output=True, text=True,
env=dict(os.environ), timeout=120)
env["lint_exit"] = r.returncode
env["lint_report"] = (r.stdout or r.stderr)[-6000:]
return env
@tool
def echo_capture(title: str, kind: str = "", body: str = "", tags: list[str] = [],
aliases: list[str] = [], sources: list[str] = [], status: str = "",
date: str = "", domain: str = "business", merge_into: str = "",
force: bool = False, dry_run: bool = False) -> dict:
"""The default memory write: routes by kind (person, company, concept, reference,
meeting, project, area, semantic, episodic, working, skill, decision), stamps
complete frontmatter, indexes, auto-links, and writes the Agent-Log line one
call. Omit kind for an inbox capture. If the result is action='duplicate-gate',
do NOT retry blindly: call again with merge_into=<candidate slug> to update the
existing entity, or force=true only after confirming they are genuinely distinct.
Offline, the whole capture queues durably (queued=true)."""
if kind and kind not in KINDS:
return {"ok": False, "error": f"unknown kind '{kind}' — one of: {', '.join(KINDS)}"}
return _guard(True, ops_mod.capture_op, kind or None, title,
body_text=body or "", status_v=status, aliases=aliases,
sources=sources, tags=tags, date=date or None, domain=domain,
inbox=not kind, dry_run=dry_run, force=force,
merge_into=merge_into or None)
@tool
def echo_link(a: str, b: str) -> dict:
"""Add reciprocal '## Related' links between two notes. Accepts vault paths or
resolvable entity names (aliases work)."""
def _to_path(x: str) -> str | None:
if "/" in x:
return x if x.endswith(".md") else x + ".md"
nmap = idx_mod.name_map(idx_mod.load())
return nmap.get(idx_mod.slugify(x))
def _do() -> dict:
pa, pb = _to_path(a), _to_path(b)
if not pa or not pb:
missing = a if not pa else b
return {"ok": False, "error": f"'{missing}' resolves to no note — "
"pass a vault path or a known entity name"}
return ops_mod.link_op(pa, pb)
return _guard(True, _do)
@tool
def echo_append_note(path: str, line: str) -> dict:
"""Append one line to a note, idempotently (skipped if the exact line already
exists). For inbox lines, Agent-Log entries, Observations bullets."""
bad = _check_path(path)
if bad:
return {"ok": False, "error": bad}
def _do() -> dict:
rc = echo.cmd_append(path, line)
return {"ok": rc == 0, "action": "append", "path": path}
return _guard(True, _do)
@tool
def echo_patch_note(path: str, operation: str, target_type: str, target: str,
content: str) -> dict:
"""Targeted edit: append/prepend/replace under a heading, frontmatter field, or
block. HEADING TARGETS must be the full '::'-delimited path from the top-level
heading (e.g. 'Operator Preferences::Fact / Pattern') on an invalid target the
error includes the note's actual headings. replace overwrites the section."""
bad = _check_path(path)
if bad:
return {"ok": False, "error": bad}
def _do() -> dict:
rc = echo.cmd_patch(path, operation, target_type, target,
echo.temp_file(content.encode("utf-8")))
return {"ok": rc == 0, "action": f"patch:{operation}", "path": path,
"target": target}
env = _guard(True, _do)
if not env.get("ok") and "HTTP 400" in str(env.get("error", "")) and target_type == "heading":
st, body = echo.request("GET", echo.vault_url(path),
headers={"Accept": "application/vnd.olrapi.document-map+json"})
if st == 200:
try:
headings = [h.get("heading") or h for h in
json.loads(body).get("headings", [])][:40]
env["available_headings"] = headings
env["hint"] = "use one of available_headings verbatim as the Target"
except Exception: # noqa: BLE001
pass
return env
@tool
def echo_triage_inbox(proposals: list[dict] = [], apply: bool = False) -> dict:
"""Inbox triage. No proposals => structured listing of captures (line, date, text,
age_days). With proposals (reflect schema + optional 'line'): apply=false previews
the routing; apply=true routes via capture and writes the processing-log audit.
List first, propose, preview, then apply only after the operator confirms."""
if not proposals:
return _guard(False, triage_mod.list_op)
return _guard(True, triage_mod.route_op, proposals, apply)
@tool
def echo_reflect(proposals: list[dict], apply: bool = False) -> dict:
"""Session-reflection proposals: validate -> classify against the entity index ->
preview (apply=false) -> apply (routes each via capture). Never apply without the
operator's go-ahead; never invent memories to have something to save."""
return _guard(True, reflect_mod.apply_op, proposals, apply)
@tool
def echo_log_session(slug: str, log_body: str, agent_log_line: str = "",
scope: str = "", reflect: list[dict] = [],
apply: bool = False, hhmm: str = "") -> dict:
"""End a substantive session in ONE call: session log -> Agent-Log line -> reflect
proposals -> optional scope switch -> heartbeat LAST (the commit marker).
apply=false previews the plan. Pass hhmm (local time, e.g. '1430') so the log
filename sorts truthfully."""
bundle = {"slug": slug, "log_body": log_body}
if agent_log_line:
bundle["agent_log_line"] = agent_log_line
if scope:
bundle["scope"] = scope
if reflect:
bundle["reflect"] = reflect
def _do() -> dict:
prev = os.environ.get("ECHO_NOW")
try:
if hhmm:
os.environ["ECHO_NOW"] = hhmm
return session_mod.session_end_op(bundle, apply=apply)
finally:
if hhmm:
if prev is None:
os.environ.pop("ECHO_NOW", None)
else:
os.environ["ECHO_NOW"] = prev
return _guard(True, _do)
# ------------------------------------------------------------- health + auth ----
@mcp.custom_route("/health", methods=["GET"])
async def health(_request: Request) -> JSONResponse:
"""Unauthenticated liveness for Docker HEALTHCHECK + Kuma. No vault data."""
st, _ = echo.request("GET", echo.vault_url("_agent/echo-vault.md"))
try:
outbox = len(queue_mod.pending())
except Exception: # noqa: BLE001
outbox = -1
return JSONResponse({"ok": True, "vault_reachable": st != 0,
"outbox_depth": outbox, "profile": PROFILE})
class BearerAuth(BaseHTTPMiddleware):
async def dispatch(self, request, call_next):
if request.url.path == "/health":
return await call_next(request)
auth = request.headers.get("authorization", "")
if not hmac.compare_digest(auth, f"Bearer {TOKEN}"):
return Response(status_code=401)
return await call_next(request)
def main() -> None:
import uvicorn
app = mcp.streamable_http_app()
app.add_middleware(BearerAuth)
print(f"echo-mcp: serving on :{PORT} (profile={PROFILE}, vault={echo.BASE})",
file=sys.stderr)
uvicorn.run(app, host="0.0.0.0", port=PORT, log_level="info")
if __name__ == "__main__":
main()
+4
View File
@@ -0,0 +1,4 @@
# echo-mcp server deps. The PLUGIN stays pure-stdlib; the dependency budget for the
# container is deliberately tiny: the official MCP SDK (brings starlette/uvicorn/
# pydantic) and nothing else.
mcp>=1.9,<2