echo/docs/IMPROVEMENT-PLANS.md

# echo-memory — Improvement Plans (post-1.5.1 review)

> Status: **planned, not started.** Source: full-code review 2026-07-28.
> Nine improvements, each scoped for an independent build session. The tenth item
> from that review — the **MCP server** — has its own detailed spec:
> `docs/MCP-SERVER-SPEC.md`.
>
> Sequencing is at the bottom. Nothing here conflicts with `TODO-1.6.md` or
> `ROADMAP-2.0.md`; release targets assume 1.6 (skills migration) ships first.

---

## 1. Token-budgeted load (`load --brief`) + parallel orientation reads

**Problem.** The SessionStart hook injects the *full text* of six files into every
session. Live measurement: 15.9KB of hook context on a single cold start, and two of
the six files grow without bound (operator-preferences `## Observations`,
current-context `## Scope History`). Every session pays this tax before any work
happens. Separately, `cmd_load` (echo.py) issues its six GETs serially even though
`read_many()` exists.

**Design.**
- New `echo.py load --brief` (and make the SessionStart hook use it):
  - marker → one line (`schema_version`, bootstrapped yes/no).
  - operator-preferences → `## Fact / Pattern` in full; `## Observations` capped at
    the last 10 lines with a `(+N older — /echo-load for full)` note.
  - current-context → `## Scope` + `scope_updated` + sessions-since (i.e. the
    `scope show` output); omit `## Scope History`.
  - heartbeat → the pointer line + only the `## Summary`/`## Outcomes` sections of
    the pointed-at session log (fetch it, extract those headings, cap ~30 lines).
  - today's daily note → `## Agent Log` lines only (or "absent").
  - inbox → count + age of oldest item, not the contents (that's all the reconcile
    needs; `/echo-triage --list` has the details).
- Budget guard: after assembly, if the brief output still exceeds
  `ECHO_LOAD_BUDGET` chars (default ~8000), truncate lowest-priority sections first
  (observations → agent log → session summary) with explicit `(truncated)` markers.
- Full `load` unchanged; `/echo-load` keeps using it.
- Fetch all reads via `read_many()` (the 6 targets + the heartbeat-pointed session
  log + the sessions listing fallback) instead of the serial loop at
  `echo.py:640`. Cache-put still applies per path.

**Files.** `echo.py` (cmd_load), `echo_hook_session_start.py`, SKILL.md (document
brief-vs-full), `eval/test_features.py`.

**Tests.** Mock vault with oversized preferences/history files → assert brief output
under budget, sections present, truncation markers correct; hook emits brief.

**Size/target.** Small-medium. **1.6.x point release** — highest leverage per line.

---

## 2. Local-first recall index

**Problem.** `echo_recall.update_note()` does GET-whole-index → add → PUT-whole-index
against the vault, under the global advisory lock, on every capture and every corpus
`put`. The index carries full BM25 postings for the whole corpus, so this round-trip
is O(vault) network per write and is the emerging bottleneck + lock hot-spot as the
vault grows past a few hundred notes.

**Design.**
- The **live** recall index moves to the local state dir
  (`~/.echo-memory/recall-index.json`, honoring `ECHO_STATE_DIR`) keyed by endpoint
  (hash of `BASE`) so multiple vaults don't collide.
- `update_note()` becomes a local read-modify-write (file lock via `os.O_EXCL`
  sidecar or atomic `os.replace`), **no vault round-trip, no advisory lock**.
- The vault copy (`_agent/index/recall-index.json`) becomes a **snapshot**, written
  by `sweep.py` and at session end (piggyback on the heartbeat write / future
  `session-end` verb). It exists so a *fresh machine* can seed its local index
  without a full rebuild.
- Freshness rule on recall: local index used when present; if the vault snapshot's
  embedded `built` stamp is newer than the local one (another client swept), pull it.
  Staleness across clients is tolerable — recall degrades gracefully and improvement
  #4 (incremental sweep) trues it up cheaply.
- Schema bump: recall-index schema 3 adds `built` (ISO timestamp) + `endpoint_hash`.
  Old schema-2 vault copies are still readable as seeds.
- The **entity index stays in the vault** (it is small, and it is the shared routing
  authority for concurrent clients) — only the BM25 index moves.

**Files.** `echo_recall.py` (load/save/update_note/rebuild), `echo_queue.py`
(state-dir helpers shared), `sweep.py`, `echo.py` (cmd_put upkeep path), routing.json
(the vault snapshot path is unchanged), README performance section.

**Tests.** Capture with vault mock → assert zero recall-index PUTs; sweep → snapshot
written; fresh state dir + existing snapshot → seeded without rebuild; two-vault
(endpoint) separation.

**Size/target.** Medium. **Pair with #4 in one index-focused minor** (they touch the
same maintenance loop). Fine to land before or after 2.0.

---

## 3. Offline durability for the high-level ops (capture path)

**Problem.** The low-level verbs queue on outage via `safe_request`, but
`_append_to_existing()` and `ensure_daily_log()` (echo_ops.py) call `echo.request`
directly — so during an outage a raw `append` survives, while a `capture` **update**
and its Agent Log line silently vanish. Inverted from the documented contract
("capture is the default write").

**Design.** Queue *semantic* records, not byte-level requests, for the high-level
path:
- New outbox record type `{"op": "capture", "args": {...}}` alongside the existing
  raw records. On an unreachable vault, `capture` short-circuits: enqueue the full
  argument set (title/kind/body/tags/aliases/sources/date/domain/merge_into) and
  report `queued (offline): capture …`.
- `flush()` replays `op:capture` records by calling `echo_ops.capture(...)` against
  the *current* index — so routing, the duplicate gate, aliasing, and linking all
  re-run with fresh state instead of replaying stale byte writes. A gate stop on
  replay (exit 76) keeps the record queued with a `needs_attention` flag and is
  surfaced by `load`/`flush` output ("1 queued capture needs --merge-into/--force").
- Rationale for semantic-over-byte: a byte replay of `_append_to_existing`'s PATCH
  could target a heading that moved, and the create-vs-update decision made offline
  may be wrong by replay time. Re-running the op is the idempotent, correct unit
  (the dated-bullet idempotency key already prevents double-append).
- `ensure_daily_log` stays best-effort but routes its POST/PATCH through
  `safe_request` so a standalone agent-log line survives an outage too.
- reflect/triage `--apply` inherit this for free (they call capture).

**Files.** `echo_ops.py`, `echo_queue.py` (record schema v2 + replay dispatch),
`eval/test_offline_queue.py`.

**Tests.** Fault-injected mock: capture-update offline → queued; flush replays via
capture; gate-on-replay keeps record + flags; agent-log line queued; no duplicate
bullets on double flush.

**Size/target.** Small-medium. **1.6.x point release** — it's a correctness gap.

---

## 4. Incremental sweep via content hashes

**Problem.** Human edits in Obsidian and writes by other clients never touch the
entity/recall indexes; drift is only corrected by a full sweep someone must remember
to run. Full sweep re-reads the whole vault (fast now, but O(vault)).

**Design.**
- Entity index entries and the recall-index snapshot gain a per-note
  `h` — SHA-1 of the note body (entity index schema 2; tolerated-absent for old
  entries).
- `sweep.py --fast`: walk the listing (cheap), `read_many` only paths that are
  (a) new, (b) missing a stored hash, or (c) whose hash mismatches after fetch — to
  avoid fetching everything just to hash it, fast mode fetches only paths whose
  **listing is new/gone** plus a rotating shard (e.g. `hash(path) % 7 == weekday`)
  of existing notes, so the whole vault is verified over a week of fast sweeps while
  each run stays tiny. `--fast --all-shards` forces full verification.
- Deletions: listing walk catches removed files → drop index + recall entries.
- Auto-run: `load` (brief or full) runs `sweep --fast` opportunistically when the
  last fast-sweep stamp (state dir) is older than `ECHO_FAST_SWEEP_DAYS` (default 7),
  in the background-tolerant sense: bounded by the same read_many concurrency, and
  skipped entirely when offline.
- `/echo-health` reports last fast/full sweep ages.

**Files.** `sweep.py`, `echo_index.py` (schema 2 + hash field), `echo_recall.py`
(snapshot hash reuse), `echo.py` (load hook-in), `vault_lint.py` (report), migrate
note (index schema is machine-owned; no vault migration needed).

**Tests.** Mock: edit a note out-of-band → fast sweep in that shard re-indexes it;
delete → entry dropped; stamp gating; full sweep unchanged.

**Size/target.** Medium. **Ship with #2** (same schema-bump train).

---

## 5. Memory lifecycle — enforce forgetting

**Problem.** The vault only accretes. Working memory is "time-boxed" by convention
only; Observations trimming is manual prose; stale-active detection only reports.
Recall precision degrades as dead weight accumulates.

**Design.**
- `capture --kind working` stamps `expires: <today + ECHO_WORKING_TTL_DAYS>`
  (default 14; `--ttl <days>` overrides). Template updated to carry the field.
- `vault_lint.py` new checks: `expired-working` (past `expires`),
  `stale-observations` (Observations > 30 bullets) — advisory, like the rest.
- `sweep.py --decay` (dry-run by default, `--apply` to write; **always
  preview-first**, honoring the operating contract):
  - expired working notes → `status: archived` + move body under a dated
    `## Archived` marker (never delete);
  - `projects/active/` untouched > 30 days → *propose* `on-hold` (folder move +
    status, the usual paired change) — apply only the ones confirmed;
  - Observations > 30 → trim oldest into a dated `_agent/memory/episodic/`
    overflow note (`observations-archive-YYYY.md` append), keeping the last 30
    in place. Nothing is lost, the hot file stays small (compounds with #1).
- `/echo-health` output nudges: "run `sweep --decay` to review N aging items".

**Files.** `echo_ops.py` (`--ttl`/expires stamp), `echo_index.py`
(KIND_REQUIRED_FM unchanged; working gains optional expires), `vault_lint.py`,
`sweep.py`, scaffold working-memory template, SKILL.md (decay section),
routing-map note.

**Tests.** Mock: expired note proposed+archived on apply; observations trim keeps
last 30 and appends overflow; stale-active proposal list; dry-run writes nothing.

**Size/target.** Medium. Feature minor (1.7.x).

---

## 6. Contradiction / supersession handling

**Problem.** A new fact that contradicts a stored one lands beside it; both surface
in recall with no signal about which is current. Biggest remaining trust gap in the
write path.

**Design.**
- **Write side:** `capture --supersedes "<slug-or-quoted-line>"`:
  - if it names an entity slug → the old entity gets `status: superseded` +
    a `superseded_by:` frontmatter path; new note carries `supersedes:` back-link
    in `## Related`. Recall's STATUS_FACTOR gains `superseded: 0.5`.
  - if it quotes a Fact/Pattern or Observations line in operator-preferences →
    the line is struck (`~~old~~ (superseded YYYY-MM-DD)`) and the new line
    appended, one PATCH each, in the same invocation.
- **Detect side (deterministic assist, model decides):** `echo_reflect.classify`
  gains a `conflict scan` — for each proposal, run its distinctive tokens
  (`_sig_tokens`) against the Fact/Pattern + recent Observations lines; overlapping
  lines are attached as `_conflicts: [...]` and the preview row renders
  `update-or-conflict?` with the matched line(s). The model (which has the
  conversation) decides supersede-vs-append; the script only guarantees the
  question gets asked before the write.
- Lint: `superseded_by` pointing at a missing note → violation.

**Files.** `echo_ops.py`, `echo_reflect.py`, `echo_recall.py` (STATUS_FACTOR),
`echo.py` (flag plumbing), `vault_lint.py`, SKILL.md + operating-contract
(supersession is additive: strike, never delete).

**Tests.** Mock: supersede entity → status/backlinks both sides; supersede line →
strike + append idempotent; reflect preview flags a planted conflict; recall ranks
superseded below active twin.

**Size/target.** Medium. Feature minor (1.7.x/1.8) — after #5 (shares the
status-vocabulary touchpoints).

---

## 7. `session-end` bundle verb

**Problem.** Ending a session correctly is 4+ invocations (session-log PUT, Agent
Log append, heartbeat PUT, optional scope set, optional reflect apply). Cost per
session, and a partial failure leaves broken orientation state (log written,
heartbeat stale).

**Design.**
- `echo.py session-end <bundle.json> [--apply]`, bundle:
  ```json
  {
    "slug": "echo-mcp-spec",
    "log_body": "<full session-log markdown, frontmatter included>",
    "agent_log_line": "- 2026-07-28: wrote MCP spec [[...]]",
    "scope": "optional new scope text",
    "reflect": [ { ...PROPOSAL_SCHEMA... } ]
  }
  ```
- Dry-run previews the whole plan (paths + reflect preview). `--apply`, in order,
  under **one** lock acquisition: PUT `_agent/sessions/<date>-<HHMM>-<slug>.md`
  (HHMM from `ECHO_NOW` env or wall clock, validated against the canonical regex) →
  agent-log line (ensure_daily_log) → reflect proposals via capture → scope set if
  given → heartbeat PUT **last** (it's the commit marker: a crash before it leaves
  the previous pointer intact, never a dangling one).
- Reports per-step ok/queued/gated; offline → every step queues (rides on #3).
- Stop hook reason text updated to name the one command; `/echo-reflect` docs point
  session logging at it. **Prerequisite for the MCP `echo_log_session` tool** —
  build this verb first so CLI and MCP share one implementation.

**Files.** new `echo_session.py`, `echo.py` (subcommand), `echo_hook_stop.py`
(REASON text), SKILL.md (Session Logging section), commands/echo-reflect.md.

**Tests.** Mock: full bundle lands all five artifacts; heartbeat-last ordering
(inject failure mid-way → heartbeat untouched); dry-run writes nothing; offline
queues all steps.

**Size/target.** Small-medium. **1.6.x point release** (and before the MCP build).

---

## 8. Recall robustness — stemming + alias query expansion

**Problem.** BM25 is purely lexical: "deploy"/"deployment"/"deployed" are distinct
terms; a paraphrased query misses. The 8-query gold set can't see this class of miss.

**Design.**
- **Stemming:** a light suffix-stripper (Porter-lite: s/es/ed/ing/ly/tion/ment/ness
  families, ~40 lines, pure stdlib) applied in `tokenize()` at **both** index and
  query time. Index schema bump (stemmed postings) → auto-rebuild on next
  recall/sweep, exactly like the 1→2 transition.
- **Alias query expansion:** at recall time, run the query against
  `fuzzy_candidates()` (top 2, score ≥ 0.5); fold the matched entities' title +
  alias tokens into the BM25 query as **down-weighted expansion terms** (×0.5 idf
  contribution) so "the ECHO plugin" also scores `echo-memory`-alias vocabulary.
  Exact-match seeding stays as-is; expansion never *creates* hits ranked above
  genuine lexical matches without corpus support.
- **Eval:** grow the gold set with ≥6 paraphrase queries (morphological variants +
  alias phrasings); publish before/after recall@5/MRR in the README table.

**Files.** `echo_recall.py` (tokenize, INDEX_SCHEMA, score, recall), new
`echo_stem.py` (pure function, unit-testable), `eval/run_eval.py` + gold set.

**Tests.** Stemmer unit table (word → stem, incl. non-cases); paraphrase gold
queries pass; schema-2 index discarded and rebuilt cleanly.

**Size/target.** Small-medium. Ride the **same index-schema train as #2/#4**.

---

## 9. Resolve-miss telemetry → alias suggestions

**Problem.** Every resolve that misses but ends in a confirmed candidate
(`--merge-into`, or the operator picking a candidate) is a free mention→entity
training pair, currently discarded.

**Design.**
- `echo_index.resolve()` miss + `fuzzy_candidates` non-empty → append one NDJSON
  record to the local state dir (`~/.echo-memory/resolve-misses.ndjson`):
  `{mention, candidates:[slugs], ts?}` (no wall clock needed — session date via
  ECHO_TODAY). `capture --merge-into <slug>` appends the *confirmation* record
  `{mention, resolved: slug}` (this already learns the alias immediately — that
  path stays; the log covers the cases that never reach --merge-into).
- `sweep.py` (and `/echo-health`) read the log: any `{mention → slug}` pair
  confirmed ≥2 times, or any mention that repeatedly fuzzy-matched a single
  candidate ≥3 times without confirmation, is **proposed** as an alias
  (`sweep --apply` writes it into the note's `aliases:` frontmatter + index, the
  existing fold-back path). Log entries consumed on apply; file capped (rotate at
  ~500 lines).
- Privacy note: mentions can contain operator phrasing — the log lives in the
  state dir (never the vault), same trust level as the offline queue.

**Files.** `echo_index.py` (log hooks), `echo_ops.py` (confirmation hook),
`sweep.py` (suggest/apply), `vault_lint.py` or sweep report (surface count),
SKILL.md one-liner.

**Tests.** Miss → record written; confirm ×2 → sweep proposes; apply folds alias +
consumes log; cap/rotation.

**Size/target.** Small. Feature minor, pairs naturally with #8.

---

## Sequencing (unified with TODO-1.6 and ROADMAP-2.0)

TODO-1.6 folds in as the opening train — the skills migration restructures the
files everything below touches, so it goes first. ROADMAP-2.0 stays a **pure
packaging major** (its own stated thesis) and is *not* folded into any feature
train; it slots between the quick wins and the MCP server so the MCP work lands
on the final packaging baseline (single command format, Gitea releases, one
manifest-verification pass).

| Release | Items | Rationale |
|---|---|---|
| **1.6.0** | TODO-1.6: skills-format migration (+ per-skill `allowed-tools`, `disable-model-invocation` on write-heavy entries, `$ECHO` block dedupe) · lint blind-spot fix (retired-dir/leaf-README) · plugin.json `homepage` | Restructures the command/SKILL files later work edits; allowed-tools removes prompt friction for every verb added below. Vault link repointing (TODO-1.6 §3) is vault data, not plugin code — any session. |
| **1.6.0 — SHIPPED 2026-07-28** | Skills migration + lint fix + homepage | Verified on desktop + CoWork. |
| **2.0.0 — SHIPPED 2026-07-28** | ROADMAP-2.0: commands/ deleted · release-based artifacts · gitignore · Codex note | Packaging-only major; skills-only install re-verified. |
| **2.1.0 quick wins — SHIPPED 2026-07-28** | #1 brief load · #3 offline capture · #7 session-end | All three landed with +19 e2e checks; see CHANGELOG 2.1.0. The MCP spec's Phase 0 (return-not-print) did NOT ride along — it's the first phase of the MCP build session. |
| **2.2** | MCP server (containerized, `echomcp.alwisp.com`) — `docs/MCP-SERVER-SPEC.md` | Needs Phase 0 (return-not-print refactor, first step of the build session). #7 session-end shipped, so `echo_log_session` wraps a real verb. Deployed via CI + PORT, not the plugin manifest. |
| **2.3 index train** | #2 local-first recall (CLI-path half) · #4 incremental sweep · #8 stemming+expansion | One schema-bump train; all touch the same index/sweep loop. The container already keeps its index warm in memory (spec §7), so #2 here covers the CLI fallback path; #4's fast sweep also becomes the container's background job. |
| **2.4 memory train** | #5 lifecycle/decay · #6 supersession · #9 resolve-miss learning | Capability tier; #6 after #5 (status vocabulary). |