2.3.0 — the index train: local-first recall index, incremental sweep, stemming + alias expansion

One schema-bump release (recall-index schema 3, entity-index schema 2; old
recall indexes rebuild automatically, no vault migration):

- Local-first recall index: the live BM25 index lives in the machine state dir
  (keyed by endpoint hash); capture's upkeep is a zero-network atomic file
  write, no advisory lock — replacing the O(vault) GET/PUT-whole-index round
  trip per write. The vault copy is a snapshot (sweep + session-end) used to
  seed fresh machines / catch up after another client swept
  (ECHO_RECALL_SYNC_HOURS, default 24). The read path never PUTs the vault.
- Incremental sweep: entity + recall meta carry 16-char content hashes;
  `sweep --fast` fetches only new/gone/hash-missing notes plus a rotating
  weekday shard (--all-shards forces everything); deletions drop from both
  indexes; index-only so no --apply gate. `load` auto-runs it past
  ECHO_FAST_SWEEP_DAYS (7); doctor reports index freshness. Obsidian-side
  edits now reach the indexes without manual maintenance.
- Stemming + alias expansion: echo_stem.py (conservative Porter-lite, unit-
  tested families + over-stemming guards) applied at index AND query time;
  a query fuzzy-matching an entity folds its title/alias vocabulary into the
  BM25 query at half weight — expansion can only boost docs containing the
  terms. capture hashes the note's FINAL content (auto-link reordered first).

Eval gold set 8 -> 14 queries (6 paraphrases): recall@5/MRR 1.00/1.00 vs
keyword baseline 0.75/0.79. +3 unit tests, +10 e2e; test harnesses now isolate
ECHO_STATE_DIR. All seven suites green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

echo-memory.plugin.src/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "echo-memory",
-  "version": "2.2.0",
+  "version": "2.3.0",
   "homepage": "https://git.alwisp.com/jason/echo",
   "repository": "https://git.alwisp.com/jason/echo",
   "description": "Persistent memory via the ECHO Obsidian vault over the Obsidian Local REST API. Cross-platform Python client: one-call capture/resolve/recall/link/triage over an entity index, hybrid BM25 + graph recall spanning entities + sessions/journal (recency/status-aware), a pre-write duplicate gate, complete-frontmatter capture, session hooks that self-fire load/reflect, offline write-ahead queue, lock-guarded concurrency, linter-enforced routing, and /echo-* commands.",

echo-memory.plugin.src/skills/echo-memory/SKILL.md

@@ -358,6 +358,8 @@ Run it after `migrate.py` (which handles structural/schema changes) — or any t
 
 The pass is cheap and pays for itself by catching drift before it requires a reorg. Write the findings as a digest; act on them only with the operator's go-ahead.
 
+**Incremental maintenance (2.3).** `sweep.py --fast` is the cheap, index-only mode: it fetches just what's new, gone, or changed (content hashes) plus a rotating daily verification shard, so edits made directly in Obsidian or by other clients reach the entity + recall indexes without a full pass. It runs **automatically at load** when this machine's last sweep is older than `ECHO_FAST_SWEEP_DAYS` (default 7) — you rarely need to invoke it. The recall index itself is **local-first**: capture maintains a machine-local copy with zero vault round-trips; the vault's `recall-index.json` is a snapshot (sweep/session-end) used to seed fresh machines.
+
 **Performance.** Full-vault scripts (`sweep.py`, `vault_lint.py`, recall rebuild) are connection-pooled (keep-alive) and read the whole vault concurrently via `echo.read_many`, so they finish in about a second on a few-hundred-note vault instead of timing out on hundreds of serial TLS handshakes. Tune with `ECHO_WORKERS` (default 8, bulk-read concurrency) and `ECHO_TIMEOUT` (default 30s, per-request). For a vault large enough that even the concurrent pass nears the tool timeout, run the script with the harness's background-execution option rather than blocking the turn.
 
 ## Daily Note — Agent Log

echo-memory.plugin.src/skills/echo-memory/scripts/echo.py

@@ -790,6 +790,24 @@ def _load_gather(targets):
     return results, texts, listing_files, offline, marker_missing, heartbeat_absent
 
 
+def _auto_fast_sweep(offline: bool) -> str | None:
+    """Opportunistic index maintenance at load (2.3): when this machine's last fast
+    sweep is older than ECHO_FAST_SWEEP_DAYS (default 7), run `sweep --fast` so the
+    local recall + entity indexes true up against edits made by Obsidian or other
+    clients. Best-effort; skipped offline; never raises."""
+    if offline:
+        return None
+    try:
+        import sweep as sweep_mod
+        days = float(os.environ.get("ECHO_FAST_SWEEP_DAYS", "7"))
+        age = sweep_mod.fast_sweep_age_days()
+        if age is not None and age < days:
+            return None
+        return sweep_mod.fast()
+    except Exception:  # noqa: BLE001 — maintenance must never block a load
+        return None
+
+
 def _fetch_pointed_session(texts: dict) -> None:
     """Follow the heartbeat pointer and stash the pointed session log for the digest."""
     hb = texts.get("heartbeat")
@@ -818,9 +836,10 @@ def load_op(brief: bool = True) -> dict:
     targets = LOAD_TARGETS + [("today", f"journal/daily/{today()}.md"),
                               ("inbox", "inbox/captures/inbox.md")]
     _, texts, listing_files, offline, marker_missing, _ = _load_gather(targets)
+    swept = _auto_fast_sweep(offline)
     data = {"ok": True, "action": "load", "offline": offline,
             "marker_missing": marker_missing, "synced": synced,
-            "needs_attention": flagged,
+            "needs_attention": flagged, "fast_sweep": swept,
             "sections": {k: v for k, v in texts.items() if not k.startswith("_")},
             "recent_sessions": sorted((f for f in listing_files if f.endswith(".md")),
                                       reverse=True)[:5]}
@@ -863,6 +882,10 @@ def cmd_load(brief: bool = False) -> int:
     results, texts, listing_files, offline, marker_missing, heartbeat_absent = \
         _load_gather(targets)
 
+    swept = _auto_fast_sweep(offline)
+    if swept:
+        print(f"({swept})\n")
+
     if brief:
         _fetch_pointed_session(texts)
         print(_render_brief(texts, listing_files, offline))

echo-memory.plugin.src/skills/echo-memory/scripts/echo_doctor.py

@@ -74,6 +74,18 @@ def run_op() -> dict:
             else:
                 line(status < 400, "marker fetch", f"HTTP {status}")
 
+    # Index freshness (2.3) — informational, never red: how stale this machine's
+    # local indexes are (the fast sweep trues them up at load past 7 days).
+    if fatal is None:
+        try:
+            import sweep as sweep_mod
+            age = sweep_mod.fast_sweep_age_days()
+            line(True, "index freshness",
+                 f"last local sweep {age:.1f}d ago" if age is not None
+                 else "no local sweep yet — runs automatically at the next load")
+        except Exception:  # noqa: BLE001
+            pass
+
     reds = sum(1 for c in checks if not c["ok"])
     return {"ok": reds == 0 and fatal is None, "action": "doctor",
             "endpoint": cfg["endpoint"], "fatal": fatal, "reds": reds, "checks": checks}

echo-memory.plugin.src/skills/echo-memory/scripts/echo_index.py

@@ -168,7 +168,10 @@ def kind_for_path(path: str) -> str | None:
 
 
 def empty_index() -> dict:
-    return {"schema": 1, "updated": echo.today(), "entities": {}}
+    # Schema 2 (2.3): entries may carry `h`, a short content hash of the note body —
+    # the incremental sweep's change detector. Tolerated-absent (schema-1 entries
+    # simply get fetched and hashed on their first fast sweep).
+    return {"schema": 2, "updated": echo.today(), "entities": {}}
 
 
 def load() -> dict:
@@ -287,13 +290,20 @@ def gate_candidates(index: dict, mention: str, kind: str | None = None,
     return out
 
 
+def content_hash(text: str) -> str:
+    import hashlib
+    return hashlib.sha1(str(text).encode("utf-8", "replace")).hexdigest()[:16]
+
+
 def upsert(index: dict, slug: str, path: str, kind: str, title: str | None = None,
-           aliases=None) -> dict:
+           aliases=None, h: str | None = None) -> dict:
     ents = index.setdefault("entities", {})
     e = ents.get(slug, {})
     e["path"] = path
     e["kind"] = kind
     e["title"] = title or e.get("title") or slug
+    if h:
+        e["h"] = h
     # Keep any prior + explicit aliases, and ALWAYS fold in the safe title variants so a
     # hyphen/space/case form of the name resolves even if no one passed --aliases. Drop any
     # alias that just equals the slug (resolve already checks the slug).

echo-memory.plugin.src/skills/echo-memory/scripts/echo_ops.py

@@ -333,15 +333,11 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status
             echo.cmd_put(path, echo.temp_file(note.encode()))
             action = "created"
 
-        # H3: the entity-index write is the race the review flagged — a bare load->save lets
-        # two concurrent captures clobber each other's entry. Route it through a lock-guarded,
-        # fresh-re-read transaction. Returns the fresh index (used by auto-link below).
-        import echo_concurrency
-        index = echo_concurrency.atomic_index_update(
-            lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases))
-
-        # auto-link: any other known entity CONFIDENTLY named in the body (M2: is_confident_link
-        # rejects short/common tokens so a 3-char alias or a word like "API" can't link everywhere).
+        # auto-link FIRST (it mutates the note's ## Related section), so the content
+        # hash stored below reflects the note's final state. Uses the pre-update index
+        # — it only needs to find OTHER entities named in the body. (M2: is_confident_link
+        # rejects short/common tokens so a 3-char alias or a word like "API" can't link
+        # everywhere.)
         linked = 0
         for s2, e2 in index.get("entities", {}).items():
             if e2.get("path") in (None, path):
@@ -351,11 +347,24 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status
                 ca, cb = links.link_bidirectional(path, e2["path"])
                 linked += int(ca or cb)
 
-        # Keep the BM25 recall index current for this note (best-effort; lock-guarded inside
-        # update_note, so it can't clobber a concurrent writer either).
+        # The note's final content: hashed into the entity index (the incremental
+        # sweep's change detector, 2.3) and fed to the local recall index.
+        final_text = links.get_text(path) or ""
+        note_h = idx_mod.content_hash(final_text) if final_text else None
+
+        # H3: the entity-index write is the race the review flagged — a bare load->save lets
+        # two concurrent captures clobber each other's entry. Route it through a lock-guarded,
+        # fresh-re-read transaction.
+        import echo_concurrency
+        index = echo_concurrency.atomic_index_update(
+            lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases,
+                                       h=note_h))
+
+        # Keep the local BM25 recall index current for this note (best-effort; a
+        # zero-network atomic file write since 2.3).
         try:
             import echo_recall
-            echo_recall.update_note(path, links.get_text(path) or "")
+            echo_recall.update_note(path, final_text)
         except Exception as exc:  # never let recall-index upkeep fail a capture
             print(f"echo_ops: recall-index update skipped ({exc})", file=sys.stderr)
 

echo-memory.plugin.src/skills/echo-memory/scripts/echo_recall.py

@@ -52,9 +52,17 @@ sys.path.insert(0, str(Path(__file__).resolve().parent))
 import echo                      # noqa: E402
 import echo_index as idx_mod     # noqa: E402
 import echo_links as links       # noqa: E402
+import echo_stem                 # noqa: E402
 
 RECALL_INDEX_PATH = "_agent/index/recall-index.json"
-INDEX_SCHEMA = 2                 # bumped: schema 2 adds per-doc meta (weight/updated/status)
+# Schema 3 (2.3, the "index train"): postings are STEMMED (echo_stem), per-doc meta
+# gains a content hash `h` (incremental sweep), and the index carries a `built`
+# stamp + endpoint hash. The LIVE index is LOCAL-FIRST — it lives in the state dir
+# and is updated with zero vault round-trips; the vault copy at RECALL_INDEX_PATH
+# is a snapshot written by sweep/session-end, used to seed fresh machines. Older
+# schemas are discarded and rebuilt (sub-second since the 1.1 network work).
+INDEX_SCHEMA = 3
+SYNC_HOURS = float(os.environ.get("ECHO_RECALL_SYNC_HOURS", "24"))
 
 # Tuning knobs. Defaults are standard BM25 (k1/b) + a 0.6 hop decay over <=2 hops.
 K1 = 1.5
@@ -131,8 +139,11 @@ _SKIP_BASENAMES = {"README.md"}
 
 
 def tokenize(text: str) -> list[str]:
-    """Lowercase word tokens, stopworded. Frontmatter should be stripped by the caller."""
-    return [t for t in _WORD.findall(text.lower()) if t not in _STOP and len(t) > 1]
+    """Lowercase word tokens, stopworded, STEMMED (2.3) — applied identically at
+    index and query time, so 'deployed'/'deployment' meet at 'deploy'.
+    Frontmatter should be stripped by the caller."""
+    return [echo_stem.stem(t) for t in _WORD.findall(text.lower())
+            if t not in _STOP and len(t) > 1]
 
 
 def strip_frontmatter(text: str) -> str:
@@ -152,9 +163,10 @@ class Bm25Index:
         self.df: Counter[str] = Counter()              # term -> #docs containing it
         self.postings: dict[str, dict[str, int]] = {}  # term -> {doc_path: tf}
         self.length: dict[str, int] = {}               # doc_path -> token count
-        self.meta: dict[str, dict] = {}                # doc_path -> {w, u, s} priors
+        self.meta: dict[str, dict] = {}                # doc_path -> {w, u, s, h} priors
         self.n_docs = 0
         self.avg_len = 0.0
+        self.built = ""                                # ISO stamp of last full (re)build
 
     def _recompute(self) -> None:
         self.n_docs = len(self.length)
@@ -165,12 +177,19 @@ class Bm25Index:
             self.remove(path)
         toks = tokenize(strip_frontmatter(raw_text))
         self.length[path] = len(toks)
-        self.meta[path] = doc_meta(path, raw_text)
+        meta = doc_meta(path, raw_text)
+        # Content hash: the incremental sweep's change detector (2.3).
+        import hashlib
+        meta["h"] = hashlib.sha1(raw_text.encode("utf-8", "replace")).hexdigest()[:16]
+        self.meta[path] = meta
         for term, tf in Counter(toks).items():
             self.postings.setdefault(term, {})[path] = tf
             self.df[term] += 1
         self._recompute()
 
+    def content_hash(self, path: str) -> str | None:
+        return (self.meta.get(path) or {}).get("h")
+
     def remove(self, path: str) -> None:
         if path not in self.length:
             return
@@ -192,11 +211,17 @@ class Bm25Index:
                 * freshness(m.get("u"), today_s)
                 * STATUS_FACTOR.get(m.get("s", ""), 1.0))
 
-    def score(self, query: str, limit: int = 10,
-              today_s: str | None = None) -> list[tuple[str, float]]:
+    def score(self, query: str, limit: int = 10, today_s: str | None = None,
+              extra_terms: dict[str, float] | None = None) -> list[tuple[str, float]]:
+        """BM25 x priors. `extra_terms` (term -> weight) are alias-expansion terms
+        (2.3): they contribute at reduced weight and can only boost documents that
+        actually contain them — expansion never invents corpus-free hits."""
         today_s = today_s or echo.today()
+        weighted: dict[str, float] = {t: 1.0 for t in set(tokenize(query))}
+        for t, w in (extra_terms or {}).items():
+            weighted.setdefault(t, w)
         scores: Counter[str] = Counter()
-        for term in set(tokenize(query)):
+        for term, weight in weighted.items():
             posting = self.postings.get(term)
             if not posting:
                 continue
@@ -204,26 +229,29 @@ class Bm25Index:
             for path, tf in posting.items():
                 dl = self.length.get(path, 0)
                 denom = tf + K1 * (1 - B + B * (dl / self.avg_len if self.avg_len else 1))
-                scores[path] += idf * (tf * (K1 + 1) / denom if denom else 0)
+                scores[path] += weight * idf * (tf * (K1 + 1) / denom if denom else 0)
         fused = {p: s * self._doc_factor(p, today_s) for p, s in scores.items()}
         ranked = sorted(fused.items(), key=lambda kv: -kv[1])[:limit]
         return [(p, s) for p, s in ranked if s > MIN_SCORE]
 
     # ---- serialization (compact; rebuildable, so the format can change freely) ----
     def to_json(self) -> dict:
-        return {"schema": INDEX_SCHEMA, "n_docs": self.n_docs, "avg_len": self.avg_len,
-                "length": self.length, "postings": self.postings, "meta": self.meta}
+        return {"schema": INDEX_SCHEMA, "built": self.built, "n_docs": self.n_docs,
+                "avg_len": self.avg_len, "length": self.length,
+                "postings": self.postings, "meta": self.meta}
 
     @classmethod
     def from_json(cls, d: dict) -> "Bm25Index":
         if d.get("schema") != INDEX_SCHEMA:
             return cls()  # older/newer format -> empty; recall's cold path rebuilds
+            # (a pre-stemming index MUST NOT be reused: its postings are unstemmed)
         ix = cls()
         ix.length = d.get("length", {})
         ix.postings = d.get("postings", {})
         ix.meta = d.get("meta", {})
         ix.n_docs = d.get("n_docs", len(ix.length))
         ix.avg_len = d.get("avg_len", 0.0)
+        ix.built = d.get("built", "")
         ix.df = Counter({term: len(posting) for term, posting in ix.postings.items()})
         return ix
 
@@ -271,27 +299,84 @@ def _indexable(path: str) -> bool:
 
 
 # ------------------------------------------------------------------- persistence
-def load_index() -> Bm25Index:
-    status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH))
-    if status == 404:
-        return Bm25Index()
-    echo.check(status, body, "recall-index load")
+# LOCAL-FIRST (2.3). The live index is a machine-local file — updating it on capture
+# is a zero-network atomic file write instead of GET-whole-index -> PUT-whole-index
+# under the vault lock (the old per-write O(vault) round trip). The vault copy is a
+# SNAPSHOT: written by sweep (and best-effort at session end), read only to seed a
+# fresh machine or to catch up after another client swept (checked at most every
+# ECHO_RECALL_SYNC_HOURS). Staleness across clients is tolerable — recall degrades
+# gracefully and the fast sweep trues it up.
+
+def _local_path() -> Path:
+    import hashlib
+    import echo_queue
+    h = hashlib.sha1((echo.BASE or "").encode()).hexdigest()[:12]
+    return echo_queue.state_dir() / f"recall-index-{h}.json"
+
+
+def _load_local() -> Bm25Index | None:
+    p = _local_path()
+    if not p.exists():
+        return None
     try:
+        return Bm25Index.from_json(json.loads(p.read_text(encoding="utf-8")))
+    except (json.JSONDecodeError, KeyError, TypeError, OSError):
+        return None
+
+
+def save_local(ix: Bm25Index) -> None:
+    p = _local_path()
+    p.parent.mkdir(parents=True, exist_ok=True)
+    tmp = p.with_suffix(".json.tmp")
+    tmp.write_text(json.dumps(ix.to_json(), ensure_ascii=False), encoding="utf-8")
+    os.replace(tmp, p)  # atomic: a crash mid-write can't corrupt the live index
+
+
+def _fetch_snapshot() -> Bm25Index | None:
+    try:
+        status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH))
+        if status != 200:
+            return None
         return Bm25Index.from_json(json.loads(body))
-    except (json.JSONDecodeError, KeyError, TypeError):
-        return Bm25Index()
+    except Exception:  # noqa: BLE001 — the snapshot is an optimization, never required
+        return None
 
 
-def save_index(ix: Bm25Index) -> None:
+def save_snapshot(ix: Bm25Index) -> None:
+    """PUT the vault snapshot — sweep + session-end only, never the per-capture path."""
     body = json.dumps(ix.to_json(), ensure_ascii=False).encode("utf-8")
     status, b = echo.request("PUT", echo.vault_url(RECALL_INDEX_PATH), data=body,
                              headers={"Content-Type": "application/json"})
-    echo.check(status, b, "recall-index save")
+    echo.check(status, b, "recall-index snapshot save")
 
 
-def rebuild(prefetched: dict | None = None) -> Bm25Index:
-    """Full rebuild from every indexable entity note. Called by sweep.py and as the
-    cold-cache path inside recall(). Saves and returns the index.
+def load_index() -> Bm25Index:
+    """The live index: local file first; the vault snapshot only seeds a fresh
+    machine or replaces a local copy that a newer sweep elsewhere has superseded
+    (checked when the local build stamp is older than SYNC_HOURS)."""
+    local = _load_local()
+    if local and local.n_docs:
+        try:
+            age_ok = local.built and (
+                _dt.datetime.now(_dt.timezone.utc)
+                - _dt.datetime.strptime(local.built, "%Y-%m-%dT%H:%M:%SZ").replace(
+                    tzinfo=_dt.timezone.utc)
+            ).total_seconds() < SYNC_HOURS * 3600
+        except ValueError:
+            age_ok = False
+        if age_ok:
+            return local
+    snap = _fetch_snapshot()
+    if snap and snap.n_docs and (not local or (snap.built or "") > (local.built or "")):
+        save_local(snap)
+        return snap
+    return local or Bm25Index()
+
+
+def rebuild(prefetched: dict | None = None, snapshot: bool = False) -> Bm25Index:
+    """Full rebuild from every indexable note. Saves the LOCAL index always; writes
+    the vault snapshot only when `snapshot=True` (sweep) — the read path never
+    surprises the vault with a PUT.
 
     `prefetched` (a {path: text} map, e.g. from echo.read_many) lets sweep.py reuse the
     content it already pulled instead of walking and re-fetching the whole vault again."""
@@ -299,27 +384,30 @@ def rebuild(prefetched: dict | None = None) -> Bm25Index:
     if prefetched is not None:
         items = ((p, t) for p, t in prefetched.items() if _indexable(p))
     else:
-        items = ((p, _get(p)) for p in _walk() if _indexable(p))
+        items = echo.read_many([p for p in _walk() if _indexable(p)]).items()
     for path, text in items:
         if text is not None:
             ix.add(path, text)   # add() strips frontmatter + extracts doc meta itself
-    save_index(ix)
+    ix.built = echo.now_iso()
+    save_local(ix)
+    if snapshot:
+        save_snapshot(ix)
     return ix
 
 
 def update_note(path: str, text: str) -> None:
-    """Best-effort incremental upkeep for a single note (mirrors how entities.json is
-    maintained on capture). Never raises into the caller. The load->save is wrapped in the
-    advisory lock with a fresh re-read (H3), so concurrent captures can't clobber the
-    recall index either."""
+    """Best-effort incremental upkeep for a single note — a LOCAL atomic file write,
+    no vault round-trip, no advisory lock (2.3). Maintains an existing local index
+    only: with no local index yet, the next recall's cold path (or a sweep) builds
+    one, and a one-note index must never shadow the snapshot. Never raises."""
     try:
         if not _indexable(path):
-            return  # only entity notes are part of the recall corpus
-        import echo_concurrency
-        with echo_concurrency.vault_lock():
-            ix = load_index()                      # fresh read inside the lock
-            ix.add(path, text)                     # raw text: add() strips + extracts meta
-            save_index(ix)
+            return  # only corpus notes are part of the recall index
+        ix = _load_local()
+        if ix is None or not ix.n_docs:
+            return
+        ix.add(path, text)                     # raw text: add() strips + extracts meta
+        save_local(ix)
     except Exception as exc:  # noqa: BLE001 — upkeep must never fail a capture
         print(f"echo_recall: index update skipped ({exc})", file=sys.stderr)
 
@@ -405,13 +493,27 @@ def recall_op(query, limit: int = 8) -> dict:
     index = idx_mod.load()
     nmap = idx_mod.name_map(index)
 
+    # --- alias query expansion (2.3): a query phrased by an entity's alias or a
+    # partial name folds that entity's title/alias vocabulary into the BM25 query at
+    # half weight — "the ECHO plugin" also scores `echo-memory` terms. Expansion can
+    # only boost documents that actually contain the terms.
+    extra_terms: dict[str, float] = {}
+    q_toks = set(tokenize(q))
+    for _slug, e, sc in idx_mod.fuzzy_candidates(index, q)[:2]:
+        if sc < 0.5:
+            continue
+        for name in (e.get("title", ""), _slug, *e.get("aliases", [])):
+            for tok in tokenize(str(name).replace("-", " ")):
+                if tok not in q_toks:
+                    extra_terms[tok] = 0.5
+
     # --- lexical layer (BM25); rebuild the cache on a cold miss --------------
     lexical: list[tuple[str, float]] = []
     try:
         ix = load_index()
         if ix.n_docs == 0:
             ix = rebuild()
-        lexical = ix.score(q, limit=limit, today_s=today_s)
+        lexical = ix.score(q, limit=limit, today_s=today_s, extra_terms=extra_terms)
     except echo.EchoError as exc:
         print(f"recall: BM25 index unavailable ({exc}); falling back to server search",
               file=sys.stderr)

echo-memory.plugin.src/skills/echo-memory/scripts/echo_session.py

@@ -135,6 +135,18 @@ def session_end_op(bundle: dict, apply: bool = False) -> dict:
         echo.cmd_put("_agent/heartbeat/last-session.md",
                      echo.temp_file(f"{path} @ {echo.now_iso()}\n".encode("utf-8")))
         steps["heartbeat"] = "ok"
+        # Index maintenance (2.3), NOT part of the bundle contract: push this
+        # machine's local recall index to the vault snapshot so other machines
+        # can seed/catch up. Best-effort — a failure never taints the bundle.
+        try:
+            import echo_recall
+            rix = echo_recall._load_local()
+            if rix and rix.n_docs:
+                rix.built = rix.built or echo.now_iso()
+                echo_recall.save_snapshot(rix)
+                steps["recall_snapshot"] = "ok"
+        except Exception:  # noqa: BLE001
+            steps["recall_snapshot"] = "skipped"
 
     data["steps"] = steps
     return echo_output.envelope("session-end", data)

echo-memory.plugin.src/skills/echo-memory/scripts/echo_stem.py

@@ -0,0 +1,78 @@
+#!/usr/bin/env python3
+"""echo_stem.py — a deliberately light, dependency-free English stemmer.  [2.3]
+
+BM25 is purely lexical: "deploy", "deployed", and "deployment" are three unrelated
+terms to it, so a paraphrased recall query misses notes stored with a different
+inflection. This Porter-LITE stemmer conflates the common inflection families while
+staying conservative — when in doubt it leaves the token alone, because an
+over-stemmed index silently merges unrelated words (far worse than a missed match).
+
+Applied by echo_recall.tokenize() at BOTH index and query time (recall-index
+schema 3; older indexes are discarded and rebuilt). Pure function, unit-tested in
+test_echo_client.py.
+
+Design notes:
+  * plural / -ed / -ing families first (with the classic restore rules: doubled
+    consonant undoubling, at/bl/iz +e, cvc +e), then one derivational suffix pass,
+    then y->i and final-e normalization so "navigate"/"navigation"/"navigating"
+    all land on "navigat".
+  * every rule guards a minimum remaining stem length — "sing" is not "s".
+"""
+from __future__ import annotations
+
+import re
+
+_VOWEL = re.compile(r"[aeiouy]")
+_DOUBLE = re.compile(r"([^aeiouylsz])\1$")
+_CVC = re.compile(r"[^aeiou][aeiouy][^aeiouwxy]$")
+
+# One derivational pass, most-specific first. Replacements keep families together:
+# navigational/navigation -> navigat · saturation/saturated -> saturat ·
+# deployment/deployed -> deploy · usefulness/useful -> use... (guarded by length).
+_SUFFIXES = (
+    ("ization", "ize"), ("ational", "ate"), ("fulness", "ful"), ("ousness", "ous"),
+    ("iveness", "ive"), ("tional", "t"), ("biliti", "ble"), ("ements", ""),
+    ("ment", ""), ("ness", ""), ("tion", "t"), ("sion", "s"), ("ance", ""),
+    ("ence", ""), ("able", ""), ("ible", ""), ("ally", "al"), ("ously", "ous"),
+    ("ively", "ive"), ("ly", ""),
+)
+
+
+def stem(t: str) -> str:
+    """Stem one lowercase token. Never raises; returns the token unchanged when no
+    rule applies safely."""
+    if len(t) <= 3 or not t.isascii():
+        return t
+    # --- plurals -------------------------------------------------------------
+    if t.endswith("sses"):
+        t = t[:-2]
+    elif t.endswith("ies") and len(t) > 4:
+        t = t[:-3] + "i"
+    elif t.endswith("s") and not t.endswith(("ss", "us", "is")):
+        t = t[:-1]
+    # --- -ed / -ing (with restore rules) --------------------------------------
+    for suf in ("ingly", "edly", "ing", "ed"):
+        if t.endswith(suf):
+            base = t[: -len(suf)]
+            if len(base) >= 3 and _VOWEL.search(base):
+                t = base
+                if t.endswith(("at", "bl", "iz")):
+                    t += "e"
+                elif _DOUBLE.search(t):
+                    t = t[:-1]
+                elif len(t) == 3 and _CVC.search(t):
+                    t += "e"
+            break
+    # --- one derivational suffix pass -----------------------------------------
+    for suf, rep in _SUFFIXES:
+        if t.endswith(suf):
+            base = t[: -len(suf)]
+            if len(base) >= 3 and len(base + rep) >= 3:
+                t = base + rep
+            break
+    # --- normalize: y->i (so penalty/penalties agree), then drop a final e -----
+    if len(t) > 3 and t.endswith("y") and t[-2] not in "aeiou":
+        t = t[:-1] + "i"
+    if len(t) > 4 and t.endswith("e"):
+        t = t[:-1]
+    return t

echo-memory.plugin.src/skills/echo-memory/scripts/sweep.py

@@ -18,9 +18,19 @@ backfill what older vaults don't have yet:
   5. stamp the marker `schema_version` to the current schema (4).
 
 Dry-run by default; pass --apply to write. READ-ONLY without --apply.
+
+**--fast (2.3)** is the incremental mode: it walks the listing (cheap), then fetches
+only what's new, gone, missing a content hash, or in today's rotating verification
+shard (hash(path) % 7 == weekday — the whole vault gets verified over a week of
+fast sweeps while each run stays tiny; --all-shards forces everything). It updates
+the LOCAL recall index and the entity index (both machine-owned), so it needs no
+--apply gate and writes no notes. `load` auto-runs it when the last fast sweep is
+older than ECHO_FAST_SWEEP_DAYS (default 7). This is what keeps the indexes honest
+against edits made directly in Obsidian or by other clients.
+
 Cross-platform: pure Python via echo.py.
 
-Usage:  sweep.py [--apply]
+Usage:  sweep.py [--apply] | sweep.py --fast [--all-shards]
 """
 
 from __future__ import annotations
@@ -102,10 +112,128 @@ def walk(prefix: str = ""):
         yield from walk(f"{prefix}{d}/")
 
 
+def _stamp_path() -> Path:
+    import echo_queue
+    return echo_queue.state_dir() / "fast-sweep.stamp"
+
+
+def _stamp_fast_sweep() -> None:
+    try:
+        p = _stamp_path()
+        p.parent.mkdir(parents=True, exist_ok=True)
+        p.write_text(echo.now_iso() + "\n", encoding="utf-8")
+    except OSError:
+        pass
+
+
+def fast_sweep_age_days() -> float | None:
+    """Days since the last fast/full sweep on THIS machine, or None if never."""
+    import datetime as dt
+    try:
+        stamp = _stamp_path().read_text(encoding="utf-8").strip()
+        then = dt.datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=dt.timezone.utc)
+        return (dt.datetime.now(dt.timezone.utc) - then).total_seconds() / 86400
+    except (OSError, ValueError):
+        return None
+
+
+def fast(all_shards: bool = False) -> str:
+    """Incremental index maintenance (2.3): true up the LOCAL recall index and the
+    entity index against the vault, fetching only new / gone / hash-missing notes
+    plus today's rotating verification shard. Index-only — writes no notes; the
+    indexes are machine-owned, so no --apply gate. Returns a one-line summary."""
+    import datetime as dt
+    import echo_recall
+
+    if get("_agent/echo-vault.md") is None:
+        return "fast-sweep skipped: vault not bootstrapped"
+
+    listing = [p for p in walk()
+               if p.endswith(".md") and p.rsplit("/", 1)[-1] not in SKIP_BASENAMES
+               and not TEMPLATE_RE.search(p)]
+    lset = set(listing)
+
+    rix = echo_recall._load_local()
+    if rix is None or not rix.n_docs:
+        # Nothing to maintain incrementally — build the local index outright
+        # (sub-second on a few-hundred-note vault; no vault snapshot write).
+        rix = echo_recall.rebuild()
+        _stamp_fast_sweep()
+        return f"fast-sweep: no local index — built one ({rix.n_docs} docs)"
+
+    index = idx_mod.load()
+    ents = index.get("entities", {})
+    removed = 0
+    for slug in [s for s, e in list(ents.items()) if e.get("path") not in lset]:
+        del ents[slug]
+        removed += 1
+    for p in [p for p in list(rix.length) if p not in lset]:
+        rix.remove(p)
+        removed += 1
+
+    try:
+        weekday = dt.date.fromisoformat(echo.today()).weekday()
+    except ValueError:
+        weekday = dt.date.today().weekday()
+
+    def in_shard(p: str) -> bool:
+        return int(idx_mod.content_hash(p), 16) % 7 == weekday
+
+    cands: set[str] = set()
+    for p in listing:
+        if not echo_recall._indexable(p):
+            continue
+        if p not in rix.length:            # new note (any client, any tool)
+            cands.add(p)
+            continue
+        kind = kind_for(p)
+        if kind:
+            slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3])
+            if not (ents.get(slug) or {}).get("h"):   # pre-2.3 entry: hash it once
+                cands.add(p)
+                continue
+        if all_shards or in_shard(p):      # rotating verification shard
+            cands.add(p)
+
+    texts = echo.read_many(sorted(cands))
+    changed = 0
+    ents_dirty = removed > 0
+    for p, t in texts.items():
+        if t is None:
+            continue
+        h = idx_mod.content_hash(t)
+        if rix.content_hash(p) != h:
+            rix.add(p, t)
+            changed += 1
+        kind = kind_for(p)
+        if kind:
+            slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3])
+            prev = ents.get(slug) or {}
+            if prev.get("h") != h or prev.get("path") != p:
+                title = links.first_h1(t) or p.rsplit("/", 1)[-1][:-3]
+                aliases = list(prev.get("aliases", [])) + idx_mod.aliases_in_frontmatter(t)
+                idx_mod.upsert(index, slug, p, kind, title, aliases, h=h)
+                ents_dirty = True
+
+    echo_recall.save_local(rix)
+    if ents_dirty:
+        idx_mod.save(index)
+    _stamp_fast_sweep()
+    return (f"fast-sweep: {len(cands)} note(s) checked, {changed} reindexed, "
+            f"{removed} removed" + (" (all shards)" if all_shards else ""))
+
+
 def main(argv: list[str] | None = None) -> int:
     ap = argparse.ArgumentParser(description="Bring an ECHO vault up to the current feature spec")
     ap.add_argument("--apply", action="store_true")
+    ap.add_argument("--fast", action="store_true",
+                    help="incremental index maintenance (local recall + entity index only)")
+    ap.add_argument("--all-shards", action="store_true",
+                    help="with --fast: verify every indexed note, not just today's shard")
     args = ap.parse_args(argv)
+    if args.fast:
+        print(fast(all_shards=args.all_shards))
+        return 0
     apply = args.apply
     tag = "APPLY" if apply else "PLAN "
 
@@ -151,7 +279,8 @@ def main(argv: list[str] | None = None) -> int:
         # aliases (e.g. mentions learned by capture) are preserved; upsert adds title variants.
         prev_aliases = prev.get("aliases", []) if prev else []
         aliases = list(prev_aliases) + idx_mod.aliases_in_frontmatter(text)
-        idx_mod.upsert(rebuilt, slug, path, kind, title, aliases)
+        idx_mod.upsert(rebuilt, slug, path, kind, title, aliases,
+                       h=idx_mod.content_hash(text))
         if not prev:
             added += 1
         elif prev.get("path") != path or prev.get("title") != title:
@@ -163,8 +292,9 @@ def main(argv: list[str] | None = None) -> int:
 
     # ---- 2b. (re)build the recall (BM25) index -------------------------------
     if apply:
-        rix = echo_recall.rebuild(prefetched=texts)
-        print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25)")
+        rix = echo_recall.rebuild(prefetched=texts, snapshot=True)
+        _stamp_fast_sweep()
+        print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25; local + vault snapshot)")
     else:
         n_idx = sum(1 for p in all_files if echo_recall._indexable(p))
         print(f"sweep: {tag} recall index — would rebuild BM25 over {n_idx} corpus notes "

echo-memory.plugin.src/skills/echo-memory/scripts/test_echo_client.py

@@ -215,6 +215,42 @@ def test_links_create_related_section_when_absent() -> None:
     assert changed and "## Related" in new and "[[resources/concepts/beta]]" in new
 
 
+def test_stemmer_conflates_inflection_families() -> None:
+    import echo_stem
+    families = [
+        ["deploy", "deploys", "deployed", "deploying", "deployment"],
+        ["navigate", "navigation", "navigating", "navigational"],
+        ["renew", "renews", "renewing"],
+        ["prefer", "prefers", "preferring", "preferred"],
+        ["penalty", "penalties"],
+        ["saturate", "saturation", "saturated"],
+        ["frequency", "frequencies"],
+        ["expand", "expanded", "expanding"],
+        ["meet", "meeting", "meetings"],
+        ["service", "services"],
+    ]
+    for fam in families:
+        stems = {echo_stem.stem(w) for w in fam}
+        assert len(stems) == 1, f"{fam} -> {stems}"
+
+
+def test_stemmer_leaves_short_and_risky_tokens_alone() -> None:
+    import echo_stem
+    # over-stemming merges unrelated words — worse than a missed match
+    for w in ("sing", "thing", "was", "is", "bus", "this", "echo", "vault", "mpm"):
+        assert echo_stem.stem(w) == w, f"{w} mangled to {echo_stem.stem(w)}"
+    # documented non-conflations (the -al / d~s irregulars full Porter also skips)
+    assert echo_stem.stem("renewal") != echo_stem.stem("renew")
+    assert echo_stem.stem("expansion") != echo_stem.stem("expand")
+
+
+def test_content_hash_stable_and_short() -> None:
+    import echo_index as ix
+    a = ix.content_hash("hello world")
+    assert a == ix.content_hash("hello world") and len(a) == 16
+    assert a != ix.content_hash("hello world!")
+
+
 def _run_all() -> int:
     tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
     failed = 0