2.3.0 — the index train: local-first recall index, incremental sweep, stemming + alias expansion
Build and Push Docker Image / build (push) Successful in 14s

One schema-bump release (recall-index schema 3, entity-index schema 2; old
recall indexes rebuild automatically, no vault migration):

- Local-first recall index: the live BM25 index lives in the machine state dir
  (keyed by endpoint hash); capture's upkeep is a zero-network atomic file
  write, no advisory lock — replacing the O(vault) GET/PUT-whole-index round
  trip per write. The vault copy is a snapshot (sweep + session-end) used to
  seed fresh machines / catch up after another client swept
  (ECHO_RECALL_SYNC_HOURS, default 24). The read path never PUTs the vault.
- Incremental sweep: entity + recall meta carry 16-char content hashes;
  `sweep --fast` fetches only new/gone/hash-missing notes plus a rotating
  weekday shard (--all-shards forces everything); deletions drop from both
  indexes; index-only so no --apply gate. `load` auto-runs it past
  ECHO_FAST_SWEEP_DAYS (7); doctor reports index freshness. Obsidian-side
  edits now reach the indexes without manual maintenance.
- Stemming + alias expansion: echo_stem.py (conservative Porter-lite, unit-
  tested families + over-stemming guards) applied at index AND query time;
  a query fuzzy-matching an entity folds its title/alias vocabulary into the
  BM25 query at half weight — expansion can only boost docs containing the
  terms. capture hashes the note's FINAL content (auto-link reordered first).

Eval gold set 8 -> 14 queries (6 paraphrases): recall@5/MRR 1.00/1.00 vs
keyword baseline 0.75/0.79. +3 unit tests, +10 e2e; test harnesses now isolate
ECHO_STATE_DIR. All seven suites green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jason Stedwell
2026-07-29 00:08:48 -05:00
parent 882d21dd96
commit b4e923c2e7
19 changed files with 611 additions and 72 deletions
@@ -790,6 +790,24 @@ def _load_gather(targets):
return results, texts, listing_files, offline, marker_missing, heartbeat_absent
def _auto_fast_sweep(offline: bool) -> str | None:
"""Opportunistic index maintenance at load (2.3): when this machine's last fast
sweep is older than ECHO_FAST_SWEEP_DAYS (default 7), run `sweep --fast` so the
local recall + entity indexes true up against edits made by Obsidian or other
clients. Best-effort; skipped offline; never raises."""
if offline:
return None
try:
import sweep as sweep_mod
days = float(os.environ.get("ECHO_FAST_SWEEP_DAYS", "7"))
age = sweep_mod.fast_sweep_age_days()
if age is not None and age < days:
return None
return sweep_mod.fast()
except Exception: # noqa: BLE001 — maintenance must never block a load
return None
def _fetch_pointed_session(texts: dict) -> None:
"""Follow the heartbeat pointer and stash the pointed session log for the digest."""
hb = texts.get("heartbeat")
@@ -818,9 +836,10 @@ def load_op(brief: bool = True) -> dict:
targets = LOAD_TARGETS + [("today", f"journal/daily/{today()}.md"),
("inbox", "inbox/captures/inbox.md")]
_, texts, listing_files, offline, marker_missing, _ = _load_gather(targets)
swept = _auto_fast_sweep(offline)
data = {"ok": True, "action": "load", "offline": offline,
"marker_missing": marker_missing, "synced": synced,
"needs_attention": flagged,
"needs_attention": flagged, "fast_sweep": swept,
"sections": {k: v for k, v in texts.items() if not k.startswith("_")},
"recent_sessions": sorted((f for f in listing_files if f.endswith(".md")),
reverse=True)[:5]}
@@ -863,6 +882,10 @@ def cmd_load(brief: bool = False) -> int:
results, texts, listing_files, offline, marker_missing, heartbeat_absent = \
_load_gather(targets)
swept = _auto_fast_sweep(offline)
if swept:
print(f"({swept})\n")
if brief:
_fetch_pointed_session(texts)
print(_render_brief(texts, listing_files, offline))