2.3.0 — the index train: local-first recall index, incremental sweep, stemming + alias expansion
Build and Push Docker Image / build (push) Successful in 14s

One schema-bump release (recall-index schema 3, entity-index schema 2; old
recall indexes rebuild automatically, no vault migration):

- Local-first recall index: the live BM25 index lives in the machine state dir
  (keyed by endpoint hash); capture's upkeep is a zero-network atomic file
  write, no advisory lock — replacing the O(vault) GET/PUT-whole-index round
  trip per write. The vault copy is a snapshot (sweep + session-end) used to
  seed fresh machines / catch up after another client swept
  (ECHO_RECALL_SYNC_HOURS, default 24). The read path never PUTs the vault.
- Incremental sweep: entity + recall meta carry 16-char content hashes;
  `sweep --fast` fetches only new/gone/hash-missing notes plus a rotating
  weekday shard (--all-shards forces everything); deletions drop from both
  indexes; index-only so no --apply gate. `load` auto-runs it past
  ECHO_FAST_SWEEP_DAYS (7); doctor reports index freshness. Obsidian-side
  edits now reach the indexes without manual maintenance.
- Stemming + alias expansion: echo_stem.py (conservative Porter-lite, unit-
  tested families + over-stemming guards) applied at index AND query time;
  a query fuzzy-matching an entity folds its title/alias vocabulary into the
  BM25 query at half weight — expansion can only boost docs containing the
  terms. capture hashes the note's FINAL content (auto-link reordered first).

Eval gold set 8 -> 14 queries (6 paraphrases): recall@5/MRR 1.00/1.00 vs
keyword baseline 0.75/0.79. +3 unit tests, +10 e2e; test harnesses now isolate
ECHO_STATE_DIR. All seven suites green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jason Stedwell
2026-07-29 00:08:48 -05:00
parent 882d21dd96
commit b4e923c2e7
19 changed files with 611 additions and 72 deletions
@@ -333,15 +333,11 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status
echo.cmd_put(path, echo.temp_file(note.encode()))
action = "created"
# H3: the entity-index write is the race the review flagged — a bare load->save lets
# two concurrent captures clobber each other's entry. Route it through a lock-guarded,
# fresh-re-read transaction. Returns the fresh index (used by auto-link below).
import echo_concurrency
index = echo_concurrency.atomic_index_update(
lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases))
# auto-link: any other known entity CONFIDENTLY named in the body (M2: is_confident_link
# rejects short/common tokens so a 3-char alias or a word like "API" can't link everywhere).
# auto-link FIRST (it mutates the note's ## Related section), so the content
# hash stored below reflects the note's final state. Uses the pre-update index
# — it only needs to find OTHER entities named in the body. (M2: is_confident_link
# rejects short/common tokens so a 3-char alias or a word like "API" can't link
# everywhere.)
linked = 0
for s2, e2 in index.get("entities", {}).items():
if e2.get("path") in (None, path):
@@ -351,11 +347,24 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status
ca, cb = links.link_bidirectional(path, e2["path"])
linked += int(ca or cb)
# Keep the BM25 recall index current for this note (best-effort; lock-guarded inside
# update_note, so it can't clobber a concurrent writer either).
# The note's final content: hashed into the entity index (the incremental
# sweep's change detector, 2.3) and fed to the local recall index.
final_text = links.get_text(path) or ""
note_h = idx_mod.content_hash(final_text) if final_text else None
# H3: the entity-index write is the race the review flagged — a bare load->save lets
# two concurrent captures clobber each other's entry. Route it through a lock-guarded,
# fresh-re-read transaction.
import echo_concurrency
index = echo_concurrency.atomic_index_update(
lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases,
h=note_h))
# Keep the local BM25 recall index current for this note (best-effort; a
# zero-network atomic file write since 2.3).
try:
import echo_recall
echo_recall.update_note(path, links.get_text(path) or "")
echo_recall.update_note(path, final_text)
except Exception as exc: # never let recall-index upkeep fail a capture
print(f"echo_ops: recall-index update skipped ({exc})", file=sys.stderr)