2.3.0 — the index train: local-first recall index, incremental sweep, stemming + alias expansion
Build and Push Docker Image / build (push) Successful in 14s
Build and Push Docker Image / build (push) Successful in 14s
One schema-bump release (recall-index schema 3, entity-index schema 2; old recall indexes rebuild automatically, no vault migration): - Local-first recall index: the live BM25 index lives in the machine state dir (keyed by endpoint hash); capture's upkeep is a zero-network atomic file write, no advisory lock — replacing the O(vault) GET/PUT-whole-index round trip per write. The vault copy is a snapshot (sweep + session-end) used to seed fresh machines / catch up after another client swept (ECHO_RECALL_SYNC_HOURS, default 24). The read path never PUTs the vault. - Incremental sweep: entity + recall meta carry 16-char content hashes; `sweep --fast` fetches only new/gone/hash-missing notes plus a rotating weekday shard (--all-shards forces everything); deletions drop from both indexes; index-only so no --apply gate. `load` auto-runs it past ECHO_FAST_SWEEP_DAYS (7); doctor reports index freshness. Obsidian-side edits now reach the indexes without manual maintenance. - Stemming + alias expansion: echo_stem.py (conservative Porter-lite, unit- tested families + over-stemming guards) applied at index AND query time; a query fuzzy-matching an entity folds its title/alias vocabulary into the BM25 query at half weight — expansion can only boost docs containing the terms. capture hashes the note's FINAL content (auto-link reordered first). Eval gold set 8 -> 14 queries (6 paraphrases): recall@5/MRR 1.00/1.00 vs keyword baseline 0.75/0.79. +3 unit tests, +10 e2e; test harnesses now isolate ECHO_STATE_DIR. All seven suites green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -333,15 +333,11 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status
|
||||
echo.cmd_put(path, echo.temp_file(note.encode()))
|
||||
action = "created"
|
||||
|
||||
# H3: the entity-index write is the race the review flagged — a bare load->save lets
|
||||
# two concurrent captures clobber each other's entry. Route it through a lock-guarded,
|
||||
# fresh-re-read transaction. Returns the fresh index (used by auto-link below).
|
||||
import echo_concurrency
|
||||
index = echo_concurrency.atomic_index_update(
|
||||
lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases))
|
||||
|
||||
# auto-link: any other known entity CONFIDENTLY named in the body (M2: is_confident_link
|
||||
# rejects short/common tokens so a 3-char alias or a word like "API" can't link everywhere).
|
||||
# auto-link FIRST (it mutates the note's ## Related section), so the content
|
||||
# hash stored below reflects the note's final state. Uses the pre-update index
|
||||
# — it only needs to find OTHER entities named in the body. (M2: is_confident_link
|
||||
# rejects short/common tokens so a 3-char alias or a word like "API" can't link
|
||||
# everywhere.)
|
||||
linked = 0
|
||||
for s2, e2 in index.get("entities", {}).items():
|
||||
if e2.get("path") in (None, path):
|
||||
@@ -351,11 +347,24 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status
|
||||
ca, cb = links.link_bidirectional(path, e2["path"])
|
||||
linked += int(ca or cb)
|
||||
|
||||
# Keep the BM25 recall index current for this note (best-effort; lock-guarded inside
|
||||
# update_note, so it can't clobber a concurrent writer either).
|
||||
# The note's final content: hashed into the entity index (the incremental
|
||||
# sweep's change detector, 2.3) and fed to the local recall index.
|
||||
final_text = links.get_text(path) or ""
|
||||
note_h = idx_mod.content_hash(final_text) if final_text else None
|
||||
|
||||
# H3: the entity-index write is the race the review flagged — a bare load->save lets
|
||||
# two concurrent captures clobber each other's entry. Route it through a lock-guarded,
|
||||
# fresh-re-read transaction.
|
||||
import echo_concurrency
|
||||
index = echo_concurrency.atomic_index_update(
|
||||
lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases,
|
||||
h=note_h))
|
||||
|
||||
# Keep the local BM25 recall index current for this note (best-effort; a
|
||||
# zero-network atomic file write since 2.3).
|
||||
try:
|
||||
import echo_recall
|
||||
echo_recall.update_note(path, links.get_text(path) or "")
|
||||
echo_recall.update_note(path, final_text)
|
||||
except Exception as exc: # never let recall-index upkeep fail a capture
|
||||
print(f"echo_ops: recall-index update skipped ({exc})", file=sys.stderr)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user