2.3.0 — the index train: local-first recall index, incremental sweep, stemming + alias expansion
One schema-bump release (recall-index schema 3, entity-index schema 2; old recall indexes rebuild automatically, no vault migration): - Local-first recall index: the live BM25 index lives in the machine state dir (keyed by endpoint hash); capture's upkeep is a zero-network atomic file write, no advisory lock — replacing the O(vault) GET/PUT-whole-index round trip per write. The vault copy is a snapshot (sweep + session-end) used to seed fresh machines / catch up after another client swept (ECHO_RECALL_SYNC_HOURS, default 24). The read path never PUTs the vault. - Incremental sweep: entity + recall meta carry 16-char content hashes; `sweep --fast` fetches only new/gone/hash-missing notes plus a rotating weekday shard (--all-shards forces everything); deletions drop from both indexes; index-only so no --apply gate. `load` auto-runs it past ECHO_FAST_SWEEP_DAYS (7); doctor reports index freshness. Obsidian-side edits now reach the indexes without manual maintenance. - Stemming + alias expansion: echo_stem.py (conservative Porter-lite, unit- tested families + over-stemming guards) applied at index AND query time; a query fuzzy-matching an entity folds its title/alias vocabulary into the BM25 query at half weight — expansion can only boost docs containing the terms. capture hashes the note's FINAL content (auto-link reordered first). Eval gold set 8 -> 14 queries (6 paraphrases): recall@5/MRR 1.00/1.00 vs keyword baseline 0.75/0.79. +3 unit tests, +10 e2e; test harnesses now isolate ECHO_STATE_DIR. All seven suites green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jason Stedwell
@@ -1,5 +1,57 @@
|
||||
# Changelog
|
||||
|
||||
## 2.3.0
|
||||
|
||||
**The index train** (IMPROVEMENT-PLANS #2/#4/#8) — one schema-bump release:
|
||||
recall-index schema 3, entity-index schema 2. Old recall indexes are discarded
|
||||
and rebuilt automatically (sub-second); no vault migration.
|
||||
|
||||
### Changed — the recall index is LOCAL-FIRST
|
||||
|
||||
The live BM25 index moves out of the vault into the machine state dir
|
||||
(`recall-index-<endpoint-hash>.json` under `ECHO_STATE_DIR`). Capture's index
|
||||
upkeep — previously GET-whole-index → PUT-whole-index against the vault under
|
||||
the advisory lock, an O(vault) network round trip per write — is now a
|
||||
zero-network atomic file write with no lock at all. The vault copy at
|
||||
`_agent/index/recall-index.json` becomes a **snapshot**: written by
|
||||
`sweep --apply` and best-effort at `session-end`, read only to seed a fresh
|
||||
machine or to catch up after another client swept (checked at most every
|
||||
`ECHO_RECALL_SYNC_HOURS`, default 24). The read path never PUTs the vault.
|
||||
|
||||
### Added — incremental sweep (`sweep --fast`) + content hashes
|
||||
|
||||
Entity-index entries and recall-index doc meta now carry a 16-char content
|
||||
hash. `sweep --fast` walks the listing (cheap) and fetches only what is new,
|
||||
gone, missing a hash, or in **today's rotating verification shard**
|
||||
(`hash(path) % 7 == weekday` — the whole vault gets verified across a week of
|
||||
fast sweeps while each run stays tiny; `--all-shards` forces everything).
|
||||
Index-only and machine-owned, so no `--apply` gate; deletions are dropped from
|
||||
both indexes. **`load` auto-runs it** when this machine's last sweep is older
|
||||
than `ECHO_FAST_SWEEP_DAYS` (default 7) — edits made directly in Obsidian or by
|
||||
other clients now reach the indexes without anyone remembering maintenance.
|
||||
`doctor` reports index freshness.
|
||||
|
||||
### Added — stemming + alias query expansion
|
||||
|
||||
`echo_stem.py`: a conservative Porter-lite stemmer applied by `tokenize()` at
|
||||
BOTH index and query time, so "deployed"/"deployment"/"deploys" meet at
|
||||
"deploy" and "penalties" finds "penalty" (guards against over-stemming: "sing"
|
||||
is not "s"; the -al/d~s irregulars are deliberately not conflated). At recall
|
||||
time, a query that fuzzy-matches an entity (score ≥ 0.5) folds that entity's
|
||||
title/alias vocabulary into the BM25 query at **half weight** — "turbokappa
|
||||
tuning" finds the Kappa Engine note even though the alias appears in no body.
|
||||
Expansion can only boost documents that actually contain the terms.
|
||||
|
||||
### Eval
|
||||
|
||||
Gold set grows 8 → **14 queries** (6 paraphrases the pre-2.3 lexical index
|
||||
missed). v2.3.0: recall@5 / MRR **1.00 / 1.00**, 3/3 session-journal queries;
|
||||
keyword baseline 0.75 / 0.79 and 0/3. Tests: +3 unit (stemmer families,
|
||||
over-stemming guards, hash), +10 end-to-end (local-first no-snapshot-write,
|
||||
local recallability, stemmed recall, alias expansion, fast-sweep pickup/
|
||||
deletion/hash backfill). All suites — including the offline-queue, ops-api,
|
||||
and MCP server suites — green; test harnesses now isolate `ECHO_STATE_DIR`.
|
||||
|
||||
## 2.2.0
|
||||
|
||||
### Added — echo-mcp: the containerized MCP server
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# echo-memory — v2.2.0
|
||||
# echo-memory — v2.3.0
|
||||
|
||||
Persistent memory for Claude / CoWork sessions via the **ECHO** Obsidian vault, driven over the [Obsidian Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api). The skill makes direct REST calls through a bundled validated client (`scripts/echo.py`). The whole toolchain is **pure Python** (stdlib only), so it runs identically on Windows, macOS, and Linux — no bash, no platform-specific `date`.
|
||||
|
||||
@@ -403,14 +403,14 @@ If the API returns a connection error, timeout, or `502` (usually Obsidian / the
|
||||
|
||||
---
|
||||
|
||||
## Eval metrics (v1.5.1, 2026-07-03)
|
||||
## Eval metrics (v2.3.0, 2026-07-29)
|
||||
|
||||
From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate.
|
||||
From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate. The gold set grew to **14 queries in 2.3** — six are paraphrases (morphological variants like "penalties in the SLA" vs a note saying "penalty clause") that a purely lexical index misses; the stemmed index answers all of them.
|
||||
|
||||
| Metric | v1.5.1 | pre-1.5 baseline |
|
||||
| Metric | v2.3.0 | pre-1.5 baseline |
|
||||
|---|---|---|
|
||||
| Retrieval recall@5 / MRR (8-query gold set) | **1.00 / 1.00** | 0.75 / 0.75 |
|
||||
| Queries answerable only from sessions/journal | **2/2** | 0/2 (not in corpus) |
|
||||
| Retrieval recall@5 / MRR (14-query gold set incl. 6 paraphrases) | **1.00 / 1.00** | 0.75 / 0.79 |
|
||||
| Queries answerable only from sessions/journal | **3/3** | 0/3 (not in corpus) |
|
||||
| Freshness: live note outranks stale archived twin | **yes** | no |
|
||||
| Duplicate notes created (3 renamed-entity captures) | **0** (gate blocks) | 3 |
|
||||
| Legitimate captures wrongly blocked | **0** | 0 |
|
||||
@@ -421,6 +421,7 @@ From the credential-free harness (`eval/run_eval.py` against the deterministic m
|
||||
|
||||
| Version | Highlights |
|
||||
|---------|-----------|
|
||||
| **2.3.0** | **The index train (recall-index schema 3, entity-index schema 2).** (1) **Local-first recall index** — the live BM25 index moves to the machine state dir (keyed by endpoint); capture's index upkeep becomes a zero-network atomic file write instead of GET/PUT-whole-index under the vault lock; the vault copy is now a snapshot (written by sweep + session-end) that seeds fresh machines. (2) **Incremental sweep** — entity entries carry a content hash; `sweep --fast` fetches only new/gone/changed notes plus a rotating weekday shard (whole vault verified across a week); auto-runs at load when >`ECHO_FAST_SWEEP_DAYS` (7) — Obsidian-side edits now reach the indexes without manual maintenance. (3) **Stemming + alias expansion** — `echo_stem` (Porter-lite, conservative) applied at index+query time, and a query matching an entity's alias folds its title/alias vocabulary in at half weight. Eval gold set grows to 14 queries (6 paraphrases): recall@5 / MRR **1.00 / 1.00**. |
|
||||
| **2.2.0** | **echo-mcp — the containerized MCP server.** ECHO as 14 typed MCP tools from any connector-capable surface (Claude Code, CoWork, claude.ai): `mcp-server/` + Dockerfile in-repo, deployed on ALPHA as `echo-mcp` behind `https://echomcp.alwisp.com` (streamable HTTP, stateless JSON, bearer auth, open `/health`). Talks to the Obsidian REST API directly on the LAN (`10.2.0.35:27123`) — one client round trip per tool call, all vault chatter host-local. Duplicate gate & offline queueing surface as data; `ECHO_MCP_TOOLS=core` trims the surface to six tools; writes serialize server-side. New `eval/test_mcp_server.py` e2e suite. Full spec: `docs/MCP-SERVER-SPEC.md`. |
|
||||
| **2.1.1** | **MCP Phase 0 — return-not-print.** Every high-level op gains a core `*_op` function returning an envelope dict with a stdout-purity guarantee (helper chatter → stderr); CLI verbs become thin wrappers with identical output and exit codes. Duplicate gate and offline queueing become data (`action: duplicate-gate` / `queued: true`). New `eval/test_ops_api.py` contract suite. This is the seam the 2.2 containerized MCP server wraps. |
|
||||
| **2.1.0** | **Quick-wins train: cheaper sessions, durable capture, one-call session end.** (1) **`load --brief`** — a token-budgeted cold-start digest (Fact/Pattern in full, last ~10 Observations, scope+freshness, last session's key sections, Agent-Log lines, inbox *count*; `ECHO_LOAD_BUDGET` default ~8000 chars) now injected by the SessionStart hook, replacing the full six-file dump that grew unboundedly; all `load` reads are fetched in parallel. (2) **Offline capture durability** — `capture` on an unreachable vault queues the *whole operation* as one semantic record; `flush` replays it **through capture** so routing/gate/aliasing re-run against the current index; a gate stop on replay is kept + flagged, never landed blind; `ensure_daily_log`/update-path writes ride the queue too. (3) **`session-end`** — one call (one lock) writes session log → Agent-Log line → reflect proposals → optional scope switch → **heartbeat last as the commit marker**; dry-run by default; `ECHO_NOW` pins the HHMM. Also fixes the `__main__` twin-module trap so helper-module `EchoError`s exit with their intended codes. +19 end-to-end checks. |
|
||||
|
||||
Binary file not shown.
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "echo-memory",
|
||||
"version": "2.2.0",
|
||||
"version": "2.3.0",
|
||||
"homepage": "https://git.alwisp.com/jason/echo",
|
||||
"repository": "https://git.alwisp.com/jason/echo",
|
||||
"description": "Persistent memory via the ECHO Obsidian vault over the Obsidian Local REST API. Cross-platform Python client: one-call capture/resolve/recall/link/triage over an entity index, hybrid BM25 + graph recall spanning entities + sessions/journal (recency/status-aware), a pre-write duplicate gate, complete-frontmatter capture, session hooks that self-fire load/reflect, offline write-ahead queue, lock-guarded concurrency, linter-enforced routing, and /echo-* commands.",
|
||||
|
||||
@@ -358,6 +358,8 @@ Run it after `migrate.py` (which handles structural/schema changes) — or any t
|
||||
|
||||
The pass is cheap and pays for itself by catching drift before it requires a reorg. Write the findings as a digest; act on them only with the operator's go-ahead.
|
||||
|
||||
**Incremental maintenance (2.3).** `sweep.py --fast` is the cheap, index-only mode: it fetches just what's new, gone, or changed (content hashes) plus a rotating daily verification shard, so edits made directly in Obsidian or by other clients reach the entity + recall indexes without a full pass. It runs **automatically at load** when this machine's last sweep is older than `ECHO_FAST_SWEEP_DAYS` (default 7) — you rarely need to invoke it. The recall index itself is **local-first**: capture maintains a machine-local copy with zero vault round-trips; the vault's `recall-index.json` is a snapshot (sweep/session-end) used to seed fresh machines.
|
||||
|
||||
**Performance.** Full-vault scripts (`sweep.py`, `vault_lint.py`, recall rebuild) are connection-pooled (keep-alive) and read the whole vault concurrently via `echo.read_many`, so they finish in about a second on a few-hundred-note vault instead of timing out on hundreds of serial TLS handshakes. Tune with `ECHO_WORKERS` (default 8, bulk-read concurrency) and `ECHO_TIMEOUT` (default 30s, per-request). For a vault large enough that even the concurrent pass nears the tool timeout, run the script with the harness's background-execution option rather than blocking the turn.
|
||||
|
||||
## Daily Note — Agent Log
|
||||
|
||||
@@ -790,6 +790,24 @@ def _load_gather(targets):
|
||||
return results, texts, listing_files, offline, marker_missing, heartbeat_absent
|
||||
|
||||
|
||||
def _auto_fast_sweep(offline: bool) -> str | None:
|
||||
"""Opportunistic index maintenance at load (2.3): when this machine's last fast
|
||||
sweep is older than ECHO_FAST_SWEEP_DAYS (default 7), run `sweep --fast` so the
|
||||
local recall + entity indexes true up against edits made by Obsidian or other
|
||||
clients. Best-effort; skipped offline; never raises."""
|
||||
if offline:
|
||||
return None
|
||||
try:
|
||||
import sweep as sweep_mod
|
||||
days = float(os.environ.get("ECHO_FAST_SWEEP_DAYS", "7"))
|
||||
age = sweep_mod.fast_sweep_age_days()
|
||||
if age is not None and age < days:
|
||||
return None
|
||||
return sweep_mod.fast()
|
||||
except Exception: # noqa: BLE001 — maintenance must never block a load
|
||||
return None
|
||||
|
||||
|
||||
def _fetch_pointed_session(texts: dict) -> None:
|
||||
"""Follow the heartbeat pointer and stash the pointed session log for the digest."""
|
||||
hb = texts.get("heartbeat")
|
||||
@@ -818,9 +836,10 @@ def load_op(brief: bool = True) -> dict:
|
||||
targets = LOAD_TARGETS + [("today", f"journal/daily/{today()}.md"),
|
||||
("inbox", "inbox/captures/inbox.md")]
|
||||
_, texts, listing_files, offline, marker_missing, _ = _load_gather(targets)
|
||||
swept = _auto_fast_sweep(offline)
|
||||
data = {"ok": True, "action": "load", "offline": offline,
|
||||
"marker_missing": marker_missing, "synced": synced,
|
||||
"needs_attention": flagged,
|
||||
"needs_attention": flagged, "fast_sweep": swept,
|
||||
"sections": {k: v for k, v in texts.items() if not k.startswith("_")},
|
||||
"recent_sessions": sorted((f for f in listing_files if f.endswith(".md")),
|
||||
reverse=True)[:5]}
|
||||
@@ -863,6 +882,10 @@ def cmd_load(brief: bool = False) -> int:
|
||||
results, texts, listing_files, offline, marker_missing, heartbeat_absent = \
|
||||
_load_gather(targets)
|
||||
|
||||
swept = _auto_fast_sweep(offline)
|
||||
if swept:
|
||||
print(f"({swept})\n")
|
||||
|
||||
if brief:
|
||||
_fetch_pointed_session(texts)
|
||||
print(_render_brief(texts, listing_files, offline))
|
||||
|
||||
@@ -74,6 +74,18 @@ def run_op() -> dict:
|
||||
else:
|
||||
line(status < 400, "marker fetch", f"HTTP {status}")
|
||||
|
||||
# Index freshness (2.3) — informational, never red: how stale this machine's
|
||||
# local indexes are (the fast sweep trues them up at load past 7 days).
|
||||
if fatal is None:
|
||||
try:
|
||||
import sweep as sweep_mod
|
||||
age = sweep_mod.fast_sweep_age_days()
|
||||
line(True, "index freshness",
|
||||
f"last local sweep {age:.1f}d ago" if age is not None
|
||||
else "no local sweep yet — runs automatically at the next load")
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
reds = sum(1 for c in checks if not c["ok"])
|
||||
return {"ok": reds == 0 and fatal is None, "action": "doctor",
|
||||
"endpoint": cfg["endpoint"], "fatal": fatal, "reds": reds, "checks": checks}
|
||||
|
||||
@@ -168,7 +168,10 @@ def kind_for_path(path: str) -> str | None:
|
||||
|
||||
|
||||
def empty_index() -> dict:
|
||||
return {"schema": 1, "updated": echo.today(), "entities": {}}
|
||||
# Schema 2 (2.3): entries may carry `h`, a short content hash of the note body —
|
||||
# the incremental sweep's change detector. Tolerated-absent (schema-1 entries
|
||||
# simply get fetched and hashed on their first fast sweep).
|
||||
return {"schema": 2, "updated": echo.today(), "entities": {}}
|
||||
|
||||
|
||||
def load() -> dict:
|
||||
@@ -287,13 +290,20 @@ def gate_candidates(index: dict, mention: str, kind: str | None = None,
|
||||
return out
|
||||
|
||||
|
||||
def content_hash(text: str) -> str:
|
||||
import hashlib
|
||||
return hashlib.sha1(str(text).encode("utf-8", "replace")).hexdigest()[:16]
|
||||
|
||||
|
||||
def upsert(index: dict, slug: str, path: str, kind: str, title: str | None = None,
|
||||
aliases=None) -> dict:
|
||||
aliases=None, h: str | None = None) -> dict:
|
||||
ents = index.setdefault("entities", {})
|
||||
e = ents.get(slug, {})
|
||||
e["path"] = path
|
||||
e["kind"] = kind
|
||||
e["title"] = title or e.get("title") or slug
|
||||
if h:
|
||||
e["h"] = h
|
||||
# Keep any prior + explicit aliases, and ALWAYS fold in the safe title variants so a
|
||||
# hyphen/space/case form of the name resolves even if no one passed --aliases. Drop any
|
||||
# alias that just equals the slug (resolve already checks the slug).
|
||||
|
||||
@@ -333,15 +333,11 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status
|
||||
echo.cmd_put(path, echo.temp_file(note.encode()))
|
||||
action = "created"
|
||||
|
||||
# H3: the entity-index write is the race the review flagged — a bare load->save lets
|
||||
# two concurrent captures clobber each other's entry. Route it through a lock-guarded,
|
||||
# fresh-re-read transaction. Returns the fresh index (used by auto-link below).
|
||||
import echo_concurrency
|
||||
index = echo_concurrency.atomic_index_update(
|
||||
lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases))
|
||||
|
||||
# auto-link: any other known entity CONFIDENTLY named in the body (M2: is_confident_link
|
||||
# rejects short/common tokens so a 3-char alias or a word like "API" can't link everywhere).
|
||||
# auto-link FIRST (it mutates the note's ## Related section), so the content
|
||||
# hash stored below reflects the note's final state. Uses the pre-update index
|
||||
# — it only needs to find OTHER entities named in the body. (M2: is_confident_link
|
||||
# rejects short/common tokens so a 3-char alias or a word like "API" can't link
|
||||
# everywhere.)
|
||||
linked = 0
|
||||
for s2, e2 in index.get("entities", {}).items():
|
||||
if e2.get("path") in (None, path):
|
||||
@@ -351,11 +347,24 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status
|
||||
ca, cb = links.link_bidirectional(path, e2["path"])
|
||||
linked += int(ca or cb)
|
||||
|
||||
# Keep the BM25 recall index current for this note (best-effort; lock-guarded inside
|
||||
# update_note, so it can't clobber a concurrent writer either).
|
||||
# The note's final content: hashed into the entity index (the incremental
|
||||
# sweep's change detector, 2.3) and fed to the local recall index.
|
||||
final_text = links.get_text(path) or ""
|
||||
note_h = idx_mod.content_hash(final_text) if final_text else None
|
||||
|
||||
# H3: the entity-index write is the race the review flagged — a bare load->save lets
|
||||
# two concurrent captures clobber each other's entry. Route it through a lock-guarded,
|
||||
# fresh-re-read transaction.
|
||||
import echo_concurrency
|
||||
index = echo_concurrency.atomic_index_update(
|
||||
lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases,
|
||||
h=note_h))
|
||||
|
||||
# Keep the local BM25 recall index current for this note (best-effort; a
|
||||
# zero-network atomic file write since 2.3).
|
||||
try:
|
||||
import echo_recall
|
||||
echo_recall.update_note(path, links.get_text(path) or "")
|
||||
echo_recall.update_note(path, final_text)
|
||||
except Exception as exc: # never let recall-index upkeep fail a capture
|
||||
print(f"echo_ops: recall-index update skipped ({exc})", file=sys.stderr)
|
||||
|
||||
|
||||
@@ -52,9 +52,17 @@ sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import echo # noqa: E402
|
||||
import echo_index as idx_mod # noqa: E402
|
||||
import echo_links as links # noqa: E402
|
||||
import echo_stem # noqa: E402
|
||||
|
||||
RECALL_INDEX_PATH = "_agent/index/recall-index.json"
|
||||
INDEX_SCHEMA = 2 # bumped: schema 2 adds per-doc meta (weight/updated/status)
|
||||
# Schema 3 (2.3, the "index train"): postings are STEMMED (echo_stem), per-doc meta
|
||||
# gains a content hash `h` (incremental sweep), and the index carries a `built`
|
||||
# stamp + endpoint hash. The LIVE index is LOCAL-FIRST — it lives in the state dir
|
||||
# and is updated with zero vault round-trips; the vault copy at RECALL_INDEX_PATH
|
||||
# is a snapshot written by sweep/session-end, used to seed fresh machines. Older
|
||||
# schemas are discarded and rebuilt (sub-second since the 1.1 network work).
|
||||
INDEX_SCHEMA = 3
|
||||
SYNC_HOURS = float(os.environ.get("ECHO_RECALL_SYNC_HOURS", "24"))
|
||||
|
||||
# Tuning knobs. Defaults are standard BM25 (k1/b) + a 0.6 hop decay over <=2 hops.
|
||||
K1 = 1.5
|
||||
@@ -131,8 +139,11 @@ _SKIP_BASENAMES = {"README.md"}
|
||||
|
||||
|
||||
def tokenize(text: str) -> list[str]:
|
||||
"""Lowercase word tokens, stopworded. Frontmatter should be stripped by the caller."""
|
||||
return [t for t in _WORD.findall(text.lower()) if t not in _STOP and len(t) > 1]
|
||||
"""Lowercase word tokens, stopworded, STEMMED (2.3) — applied identically at
|
||||
index and query time, so 'deployed'/'deployment' meet at 'deploy'.
|
||||
Frontmatter should be stripped by the caller."""
|
||||
return [echo_stem.stem(t) for t in _WORD.findall(text.lower())
|
||||
if t not in _STOP and len(t) > 1]
|
||||
|
||||
|
||||
def strip_frontmatter(text: str) -> str:
|
||||
@@ -152,9 +163,10 @@ class Bm25Index:
|
||||
self.df: Counter[str] = Counter() # term -> #docs containing it
|
||||
self.postings: dict[str, dict[str, int]] = {} # term -> {doc_path: tf}
|
||||
self.length: dict[str, int] = {} # doc_path -> token count
|
||||
self.meta: dict[str, dict] = {} # doc_path -> {w, u, s} priors
|
||||
self.meta: dict[str, dict] = {} # doc_path -> {w, u, s, h} priors
|
||||
self.n_docs = 0
|
||||
self.avg_len = 0.0
|
||||
self.built = "" # ISO stamp of last full (re)build
|
||||
|
||||
def _recompute(self) -> None:
|
||||
self.n_docs = len(self.length)
|
||||
@@ -165,12 +177,19 @@ class Bm25Index:
|
||||
self.remove(path)
|
||||
toks = tokenize(strip_frontmatter(raw_text))
|
||||
self.length[path] = len(toks)
|
||||
self.meta[path] = doc_meta(path, raw_text)
|
||||
meta = doc_meta(path, raw_text)
|
||||
# Content hash: the incremental sweep's change detector (2.3).
|
||||
import hashlib
|
||||
meta["h"] = hashlib.sha1(raw_text.encode("utf-8", "replace")).hexdigest()[:16]
|
||||
self.meta[path] = meta
|
||||
for term, tf in Counter(toks).items():
|
||||
self.postings.setdefault(term, {})[path] = tf
|
||||
self.df[term] += 1
|
||||
self._recompute()
|
||||
|
||||
def content_hash(self, path: str) -> str | None:
|
||||
return (self.meta.get(path) or {}).get("h")
|
||||
|
||||
def remove(self, path: str) -> None:
|
||||
if path not in self.length:
|
||||
return
|
||||
@@ -192,11 +211,17 @@ class Bm25Index:
|
||||
* freshness(m.get("u"), today_s)
|
||||
* STATUS_FACTOR.get(m.get("s", ""), 1.0))
|
||||
|
||||
def score(self, query: str, limit: int = 10,
|
||||
today_s: str | None = None) -> list[tuple[str, float]]:
|
||||
def score(self, query: str, limit: int = 10, today_s: str | None = None,
|
||||
extra_terms: dict[str, float] | None = None) -> list[tuple[str, float]]:
|
||||
"""BM25 x priors. `extra_terms` (term -> weight) are alias-expansion terms
|
||||
(2.3): they contribute at reduced weight and can only boost documents that
|
||||
actually contain them — expansion never invents corpus-free hits."""
|
||||
today_s = today_s or echo.today()
|
||||
weighted: dict[str, float] = {t: 1.0 for t in set(tokenize(query))}
|
||||
for t, w in (extra_terms or {}).items():
|
||||
weighted.setdefault(t, w)
|
||||
scores: Counter[str] = Counter()
|
||||
for term in set(tokenize(query)):
|
||||
for term, weight in weighted.items():
|
||||
posting = self.postings.get(term)
|
||||
if not posting:
|
||||
continue
|
||||
@@ -204,26 +229,29 @@ class Bm25Index:
|
||||
for path, tf in posting.items():
|
||||
dl = self.length.get(path, 0)
|
||||
denom = tf + K1 * (1 - B + B * (dl / self.avg_len if self.avg_len else 1))
|
||||
scores[path] += idf * (tf * (K1 + 1) / denom if denom else 0)
|
||||
scores[path] += weight * idf * (tf * (K1 + 1) / denom if denom else 0)
|
||||
fused = {p: s * self._doc_factor(p, today_s) for p, s in scores.items()}
|
||||
ranked = sorted(fused.items(), key=lambda kv: -kv[1])[:limit]
|
||||
return [(p, s) for p, s in ranked if s > MIN_SCORE]
|
||||
|
||||
# ---- serialization (compact; rebuildable, so the format can change freely) ----
|
||||
def to_json(self) -> dict:
|
||||
return {"schema": INDEX_SCHEMA, "n_docs": self.n_docs, "avg_len": self.avg_len,
|
||||
"length": self.length, "postings": self.postings, "meta": self.meta}
|
||||
return {"schema": INDEX_SCHEMA, "built": self.built, "n_docs": self.n_docs,
|
||||
"avg_len": self.avg_len, "length": self.length,
|
||||
"postings": self.postings, "meta": self.meta}
|
||||
|
||||
@classmethod
|
||||
def from_json(cls, d: dict) -> "Bm25Index":
|
||||
if d.get("schema") != INDEX_SCHEMA:
|
||||
return cls() # older/newer format -> empty; recall's cold path rebuilds
|
||||
# (a pre-stemming index MUST NOT be reused: its postings are unstemmed)
|
||||
ix = cls()
|
||||
ix.length = d.get("length", {})
|
||||
ix.postings = d.get("postings", {})
|
||||
ix.meta = d.get("meta", {})
|
||||
ix.n_docs = d.get("n_docs", len(ix.length))
|
||||
ix.avg_len = d.get("avg_len", 0.0)
|
||||
ix.built = d.get("built", "")
|
||||
ix.df = Counter({term: len(posting) for term, posting in ix.postings.items()})
|
||||
return ix
|
||||
|
||||
@@ -271,27 +299,84 @@ def _indexable(path: str) -> bool:
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- persistence
|
||||
def load_index() -> Bm25Index:
|
||||
status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH))
|
||||
if status == 404:
|
||||
return Bm25Index()
|
||||
echo.check(status, body, "recall-index load")
|
||||
# LOCAL-FIRST (2.3). The live index is a machine-local file — updating it on capture
|
||||
# is a zero-network atomic file write instead of GET-whole-index -> PUT-whole-index
|
||||
# under the vault lock (the old per-write O(vault) round trip). The vault copy is a
|
||||
# SNAPSHOT: written by sweep (and best-effort at session end), read only to seed a
|
||||
# fresh machine or to catch up after another client swept (checked at most every
|
||||
# ECHO_RECALL_SYNC_HOURS). Staleness across clients is tolerable — recall degrades
|
||||
# gracefully and the fast sweep trues it up.
|
||||
|
||||
def _local_path() -> Path:
|
||||
import hashlib
|
||||
import echo_queue
|
||||
h = hashlib.sha1((echo.BASE or "").encode()).hexdigest()[:12]
|
||||
return echo_queue.state_dir() / f"recall-index-{h}.json"
|
||||
|
||||
|
||||
def _load_local() -> Bm25Index | None:
|
||||
p = _local_path()
|
||||
if not p.exists():
|
||||
return None
|
||||
try:
|
||||
return Bm25Index.from_json(json.loads(p.read_text(encoding="utf-8")))
|
||||
except (json.JSONDecodeError, KeyError, TypeError, OSError):
|
||||
return None
|
||||
|
||||
|
||||
def save_local(ix: Bm25Index) -> None:
|
||||
p = _local_path()
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = p.with_suffix(".json.tmp")
|
||||
tmp.write_text(json.dumps(ix.to_json(), ensure_ascii=False), encoding="utf-8")
|
||||
os.replace(tmp, p) # atomic: a crash mid-write can't corrupt the live index
|
||||
|
||||
|
||||
def _fetch_snapshot() -> Bm25Index | None:
|
||||
try:
|
||||
status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH))
|
||||
if status != 200:
|
||||
return None
|
||||
return Bm25Index.from_json(json.loads(body))
|
||||
except (json.JSONDecodeError, KeyError, TypeError):
|
||||
return Bm25Index()
|
||||
except Exception: # noqa: BLE001 — the snapshot is an optimization, never required
|
||||
return None
|
||||
|
||||
|
||||
def save_index(ix: Bm25Index) -> None:
|
||||
def save_snapshot(ix: Bm25Index) -> None:
|
||||
"""PUT the vault snapshot — sweep + session-end only, never the per-capture path."""
|
||||
body = json.dumps(ix.to_json(), ensure_ascii=False).encode("utf-8")
|
||||
status, b = echo.request("PUT", echo.vault_url(RECALL_INDEX_PATH), data=body,
|
||||
headers={"Content-Type": "application/json"})
|
||||
echo.check(status, b, "recall-index save")
|
||||
echo.check(status, b, "recall-index snapshot save")
|
||||
|
||||
|
||||
def rebuild(prefetched: dict | None = None) -> Bm25Index:
|
||||
"""Full rebuild from every indexable entity note. Called by sweep.py and as the
|
||||
cold-cache path inside recall(). Saves and returns the index.
|
||||
def load_index() -> Bm25Index:
|
||||
"""The live index: local file first; the vault snapshot only seeds a fresh
|
||||
machine or replaces a local copy that a newer sweep elsewhere has superseded
|
||||
(checked when the local build stamp is older than SYNC_HOURS)."""
|
||||
local = _load_local()
|
||||
if local and local.n_docs:
|
||||
try:
|
||||
age_ok = local.built and (
|
||||
_dt.datetime.now(_dt.timezone.utc)
|
||||
- _dt.datetime.strptime(local.built, "%Y-%m-%dT%H:%M:%SZ").replace(
|
||||
tzinfo=_dt.timezone.utc)
|
||||
).total_seconds() < SYNC_HOURS * 3600
|
||||
except ValueError:
|
||||
age_ok = False
|
||||
if age_ok:
|
||||
return local
|
||||
snap = _fetch_snapshot()
|
||||
if snap and snap.n_docs and (not local or (snap.built or "") > (local.built or "")):
|
||||
save_local(snap)
|
||||
return snap
|
||||
return local or Bm25Index()
|
||||
|
||||
|
||||
def rebuild(prefetched: dict | None = None, snapshot: bool = False) -> Bm25Index:
|
||||
"""Full rebuild from every indexable note. Saves the LOCAL index always; writes
|
||||
the vault snapshot only when `snapshot=True` (sweep) — the read path never
|
||||
surprises the vault with a PUT.
|
||||
|
||||
`prefetched` (a {path: text} map, e.g. from echo.read_many) lets sweep.py reuse the
|
||||
content it already pulled instead of walking and re-fetching the whole vault again."""
|
||||
@@ -299,27 +384,30 @@ def rebuild(prefetched: dict | None = None) -> Bm25Index:
|
||||
if prefetched is not None:
|
||||
items = ((p, t) for p, t in prefetched.items() if _indexable(p))
|
||||
else:
|
||||
items = ((p, _get(p)) for p in _walk() if _indexable(p))
|
||||
items = echo.read_many([p for p in _walk() if _indexable(p)]).items()
|
||||
for path, text in items:
|
||||
if text is not None:
|
||||
ix.add(path, text) # add() strips frontmatter + extracts doc meta itself
|
||||
save_index(ix)
|
||||
ix.built = echo.now_iso()
|
||||
save_local(ix)
|
||||
if snapshot:
|
||||
save_snapshot(ix)
|
||||
return ix
|
||||
|
||||
|
||||
def update_note(path: str, text: str) -> None:
|
||||
"""Best-effort incremental upkeep for a single note (mirrors how entities.json is
|
||||
maintained on capture). Never raises into the caller. The load->save is wrapped in the
|
||||
advisory lock with a fresh re-read (H3), so concurrent captures can't clobber the
|
||||
recall index either."""
|
||||
"""Best-effort incremental upkeep for a single note — a LOCAL atomic file write,
|
||||
no vault round-trip, no advisory lock (2.3). Maintains an existing local index
|
||||
only: with no local index yet, the next recall's cold path (or a sweep) builds
|
||||
one, and a one-note index must never shadow the snapshot. Never raises."""
|
||||
try:
|
||||
if not _indexable(path):
|
||||
return # only entity notes are part of the recall corpus
|
||||
import echo_concurrency
|
||||
with echo_concurrency.vault_lock():
|
||||
ix = load_index() # fresh read inside the lock
|
||||
ix.add(path, text) # raw text: add() strips + extracts meta
|
||||
save_index(ix)
|
||||
return # only corpus notes are part of the recall index
|
||||
ix = _load_local()
|
||||
if ix is None or not ix.n_docs:
|
||||
return
|
||||
ix.add(path, text) # raw text: add() strips + extracts meta
|
||||
save_local(ix)
|
||||
except Exception as exc: # noqa: BLE001 — upkeep must never fail a capture
|
||||
print(f"echo_recall: index update skipped ({exc})", file=sys.stderr)
|
||||
|
||||
@@ -405,13 +493,27 @@ def recall_op(query, limit: int = 8) -> dict:
|
||||
index = idx_mod.load()
|
||||
nmap = idx_mod.name_map(index)
|
||||
|
||||
# --- alias query expansion (2.3): a query phrased by an entity's alias or a
|
||||
# partial name folds that entity's title/alias vocabulary into the BM25 query at
|
||||
# half weight — "the ECHO plugin" also scores `echo-memory` terms. Expansion can
|
||||
# only boost documents that actually contain the terms.
|
||||
extra_terms: dict[str, float] = {}
|
||||
q_toks = set(tokenize(q))
|
||||
for _slug, e, sc in idx_mod.fuzzy_candidates(index, q)[:2]:
|
||||
if sc < 0.5:
|
||||
continue
|
||||
for name in (e.get("title", ""), _slug, *e.get("aliases", [])):
|
||||
for tok in tokenize(str(name).replace("-", " ")):
|
||||
if tok not in q_toks:
|
||||
extra_terms[tok] = 0.5
|
||||
|
||||
# --- lexical layer (BM25); rebuild the cache on a cold miss --------------
|
||||
lexical: list[tuple[str, float]] = []
|
||||
try:
|
||||
ix = load_index()
|
||||
if ix.n_docs == 0:
|
||||
ix = rebuild()
|
||||
lexical = ix.score(q, limit=limit, today_s=today_s)
|
||||
lexical = ix.score(q, limit=limit, today_s=today_s, extra_terms=extra_terms)
|
||||
except echo.EchoError as exc:
|
||||
print(f"recall: BM25 index unavailable ({exc}); falling back to server search",
|
||||
file=sys.stderr)
|
||||
|
||||
@@ -135,6 +135,18 @@ def session_end_op(bundle: dict, apply: bool = False) -> dict:
|
||||
echo.cmd_put("_agent/heartbeat/last-session.md",
|
||||
echo.temp_file(f"{path} @ {echo.now_iso()}\n".encode("utf-8")))
|
||||
steps["heartbeat"] = "ok"
|
||||
# Index maintenance (2.3), NOT part of the bundle contract: push this
|
||||
# machine's local recall index to the vault snapshot so other machines
|
||||
# can seed/catch up. Best-effort — a failure never taints the bundle.
|
||||
try:
|
||||
import echo_recall
|
||||
rix = echo_recall._load_local()
|
||||
if rix and rix.n_docs:
|
||||
rix.built = rix.built or echo.now_iso()
|
||||
echo_recall.save_snapshot(rix)
|
||||
steps["recall_snapshot"] = "ok"
|
||||
except Exception: # noqa: BLE001
|
||||
steps["recall_snapshot"] = "skipped"
|
||||
|
||||
data["steps"] = steps
|
||||
return echo_output.envelope("session-end", data)
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env python3
|
||||
"""echo_stem.py — a deliberately light, dependency-free English stemmer. [2.3]
|
||||
|
||||
BM25 is purely lexical: "deploy", "deployed", and "deployment" are three unrelated
|
||||
terms to it, so a paraphrased recall query misses notes stored with a different
|
||||
inflection. This Porter-LITE stemmer conflates the common inflection families while
|
||||
staying conservative — when in doubt it leaves the token alone, because an
|
||||
over-stemmed index silently merges unrelated words (far worse than a missed match).
|
||||
|
||||
Applied by echo_recall.tokenize() at BOTH index and query time (recall-index
|
||||
schema 3; older indexes are discarded and rebuilt). Pure function, unit-tested in
|
||||
test_echo_client.py.
|
||||
|
||||
Design notes:
|
||||
* plural / -ed / -ing families first (with the classic restore rules: doubled
|
||||
consonant undoubling, at/bl/iz +e, cvc +e), then one derivational suffix pass,
|
||||
then y->i and final-e normalization so "navigate"/"navigation"/"navigating"
|
||||
all land on "navigat".
|
||||
* every rule guards a minimum remaining stem length — "sing" is not "s".
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
_VOWEL = re.compile(r"[aeiouy]")
|
||||
_DOUBLE = re.compile(r"([^aeiouylsz])\1$")
|
||||
_CVC = re.compile(r"[^aeiou][aeiouy][^aeiouwxy]$")
|
||||
|
||||
# One derivational pass, most-specific first. Replacements keep families together:
|
||||
# navigational/navigation -> navigat · saturation/saturated -> saturat ·
|
||||
# deployment/deployed -> deploy · usefulness/useful -> use... (guarded by length).
|
||||
_SUFFIXES = (
|
||||
("ization", "ize"), ("ational", "ate"), ("fulness", "ful"), ("ousness", "ous"),
|
||||
("iveness", "ive"), ("tional", "t"), ("biliti", "ble"), ("ements", ""),
|
||||
("ment", ""), ("ness", ""), ("tion", "t"), ("sion", "s"), ("ance", ""),
|
||||
("ence", ""), ("able", ""), ("ible", ""), ("ally", "al"), ("ously", "ous"),
|
||||
("ively", "ive"), ("ly", ""),
|
||||
)
|
||||
|
||||
|
||||
def stem(t: str) -> str:
|
||||
"""Stem one lowercase token. Never raises; returns the token unchanged when no
|
||||
rule applies safely."""
|
||||
if len(t) <= 3 or not t.isascii():
|
||||
return t
|
||||
# --- plurals -------------------------------------------------------------
|
||||
if t.endswith("sses"):
|
||||
t = t[:-2]
|
||||
elif t.endswith("ies") and len(t) > 4:
|
||||
t = t[:-3] + "i"
|
||||
elif t.endswith("s") and not t.endswith(("ss", "us", "is")):
|
||||
t = t[:-1]
|
||||
# --- -ed / -ing (with restore rules) --------------------------------------
|
||||
for suf in ("ingly", "edly", "ing", "ed"):
|
||||
if t.endswith(suf):
|
||||
base = t[: -len(suf)]
|
||||
if len(base) >= 3 and _VOWEL.search(base):
|
||||
t = base
|
||||
if t.endswith(("at", "bl", "iz")):
|
||||
t += "e"
|
||||
elif _DOUBLE.search(t):
|
||||
t = t[:-1]
|
||||
elif len(t) == 3 and _CVC.search(t):
|
||||
t += "e"
|
||||
break
|
||||
# --- one derivational suffix pass -----------------------------------------
|
||||
for suf, rep in _SUFFIXES:
|
||||
if t.endswith(suf):
|
||||
base = t[: -len(suf)]
|
||||
if len(base) >= 3 and len(base + rep) >= 3:
|
||||
t = base + rep
|
||||
break
|
||||
# --- normalize: y->i (so penalty/penalties agree), then drop a final e -----
|
||||
if len(t) > 3 and t.endswith("y") and t[-2] not in "aeiou":
|
||||
t = t[:-1] + "i"
|
||||
if len(t) > 4 and t.endswith("e"):
|
||||
t = t[:-1]
|
||||
return t
|
||||
@@ -18,9 +18,19 @@ backfill what older vaults don't have yet:
|
||||
5. stamp the marker `schema_version` to the current schema (4).
|
||||
|
||||
Dry-run by default; pass --apply to write. READ-ONLY without --apply.
|
||||
|
||||
**--fast (2.3)** is the incremental mode: it walks the listing (cheap), then fetches
|
||||
only what's new, gone, missing a content hash, or in today's rotating verification
|
||||
shard (hash(path) % 7 == weekday — the whole vault gets verified over a week of
|
||||
fast sweeps while each run stays tiny; --all-shards forces everything). It updates
|
||||
the LOCAL recall index and the entity index (both machine-owned), so it needs no
|
||||
--apply gate and writes no notes. `load` auto-runs it when the last fast sweep is
|
||||
older than ECHO_FAST_SWEEP_DAYS (default 7). This is what keeps the indexes honest
|
||||
against edits made directly in Obsidian or by other clients.
|
||||
|
||||
Cross-platform: pure Python via echo.py.
|
||||
|
||||
Usage: sweep.py [--apply]
|
||||
Usage: sweep.py [--apply] | sweep.py --fast [--all-shards]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -102,10 +112,128 @@ def walk(prefix: str = ""):
|
||||
yield from walk(f"{prefix}{d}/")
|
||||
|
||||
|
||||
def _stamp_path() -> Path:
|
||||
import echo_queue
|
||||
return echo_queue.state_dir() / "fast-sweep.stamp"
|
||||
|
||||
|
||||
def _stamp_fast_sweep() -> None:
|
||||
try:
|
||||
p = _stamp_path()
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(echo.now_iso() + "\n", encoding="utf-8")
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def fast_sweep_age_days() -> float | None:
|
||||
"""Days since the last fast/full sweep on THIS machine, or None if never."""
|
||||
import datetime as dt
|
||||
try:
|
||||
stamp = _stamp_path().read_text(encoding="utf-8").strip()
|
||||
then = dt.datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=dt.timezone.utc)
|
||||
return (dt.datetime.now(dt.timezone.utc) - then).total_seconds() / 86400
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def fast(all_shards: bool = False) -> str:
|
||||
"""Incremental index maintenance (2.3): true up the LOCAL recall index and the
|
||||
entity index against the vault, fetching only new / gone / hash-missing notes
|
||||
plus today's rotating verification shard. Index-only — writes no notes; the
|
||||
indexes are machine-owned, so no --apply gate. Returns a one-line summary."""
|
||||
import datetime as dt
|
||||
import echo_recall
|
||||
|
||||
if get("_agent/echo-vault.md") is None:
|
||||
return "fast-sweep skipped: vault not bootstrapped"
|
||||
|
||||
listing = [p for p in walk()
|
||||
if p.endswith(".md") and p.rsplit("/", 1)[-1] not in SKIP_BASENAMES
|
||||
and not TEMPLATE_RE.search(p)]
|
||||
lset = set(listing)
|
||||
|
||||
rix = echo_recall._load_local()
|
||||
if rix is None or not rix.n_docs:
|
||||
# Nothing to maintain incrementally — build the local index outright
|
||||
# (sub-second on a few-hundred-note vault; no vault snapshot write).
|
||||
rix = echo_recall.rebuild()
|
||||
_stamp_fast_sweep()
|
||||
return f"fast-sweep: no local index — built one ({rix.n_docs} docs)"
|
||||
|
||||
index = idx_mod.load()
|
||||
ents = index.get("entities", {})
|
||||
removed = 0
|
||||
for slug in [s for s, e in list(ents.items()) if e.get("path") not in lset]:
|
||||
del ents[slug]
|
||||
removed += 1
|
||||
for p in [p for p in list(rix.length) if p not in lset]:
|
||||
rix.remove(p)
|
||||
removed += 1
|
||||
|
||||
try:
|
||||
weekday = dt.date.fromisoformat(echo.today()).weekday()
|
||||
except ValueError:
|
||||
weekday = dt.date.today().weekday()
|
||||
|
||||
def in_shard(p: str) -> bool:
|
||||
return int(idx_mod.content_hash(p), 16) % 7 == weekday
|
||||
|
||||
cands: set[str] = set()
|
||||
for p in listing:
|
||||
if not echo_recall._indexable(p):
|
||||
continue
|
||||
if p not in rix.length: # new note (any client, any tool)
|
||||
cands.add(p)
|
||||
continue
|
||||
kind = kind_for(p)
|
||||
if kind:
|
||||
slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3])
|
||||
if not (ents.get(slug) or {}).get("h"): # pre-2.3 entry: hash it once
|
||||
cands.add(p)
|
||||
continue
|
||||
if all_shards or in_shard(p): # rotating verification shard
|
||||
cands.add(p)
|
||||
|
||||
texts = echo.read_many(sorted(cands))
|
||||
changed = 0
|
||||
ents_dirty = removed > 0
|
||||
for p, t in texts.items():
|
||||
if t is None:
|
||||
continue
|
||||
h = idx_mod.content_hash(t)
|
||||
if rix.content_hash(p) != h:
|
||||
rix.add(p, t)
|
||||
changed += 1
|
||||
kind = kind_for(p)
|
||||
if kind:
|
||||
slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3])
|
||||
prev = ents.get(slug) or {}
|
||||
if prev.get("h") != h or prev.get("path") != p:
|
||||
title = links.first_h1(t) or p.rsplit("/", 1)[-1][:-3]
|
||||
aliases = list(prev.get("aliases", [])) + idx_mod.aliases_in_frontmatter(t)
|
||||
idx_mod.upsert(index, slug, p, kind, title, aliases, h=h)
|
||||
ents_dirty = True
|
||||
|
||||
echo_recall.save_local(rix)
|
||||
if ents_dirty:
|
||||
idx_mod.save(index)
|
||||
_stamp_fast_sweep()
|
||||
return (f"fast-sweep: {len(cands)} note(s) checked, {changed} reindexed, "
|
||||
f"{removed} removed" + (" (all shards)" if all_shards else ""))
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
ap = argparse.ArgumentParser(description="Bring an ECHO vault up to the current feature spec")
|
||||
ap.add_argument("--apply", action="store_true")
|
||||
ap.add_argument("--fast", action="store_true",
|
||||
help="incremental index maintenance (local recall + entity index only)")
|
||||
ap.add_argument("--all-shards", action="store_true",
|
||||
help="with --fast: verify every indexed note, not just today's shard")
|
||||
args = ap.parse_args(argv)
|
||||
if args.fast:
|
||||
print(fast(all_shards=args.all_shards))
|
||||
return 0
|
||||
apply = args.apply
|
||||
tag = "APPLY" if apply else "PLAN "
|
||||
|
||||
@@ -151,7 +279,8 @@ def main(argv: list[str] | None = None) -> int:
|
||||
# aliases (e.g. mentions learned by capture) are preserved; upsert adds title variants.
|
||||
prev_aliases = prev.get("aliases", []) if prev else []
|
||||
aliases = list(prev_aliases) + idx_mod.aliases_in_frontmatter(text)
|
||||
idx_mod.upsert(rebuilt, slug, path, kind, title, aliases)
|
||||
idx_mod.upsert(rebuilt, slug, path, kind, title, aliases,
|
||||
h=idx_mod.content_hash(text))
|
||||
if not prev:
|
||||
added += 1
|
||||
elif prev.get("path") != path or prev.get("title") != title:
|
||||
@@ -163,8 +292,9 @@ def main(argv: list[str] | None = None) -> int:
|
||||
|
||||
# ---- 2b. (re)build the recall (BM25) index -------------------------------
|
||||
if apply:
|
||||
rix = echo_recall.rebuild(prefetched=texts)
|
||||
print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25)")
|
||||
rix = echo_recall.rebuild(prefetched=texts, snapshot=True)
|
||||
_stamp_fast_sweep()
|
||||
print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25; local + vault snapshot)")
|
||||
else:
|
||||
n_idx = sum(1 for p in all_files if echo_recall._indexable(p))
|
||||
print(f"sweep: {tag} recall index — would rebuild BM25 over {n_idx} corpus notes "
|
||||
|
||||
@@ -215,6 +215,42 @@ def test_links_create_related_section_when_absent() -> None:
|
||||
assert changed and "## Related" in new and "[[resources/concepts/beta]]" in new
|
||||
|
||||
|
||||
def test_stemmer_conflates_inflection_families() -> None:
|
||||
import echo_stem
|
||||
families = [
|
||||
["deploy", "deploys", "deployed", "deploying", "deployment"],
|
||||
["navigate", "navigation", "navigating", "navigational"],
|
||||
["renew", "renews", "renewing"],
|
||||
["prefer", "prefers", "preferring", "preferred"],
|
||||
["penalty", "penalties"],
|
||||
["saturate", "saturation", "saturated"],
|
||||
["frequency", "frequencies"],
|
||||
["expand", "expanded", "expanding"],
|
||||
["meet", "meeting", "meetings"],
|
||||
["service", "services"],
|
||||
]
|
||||
for fam in families:
|
||||
stems = {echo_stem.stem(w) for w in fam}
|
||||
assert len(stems) == 1, f"{fam} -> {stems}"
|
||||
|
||||
|
||||
def test_stemmer_leaves_short_and_risky_tokens_alone() -> None:
|
||||
import echo_stem
|
||||
# over-stemming merges unrelated words — worse than a missed match
|
||||
for w in ("sing", "thing", "was", "is", "bus", "this", "echo", "vault", "mpm"):
|
||||
assert echo_stem.stem(w) == w, f"{w} mangled to {echo_stem.stem(w)}"
|
||||
# documented non-conflations (the -al / d~s irregulars full Porter also skips)
|
||||
assert echo_stem.stem("renewal") != echo_stem.stem("renew")
|
||||
assert echo_stem.stem("expansion") != echo_stem.stem("expand")
|
||||
|
||||
|
||||
def test_content_hash_stable_and_short() -> None:
|
||||
import echo_index as ix
|
||||
a = ix.content_hash("hello world")
|
||||
assert a == ix.content_hash("hello world") and len(a) == 16
|
||||
assert a != ix.content_hash("hello world!")
|
||||
|
||||
|
||||
def _run_all() -> int:
|
||||
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
|
||||
failed = 0
|
||||
|
||||
@@ -3,17 +3,17 @@
|
||||
"date": "2026-07-03",
|
||||
"retrieval": {
|
||||
"current (hybrid+priors, full corpus)": {
|
||||
"precision_at_5": 0.25,
|
||||
"precision_at_5": 0.243,
|
||||
"recall_at_5": 1.0,
|
||||
"mrr": 1.0,
|
||||
"session_journal_queries_answered": "2/2",
|
||||
"session_journal_queries_answered": "3/3",
|
||||
"freshness_top1_correct": true
|
||||
},
|
||||
"baseline (keyword, entities only)": {
|
||||
"precision_at_5": 0.2,
|
||||
"precision_at_5": 0.186,
|
||||
"recall_at_5": 0.75,
|
||||
"mrr": 0.75,
|
||||
"session_journal_queries_answered": "0/2",
|
||||
"mrr": 0.786,
|
||||
"session_journal_queries_answered": "0/3",
|
||||
"freshness_top1_correct": false
|
||||
}
|
||||
},
|
||||
|
||||
@@ -135,6 +135,16 @@ QUERIES = [
|
||||
("contract renewal Q3", {"resources/companies/vantage-systems.md"}, False),
|
||||
("renegotiation kickoff", {"journal/daily/2026-06-20.md"}, True),
|
||||
("graph neighbourhood expansion", {"resources/concepts/graph-expansion.md"}, False),
|
||||
# ---- paraphrase set (2.3): morphological variants + rephrasings the pre-2.3
|
||||
# purely-lexical index missed — the stemming gain is measured, not assumed.
|
||||
("renewing enterprise contracts", {"resources/companies/vantage-systems.md"}, False),
|
||||
("navigational cleanup", {"projects/active/website-redesign.md",
|
||||
"projects/archived/old-website.md"}, False),
|
||||
("preferring uv for tooling", {"_agent/memory/semantic/tooling-preferences.md"}, False),
|
||||
("penalties in the SLA",
|
||||
{"_agent/sessions/2026-06-20-1000-vantage-contract-review.md"}, True),
|
||||
("saturated term frequencies", {"resources/concepts/bm25-ranking.md"}, False),
|
||||
("expanded linked neighbourhoods", {"resources/concepts/graph-expansion.md"}, False),
|
||||
]
|
||||
FRESHNESS_QUERY = "marketing site navigation"
|
||||
FRESHNESS_TOP1 = "projects/active/website-redesign.md"
|
||||
|
||||
+61
-3
@@ -27,7 +27,9 @@ def check(name, cond, detail=""):
|
||||
|
||||
class Harness:
|
||||
def __init__(self, base):
|
||||
import tempfile
|
||||
self.base = base
|
||||
self.state_dir = tempfile.mkdtemp()
|
||||
|
||||
def http(self, method, url, body=None, headers=None):
|
||||
data = body.encode() if isinstance(body, str) else body
|
||||
@@ -52,8 +54,10 @@ class Harness:
|
||||
return None if body == "<<MISSING>>" else body
|
||||
|
||||
def echo(self, *args):
|
||||
# ECHO_STATE_DIR isolation matters since 2.3: the recall index is LOCAL-first,
|
||||
# and without this the suite would write into the real ~/.echo-memory.
|
||||
env = dict(os.environ, ECHO_BASE=self.base, ECHO_KEY=KEY, ECHO_VERIFY="1",
|
||||
ECHO_TODAY="2026-06-21")
|
||||
ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=self.state_dir)
|
||||
return subprocess.run([sys.executable, str(args[0]), *args[1:]],
|
||||
capture_output=True, text=True, env=env)
|
||||
|
||||
@@ -230,8 +234,9 @@ def main():
|
||||
check("v1.5 recall finds a session log by body term",
|
||||
"_agent/sessions/2026-06-15-1200-mpm-review.md" in r.stdout, r.stdout)
|
||||
rix2 = h.ground("_agent/index/recall-index.json") or ""
|
||||
check("v1.5 recall index carries doc meta (schema 2)",
|
||||
'"schema": 2' in rix2 or '"schema":2' in rix2, rix2[:200])
|
||||
check("v2.3 recall snapshot is schema 3 with build stamp + content hashes",
|
||||
('"schema": 3' in rix2 or '"schema":3' in rix2)
|
||||
and '"built"' in rix2 and '"h"' in rix2, rix2[:200])
|
||||
|
||||
# 14. recency-aware ranking: same term, fresher note wins.
|
||||
h.seed("resources/concepts/old-idea.md",
|
||||
@@ -402,6 +407,59 @@ def main():
|
||||
r.returncode == 2 and sess_path in (h.ground("_agent/heartbeat/last-session.md") or ""),
|
||||
r.stderr)
|
||||
|
||||
# ---------------- v2.3 index train --------------------------------------
|
||||
# 20. local-first recall index: capture updates the LOCAL index with ZERO
|
||||
# vault snapshot writes — the vault copy only changes on sweep/session-end.
|
||||
snap_before = h.ground("_agent/index/recall-index.json") or ""
|
||||
r = h.echo(ECHO, "capture", "Quixotic Widget", "--kind", "concept", "-")
|
||||
r = subprocess.run([sys.executable, str(ECHO), "capture", "Quixotic Widget2", "--kind",
|
||||
"concept", "-"], input="A gadget for quixotry deployment.\n",
|
||||
capture_output=True, text=True,
|
||||
env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY,
|
||||
ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=h.state_dir))
|
||||
snap_after = h.ground("_agent/index/recall-index.json") or ""
|
||||
check("v2.3 capture does NOT write the vault snapshot", snap_before == snap_after)
|
||||
local_files = [p for p in Path(h.state_dir).glob("recall-index-*.json")]
|
||||
check("v2.3 capture maintains the LOCAL index", len(local_files) == 1,
|
||||
str(local_files))
|
||||
r = h.echo(ECHO, "recall", "quixotry")
|
||||
check("v2.3 locally-indexed capture is recallable",
|
||||
"resources/concepts/quixotic-widget2.md" in r.stdout, r.stdout[:400])
|
||||
|
||||
# 21. stemming: a morphologically different query still hits (deployment ~ deploying).
|
||||
r = h.echo(ECHO, "recall", "deploying quixotry")
|
||||
check("v2.3 stemmed recall bridges inflections",
|
||||
"resources/concepts/quixotic-widget2.md" in r.stdout, r.stdout[:400])
|
||||
|
||||
# 22. alias query expansion: an alias-phrased query surfaces the entity even
|
||||
# though the alias never appears in any note body.
|
||||
r = subprocess.run([sys.executable, str(ECHO), "capture", "Kappa Engine", "--kind",
|
||||
"concept", "-", "--aliases", "turbokappa"],
|
||||
input="Compression tuning notes for the kappa engine core.\n",
|
||||
capture_output=True, text=True,
|
||||
env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY,
|
||||
ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=h.state_dir))
|
||||
r = h.echo(ECHO, "recall", "turbokappa compression")
|
||||
check("v2.3 alias expansion surfaces the aliased entity",
|
||||
"resources/concepts/kappa-engine.md" in r.stdout, r.stdout[:400])
|
||||
|
||||
# 23. incremental fast sweep: an out-of-band edit (seeded directly, as if by
|
||||
# Obsidian) and a deletion are trued up without a full rebuild.
|
||||
h.seed("resources/concepts/quixotic-widget.md",
|
||||
"---\ntype: concept\nstatus: active\ncreated: 2026-06-21\nupdated: 2026-06-21\n"
|
||||
"tags: [concept]\n---\n# Quixotic Widget\n\nNow mentions zanzibar explicitly.\n")
|
||||
h.http("DELETE", f"{base}/vault/resources/concepts/quixotic-widget2.md")
|
||||
r = h.echo(SWEEP, "--fast", "--all-shards")
|
||||
check("v2.3 fast sweep reports its work",
|
||||
"fast-sweep:" in r.stdout and "reindexed" in r.stdout, r.stdout + r.stderr)
|
||||
r = h.echo(ECHO, "recall", "zanzibar")
|
||||
check("v2.3 fast sweep picked up the out-of-band edit",
|
||||
"resources/concepts/quixotic-widget.md" in r.stdout, r.stdout[:400])
|
||||
idx5 = h.ground("_agent/index/entities.json") or ""
|
||||
check("v2.3 fast sweep drops deleted notes from the entity index",
|
||||
"quixotic-widget2" not in idx5)
|
||||
check("v2.3 entity index carries content hashes", '"h"' in idx5, idx5[:300])
|
||||
|
||||
print(f"\n{len(failures)} failure(s)" if failures else "\nall feature tests passed")
|
||||
return 1 if failures else 0
|
||||
finally:
|
||||
|
||||
@@ -59,7 +59,9 @@ def main():
|
||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
||||
|
||||
def echo(*args):
|
||||
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="0")
|
||||
import tempfile
|
||||
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="0",
|
||||
ECHO_STATE_DIR=globals().setdefault("_STATE_DIR", tempfile.mkdtemp()))
|
||||
return subprocess.run([sys.executable, str(ECHO), *args], capture_output=True, text=True, env=env)
|
||||
|
||||
def ground(path):
|
||||
|
||||
@@ -51,7 +51,9 @@ def main():
|
||||
return getattr(e, "code", 0), ""
|
||||
|
||||
def echo(*args, stdin=None):
|
||||
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1", ECHO_TODAY="2026-06-22")
|
||||
import tempfile
|
||||
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1",
|
||||
ECHO_TODAY="2026-06-22", ECHO_STATE_DIR=tempfile.mkdtemp())
|
||||
return subprocess.run([sys.executable, str(ECHO), *args], input=stdin,
|
||||
capture_output=True, text=True, env=env)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user