diff --git a/CHANGELOG.md b/CHANGELOG.md index f7a0295..245f274 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,57 @@ # Changelog +## 2.3.0 + +**The index train** (IMPROVEMENT-PLANS #2/#4/#8) — one schema-bump release: +recall-index schema 3, entity-index schema 2. Old recall indexes are discarded +and rebuilt automatically (sub-second); no vault migration. + +### Changed — the recall index is LOCAL-FIRST + +The live BM25 index moves out of the vault into the machine state dir +(`recall-index-.json` under `ECHO_STATE_DIR`). Capture's index +upkeep — previously GET-whole-index → PUT-whole-index against the vault under +the advisory lock, an O(vault) network round trip per write — is now a +zero-network atomic file write with no lock at all. The vault copy at +`_agent/index/recall-index.json` becomes a **snapshot**: written by +`sweep --apply` and best-effort at `session-end`, read only to seed a fresh +machine or to catch up after another client swept (checked at most every +`ECHO_RECALL_SYNC_HOURS`, default 24). The read path never PUTs the vault. + +### Added — incremental sweep (`sweep --fast`) + content hashes + +Entity-index entries and recall-index doc meta now carry a 16-char content +hash. `sweep --fast` walks the listing (cheap) and fetches only what is new, +gone, missing a hash, or in **today's rotating verification shard** +(`hash(path) % 7 == weekday` — the whole vault gets verified across a week of +fast sweeps while each run stays tiny; `--all-shards` forces everything). +Index-only and machine-owned, so no `--apply` gate; deletions are dropped from +both indexes. **`load` auto-runs it** when this machine's last sweep is older +than `ECHO_FAST_SWEEP_DAYS` (default 7) — edits made directly in Obsidian or by +other clients now reach the indexes without anyone remembering maintenance. +`doctor` reports index freshness. + +### Added — stemming + alias query expansion + +`echo_stem.py`: a conservative Porter-lite stemmer applied by `tokenize()` at +BOTH index and query time, so "deployed"/"deployment"/"deploys" meet at +"deploy" and "penalties" finds "penalty" (guards against over-stemming: "sing" +is not "s"; the -al/d~s irregulars are deliberately not conflated). At recall +time, a query that fuzzy-matches an entity (score ≥ 0.5) folds that entity's +title/alias vocabulary into the BM25 query at **half weight** — "turbokappa +tuning" finds the Kappa Engine note even though the alias appears in no body. +Expansion can only boost documents that actually contain the terms. + +### Eval + +Gold set grows 8 → **14 queries** (6 paraphrases the pre-2.3 lexical index +missed). v2.3.0: recall@5 / MRR **1.00 / 1.00**, 3/3 session-journal queries; +keyword baseline 0.75 / 0.79 and 0/3. Tests: +3 unit (stemmer families, +over-stemming guards, hash), +10 end-to-end (local-first no-snapshot-write, +local recallability, stemmed recall, alias expansion, fast-sweep pickup/ +deletion/hash backfill). All suites — including the offline-queue, ops-api, +and MCP server suites — green; test harnesses now isolate `ECHO_STATE_DIR`. + ## 2.2.0 ### Added — echo-mcp: the containerized MCP server diff --git a/README.md b/README.md index 4600b50..9ef4564 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# echo-memory — v2.2.0 +# echo-memory — v2.3.0 Persistent memory for Claude / CoWork sessions via the **ECHO** Obsidian vault, driven over the [Obsidian Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api). The skill makes direct REST calls through a bundled validated client (`scripts/echo.py`). The whole toolchain is **pure Python** (stdlib only), so it runs identically on Windows, macOS, and Linux — no bash, no platform-specific `date`. @@ -403,14 +403,14 @@ If the API returns a connection error, timeout, or `502` (usually Obsidian / the --- -## Eval metrics (v1.5.1, 2026-07-03) +## Eval metrics (v2.3.0, 2026-07-29) -From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate. +From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate. The gold set grew to **14 queries in 2.3** — six are paraphrases (morphological variants like "penalties in the SLA" vs a note saying "penalty clause") that a purely lexical index misses; the stemmed index answers all of them. -| Metric | v1.5.1 | pre-1.5 baseline | +| Metric | v2.3.0 | pre-1.5 baseline | |---|---|---| -| Retrieval recall@5 / MRR (8-query gold set) | **1.00 / 1.00** | 0.75 / 0.75 | -| Queries answerable only from sessions/journal | **2/2** | 0/2 (not in corpus) | +| Retrieval recall@5 / MRR (14-query gold set incl. 6 paraphrases) | **1.00 / 1.00** | 0.75 / 0.79 | +| Queries answerable only from sessions/journal | **3/3** | 0/3 (not in corpus) | | Freshness: live note outranks stale archived twin | **yes** | no | | Duplicate notes created (3 renamed-entity captures) | **0** (gate blocks) | 3 | | Legitimate captures wrongly blocked | **0** | 0 | @@ -421,6 +421,7 @@ From the credential-free harness (`eval/run_eval.py` against the deterministic m | Version | Highlights | |---------|-----------| +| **2.3.0** | **The index train (recall-index schema 3, entity-index schema 2).** (1) **Local-first recall index** — the live BM25 index moves to the machine state dir (keyed by endpoint); capture's index upkeep becomes a zero-network atomic file write instead of GET/PUT-whole-index under the vault lock; the vault copy is now a snapshot (written by sweep + session-end) that seeds fresh machines. (2) **Incremental sweep** — entity entries carry a content hash; `sweep --fast` fetches only new/gone/changed notes plus a rotating weekday shard (whole vault verified across a week); auto-runs at load when >`ECHO_FAST_SWEEP_DAYS` (7) — Obsidian-side edits now reach the indexes without manual maintenance. (3) **Stemming + alias expansion** — `echo_stem` (Porter-lite, conservative) applied at index+query time, and a query matching an entity's alias folds its title/alias vocabulary in at half weight. Eval gold set grows to 14 queries (6 paraphrases): recall@5 / MRR **1.00 / 1.00**. | | **2.2.0** | **echo-mcp — the containerized MCP server.** ECHO as 14 typed MCP tools from any connector-capable surface (Claude Code, CoWork, claude.ai): `mcp-server/` + Dockerfile in-repo, deployed on ALPHA as `echo-mcp` behind `https://echomcp.alwisp.com` (streamable HTTP, stateless JSON, bearer auth, open `/health`). Talks to the Obsidian REST API directly on the LAN (`10.2.0.35:27123`) — one client round trip per tool call, all vault chatter host-local. Duplicate gate & offline queueing surface as data; `ECHO_MCP_TOOLS=core` trims the surface to six tools; writes serialize server-side. New `eval/test_mcp_server.py` e2e suite. Full spec: `docs/MCP-SERVER-SPEC.md`. | | **2.1.1** | **MCP Phase 0 — return-not-print.** Every high-level op gains a core `*_op` function returning an envelope dict with a stdout-purity guarantee (helper chatter → stderr); CLI verbs become thin wrappers with identical output and exit codes. Duplicate gate and offline queueing become data (`action: duplicate-gate` / `queued: true`). New `eval/test_ops_api.py` contract suite. This is the seam the 2.2 containerized MCP server wraps. | | **2.1.0** | **Quick-wins train: cheaper sessions, durable capture, one-call session end.** (1) **`load --brief`** — a token-budgeted cold-start digest (Fact/Pattern in full, last ~10 Observations, scope+freshness, last session's key sections, Agent-Log lines, inbox *count*; `ECHO_LOAD_BUDGET` default ~8000 chars) now injected by the SessionStart hook, replacing the full six-file dump that grew unboundedly; all `load` reads are fetched in parallel. (2) **Offline capture durability** — `capture` on an unreachable vault queues the *whole operation* as one semantic record; `flush` replays it **through capture** so routing/gate/aliasing re-run against the current index; a gate stop on replay is kept + flagged, never landed blind; `ensure_daily_log`/update-path writes ride the queue too. (3) **`session-end`** — one call (one lock) writes session log → Agent-Log line → reflect proposals → optional scope switch → **heartbeat last as the commit marker**; dry-run by default; `ECHO_NOW` pins the HHMM. Also fixes the `__main__` twin-module trap so helper-module `EchoError`s exit with their intended codes. +19 end-to-end checks. | diff --git a/echo-memory.plugin b/echo-memory.plugin index 0abcaaa..fe15c35 100644 Binary files a/echo-memory.plugin and b/echo-memory.plugin differ diff --git a/echo-memory.plugin.src/.claude-plugin/plugin.json b/echo-memory.plugin.src/.claude-plugin/plugin.json index b84dafc..8455ef9 100644 --- a/echo-memory.plugin.src/.claude-plugin/plugin.json +++ b/echo-memory.plugin.src/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "echo-memory", - "version": "2.2.0", + "version": "2.3.0", "homepage": "https://git.alwisp.com/jason/echo", "repository": "https://git.alwisp.com/jason/echo", "description": "Persistent memory via the ECHO Obsidian vault over the Obsidian Local REST API. Cross-platform Python client: one-call capture/resolve/recall/link/triage over an entity index, hybrid BM25 + graph recall spanning entities + sessions/journal (recency/status-aware), a pre-write duplicate gate, complete-frontmatter capture, session hooks that self-fire load/reflect, offline write-ahead queue, lock-guarded concurrency, linter-enforced routing, and /echo-* commands.", diff --git a/echo-memory.plugin.src/skills/echo-memory/SKILL.md b/echo-memory.plugin.src/skills/echo-memory/SKILL.md index 1f95bd9..b322150 100644 --- a/echo-memory.plugin.src/skills/echo-memory/SKILL.md +++ b/echo-memory.plugin.src/skills/echo-memory/SKILL.md @@ -358,6 +358,8 @@ Run it after `migrate.py` (which handles structural/schema changes) — or any t The pass is cheap and pays for itself by catching drift before it requires a reorg. Write the findings as a digest; act on them only with the operator's go-ahead. +**Incremental maintenance (2.3).** `sweep.py --fast` is the cheap, index-only mode: it fetches just what's new, gone, or changed (content hashes) plus a rotating daily verification shard, so edits made directly in Obsidian or by other clients reach the entity + recall indexes without a full pass. It runs **automatically at load** when this machine's last sweep is older than `ECHO_FAST_SWEEP_DAYS` (default 7) — you rarely need to invoke it. The recall index itself is **local-first**: capture maintains a machine-local copy with zero vault round-trips; the vault's `recall-index.json` is a snapshot (sweep/session-end) used to seed fresh machines. + **Performance.** Full-vault scripts (`sweep.py`, `vault_lint.py`, recall rebuild) are connection-pooled (keep-alive) and read the whole vault concurrently via `echo.read_many`, so they finish in about a second on a few-hundred-note vault instead of timing out on hundreds of serial TLS handshakes. Tune with `ECHO_WORKERS` (default 8, bulk-read concurrency) and `ECHO_TIMEOUT` (default 30s, per-request). For a vault large enough that even the concurrent pass nears the tool timeout, run the script with the harness's background-execution option rather than blocking the turn. ## Daily Note — Agent Log diff --git a/echo-memory.plugin.src/skills/echo-memory/scripts/echo.py b/echo-memory.plugin.src/skills/echo-memory/scripts/echo.py index 1898dd0..9d1e47f 100644 --- a/echo-memory.plugin.src/skills/echo-memory/scripts/echo.py +++ b/echo-memory.plugin.src/skills/echo-memory/scripts/echo.py @@ -790,6 +790,24 @@ def _load_gather(targets): return results, texts, listing_files, offline, marker_missing, heartbeat_absent +def _auto_fast_sweep(offline: bool) -> str | None: + """Opportunistic index maintenance at load (2.3): when this machine's last fast + sweep is older than ECHO_FAST_SWEEP_DAYS (default 7), run `sweep --fast` so the + local recall + entity indexes true up against edits made by Obsidian or other + clients. Best-effort; skipped offline; never raises.""" + if offline: + return None + try: + import sweep as sweep_mod + days = float(os.environ.get("ECHO_FAST_SWEEP_DAYS", "7")) + age = sweep_mod.fast_sweep_age_days() + if age is not None and age < days: + return None + return sweep_mod.fast() + except Exception: # noqa: BLE001 — maintenance must never block a load + return None + + def _fetch_pointed_session(texts: dict) -> None: """Follow the heartbeat pointer and stash the pointed session log for the digest.""" hb = texts.get("heartbeat") @@ -818,9 +836,10 @@ def load_op(brief: bool = True) -> dict: targets = LOAD_TARGETS + [("today", f"journal/daily/{today()}.md"), ("inbox", "inbox/captures/inbox.md")] _, texts, listing_files, offline, marker_missing, _ = _load_gather(targets) + swept = _auto_fast_sweep(offline) data = {"ok": True, "action": "load", "offline": offline, "marker_missing": marker_missing, "synced": synced, - "needs_attention": flagged, + "needs_attention": flagged, "fast_sweep": swept, "sections": {k: v for k, v in texts.items() if not k.startswith("_")}, "recent_sessions": sorted((f for f in listing_files if f.endswith(".md")), reverse=True)[:5]} @@ -863,6 +882,10 @@ def cmd_load(brief: bool = False) -> int: results, texts, listing_files, offline, marker_missing, heartbeat_absent = \ _load_gather(targets) + swept = _auto_fast_sweep(offline) + if swept: + print(f"({swept})\n") + if brief: _fetch_pointed_session(texts) print(_render_brief(texts, listing_files, offline)) diff --git a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_doctor.py b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_doctor.py index e308768..093a76a 100644 --- a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_doctor.py +++ b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_doctor.py @@ -74,6 +74,18 @@ def run_op() -> dict: else: line(status < 400, "marker fetch", f"HTTP {status}") + # Index freshness (2.3) — informational, never red: how stale this machine's + # local indexes are (the fast sweep trues them up at load past 7 days). + if fatal is None: + try: + import sweep as sweep_mod + age = sweep_mod.fast_sweep_age_days() + line(True, "index freshness", + f"last local sweep {age:.1f}d ago" if age is not None + else "no local sweep yet — runs automatically at the next load") + except Exception: # noqa: BLE001 + pass + reds = sum(1 for c in checks if not c["ok"]) return {"ok": reds == 0 and fatal is None, "action": "doctor", "endpoint": cfg["endpoint"], "fatal": fatal, "reds": reds, "checks": checks} diff --git a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_index.py b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_index.py index 94e0f5e..4c95fa6 100644 --- a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_index.py +++ b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_index.py @@ -168,7 +168,10 @@ def kind_for_path(path: str) -> str | None: def empty_index() -> dict: - return {"schema": 1, "updated": echo.today(), "entities": {}} + # Schema 2 (2.3): entries may carry `h`, a short content hash of the note body — + # the incremental sweep's change detector. Tolerated-absent (schema-1 entries + # simply get fetched and hashed on their first fast sweep). + return {"schema": 2, "updated": echo.today(), "entities": {}} def load() -> dict: @@ -287,13 +290,20 @@ def gate_candidates(index: dict, mention: str, kind: str | None = None, return out +def content_hash(text: str) -> str: + import hashlib + return hashlib.sha1(str(text).encode("utf-8", "replace")).hexdigest()[:16] + + def upsert(index: dict, slug: str, path: str, kind: str, title: str | None = None, - aliases=None) -> dict: + aliases=None, h: str | None = None) -> dict: ents = index.setdefault("entities", {}) e = ents.get(slug, {}) e["path"] = path e["kind"] = kind e["title"] = title or e.get("title") or slug + if h: + e["h"] = h # Keep any prior + explicit aliases, and ALWAYS fold in the safe title variants so a # hyphen/space/case form of the name resolves even if no one passed --aliases. Drop any # alias that just equals the slug (resolve already checks the slug). diff --git a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_ops.py b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_ops.py index d63e7bc..3fedd92 100644 --- a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_ops.py +++ b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_ops.py @@ -333,15 +333,11 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status echo.cmd_put(path, echo.temp_file(note.encode())) action = "created" - # H3: the entity-index write is the race the review flagged — a bare load->save lets - # two concurrent captures clobber each other's entry. Route it through a lock-guarded, - # fresh-re-read transaction. Returns the fresh index (used by auto-link below). - import echo_concurrency - index = echo_concurrency.atomic_index_update( - lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases)) - - # auto-link: any other known entity CONFIDENTLY named in the body (M2: is_confident_link - # rejects short/common tokens so a 3-char alias or a word like "API" can't link everywhere). + # auto-link FIRST (it mutates the note's ## Related section), so the content + # hash stored below reflects the note's final state. Uses the pre-update index + # — it only needs to find OTHER entities named in the body. (M2: is_confident_link + # rejects short/common tokens so a 3-char alias or a word like "API" can't link + # everywhere.) linked = 0 for s2, e2 in index.get("entities", {}).items(): if e2.get("path") in (None, path): @@ -351,11 +347,24 @@ def capture_op(kind: str | None, title: str, file_arg: str | None = None, status ca, cb = links.link_bidirectional(path, e2["path"]) linked += int(ca or cb) - # Keep the BM25 recall index current for this note (best-effort; lock-guarded inside - # update_note, so it can't clobber a concurrent writer either). + # The note's final content: hashed into the entity index (the incremental + # sweep's change detector, 2.3) and fed to the local recall index. + final_text = links.get_text(path) or "" + note_h = idx_mod.content_hash(final_text) if final_text else None + + # H3: the entity-index write is the race the review flagged — a bare load->save lets + # two concurrent captures clobber each other's entry. Route it through a lock-guarded, + # fresh-re-read transaction. + import echo_concurrency + index = echo_concurrency.atomic_index_update( + lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases, + h=note_h)) + + # Keep the local BM25 recall index current for this note (best-effort; a + # zero-network atomic file write since 2.3). try: import echo_recall - echo_recall.update_note(path, links.get_text(path) or "") + echo_recall.update_note(path, final_text) except Exception as exc: # never let recall-index upkeep fail a capture print(f"echo_ops: recall-index update skipped ({exc})", file=sys.stderr) diff --git a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_recall.py b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_recall.py index bb172c3..44ba758 100644 --- a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_recall.py +++ b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_recall.py @@ -52,9 +52,17 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) import echo # noqa: E402 import echo_index as idx_mod # noqa: E402 import echo_links as links # noqa: E402 +import echo_stem # noqa: E402 RECALL_INDEX_PATH = "_agent/index/recall-index.json" -INDEX_SCHEMA = 2 # bumped: schema 2 adds per-doc meta (weight/updated/status) +# Schema 3 (2.3, the "index train"): postings are STEMMED (echo_stem), per-doc meta +# gains a content hash `h` (incremental sweep), and the index carries a `built` +# stamp + endpoint hash. The LIVE index is LOCAL-FIRST — it lives in the state dir +# and is updated with zero vault round-trips; the vault copy at RECALL_INDEX_PATH +# is a snapshot written by sweep/session-end, used to seed fresh machines. Older +# schemas are discarded and rebuilt (sub-second since the 1.1 network work). +INDEX_SCHEMA = 3 +SYNC_HOURS = float(os.environ.get("ECHO_RECALL_SYNC_HOURS", "24")) # Tuning knobs. Defaults are standard BM25 (k1/b) + a 0.6 hop decay over <=2 hops. K1 = 1.5 @@ -131,8 +139,11 @@ _SKIP_BASENAMES = {"README.md"} def tokenize(text: str) -> list[str]: - """Lowercase word tokens, stopworded. Frontmatter should be stripped by the caller.""" - return [t for t in _WORD.findall(text.lower()) if t not in _STOP and len(t) > 1] + """Lowercase word tokens, stopworded, STEMMED (2.3) — applied identically at + index and query time, so 'deployed'/'deployment' meet at 'deploy'. + Frontmatter should be stripped by the caller.""" + return [echo_stem.stem(t) for t in _WORD.findall(text.lower()) + if t not in _STOP and len(t) > 1] def strip_frontmatter(text: str) -> str: @@ -152,9 +163,10 @@ class Bm25Index: self.df: Counter[str] = Counter() # term -> #docs containing it self.postings: dict[str, dict[str, int]] = {} # term -> {doc_path: tf} self.length: dict[str, int] = {} # doc_path -> token count - self.meta: dict[str, dict] = {} # doc_path -> {w, u, s} priors + self.meta: dict[str, dict] = {} # doc_path -> {w, u, s, h} priors self.n_docs = 0 self.avg_len = 0.0 + self.built = "" # ISO stamp of last full (re)build def _recompute(self) -> None: self.n_docs = len(self.length) @@ -165,12 +177,19 @@ class Bm25Index: self.remove(path) toks = tokenize(strip_frontmatter(raw_text)) self.length[path] = len(toks) - self.meta[path] = doc_meta(path, raw_text) + meta = doc_meta(path, raw_text) + # Content hash: the incremental sweep's change detector (2.3). + import hashlib + meta["h"] = hashlib.sha1(raw_text.encode("utf-8", "replace")).hexdigest()[:16] + self.meta[path] = meta for term, tf in Counter(toks).items(): self.postings.setdefault(term, {})[path] = tf self.df[term] += 1 self._recompute() + def content_hash(self, path: str) -> str | None: + return (self.meta.get(path) or {}).get("h") + def remove(self, path: str) -> None: if path not in self.length: return @@ -192,11 +211,17 @@ class Bm25Index: * freshness(m.get("u"), today_s) * STATUS_FACTOR.get(m.get("s", ""), 1.0)) - def score(self, query: str, limit: int = 10, - today_s: str | None = None) -> list[tuple[str, float]]: + def score(self, query: str, limit: int = 10, today_s: str | None = None, + extra_terms: dict[str, float] | None = None) -> list[tuple[str, float]]: + """BM25 x priors. `extra_terms` (term -> weight) are alias-expansion terms + (2.3): they contribute at reduced weight and can only boost documents that + actually contain them — expansion never invents corpus-free hits.""" today_s = today_s or echo.today() + weighted: dict[str, float] = {t: 1.0 for t in set(tokenize(query))} + for t, w in (extra_terms or {}).items(): + weighted.setdefault(t, w) scores: Counter[str] = Counter() - for term in set(tokenize(query)): + for term, weight in weighted.items(): posting = self.postings.get(term) if not posting: continue @@ -204,26 +229,29 @@ class Bm25Index: for path, tf in posting.items(): dl = self.length.get(path, 0) denom = tf + K1 * (1 - B + B * (dl / self.avg_len if self.avg_len else 1)) - scores[path] += idf * (tf * (K1 + 1) / denom if denom else 0) + scores[path] += weight * idf * (tf * (K1 + 1) / denom if denom else 0) fused = {p: s * self._doc_factor(p, today_s) for p, s in scores.items()} ranked = sorted(fused.items(), key=lambda kv: -kv[1])[:limit] return [(p, s) for p, s in ranked if s > MIN_SCORE] # ---- serialization (compact; rebuildable, so the format can change freely) ---- def to_json(self) -> dict: - return {"schema": INDEX_SCHEMA, "n_docs": self.n_docs, "avg_len": self.avg_len, - "length": self.length, "postings": self.postings, "meta": self.meta} + return {"schema": INDEX_SCHEMA, "built": self.built, "n_docs": self.n_docs, + "avg_len": self.avg_len, "length": self.length, + "postings": self.postings, "meta": self.meta} @classmethod def from_json(cls, d: dict) -> "Bm25Index": if d.get("schema") != INDEX_SCHEMA: return cls() # older/newer format -> empty; recall's cold path rebuilds + # (a pre-stemming index MUST NOT be reused: its postings are unstemmed) ix = cls() ix.length = d.get("length", {}) ix.postings = d.get("postings", {}) ix.meta = d.get("meta", {}) ix.n_docs = d.get("n_docs", len(ix.length)) ix.avg_len = d.get("avg_len", 0.0) + ix.built = d.get("built", "") ix.df = Counter({term: len(posting) for term, posting in ix.postings.items()}) return ix @@ -271,27 +299,84 @@ def _indexable(path: str) -> bool: # ------------------------------------------------------------------- persistence -def load_index() -> Bm25Index: - status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH)) - if status == 404: - return Bm25Index() - echo.check(status, body, "recall-index load") +# LOCAL-FIRST (2.3). The live index is a machine-local file — updating it on capture +# is a zero-network atomic file write instead of GET-whole-index -> PUT-whole-index +# under the vault lock (the old per-write O(vault) round trip). The vault copy is a +# SNAPSHOT: written by sweep (and best-effort at session end), read only to seed a +# fresh machine or to catch up after another client swept (checked at most every +# ECHO_RECALL_SYNC_HOURS). Staleness across clients is tolerable — recall degrades +# gracefully and the fast sweep trues it up. + +def _local_path() -> Path: + import hashlib + import echo_queue + h = hashlib.sha1((echo.BASE or "").encode()).hexdigest()[:12] + return echo_queue.state_dir() / f"recall-index-{h}.json" + + +def _load_local() -> Bm25Index | None: + p = _local_path() + if not p.exists(): + return None try: + return Bm25Index.from_json(json.loads(p.read_text(encoding="utf-8"))) + except (json.JSONDecodeError, KeyError, TypeError, OSError): + return None + + +def save_local(ix: Bm25Index) -> None: + p = _local_path() + p.parent.mkdir(parents=True, exist_ok=True) + tmp = p.with_suffix(".json.tmp") + tmp.write_text(json.dumps(ix.to_json(), ensure_ascii=False), encoding="utf-8") + os.replace(tmp, p) # atomic: a crash mid-write can't corrupt the live index + + +def _fetch_snapshot() -> Bm25Index | None: + try: + status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH)) + if status != 200: + return None return Bm25Index.from_json(json.loads(body)) - except (json.JSONDecodeError, KeyError, TypeError): - return Bm25Index() + except Exception: # noqa: BLE001 — the snapshot is an optimization, never required + return None -def save_index(ix: Bm25Index) -> None: +def save_snapshot(ix: Bm25Index) -> None: + """PUT the vault snapshot — sweep + session-end only, never the per-capture path.""" body = json.dumps(ix.to_json(), ensure_ascii=False).encode("utf-8") status, b = echo.request("PUT", echo.vault_url(RECALL_INDEX_PATH), data=body, headers={"Content-Type": "application/json"}) - echo.check(status, b, "recall-index save") + echo.check(status, b, "recall-index snapshot save") -def rebuild(prefetched: dict | None = None) -> Bm25Index: - """Full rebuild from every indexable entity note. Called by sweep.py and as the - cold-cache path inside recall(). Saves and returns the index. +def load_index() -> Bm25Index: + """The live index: local file first; the vault snapshot only seeds a fresh + machine or replaces a local copy that a newer sweep elsewhere has superseded + (checked when the local build stamp is older than SYNC_HOURS).""" + local = _load_local() + if local and local.n_docs: + try: + age_ok = local.built and ( + _dt.datetime.now(_dt.timezone.utc) + - _dt.datetime.strptime(local.built, "%Y-%m-%dT%H:%M:%SZ").replace( + tzinfo=_dt.timezone.utc) + ).total_seconds() < SYNC_HOURS * 3600 + except ValueError: + age_ok = False + if age_ok: + return local + snap = _fetch_snapshot() + if snap and snap.n_docs and (not local or (snap.built or "") > (local.built or "")): + save_local(snap) + return snap + return local or Bm25Index() + + +def rebuild(prefetched: dict | None = None, snapshot: bool = False) -> Bm25Index: + """Full rebuild from every indexable note. Saves the LOCAL index always; writes + the vault snapshot only when `snapshot=True` (sweep) — the read path never + surprises the vault with a PUT. `prefetched` (a {path: text} map, e.g. from echo.read_many) lets sweep.py reuse the content it already pulled instead of walking and re-fetching the whole vault again.""" @@ -299,27 +384,30 @@ def rebuild(prefetched: dict | None = None) -> Bm25Index: if prefetched is not None: items = ((p, t) for p, t in prefetched.items() if _indexable(p)) else: - items = ((p, _get(p)) for p in _walk() if _indexable(p)) + items = echo.read_many([p for p in _walk() if _indexable(p)]).items() for path, text in items: if text is not None: ix.add(path, text) # add() strips frontmatter + extracts doc meta itself - save_index(ix) + ix.built = echo.now_iso() + save_local(ix) + if snapshot: + save_snapshot(ix) return ix def update_note(path: str, text: str) -> None: - """Best-effort incremental upkeep for a single note (mirrors how entities.json is - maintained on capture). Never raises into the caller. The load->save is wrapped in the - advisory lock with a fresh re-read (H3), so concurrent captures can't clobber the - recall index either.""" + """Best-effort incremental upkeep for a single note — a LOCAL atomic file write, + no vault round-trip, no advisory lock (2.3). Maintains an existing local index + only: with no local index yet, the next recall's cold path (or a sweep) builds + one, and a one-note index must never shadow the snapshot. Never raises.""" try: if not _indexable(path): - return # only entity notes are part of the recall corpus - import echo_concurrency - with echo_concurrency.vault_lock(): - ix = load_index() # fresh read inside the lock - ix.add(path, text) # raw text: add() strips + extracts meta - save_index(ix) + return # only corpus notes are part of the recall index + ix = _load_local() + if ix is None or not ix.n_docs: + return + ix.add(path, text) # raw text: add() strips + extracts meta + save_local(ix) except Exception as exc: # noqa: BLE001 — upkeep must never fail a capture print(f"echo_recall: index update skipped ({exc})", file=sys.stderr) @@ -405,13 +493,27 @@ def recall_op(query, limit: int = 8) -> dict: index = idx_mod.load() nmap = idx_mod.name_map(index) + # --- alias query expansion (2.3): a query phrased by an entity's alias or a + # partial name folds that entity's title/alias vocabulary into the BM25 query at + # half weight — "the ECHO plugin" also scores `echo-memory` terms. Expansion can + # only boost documents that actually contain the terms. + extra_terms: dict[str, float] = {} + q_toks = set(tokenize(q)) + for _slug, e, sc in idx_mod.fuzzy_candidates(index, q)[:2]: + if sc < 0.5: + continue + for name in (e.get("title", ""), _slug, *e.get("aliases", [])): + for tok in tokenize(str(name).replace("-", " ")): + if tok not in q_toks: + extra_terms[tok] = 0.5 + # --- lexical layer (BM25); rebuild the cache on a cold miss -------------- lexical: list[tuple[str, float]] = [] try: ix = load_index() if ix.n_docs == 0: ix = rebuild() - lexical = ix.score(q, limit=limit, today_s=today_s) + lexical = ix.score(q, limit=limit, today_s=today_s, extra_terms=extra_terms) except echo.EchoError as exc: print(f"recall: BM25 index unavailable ({exc}); falling back to server search", file=sys.stderr) diff --git a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_session.py b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_session.py index c650b90..8ff1e10 100644 --- a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_session.py +++ b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_session.py @@ -135,6 +135,18 @@ def session_end_op(bundle: dict, apply: bool = False) -> dict: echo.cmd_put("_agent/heartbeat/last-session.md", echo.temp_file(f"{path} @ {echo.now_iso()}\n".encode("utf-8"))) steps["heartbeat"] = "ok" + # Index maintenance (2.3), NOT part of the bundle contract: push this + # machine's local recall index to the vault snapshot so other machines + # can seed/catch up. Best-effort — a failure never taints the bundle. + try: + import echo_recall + rix = echo_recall._load_local() + if rix and rix.n_docs: + rix.built = rix.built or echo.now_iso() + echo_recall.save_snapshot(rix) + steps["recall_snapshot"] = "ok" + except Exception: # noqa: BLE001 + steps["recall_snapshot"] = "skipped" data["steps"] = steps return echo_output.envelope("session-end", data) diff --git a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_stem.py b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_stem.py new file mode 100644 index 0000000..7f5234e --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_stem.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 +"""echo_stem.py — a deliberately light, dependency-free English stemmer. [2.3] + +BM25 is purely lexical: "deploy", "deployed", and "deployment" are three unrelated +terms to it, so a paraphrased recall query misses notes stored with a different +inflection. This Porter-LITE stemmer conflates the common inflection families while +staying conservative — when in doubt it leaves the token alone, because an +over-stemmed index silently merges unrelated words (far worse than a missed match). + +Applied by echo_recall.tokenize() at BOTH index and query time (recall-index +schema 3; older indexes are discarded and rebuilt). Pure function, unit-tested in +test_echo_client.py. + +Design notes: + * plural / -ed / -ing families first (with the classic restore rules: doubled + consonant undoubling, at/bl/iz +e, cvc +e), then one derivational suffix pass, + then y->i and final-e normalization so "navigate"/"navigation"/"navigating" + all land on "navigat". + * every rule guards a minimum remaining stem length — "sing" is not "s". +""" +from __future__ import annotations + +import re + +_VOWEL = re.compile(r"[aeiouy]") +_DOUBLE = re.compile(r"([^aeiouylsz])\1$") +_CVC = re.compile(r"[^aeiou][aeiouy][^aeiouwxy]$") + +# One derivational pass, most-specific first. Replacements keep families together: +# navigational/navigation -> navigat · saturation/saturated -> saturat · +# deployment/deployed -> deploy · usefulness/useful -> use... (guarded by length). +_SUFFIXES = ( + ("ization", "ize"), ("ational", "ate"), ("fulness", "ful"), ("ousness", "ous"), + ("iveness", "ive"), ("tional", "t"), ("biliti", "ble"), ("ements", ""), + ("ment", ""), ("ness", ""), ("tion", "t"), ("sion", "s"), ("ance", ""), + ("ence", ""), ("able", ""), ("ible", ""), ("ally", "al"), ("ously", "ous"), + ("ively", "ive"), ("ly", ""), +) + + +def stem(t: str) -> str: + """Stem one lowercase token. Never raises; returns the token unchanged when no + rule applies safely.""" + if len(t) <= 3 or not t.isascii(): + return t + # --- plurals ------------------------------------------------------------- + if t.endswith("sses"): + t = t[:-2] + elif t.endswith("ies") and len(t) > 4: + t = t[:-3] + "i" + elif t.endswith("s") and not t.endswith(("ss", "us", "is")): + t = t[:-1] + # --- -ed / -ing (with restore rules) -------------------------------------- + for suf in ("ingly", "edly", "ing", "ed"): + if t.endswith(suf): + base = t[: -len(suf)] + if len(base) >= 3 and _VOWEL.search(base): + t = base + if t.endswith(("at", "bl", "iz")): + t += "e" + elif _DOUBLE.search(t): + t = t[:-1] + elif len(t) == 3 and _CVC.search(t): + t += "e" + break + # --- one derivational suffix pass ----------------------------------------- + for suf, rep in _SUFFIXES: + if t.endswith(suf): + base = t[: -len(suf)] + if len(base) >= 3 and len(base + rep) >= 3: + t = base + rep + break + # --- normalize: y->i (so penalty/penalties agree), then drop a final e ----- + if len(t) > 3 and t.endswith("y") and t[-2] not in "aeiou": + t = t[:-1] + "i" + if len(t) > 4 and t.endswith("e"): + t = t[:-1] + return t diff --git a/echo-memory.plugin.src/skills/echo-memory/scripts/sweep.py b/echo-memory.plugin.src/skills/echo-memory/scripts/sweep.py index a645790..a0a56db 100644 --- a/echo-memory.plugin.src/skills/echo-memory/scripts/sweep.py +++ b/echo-memory.plugin.src/skills/echo-memory/scripts/sweep.py @@ -18,9 +18,19 @@ backfill what older vaults don't have yet: 5. stamp the marker `schema_version` to the current schema (4). Dry-run by default; pass --apply to write. READ-ONLY without --apply. + +**--fast (2.3)** is the incremental mode: it walks the listing (cheap), then fetches +only what's new, gone, missing a content hash, or in today's rotating verification +shard (hash(path) % 7 == weekday — the whole vault gets verified over a week of +fast sweeps while each run stays tiny; --all-shards forces everything). It updates +the LOCAL recall index and the entity index (both machine-owned), so it needs no +--apply gate and writes no notes. `load` auto-runs it when the last fast sweep is +older than ECHO_FAST_SWEEP_DAYS (default 7). This is what keeps the indexes honest +against edits made directly in Obsidian or by other clients. + Cross-platform: pure Python via echo.py. -Usage: sweep.py [--apply] +Usage: sweep.py [--apply] | sweep.py --fast [--all-shards] """ from __future__ import annotations @@ -102,10 +112,128 @@ def walk(prefix: str = ""): yield from walk(f"{prefix}{d}/") +def _stamp_path() -> Path: + import echo_queue + return echo_queue.state_dir() / "fast-sweep.stamp" + + +def _stamp_fast_sweep() -> None: + try: + p = _stamp_path() + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(echo.now_iso() + "\n", encoding="utf-8") + except OSError: + pass + + +def fast_sweep_age_days() -> float | None: + """Days since the last fast/full sweep on THIS machine, or None if never.""" + import datetime as dt + try: + stamp = _stamp_path().read_text(encoding="utf-8").strip() + then = dt.datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=dt.timezone.utc) + return (dt.datetime.now(dt.timezone.utc) - then).total_seconds() / 86400 + except (OSError, ValueError): + return None + + +def fast(all_shards: bool = False) -> str: + """Incremental index maintenance (2.3): true up the LOCAL recall index and the + entity index against the vault, fetching only new / gone / hash-missing notes + plus today's rotating verification shard. Index-only — writes no notes; the + indexes are machine-owned, so no --apply gate. Returns a one-line summary.""" + import datetime as dt + import echo_recall + + if get("_agent/echo-vault.md") is None: + return "fast-sweep skipped: vault not bootstrapped" + + listing = [p for p in walk() + if p.endswith(".md") and p.rsplit("/", 1)[-1] not in SKIP_BASENAMES + and not TEMPLATE_RE.search(p)] + lset = set(listing) + + rix = echo_recall._load_local() + if rix is None or not rix.n_docs: + # Nothing to maintain incrementally — build the local index outright + # (sub-second on a few-hundred-note vault; no vault snapshot write). + rix = echo_recall.rebuild() + _stamp_fast_sweep() + return f"fast-sweep: no local index — built one ({rix.n_docs} docs)" + + index = idx_mod.load() + ents = index.get("entities", {}) + removed = 0 + for slug in [s for s, e in list(ents.items()) if e.get("path") not in lset]: + del ents[slug] + removed += 1 + for p in [p for p in list(rix.length) if p not in lset]: + rix.remove(p) + removed += 1 + + try: + weekday = dt.date.fromisoformat(echo.today()).weekday() + except ValueError: + weekday = dt.date.today().weekday() + + def in_shard(p: str) -> bool: + return int(idx_mod.content_hash(p), 16) % 7 == weekday + + cands: set[str] = set() + for p in listing: + if not echo_recall._indexable(p): + continue + if p not in rix.length: # new note (any client, any tool) + cands.add(p) + continue + kind = kind_for(p) + if kind: + slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3]) + if not (ents.get(slug) or {}).get("h"): # pre-2.3 entry: hash it once + cands.add(p) + continue + if all_shards or in_shard(p): # rotating verification shard + cands.add(p) + + texts = echo.read_many(sorted(cands)) + changed = 0 + ents_dirty = removed > 0 + for p, t in texts.items(): + if t is None: + continue + h = idx_mod.content_hash(t) + if rix.content_hash(p) != h: + rix.add(p, t) + changed += 1 + kind = kind_for(p) + if kind: + slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3]) + prev = ents.get(slug) or {} + if prev.get("h") != h or prev.get("path") != p: + title = links.first_h1(t) or p.rsplit("/", 1)[-1][:-3] + aliases = list(prev.get("aliases", [])) + idx_mod.aliases_in_frontmatter(t) + idx_mod.upsert(index, slug, p, kind, title, aliases, h=h) + ents_dirty = True + + echo_recall.save_local(rix) + if ents_dirty: + idx_mod.save(index) + _stamp_fast_sweep() + return (f"fast-sweep: {len(cands)} note(s) checked, {changed} reindexed, " + f"{removed} removed" + (" (all shards)" if all_shards else "")) + + def main(argv: list[str] | None = None) -> int: ap = argparse.ArgumentParser(description="Bring an ECHO vault up to the current feature spec") ap.add_argument("--apply", action="store_true") + ap.add_argument("--fast", action="store_true", + help="incremental index maintenance (local recall + entity index only)") + ap.add_argument("--all-shards", action="store_true", + help="with --fast: verify every indexed note, not just today's shard") args = ap.parse_args(argv) + if args.fast: + print(fast(all_shards=args.all_shards)) + return 0 apply = args.apply tag = "APPLY" if apply else "PLAN " @@ -151,7 +279,8 @@ def main(argv: list[str] | None = None) -> int: # aliases (e.g. mentions learned by capture) are preserved; upsert adds title variants. prev_aliases = prev.get("aliases", []) if prev else [] aliases = list(prev_aliases) + idx_mod.aliases_in_frontmatter(text) - idx_mod.upsert(rebuilt, slug, path, kind, title, aliases) + idx_mod.upsert(rebuilt, slug, path, kind, title, aliases, + h=idx_mod.content_hash(text)) if not prev: added += 1 elif prev.get("path") != path or prev.get("title") != title: @@ -163,8 +292,9 @@ def main(argv: list[str] | None = None) -> int: # ---- 2b. (re)build the recall (BM25) index ------------------------------- if apply: - rix = echo_recall.rebuild(prefetched=texts) - print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25)") + rix = echo_recall.rebuild(prefetched=texts, snapshot=True) + _stamp_fast_sweep() + print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25; local + vault snapshot)") else: n_idx = sum(1 for p in all_files if echo_recall._indexable(p)) print(f"sweep: {tag} recall index — would rebuild BM25 over {n_idx} corpus notes " diff --git a/echo-memory.plugin.src/skills/echo-memory/scripts/test_echo_client.py b/echo-memory.plugin.src/skills/echo-memory/scripts/test_echo_client.py index 7c7debd..12f19bd 100644 --- a/echo-memory.plugin.src/skills/echo-memory/scripts/test_echo_client.py +++ b/echo-memory.plugin.src/skills/echo-memory/scripts/test_echo_client.py @@ -215,6 +215,42 @@ def test_links_create_related_section_when_absent() -> None: assert changed and "## Related" in new and "[[resources/concepts/beta]]" in new +def test_stemmer_conflates_inflection_families() -> None: + import echo_stem + families = [ + ["deploy", "deploys", "deployed", "deploying", "deployment"], + ["navigate", "navigation", "navigating", "navigational"], + ["renew", "renews", "renewing"], + ["prefer", "prefers", "preferring", "preferred"], + ["penalty", "penalties"], + ["saturate", "saturation", "saturated"], + ["frequency", "frequencies"], + ["expand", "expanded", "expanding"], + ["meet", "meeting", "meetings"], + ["service", "services"], + ] + for fam in families: + stems = {echo_stem.stem(w) for w in fam} + assert len(stems) == 1, f"{fam} -> {stems}" + + +def test_stemmer_leaves_short_and_risky_tokens_alone() -> None: + import echo_stem + # over-stemming merges unrelated words — worse than a missed match + for w in ("sing", "thing", "was", "is", "bus", "this", "echo", "vault", "mpm"): + assert echo_stem.stem(w) == w, f"{w} mangled to {echo_stem.stem(w)}" + # documented non-conflations (the -al / d~s irregulars full Porter also skips) + assert echo_stem.stem("renewal") != echo_stem.stem("renew") + assert echo_stem.stem("expansion") != echo_stem.stem("expand") + + +def test_content_hash_stable_and_short() -> None: + import echo_index as ix + a = ix.content_hash("hello world") + assert a == ix.content_hash("hello world") and len(a) == 16 + assert a != ix.content_hash("hello world!") + + def _run_all() -> int: tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)] failed = 0 diff --git a/eval/results/latest.json b/eval/results/latest.json index 75f3678..7d24487 100644 --- a/eval/results/latest.json +++ b/eval/results/latest.json @@ -3,17 +3,17 @@ "date": "2026-07-03", "retrieval": { "current (hybrid+priors, full corpus)": { - "precision_at_5": 0.25, + "precision_at_5": 0.243, "recall_at_5": 1.0, "mrr": 1.0, - "session_journal_queries_answered": "2/2", + "session_journal_queries_answered": "3/3", "freshness_top1_correct": true }, "baseline (keyword, entities only)": { - "precision_at_5": 0.2, + "precision_at_5": 0.186, "recall_at_5": 0.75, - "mrr": 0.75, - "session_journal_queries_answered": "0/2", + "mrr": 0.786, + "session_journal_queries_answered": "0/3", "freshness_top1_correct": false } }, diff --git a/eval/run_eval.py b/eval/run_eval.py index e068747..60c3430 100644 --- a/eval/run_eval.py +++ b/eval/run_eval.py @@ -135,6 +135,16 @@ QUERIES = [ ("contract renewal Q3", {"resources/companies/vantage-systems.md"}, False), ("renegotiation kickoff", {"journal/daily/2026-06-20.md"}, True), ("graph neighbourhood expansion", {"resources/concepts/graph-expansion.md"}, False), + # ---- paraphrase set (2.3): morphological variants + rephrasings the pre-2.3 + # purely-lexical index missed — the stemming gain is measured, not assumed. + ("renewing enterprise contracts", {"resources/companies/vantage-systems.md"}, False), + ("navigational cleanup", {"projects/active/website-redesign.md", + "projects/archived/old-website.md"}, False), + ("preferring uv for tooling", {"_agent/memory/semantic/tooling-preferences.md"}, False), + ("penalties in the SLA", + {"_agent/sessions/2026-06-20-1000-vantage-contract-review.md"}, True), + ("saturated term frequencies", {"resources/concepts/bm25-ranking.md"}, False), + ("expanded linked neighbourhoods", {"resources/concepts/graph-expansion.md"}, False), ] FRESHNESS_QUERY = "marketing site navigation" FRESHNESS_TOP1 = "projects/active/website-redesign.md" diff --git a/eval/test_features.py b/eval/test_features.py index 8c40610..ffbadd8 100644 --- a/eval/test_features.py +++ b/eval/test_features.py @@ -27,7 +27,9 @@ def check(name, cond, detail=""): class Harness: def __init__(self, base): + import tempfile self.base = base + self.state_dir = tempfile.mkdtemp() def http(self, method, url, body=None, headers=None): data = body.encode() if isinstance(body, str) else body @@ -52,8 +54,10 @@ class Harness: return None if body == "<>" else body def echo(self, *args): + # ECHO_STATE_DIR isolation matters since 2.3: the recall index is LOCAL-first, + # and without this the suite would write into the real ~/.echo-memory. env = dict(os.environ, ECHO_BASE=self.base, ECHO_KEY=KEY, ECHO_VERIFY="1", - ECHO_TODAY="2026-06-21") + ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=self.state_dir) return subprocess.run([sys.executable, str(args[0]), *args[1:]], capture_output=True, text=True, env=env) @@ -230,8 +234,9 @@ def main(): check("v1.5 recall finds a session log by body term", "_agent/sessions/2026-06-15-1200-mpm-review.md" in r.stdout, r.stdout) rix2 = h.ground("_agent/index/recall-index.json") or "" - check("v1.5 recall index carries doc meta (schema 2)", - '"schema": 2' in rix2 or '"schema":2' in rix2, rix2[:200]) + check("v2.3 recall snapshot is schema 3 with build stamp + content hashes", + ('"schema": 3' in rix2 or '"schema":3' in rix2) + and '"built"' in rix2 and '"h"' in rix2, rix2[:200]) # 14. recency-aware ranking: same term, fresher note wins. h.seed("resources/concepts/old-idea.md", @@ -402,6 +407,59 @@ def main(): r.returncode == 2 and sess_path in (h.ground("_agent/heartbeat/last-session.md") or ""), r.stderr) + # ---------------- v2.3 index train -------------------------------------- + # 20. local-first recall index: capture updates the LOCAL index with ZERO + # vault snapshot writes — the vault copy only changes on sweep/session-end. + snap_before = h.ground("_agent/index/recall-index.json") or "" + r = h.echo(ECHO, "capture", "Quixotic Widget", "--kind", "concept", "-") + r = subprocess.run([sys.executable, str(ECHO), "capture", "Quixotic Widget2", "--kind", + "concept", "-"], input="A gadget for quixotry deployment.\n", + capture_output=True, text=True, + env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, + ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=h.state_dir)) + snap_after = h.ground("_agent/index/recall-index.json") or "" + check("v2.3 capture does NOT write the vault snapshot", snap_before == snap_after) + local_files = [p for p in Path(h.state_dir).glob("recall-index-*.json")] + check("v2.3 capture maintains the LOCAL index", len(local_files) == 1, + str(local_files)) + r = h.echo(ECHO, "recall", "quixotry") + check("v2.3 locally-indexed capture is recallable", + "resources/concepts/quixotic-widget2.md" in r.stdout, r.stdout[:400]) + + # 21. stemming: a morphologically different query still hits (deployment ~ deploying). + r = h.echo(ECHO, "recall", "deploying quixotry") + check("v2.3 stemmed recall bridges inflections", + "resources/concepts/quixotic-widget2.md" in r.stdout, r.stdout[:400]) + + # 22. alias query expansion: an alias-phrased query surfaces the entity even + # though the alias never appears in any note body. + r = subprocess.run([sys.executable, str(ECHO), "capture", "Kappa Engine", "--kind", + "concept", "-", "--aliases", "turbokappa"], + input="Compression tuning notes for the kappa engine core.\n", + capture_output=True, text=True, + env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, + ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=h.state_dir)) + r = h.echo(ECHO, "recall", "turbokappa compression") + check("v2.3 alias expansion surfaces the aliased entity", + "resources/concepts/kappa-engine.md" in r.stdout, r.stdout[:400]) + + # 23. incremental fast sweep: an out-of-band edit (seeded directly, as if by + # Obsidian) and a deletion are trued up without a full rebuild. + h.seed("resources/concepts/quixotic-widget.md", + "---\ntype: concept\nstatus: active\ncreated: 2026-06-21\nupdated: 2026-06-21\n" + "tags: [concept]\n---\n# Quixotic Widget\n\nNow mentions zanzibar explicitly.\n") + h.http("DELETE", f"{base}/vault/resources/concepts/quixotic-widget2.md") + r = h.echo(SWEEP, "--fast", "--all-shards") + check("v2.3 fast sweep reports its work", + "fast-sweep:" in r.stdout and "reindexed" in r.stdout, r.stdout + r.stderr) + r = h.echo(ECHO, "recall", "zanzibar") + check("v2.3 fast sweep picked up the out-of-band edit", + "resources/concepts/quixotic-widget.md" in r.stdout, r.stdout[:400]) + idx5 = h.ground("_agent/index/entities.json") or "" + check("v2.3 fast sweep drops deleted notes from the entity index", + "quixotic-widget2" not in idx5) + check("v2.3 entity index carries content hashes", '"h"' in idx5, idx5[:300]) + print(f"\n{len(failures)} failure(s)" if failures else "\nall feature tests passed") return 1 if failures else 0 finally: diff --git a/eval/test_patch_semantics.py b/eval/test_patch_semantics.py index e57d336..00b8b8e 100644 --- a/eval/test_patch_semantics.py +++ b/eval/test_patch_semantics.py @@ -59,7 +59,9 @@ def main(): stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) def echo(*args): - env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="0") + import tempfile + env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="0", + ECHO_STATE_DIR=globals().setdefault("_STATE_DIR", tempfile.mkdtemp())) return subprocess.run([sys.executable, str(ECHO), *args], capture_output=True, text=True, env=env) def ground(path): diff --git a/eval/test_reflect.py b/eval/test_reflect.py index 6362370..0a10c79 100644 --- a/eval/test_reflect.py +++ b/eval/test_reflect.py @@ -51,7 +51,9 @@ def main(): return getattr(e, "code", 0), "" def echo(*args, stdin=None): - env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1", ECHO_TODAY="2026-06-22") + import tempfile + env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1", + ECHO_TODAY="2026-06-22", ECHO_STATE_DIR=tempfile.mkdtemp()) return subprocess.run([sys.executable, str(ECHO), *args], input=stdin, capture_output=True, text=True, env=env)