2.3.0 — the index train: local-first recall index, incremental sweep, stemming + alias expansion
Build and Push Docker Image / build (push) Successful in 14s
Build and Push Docker Image / build (push) Successful in 14s
One schema-bump release (recall-index schema 3, entity-index schema 2; old recall indexes rebuild automatically, no vault migration): - Local-first recall index: the live BM25 index lives in the machine state dir (keyed by endpoint hash); capture's upkeep is a zero-network atomic file write, no advisory lock — replacing the O(vault) GET/PUT-whole-index round trip per write. The vault copy is a snapshot (sweep + session-end) used to seed fresh machines / catch up after another client swept (ECHO_RECALL_SYNC_HOURS, default 24). The read path never PUTs the vault. - Incremental sweep: entity + recall meta carry 16-char content hashes; `sweep --fast` fetches only new/gone/hash-missing notes plus a rotating weekday shard (--all-shards forces everything); deletions drop from both indexes; index-only so no --apply gate. `load` auto-runs it past ECHO_FAST_SWEEP_DAYS (7); doctor reports index freshness. Obsidian-side edits now reach the indexes without manual maintenance. - Stemming + alias expansion: echo_stem.py (conservative Porter-lite, unit- tested families + over-stemming guards) applied at index AND query time; a query fuzzy-matching an entity folds its title/alias vocabulary into the BM25 query at half weight — expansion can only boost docs containing the terms. capture hashes the note's FINAL content (auto-link reordered first). Eval gold set 8 -> 14 queries (6 paraphrases): recall@5/MRR 1.00/1.00 vs keyword baseline 0.75/0.79. +3 unit tests, +10 e2e; test harnesses now isolate ECHO_STATE_DIR. All seven suites green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,57 @@
|
||||
# Changelog
|
||||
|
||||
## 2.3.0
|
||||
|
||||
**The index train** (IMPROVEMENT-PLANS #2/#4/#8) — one schema-bump release:
|
||||
recall-index schema 3, entity-index schema 2. Old recall indexes are discarded
|
||||
and rebuilt automatically (sub-second); no vault migration.
|
||||
|
||||
### Changed — the recall index is LOCAL-FIRST
|
||||
|
||||
The live BM25 index moves out of the vault into the machine state dir
|
||||
(`recall-index-<endpoint-hash>.json` under `ECHO_STATE_DIR`). Capture's index
|
||||
upkeep — previously GET-whole-index → PUT-whole-index against the vault under
|
||||
the advisory lock, an O(vault) network round trip per write — is now a
|
||||
zero-network atomic file write with no lock at all. The vault copy at
|
||||
`_agent/index/recall-index.json` becomes a **snapshot**: written by
|
||||
`sweep --apply` and best-effort at `session-end`, read only to seed a fresh
|
||||
machine or to catch up after another client swept (checked at most every
|
||||
`ECHO_RECALL_SYNC_HOURS`, default 24). The read path never PUTs the vault.
|
||||
|
||||
### Added — incremental sweep (`sweep --fast`) + content hashes
|
||||
|
||||
Entity-index entries and recall-index doc meta now carry a 16-char content
|
||||
hash. `sweep --fast` walks the listing (cheap) and fetches only what is new,
|
||||
gone, missing a hash, or in **today's rotating verification shard**
|
||||
(`hash(path) % 7 == weekday` — the whole vault gets verified across a week of
|
||||
fast sweeps while each run stays tiny; `--all-shards` forces everything).
|
||||
Index-only and machine-owned, so no `--apply` gate; deletions are dropped from
|
||||
both indexes. **`load` auto-runs it** when this machine's last sweep is older
|
||||
than `ECHO_FAST_SWEEP_DAYS` (default 7) — edits made directly in Obsidian or by
|
||||
other clients now reach the indexes without anyone remembering maintenance.
|
||||
`doctor` reports index freshness.
|
||||
|
||||
### Added — stemming + alias query expansion
|
||||
|
||||
`echo_stem.py`: a conservative Porter-lite stemmer applied by `tokenize()` at
|
||||
BOTH index and query time, so "deployed"/"deployment"/"deploys" meet at
|
||||
"deploy" and "penalties" finds "penalty" (guards against over-stemming: "sing"
|
||||
is not "s"; the -al/d~s irregulars are deliberately not conflated). At recall
|
||||
time, a query that fuzzy-matches an entity (score ≥ 0.5) folds that entity's
|
||||
title/alias vocabulary into the BM25 query at **half weight** — "turbokappa
|
||||
tuning" finds the Kappa Engine note even though the alias appears in no body.
|
||||
Expansion can only boost documents that actually contain the terms.
|
||||
|
||||
### Eval
|
||||
|
||||
Gold set grows 8 → **14 queries** (6 paraphrases the pre-2.3 lexical index
|
||||
missed). v2.3.0: recall@5 / MRR **1.00 / 1.00**, 3/3 session-journal queries;
|
||||
keyword baseline 0.75 / 0.79 and 0/3. Tests: +3 unit (stemmer families,
|
||||
over-stemming guards, hash), +10 end-to-end (local-first no-snapshot-write,
|
||||
local recallability, stemmed recall, alias expansion, fast-sweep pickup/
|
||||
deletion/hash backfill). All suites — including the offline-queue, ops-api,
|
||||
and MCP server suites — green; test harnesses now isolate `ECHO_STATE_DIR`.
|
||||
|
||||
## 2.2.0
|
||||
|
||||
### Added — echo-mcp: the containerized MCP server
|
||||
|
||||
Reference in New Issue
Block a user