From 993abdc846c084154933542d69011edb73094da1 Mon Sep 17 00:00:00 2001 From: Jason Stedwell Date: Fri, 3 Jul 2026 13:30:01 -0500 Subject: [PATCH] Retire ROADMAP-1.0; rewrite run_eval.py as current-version metrics harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Knocks two items off ROADMAP-2.0 (§4 eval refresh, §5 roadmap retirement): - run_eval.py: the stale 0.6-vs-0.7 A/B is replaced (old version in git history) with a four-part current-version eval against the mock — retrieval (hybrid+priors vs keyword/entities-only baseline: R@5 1.00 vs 0.75, MRR 1.00 vs 0.75, session/journal queries 2/2 vs 0/2, freshness top-1 correct vs wrong), dedup (0 dupes gate-on vs 3 gate-off, 0 false blocks), write safety (0 silent failures across 4 fault scenarios), durability (3/3 offline writes queued+landed, 0 lost, 0 re-flush dupes). Results in eval/results/latest.json. - README: metrics table published; repo-layout eval descriptor updated. - eval/README: new harness documented; A/B framing marked historical. - ROADMAP-1.0.md removed (fully shipped; story lives in the README version table + git history); dangling docstring pointer fixed in echo_concurrency.py; ROADMAP-2.0 checkboxes ticked. All suites green; artifact rebuilt (1.5.1, docstring-only source change). Co-Authored-By: Claude Fable 5 --- README.md | 14 + ROADMAP-1.0.md | 176 ------ ROADMAP-2.0.md | 21 +- echo-memory-1.5.1.plugin | Bin 183493 -> 183508 bytes echo-memory.plugin | Bin 183493 -> 183508 bytes .../echo-memory/scripts/echo_concurrency.py | 2 +- eval/README.md | 24 +- eval/results/latest.json | 186 ++---- eval/run_eval.py | 577 ++++++++++-------- 9 files changed, 405 insertions(+), 595 deletions(-) delete mode 100644 ROADMAP-1.0.md diff --git a/README.md b/README.md index e12e907..8d8e768 100644 --- a/README.md +++ b/README.md @@ -380,6 +380,20 @@ If the API returns a connection error, timeout, or `502` (usually Obsidian / the --- +## Eval metrics (v1.5.1, 2026-07-03) + +From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate. + +| Metric | v1.5.1 | pre-1.5 baseline | +|---|---|---| +| Retrieval recall@5 / MRR (8-query gold set) | **1.00 / 1.00** | 0.75 / 0.75 | +| Queries answerable only from sessions/journal | **2/2** | 0/2 (not in corpus) | +| Freshness: live note outranks stale archived twin | **yes** | no | +| Duplicate notes created (3 renamed-entity captures) | **0** (gate blocks) | 3 | +| Legitimate captures wrongly blocked | **0** | 0 | +| Silent write failures (4 fault scenarios) | **0** (failures loud) | — | +| Offline writes lost / duplicated on re-flush | **0 / 0** (3/3 queued + landed) | — | + ## Version history | Version | Highlights | diff --git a/ROADMAP-1.0.md b/ROADMAP-1.0.md deleted file mode 100644 index 2ba46d4..0000000 --- a/ROADMAP-1.0.md +++ /dev/null @@ -1,176 +0,0 @@ -# echo-memory — Roadmap to v1.0 - -> Status: **draft / in progress.** Current shipping version: **0.9.0 (schema 3)**. -> Target: **1.0.0 (schema 4)** — *"memory you can actually trust and retrieve."* - -This roadmap turns the 10 code-review suggestions into a sequenced plan and tracks the -scaffold for each. It is the single index: every v1.0 workstream maps to one or more -files under `echo-memory.plugin.src/skills/echo-memory/scripts/` (plus tests and CI). - ---- - -## 1.0 thesis - -0.x made the plugin **correct and self-describing**: status-checked writes, idempotent -appends, a routing manifest the linter enforces, an entity index, and one-call -`capture/recall`. 1.0 makes it **trustworthy and genuinely recall-driven**: - -1. You can **retrieve** a fact even when you phrase the query differently than you stored it. *(H1)* -2. Nothing is **lost** when the backend is down. *(H2)* -3. Two clients writing at once **cannot corrupt** the index or clobber state. *(H3)* -4. Every feature is **proven** by tests that model the real API, gated in CI. *(H4)* -5. Memory **accrues on its own** instead of only when the agent remembers to write. *(H5)* - -…on a base that is secure (M1), clean-linking (M2), self-diagnosing (M3), -machine-parseable (M4), and maintainable (M5). - ---- - -## Non-negotiable constraints (these bound every solution) - -These are inherited from the 0.x design and **must not regress**: - -| Constraint | Consequence for 1.0 | -|---|---| -| **Pure-Python stdlib, no third-party deps** | H1 recall = local BM25 + graph fusion, **not** an embedding API. No `numpy`, `keyring`, `requests`. | -| **Cross-platform (Win/macOS/Linux)** | No bash, no platform `date`; `pathlib`, `os`, UTF-8-safe streams. | -| **Plugin is the single source of truth; vault holds data only** | Derived artifacts (recall index) live in `_agent/index/` (machine-maintained, rebuildable) — never new control docs in the vault. | -| **Fail loud, never silent** | New write paths keep status-checking and non-zero exits. | -| **Local state can't depend on the vault being up** | H2 queue/cache lives in `ECHO_STATE_DIR` (default `~/.echo-memory/`), not the vault. | - ---- - -## Build order vs. scaffold order - -The user asked to **scaffold heaviest → lightest** (H1→M5). That is the order the -files below are stubbed. The **recommended build/merge order is different** — safety -and proof first, so the heavy features land on a tested, secure base: - -``` -Phase A (land first): M1 secrets · H4 tests+CI -Phase B (correctness): H3 concurrency · M2 link/slug quality -Phase C (durability): H2 offline queue + cache -Phase D (intelligence): H1 hybrid recall · H5 reflection capture -Phase E (polish): M3 doctor · M4 --json · M5 hygiene → tag 1.0.0 -``` - -Scaffolds are independent stubs, so stubbing heaviest-first is safe; only the -*integration* (wiring into `echo.py`, schema bump, migration) follows Phase order. - ---- - -## Workstreams (heaviest → lightest) - -Each item: **goal · why · scope · files · storage/schema · acceptance · depends-on · lift.** - -### H1 — Hybrid/semantic recall · lift: ●●●●● -- **Goal:** `recall "X"` returns relevant notes even when the wording differs, ranked, with scored multi-hop graph expansion. -- **Why:** Recall is the product. Today it's keyword `/search/simple` + fixed one-hop (`echo_ops.py:88`). Differently-phrased memory is unfindable → dead weight. -- **Scope:** local BM25 index over note bodies (stdlib only); fuse lexical score + graph distance (decay per hop); relevance threshold; configurable hops. -- **Files:** `scripts/echo_recall.py` (new) → later replaces `echo_ops.recall`. -- **Storage/schema:** `_agent/index/recall-index.json` (postings + doc stats), machine-maintained, rebuilt by `sweep.py`. **Schema 3→4.** -- **Acceptance:** retrieval precision/recall measured in eval beats keyword-only baseline; recall returns scored, deduped hits + neighbourhood; index rebuildable offline. -- **Depends-on:** entity index (have), H4 eval harness for the precision/recall numbers. - -### H2 — Offline durability: write-ahead queue + read cache · lift: ●●●●● -- **Goal:** No durable write is lost when the API/Obsidian is down; cold-start degrades to last-known context instead of "no memory." -- **Why:** Today vault-unreachable = silently proceed without memory (`SKILL.md:383`). 502 (Obsidian not running) is the most common real failure. -- **Scope:** append-only NDJSON outbox; replay on next reachable session (idempotency makes replay safe); last-known-good read cache for `load`. -- **Files:** `scripts/echo_queue.py` (new); integration hook `safe_request()` wrapping `echo.request`. -- **Storage/schema:** local `ECHO_STATE_DIR/outbox.ndjson` + `ECHO_STATE_DIR/cache/`. No vault schema change. -- **Acceptance:** kill the API mid-session → writes queue; restart → replay lands exactly once (no dupes); `load` serves cache when offline and says so. -- **Depends-on:** existing idempotent-append discipline; H4 for fault-injection tests. - -### H3 — Concurrency correctness · lift: ●●●●○ -- **Goal:** Make the advertised multi-writer (Claude + CoWork) story actually safe. -- **Why:** Index is full-file load→PUT with no lock (`echo_index.py:105`, `echo_ops.py:230`) → concurrent `capture` silently drops an entity. Lock is manual/advisory only (`echo.py:293`). -- **Scope:** auto-lock context manager around `capture`/`scope set`/index writes; atomic index update (re-read under lock, merge, save); per-resource locks; crash-safe TTL reclaim (have). -- **Files:** `scripts/echo_concurrency.py` (new); refactor `echo_index.save` + `echo_ops.capture` to route through it. -- **Storage/schema:** reuse `_agent/locks/`. No schema change. -- **Acceptance:** two concurrent captures both land in the index; no lost entries; lock auto-released on normal + error exit. -- **Depends-on:** none (pure refactor); H4 to prove it. - -### H4 — High-fidelity tests + CI + current eval · lift: ●●●●○ -- **Goal:** Prove every 0.9 + 1.0 feature against an API model that reproduces real failure modes; gate on CI. -- **Why:** Mock PATCH is "naive append" (`mock_olrapi.py:138`) — can't model heading replace/insert; eval still benchmarks **0.6 vs 0.7** (two versions stale). -- **Scope:** higher-fidelity mock (real heading/frontmatter PATCH semantics) or containerized real Obsidian Local REST API; refresh eval to measure retrieval precision/recall, link correctness, dup rate; wire `test_echo_client.py` + `check_routing.py` + `test_features.py` + new module tests into GitHub Actions. -- **Files:** `.github/workflows/ci.yml` (new); `eval/mock_olrapi_hifi.py` (new); `scripts/test_v1_scaffold.py` (interface guard, new); refreshed `eval/run_eval.py`. -- **Acceptance:** CI green on every push; eval reports current-version metrics; mock reproduces the 40080 invalid-target and replace/insert paths. -- **Depends-on:** none — **foundational, build in Phase A.** - -### H5 — Automatic session-reflection capture · lift: ●●●●○ -- **Goal:** At session end, propose durable captures extracted from the conversation for one-tap confirm — memory that fills itself. -- **Why:** Memory only accrues when the agent decides to write. "As useful as possible" means not depending on that judgment firing. -- **Scope:** model-side extraction emits a JSON proposal set; script dedups against the entity index, previews, and applies on confirm (respects "show before large writes" safety rule). -- **Files:** `scripts/echo_reflect.py` (new); `/echo-reflect` command (later). -- **Storage/schema:** none new (routes through `capture`). -- **Acceptance:** given a transcript, proposals are deduped vs index, previewed, and only applied on explicit confirm; nothing written without go-ahead. -- **Depends-on:** H1 index/recall for dedup quality; H3 for safe concurrent apply. - -### M1 — Harden secret handling · lift: ●●○○○ *(do now — Phase A)* -- **Goal:** Stop shipping a live bearer token in source, docs, and every `.plugin` zip. -- **Why:** Token hardcoded at `echo.py:70` and repeated through `api-reference.md`/README — committed to git, only rotatable by rebuild. Violates the plugin's own "never store secrets" rule. -- **Scope:** resolve key env → local `ECHO_STATE_DIR/credentials` (0600) → deprecated baked default **with a loud warning**; scrub literal from docs to a placeholder; document rotation. -- **Files:** `scripts/echo_secrets.py` (new); `echo.py` `KEY=` → `echo_secrets.resolve_key()`. -- **Acceptance:** no live token in tracked source/docs/package; `ECHO_KEY` still overrides; rotation documented. -- **Depends-on:** none. - -### M2 — Tame auto-link false positives & slug collisions · lift: ●●○○○ -- **Goal:** Keep the graph (which recall depends on) clean. -- **Why:** Auto-link fires on any ≥3-char name/alias word-match (`echo_ops.py:236`) → a concept "API" or alias "rs" links everywhere; `slugify` truncates to 40 chars with no collision check (`echo_index.py:58`) → distinct titles silently share a note. -- **Scope:** raise alias floor / require multi-token or kind-aware match / stopword guard; collision-aware slug disambiguation in `derive_path`. -- **Files:** `scripts/echo_quality.py` (new); patches to `echo_links`/`echo_index`. -- **Acceptance:** known false-positive cases no longer link; colliding titles get distinct slugs; covered by tests. -- **Depends-on:** none. - -### M3 — `echo.py doctor` + complete `load` fallback · lift: ●●○○○ -- **Goal:** One-call readiness check; make `load` self-sufficient. -- **Why:** No single "is everything OK" check. `cmd_load` reads the heartbeat but never falls back to listing `_agent/sessions/` though `SKILL.md:122` documents it (`echo.py:395`). -- **Scope:** doctor = Python version + reachability + auth + marker/`schema_version` + lint summary, green/red; `load` lists recent sessions when heartbeat missing/stale. -- **Files:** `scripts/echo_doctor.py` (new); patch `echo.cmd_load`. -- **Acceptance:** `doctor` prints actionable status and exits non-zero on any red; `load` orients with no heartbeat. -- **Depends-on:** none. - -### M4 — Machine output (`--json`) + `--dry-run` · lift: ●○○○○ -- **Goal:** Let the agent act on structured results and preview writes. -- **Why:** `capture/recall/resolve/scope` print prose; agent re-parses free text. -- **Scope:** `--json` result envelope (action, path, links_added, hits); `--dry-run` for `capture`/`scope set`. -- **Files:** `scripts/echo_output.py` (new helpers); thread through `echo_ops`/`echo.py`. -- **Acceptance:** every high-level op emits valid JSON under `--json`; `--dry-run` writes nothing. -- **Depends-on:** none. - -### M5 — Repo hygiene & manifest completeness · lift: ●○○○○ -- **Goal:** Remove "which tree is canonical?" traps; complete the manifest. -- **Why:** Two plugin trees (`codex plugin/` vs `echo-memory.plugin.src/`) have already drifted (codex lacks the 0.9 modules); stack of versioned `.plugin` artifacts; sparse `plugin.json`. -- **Scope:** consolidate/mark canonical tree; prune old artifacts; add `license`/`homepage`/command registration to `plugin.json`; refresh version-history + eval references. -- **Files:** `MAINTENANCE.md` (new checklist); `plugin.json`. -- **Acceptance:** one canonical source tree; complete manifest; docs reference current version. -- **Depends-on:** none. - ---- - -## Schema 3 → 4 migration (owned by H1/H2) - -- Add `_agent/index/recall-index.json` (H1), rebuilt by `sweep.py`. -- `migrate.py`: add `[3→4]` step (create recall-index placeholder; no destructive moves). -- `routing.json`: add a route for `_agent/index/recall-index.json` (already covered by the generic `^_agent/index/[^/]+\.json$` route — verify in `check_routing.py`). -- Bump `CURRENT_SCHEMA` to 4 in `sweep.py`/`migrate.py`; `vault_lint.py` unaffected. - ---- - -## Scaffold status - -| # | Workstream | Lift | Scaffold file(s) | State | -|---|---|---|---|---| -| H1 | Hybrid recall | ●●●●● | `scripts/echo_recall.py` | **✅ WIRED** — BM25 + vault crawl + `recall-index.json` persistence + graph fusion; maintained on `capture`, rebuilt by `sweep`; replaces `echo_ops.recall`; **schema 4**. Covered by `test_features.py`. | -| H2 | Offline queue/cache | ●●●●● | `scripts/echo_queue.py`, `eval/test_offline_queue.py` | **✅ WIRED** — write verbs (put/post/append/patch/delete) queue on outage and report "queued"; `flush` (+ flush-on-`load`) replays idempotently and **re-bases to the current endpoint** (survives base migration); `load` degrades to last-known-good cache and flags OFFLINE. Proven end-to-end (down→queue→up→flush→land, idempotent, cache fallback). Follow-up: route capture's internal index/link writes through the queue for full offline-atomic captures. | -| H3 | Concurrency | ●●●●○ | `scripts/echo_concurrency.py` | **✅ WIRED** — `vault_lock` CM (auto acquire/release, crash-safe, advisory) + `atomic_index_update` (lock + fresh re-read-merge). `capture` entity write and `echo_recall.update_note` both routed through it. Proven by `test_features.py` (merge-no-clobber + lock-release). Follow-up: deeper resolve-level title-collision detection; true concurrent stress test. | -| H4 | Tests + CI | ●●●●○ | `.github/workflows/ci.yml`, `eval/mock_olrapi_hifi.py`, `eval/test_patch_semantics.py`, `scripts/test_v1_scaffold.py` | **✅ hi-fi PATCH mock + semantics test + CI matrix (Win/macOS/Linux × 3.10/3.12) live.** Remaining: refresh `run_eval.py` (still 0.6-vs-0.7) → publish current metrics. | -| H5 | Reflection capture | ●●●●○ | `scripts/echo_reflect.py`, `eval/test_reflect.py`, `commands/echo-reflect.md` | **✅ WIRED** — `validate`/`classify`/`preview`/`apply` + `echo.py reflect` (dry-run unless `--apply`, reads file or stdin) + `/echo-reflect` command. Dedups proposals vs the entity index, drops below-confidence, routes `inbox`, applies via `capture` (indexed/linked/logged, self-lock-guarded). Proven by `test_reflect.py`. | -| M1 | Secret handling | ●●○○○ | `scripts/echo_secrets.py` | **✅ WIRED** — `resolve_key` (env → `~/.echo-memory/credentials` → deprecated fallback + loud warning), `write-key` CLI, docs scrubbed. Live token now only in `echo.py` `DEFAULT_KEY` (remove at 1.0; operator env already sets `ECHO_KEY`). | -| M2 | Link/slug quality | ●●○○○ | `scripts/echo_quality.py` | **✅ WIRED** — `is_confident_link` gates `capture` auto-linking (rejects <4-char / common tokens, trusts multi-word names); `safe_slug` disambiguates a colliding 40-char-truncated slug in the create path. Follow-up: resolve-level title collision (two long titles → same slug → false merge). | -| M3 | Doctor + load fallback | ●●○○○ | `scripts/echo_doctor.py`, `commands/echo-doctor.md` | **✅ WIRED** — `echo.py doctor` (Python/reachability/auth/bootstrap/schema/key-source) + `/echo-doctor`; `cmd_load` now falls back to a recent-sessions listing when the heartbeat pointer is absent. Verified against the live vault. | -| M4 | `--json` / `--dry-run` | ●○○○○ | `scripts/echo_output.py` | **✅ WIRED** — `capture --json` emits a clean result envelope (helper chatter redirected) and `--dry-run` previews the create/update plan without writing; `resolve` already emits JSON. Proven by `test_features.py`. Follow-up: `--json` for recall/scope/link. | -| M5 | Hygiene | ●○○○○ | `plugin.json`, `.gitignore`, `MAINTENANCE.md` | **✅ DONE** — `plugin.json` → **1.0.0** + `license`; `.gitignore` (`.DS_Store`, `__pycache__`, local `.echo-memory/` state, eval output). Remaining (operator judgment, not code): consolidate/retire the drifted `codex plugin/` tree and prune old `.plugin` artifacts — see `MAINTENANCE.md`. | - -**Wiring:** scaffolds are **not** imported by the live `echo.py` yet — 0.9.0 behavior is unchanged. Integration happens per Phase, each gated by its tests (H4). diff --git a/ROADMAP-2.0.md b/ROADMAP-2.0.md index 770f662..51712b0 100644 --- a/ROADMAP-2.0.md +++ b/ROADMAP-2.0.md @@ -51,21 +51,22 @@ regenerated after 2.0 if needed). What remains for 2.0: documenting what a Codex/other-runtime port needs (manifest shape, skill entry point, script invocation). -## 4. Eval refresh — publish current-version metrics (from MAINTENANCE › Docs freshness; ROADMAP-1.0 H4 leftover) +## 4. Eval refresh — publish current-version metrics (from MAINTENANCE › Docs freshness; ROADMAP-1.0 H4 leftover) — ✅ DONE -`eval/run_eval.py` still benchmarks **0.6 vs 0.7**. The current-version suites -(`test_features` et al.) prove correctness but publish no *metrics*. +`run_eval.py` was rewritten from the stale 0.6-vs-0.7 A/B into the current-version +metrics harness (retrieval / dedup / write-safety / durability); numbers published in +the README. Re-run it per release and refresh the README table. -- [ ] Refresh `run_eval.py` to measure the current version: retrieval precision/recall - (BM25 + freshness/status fusion vs keyword baseline), duplicate rate with the - gate on/off, offline-queue durability, silent-write-failure rate. -- [ ] Publish the numbers in the README (replacing the 0.6-vs-0.7 claims) and archive - the old A/B harness docs as historical. +- [x] Refresh `run_eval.py` to measure the current version — **done 2026-07-03**: + retrieval P@5/R@5/MRR + session/journal answerability + freshness top-1 (vs a + keyword/entities-only baseline), duplicate rate gate-on/off, silent-write-failure + rate under fault injection, offline-queue durability. `results/latest.json`. +- [x] Publish the numbers in the README and mark the old A/B harness historical + (git history) — **done 2026-07-03**. ## 5. Repo hygiene odds & ends -- [ ] Retire `ROADMAP-1.0.md` (fully shipped; keep in git history, drop from the tree) - once this roadmap and the README version table carry the story. +- [x] Retire `ROADMAP-1.0.md` (fully shipped; in git history) — **done 2026-07-03**. - [ ] Root README "Repository layout" section updated for the post-2.0 tree (no versioned zips, no codex tree, skills-only plugin). - [ ] Confirm `echo-improvements-prompt.md` (the 1.5.0 field report) either moves to a diff --git a/echo-memory-1.5.1.plugin b/echo-memory-1.5.1.plugin index a036c58fbe73ddf4f71c779dca9854f330b5f4ce..af302d868c729d415869d7da34f9afa695e7dd61 100644 GIT binary patch delta 2163 zcmYM#X*iS%8wYTnv1S?GvSuHqn(XVOtjS*XWl+|#i$NNU$9wEsI*G@YNt~I=TGr-_ z)TtTENQ8qSr6D5bgzWL^(skaCzyJTfuj~GHQ%m@UOZbwT*@ZigSc`mmxx>ZA7AOcs zr2z8K3k1Lk`d^RmkHuoIM)>1YBXM}Qr3fZXvIO8W4Sg%B=ceRU=u^W+iX~E0`O*D9KI(z`OTQUimPmNK zRK&nfqIZ4-os(I_GK>Cbx;odHYUIP&#)=6tM4nCC>$ zrwTB)ZVe{UN?BfrC+oHl8IQ7w;02_^g*TmsiC*@|aXO5XfEEa;iRO^fQ_txv+qn6Z zl7VkeGCTDy{1IaehqqBO=4r-8y{dEppqiRv)H5R}v#1s=-SQJvSPt1ejZenxCJY|* zX%<5?sJ0_?CcydxlfdR&P<>RtQ^!*pAgW?{;?7t&{D%rUWDpr9`5 z&ccJT$16_U7ddAn zBQwd%{=L5osV_0I(-0;%4X)P1@D!ZZKBhdUrtr#HADFOwE^iv5XRTY_bBU#0o8BEC z6^aUN1CYkdm4%Q!-zZ$P4#^f==sABkFT^2Fy-lMwceN|$ibcKhv{KN;I5z1>Ebg%LbMr(INM)+h;*~*DVuR=U&M^tIRcf zMxQ17d()Llh_Nz#m~OWV%rNo1?N9@13r4`rOFStJZ%OnJ(L-yDZ6$-C1!7WV)0|yu z(CVqy$D??fp4O&$o-%|R95pLqDkE;srNp}G;b&RL71yoa1ns>VkF$GAM4BAYaWPPq zADaubsIqS0B>pjhS!yaTwu&|?Rc`E;A>O_VHHz9(O+sOt!Fa&KAgvtgZ$7Q-rZ(J@QV2h zmmDKxJW|N@Et4xIv(jh1+C)B`&)&huRh$(TMe^6J@a{%TblA6Trg1m~DiFYO8 znXSz-AT8RPNcbkzk9e%SugRqfo^AA6GupW$+FiY5lhMNg=1QcP?-P#)>OA<^z?*1E zg%Jy*byPcDVh6Smt*rK0n%GjV0~)ovS~-CEtJb|YS7j0Cw0WP69+aXkG?57aLaZ$eF5XO3NFNmCzw0-|W4v)B(&l*A zl+eH>r&)FIrf9RDUDNR62yPBl#Z^zk);S((HOHAr-|W!ltLZ5g?B$sZ`?hg5yGzE` zNE8DwB5C^UlZuBf;)9$&CjHZ^-Gq9b8%7^JsCmAjhngfIDcF%cvjBW@v$*>mR`q7i zUKww^qgGF7)`VtK`gp-z4f)p(t5SLE z|NGOP`EY$R6|!J{)4il&shOA+jNca4v}AMOMIyIC?5zizbb^!DAPc^iisZFat;?J3 z_!22AoI{gQqcp1T8u+~%-_Ph}M#(3ChQXW8)3@r=-ZB zFlb+6Dy~o9rw=2cVE1dmbAwcNDRuqMNEcMp%wjcFVXd6m7jd zKD1RNI$R*tTo9q2MsFw4nsf|Z8P!SHvb=K*mr-TTgDOWOUUL3!(I z`Mh@)DSkeoLb9Jt&}PCOe+$=VO*2*9Q3?MssrT_6 zjc=SJA}}&C3vew(wPk7YK$HZ+F$c4~TNipmdF3wlLg#M#%?%kfxY~({%?m0Di(-Ac z*}PIbhDP%%4v~3Jh#LEn4__y^h-p+LQgyF8i&0|I+n0Zjl6Kyp2R0t9sf>cE8LAr`vWv9bLRg6#a$ delta 2194 zcmYk-c{G#_8wPNmv5#dir0hx}`!*(xWGRFq8cT`BzLWKh8I6+cq#pa0ZJKwiF|tNk z8Y277YmS`~q9Wt-oz8LmdHv3HU;o}i>CpCcXvAB3?iRLwlj`VoCOW#SoM1#GAP#;J z0T^)@A8$;czrT|=CRE-xG#owYJxSpUA4RXPUVdUMIe{$Er$4P#x_Y6DUP#k`PYA9r zp4gSFrilE|!!Eg2NWaH@(1tY}ozb;vE*y@OpZC2#VzXZ$iVYrM#8s*9>P-sQJgW=b zE$I)rJhCuCGIHteedN`x+b^%OQ^~sCsg~q*dmK@MAYV_A zX$kmZpnr5L;;Ft2A;rsojra%GkEP` zjN}1|@7Y%Ro?|I5=?Tp1L;IY{{rIlVW-T6~4&!c1yA>jlE`oF<9+0AjVb%)Xx=vq5 z-ZQxND9H1Rg4@JcMyIPDxV-=rZr_3C7c(G}5re&ExhVUKklOY$E*9ot+BVj?Rmmx? z6Qi--Y9xP+M=Q=X7u)V>y(Bm^_GeYjZ6n*o{wz5r+10;hB4FL4Mq==7ZzM0&;f-q+ z$7?ejNvd3ZZ7v4-f$Yoj$%07bmtxWs-F&FpHjVdzYqOvvG#Qg{{VriDkv^rk8cM9) zSL=l8IPDy+5o_{~>`^X*_1KJlDp$!BEv-ySF8obV{r#&keSHbnUe2%>UnG+!T##h0 zkE*-`NE#AL2qcw|c5jyod}gg*s;)pnTB49KtNa`!HOs5p*(ynW%%+2pvZGmggUc%! zNEop$yu;<4geK0TIbdq$Zg7L70sC^n?q#uA$pt?o>k|Eh9-Srn%*V55zKBQR$6t&@ z3iMvpe=Kd^Slv!a7!UE=r2P}<_q&_wj?DQ|C)@e^LJb_|<)%W5S!y_e4! zg)@AwzZVwvMe#VZRzHwdsr?rkrVx0MsyE3?#HrLX?ICQ#&XEy>Q=D?&l^Rc@FGWjl z6dU+q!d??u07{$Uzolrn*CL&~}igxRRhUZ>(?XkEiRh-K;+*~@4;{8$ToHEKocB){%Eh&xB zi8i~=s+2wd*6)?5M#-QBkKp))c~0hEd0G9ps3|y$s+RYY_y%}}cf-i~=ymQ#TdD5)&OLoPw-aIL zawsap)%ldphW&C+g4~({7&|zKP$nJ(e?nn4%@yqIQhb-c>1&hN9G1hBnQmv^EmBG= zQp36H91_bCcQk!$J=G8^kB#Hz5orf*j#p;JtjaiTMHW+kRCjL6k#frMRtu+t(u$s9 z(wcbxIj9%yPVAUr!aUgDQ%h{=igmUafR|Y5)V;8eME)pRYrvY!X$a>nnfNMecRg(* zDO%#KNL*A;Hc8Xpo-La@a?VI0!63#d4&0cR`HIBp$UXgM7t&GZ4a&B+BEq%9(jLvh z^j)p5*PV~b4Qe$XM|01!@$;MfXT8>%k|6TxrlXqEEtAMyr}qpRhLc9-J9ldNIrGab zH9aj-#LgFt3k_`b*?KPQ>WC_cRqbm{IO@dS(++CTnnGpdiq!IsM^~UF;~|yU5-M)z z)D3qN!G(p5C+q(_PaF|U4}OWA@F*msL=}-YH%s`y0AH=1pz*D0j1nq&VKbkmsJlg- zsf42BFnaDvH*ybiGf;e?M(+ylGwI{#K|2(zesEou3P+EXAKb1Ny91B} zQ#t@epmFg3K>}Rt08{|z?t1*kxxMNWv>CqNbQ?Cc34d^*}`RyY_+N6C#of@Pe*AM`I3|Czy~4aDdG{0E{W50-&R#`yWH51bF}e diff --git a/echo-memory.plugin b/echo-memory.plugin index a036c58fbe73ddf4f71c779dca9854f330b5f4ce..af302d868c729d415869d7da34f9afa695e7dd61 100644 GIT binary patch delta 2163 zcmYM#X*iS%8wYTnv1S?GvSuHqn(XVOtjS*XWl+|#i$NNU$9wEsI*G@YNt~I=TGr-_ z)TtTENQ8qSr6D5bgzWL^(skaCzyJTfuj~GHQ%m@UOZbwT*@ZigSc`mmxx>ZA7AOcs zr2z8K3k1Lk`d^RmkHuoIM)>1YBXM}Qr3fZXvIO8W4Sg%B=ceRU=u^W+iX~E0`O*D9KI(z`OTQUimPmNK zRK&nfqIZ4-os(I_GK>Cbx;odHYUIP&#)=6tM4nCC>$ zrwTB)ZVe{UN?BfrC+oHl8IQ7w;02_^g*TmsiC*@|aXO5XfEEa;iRO^fQ_txv+qn6Z zl7VkeGCTDy{1IaehqqBO=4r-8y{dEppqiRv)H5R}v#1s=-SQJvSPt1ejZenxCJY|* zX%<5?sJ0_?CcydxlfdR&P<>RtQ^!*pAgW?{;?7t&{D%rUWDpr9`5 z&ccJT$16_U7ddAn zBQwd%{=L5osV_0I(-0;%4X)P1@D!ZZKBhdUrtr#HADFOwE^iv5XRTY_bBU#0o8BEC z6^aUN1CYkdm4%Q!-zZ$P4#^f==sABkFT^2Fy-lMwceN|$ibcKhv{KN;I5z1>Ebg%LbMr(INM)+h;*~*DVuR=U&M^tIRcf zMxQ17d()Llh_Nz#m~OWV%rNo1?N9@13r4`rOFStJZ%OnJ(L-yDZ6$-C1!7WV)0|yu z(CVqy$D??fp4O&$o-%|R95pLqDkE;srNp}G;b&RL71yoa1ns>VkF$GAM4BAYaWPPq zADaubsIqS0B>pjhS!yaTwu&|?Rc`E;A>O_VHHz9(O+sOt!Fa&KAgvtgZ$7Q-rZ(J@QV2h zmmDKxJW|N@Et4xIv(jh1+C)B`&)&huRh$(TMe^6J@a{%TblA6Trg1m~DiFYO8 znXSz-AT8RPNcbkzk9e%SugRqfo^AA6GupW$+FiY5lhMNg=1QcP?-P#)>OA<^z?*1E zg%Jy*byPcDVh6Smt*rK0n%GjV0~)ovS~-CEtJb|YS7j0Cw0WP69+aXkG?57aLaZ$eF5XO3NFNmCzw0-|W4v)B(&l*A zl+eH>r&)FIrf9RDUDNR62yPBl#Z^zk);S((HOHAr-|W!ltLZ5g?B$sZ`?hg5yGzE` zNE8DwB5C^UlZuBf;)9$&CjHZ^-Gq9b8%7^JsCmAjhngfIDcF%cvjBW@v$*>mR`q7i zUKww^qgGF7)`VtK`gp-z4f)p(t5SLE z|NGOP`EY$R6|!J{)4il&shOA+jNca4v}AMOMIyIC?5zizbb^!DAPc^iisZFat;?J3 z_!22AoI{gQqcp1T8u+~%-_Ph}M#(3ChQXW8)3@r=-ZB zFlb+6Dy~o9rw=2cVE1dmbAwcNDRuqMNEcMp%wjcFVXd6m7jd zKD1RNI$R*tTo9q2MsFw4nsf|Z8P!SHvb=K*mr-TTgDOWOUUL3!(I z`Mh@)DSkeoLb9Jt&}PCOe+$=VO*2*9Q3?MssrT_6 zjc=SJA}}&C3vew(wPk7YK$HZ+F$c4~TNipmdF3wlLg#M#%?%kfxY~({%?m0Di(-Ac z*}PIbhDP%%4v~3Jh#LEn4__y^h-p+LQgyF8i&0|I+n0Zjl6Kyp2R0t9sf>cE8LAr`vWv9bLRg6#a$ delta 2194 zcmYk-c{G#_8wPNmv5#dir0hx}`!*(xWGRFq8cT`BzLWKh8I6+cq#pa0ZJKwiF|tNk z8Y277YmS`~q9Wt-oz8LmdHv3HU;o}i>CpCcXvAB3?iRLwlj`VoCOW#SoM1#GAP#;J z0T^)@A8$;czrT|=CRE-xG#owYJxSpUA4RXPUVdUMIe{$Er$4P#x_Y6DUP#k`PYA9r zp4gSFrilE|!!Eg2NWaH@(1tY}ozb;vE*y@OpZC2#VzXZ$iVYrM#8s*9>P-sQJgW=b zE$I)rJhCuCGIHteedN`x+b^%OQ^~sCsg~q*dmK@MAYV_A zX$kmZpnr5L;;Ft2A;rsojra%GkEP` zjN}1|@7Y%Ro?|I5=?Tp1L;IY{{rIlVW-T6~4&!c1yA>jlE`oF<9+0AjVb%)Xx=vq5 z-ZQxND9H1Rg4@JcMyIPDxV-=rZr_3C7c(G}5re&ExhVUKklOY$E*9ot+BVj?Rmmx? z6Qi--Y9xP+M=Q=X7u)V>y(Bm^_GeYjZ6n*o{wz5r+10;hB4FL4Mq==7ZzM0&;f-q+ z$7?ejNvd3ZZ7v4-f$Yoj$%07bmtxWs-F&FpHjVdzYqOvvG#Qg{{VriDkv^rk8cM9) zSL=l8IPDy+5o_{~>`^X*_1KJlDp$!BEv-ySF8obV{r#&keSHbnUe2%>UnG+!T##h0 zkE*-`NE#AL2qcw|c5jyod}gg*s;)pnTB49KtNa`!HOs5p*(ynW%%+2pvZGmggUc%! zNEop$yu;<4geK0TIbdq$Zg7L70sC^n?q#uA$pt?o>k|Eh9-Srn%*V55zKBQR$6t&@ z3iMvpe=Kd^Slv!a7!UE=r2P}<_q&_wj?DQ|C)@e^LJb_|<)%W5S!y_e4! zg)@AwzZVwvMe#VZRzHwdsr?rkrVx0MsyE3?#HrLX?ICQ#&XEy>Q=D?&l^Rc@FGWjl z6dU+q!d??u07{$Uzolrn*CL&~}igxRRhUZ>(?XkEiRh-K;+*~@4;{8$ToHEKocB){%Eh&xB zi8i~=s+2wd*6)?5M#-QBkKp))c~0hEd0G9ps3|y$s+RYY_y%}}cf-i~=ymQ#TdD5)&OLoPw-aIL zawsap)%ldphW&C+g4~({7&|zKP$nJ(e?nn4%@yqIQhb-c>1&hN9G1hBnQmv^EmBG= zQp36H91_bCcQk!$J=G8^kB#Hz5orf*j#p;JtjaiTMHW+kRCjL6k#frMRtu+t(u$s9 z(wcbxIj9%yPVAUr!aUgDQ%h{=igmUafR|Y5)V;8eME)pRYrvY!X$a>nnfNMecRg(* zDO%#KNL*A;Hc8Xpo-La@a?VI0!63#d4&0cR`HIBp$UXgM7t&GZ4a&B+BEq%9(jLvh z^j)p5*PV~b4Qe$XM|01!@$;MfXT8>%k|6TxrlXqEEtAMyr}qpRhLc9-J9ldNIrGab zH9aj-#LgFt3k_`b*?KPQ>WC_cRqbm{IO@dS(++CTnnGpdiq!IsM^~UF;~|yU5-M)z z)D3qN!G(p5C+q(_PaF|U4}OWA@F*msL=}-YH%s`y0AH=1pz*D0j1nq&VKbkmsJlg- zsf42BFnaDvH*ybiGf;e?M(+ylGwI{#K|2(zesEou3P+EXAKb1Ny91B} zQ#t@epmFg3K>}Rt08{|z?t1*kxxMNWv>CqNbQ?Cc34d^*}`RyY_+N6C#of@Pe*AM`I3|Czy~4aDdG{0E{W50-&R#`yWH51bF}e diff --git a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_concurrency.py b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_concurrency.py index de03332..4f30a24 100644 --- a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_concurrency.py +++ b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_concurrency.py @@ -14,7 +14,7 @@ advisory lock around a write burst, with crash-safe release, and (b) an atomic index-update that re-reads UNDER the lock before saving, so concurrent captures merge instead of clobber. -ACCEPTANCE (see ROADMAP-1.0.md > H3): +ACCEPTANCE (v1.0 roadmap H3, shipped; roadmap retired — see git history): - two concurrent captures both land in the index (no lost entries) - lock auto-released on normal AND error exit """ diff --git a/eval/README.md b/eval/README.md index d0ccab1..dfd4da8 100644 --- a/eval/README.md +++ b/eval/README.md @@ -1,9 +1,23 @@ -# echo-memory eval — 0.6 vs 0.7 A/B harness +# echo-memory eval — current-version metrics harness -A reproducible, credential-free A/B comparison of the plugin **before** (0.6: raw-curl -recipes from `SKILL.md`) and **after** (the shipped `scripts/echo.py` client) the -hardening work. It quantifies the claims in the comparative analysis: token cost of the -I/O layer, and the rate of **silent write failures**. +A reproducible, credential-free eval of the shipped plugin against the deterministic +mock REST API. `run_eval.py` measures the four things the plugin claims (results print +as a table and land in `results/latest.json`): + +1. **Retrieval** — hybrid recall (BM25 over entities **and** sessions/journal, fused + with freshness/status priors) vs a keyword-only baseline modeling pre-1.5 recall + (per-word substring ranking over entity notes only). Precision@5 / recall@5 / MRR, + session/journal answerability, freshness top-1. +2. **Dedup** — duplicate notes created replaying a capture stream with the pre-write + gate ON (default) vs OFF (`ECHO_DUP_GATE=99` ≈ pre-1.5 warn-only), plus legitimate + captures wrongly blocked (must be 0). +3. **Write safety** — silent write failures under fault injection (transient 503, + phantom PUT, PATCH to a missing heading, replayed append). Must be 0; must be loud. +4. **Durability** — writes during an outage queue, replay exactly once, never + duplicate on re-flush. + +> The original **0.6-vs-0.7 A/B harness** (token cost + silent-failure comparison that +> motivated the 0.7 client) lives in git history — `git log -- eval/run_eval.py`. ## Current test suites (run these for any change) diff --git a/eval/results/latest.json b/eval/results/latest.json index c9ee76f..75f3678 100644 --- a/eval/results/latest.json +++ b/eval/results/latest.json @@ -1,154 +1,46 @@ { - "params": { - "port": 8799, - "cpt": 4.0, - "recovery": 1500, - "detect_cost": 80 - }, - "rows": [ - { - "scenario": "agent-log-missing-heading", - "fault": "bad heading -> 400 invalid-target", - "06": { - "emit_tokens": 82, - "claimed_ok": true, - "detected": false, - "persisted": false, - "duplicates": 0, - "silent_failure": true - }, - "07": { - "emit_tokens": 22, - "claimed_ok": false, - "detected": true, - "persisted": false, - "duplicates": 0, - "silent_failure": false - } + "version": "1.5.1", + "date": "2026-07-03", + "retrieval": { + "current (hybrid+priors, full corpus)": { + "precision_at_5": 0.25, + "recall_at_5": 1.0, + "mrr": 1.0, + "session_journal_queries_answered": "2/2", + "freshness_top1_correct": true }, - { - "scenario": "scope-switch", - "fault": "(none)", - "06": { - "emit_tokens": 84, - "claimed_ok": true, - "detected": false, - "persisted": true, - "duplicates": 0, - "silent_failure": false - }, - "07": { - "emit_tokens": 24, - "claimed_ok": true, - "detected": false, - "persisted": true, - "duplicates": 0, - "silent_failure": false - } - }, - { - "scenario": "inbox-capture-replayed", - "fault": "duplicate attempt (retry/replay)", - "06": { - "emit_tokens": 125, - "claimed_ok": true, - "detected": false, - "persisted": true, - "duplicates": 1, - "silent_failure": true - }, - "07": { - "emit_tokens": 33, - "claimed_ok": true, - "detected": false, - "persisted": true, - "duplicates": 0, - "silent_failure": false - } - }, - { - "scenario": "session-log-flaky-network", - "fault": "one-time 503 (transient)", - "06": { - "emit_tokens": 64, - "claimed_ok": true, - "detected": false, - "persisted": false, - "duplicates": 0, - "silent_failure": true - }, - "07": { - "emit_tokens": 17, - "claimed_ok": true, - "detected": false, - "persisted": true, - "duplicates": 0, - "silent_failure": false - } - }, - { - "scenario": "heartbeat-phantom-write", - "fault": "accepted-but-not-persisted", - "06": { - "emit_tokens": 61, - "claimed_ok": true, - "detected": false, - "persisted": false, - "duplicates": 0, - "silent_failure": true - }, - "07": { - "emit_tokens": 14, - "claimed_ok": false, - "detected": true, - "persisted": false, - "duplicates": 0, - "silent_failure": false - } - }, - { - "scenario": "cold-start-load-6-reads", - "fault": "(none)", - "06": { - "emit_tokens": 192, - "claimed_ok": true, - "detected": false, - "persisted": true, - "duplicates": 0, - "silent_failure": false - }, - "07": { - "emit_tokens": 64, - "claimed_ok": true, - "detected": false, - "persisted": true, - "duplicates": 0, - "silent_failure": false - } - } - ], - "totals": { - "0.6": { - "gen_tokens": 608, - "silent_failures": 4, - "detected_failures": 0, - "duplicates": 1, - "effective_tokens": 6608 - }, - "0.7": { - "gen_tokens": 174, - "silent_failures": 0, - "detected_failures": 2, - "duplicates": 0, - "effective_tokens": 334 + "baseline (keyword, entities only)": { + "precision_at_5": 0.2, + "recall_at_5": 0.75, + "mrr": 0.75, + "session_journal_queries_answered": "0/2", + "freshness_top1_correct": false } }, - "silent_error_free": { - "0.6": "1/5", - "0.7": "5/5" + "dedup": { + "gate ON (v1.5.1 default)": { + "duplicate_notes_created": 0, + "legitimate_captures_blocked": 0, + "referring_attempts": 3, + "legit_attempts": 2 + }, + "gate OFF (pre-1.5, warn-only)": { + "duplicate_notes_created": 3, + "legitimate_captures_blocked": 0, + "referring_attempts": 3, + "legit_attempts": 2 + } }, - "writes_persisted": { - "0.6": "2/5", - "0.7": "3/5" + "write_safety": { + "fault_scenarios": 4, + "silent_write_failures": 0, + "failures_surfaced_loud": 2 + }, + "durability": { + "writes_during_outage": 3, + "reported_queued": 3, + "landed_after_flush": 3, + "lost": 0, + "duplicated_on_reflush": 0 } } \ No newline at end of file diff --git a/eval/run_eval.py b/eval/run_eval.py index 9c5f342..e068747 100644 --- a/eval/run_eval.py +++ b/eval/run_eval.py @@ -1,41 +1,52 @@ #!/usr/bin/env python3 -"""run_eval.py — A/B eval of echo-memory 0.6 (raw curl, documented recipes) vs -0.7 (the shipped echo.py client) over representative memory operations, with -fault injection. +"""run_eval.py — current-version metrics for echo-memory (v1.5+). -Design ------- -* A mock Obsidian REST API (mock_olrapi.py) gives deterministic behavior + faults, - so no credentials are needed and the real vault is never touched. -* The 0.7 side runs the ACTUAL shipped scripts/echo.py (status-checked, retry, verify, - idempotent append). -* The 0.6 side faithfully models the documented raw-curl recipes: it performs the - same HTTP but does NOT inspect status, does NOT retry, does NOT verify, and does - NOT dedupe — exactly what SKILL.md 0.6 told the model to emit. -* Ground truth is read back independently from the mock after each method, so a - "silent failure" (method reported success but the vault is actually wrong, and - nobody noticed) is detected regardless of what the method claimed. -* Each (scenario, method) runs against a freshly reset + re-seeded server, so faults - (e.g. one-time 503) are identical for both methods. +Replaces the historical 0.6-vs-0.7 A/B harness (see git history for that version). +Measures the four things the plugin now claims, against the deterministic mock +REST API (mock_olrapi.py) — no credentials, no live vault: -Metrics -------- -* gen_tokens : output tokens the MODEL must generate for the op (len(emitted)/CPT). -* silent_failure : claimed success BUT ground truth is wrong (lost write or duplicate). -* detected : the method surfaced the failure (loud, retryable) instead of hiding it. -* effective_tokens = gen_tokens + silent_failures*RECOVERY + detected*DETECT_COST - (RECOVERY/DETECT_COST are labeled assumptions, tune via env). + 1. RETRIEVAL — hybrid recall (BM25 over entities + sessions/journal, fused with + freshness/status priors) vs a keyword-only baseline modeling the + pre-1.5 recall (per-word substring ranking over ENTITY notes only). + Metrics: precision@5, recall@5, MRR, session/journal queries + answered, freshness top-1 correctness. + 2. DEDUP — duplicate notes created replaying a capture stream with the + pre-write gate ON (default) vs OFF (ECHO_DUP_GATE=99, i.e. the + pre-1.5 warn-only behavior). Also counts legitimate captures + wrongly blocked (must be 0). + 3. WRITE SAFETY— silent write failures (claimed success but ground truth wrong) + across fault injection: transient 503, phantom PUT, PATCH to a + missing heading, replayed append. Must be 0; failures must be LOUD. + 4. DURABILITY — writes issued while the vault is unreachable must queue, replay + exactly once when it returns, and never duplicate on re-flush. -Usage: python3 run_eval.py [--port 8799] [--cpt 4] [--recovery 1500] [--detect-cost 80] +Ground truth is always read back independently via the mock's __debug__ endpoint. +Results print as a table and land in results/latest.json. + +Usage: python3 run_eval.py [--port 8797] """ -import os, sys, json, time, subprocess, tempfile, argparse, urllib.request, urllib.error +from __future__ import annotations + +import argparse +import json +import os +import re +import subprocess +import sys +import tempfile +import time +import urllib.error +import urllib.request from pathlib import Path HERE = Path(__file__).resolve().parent -ECHO = HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts" / "echo.py" -KEY = "test-key-not-a-real-secret" +SCRIPTS = HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts" +ECHO = SCRIPTS / "echo.py" +SWEEP = SCRIPTS / "sweep.py" +KEY = "test-key-not-a-real-secret" +TODAY = "2026-07-03" + -# ----- tiny HTTP helpers (used for setup, ground truth, and the 0.6 model) ----- def http(method, url, body=None, headers=None): data = body.encode() if isinstance(body, str) else body req = urllib.request.Request(url, data=data, method=method, @@ -45,259 +56,313 @@ def http(method, url, body=None, headers=None): return r.status, r.read().decode("utf-8", "replace") except urllib.error.HTTPError as e: return e.code, e.read().decode("utf-8", "replace") - except Exception as e: + except Exception as e: # noqa: BLE001 return 0, str(e) -class Eval: - def __init__(self, base, cpt): - self.base, self.cpt = base, cpt - def reset(self): http("POST", f"{self.base}/__debug__reset") - def seed(self, path, content): http("PUT", f"{self.base}/vault/{path}", content) + +class Harness: + def __init__(self, base, state_dir): + self.base = base + self.state_dir = state_dir + + def reset(self): + http("POST", f"{self.base}/__debug__reset") + + def seed(self, path, content): + http("PUT", f"{self.base}/vault/{path}", content) + def ground(self, path): - st, body = http("GET", f"{self.base}/__debug__?path={path}") + _, body = http("GET", f"{self.base}/__debug__?path={path}") return None if body == "<>" else body - def toks(self, text): return round(len(text) / self.cpt) - def echo(self, *args): - env = dict(os.environ, ECHO_BASE=self.base, ECHO_KEY=KEY, ECHO_VERIFY="1", ECHO_LOCK_TTL="900") - p = subprocess.run([sys.executable, str(ECHO), *args], capture_output=True, text=True, env=env) - return p.returncode -# ---- faithful 0.6 recipe text (what the model emitted), for token accounting ---- -def recipe_get(path): - return (f'curl -s -H "Authorization: Bearer {KEY}" ' - f'"https://echoapi.alwisp.com/vault/{path}"') -def recipe_put(path): - return (f"cat > /tmp/obs_file.md << 'EOF'\n\nEOF\n\n" - f'curl -s -X PUT -H "Authorization: Bearer {KEY}" ' - f'-H "Content-Type: text/markdown" --data-binary @/tmp/obs_file.md ' - f'"https://echoapi.alwisp.com/vault/{path}"') -def recipe_post(path): - return (f"cat > /tmp/obs_entry.md << 'EOF'\n\nEOF\n\n" - f'curl -s -X POST -H "Authorization: Bearer {KEY}" ' - f'-H "Content-Type: text/markdown" --data-binary @/tmp/obs_entry.md ' - f'"https://echoapi.alwisp.com/vault/{path}"') -def recipe_patch(path, target): - return (f"cat > /tmp/obs_patch.md << 'EOF'\n\nEOF\n\n" - f'curl -s -X PATCH -H "Authorization: Bearer {KEY}" ' - f'-H "Operation: append" -H "Target-Type: heading" -H "Target: {target}" ' - f'-H "Content-Type: text/markdown" --data-binary @/tmp/obs_patch.md ' - f'"https://echoapi.alwisp.com/vault/{path}"') + def run(self, script, *args, env_extra=None, base=None): + env = dict(os.environ, ECHO_BASE=base or self.base, ECHO_KEY=KEY, + ECHO_VERIFY="1", ECHO_TODAY=TODAY, ECHO_STATE_DIR=self.state_dir, + **(env_extra or {})) + return subprocess.run([sys.executable, str(script), *args], + capture_output=True, text=True, env=env) -def tmpfile(content): - f = tempfile.NamedTemporaryFile("w", suffix=".md", delete=False); f.write(content); f.close() - return f.name -# --------------------------------------------------------------------------- -# Scenarios. Each defines seed(), and a run for each method that returns: -# {emit: , claimed_ok: bool, detected: bool} -# plus a ground-truth check producing {persisted, duplicates}. -# --------------------------------------------------------------------------- -def scenarios(ev): - out = [] +def note(fm_type, title, body, status="active", created=TODAY, updated=TODAY, tags=None): + tags_s = "[" + ", ".join(tags or [fm_type]) + "]" + return (f"---\ntype: {fm_type}\nstatus: {status}\ncreated: {created}\n" + f"updated: {updated}\ntags: {tags_s}\nagent_written: true\nsource_notes: []\n---\n\n" + f"# {title}\n\n{body}\n\n## Related\n") - # S1 — agent-log append where the heading is MISSING (-> 400 invalid-target) - def s1(): - path, tgt = "journal/daily/2026-06-19.md", "2026-06-19::Agent Log" - line = "- 2026-06-19: EVALMARK-s1" - def seed(): ev.seed(path, "---\ntype: daily-note\n---\n\n# 2026-06-19\n\n## Notes\n") - def m06(): - http("PATCH", f"{ev.base}/vault/{path}", line, - {"Operation": "append", "Target-Type": "heading", "Target": tgt}) # status ignored - return {"emit": recipe_patch(path, tgt), "claimed_ok": True, "detected": False} - def m07(): - rc = ev.echo("patch", path, "append", "heading", tgt, tmpfile(line)) - return {"emit": f'"$ECHO" patch {path} append heading "{tgt}" /tmp/x.md', - "claimed_ok": rc == 0, "detected": rc != 0} - def gt(): - c = ev.ground(path) or "" - return {"persisted": "EVALMARK-s1" in c, "duplicates": max(0, c.count("EVALMARK-s1") - 1)} - return dict(name="agent-log-missing-heading", fault="bad heading -> 400 invalid-target", - seed=seed, m06=m06, m07=m07, gt=gt) - out.append(s1()) - # S2 — scope switch (no fault; pure token comparison on a PATCH replace) - def s2(): - path, tgt = "_agent/context/current-context.md", "Current Context::Scope" - def seed(): ev.seed(path, "---\ntype: context-bundle\n---\n\n# Current Context\n\n## Scope\nold scope\n") - def m06(): - http("PATCH", f"{ev.base}/vault/{path}", "EVALMARK-s2 new scope", - {"Operation": "replace", "Target-Type": "heading", "Target": tgt}) - return {"emit": recipe_patch(path, tgt), "claimed_ok": True, "detected": False} - def m07(): - rc = ev.echo("patch", path, "replace", "heading", tgt, tmpfile("EVALMARK-s2 new scope")) - return {"emit": f'"$ECHO" patch {path} replace heading "{tgt}" /tmp/x.md', - "claimed_ok": rc == 0, "detected": rc != 0} - def gt(): - c = ev.ground(path) or "" - return {"persisted": "EVALMARK-s2" in c, "duplicates": 0} - return dict(name="scope-switch", fault="(none)", seed=seed, m06=m06, m07=m07, gt=gt) - out.append(s2()) +# ---------------------------------------------------------------- 1. RETRIEVAL +ENTITY_CORPUS = { + "resources/people/maria-lopez.md": note( + "person", "Maria Lopez", "Runs procurement at Vantage Systems. Prefers email over calls."), + "resources/companies/vantage-systems.md": note( + "company", "Vantage Systems", "Enterprise procurement platform vendor; the contract renews in Q3."), + "resources/concepts/bm25-ranking.md": note( + "concept", "BM25 Ranking", "Lexical retrieval scoring with term frequency saturation and document length normalization."), + "resources/concepts/graph-expansion.md": note( + "concept", "Graph Expansion", "Expanding a hit into its linked neighbourhood along Related links and source notes."), + "projects/active/website-redesign.md": note( + "project", "Website Redesign", + "Rebuild the marketing site. Navigation cleanup and new hero copy.", + status="active", created="2026-06-25", updated="2026-07-01"), + "projects/archived/old-website.md": note( + "project", "Old Website", + "Legacy marketing site. Navigation used nested dropdown menus. Navigation was the top complaint.", + status="archived", created="2024-03-01", updated="2024-08-10"), + "_agent/memory/semantic/tooling-preferences.md": note( + "semantic-memory", "Tooling Preferences", + "The operator prefers uv over pip for Python tooling, and ruff for linting."), +} +TIME_SERIES_CORPUS = { + "_agent/sessions/2026-06-20-1000-vantage-contract-review.md": note( + "session-log", "Vantage contract review", + "Reviewed the Vantage contract. Decided to renegotiate the SLA penalty clause before renewal.", + status="complete", created="2026-06-20", updated="2026-06-20", tags=["session-log"]), + "journal/daily/2026-06-20.md": ( + "---\ntype: daily-note\ncreated: 2026-06-20\nupdated: 2026-06-20\ntags: [daily]\n---\n\n" + "# 2026-06-20\n\n## Agent Log\n- 2026-06-20: renegotiation kickoff email sent to Vantage\n"), +} - # S3 — inbox capture issued TWICE (retry/replay): dedup vs duplicate line - def s3(): - path = "inbox/captures/inbox.md"; line = "- 2026-06-19: EVALMARK-s3" - def seed(): ev.seed(path, "# Inbox\n") - def m06(): - http("POST", f"{ev.base}/vault/{path}", line + "\n") # no idempotency check - http("POST", f"{ev.base}/vault/{path}", line + "\n") - return {"emit": recipe_post(path) + "\n# (repeated on retry)\n" + recipe_post(path), - "claimed_ok": True, "detected": False} - def m07(): - rc1 = ev.echo("append", path, line) - rc2 = ev.echo("append", path, line) # second is skipped by read-before-POST - return {"emit": f'"$ECHO" append {path} "{line}"\n"$ECHO" append {path} "{line}"', - "claimed_ok": rc1 == 0 and rc2 == 0, "detected": False} - def gt(): - c = ev.ground(path) or "" - return {"persisted": "EVALMARK-s3" in c, "duplicates": max(0, c.count("EVALMARK-s3") - 1)} - return dict(name="inbox-capture-replayed", fault="duplicate attempt (retry/replay)", - seed=seed, m06=m06, m07=m07, gt=gt) - out.append(s3()) +# (query, gold paths, only answerable from sessions/journal) +QUERIES = [ + ("uv or pip for python tooling", {"_agent/memory/semantic/tooling-preferences.md"}, False), + ("procurement contact", {"resources/people/maria-lopez.md", + "resources/companies/vantage-systems.md"}, False), + ("SLA penalty clause decision", + {"_agent/sessions/2026-06-20-1000-vantage-contract-review.md"}, True), + ("marketing site navigation", {"projects/active/website-redesign.md", + "projects/archived/old-website.md"}, False), + ("term frequency saturation", {"resources/concepts/bm25-ranking.md"}, False), + ("contract renewal Q3", {"resources/companies/vantage-systems.md"}, False), + ("renegotiation kickoff", {"journal/daily/2026-06-20.md"}, True), + ("graph neighbourhood expansion", {"resources/concepts/graph-expansion.md"}, False), +] +FRESHNESS_QUERY = "marketing site navigation" +FRESHNESS_TOP1 = "projects/active/website-redesign.md" - # S4 — session-log PUT under a flaky network (one-time 503 then success) - def s4(): - path = "_agent/sessions/2026-06-19-1430-flaky-eval.md" - body = "---\ntype: session-log\n---\n\n# Session\nEVALMARK-s4\n" - def seed(): pass # file does not pre-exist - def m06(): - http("PUT", f"{ev.base}/vault/{path}", body) # single shot, no retry, status ignored - return {"emit": recipe_put(path), "claimed_ok": True, "detected": False} - def m07(): - rc = ev.echo("put", path, tmpfile(body)) # echo.py retries the 503 once - return {"emit": f'"$ECHO" put {path} /tmp/x.md', "claimed_ok": rc == 0, "detected": rc != 0} - def gt(): - c = ev.ground(path) or "" - return {"persisted": "EVALMARK-s4" in c, "duplicates": 0} - return dict(name="session-log-flaky-network", fault="one-time 503 (transient)", - seed=seed, m06=m06, m07=m07, gt=gt) - out.append(s4()) - # S5 — heartbeat PUT accepted-but-not-persisted (proxy hiccup): verify catches it - def s5(): - path = "_agent/heartbeat/phantom-eval.md"; body = "EVALMARK-s5 @ 2026-06-19T14:30:00Z\n" - def seed(): pass - def m06(): - http("PUT", f"{ev.base}/vault/{path}", body) # 200 returned; no read-back verify - return {"emit": recipe_put(path), "claimed_ok": True, "detected": False} - def m07(): - rc = ev.echo("put", path, tmpfile(body)) # verify GET -> 404 -> die - return {"emit": f'"$ECHO" put {path} /tmp/x.md', "claimed_ok": rc == 0, "detected": rc != 0} - def gt(): - c = ev.ground(path) or "" - return {"persisted": "EVALMARK-s5" in c, "duplicates": 0} - return dict(name="heartbeat-phantom-write", fault="accepted-but-not-persisted", - seed=seed, m06=m06, m07=m07, gt=gt) - out.append(s5()) +def keyword_baseline(query, corpus, k=5): + """Models pre-1.5 keyword recall: per-word substring counting over ENTITY note + bodies only (sessions/journal were not in the corpus before 1.5).""" + words = [w for w in re.findall(r"[a-z0-9]+", query.lower()) if len(w) > 2] + scored = [] + for path, text in corpus.items(): + body = text.lower() + s = sum(body.count(w) for w in words) + if s > 0: + scored.append((s, path)) + scored.sort(key=lambda x: (-x[0], x[1])) + return [p for _, p in scored[:k]] - # S6 — cold-start load: 6 reads (no fault; pure token comparison) - def s6(): - paths = ["_agent/echo-vault.md", "_agent/memory/semantic/operator-preferences.md", - "_agent/context/current-context.md", "_agent/heartbeat/last-session.md", - "journal/daily/2026-06-19.md", "inbox/captures/inbox.md"] - def seed(): - for p in paths: ev.seed(p, f"# {p}\nEVALMARK-s6\n") - def m06(): - for p in paths: http("GET", f"{ev.base}/vault/{p}") - return {"emit": "\n".join(recipe_get(p) for p in paths), "claimed_ok": True, "detected": False} - def m07(): - for p in paths: ev.echo("get", p) - return {"emit": "\n".join(f'"$ECHO" get {p}' for p in paths), "claimed_ok": True, "detected": False} - def gt(): return {"persisted": True, "duplicates": 0} - return dict(name="cold-start-load-6-reads", fault="(none)", seed=seed, m06=m06, m07=m07, gt=gt) - out.append(s6()) - return out +def metrics_for(ranked_by_query): + p_sum = r_sum = mrr_sum = 0.0 + ts_answered = ts_total = 0 + for (q, gold, ts_only), ranked in ranked_by_query: + top = ranked[:5] + hits = [p for p in top if p in gold] + p_sum += len(hits) / 5 + r_sum += len(hits) / len(gold) + rr = 0.0 + for i, p in enumerate(top): + if p in gold: + rr = 1.0 / (i + 1) + break + mrr_sum += rr + if ts_only: + ts_total += 1 + ts_answered += int(bool(hits)) + n = len(ranked_by_query) + return {"precision_at_5": round(p_sum / n, 3), "recall_at_5": round(r_sum / n, 3), + "mrr": round(mrr_sum / n, 3), + "session_journal_queries_answered": f"{ts_answered}/{ts_total}"} -def run_method(ev, scn, key): - ev.reset(); scn["seed"]() - res = scn[key]() - g = scn["gt"]() - # a write op "should persist"; reads (s6) and replays are judged by their own gt - bad = (not g["persisted"] and scn["name"] != "cold-start-load-6-reads") or g["duplicates"] > 0 - silent = bool(res["claimed_ok"] and bad) - return {"emit_tokens": ev.toks(res["emit"]), "claimed_ok": res["claimed_ok"], - "detected": res["detected"], "persisted": g["persisted"], - "duplicates": g["duplicates"], "silent_failure": silent} +def eval_retrieval(h): + h.reset() + h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n") + for path, text in {**ENTITY_CORPUS, **TIME_SERIES_CORPUS}.items(): + h.seed(path, text) + r = h.run(SWEEP, "--apply") + assert r.returncode == 0, f"sweep failed: {r.stdout}{r.stderr}" + + current_ranked, baseline_ranked = [], [] + fresh_top1_current = fresh_top1_baseline = None + for q, gold, ts_only in QUERIES: + r = h.run(ECHO, "recall", q, "--json", "--limit", "5") + try: + primary = [hit["path"] for hit in json.loads(r.stdout).get("primary", [])] + except Exception: # noqa: BLE001 + primary = [] + current_ranked.append(((q, gold, ts_only), primary)) + base = keyword_baseline(q, ENTITY_CORPUS) + baseline_ranked.append(((q, gold, ts_only), base)) + if q == FRESHNESS_QUERY: + fresh_top1_current = primary[0] if primary else None + fresh_top1_baseline = base[0] if base else None + + cur = metrics_for(current_ranked) + cur["freshness_top1_correct"] = fresh_top1_current == FRESHNESS_TOP1 + bas = metrics_for(baseline_ranked) + bas["freshness_top1_correct"] = fresh_top1_baseline == FRESHNESS_TOP1 + return {"current (hybrid+priors, full corpus)": cur, + "baseline (keyword, entities only)": bas} + + +# ------------------------------------------------------------------- 2. DEDUP +# Attempts referring to an EXISTING entity under another name; a create == duplicate. +DUP_ATTEMPTS = [ + ("Robert Smith", "person", "resources/people/robert-smith.md"), + ("Echo Memory", "project", "projects/active/echo-memory.md"), + ("Acme Consulting", "company", "resources/companies/acme-consulting.md"), +] +# Legitimate NEW entities; a block == false positive (must create in both modes). +LEGIT_ATTEMPTS = [ + ("Echo Review Meeting", "meeting", + f"resources/meetings/{TODAY}-echo-review-meeting.md"), # cross-kind lookalike + ("Zephyr", "concept", "resources/concepts/zephyr.md"), # unrelated +] + + +def _dedup_pass(h, gate_env): + h.reset() + h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n") + for title, kind in [("Bob Smith", "person"), ("Echo", "project"), ("Acme", "company")]: + h.run(ECHO, "capture", title, "--kind", kind, env_extra=gate_env) + dupes = blocked_legit = 0 + for title, kind, would_be in DUP_ATTEMPTS: + h.run(ECHO, "capture", title, "--kind", kind, env_extra=gate_env) + dupes += int(h.ground(would_be) is not None) + for title, kind, path in LEGIT_ATTEMPTS: + h.run(ECHO, "capture", title, "--kind", kind, env_extra=gate_env) + blocked_legit += int(h.ground(path) is None) + return {"duplicate_notes_created": dupes, "legitimate_captures_blocked": blocked_legit, + "referring_attempts": len(DUP_ATTEMPTS), "legit_attempts": len(LEGIT_ATTEMPTS)} + + +def eval_dedup(h): + return {"gate ON (v1.5.1 default)": _dedup_pass(h, {}), + "gate OFF (pre-1.5, warn-only)": _dedup_pass(h, {"ECHO_DUP_GATE": "99"})} + + +# ------------------------------------------------------------- 3. WRITE SAFETY +def eval_write_safety(h): + h.reset() + h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n") + silent = loud = 0 + ops = 0 + + def bodyfile(text): + f = tempfile.NamedTemporaryFile("w", suffix=".md", delete=False, encoding="utf-8") + f.write(text) + f.close() + return f.name + + # transient 503 on first write ("flaky" marker) -> client must retry and land it + ops += 1 + r = h.run(ECHO, "put", "resources/concepts/flaky-note.md", bodyfile("# Flaky\nok\n")) + landed = h.ground("resources/concepts/flaky-note.md") is not None + if r.returncode == 0 and not landed: + silent += 1 + if r.returncode != 0: + loud += 1 + + # phantom PUT ("phantom" marker: 200 but not persisted) -> verify must fail LOUD + ops += 1 + r = h.run(ECHO, "put", "resources/concepts/phantom-note.md", bodyfile("# Phantom\n")) + landed = h.ground("resources/concepts/phantom-note.md") is not None + if r.returncode == 0 and not landed: + silent += 1 + if r.returncode != 0 and not landed: + loud += 1 + + # PATCH to a missing heading -> 400 must be surfaced, nothing written + ops += 1 + h.seed("resources/concepts/target.md", "# Target\n\n## Real\nx\n") + r = h.run(ECHO, "patch", "resources/concepts/target.md", "append", "heading", + "Target::Nope", bodyfile("lost?")) + if r.returncode == 0 and "lost?" not in (h.ground("resources/concepts/target.md") or ""): + silent += 1 + if r.returncode != 0: + loud += 1 + + # replayed append -> whole-line idempotency, no duplicate + ops += 1 + h.run(ECHO, "append", "inbox/captures/inbox.md", f"- {TODAY}: replay me") + h.run(ECHO, "append", "inbox/captures/inbox.md", f"- {TODAY}: replay me") + n = (h.ground("inbox/captures/inbox.md") or "").count("replay me") + if n != 1: + silent += 1 + + return {"fault_scenarios": ops, "silent_write_failures": silent, + "failures_surfaced_loud": loud} + + +# --------------------------------------------------------------- 4. DURABILITY +def eval_durability(h, dead_base): + h.reset() + h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n") + queued_ok = 0 + for i in range(3): + r = h.run(ECHO, "append", "inbox/captures/inbox.md", + f"- {TODAY}: offline capture {i}", base=dead_base) + queued_ok += int(r.returncode == 0 and "queued" in r.stdout) + # vault returns: flush replays; a second flush must not duplicate + h.run(ECHO, "flush") + h.run(ECHO, "flush") + inbox = h.ground("inbox/captures/inbox.md") or "" + landed = sum(1 for i in range(3) if f"offline capture {i}" in inbox) + dupes = sum(inbox.count(f"offline capture {i}") - 1 + for i in range(3) if f"offline capture {i}" in inbox) + return {"writes_during_outage": 3, "reported_queued": queued_ok, + "landed_after_flush": landed, "lost": 3 - landed, "duplicated_on_reflush": dupes} + + +# ------------------------------------------------------------------------ main def main(): ap = argparse.ArgumentParser() - ap.add_argument("--port", type=int, default=8799) - ap.add_argument("--cpt", type=float, default=4.0, help="chars per token") - ap.add_argument("--recovery", type=int, default=1500, help="token penalty per silent failure (recovery loop)") - ap.add_argument("--detect-cost", type=int, default=80, help="token cost to re-issue a corrected call after a detected failure") + ap.add_argument("--port", type=int, default=8797) a = ap.parse_args() - base = f"http://127.0.0.1:{a.port}" srv = subprocess.Popen([sys.executable, str(HERE / "mock_olrapi.py"), "--port", str(a.port)], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + state = tempfile.mkdtemp(prefix="echo-eval-state-") try: - for _ in range(50): # wait for ready + for _ in range(50): try: - urllib.request.urlopen(f"{base}/__debug__reset", data=b"", timeout=1); break - except Exception: time.sleep(0.1) - ev = Eval(base, a.cpt) - rows, agg = [], {"06": {}, "07": {}} - for scn in scenarios(ev): - r06 = run_method(ev, scn, "m06") - r07 = run_method(ev, scn, "m07") - rows.append({"scenario": scn["name"], "fault": scn["fault"], "06": r06, "07": r07}) + urllib.request.urlopen(f"{base}/__debug__reset", data=b"", timeout=1) + break + except Exception: # noqa: BLE001 + time.sleep(0.1) + h = Harness(base, state) + results = { + "version": "1.5.1", "date": TODAY, + "retrieval": eval_retrieval(h), + "dedup": eval_dedup(h), + "write_safety": eval_write_safety(h), + "durability": eval_durability(h, dead_base=f"http://127.0.0.1:{a.port + 1}"), + } + out = HERE / "results" / "latest.json" + out.parent.mkdir(exist_ok=True) + out.write_text(json.dumps(results, indent=2), encoding="utf-8") - def totals(side): - gen = sum(r[side]["emit_tokens"] for r in rows) - sil = sum(1 for r in rows if r[side]["silent_failure"]) - det = sum(1 for r in rows if r[side]["detected"]) - dup = sum(r[side]["duplicates"] for r in rows) - eff = gen + sil * a.recovery + det * a.detect_cost - return dict(gen_tokens=gen, silent_failures=sil, detected_failures=det, - duplicates=dup, effective_tokens=eff) - T06, T07 = totals("06"), totals("07") - - # ---- report ---- - line = "=" * 78 - print(f"\n{line}\nECHO MEMORY — 0.6 vs 0.7 A/B EVAL") - print(f"(mock OLRAPI; chars/token={a.cpt}, recovery-penalty={a.recovery}, detect-cost={a.detect_cost})\n{line}") - hdr = f"{'scenario':28} {'fault':32} {'gen06':>5} {'gen07':>5} {'silent06':>8} {'silent07':>8}" - print(hdr); print("-" * len(hdr)) - for r in rows: - print(f"{r['scenario']:28} {r['fault']:32} " - f"{r['06']['emit_tokens']:>5} {r['07']['emit_tokens']:>5} " - f"{('YES' if r['06']['silent_failure'] else '-'):>8} " - f"{('YES' if r['07']['silent_failure'] else '-'):>8}") - print("-" * len(hdr)) - - def pct(old, new): return f"{(old-new)/old*100:+.0f}%" if old else "n/a" - print(f"\n{'metric':30} {'0.6':>10} {'0.7':>10} {'delta':>10}") - print("-" * 62) - for label, k in [("generated tokens", "gen_tokens"), - ("silent failures", "silent_failures"), - ("duplicate lines", "duplicates"), - ("detected (loud) failures", "detected_failures"), - ("effective tokens (incl. recovery)", "effective_tokens")]: - o, n = T06[k], T07[k] - d = pct(o, n) if "token" in k else f"{n-o:+d}" - print(f"{label:30} {o:>10} {n:>10} {d:>10}") - wrows = [r for r in rows if r['scenario'] != 'cold-start-load-6-reads'] - denom = len(wrows) - sf06 = sum(1 for r in wrows if not r['06']['silent_failure']) # silent-error-free - sf07 = sum(1 for r in wrows if not r['07']['silent_failure']) - pr06 = sum(1 for r in wrows if r['06']['persisted']) # write actually landed - pr07 = sum(1 for r in wrows if r['07']['persisted']) - print("-" * 62) - print(f"{'silent-error-free ops':30} {str(sf06)+'/'+str(denom):>10} {str(sf07)+'/'+str(denom):>10}") - print(f"{'writes actually persisted':30} {str(pr06)+'/'+str(denom):>10} {str(pr07)+'/'+str(denom):>10}") - print(f" (the 2 un-persisted 0.7 ops are bad-heading + phantom: not persistable in a single") - print(f" call, but 0.7 fails LOUD so the agent's ensure-heading/retry path can recover.)") - print(f"\nNOTE: recovery-penalty and detect-cost are tunable ASSUMPTIONS, not measured.") - print(f" gen tokens are a chars/{a.cpt} proxy for output tokens of the I/O layer only.") - print(f" This harness measures mechanics, not model reasoning quality.\n") - - report = {"params": vars(a), "rows": rows, "totals": {"0.6": T06, "0.7": T07}, - "silent_error_free": {"0.6": f"{sf06}/{denom}", "0.7": f"{sf07}/{denom}"}, - "writes_persisted": {"0.6": f"{pr06}/{denom}", "0.7": f"{pr07}/{denom}"}} - (HERE / "results" / "latest.json").write_text(json.dumps(report, indent=2)) - print(f"wrote {HERE/'results'/'latest.json'}") + print(f"\n== echo-memory eval — v{results['version']} ({results['date']}) ==\n") + for section, data in results.items(): + if section in ("version", "date"): + continue + print(f"## {section}") + for k, v in data.items(): + if isinstance(v, dict): + print(f" {k}:") + for k2, v2 in v.items(): + print(f" {k2:38} {v2}") + else: + print(f" {k:40} {v}") + print() + print(f"results -> {out}") + return 0 finally: srv.terminate() + if __name__ == "__main__": - main() + raise SystemExit(main())