Retire ROADMAP-1.0; rewrite run_eval.py as current-version metrics harness

Knocks two items off ROADMAP-2.0 (§4 eval refresh, §5 roadmap retirement):

- run_eval.py: the stale 0.6-vs-0.7 A/B is replaced (old version in git
  history) with a four-part current-version eval against the mock —
  retrieval (hybrid+priors vs keyword/entities-only baseline: R@5 1.00
  vs 0.75, MRR 1.00 vs 0.75, session/journal queries 2/2 vs 0/2,
  freshness top-1 correct vs wrong), dedup (0 dupes gate-on vs 3
  gate-off, 0 false blocks), write safety (0 silent failures across 4
  fault scenarios), durability (3/3 offline writes queued+landed, 0
  lost, 0 re-flush dupes). Results in eval/results/latest.json.
- README: metrics table published; repo-layout eval descriptor updated.
- eval/README: new harness documented; A/B framing marked historical.
- ROADMAP-1.0.md removed (fully shipped; story lives in the README
  version table + git history); dangling docstring pointer fixed in
  echo_concurrency.py; ROADMAP-2.0 checkboxes ticked.

All suites green; artifact rebuilt (1.5.1, docstring-only source change).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jason Stedwell
2026-07-03 13:30:01 -05:00
parent 919e411136
commit 993abdc846
9 changed files with 405 additions and 595 deletions
+19 -5
View File
@@ -1,9 +1,23 @@
# echo-memory eval — 0.6 vs 0.7 A/B harness
# echo-memory eval — current-version metrics harness
A reproducible, credential-free A/B comparison of the plugin **before** (0.6: raw-curl
recipes from `SKILL.md`) and **after** (the shipped `scripts/echo.py` client) the
hardening work. It quantifies the claims in the comparative analysis: token cost of the
I/O layer, and the rate of **silent write failures**.
A reproducible, credential-free eval of the shipped plugin against the deterministic
mock REST API. `run_eval.py` measures the four things the plugin claims (results print
as a table and land in `results/latest.json`):
1. **Retrieval** — hybrid recall (BM25 over entities **and** sessions/journal, fused
with freshness/status priors) vs a keyword-only baseline modeling pre-1.5 recall
(per-word substring ranking over entity notes only). Precision@5 / recall@5 / MRR,
session/journal answerability, freshness top-1.
2. **Dedup** — duplicate notes created replaying a capture stream with the pre-write
gate ON (default) vs OFF (`ECHO_DUP_GATE=99` ≈ pre-1.5 warn-only), plus legitimate
captures wrongly blocked (must be 0).
3. **Write safety** — silent write failures under fault injection (transient 503,
phantom PUT, PATCH to a missing heading, replayed append). Must be 0; must be loud.
4. **Durability** — writes during an outage queue, replay exactly once, never
duplicate on re-flush.
> The original **0.6-vs-0.7 A/B harness** (token cost + silent-failure comparison that
> motivated the 0.7 client) lives in git history — `git log -- eval/run_eval.py`.
## Current test suites (run these for any change)