Retire ROADMAP-1.0; rewrite run_eval.py as current-version metrics harness
Knocks two items off ROADMAP-2.0 (§4 eval refresh, §5 roadmap retirement): - run_eval.py: the stale 0.6-vs-0.7 A/B is replaced (old version in git history) with a four-part current-version eval against the mock — retrieval (hybrid+priors vs keyword/entities-only baseline: R@5 1.00 vs 0.75, MRR 1.00 vs 0.75, session/journal queries 2/2 vs 0/2, freshness top-1 correct vs wrong), dedup (0 dupes gate-on vs 3 gate-off, 0 false blocks), write safety (0 silent failures across 4 fault scenarios), durability (3/3 offline writes queued+landed, 0 lost, 0 re-flush dupes). Results in eval/results/latest.json. - README: metrics table published; repo-layout eval descriptor updated. - eval/README: new harness documented; A/B framing marked historical. - ROADMAP-1.0.md removed (fully shipped; story lives in the README version table + git history); dangling docstring pointer fixed in echo_concurrency.py; ROADMAP-2.0 checkboxes ticked. All suites green; artifact rebuilt (1.5.1, docstring-only source change). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+19
-5
@@ -1,9 +1,23 @@
|
||||
# echo-memory eval — 0.6 vs 0.7 A/B harness
|
||||
# echo-memory eval — current-version metrics harness
|
||||
|
||||
A reproducible, credential-free A/B comparison of the plugin **before** (0.6: raw-curl
|
||||
recipes from `SKILL.md`) and **after** (the shipped `scripts/echo.py` client) the
|
||||
hardening work. It quantifies the claims in the comparative analysis: token cost of the
|
||||
I/O layer, and the rate of **silent write failures**.
|
||||
A reproducible, credential-free eval of the shipped plugin against the deterministic
|
||||
mock REST API. `run_eval.py` measures the four things the plugin claims (results print
|
||||
as a table and land in `results/latest.json`):
|
||||
|
||||
1. **Retrieval** — hybrid recall (BM25 over entities **and** sessions/journal, fused
|
||||
with freshness/status priors) vs a keyword-only baseline modeling pre-1.5 recall
|
||||
(per-word substring ranking over entity notes only). Precision@5 / recall@5 / MRR,
|
||||
session/journal answerability, freshness top-1.
|
||||
2. **Dedup** — duplicate notes created replaying a capture stream with the pre-write
|
||||
gate ON (default) vs OFF (`ECHO_DUP_GATE=99` ≈ pre-1.5 warn-only), plus legitimate
|
||||
captures wrongly blocked (must be 0).
|
||||
3. **Write safety** — silent write failures under fault injection (transient 503,
|
||||
phantom PUT, PATCH to a missing heading, replayed append). Must be 0; must be loud.
|
||||
4. **Durability** — writes during an outage queue, replay exactly once, never
|
||||
duplicate on re-flush.
|
||||
|
||||
> The original **0.6-vs-0.7 A/B harness** (token cost + silent-failure comparison that
|
||||
> motivated the 0.7 client) lives in git history — `git log -- eval/run_eval.py`.
|
||||
|
||||
## Current test suites (run these for any change)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user