jason/chorus
1
0
Fork 0
forked from jason/echo
Code Pull Requests Activity

Retire ROADMAP-1.0; rewrite run_eval.py as current-version metrics harness

Browse Source 
Knocks two items off ROADMAP-2.0 (§4 eval refresh, §5 roadmap retirement):

- run_eval.py: the stale 0.6-vs-0.7 A/B is replaced (old version in git
  history) with a four-part current-version eval against the mock —
  retrieval (hybrid+priors vs keyword/entities-only baseline: R@5 1.00
  vs 0.75, MRR 1.00 vs 0.75, session/journal queries 2/2 vs 0/2,
  freshness top-1 correct vs wrong), dedup (0 dupes gate-on vs 3
  gate-off, 0 false blocks), write safety (0 silent failures across 4
  fault scenarios), durability (3/3 offline writes queued+landed, 0
  lost, 0 re-flush dupes). Results in eval/results/latest.json.
- README: metrics table published; repo-layout eval descriptor updated.
- eval/README: new harness documented; A/B framing marked historical.
- ROADMAP-1.0.md removed (fully shipped; story lives in the README
  version table + git history); dangling docstring pointer fixed in
  echo_concurrency.py; ROADMAP-2.0 checkboxes ticked.

All suites green; artifact rebuilt (1.5.1, docstring-only source change).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jason Stedwell
2026-07-03 13:30:01 -05:00
parent 919e411136
commit 993abdc846
9 changed files with 405 additions and 595 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+14
View File
@@ -380,6 +380,20 @@ If the API returns a connection error, timeout, or `502` (usually Obsidian / the
---
## Eval metrics (v1.5.1, 2026-07-03)
From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate.
| Metric | v1.5.1 | pre-1.5 baseline |
|---|---|---|
| Retrieval recall@5 / MRR (8-query gold set) | **1.00 / 1.00** | 0.75 / 0.75 |
| Queries answerable only from sessions/journal | **2/2** | 0/2 (not in corpus) |
| Freshness: live note outranks stale archived twin | **yes** | no |
| Duplicate notes created (3 renamed-entity captures) | **0** (gate blocks) | 3 |
| Legitimate captures wrongly blocked | **0** | 0 |
| Silent write failures (4 fault scenarios) | **0** (failures loud) | — |
| Offline writes lost / duplicated on re-flush | **0 / 0** (3/3 queued + landed) | — |
## Version history
| Version | Highlights |