Retire ROADMAP-1.0; rewrite run_eval.py as current-version metrics harness

Knocks two items off ROADMAP-2.0 (§4 eval refresh, §5 roadmap retirement):

- run_eval.py: the stale 0.6-vs-0.7 A/B is replaced (old version in git
  history) with a four-part current-version eval against the mock —
  retrieval (hybrid+priors vs keyword/entities-only baseline: R@5 1.00
  vs 0.75, MRR 1.00 vs 0.75, session/journal queries 2/2 vs 0/2,
  freshness top-1 correct vs wrong), dedup (0 dupes gate-on vs 3
  gate-off, 0 false blocks), write safety (0 silent failures across 4
  fault scenarios), durability (3/3 offline writes queued+landed, 0
  lost, 0 re-flush dupes). Results in eval/results/latest.json.
- README: metrics table published; repo-layout eval descriptor updated.
- eval/README: new harness documented; A/B framing marked historical.
- ROADMAP-1.0.md removed (fully shipped; story lives in the README
  version table + git history); dangling docstring pointer fixed in
  echo_concurrency.py; ROADMAP-2.0 checkboxes ticked.

All suites green; artifact rebuilt (1.5.1, docstring-only source change).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jason Stedwell
2026-07-03 13:30:01 -05:00
parent 919e411136
commit 993abdc846
9 changed files with 405 additions and 595 deletions
+11 -10
View File
@@ -51,21 +51,22 @@ regenerated after 2.0 if needed). What remains for 2.0:
documenting what a Codex/other-runtime port needs (manifest shape, skill entry
point, script invocation).
## 4. Eval refresh — publish current-version metrics (from MAINTENANCE Docs freshness; ROADMAP-1.0 H4 leftover)
## 4. Eval refresh — publish current-version metrics (from MAINTENANCE Docs freshness; ROADMAP-1.0 H4 leftover) — ✅ DONE
`eval/run_eval.py` still benchmarks **0.6 vs 0.7**. The current-version suites
(`test_features` et al.) prove correctness but publish no *metrics*.
`run_eval.py` was rewritten from the stale 0.6-vs-0.7 A/B into the current-version
metrics harness (retrieval / dedup / write-safety / durability); numbers published in
the README. Re-run it per release and refresh the README table.
- [ ] Refresh `run_eval.py` to measure the current version: retrieval precision/recall
(BM25 + freshness/status fusion vs keyword baseline), duplicate rate with the
gate on/off, offline-queue durability, silent-write-failure rate.
- [ ] Publish the numbers in the README (replacing the 0.6-vs-0.7 claims) and archive
the old A/B harness docs as historical.
- [x] Refresh `run_eval.py` to measure the current version**done 2026-07-03**:
retrieval P@5/R@5/MRR + session/journal answerability + freshness top-1 (vs a
keyword/entities-only baseline), duplicate rate gate-on/off, silent-write-failure
rate under fault injection, offline-queue durability. `results/latest.json`.
- [x] Publish the numbers in the README and mark the old A/B harness historical
(git history) — **done 2026-07-03**.
## 5. Repo hygiene odds & ends
- [ ] Retire `ROADMAP-1.0.md` (fully shipped; keep in git history, drop from the tree)
once this roadmap and the README version table carry the story.
- [x] Retire `ROADMAP-1.0.md` (fully shipped; in git history) — **done 2026-07-03**.
- [ ] Root README "Repository layout" section updated for the post-2.0 tree
(no versioned zips, no codex tree, skills-only plugin).
- [ ] Confirm `echo-improvements-prompt.md` (the 1.5.0 field report) either moves to a