Retire ROADMAP-1.0; rewrite run_eval.py as current-version metrics harness
Knocks two items off ROADMAP-2.0 (§4 eval refresh, §5 roadmap retirement): - run_eval.py: the stale 0.6-vs-0.7 A/B is replaced (old version in git history) with a four-part current-version eval against the mock — retrieval (hybrid+priors vs keyword/entities-only baseline: R@5 1.00 vs 0.75, MRR 1.00 vs 0.75, session/journal queries 2/2 vs 0/2, freshness top-1 correct vs wrong), dedup (0 dupes gate-on vs 3 gate-off, 0 false blocks), write safety (0 silent failures across 4 fault scenarios), durability (3/3 offline writes queued+landed, 0 lost, 0 re-flush dupes). Results in eval/results/latest.json. - README: metrics table published; repo-layout eval descriptor updated. - eval/README: new harness documented; A/B framing marked historical. - ROADMAP-1.0.md removed (fully shipped; story lives in the README version table + git history); dangling docstring pointer fixed in echo_concurrency.py; ROADMAP-2.0 checkboxes ticked. All suites green; artifact rebuilt (1.5.1, docstring-only source change). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+11
-10
@@ -51,21 +51,22 @@ regenerated after 2.0 if needed). What remains for 2.0:
|
||||
documenting what a Codex/other-runtime port needs (manifest shape, skill entry
|
||||
point, script invocation).
|
||||
|
||||
## 4. Eval refresh — publish current-version metrics (from MAINTENANCE › Docs freshness; ROADMAP-1.0 H4 leftover)
|
||||
## 4. Eval refresh — publish current-version metrics (from MAINTENANCE › Docs freshness; ROADMAP-1.0 H4 leftover) — ✅ DONE
|
||||
|
||||
`eval/run_eval.py` still benchmarks **0.6 vs 0.7**. The current-version suites
|
||||
(`test_features` et al.) prove correctness but publish no *metrics*.
|
||||
`run_eval.py` was rewritten from the stale 0.6-vs-0.7 A/B into the current-version
|
||||
metrics harness (retrieval / dedup / write-safety / durability); numbers published in
|
||||
the README. Re-run it per release and refresh the README table.
|
||||
|
||||
- [ ] Refresh `run_eval.py` to measure the current version: retrieval precision/recall
|
||||
(BM25 + freshness/status fusion vs keyword baseline), duplicate rate with the
|
||||
gate on/off, offline-queue durability, silent-write-failure rate.
|
||||
- [ ] Publish the numbers in the README (replacing the 0.6-vs-0.7 claims) and archive
|
||||
the old A/B harness docs as historical.
|
||||
- [x] Refresh `run_eval.py` to measure the current version — **done 2026-07-03**:
|
||||
retrieval P@5/R@5/MRR + session/journal answerability + freshness top-1 (vs a
|
||||
keyword/entities-only baseline), duplicate rate gate-on/off, silent-write-failure
|
||||
rate under fault injection, offline-queue durability. `results/latest.json`.
|
||||
- [x] Publish the numbers in the README and mark the old A/B harness historical
|
||||
(git history) — **done 2026-07-03**.
|
||||
|
||||
## 5. Repo hygiene odds & ends
|
||||
|
||||
- [ ] Retire `ROADMAP-1.0.md` (fully shipped; keep in git history, drop from the tree)
|
||||
once this roadmap and the README version table carry the story.
|
||||
- [x] Retire `ROADMAP-1.0.md` (fully shipped; in git history) — **done 2026-07-03**.
|
||||
- [ ] Root README "Repository layout" section updated for the post-2.0 tree
|
||||
(no versioned zips, no codex tree, skills-only plugin).
|
||||
- [ ] Confirm `echo-improvements-prompt.md` (the 1.5.0 field report) either moves to a
|
||||
|
||||
Reference in New Issue
Block a user