forked from jason/echo
Retire ROADMAP-1.0; rewrite run_eval.py as current-version metrics harness
Knocks two items off ROADMAP-2.0 (§4 eval refresh, §5 roadmap retirement): - run_eval.py: the stale 0.6-vs-0.7 A/B is replaced (old version in git history) with a four-part current-version eval against the mock — retrieval (hybrid+priors vs keyword/entities-only baseline: R@5 1.00 vs 0.75, MRR 1.00 vs 0.75, session/journal queries 2/2 vs 0/2, freshness top-1 correct vs wrong), dedup (0 dupes gate-on vs 3 gate-off, 0 false blocks), write safety (0 silent failures across 4 fault scenarios), durability (3/3 offline writes queued+landed, 0 lost, 0 re-flush dupes). Results in eval/results/latest.json. - README: metrics table published; repo-layout eval descriptor updated. - eval/README: new harness documented; A/B framing marked historical. - ROADMAP-1.0.md removed (fully shipped; story lives in the README version table + git history); dangling docstring pointer fixed in echo_concurrency.py; ROADMAP-2.0 checkboxes ticked. All suites green; artifact rebuilt (1.5.1, docstring-only source change). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jason Stedwell
@@ -380,6 +380,20 @@ If the API returns a connection error, timeout, or `502` (usually Obsidian / the
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Eval metrics (v1.5.1, 2026-07-03)
|
||||||
|
|
||||||
|
From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate.
|
||||||
|
|
||||||
|
| Metric | v1.5.1 | pre-1.5 baseline |
|
||||||
|
|---|---|---|
|
||||||
|
| Retrieval recall@5 / MRR (8-query gold set) | **1.00 / 1.00** | 0.75 / 0.75 |
|
||||||
|
| Queries answerable only from sessions/journal | **2/2** | 0/2 (not in corpus) |
|
||||||
|
| Freshness: live note outranks stale archived twin | **yes** | no |
|
||||||
|
| Duplicate notes created (3 renamed-entity captures) | **0** (gate blocks) | 3 |
|
||||||
|
| Legitimate captures wrongly blocked | **0** | 0 |
|
||||||
|
| Silent write failures (4 fault scenarios) | **0** (failures loud) | — |
|
||||||
|
| Offline writes lost / duplicated on re-flush | **0 / 0** (3/3 queued + landed) | — |
|
||||||
|
|
||||||
## Version history
|
## Version history
|
||||||
|
|
||||||
| Version | Highlights |
|
| Version | Highlights |
|
||||||
|
|||||||
-176
@@ -1,176 +0,0 @@
|
|||||||
# echo-memory — Roadmap to v1.0
|
|
||||||
|
|
||||||
> Status: **draft / in progress.** Current shipping version: **0.9.0 (schema 3)**.
|
|
||||||
> Target: **1.0.0 (schema 4)** — *"memory you can actually trust and retrieve."*
|
|
||||||
|
|
||||||
This roadmap turns the 10 code-review suggestions into a sequenced plan and tracks the
|
|
||||||
scaffold for each. It is the single index: every v1.0 workstream maps to one or more
|
|
||||||
files under `echo-memory.plugin.src/skills/echo-memory/scripts/` (plus tests and CI).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1.0 thesis
|
|
||||||
|
|
||||||
0.x made the plugin **correct and self-describing**: status-checked writes, idempotent
|
|
||||||
appends, a routing manifest the linter enforces, an entity index, and one-call
|
|
||||||
`capture/recall`. 1.0 makes it **trustworthy and genuinely recall-driven**:
|
|
||||||
|
|
||||||
1. You can **retrieve** a fact even when you phrase the query differently than you stored it. *(H1)*
|
|
||||||
2. Nothing is **lost** when the backend is down. *(H2)*
|
|
||||||
3. Two clients writing at once **cannot corrupt** the index or clobber state. *(H3)*
|
|
||||||
4. Every feature is **proven** by tests that model the real API, gated in CI. *(H4)*
|
|
||||||
5. Memory **accrues on its own** instead of only when the agent remembers to write. *(H5)*
|
|
||||||
|
|
||||||
…on a base that is secure (M1), clean-linking (M2), self-diagnosing (M3),
|
|
||||||
machine-parseable (M4), and maintainable (M5).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Non-negotiable constraints (these bound every solution)
|
|
||||||
|
|
||||||
These are inherited from the 0.x design and **must not regress**:
|
|
||||||
|
|
||||||
| Constraint | Consequence for 1.0 |
|
|
||||||
|---|---|
|
|
||||||
| **Pure-Python stdlib, no third-party deps** | H1 recall = local BM25 + graph fusion, **not** an embedding API. No `numpy`, `keyring`, `requests`. |
|
|
||||||
| **Cross-platform (Win/macOS/Linux)** | No bash, no platform `date`; `pathlib`, `os`, UTF-8-safe streams. |
|
|
||||||
| **Plugin is the single source of truth; vault holds data only** | Derived artifacts (recall index) live in `_agent/index/` (machine-maintained, rebuildable) — never new control docs in the vault. |
|
|
||||||
| **Fail loud, never silent** | New write paths keep status-checking and non-zero exits. |
|
|
||||||
| **Local state can't depend on the vault being up** | H2 queue/cache lives in `ECHO_STATE_DIR` (default `~/.echo-memory/`), not the vault. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Build order vs. scaffold order
|
|
||||||
|
|
||||||
The user asked to **scaffold heaviest → lightest** (H1→M5). That is the order the
|
|
||||||
files below are stubbed. The **recommended build/merge order is different** — safety
|
|
||||||
and proof first, so the heavy features land on a tested, secure base:
|
|
||||||
|
|
||||||
```
|
|
||||||
Phase A (land first): M1 secrets · H4 tests+CI
|
|
||||||
Phase B (correctness): H3 concurrency · M2 link/slug quality
|
|
||||||
Phase C (durability): H2 offline queue + cache
|
|
||||||
Phase D (intelligence): H1 hybrid recall · H5 reflection capture
|
|
||||||
Phase E (polish): M3 doctor · M4 --json · M5 hygiene → tag 1.0.0
|
|
||||||
```
|
|
||||||
|
|
||||||
Scaffolds are independent stubs, so stubbing heaviest-first is safe; only the
|
|
||||||
*integration* (wiring into `echo.py`, schema bump, migration) follows Phase order.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Workstreams (heaviest → lightest)
|
|
||||||
|
|
||||||
Each item: **goal · why · scope · files · storage/schema · acceptance · depends-on · lift.**
|
|
||||||
|
|
||||||
### H1 — Hybrid/semantic recall · lift: ●●●●●
|
|
||||||
- **Goal:** `recall "X"` returns relevant notes even when the wording differs, ranked, with scored multi-hop graph expansion.
|
|
||||||
- **Why:** Recall is the product. Today it's keyword `/search/simple` + fixed one-hop (`echo_ops.py:88`). Differently-phrased memory is unfindable → dead weight.
|
|
||||||
- **Scope:** local BM25 index over note bodies (stdlib only); fuse lexical score + graph distance (decay per hop); relevance threshold; configurable hops.
|
|
||||||
- **Files:** `scripts/echo_recall.py` (new) → later replaces `echo_ops.recall`.
|
|
||||||
- **Storage/schema:** `_agent/index/recall-index.json` (postings + doc stats), machine-maintained, rebuilt by `sweep.py`. **Schema 3→4.**
|
|
||||||
- **Acceptance:** retrieval precision/recall measured in eval beats keyword-only baseline; recall returns scored, deduped hits + neighbourhood; index rebuildable offline.
|
|
||||||
- **Depends-on:** entity index (have), H4 eval harness for the precision/recall numbers.
|
|
||||||
|
|
||||||
### H2 — Offline durability: write-ahead queue + read cache · lift: ●●●●●
|
|
||||||
- **Goal:** No durable write is lost when the API/Obsidian is down; cold-start degrades to last-known context instead of "no memory."
|
|
||||||
- **Why:** Today vault-unreachable = silently proceed without memory (`SKILL.md:383`). 502 (Obsidian not running) is the most common real failure.
|
|
||||||
- **Scope:** append-only NDJSON outbox; replay on next reachable session (idempotency makes replay safe); last-known-good read cache for `load`.
|
|
||||||
- **Files:** `scripts/echo_queue.py` (new); integration hook `safe_request()` wrapping `echo.request`.
|
|
||||||
- **Storage/schema:** local `ECHO_STATE_DIR/outbox.ndjson` + `ECHO_STATE_DIR/cache/`. No vault schema change.
|
|
||||||
- **Acceptance:** kill the API mid-session → writes queue; restart → replay lands exactly once (no dupes); `load` serves cache when offline and says so.
|
|
||||||
- **Depends-on:** existing idempotent-append discipline; H4 for fault-injection tests.
|
|
||||||
|
|
||||||
### H3 — Concurrency correctness · lift: ●●●●○
|
|
||||||
- **Goal:** Make the advertised multi-writer (Claude + CoWork) story actually safe.
|
|
||||||
- **Why:** Index is full-file load→PUT with no lock (`echo_index.py:105`, `echo_ops.py:230`) → concurrent `capture` silently drops an entity. Lock is manual/advisory only (`echo.py:293`).
|
|
||||||
- **Scope:** auto-lock context manager around `capture`/`scope set`/index writes; atomic index update (re-read under lock, merge, save); per-resource locks; crash-safe TTL reclaim (have).
|
|
||||||
- **Files:** `scripts/echo_concurrency.py` (new); refactor `echo_index.save` + `echo_ops.capture` to route through it.
|
|
||||||
- **Storage/schema:** reuse `_agent/locks/`. No schema change.
|
|
||||||
- **Acceptance:** two concurrent captures both land in the index; no lost entries; lock auto-released on normal + error exit.
|
|
||||||
- **Depends-on:** none (pure refactor); H4 to prove it.
|
|
||||||
|
|
||||||
### H4 — High-fidelity tests + CI + current eval · lift: ●●●●○
|
|
||||||
- **Goal:** Prove every 0.9 + 1.0 feature against an API model that reproduces real failure modes; gate on CI.
|
|
||||||
- **Why:** Mock PATCH is "naive append" (`mock_olrapi.py:138`) — can't model heading replace/insert; eval still benchmarks **0.6 vs 0.7** (two versions stale).
|
|
||||||
- **Scope:** higher-fidelity mock (real heading/frontmatter PATCH semantics) or containerized real Obsidian Local REST API; refresh eval to measure retrieval precision/recall, link correctness, dup rate; wire `test_echo_client.py` + `check_routing.py` + `test_features.py` + new module tests into GitHub Actions.
|
|
||||||
- **Files:** `.github/workflows/ci.yml` (new); `eval/mock_olrapi_hifi.py` (new); `scripts/test_v1_scaffold.py` (interface guard, new); refreshed `eval/run_eval.py`.
|
|
||||||
- **Acceptance:** CI green on every push; eval reports current-version metrics; mock reproduces the 40080 invalid-target and replace/insert paths.
|
|
||||||
- **Depends-on:** none — **foundational, build in Phase A.**
|
|
||||||
|
|
||||||
### H5 — Automatic session-reflection capture · lift: ●●●●○
|
|
||||||
- **Goal:** At session end, propose durable captures extracted from the conversation for one-tap confirm — memory that fills itself.
|
|
||||||
- **Why:** Memory only accrues when the agent decides to write. "As useful as possible" means not depending on that judgment firing.
|
|
||||||
- **Scope:** model-side extraction emits a JSON proposal set; script dedups against the entity index, previews, and applies on confirm (respects "show before large writes" safety rule).
|
|
||||||
- **Files:** `scripts/echo_reflect.py` (new); `/echo-reflect` command (later).
|
|
||||||
- **Storage/schema:** none new (routes through `capture`).
|
|
||||||
- **Acceptance:** given a transcript, proposals are deduped vs index, previewed, and only applied on explicit confirm; nothing written without go-ahead.
|
|
||||||
- **Depends-on:** H1 index/recall for dedup quality; H3 for safe concurrent apply.
|
|
||||||
|
|
||||||
### M1 — Harden secret handling · lift: ●●○○○ *(do now — Phase A)*
|
|
||||||
- **Goal:** Stop shipping a live bearer token in source, docs, and every `.plugin` zip.
|
|
||||||
- **Why:** Token hardcoded at `echo.py:70` and repeated through `api-reference.md`/README — committed to git, only rotatable by rebuild. Violates the plugin's own "never store secrets" rule.
|
|
||||||
- **Scope:** resolve key env → local `ECHO_STATE_DIR/credentials` (0600) → deprecated baked default **with a loud warning**; scrub literal from docs to a placeholder; document rotation.
|
|
||||||
- **Files:** `scripts/echo_secrets.py` (new); `echo.py` `KEY=` → `echo_secrets.resolve_key()`.
|
|
||||||
- **Acceptance:** no live token in tracked source/docs/package; `ECHO_KEY` still overrides; rotation documented.
|
|
||||||
- **Depends-on:** none.
|
|
||||||
|
|
||||||
### M2 — Tame auto-link false positives & slug collisions · lift: ●●○○○
|
|
||||||
- **Goal:** Keep the graph (which recall depends on) clean.
|
|
||||||
- **Why:** Auto-link fires on any ≥3-char name/alias word-match (`echo_ops.py:236`) → a concept "API" or alias "rs" links everywhere; `slugify` truncates to 40 chars with no collision check (`echo_index.py:58`) → distinct titles silently share a note.
|
|
||||||
- **Scope:** raise alias floor / require multi-token or kind-aware match / stopword guard; collision-aware slug disambiguation in `derive_path`.
|
|
||||||
- **Files:** `scripts/echo_quality.py` (new); patches to `echo_links`/`echo_index`.
|
|
||||||
- **Acceptance:** known false-positive cases no longer link; colliding titles get distinct slugs; covered by tests.
|
|
||||||
- **Depends-on:** none.
|
|
||||||
|
|
||||||
### M3 — `echo.py doctor` + complete `load` fallback · lift: ●●○○○
|
|
||||||
- **Goal:** One-call readiness check; make `load` self-sufficient.
|
|
||||||
- **Why:** No single "is everything OK" check. `cmd_load` reads the heartbeat but never falls back to listing `_agent/sessions/` though `SKILL.md:122` documents it (`echo.py:395`).
|
|
||||||
- **Scope:** doctor = Python version + reachability + auth + marker/`schema_version` + lint summary, green/red; `load` lists recent sessions when heartbeat missing/stale.
|
|
||||||
- **Files:** `scripts/echo_doctor.py` (new); patch `echo.cmd_load`.
|
|
||||||
- **Acceptance:** `doctor` prints actionable status and exits non-zero on any red; `load` orients with no heartbeat.
|
|
||||||
- **Depends-on:** none.
|
|
||||||
|
|
||||||
### M4 — Machine output (`--json`) + `--dry-run` · lift: ●○○○○
|
|
||||||
- **Goal:** Let the agent act on structured results and preview writes.
|
|
||||||
- **Why:** `capture/recall/resolve/scope` print prose; agent re-parses free text.
|
|
||||||
- **Scope:** `--json` result envelope (action, path, links_added, hits); `--dry-run` for `capture`/`scope set`.
|
|
||||||
- **Files:** `scripts/echo_output.py` (new helpers); thread through `echo_ops`/`echo.py`.
|
|
||||||
- **Acceptance:** every high-level op emits valid JSON under `--json`; `--dry-run` writes nothing.
|
|
||||||
- **Depends-on:** none.
|
|
||||||
|
|
||||||
### M5 — Repo hygiene & manifest completeness · lift: ●○○○○
|
|
||||||
- **Goal:** Remove "which tree is canonical?" traps; complete the manifest.
|
|
||||||
- **Why:** Two plugin trees (`codex plugin/` vs `echo-memory.plugin.src/`) have already drifted (codex lacks the 0.9 modules); stack of versioned `.plugin` artifacts; sparse `plugin.json`.
|
|
||||||
- **Scope:** consolidate/mark canonical tree; prune old artifacts; add `license`/`homepage`/command registration to `plugin.json`; refresh version-history + eval references.
|
|
||||||
- **Files:** `MAINTENANCE.md` (new checklist); `plugin.json`.
|
|
||||||
- **Acceptance:** one canonical source tree; complete manifest; docs reference current version.
|
|
||||||
- **Depends-on:** none.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Schema 3 → 4 migration (owned by H1/H2)
|
|
||||||
|
|
||||||
- Add `_agent/index/recall-index.json` (H1), rebuilt by `sweep.py`.
|
|
||||||
- `migrate.py`: add `[3→4]` step (create recall-index placeholder; no destructive moves).
|
|
||||||
- `routing.json`: add a route for `_agent/index/recall-index.json` (already covered by the generic `^_agent/index/[^/]+\.json$` route — verify in `check_routing.py`).
|
|
||||||
- Bump `CURRENT_SCHEMA` to 4 in `sweep.py`/`migrate.py`; `vault_lint.py` unaffected.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Scaffold status
|
|
||||||
|
|
||||||
| # | Workstream | Lift | Scaffold file(s) | State |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| H1 | Hybrid recall | ●●●●● | `scripts/echo_recall.py` | **✅ WIRED** — BM25 + vault crawl + `recall-index.json` persistence + graph fusion; maintained on `capture`, rebuilt by `sweep`; replaces `echo_ops.recall`; **schema 4**. Covered by `test_features.py`. |
|
|
||||||
| H2 | Offline queue/cache | ●●●●● | `scripts/echo_queue.py`, `eval/test_offline_queue.py` | **✅ WIRED** — write verbs (put/post/append/patch/delete) queue on outage and report "queued"; `flush` (+ flush-on-`load`) replays idempotently and **re-bases to the current endpoint** (survives base migration); `load` degrades to last-known-good cache and flags OFFLINE. Proven end-to-end (down→queue→up→flush→land, idempotent, cache fallback). Follow-up: route capture's internal index/link writes through the queue for full offline-atomic captures. |
|
|
||||||
| H3 | Concurrency | ●●●●○ | `scripts/echo_concurrency.py` | **✅ WIRED** — `vault_lock` CM (auto acquire/release, crash-safe, advisory) + `atomic_index_update` (lock + fresh re-read-merge). `capture` entity write and `echo_recall.update_note` both routed through it. Proven by `test_features.py` (merge-no-clobber + lock-release). Follow-up: deeper resolve-level title-collision detection; true concurrent stress test. |
|
|
||||||
| H4 | Tests + CI | ●●●●○ | `.github/workflows/ci.yml`, `eval/mock_olrapi_hifi.py`, `eval/test_patch_semantics.py`, `scripts/test_v1_scaffold.py` | **✅ hi-fi PATCH mock + semantics test + CI matrix (Win/macOS/Linux × 3.10/3.12) live.** Remaining: refresh `run_eval.py` (still 0.6-vs-0.7) → publish current metrics. |
|
|
||||||
| H5 | Reflection capture | ●●●●○ | `scripts/echo_reflect.py`, `eval/test_reflect.py`, `commands/echo-reflect.md` | **✅ WIRED** — `validate`/`classify`/`preview`/`apply` + `echo.py reflect` (dry-run unless `--apply`, reads file or stdin) + `/echo-reflect` command. Dedups proposals vs the entity index, drops below-confidence, routes `inbox`, applies via `capture` (indexed/linked/logged, self-lock-guarded). Proven by `test_reflect.py`. |
|
|
||||||
| M1 | Secret handling | ●●○○○ | `scripts/echo_secrets.py` | **✅ WIRED** — `resolve_key` (env → `~/.echo-memory/credentials` → deprecated fallback + loud warning), `write-key` CLI, docs scrubbed. Live token now only in `echo.py` `DEFAULT_KEY` (remove at 1.0; operator env already sets `ECHO_KEY`). |
|
|
||||||
| M2 | Link/slug quality | ●●○○○ | `scripts/echo_quality.py` | **✅ WIRED** — `is_confident_link` gates `capture` auto-linking (rejects <4-char / common tokens, trusts multi-word names); `safe_slug` disambiguates a colliding 40-char-truncated slug in the create path. Follow-up: resolve-level title collision (two long titles → same slug → false merge). |
|
|
||||||
| M3 | Doctor + load fallback | ●●○○○ | `scripts/echo_doctor.py`, `commands/echo-doctor.md` | **✅ WIRED** — `echo.py doctor` (Python/reachability/auth/bootstrap/schema/key-source) + `/echo-doctor`; `cmd_load` now falls back to a recent-sessions listing when the heartbeat pointer is absent. Verified against the live vault. |
|
|
||||||
| M4 | `--json` / `--dry-run` | ●○○○○ | `scripts/echo_output.py` | **✅ WIRED** — `capture --json` emits a clean result envelope (helper chatter redirected) and `--dry-run` previews the create/update plan without writing; `resolve` already emits JSON. Proven by `test_features.py`. Follow-up: `--json` for recall/scope/link. |
|
|
||||||
| M5 | Hygiene | ●○○○○ | `plugin.json`, `.gitignore`, `MAINTENANCE.md` | **✅ DONE** — `plugin.json` → **1.0.0** + `license`; `.gitignore` (`.DS_Store`, `__pycache__`, local `.echo-memory/` state, eval output). Remaining (operator judgment, not code): consolidate/retire the drifted `codex plugin/` tree and prune old `.plugin` artifacts — see `MAINTENANCE.md`. |
|
|
||||||
|
|
||||||
**Wiring:** scaffolds are **not** imported by the live `echo.py` yet — 0.9.0 behavior is unchanged. Integration happens per Phase, each gated by its tests (H4).
|
|
||||||
+11
-10
@@ -51,21 +51,22 @@ regenerated after 2.0 if needed). What remains for 2.0:
|
|||||||
documenting what a Codex/other-runtime port needs (manifest shape, skill entry
|
documenting what a Codex/other-runtime port needs (manifest shape, skill entry
|
||||||
point, script invocation).
|
point, script invocation).
|
||||||
|
|
||||||
## 4. Eval refresh — publish current-version metrics (from MAINTENANCE › Docs freshness; ROADMAP-1.0 H4 leftover)
|
## 4. Eval refresh — publish current-version metrics (from MAINTENANCE › Docs freshness; ROADMAP-1.0 H4 leftover) — ✅ DONE
|
||||||
|
|
||||||
`eval/run_eval.py` still benchmarks **0.6 vs 0.7**. The current-version suites
|
`run_eval.py` was rewritten from the stale 0.6-vs-0.7 A/B into the current-version
|
||||||
(`test_features` et al.) prove correctness but publish no *metrics*.
|
metrics harness (retrieval / dedup / write-safety / durability); numbers published in
|
||||||
|
the README. Re-run it per release and refresh the README table.
|
||||||
|
|
||||||
- [ ] Refresh `run_eval.py` to measure the current version: retrieval precision/recall
|
- [x] Refresh `run_eval.py` to measure the current version — **done 2026-07-03**:
|
||||||
(BM25 + freshness/status fusion vs keyword baseline), duplicate rate with the
|
retrieval P@5/R@5/MRR + session/journal answerability + freshness top-1 (vs a
|
||||||
gate on/off, offline-queue durability, silent-write-failure rate.
|
keyword/entities-only baseline), duplicate rate gate-on/off, silent-write-failure
|
||||||
- [ ] Publish the numbers in the README (replacing the 0.6-vs-0.7 claims) and archive
|
rate under fault injection, offline-queue durability. `results/latest.json`.
|
||||||
the old A/B harness docs as historical.
|
- [x] Publish the numbers in the README and mark the old A/B harness historical
|
||||||
|
(git history) — **done 2026-07-03**.
|
||||||
|
|
||||||
## 5. Repo hygiene odds & ends
|
## 5. Repo hygiene odds & ends
|
||||||
|
|
||||||
- [ ] Retire `ROADMAP-1.0.md` (fully shipped; keep in git history, drop from the tree)
|
- [x] Retire `ROADMAP-1.0.md` (fully shipped; in git history) — **done 2026-07-03**.
|
||||||
once this roadmap and the README version table carry the story.
|
|
||||||
- [ ] Root README "Repository layout" section updated for the post-2.0 tree
|
- [ ] Root README "Repository layout" section updated for the post-2.0 tree
|
||||||
(no versioned zips, no codex tree, skills-only plugin).
|
(no versioned zips, no codex tree, skills-only plugin).
|
||||||
- [ ] Confirm `echo-improvements-prompt.md` (the 1.5.0 field report) either moves to a
|
- [ ] Confirm `echo-improvements-prompt.md` (the 1.5.0 field report) either moves to a
|
||||||
|
|||||||
Binary file not shown.
Binary file not shown.
@@ -14,7 +14,7 @@ advisory lock around a write burst, with crash-safe release, and (b) an atomic
|
|||||||
index-update that re-reads UNDER the lock before saving, so concurrent captures merge
|
index-update that re-reads UNDER the lock before saving, so concurrent captures merge
|
||||||
instead of clobber.
|
instead of clobber.
|
||||||
|
|
||||||
ACCEPTANCE (see ROADMAP-1.0.md > H3):
|
ACCEPTANCE (v1.0 roadmap H3, shipped; roadmap retired — see git history):
|
||||||
- two concurrent captures both land in the index (no lost entries)
|
- two concurrent captures both land in the index (no lost entries)
|
||||||
- lock auto-released on normal AND error exit
|
- lock auto-released on normal AND error exit
|
||||||
"""
|
"""
|
||||||
|
|||||||
+19
-5
@@ -1,9 +1,23 @@
|
|||||||
# echo-memory eval — 0.6 vs 0.7 A/B harness
|
# echo-memory eval — current-version metrics harness
|
||||||
|
|
||||||
A reproducible, credential-free A/B comparison of the plugin **before** (0.6: raw-curl
|
A reproducible, credential-free eval of the shipped plugin against the deterministic
|
||||||
recipes from `SKILL.md`) and **after** (the shipped `scripts/echo.py` client) the
|
mock REST API. `run_eval.py` measures the four things the plugin claims (results print
|
||||||
hardening work. It quantifies the claims in the comparative analysis: token cost of the
|
as a table and land in `results/latest.json`):
|
||||||
I/O layer, and the rate of **silent write failures**.
|
|
||||||
|
1. **Retrieval** — hybrid recall (BM25 over entities **and** sessions/journal, fused
|
||||||
|
with freshness/status priors) vs a keyword-only baseline modeling pre-1.5 recall
|
||||||
|
(per-word substring ranking over entity notes only). Precision@5 / recall@5 / MRR,
|
||||||
|
session/journal answerability, freshness top-1.
|
||||||
|
2. **Dedup** — duplicate notes created replaying a capture stream with the pre-write
|
||||||
|
gate ON (default) vs OFF (`ECHO_DUP_GATE=99` ≈ pre-1.5 warn-only), plus legitimate
|
||||||
|
captures wrongly blocked (must be 0).
|
||||||
|
3. **Write safety** — silent write failures under fault injection (transient 503,
|
||||||
|
phantom PUT, PATCH to a missing heading, replayed append). Must be 0; must be loud.
|
||||||
|
4. **Durability** — writes during an outage queue, replay exactly once, never
|
||||||
|
duplicate on re-flush.
|
||||||
|
|
||||||
|
> The original **0.6-vs-0.7 A/B harness** (token cost + silent-failure comparison that
|
||||||
|
> motivated the 0.7 client) lives in git history — `git log -- eval/run_eval.py`.
|
||||||
|
|
||||||
## Current test suites (run these for any change)
|
## Current test suites (run these for any change)
|
||||||
|
|
||||||
|
|||||||
+36
-144
@@ -1,154 +1,46 @@
|
|||||||
{
|
{
|
||||||
"params": {
|
"version": "1.5.1",
|
||||||
"port": 8799,
|
"date": "2026-07-03",
|
||||||
"cpt": 4.0,
|
"retrieval": {
|
||||||
"recovery": 1500,
|
"current (hybrid+priors, full corpus)": {
|
||||||
"detect_cost": 80
|
"precision_at_5": 0.25,
|
||||||
|
"recall_at_5": 1.0,
|
||||||
|
"mrr": 1.0,
|
||||||
|
"session_journal_queries_answered": "2/2",
|
||||||
|
"freshness_top1_correct": true
|
||||||
},
|
},
|
||||||
"rows": [
|
"baseline (keyword, entities only)": {
|
||||||
{
|
"precision_at_5": 0.2,
|
||||||
"scenario": "agent-log-missing-heading",
|
"recall_at_5": 0.75,
|
||||||
"fault": "bad heading -> 400 invalid-target",
|
"mrr": 0.75,
|
||||||
"06": {
|
"session_journal_queries_answered": "0/2",
|
||||||
"emit_tokens": 82,
|
"freshness_top1_correct": false
|
||||||
"claimed_ok": true,
|
|
||||||
"detected": false,
|
|
||||||
"persisted": false,
|
|
||||||
"duplicates": 0,
|
|
||||||
"silent_failure": true
|
|
||||||
},
|
|
||||||
"07": {
|
|
||||||
"emit_tokens": 22,
|
|
||||||
"claimed_ok": false,
|
|
||||||
"detected": true,
|
|
||||||
"persisted": false,
|
|
||||||
"duplicates": 0,
|
|
||||||
"silent_failure": false
|
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
{
|
"dedup": {
|
||||||
"scenario": "scope-switch",
|
"gate ON (v1.5.1 default)": {
|
||||||
"fault": "(none)",
|
"duplicate_notes_created": 0,
|
||||||
"06": {
|
"legitimate_captures_blocked": 0,
|
||||||
"emit_tokens": 84,
|
"referring_attempts": 3,
|
||||||
"claimed_ok": true,
|
"legit_attempts": 2
|
||||||
"detected": false,
|
|
||||||
"persisted": true,
|
|
||||||
"duplicates": 0,
|
|
||||||
"silent_failure": false
|
|
||||||
},
|
},
|
||||||
"07": {
|
"gate OFF (pre-1.5, warn-only)": {
|
||||||
"emit_tokens": 24,
|
"duplicate_notes_created": 3,
|
||||||
"claimed_ok": true,
|
"legitimate_captures_blocked": 0,
|
||||||
"detected": false,
|
"referring_attempts": 3,
|
||||||
"persisted": true,
|
"legit_attempts": 2
|
||||||
"duplicates": 0,
|
|
||||||
"silent_failure": false
|
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
{
|
"write_safety": {
|
||||||
"scenario": "inbox-capture-replayed",
|
"fault_scenarios": 4,
|
||||||
"fault": "duplicate attempt (retry/replay)",
|
"silent_write_failures": 0,
|
||||||
"06": {
|
"failures_surfaced_loud": 2
|
||||||
"emit_tokens": 125,
|
|
||||||
"claimed_ok": true,
|
|
||||||
"detected": false,
|
|
||||||
"persisted": true,
|
|
||||||
"duplicates": 1,
|
|
||||||
"silent_failure": true
|
|
||||||
},
|
},
|
||||||
"07": {
|
"durability": {
|
||||||
"emit_tokens": 33,
|
"writes_during_outage": 3,
|
||||||
"claimed_ok": true,
|
"reported_queued": 3,
|
||||||
"detected": false,
|
"landed_after_flush": 3,
|
||||||
"persisted": true,
|
"lost": 0,
|
||||||
"duplicates": 0,
|
"duplicated_on_reflush": 0
|
||||||
"silent_failure": false
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"scenario": "session-log-flaky-network",
|
|
||||||
"fault": "one-time 503 (transient)",
|
|
||||||
"06": {
|
|
||||||
"emit_tokens": 64,
|
|
||||||
"claimed_ok": true,
|
|
||||||
"detected": false,
|
|
||||||
"persisted": false,
|
|
||||||
"duplicates": 0,
|
|
||||||
"silent_failure": true
|
|
||||||
},
|
|
||||||
"07": {
|
|
||||||
"emit_tokens": 17,
|
|
||||||
"claimed_ok": true,
|
|
||||||
"detected": false,
|
|
||||||
"persisted": true,
|
|
||||||
"duplicates": 0,
|
|
||||||
"silent_failure": false
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"scenario": "heartbeat-phantom-write",
|
|
||||||
"fault": "accepted-but-not-persisted",
|
|
||||||
"06": {
|
|
||||||
"emit_tokens": 61,
|
|
||||||
"claimed_ok": true,
|
|
||||||
"detected": false,
|
|
||||||
"persisted": false,
|
|
||||||
"duplicates": 0,
|
|
||||||
"silent_failure": true
|
|
||||||
},
|
|
||||||
"07": {
|
|
||||||
"emit_tokens": 14,
|
|
||||||
"claimed_ok": false,
|
|
||||||
"detected": true,
|
|
||||||
"persisted": false,
|
|
||||||
"duplicates": 0,
|
|
||||||
"silent_failure": false
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"scenario": "cold-start-load-6-reads",
|
|
||||||
"fault": "(none)",
|
|
||||||
"06": {
|
|
||||||
"emit_tokens": 192,
|
|
||||||
"claimed_ok": true,
|
|
||||||
"detected": false,
|
|
||||||
"persisted": true,
|
|
||||||
"duplicates": 0,
|
|
||||||
"silent_failure": false
|
|
||||||
},
|
|
||||||
"07": {
|
|
||||||
"emit_tokens": 64,
|
|
||||||
"claimed_ok": true,
|
|
||||||
"detected": false,
|
|
||||||
"persisted": true,
|
|
||||||
"duplicates": 0,
|
|
||||||
"silent_failure": false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"totals": {
|
|
||||||
"0.6": {
|
|
||||||
"gen_tokens": 608,
|
|
||||||
"silent_failures": 4,
|
|
||||||
"detected_failures": 0,
|
|
||||||
"duplicates": 1,
|
|
||||||
"effective_tokens": 6608
|
|
||||||
},
|
|
||||||
"0.7": {
|
|
||||||
"gen_tokens": 174,
|
|
||||||
"silent_failures": 0,
|
|
||||||
"detected_failures": 2,
|
|
||||||
"duplicates": 0,
|
|
||||||
"effective_tokens": 334
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"silent_error_free": {
|
|
||||||
"0.6": "1/5",
|
|
||||||
"0.7": "5/5"
|
|
||||||
},
|
|
||||||
"writes_persisted": {
|
|
||||||
"0.6": "2/5",
|
|
||||||
"0.7": "3/5"
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
+320
-255
@@ -1,41 +1,52 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""run_eval.py — A/B eval of echo-memory 0.6 (raw curl, documented recipes) vs
|
"""run_eval.py — current-version metrics for echo-memory (v1.5+).
|
||||||
0.7 (the shipped echo.py client) over representative memory operations, with
|
|
||||||
fault injection.
|
|
||||||
|
|
||||||
Design
|
Replaces the historical 0.6-vs-0.7 A/B harness (see git history for that version).
|
||||||
------
|
Measures the four things the plugin now claims, against the deterministic mock
|
||||||
* A mock Obsidian REST API (mock_olrapi.py) gives deterministic behavior + faults,
|
REST API (mock_olrapi.py) — no credentials, no live vault:
|
||||||
so no credentials are needed and the real vault is never touched.
|
|
||||||
* The 0.7 side runs the ACTUAL shipped scripts/echo.py (status-checked, retry, verify,
|
|
||||||
idempotent append).
|
|
||||||
* The 0.6 side faithfully models the documented raw-curl recipes: it performs the
|
|
||||||
same HTTP but does NOT inspect status, does NOT retry, does NOT verify, and does
|
|
||||||
NOT dedupe — exactly what SKILL.md 0.6 told the model to emit.
|
|
||||||
* Ground truth is read back independently from the mock after each method, so a
|
|
||||||
"silent failure" (method reported success but the vault is actually wrong, and
|
|
||||||
nobody noticed) is detected regardless of what the method claimed.
|
|
||||||
* Each (scenario, method) runs against a freshly reset + re-seeded server, so faults
|
|
||||||
(e.g. one-time 503) are identical for both methods.
|
|
||||||
|
|
||||||
Metrics
|
1. RETRIEVAL — hybrid recall (BM25 over entities + sessions/journal, fused with
|
||||||
-------
|
freshness/status priors) vs a keyword-only baseline modeling the
|
||||||
* gen_tokens : output tokens the MODEL must generate for the op (len(emitted)/CPT).
|
pre-1.5 recall (per-word substring ranking over ENTITY notes only).
|
||||||
* silent_failure : claimed success BUT ground truth is wrong (lost write or duplicate).
|
Metrics: precision@5, recall@5, MRR, session/journal queries
|
||||||
* detected : the method surfaced the failure (loud, retryable) instead of hiding it.
|
answered, freshness top-1 correctness.
|
||||||
* effective_tokens = gen_tokens + silent_failures*RECOVERY + detected*DETECT_COST
|
2. DEDUP — duplicate notes created replaying a capture stream with the
|
||||||
(RECOVERY/DETECT_COST are labeled assumptions, tune via env).
|
pre-write gate ON (default) vs OFF (ECHO_DUP_GATE=99, i.e. the
|
||||||
|
pre-1.5 warn-only behavior). Also counts legitimate captures
|
||||||
|
wrongly blocked (must be 0).
|
||||||
|
3. WRITE SAFETY— silent write failures (claimed success but ground truth wrong)
|
||||||
|
across fault injection: transient 503, phantom PUT, PATCH to a
|
||||||
|
missing heading, replayed append. Must be 0; failures must be LOUD.
|
||||||
|
4. DURABILITY — writes issued while the vault is unreachable must queue, replay
|
||||||
|
exactly once when it returns, and never duplicate on re-flush.
|
||||||
|
|
||||||
Usage: python3 run_eval.py [--port 8799] [--cpt 4] [--recovery 1500] [--detect-cost 80]
|
Ground truth is always read back independently via the mock's __debug__ endpoint.
|
||||||
|
Results print as a table and land in results/latest.json.
|
||||||
|
|
||||||
|
Usage: python3 run_eval.py [--port 8797]
|
||||||
"""
|
"""
|
||||||
import os, sys, json, time, subprocess, tempfile, argparse, urllib.request, urllib.error
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
HERE = Path(__file__).resolve().parent
|
HERE = Path(__file__).resolve().parent
|
||||||
ECHO = HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts" / "echo.py"
|
SCRIPTS = HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts"
|
||||||
|
ECHO = SCRIPTS / "echo.py"
|
||||||
|
SWEEP = SCRIPTS / "sweep.py"
|
||||||
KEY = "test-key-not-a-real-secret"
|
KEY = "test-key-not-a-real-secret"
|
||||||
|
TODAY = "2026-07-03"
|
||||||
|
|
||||||
|
|
||||||
# ----- tiny HTTP helpers (used for setup, ground truth, and the 0.6 model) -----
|
|
||||||
def http(method, url, body=None, headers=None):
|
def http(method, url, body=None, headers=None):
|
||||||
data = body.encode() if isinstance(body, str) else body
|
data = body.encode() if isinstance(body, str) else body
|
||||||
req = urllib.request.Request(url, data=data, method=method,
|
req = urllib.request.Request(url, data=data, method=method,
|
||||||
@@ -45,259 +56,313 @@ def http(method, url, body=None, headers=None):
|
|||||||
return r.status, r.read().decode("utf-8", "replace")
|
return r.status, r.read().decode("utf-8", "replace")
|
||||||
except urllib.error.HTTPError as e:
|
except urllib.error.HTTPError as e:
|
||||||
return e.code, e.read().decode("utf-8", "replace")
|
return e.code, e.read().decode("utf-8", "replace")
|
||||||
except Exception as e:
|
except Exception as e: # noqa: BLE001
|
||||||
return 0, str(e)
|
return 0, str(e)
|
||||||
|
|
||||||
class Eval:
|
|
||||||
def __init__(self, base, cpt):
|
class Harness:
|
||||||
self.base, self.cpt = base, cpt
|
def __init__(self, base, state_dir):
|
||||||
def reset(self): http("POST", f"{self.base}/__debug__reset")
|
self.base = base
|
||||||
def seed(self, path, content): http("PUT", f"{self.base}/vault/{path}", content)
|
self.state_dir = state_dir
|
||||||
|
|
||||||
|
def reset(self):
|
||||||
|
http("POST", f"{self.base}/__debug__reset")
|
||||||
|
|
||||||
|
def seed(self, path, content):
|
||||||
|
http("PUT", f"{self.base}/vault/{path}", content)
|
||||||
|
|
||||||
def ground(self, path):
|
def ground(self, path):
|
||||||
st, body = http("GET", f"{self.base}/__debug__?path={path}")
|
_, body = http("GET", f"{self.base}/__debug__?path={path}")
|
||||||
return None if body == "<<MISSING>>" else body
|
return None if body == "<<MISSING>>" else body
|
||||||
def toks(self, text): return round(len(text) / self.cpt)
|
|
||||||
def echo(self, *args):
|
|
||||||
env = dict(os.environ, ECHO_BASE=self.base, ECHO_KEY=KEY, ECHO_VERIFY="1", ECHO_LOCK_TTL="900")
|
|
||||||
p = subprocess.run([sys.executable, str(ECHO), *args], capture_output=True, text=True, env=env)
|
|
||||||
return p.returncode
|
|
||||||
|
|
||||||
# ---- faithful 0.6 recipe text (what the model emitted), for token accounting ----
|
def run(self, script, *args, env_extra=None, base=None):
|
||||||
def recipe_get(path):
|
env = dict(os.environ, ECHO_BASE=base or self.base, ECHO_KEY=KEY,
|
||||||
return (f'curl -s -H "Authorization: Bearer {KEY}" '
|
ECHO_VERIFY="1", ECHO_TODAY=TODAY, ECHO_STATE_DIR=self.state_dir,
|
||||||
f'"https://echoapi.alwisp.com/vault/{path}"')
|
**(env_extra or {}))
|
||||||
def recipe_put(path):
|
return subprocess.run([sys.executable, str(script), *args],
|
||||||
return (f"cat > /tmp/obs_file.md << 'EOF'\n<body>\nEOF\n\n"
|
capture_output=True, text=True, env=env)
|
||||||
f'curl -s -X PUT -H "Authorization: Bearer {KEY}" '
|
|
||||||
f'-H "Content-Type: text/markdown" --data-binary @/tmp/obs_file.md '
|
|
||||||
f'"https://echoapi.alwisp.com/vault/{path}"')
|
|
||||||
def recipe_post(path):
|
|
||||||
return (f"cat > /tmp/obs_entry.md << 'EOF'\n<line>\nEOF\n\n"
|
|
||||||
f'curl -s -X POST -H "Authorization: Bearer {KEY}" '
|
|
||||||
f'-H "Content-Type: text/markdown" --data-binary @/tmp/obs_entry.md '
|
|
||||||
f'"https://echoapi.alwisp.com/vault/{path}"')
|
|
||||||
def recipe_patch(path, target):
|
|
||||||
return (f"cat > /tmp/obs_patch.md << 'EOF'\n<body>\nEOF\n\n"
|
|
||||||
f'curl -s -X PATCH -H "Authorization: Bearer {KEY}" '
|
|
||||||
f'-H "Operation: append" -H "Target-Type: heading" -H "Target: {target}" '
|
|
||||||
f'-H "Content-Type: text/markdown" --data-binary @/tmp/obs_patch.md '
|
|
||||||
f'"https://echoapi.alwisp.com/vault/{path}"')
|
|
||||||
|
|
||||||
def tmpfile(content):
|
|
||||||
f = tempfile.NamedTemporaryFile("w", suffix=".md", delete=False); f.write(content); f.close()
|
def note(fm_type, title, body, status="active", created=TODAY, updated=TODAY, tags=None):
|
||||||
|
tags_s = "[" + ", ".join(tags or [fm_type]) + "]"
|
||||||
|
return (f"---\ntype: {fm_type}\nstatus: {status}\ncreated: {created}\n"
|
||||||
|
f"updated: {updated}\ntags: {tags_s}\nagent_written: true\nsource_notes: []\n---\n\n"
|
||||||
|
f"# {title}\n\n{body}\n\n## Related\n")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 1. RETRIEVAL
|
||||||
|
ENTITY_CORPUS = {
|
||||||
|
"resources/people/maria-lopez.md": note(
|
||||||
|
"person", "Maria Lopez", "Runs procurement at Vantage Systems. Prefers email over calls."),
|
||||||
|
"resources/companies/vantage-systems.md": note(
|
||||||
|
"company", "Vantage Systems", "Enterprise procurement platform vendor; the contract renews in Q3."),
|
||||||
|
"resources/concepts/bm25-ranking.md": note(
|
||||||
|
"concept", "BM25 Ranking", "Lexical retrieval scoring with term frequency saturation and document length normalization."),
|
||||||
|
"resources/concepts/graph-expansion.md": note(
|
||||||
|
"concept", "Graph Expansion", "Expanding a hit into its linked neighbourhood along Related links and source notes."),
|
||||||
|
"projects/active/website-redesign.md": note(
|
||||||
|
"project", "Website Redesign",
|
||||||
|
"Rebuild the marketing site. Navigation cleanup and new hero copy.",
|
||||||
|
status="active", created="2026-06-25", updated="2026-07-01"),
|
||||||
|
"projects/archived/old-website.md": note(
|
||||||
|
"project", "Old Website",
|
||||||
|
"Legacy marketing site. Navigation used nested dropdown menus. Navigation was the top complaint.",
|
||||||
|
status="archived", created="2024-03-01", updated="2024-08-10"),
|
||||||
|
"_agent/memory/semantic/tooling-preferences.md": note(
|
||||||
|
"semantic-memory", "Tooling Preferences",
|
||||||
|
"The operator prefers uv over pip for Python tooling, and ruff for linting."),
|
||||||
|
}
|
||||||
|
TIME_SERIES_CORPUS = {
|
||||||
|
"_agent/sessions/2026-06-20-1000-vantage-contract-review.md": note(
|
||||||
|
"session-log", "Vantage contract review",
|
||||||
|
"Reviewed the Vantage contract. Decided to renegotiate the SLA penalty clause before renewal.",
|
||||||
|
status="complete", created="2026-06-20", updated="2026-06-20", tags=["session-log"]),
|
||||||
|
"journal/daily/2026-06-20.md": (
|
||||||
|
"---\ntype: daily-note\ncreated: 2026-06-20\nupdated: 2026-06-20\ntags: [daily]\n---\n\n"
|
||||||
|
"# 2026-06-20\n\n## Agent Log\n- 2026-06-20: renegotiation kickoff email sent to Vantage\n"),
|
||||||
|
}
|
||||||
|
|
||||||
|
# (query, gold paths, only answerable from sessions/journal)
|
||||||
|
QUERIES = [
|
||||||
|
("uv or pip for python tooling", {"_agent/memory/semantic/tooling-preferences.md"}, False),
|
||||||
|
("procurement contact", {"resources/people/maria-lopez.md",
|
||||||
|
"resources/companies/vantage-systems.md"}, False),
|
||||||
|
("SLA penalty clause decision",
|
||||||
|
{"_agent/sessions/2026-06-20-1000-vantage-contract-review.md"}, True),
|
||||||
|
("marketing site navigation", {"projects/active/website-redesign.md",
|
||||||
|
"projects/archived/old-website.md"}, False),
|
||||||
|
("term frequency saturation", {"resources/concepts/bm25-ranking.md"}, False),
|
||||||
|
("contract renewal Q3", {"resources/companies/vantage-systems.md"}, False),
|
||||||
|
("renegotiation kickoff", {"journal/daily/2026-06-20.md"}, True),
|
||||||
|
("graph neighbourhood expansion", {"resources/concepts/graph-expansion.md"}, False),
|
||||||
|
]
|
||||||
|
FRESHNESS_QUERY = "marketing site navigation"
|
||||||
|
FRESHNESS_TOP1 = "projects/active/website-redesign.md"
|
||||||
|
|
||||||
|
|
||||||
|
def keyword_baseline(query, corpus, k=5):
|
||||||
|
"""Models pre-1.5 keyword recall: per-word substring counting over ENTITY note
|
||||||
|
bodies only (sessions/journal were not in the corpus before 1.5)."""
|
||||||
|
words = [w for w in re.findall(r"[a-z0-9]+", query.lower()) if len(w) > 2]
|
||||||
|
scored = []
|
||||||
|
for path, text in corpus.items():
|
||||||
|
body = text.lower()
|
||||||
|
s = sum(body.count(w) for w in words)
|
||||||
|
if s > 0:
|
||||||
|
scored.append((s, path))
|
||||||
|
scored.sort(key=lambda x: (-x[0], x[1]))
|
||||||
|
return [p for _, p in scored[:k]]
|
||||||
|
|
||||||
|
|
||||||
|
def metrics_for(ranked_by_query):
|
||||||
|
p_sum = r_sum = mrr_sum = 0.0
|
||||||
|
ts_answered = ts_total = 0
|
||||||
|
for (q, gold, ts_only), ranked in ranked_by_query:
|
||||||
|
top = ranked[:5]
|
||||||
|
hits = [p for p in top if p in gold]
|
||||||
|
p_sum += len(hits) / 5
|
||||||
|
r_sum += len(hits) / len(gold)
|
||||||
|
rr = 0.0
|
||||||
|
for i, p in enumerate(top):
|
||||||
|
if p in gold:
|
||||||
|
rr = 1.0 / (i + 1)
|
||||||
|
break
|
||||||
|
mrr_sum += rr
|
||||||
|
if ts_only:
|
||||||
|
ts_total += 1
|
||||||
|
ts_answered += int(bool(hits))
|
||||||
|
n = len(ranked_by_query)
|
||||||
|
return {"precision_at_5": round(p_sum / n, 3), "recall_at_5": round(r_sum / n, 3),
|
||||||
|
"mrr": round(mrr_sum / n, 3),
|
||||||
|
"session_journal_queries_answered": f"{ts_answered}/{ts_total}"}
|
||||||
|
|
||||||
|
|
||||||
|
def eval_retrieval(h):
|
||||||
|
h.reset()
|
||||||
|
h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n")
|
||||||
|
for path, text in {**ENTITY_CORPUS, **TIME_SERIES_CORPUS}.items():
|
||||||
|
h.seed(path, text)
|
||||||
|
r = h.run(SWEEP, "--apply")
|
||||||
|
assert r.returncode == 0, f"sweep failed: {r.stdout}{r.stderr}"
|
||||||
|
|
||||||
|
current_ranked, baseline_ranked = [], []
|
||||||
|
fresh_top1_current = fresh_top1_baseline = None
|
||||||
|
for q, gold, ts_only in QUERIES:
|
||||||
|
r = h.run(ECHO, "recall", q, "--json", "--limit", "5")
|
||||||
|
try:
|
||||||
|
primary = [hit["path"] for hit in json.loads(r.stdout).get("primary", [])]
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
primary = []
|
||||||
|
current_ranked.append(((q, gold, ts_only), primary))
|
||||||
|
base = keyword_baseline(q, ENTITY_CORPUS)
|
||||||
|
baseline_ranked.append(((q, gold, ts_only), base))
|
||||||
|
if q == FRESHNESS_QUERY:
|
||||||
|
fresh_top1_current = primary[0] if primary else None
|
||||||
|
fresh_top1_baseline = base[0] if base else None
|
||||||
|
|
||||||
|
cur = metrics_for(current_ranked)
|
||||||
|
cur["freshness_top1_correct"] = fresh_top1_current == FRESHNESS_TOP1
|
||||||
|
bas = metrics_for(baseline_ranked)
|
||||||
|
bas["freshness_top1_correct"] = fresh_top1_baseline == FRESHNESS_TOP1
|
||||||
|
return {"current (hybrid+priors, full corpus)": cur,
|
||||||
|
"baseline (keyword, entities only)": bas}
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------- 2. DEDUP
|
||||||
|
# Attempts referring to an EXISTING entity under another name; a create == duplicate.
|
||||||
|
DUP_ATTEMPTS = [
|
||||||
|
("Robert Smith", "person", "resources/people/robert-smith.md"),
|
||||||
|
("Echo Memory", "project", "projects/active/echo-memory.md"),
|
||||||
|
("Acme Consulting", "company", "resources/companies/acme-consulting.md"),
|
||||||
|
]
|
||||||
|
# Legitimate NEW entities; a block == false positive (must create in both modes).
|
||||||
|
LEGIT_ATTEMPTS = [
|
||||||
|
("Echo Review Meeting", "meeting",
|
||||||
|
f"resources/meetings/{TODAY}-echo-review-meeting.md"), # cross-kind lookalike
|
||||||
|
("Zephyr", "concept", "resources/concepts/zephyr.md"), # unrelated
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _dedup_pass(h, gate_env):
|
||||||
|
h.reset()
|
||||||
|
h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n")
|
||||||
|
for title, kind in [("Bob Smith", "person"), ("Echo", "project"), ("Acme", "company")]:
|
||||||
|
h.run(ECHO, "capture", title, "--kind", kind, env_extra=gate_env)
|
||||||
|
dupes = blocked_legit = 0
|
||||||
|
for title, kind, would_be in DUP_ATTEMPTS:
|
||||||
|
h.run(ECHO, "capture", title, "--kind", kind, env_extra=gate_env)
|
||||||
|
dupes += int(h.ground(would_be) is not None)
|
||||||
|
for title, kind, path in LEGIT_ATTEMPTS:
|
||||||
|
h.run(ECHO, "capture", title, "--kind", kind, env_extra=gate_env)
|
||||||
|
blocked_legit += int(h.ground(path) is None)
|
||||||
|
return {"duplicate_notes_created": dupes, "legitimate_captures_blocked": blocked_legit,
|
||||||
|
"referring_attempts": len(DUP_ATTEMPTS), "legit_attempts": len(LEGIT_ATTEMPTS)}
|
||||||
|
|
||||||
|
|
||||||
|
def eval_dedup(h):
|
||||||
|
return {"gate ON (v1.5.1 default)": _dedup_pass(h, {}),
|
||||||
|
"gate OFF (pre-1.5, warn-only)": _dedup_pass(h, {"ECHO_DUP_GATE": "99"})}
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------- 3. WRITE SAFETY
|
||||||
|
def eval_write_safety(h):
|
||||||
|
h.reset()
|
||||||
|
h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n")
|
||||||
|
silent = loud = 0
|
||||||
|
ops = 0
|
||||||
|
|
||||||
|
def bodyfile(text):
|
||||||
|
f = tempfile.NamedTemporaryFile("w", suffix=".md", delete=False, encoding="utf-8")
|
||||||
|
f.write(text)
|
||||||
|
f.close()
|
||||||
return f.name
|
return f.name
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# transient 503 on first write ("flaky" marker) -> client must retry and land it
|
||||||
# Scenarios. Each defines seed(), and a run for each method that returns:
|
ops += 1
|
||||||
# {emit: <text model generates>, claimed_ok: bool, detected: bool}
|
r = h.run(ECHO, "put", "resources/concepts/flaky-note.md", bodyfile("# Flaky\nok\n"))
|
||||||
# plus a ground-truth check producing {persisted, duplicates}.
|
landed = h.ground("resources/concepts/flaky-note.md") is not None
|
||||||
# ---------------------------------------------------------------------------
|
if r.returncode == 0 and not landed:
|
||||||
def scenarios(ev):
|
silent += 1
|
||||||
out = []
|
if r.returncode != 0:
|
||||||
|
loud += 1
|
||||||
|
|
||||||
# S1 — agent-log append where the heading is MISSING (-> 400 invalid-target)
|
# phantom PUT ("phantom" marker: 200 but not persisted) -> verify must fail LOUD
|
||||||
def s1():
|
ops += 1
|
||||||
path, tgt = "journal/daily/2026-06-19.md", "2026-06-19::Agent Log"
|
r = h.run(ECHO, "put", "resources/concepts/phantom-note.md", bodyfile("# Phantom\n"))
|
||||||
line = "- 2026-06-19: EVALMARK-s1"
|
landed = h.ground("resources/concepts/phantom-note.md") is not None
|
||||||
def seed(): ev.seed(path, "---\ntype: daily-note\n---\n\n# 2026-06-19\n\n## Notes\n")
|
if r.returncode == 0 and not landed:
|
||||||
def m06():
|
silent += 1
|
||||||
http("PATCH", f"{ev.base}/vault/{path}", line,
|
if r.returncode != 0 and not landed:
|
||||||
{"Operation": "append", "Target-Type": "heading", "Target": tgt}) # status ignored
|
loud += 1
|
||||||
return {"emit": recipe_patch(path, tgt), "claimed_ok": True, "detected": False}
|
|
||||||
def m07():
|
|
||||||
rc = ev.echo("patch", path, "append", "heading", tgt, tmpfile(line))
|
|
||||||
return {"emit": f'"$ECHO" patch {path} append heading "{tgt}" /tmp/x.md',
|
|
||||||
"claimed_ok": rc == 0, "detected": rc != 0}
|
|
||||||
def gt():
|
|
||||||
c = ev.ground(path) or ""
|
|
||||||
return {"persisted": "EVALMARK-s1" in c, "duplicates": max(0, c.count("EVALMARK-s1") - 1)}
|
|
||||||
return dict(name="agent-log-missing-heading", fault="bad heading -> 400 invalid-target",
|
|
||||||
seed=seed, m06=m06, m07=m07, gt=gt)
|
|
||||||
out.append(s1())
|
|
||||||
|
|
||||||
# S2 — scope switch (no fault; pure token comparison on a PATCH replace)
|
# PATCH to a missing heading -> 400 must be surfaced, nothing written
|
||||||
def s2():
|
ops += 1
|
||||||
path, tgt = "_agent/context/current-context.md", "Current Context::Scope"
|
h.seed("resources/concepts/target.md", "# Target\n\n## Real\nx\n")
|
||||||
def seed(): ev.seed(path, "---\ntype: context-bundle\n---\n\n# Current Context\n\n## Scope\nold scope\n")
|
r = h.run(ECHO, "patch", "resources/concepts/target.md", "append", "heading",
|
||||||
def m06():
|
"Target::Nope", bodyfile("lost?"))
|
||||||
http("PATCH", f"{ev.base}/vault/{path}", "EVALMARK-s2 new scope",
|
if r.returncode == 0 and "lost?" not in (h.ground("resources/concepts/target.md") or ""):
|
||||||
{"Operation": "replace", "Target-Type": "heading", "Target": tgt})
|
silent += 1
|
||||||
return {"emit": recipe_patch(path, tgt), "claimed_ok": True, "detected": False}
|
if r.returncode != 0:
|
||||||
def m07():
|
loud += 1
|
||||||
rc = ev.echo("patch", path, "replace", "heading", tgt, tmpfile("EVALMARK-s2 new scope"))
|
|
||||||
return {"emit": f'"$ECHO" patch {path} replace heading "{tgt}" /tmp/x.md',
|
|
||||||
"claimed_ok": rc == 0, "detected": rc != 0}
|
|
||||||
def gt():
|
|
||||||
c = ev.ground(path) or ""
|
|
||||||
return {"persisted": "EVALMARK-s2" in c, "duplicates": 0}
|
|
||||||
return dict(name="scope-switch", fault="(none)", seed=seed, m06=m06, m07=m07, gt=gt)
|
|
||||||
out.append(s2())
|
|
||||||
|
|
||||||
# S3 — inbox capture issued TWICE (retry/replay): dedup vs duplicate line
|
# replayed append -> whole-line idempotency, no duplicate
|
||||||
def s3():
|
ops += 1
|
||||||
path = "inbox/captures/inbox.md"; line = "- 2026-06-19: EVALMARK-s3"
|
h.run(ECHO, "append", "inbox/captures/inbox.md", f"- {TODAY}: replay me")
|
||||||
def seed(): ev.seed(path, "# Inbox\n")
|
h.run(ECHO, "append", "inbox/captures/inbox.md", f"- {TODAY}: replay me")
|
||||||
def m06():
|
n = (h.ground("inbox/captures/inbox.md") or "").count("replay me")
|
||||||
http("POST", f"{ev.base}/vault/{path}", line + "\n") # no idempotency check
|
if n != 1:
|
||||||
http("POST", f"{ev.base}/vault/{path}", line + "\n")
|
silent += 1
|
||||||
return {"emit": recipe_post(path) + "\n# (repeated on retry)\n" + recipe_post(path),
|
|
||||||
"claimed_ok": True, "detected": False}
|
|
||||||
def m07():
|
|
||||||
rc1 = ev.echo("append", path, line)
|
|
||||||
rc2 = ev.echo("append", path, line) # second is skipped by read-before-POST
|
|
||||||
return {"emit": f'"$ECHO" append {path} "{line}"\n"$ECHO" append {path} "{line}"',
|
|
||||||
"claimed_ok": rc1 == 0 and rc2 == 0, "detected": False}
|
|
||||||
def gt():
|
|
||||||
c = ev.ground(path) or ""
|
|
||||||
return {"persisted": "EVALMARK-s3" in c, "duplicates": max(0, c.count("EVALMARK-s3") - 1)}
|
|
||||||
return dict(name="inbox-capture-replayed", fault="duplicate attempt (retry/replay)",
|
|
||||||
seed=seed, m06=m06, m07=m07, gt=gt)
|
|
||||||
out.append(s3())
|
|
||||||
|
|
||||||
# S4 — session-log PUT under a flaky network (one-time 503 then success)
|
return {"fault_scenarios": ops, "silent_write_failures": silent,
|
||||||
def s4():
|
"failures_surfaced_loud": loud}
|
||||||
path = "_agent/sessions/2026-06-19-1430-flaky-eval.md"
|
|
||||||
body = "---\ntype: session-log\n---\n\n# Session\nEVALMARK-s4\n"
|
|
||||||
def seed(): pass # file does not pre-exist
|
|
||||||
def m06():
|
|
||||||
http("PUT", f"{ev.base}/vault/{path}", body) # single shot, no retry, status ignored
|
|
||||||
return {"emit": recipe_put(path), "claimed_ok": True, "detected": False}
|
|
||||||
def m07():
|
|
||||||
rc = ev.echo("put", path, tmpfile(body)) # echo.py retries the 503 once
|
|
||||||
return {"emit": f'"$ECHO" put {path} /tmp/x.md', "claimed_ok": rc == 0, "detected": rc != 0}
|
|
||||||
def gt():
|
|
||||||
c = ev.ground(path) or ""
|
|
||||||
return {"persisted": "EVALMARK-s4" in c, "duplicates": 0}
|
|
||||||
return dict(name="session-log-flaky-network", fault="one-time 503 (transient)",
|
|
||||||
seed=seed, m06=m06, m07=m07, gt=gt)
|
|
||||||
out.append(s4())
|
|
||||||
|
|
||||||
# S5 — heartbeat PUT accepted-but-not-persisted (proxy hiccup): verify catches it
|
|
||||||
def s5():
|
|
||||||
path = "_agent/heartbeat/phantom-eval.md"; body = "EVALMARK-s5 @ 2026-06-19T14:30:00Z\n"
|
|
||||||
def seed(): pass
|
|
||||||
def m06():
|
|
||||||
http("PUT", f"{ev.base}/vault/{path}", body) # 200 returned; no read-back verify
|
|
||||||
return {"emit": recipe_put(path), "claimed_ok": True, "detected": False}
|
|
||||||
def m07():
|
|
||||||
rc = ev.echo("put", path, tmpfile(body)) # verify GET -> 404 -> die
|
|
||||||
return {"emit": f'"$ECHO" put {path} /tmp/x.md', "claimed_ok": rc == 0, "detected": rc != 0}
|
|
||||||
def gt():
|
|
||||||
c = ev.ground(path) or ""
|
|
||||||
return {"persisted": "EVALMARK-s5" in c, "duplicates": 0}
|
|
||||||
return dict(name="heartbeat-phantom-write", fault="accepted-but-not-persisted",
|
|
||||||
seed=seed, m06=m06, m07=m07, gt=gt)
|
|
||||||
out.append(s5())
|
|
||||||
|
|
||||||
# S6 — cold-start load: 6 reads (no fault; pure token comparison)
|
# --------------------------------------------------------------- 4. DURABILITY
|
||||||
def s6():
|
def eval_durability(h, dead_base):
|
||||||
paths = ["_agent/echo-vault.md", "_agent/memory/semantic/operator-preferences.md",
|
h.reset()
|
||||||
"_agent/context/current-context.md", "_agent/heartbeat/last-session.md",
|
h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n")
|
||||||
"journal/daily/2026-06-19.md", "inbox/captures/inbox.md"]
|
queued_ok = 0
|
||||||
def seed():
|
for i in range(3):
|
||||||
for p in paths: ev.seed(p, f"# {p}\nEVALMARK-s6\n")
|
r = h.run(ECHO, "append", "inbox/captures/inbox.md",
|
||||||
def m06():
|
f"- {TODAY}: offline capture {i}", base=dead_base)
|
||||||
for p in paths: http("GET", f"{ev.base}/vault/{p}")
|
queued_ok += int(r.returncode == 0 and "queued" in r.stdout)
|
||||||
return {"emit": "\n".join(recipe_get(p) for p in paths), "claimed_ok": True, "detected": False}
|
# vault returns: flush replays; a second flush must not duplicate
|
||||||
def m07():
|
h.run(ECHO, "flush")
|
||||||
for p in paths: ev.echo("get", p)
|
h.run(ECHO, "flush")
|
||||||
return {"emit": "\n".join(f'"$ECHO" get {p}' for p in paths), "claimed_ok": True, "detected": False}
|
inbox = h.ground("inbox/captures/inbox.md") or ""
|
||||||
def gt(): return {"persisted": True, "duplicates": 0}
|
landed = sum(1 for i in range(3) if f"offline capture {i}" in inbox)
|
||||||
return dict(name="cold-start-load-6-reads", fault="(none)", seed=seed, m06=m06, m07=m07, gt=gt)
|
dupes = sum(inbox.count(f"offline capture {i}") - 1
|
||||||
out.append(s6())
|
for i in range(3) if f"offline capture {i}" in inbox)
|
||||||
|
return {"writes_during_outage": 3, "reported_queued": queued_ok,
|
||||||
|
"landed_after_flush": landed, "lost": 3 - landed, "duplicated_on_reflush": dupes}
|
||||||
|
|
||||||
return out
|
|
||||||
|
|
||||||
def run_method(ev, scn, key):
|
|
||||||
ev.reset(); scn["seed"]()
|
|
||||||
res = scn[key]()
|
|
||||||
g = scn["gt"]()
|
|
||||||
# a write op "should persist"; reads (s6) and replays are judged by their own gt
|
|
||||||
bad = (not g["persisted"] and scn["name"] != "cold-start-load-6-reads") or g["duplicates"] > 0
|
|
||||||
silent = bool(res["claimed_ok"] and bad)
|
|
||||||
return {"emit_tokens": ev.toks(res["emit"]), "claimed_ok": res["claimed_ok"],
|
|
||||||
"detected": res["detected"], "persisted": g["persisted"],
|
|
||||||
"duplicates": g["duplicates"], "silent_failure": silent}
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------ main
|
||||||
def main():
|
def main():
|
||||||
ap = argparse.ArgumentParser()
|
ap = argparse.ArgumentParser()
|
||||||
ap.add_argument("--port", type=int, default=8799)
|
ap.add_argument("--port", type=int, default=8797)
|
||||||
ap.add_argument("--cpt", type=float, default=4.0, help="chars per token")
|
|
||||||
ap.add_argument("--recovery", type=int, default=1500, help="token penalty per silent failure (recovery loop)")
|
|
||||||
ap.add_argument("--detect-cost", type=int, default=80, help="token cost to re-issue a corrected call after a detected failure")
|
|
||||||
a = ap.parse_args()
|
a = ap.parse_args()
|
||||||
|
|
||||||
base = f"http://127.0.0.1:{a.port}"
|
base = f"http://127.0.0.1:{a.port}"
|
||||||
srv = subprocess.Popen([sys.executable, str(HERE / "mock_olrapi.py"), "--port", str(a.port)],
|
srv = subprocess.Popen([sys.executable, str(HERE / "mock_olrapi.py"), "--port", str(a.port)],
|
||||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
||||||
|
state = tempfile.mkdtemp(prefix="echo-eval-state-")
|
||||||
try:
|
try:
|
||||||
for _ in range(50): # wait for ready
|
for _ in range(50):
|
||||||
try:
|
try:
|
||||||
urllib.request.urlopen(f"{base}/__debug__reset", data=b"", timeout=1); break
|
urllib.request.urlopen(f"{base}/__debug__reset", data=b"", timeout=1)
|
||||||
except Exception: time.sleep(0.1)
|
break
|
||||||
ev = Eval(base, a.cpt)
|
except Exception: # noqa: BLE001
|
||||||
rows, agg = [], {"06": {}, "07": {}}
|
time.sleep(0.1)
|
||||||
for scn in scenarios(ev):
|
h = Harness(base, state)
|
||||||
r06 = run_method(ev, scn, "m06")
|
results = {
|
||||||
r07 = run_method(ev, scn, "m07")
|
"version": "1.5.1", "date": TODAY,
|
||||||
rows.append({"scenario": scn["name"], "fault": scn["fault"], "06": r06, "07": r07})
|
"retrieval": eval_retrieval(h),
|
||||||
|
"dedup": eval_dedup(h),
|
||||||
|
"write_safety": eval_write_safety(h),
|
||||||
|
"durability": eval_durability(h, dead_base=f"http://127.0.0.1:{a.port + 1}"),
|
||||||
|
}
|
||||||
|
out = HERE / "results" / "latest.json"
|
||||||
|
out.parent.mkdir(exist_ok=True)
|
||||||
|
out.write_text(json.dumps(results, indent=2), encoding="utf-8")
|
||||||
|
|
||||||
def totals(side):
|
print(f"\n== echo-memory eval — v{results['version']} ({results['date']}) ==\n")
|
||||||
gen = sum(r[side]["emit_tokens"] for r in rows)
|
for section, data in results.items():
|
||||||
sil = sum(1 for r in rows if r[side]["silent_failure"])
|
if section in ("version", "date"):
|
||||||
det = sum(1 for r in rows if r[side]["detected"])
|
continue
|
||||||
dup = sum(r[side]["duplicates"] for r in rows)
|
print(f"## {section}")
|
||||||
eff = gen + sil * a.recovery + det * a.detect_cost
|
for k, v in data.items():
|
||||||
return dict(gen_tokens=gen, silent_failures=sil, detected_failures=det,
|
if isinstance(v, dict):
|
||||||
duplicates=dup, effective_tokens=eff)
|
print(f" {k}:")
|
||||||
T06, T07 = totals("06"), totals("07")
|
for k2, v2 in v.items():
|
||||||
|
print(f" {k2:38} {v2}")
|
||||||
# ---- report ----
|
else:
|
||||||
line = "=" * 78
|
print(f" {k:40} {v}")
|
||||||
print(f"\n{line}\nECHO MEMORY — 0.6 vs 0.7 A/B EVAL")
|
print()
|
||||||
print(f"(mock OLRAPI; chars/token={a.cpt}, recovery-penalty={a.recovery}, detect-cost={a.detect_cost})\n{line}")
|
print(f"results -> {out}")
|
||||||
hdr = f"{'scenario':28} {'fault':32} {'gen06':>5} {'gen07':>5} {'silent06':>8} {'silent07':>8}"
|
return 0
|
||||||
print(hdr); print("-" * len(hdr))
|
|
||||||
for r in rows:
|
|
||||||
print(f"{r['scenario']:28} {r['fault']:32} "
|
|
||||||
f"{r['06']['emit_tokens']:>5} {r['07']['emit_tokens']:>5} "
|
|
||||||
f"{('YES' if r['06']['silent_failure'] else '-'):>8} "
|
|
||||||
f"{('YES' if r['07']['silent_failure'] else '-'):>8}")
|
|
||||||
print("-" * len(hdr))
|
|
||||||
|
|
||||||
def pct(old, new): return f"{(old-new)/old*100:+.0f}%" if old else "n/a"
|
|
||||||
print(f"\n{'metric':30} {'0.6':>10} {'0.7':>10} {'delta':>10}")
|
|
||||||
print("-" * 62)
|
|
||||||
for label, k in [("generated tokens", "gen_tokens"),
|
|
||||||
("silent failures", "silent_failures"),
|
|
||||||
("duplicate lines", "duplicates"),
|
|
||||||
("detected (loud) failures", "detected_failures"),
|
|
||||||
("effective tokens (incl. recovery)", "effective_tokens")]:
|
|
||||||
o, n = T06[k], T07[k]
|
|
||||||
d = pct(o, n) if "token" in k else f"{n-o:+d}"
|
|
||||||
print(f"{label:30} {o:>10} {n:>10} {d:>10}")
|
|
||||||
wrows = [r for r in rows if r['scenario'] != 'cold-start-load-6-reads']
|
|
||||||
denom = len(wrows)
|
|
||||||
sf06 = sum(1 for r in wrows if not r['06']['silent_failure']) # silent-error-free
|
|
||||||
sf07 = sum(1 for r in wrows if not r['07']['silent_failure'])
|
|
||||||
pr06 = sum(1 for r in wrows if r['06']['persisted']) # write actually landed
|
|
||||||
pr07 = sum(1 for r in wrows if r['07']['persisted'])
|
|
||||||
print("-" * 62)
|
|
||||||
print(f"{'silent-error-free ops':30} {str(sf06)+'/'+str(denom):>10} {str(sf07)+'/'+str(denom):>10}")
|
|
||||||
print(f"{'writes actually persisted':30} {str(pr06)+'/'+str(denom):>10} {str(pr07)+'/'+str(denom):>10}")
|
|
||||||
print(f" (the 2 un-persisted 0.7 ops are bad-heading + phantom: not persistable in a single")
|
|
||||||
print(f" call, but 0.7 fails LOUD so the agent's ensure-heading/retry path can recover.)")
|
|
||||||
print(f"\nNOTE: recovery-penalty and detect-cost are tunable ASSUMPTIONS, not measured.")
|
|
||||||
print(f" gen tokens are a chars/{a.cpt} proxy for output tokens of the I/O layer only.")
|
|
||||||
print(f" This harness measures mechanics, not model reasoning quality.\n")
|
|
||||||
|
|
||||||
report = {"params": vars(a), "rows": rows, "totals": {"0.6": T06, "0.7": T07},
|
|
||||||
"silent_error_free": {"0.6": f"{sf06}/{denom}", "0.7": f"{sf07}/{denom}"},
|
|
||||||
"writes_persisted": {"0.6": f"{pr06}/{denom}", "0.7": f"{pr07}/{denom}"}}
|
|
||||||
(HERE / "results" / "latest.json").write_text(json.dumps(report, indent=2))
|
|
||||||
print(f"wrote {HERE/'results'/'latest.json'}")
|
|
||||||
finally:
|
finally:
|
||||||
srv.terminate()
|
srv.terminate()
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
main()
|
raise SystemExit(main())
|
||||||
|
|||||||
Reference in New Issue
Block a user