chorus/eval/README.md

echo-memory eval — 0.6 vs 0.7 A/B harness

A reproducible, credential-free A/B comparison of the plugin before (0.6: raw-curl recipes from SKILL.md) and after (the shipped scripts/echo.py client) the hardening work. It quantifies the claims in the comparative analysis: token cost of the I/O layer, and the rate of silent write failures.

Current test suites (run these for any change)

The A/B harness below is the historical 0.6-vs-0.7 comparison. The current-version suites live alongside it and are what CI gates on:

python3 test_features.py         # end-to-end features vs the mock (capture/recall/gate/triage/hooks/…)
python3 test_reflect.py          # reflection pipeline
python3 test_offline_queue.py    # H2 outage queue + read cache
python3 test_patch_semantics.py  # real PATCH semantics vs mock_olrapi_hifi.py (incl. fm create-or-replace)
# plus, in the plugin tree: scripts/test_echo_client.py (offline unit + routing-sync guard)

Run it (historical A/B)

cd eval
python3 run_eval.py                 # default params
python3 run_eval.py --recovery 2500 # sensitivity-test the recovery assumption
python3 run_eval.py --cpt 3.5       # different chars/token proxy

No network, no API key, no live vault. Pure stdlib Python 3 (the shipped client is now Python — no bash required). Results table prints to stdout and a machine-readable copy lands in results/latest.json.

How it works

• mock_olrapi.py — a deterministic mock of the Obsidian Local REST API surface the plugin uses, reproducing its real behaviors and quirks (404 shape, the /vault// double-slash 400, directory listings with dir/ entries, PATCH heading targets that return 400 invalid-target / 40080 when the heading is absent). Faults are triggered by path markers so one server serves every scenario:
  • flaky in the path → first write returns 503, then succeeds (tests retry).
  • phantom in the path → PUT returns 200 but does not persist (tests read-back verify).
  • a PATCH to a missing heading → 400 (the silent-write-loss trigger).
• run_eval.py — for each scenario, runs both methods against a freshly reset + re-seeded server (so faults are identical for both), then reads ground truth back independently from the mock. The 0.7 side executes the actual shipped echo.py; the 0.6 side faithfully models the documented recipe (real HTTP, but no status check, no retry, no verify, no dedupe).

Metrics

metric	meaning
gen_tokens	output tokens the model must generate for the op (len(emitted)/cpt proxy)
silent_failure	method reported success but ground truth is wrong (lost write or dup) — and nobody noticed
detected	the method surfaced the failure (nonzero exit) instead of hiding it
effective_tokens	gen + silent_failures*recovery + detected*detect_cost
silent-error-free ops	the headline accuracy number
writes actually persisted	did the single op land (separate from "was it silent")

Scenarios

1. agent-log-missing-heading — PATCH append to a note lacking the target heading (400).
2. scope-switch — clean PATCH replace (no fault; pure token comparison).
3. inbox-capture-replayed — same capture issued twice (retry/replay): dedup vs duplicate.
4. session-log-flaky-network — one-time 503: retry vs single-shot.
5. heartbeat-phantom-write — accepted-but-not-persisted: read-back verify vs none.
6. cold-start-load-6-reads — 6 GETs (no fault; pure token comparison).

Representative result (defaults)

generated tokens             723 -> 174   (+76% fewer)
silent failures                4 ->   0   (-4)
duplicate lines                1 ->   0   (-1)
silent-error-free ops        1/5 -> 5/5
effective tokens (assumed)  6723 -> 334

Honest caveats

• Mechanics, not reasoning. This measures the deterministic plumbing differences. It does not measure model judgment (routing choices, prose quality) — that needs a live model.
• recovery and detect-cost are assumptions, not measurements. The headline "silent failures: 4 → 0" is a hard count from ground truth; the effective_tokens figure is a model on top of it — tune --recovery to see the sensitivity.
• gen_tokens is a chars/cpt proxy for the I/O layer only, not a tokenizer count, and excludes the one-time +12% SKILL.md context cost noted in the analysis (that's a per-session context cost, not per-op).

Extending to a live-model run (optional)

To measure real model behavior and true token counts:

1. Define the same six scenarios as natural-language tasks (e.g. "log a session note for X").
2. Run each twice — once with the 0.6 skill files, once with 0.7 — through the Agent SDK against the mock server (point ECHO_BASE at it) so faults stay deterministic.
3. Record usage.output_tokens per task from the API and whether the vault ended correct (same ground-truth read used here).

The mock + ground-truth checks in this harness are reusable as-is for that; only the driver changes from "scripted ops" to "model-driven ops".