chorus/eval/README.md

# echo-memory eval — 0.6 vs 0.7 A/B harness

A reproducible, credential-free A/B comparison of the plugin **before** (0.6: raw-curl
recipes from `SKILL.md`) and **after** (the shipped `scripts/echo.py` client) the
hardening work. It quantifies the claims in the comparative analysis: token cost of the
I/O layer, and the rate of **silent write failures**.

## Current test suites (run these for any change)

The A/B harness below is the historical 0.6-vs-0.7 comparison. The **current-version**
suites live alongside it and are what CI gates on:

```bash
python3 test_features.py         # end-to-end features vs the mock (capture/recall/gate/triage/hooks/…)
python3 test_reflect.py          # reflection pipeline
python3 test_offline_queue.py    # H2 outage queue + read cache
python3 test_patch_semantics.py  # real PATCH semantics vs mock_olrapi_hifi.py (incl. fm create-or-replace)
# plus, in the plugin tree: scripts/test_echo_client.py (offline unit + routing-sync guard)
```

## Run it (historical A/B)

```bash
cd eval
python3 run_eval.py                 # default params
python3 run_eval.py --recovery 2500 # sensitivity-test the recovery assumption
python3 run_eval.py --cpt 3.5       # different chars/token proxy
```

No network, no API key, no live vault. Pure stdlib Python 3 (the shipped client is now Python — no bash required).
Results table prints to stdout and a machine-readable copy lands in `results/latest.json`.

## How it works

- **`mock_olrapi.py`** — a deterministic mock of the Obsidian Local REST API surface the
  plugin uses, reproducing its real behaviors and quirks (404 shape, the `/vault//`
  double-slash 400, directory listings with `dir/` entries, `PATCH` heading targets that
  return `400 invalid-target / 40080` when the heading is absent). Faults are triggered by
  path markers so one server serves every scenario:
  - `flaky` in the path → first write returns `503`, then succeeds (tests retry).
  - `phantom` in the path → `PUT` returns `200` but does **not** persist (tests read-back verify).
  - a `PATCH` to a missing heading → `400` (the silent-write-loss trigger).
- **`run_eval.py`** — for each scenario, runs both methods against a freshly reset +
  re-seeded server (so faults are identical for both), then reads ground truth back
  **independently** from the mock. The 0.7 side executes the *actual shipped `echo.py`*;
  the 0.6 side faithfully models the documented recipe (real HTTP, but no status check,
  no retry, no verify, no dedupe).

## Metrics

| metric | meaning |
|---|---|
| `gen_tokens` | output tokens the model must generate for the op (`len(emitted)/cpt` proxy) |
| `silent_failure` | method **reported success** but ground truth is wrong (lost write or dup) — and nobody noticed |
| `detected` | the method surfaced the failure (nonzero exit) instead of hiding it |
| `effective_tokens` | `gen + silent_failures*recovery + detected*detect_cost` |
| `silent-error-free ops` | the headline accuracy number |
| `writes actually persisted` | did the single op land (separate from "was it silent") |

## Scenarios

1. **agent-log-missing-heading** — `PATCH` append to a note lacking the target heading (`400`).
2. **scope-switch** — clean `PATCH replace` (no fault; pure token comparison).
3. **inbox-capture-replayed** — same capture issued twice (retry/replay): dedup vs duplicate.
4. **session-log-flaky-network** — one-time `503`: retry vs single-shot.
5. **heartbeat-phantom-write** — accepted-but-not-persisted: read-back verify vs none.
6. **cold-start-load-6-reads** — 6 GETs (no fault; pure token comparison).

## Representative result (defaults)

```
generated tokens             723 -> 174   (+76% fewer)
silent failures                4 ->   0   (-4)
duplicate lines                1 ->   0   (-1)
silent-error-free ops        1/5 -> 5/5
effective tokens (assumed)  6723 -> 334
```

## Honest caveats

- **Mechanics, not reasoning.** This measures the deterministic plumbing differences. It
  does **not** measure model judgment (routing choices, prose quality) — that needs a live model.
- **`recovery` and `detect-cost` are assumptions**, not measurements. The headline
  "silent failures: 4 → 0" is a hard count from ground truth; the `effective_tokens` figure
  is a model on top of it — tune `--recovery` to see the sensitivity.
- **`gen_tokens` is a `chars/cpt` proxy** for the I/O layer only, not a tokenizer count, and
  excludes the one-time `+12%` SKILL.md context cost noted in the analysis (that's a
  per-session context cost, not per-op).

## Extending to a live-model run (optional)

To measure real model behavior and true token counts:
1. Define the same six scenarios as natural-language tasks (e.g. "log a session note for X").
2. Run each twice — once with the 0.6 skill files, once with 0.7 — through the Agent SDK
   against the **mock** server (point `ECHO_BASE` at it) so faults stay deterministic.
3. Record `usage.output_tokens` per task from the API and whether the vault ended correct
   (same ground-truth read used here).

The mock + ground-truth checks in this harness are reusable as-is for that; only the driver
changes from "scripted ops" to "model-driven ops".