chorus/echo-memory.plugin.src/skills/echo-memory/eval/perf/README.md

# ECHO Performance Test Suite

A timing harness for the operations the **1.1.0** (connection pooling + concurrent
`read_many` + shared cache) and **1.2.0** (fuzzy `resolve` / alias learning) updates
targeted. It reuses the live `echo` client module, so it measures the real pooled /
concurrent code path — not a reimplementation.

This is a **manual / pre-release** tool, not a CI gate: it depends on the live
configured endpoint (`echo.BASE` / `$ECHO_BASE`) and WAN latency. Gates are **relative ratios**
(portable), with absolute milliseconds kept as informational context.

See `PERF-TEST-PLAN.md` for the design rationale behind each metric.

## Run

```bash
cd eval/perf
python3 bench.py --list                 # list metrics
python3 bench.py all                     # the read-only safe set (no vault writes)
python3 bench.py read-full -n 10         # full-vault serial-vs-concurrent read
python3 bench.py subject-pull            # recall APTA / CapMetro / hubs (resolve+search+recall split)
python3 bench.py bulk-put -k 50          # write metric -> uses the _bench namespace
python3 bench.py worker-sweep            # ECHO_WORKERS = 1,2,4,8,16 scaling curve
python3 bench.py resolve                 # 1.2.0 fuzzy correctness table + timing
python3 bench.py soak --soak-seconds 120 # stability / leak watch
```

Configure this machine first (owner/endpoint/key):
`python3 ../../scripts/echo.py config init` then edit the file, or `export ECHO_BASE=... ECHO_KEY=...`.

## Metrics

| Metric | Plan § | Reads/Writes | What it proves |
|---|---|---|---|
| `read-full` | 3.1 | read-only | full-vault concurrent vs serial speedup (1.1.0 headline) |
| `subject-pull` | 3.2 | read-only | `recall` cost for APTA/CapMetro/hubs, split into resolve floor + search + expansion |
| `expand-graph` | 3.2a | read-only | recall()'s graph layer: serial vs concurrent neighbourhood expansion + ranking-parity guard (pins the real recall bottleneck) |
| `bulk-get` | 3.3 | writes seed | concurrent vs serial read at a controlled K |
| `bulk-put` | 3.3 | **writes** | read-back-verified PUT throughput (puts/sec) |
| `bulk-append` | 3.3 | **writes** | append throughput + idempotent-skip path (no O(file) blowup) |
| `pool-warmup` | 4.1 | read-only | cold vs warm single GET — keep-alive is alive |
| `worker-sweep` | 4.2 | read-only | concurrency scaling knee; validates default workers=8 |
| `resolve` | 4.3 | read-only | fuzzy-resolve timing **and** correctness (anti-duplicate guard) |
| `index` | 4.4 | read-only | `sweep.py` index rebuild + link symmetrize time |
| `lint` | 4.5 | read-only | `vault_lint.py` full-vault time vs 1.1.0 baseline |
| `lock` | 4.6 | **writes** lock | advisory-lock acquire/contend/release timing + semantics |
| `queue` | 4.7 | none (bogus base) | offline write-ahead enqueue timing (see caveat) |
| `cache` | 4.8 | read-only | `read_many` dedups a duplicated path list |
| `soak` | 4.10 | read-only | stability over a window; leak/drift detector |

`all` runs only the read-only set. Write metrics must be requested by name.

## Isolation & cleanup

Write metrics touch **only** `_agent/_bench/<run-id>/`, take the advisory lock for
the duration (and abort if another session holds it), and delete the namespace at
the end. Pass `--keep` to preserve it for debugging. Nothing is written to
`projects/`, `resources/`, `inbox/`, or `journal/`.

## Output

Each run writes `results/<date>-<commit>.json` (machine-readable, for trend diffing)
and prints a human table. The JSON header captures `ECHO_WORKERS`, `ECHO_TIMEOUT`,
note count, git commit, and date so two runs are comparable.

## Gating

`baselines.json` holds relative-ratio gates (e.g. `read-full.speedup_ratio >= 1.8`,
`pool-warmup.cold_over_warm_ratio >= 1.5`). Run once to capture a baseline, then
tune the thresholds to the observed numbers before treating a `FAIL` as blocking.
`--no-gate` reports numbers without pass/fail.

## Finding: recall() bottleneck is the graph layer, not BM25

A profiled `recall("operator preferences")` decomposes as: `load_index()` ~500 ms (the
BM25 index is already persisted + incrementally maintained, n_docs=119), `score()`
0.1 ms, and `expand_graph()` **~3,000 ms** — the graph BFS was fetching each neighbour
serially (one GET per node, ~126 nodes). The fix (shipped in `echo_recall.py`) makes the
BFS breadth-first by hop and fetches each frontier through the existing `read_many`
concurrency. Result: `expand_graph` 3.2–4.2x faster with byte-identical ranking and
scores; end-to-end `recall()` ~1.7x (e.g. APTA 3.8 s -> 2.2 s). The `expand-graph`
metric + gate above regression-guard this so it can't quietly revert to serial.

## Caveats

- **`resolve` verdicts are PASS/INFO only.** The correctness table in `fixtures.py`
  encodes expected slugs against the live vault; until confirmed, mismatches report
  `INFO` (tune-me), not `FAIL`. Promote to FAIL once the table is verified.
- **`queue` times the enqueue path only.** A true flush-replay needs a throwaway
  sandbox vault so the replay doesn't write to production; that's deferred (the
  plan's synthetic-vault item). The metric still proves enqueue works and reports
  queue depth.
- **Absolute ms depend on where you run from.** WAN latency to the configured
  endpoint dominates single-request times; that's why gates are ratios.
- **Synthetic-vault scaling (100/500/1000 notes) is deferred** per decision — the
  suite benchmarks the real ~190-note vault today.