echo/echo-memory.plugin.src/skills/echo-memory/eval/perf/PERF-TEST-PLAN.md

ECHO Performance Test Suite — Plan

Status: plan only — nothing here is to be run yet. Author: drafted for Jason Stedwell, 2026-06-23. Targets the changes that landed in: 1.1.0 (connection pooling + concurrent read_many + shared cache) and 1.2.0 (fuzzy resolve, alias derivation/learning, capture re-slug fix). Scope: a performance / timing harness. It complements — does not replace — the existing correctness tests (test_echo_client.py, test_v1_scaffold.py, the eval feature tests), which stay as the pass/fail gate for behaviour.

---

1. Goals

1. Put hard numbers on the three operations Jason called out — full endpoint read, full subject pulls, bulk get/put/append — so the 1.1.0 perf claims ("lint 0.90s / sweep 0.85s on 186 notes, ~4.5× from reuse, ~2× from concurrency") become a repeatable, checked-in measurement instead of a one-time observation.
2. Establish baselines + regression gates so a future change that quietly reintroduces the pre-1.1.0 per-request TLS handshake (the bug that was timing out sessions) fails the suite instead of silently shipping.
3. Keep every run non-destructive to the production vault — writes go to a disposable namespace and are cleaned up (per Jason's "prune test artifacts" rule).

2. Methodology (applies to every metric)

• Harness: a new bench.py under scripts/ (or eval/perf/), reusing the live echo client module so it measures the real pooled/concurrent code path, not a reimplementation.
• Timing: time.perf_counter() around each operation. Report min / median / p90 / p95 / max / mean over N iterations — not a single number. Single timings hide GC pauses and TLS renegotiation.
• Warm-up: discard the first 1–2 iterations (cold connection pool, cold server cache) and report them separately as the cold-start cost — that delta is itself one of the most interesting numbers (it's what 1.1.0's keep-alive was meant to kill).
• Iterations: default N=10 for read-heavy metrics, N=5 for write-heavy (writes are slower and we don't want to bloat the vault). Make N a CLI flag.
• Environment capture: record ECHO_WORKERS, ECHO_TIMEOUT, vault note count, git commit/plugin version, and wall-clock date in the results header so two runs are comparable.
• Output: machine-readable JSON (eval/perf/results/<date>-<commit>.json) plus a short human table to stdout. JSON lets us diff runs over time and chart trend.
• Isolation: all test writes live under a dedicated prefix, e.g. _agent/_bench/<run-id>/..., never in projects/, resources/, inbox/, or journal/. A --cleanup pass deletes the whole prefix at the end; a --keep flag preserves it for debugging. The advisory lock (echo.py lock) is taken for the write phases so a bench run can't race a real CoWork session.
• Network honesty: the endpoint is the live echoapi.alwisp.com, so absolute numbers depend on WAN latency. Report both absolute ms and a relative ratio against the same run's single-GET baseline, so the suite stays meaningful regardless of where it's run from.

---

3. Core metrics (the three Jason asked for)

3.1 Timed full endpoint read (full-vault bulk read)

What it measures: the headline 1.1.0 path — reading every note in the vault via concurrent read_many over warm pooled connections vs. serial GETs.

Procedure:

1. Enumerate all vault paths (the same listing sweep.py/vault_lint.py walk).
2. Serial baseline: GET each path one at a time on a single connection. Time total + per-note.
3. Pooled+concurrent: read_many(paths) at the default ECHO_WORKERS. Time total.
4. Report total time, notes/sec, and the speedup ratio (serial ÷ concurrent). Repeat the concurrent pass N times for percentile spread.

What "good" looks like: concurrent should be multiples faster than serial; full-vault concurrent read should stay well under the agent tool timeout (the original failure mode). Record current note count alongside — the metric is only comparable at similar vault sizes.

Regression gate (suggested): full-vault concurrent read median < a threshold (e.g. 3s at ~200 notes) AND speedup ratio ≥ ~2×. Tune thresholds after the first baseline run.

3.2 Timed full subject pulls (e.g. APTA, CapMetro statuses)

What it measures: the cost of a recall "<subject>" — search + 1-hop neighbourhood expansion along ## Related links and source_notes — which is the real "what do we know about X" operation, and exercises the 1.2.0 resolver. APTA and CapMetro are live subjects in the vault, so they're realistic fixtures.

Procedure:

1. Pick a fixed fixture set of subjects with different neighbourhood sizes: a small one, a hub note with many links (e.g. the echo project or operator-preferences), and the two named ones (APTA, CapMetro). Neighbourhood size is the real cost driver, so vary it deliberately.
2. Time end-to-end recall for each: (a) the search call, (b) the 1-hop expansion reads, (c) total. Break out the search vs. expansion split — it tells us whether latency is in the index/search or in the fan-out reads (which 1.1.0's read_many should accelerate).
3. Also time bare resolve "<subject>" (the O(1) index lookup) separately as the floor — recall should be resolve + search + N expansion reads.
4. Report per-subject and aggregate percentiles; annotate each with its neighbourhood node count so time-per-node is derivable.

What "good" looks like: resolve is near-instant (in-memory index); recall scales with neighbourhood size but the expansion reads are concurrent, so a hub note shouldn't be linearly worse than a leaf.

Live result + the expand-graph sub-metric (added 2026-06-23). The first run showed recall() at 2.8–4.7 s/subject. Profiling pinned it on the graph layer, not BM25: load_index() ~500 ms (index already persisted/incremental), score() 0.1 ms, expand_graph() ~3,000 ms doing one serial GET per neighbour. Fix shipped in echo_recall.py — the BFS now fetches each hop via read_many. A dedicated expand-graph metric times serial vs concurrent expansion (gate: ≥2× min) and asserts byte-identical ranking + scores against a serial reference, so the fix can't silently regress. Measured 3.5–4.2×; end-to-end recall() down ~1.7×.

3.3 Timed bulk get / put / append

What it measures: write-path throughput and the read-back/idempotency overhead Jason deliberately pays for.

Procedure (all in the _agent/_bench/<run-id>/ namespace):

• Bulk GET: create K fixture notes once, then time reading them back — both serially and via read_many — to compare against §3.1 at a controlled, fixed K (e.g. 25/50/100) so the read curve is isolated from whatever the live vault happens to contain.
• Bulk PUT: time creating K notes. echo.py put is read-back-verified (it re-GETs after writing), so this measures the true write+verify cost, not a fire-and-forget POST. Report puts/sec and per-put median.
• Bulk APPEND: time K appends to a single growing note. append is whole-line idempotent (it GETs first and skips duplicates), so also measure the idempotent-skip path: re-append the same K lines and confirm near-zero net writes — time the skip vs. the real append. This validates that idempotency isn't quietly O(file size) per append as the note grows.
• Concurrent write caveat: writes to single-line shared files assume one writer; the bulk-write bench must use per-note targets (or hold the lock) so it doesn't model an unsupported pattern.

What "good" looks like: PUT slower than GET (verify round-trip); append-skip much cheaper than append-write; no superlinear blowup as the appended note grows.

---

4. Additional test suggestions (recommended, beyond the three)

These target the rest of the 1.1.0/1.2.0 surface and the known historical failure modes.

1. Connection-pool cold vs. warm. Isolate the keep-alive win directly: time the first request after pool creation vs. the median of subsequent requests. This is the single clearest proof the pooling fix is alive; a regression here is the early-warning signal for the old per-request-handshake bug.

2. Concurrency scaling sweep (ECHO_WORKERS). Run §3.1 at workers = 1, 2, 4, 8, 16. Plot total time vs. workers to find the knee and confirm the default of 8 is sensible for the current vault/WAN, and that it degrades gracefully (no errors/timeouts) at high concurrency.

3. Fuzzy-resolve latency + accuracy (1.2.0). Two parts: (a) timing — resolve on exact, alias, and shortened/typo'd mentions; the fuzzy-candidate path does more work, so measure its overhead vs. an exact hit. (b) correctness — a fixture table of mention → expected canonical slug (e.g. "echo memory" → echo, alias hits, a near-miss that should return ranked candidates rather than spawn a duplicate). This guards the actual bug 1.2.0 fixed. Correctness assertions, not just timing.

4. Index build/rebuild time. Time sweep.py entity-index rebuild + link symmetrization on the full vault. This is the other full-vault script that historically neared the timeout; gate it like §3.1.

5. Lint + recall-rebuild full-vault timing. Regression-gate vault_lint.py and the recall rebuild against the 1.1.0 baselines (lint ~0.90s, sweep ~0.85s @ 186 notes). These are the concrete numbers in the changelog — turn them into asserted thresholds scaled to current note count.

6. Lock contention timing. Time lock acquisition when free, when held-fresh (should fast-fail with exit 75), and reclaim of a stale lock past ECHO_LOCK_TTL. Confirms the read-back-confirmed lock doesn't add pathological latency and behaves under contention.

7. Offline write-ahead queue throughput (1.0 carry-over). With the endpoint unreachable, time enqueue of K writes, then time flush replay on reconnect. Validates the queue doesn't degrade and replays in order. Useful because it's an untimed path today.

8. Cache hit/miss ratio. Instrument the shared single-pass cache: in a sweep/lint run, count served-from-cache vs. network reads. A correctness+efficiency check that the cache is actually collapsing duplicate reads, not just present.

9. Scaling fixture / synthetic vault. Optional but valuable: a script that mints a throwaway vault of N synthetic notes (100 / 500 / 1000) in the bench namespace to measure how all the above scale beyond the current ~190 notes — answers "when does the next timeout cliff arrive?" before it hits a real session.

10. Soak / stability pass. Run §3.1 in a loop for M minutes to surface connection leaks, pool exhaustion, or memory growth in the long-lived client — relevant because the pool is thread-local and long-running.

---

5. Proposed file layout

eval/perf/
  PERF-TEST-PLAN.md          # this file
  bench.py                   # the harness (CLI: --metric, --iterations, --workers, --cleanup/--keep, --json-out)
  fixtures.py                # subject lists, K-note generators, mention→slug resolve table
  baselines.json             # committed thresholds + reference numbers (per note-count bucket)
  results/                   # dated JSON run outputs (gitignored or kept for trend)
    2026-06-23-<commit>.json

bench.py subcommands map 1:1 to the metrics: read-full, subject-pull, bulk-get, bulk-put, bulk-append, plus pool-warmup, worker-sweep, resolve, index, lint, lock, queue, cache, soak.

6. Decisions (resolved 2026-06-23)

• Baseline policy: relative ratios are the gate (baselines.json); absolute ms are recorded as informational. Portable across machines/WAN.
• CI vs. manual: manual / pre-release. The suite is invoked by hand before tagging; it is not wired into CI (avoids WAN flakiness failing builds against the live endpoint).
• Synthetic vault (suggestion #9): deferred. Benchmark the real ~190-note vault for now; add the generator if vault growth makes it relevant.
• Cleanup: auto-delete the _agent/_bench/<run-id>/ namespace at end of each run, with --keep to override.

7. Build status

Harness built under eval/perf/:

• bench.py — the CLI harness; all 14 metrics implemented (read-full, subject-pull, bulk-get, bulk-put, bulk-append, pool-warmup, worker-sweep, resolve, index, lint, lock, queue, cache, soak).
• fixtures.py — subject list (APTA/CapMetro/hubs/leaf), the 1.2.0 resolve correctness table, and bench-note generators.
• baselines.json — relative-ratio gates (tune thresholds after the first real run).
• README.md — run instructions, metric map, isolation/cleanup, caveats.
• results/ — dated JSON run outputs.

Verified offline (compile + imports + --list/--help), then run live against the 197-note vault @ af16598 — 11/11 gate checks pass. First run surfaced the recall() finding below; the fix and a guarding metric were added the same session.

8. ECHO usage improvements (from the live run)

The benchmark wasn't just validation — it surfaced concrete improvements to how ECHO should do reads. In priority order:

1. Apply read_many everywhere a code path reads a set of notes (DONE for expand_graph). The 1.1.0 release shipped concurrent bulk reads but only sweep/lint/full-vault read adopted it. recall()'s graph expansion was still serial — one GET per neighbour — which is why it ran 3 s. The rule going forward: any loop that GETs more than ~3 notes whose paths are known up front should batch them through echo.read_many, not iterate get_text. Audit candidates: expand_graph (fixed), _brief result printing (still serial), rebuild()'s vault walk (still serial), and any future multi-note reader.

2. Prefetch _brief bodies in one read_many (NEXT). After expansion, recall prints each primary hit + neighbour via _brief(), which GETs the note again to pull its type/status. Those are known paths — fetch them all in one concurrent batch (and reuse bodies already pulled during expansion via a per-call cache, so a note fetched in the BFS isn't re-fetched to print it).

3. Reuse one in-process read cache across a recall call. load_index, expand_graph, and _brief currently re-fetch overlapping notes. A single {path: text} memo passed through the call (the same pattern sweep.py already uses via read_many once) removes the duplicate GETs — load_index's ~500 ms and the _brief re-reads are the remaining recall cost.

4. Make "batch the reads" a documented contract, not a per-site fix. Add a one-line rule to the plugin's operating contract / API reference: prefer read_many(paths) over a GET loop; serial multi-note reads are a performance bug. This is what would have caught the expand_graph regression at authoring time. The expand-graph metric now enforces it mechanically for recall; the contract generalizes it.

5. Consider indexing ## Related/source_notes adjacency into the persisted index. Longer-term: the graph edges are derivable and change only on write. Storing an adjacency list alongside the BM25 postings (maintained by capture/sweep, same as entities.json) would let expand_graph skip re-reading note bodies just to extract links — turning the graph layer into an in-memory lookup like resolve. Bigger change; revisit if recall latency still matters after #2–#3.

None of #1–#4 add dependencies or change behavior — they apply the concurrency primitive that already exists. #1 is done and gated; #2–#3 are the next cheap wins; #5 is the structural option.