# ECHO Performance Test Suite — Plan **Status:** plan only — nothing here is to be run yet. **Author:** drafted for Jason Stedwell, 2026-06-23. **Targets the changes that landed in:** 1.1.0 (connection pooling + concurrent `read_many` + shared cache) and 1.2.0 (fuzzy `resolve`, alias derivation/learning, `capture` re-slug fix). **Scope:** a *performance / timing* harness. It complements — does not replace — the existing correctness tests (`test_echo_client.py`, `test_v1_scaffold.py`, the eval feature tests), which stay as the pass/fail gate for behaviour. --- ## 1. Goals 1. Put hard numbers on the three operations Jason called out — full endpoint read, full subject pulls, bulk get/put/append — so the 1.1.0 perf claims ("lint 0.90s / sweep 0.85s on 186 notes, ~4.5× from reuse, ~2× from concurrency") become a repeatable, checked-in measurement instead of a one-time observation. 2. Establish **baselines + regression gates** so a future change that quietly reintroduces the pre-1.1.0 per-request TLS handshake (the bug that was timing out sessions) fails the suite instead of silently shipping. 3. Keep every run **non-destructive to the production vault** — writes go to a disposable namespace and are cleaned up (per Jason's "prune test artifacts" rule). ## 2. Methodology (applies to every metric) - **Harness:** a new `bench.py` under `scripts/` (or `eval/perf/`), reusing the live `echo` client module so it measures the real pooled/concurrent code path, not a reimplementation. - **Timing:** `time.perf_counter()` around each operation. Report **min / median / p90 / p95 / max / mean** over N iterations — not a single number. Single timings hide GC pauses and TLS renegotiation. - **Warm-up:** discard the first 1–2 iterations (cold connection pool, cold server cache) and report them *separately* as the cold-start cost — that delta is itself one of the most interesting numbers (it's what 1.1.0's keep-alive was meant to kill). - **Iterations:** default N=10 for read-heavy metrics, N=5 for write-heavy (writes are slower and we don't want to bloat the vault). Make N a CLI flag. - **Environment capture:** record `ECHO_WORKERS`, `ECHO_TIMEOUT`, vault note count, git commit/plugin version, and wall-clock date in the results header so two runs are comparable. - **Output:** machine-readable JSON (`eval/perf/results/-.json`) plus a short human table to stdout. JSON lets us diff runs over time and chart trend. - **Isolation:** all test writes live under a dedicated prefix, e.g. `_agent/_bench//...`, never in `projects/`, `resources/`, `inbox/`, or `journal/`. A `--cleanup` pass deletes the whole prefix at the end; a `--keep` flag preserves it for debugging. The advisory lock (`echo.py lock`) is taken for the write phases so a bench run can't race a real CoWork session. - **Network honesty:** the endpoint is the live `echoapi.alwisp.com`, so absolute numbers depend on WAN latency. Report **both** absolute ms and a relative ratio against the same run's single-GET baseline, so the suite stays meaningful regardless of where it's run from. --- ## 3. Core metrics (the three Jason asked for) ### 3.1 Timed full endpoint read (full-vault bulk read) **What it measures:** the headline 1.1.0 path — reading every note in the vault via concurrent `read_many` over warm pooled connections vs. serial GETs. **Procedure:** 1. Enumerate all vault paths (the same listing `sweep.py`/`vault_lint.py` walk). 2. **Serial baseline:** GET each path one at a time on a single connection. Time total + per-note. 3. **Pooled+concurrent:** `read_many(paths)` at the default `ECHO_WORKERS`. Time total. 4. Report total time, notes/sec, and the **speedup ratio** (serial ÷ concurrent). Repeat the concurrent pass N times for percentile spread. **What "good" looks like:** concurrent should be multiples faster than serial; full-vault concurrent read should stay well under the agent tool timeout (the original failure mode). Record current note count alongside — the metric is only comparable at similar vault sizes. **Regression gate (suggested):** full-vault concurrent read median < a threshold (e.g. 3s at ~200 notes) AND speedup ratio ≥ ~2×. Tune thresholds after the first baseline run. ### 3.2 Timed full subject pulls (e.g. APTA, CapMetro statuses) **What it measures:** the cost of a `recall ""` — search + 1-hop neighbourhood expansion along `## Related` links and `source_notes` — which is the real "what do we know about X" operation, and exercises the 1.2.0 resolver. APTA and CapMetro are live subjects in the vault, so they're realistic fixtures. **Procedure:** 1. Pick a fixed fixture set of subjects with **different neighbourhood sizes**: a small one, a hub note with many links (e.g. the `echo` project or `operator-preferences`), and the two named ones (APTA, CapMetro). Neighbourhood size is the real cost driver, so vary it deliberately. 2. Time end-to-end `recall` for each: (a) the search call, (b) the 1-hop expansion reads, (c) total. Break out the search vs. expansion split — it tells us whether latency is in the index/search or in the fan-out reads (which 1.1.0's `read_many` should accelerate). 3. Also time bare `resolve ""` (the O(1) index lookup) separately as the floor — recall should be resolve + search + N expansion reads. 4. Report per-subject and aggregate percentiles; annotate each with its neighbourhood node count so time-per-node is derivable. **What "good" looks like:** `resolve` is near-instant (in-memory index); `recall` scales with neighbourhood size but the expansion reads are concurrent, so a hub note shouldn't be linearly worse than a leaf. **Live result + the `expand-graph` sub-metric (added 2026-06-23).** The first run showed `recall()` at 2.8–4.7 s/subject. Profiling pinned it on the graph layer, not BM25: `load_index()` ~500 ms (index already persisted/incremental), `score()` 0.1 ms, `expand_graph()` ~3,000 ms doing one serial GET per neighbour. Fix shipped in `echo_recall.py` — the BFS now fetches each hop via `read_many`. A dedicated **`expand-graph`** metric times serial vs concurrent expansion (gate: ≥2× min) and asserts byte-identical ranking + scores against a serial reference, so the fix can't silently regress. Measured 3.5–4.2×; end-to-end `recall()` down ~1.7×. ### 3.3 Timed bulk get / put / append **What it measures:** write-path throughput and the read-back/idempotency overhead Jason deliberately pays for. **Procedure (all in the `_agent/_bench//` namespace):** - **Bulk GET:** create K fixture notes once, then time reading them back — both serially and via `read_many` — to compare against §3.1 at a controlled, fixed K (e.g. 25/50/100) so the read curve is isolated from whatever the live vault happens to contain. - **Bulk PUT:** time creating K notes. `echo.py put` is read-back-verified (it re-GETs after writing), so this measures the true write+verify cost, not a fire-and-forget POST. Report puts/sec and per-put median. - **Bulk APPEND:** time K appends to a single growing note. `append` is whole-line idempotent (it GETs first and skips duplicates), so also measure the **idempotent-skip path**: re-append the same K lines and confirm near-zero net writes — time the skip vs. the real append. This validates that idempotency isn't quietly O(file size) per append as the note grows. - **Concurrent write caveat:** writes to single-line shared files assume one writer; the bulk-write bench must use per-note targets (or hold the lock) so it doesn't model an unsupported pattern. **What "good" looks like:** PUT slower than GET (verify round-trip); append-skip much cheaper than append-write; no superlinear blowup as the appended note grows. --- ## 4. Additional test suggestions (recommended, beyond the three) These target the rest of the 1.1.0/1.2.0 surface and the known historical failure modes. 1. **Connection-pool cold vs. warm.** Isolate the keep-alive win directly: time the *first* request after pool creation vs. the median of subsequent requests. This is the single clearest proof the pooling fix is alive; a regression here is the early-warning signal for the old per-request-handshake bug. 2. **Concurrency scaling sweep (`ECHO_WORKERS`).** Run §3.1 at workers = 1, 2, 4, 8, 16. Plot total time vs. workers to find the knee and confirm the default of 8 is sensible for the current vault/WAN, and that it degrades gracefully (no errors/timeouts) at high concurrency. 3. **Fuzzy-resolve latency + accuracy (1.2.0).** Two parts: (a) *timing* — `resolve` on exact, alias, and shortened/typo'd mentions; the fuzzy-candidate path does more work, so measure its overhead vs. an exact hit. (b) *correctness* — a fixture table of mention → expected canonical slug (e.g. "echo memory" → `echo`, alias hits, a near-miss that *should* return ranked candidates rather than spawn a duplicate). This guards the actual bug 1.2.0 fixed. Correctness assertions, not just timing. 4. **Index build/rebuild time.** Time `sweep.py` entity-index rebuild + link symmetrization on the full vault. This is the other full-vault script that historically neared the timeout; gate it like §3.1. 5. **Lint + recall-rebuild full-vault timing.** Regression-gate `vault_lint.py` and the recall rebuild against the 1.1.0 baselines (lint ~0.90s, sweep ~0.85s @ 186 notes). These are the concrete numbers in the changelog — turn them into asserted thresholds scaled to current note count. 6. **Lock contention timing.** Time `lock` acquisition when free, when held-fresh (should fast-fail with exit 75), and reclaim of a stale lock past `ECHO_LOCK_TTL`. Confirms the read-back-confirmed lock doesn't add pathological latency and behaves under contention. 7. **Offline write-ahead queue throughput (1.0 carry-over).** With the endpoint unreachable, time enqueue of K writes, then time `flush` replay on reconnect. Validates the queue doesn't degrade and replays in order. Useful because it's an untimed path today. 8. **Cache hit/miss ratio.** Instrument the shared single-pass cache: in a sweep/lint run, count served-from-cache vs. network reads. A correctness+efficiency check that the cache is actually collapsing duplicate reads, not just present. 9. **Scaling fixture / synthetic vault.** Optional but valuable: a script that mints a throwaway vault of N synthetic notes (100 / 500 / 1000) in the bench namespace to measure how all the above scale *beyond* the current ~190 notes — answers "when does the next timeout cliff arrive?" before it hits a real session. 10. **Soak / stability pass.** Run §3.1 in a loop for M minutes to surface connection leaks, pool exhaustion, or memory growth in the long-lived client — relevant because the pool is thread-local and long-running. --- ## 5. Proposed file layout ``` eval/perf/ PERF-TEST-PLAN.md # this file bench.py # the harness (CLI: --metric, --iterations, --workers, --cleanup/--keep, --json-out) fixtures.py # subject lists, K-note generators, mention→slug resolve table baselines.json # committed thresholds + reference numbers (per note-count bucket) results/ # dated JSON run outputs (gitignored or kept for trend) 2026-06-23-.json ``` `bench.py` subcommands map 1:1 to the metrics: `read-full`, `subject-pull`, `bulk-get`, `bulk-put`, `bulk-append`, plus `pool-warmup`, `worker-sweep`, `resolve`, `index`, `lint`, `lock`, `queue`, `cache`, `soak`. ## 6. Decisions (resolved 2026-06-23) - **Baseline policy:** **relative ratios** are the gate (`baselines.json`); absolute ms are recorded as informational. Portable across machines/WAN. - **CI vs. manual:** **manual / pre-release.** The suite is invoked by hand before tagging; it is not wired into CI (avoids WAN flakiness failing builds against the live endpoint). - **Synthetic vault (suggestion #9):** **deferred.** Benchmark the real ~190-note vault for now; add the generator if vault growth makes it relevant. - **Cleanup:** **auto-delete** the `_agent/_bench//` namespace at end of each run, with `--keep` to override. ## 7. Build status Harness built under `eval/perf/`: - `bench.py` — the CLI harness; all 14 metrics implemented (`read-full`, `subject-pull`, `bulk-get`, `bulk-put`, `bulk-append`, `pool-warmup`, `worker-sweep`, `resolve`, `index`, `lint`, `lock`, `queue`, `cache`, `soak`). - `fixtures.py` — subject list (APTA/CapMetro/hubs/leaf), the 1.2.0 resolve correctness table, and bench-note generators. - `baselines.json` — relative-ratio gates (tune thresholds after the first real run). - `README.md` — run instructions, metric map, isolation/cleanup, caveats. - `results/` — dated JSON run outputs. Verified offline (compile + imports + `--list`/`--help`), then **run live** against the 197-note vault @ af16598 — 11/11 gate checks pass. First run surfaced the recall() finding below; the fix and a guarding metric were added the same session. ## 8. ECHO usage improvements (from the live run) The benchmark wasn't just validation — it surfaced concrete improvements to how ECHO should do reads. In priority order: 1. **Apply `read_many` everywhere a code path reads a *set* of notes (DONE for `expand_graph`).** The 1.1.0 release shipped concurrent bulk reads but only `sweep`/`lint`/full-vault read adopted it. `recall()`'s graph expansion was still serial — one GET per neighbour — which is why it ran 3 s. The rule going forward: *any* loop that GETs more than ~3 notes whose paths are known up front should batch them through `echo.read_many`, not iterate `get_text`. Audit candidates: `expand_graph` (fixed), `_brief` result printing (still serial), `rebuild()`'s vault walk (still serial), and any future multi-note reader. 2. **Prefetch `_brief` bodies in one `read_many` (NEXT).** After expansion, `recall` prints each primary hit + neighbour via `_brief()`, which GETs the note again to pull its type/status. Those are known paths — fetch them all in one concurrent batch (and reuse bodies already pulled during expansion via a per-call cache, so a note fetched in the BFS isn't re-fetched to print it). 3. **Reuse one in-process read cache across a `recall` call.** `load_index`, `expand_graph`, and `_brief` currently re-fetch overlapping notes. A single `{path: text}` memo passed through the call (the same pattern `sweep.py` already uses via `read_many` once) removes the duplicate GETs — `load_index`'s ~500 ms and the `_brief` re-reads are the remaining recall cost. 4. **Make "batch the reads" a documented contract, not a per-site fix.** Add a one-line rule to the plugin's operating contract / API reference: *prefer `read_many(paths)` over a GET loop; serial multi-note reads are a performance bug.* This is what would have caught the `expand_graph` regression at authoring time. The `expand-graph` metric now enforces it mechanically for recall; the contract generalizes it. 5. **Consider indexing `## Related`/`source_notes` adjacency into the persisted index.** Longer-term: the graph edges are derivable and change only on write. Storing an adjacency list alongside the BM25 postings (maintained by `capture`/`sweep`, same as `entities.json`) would let `expand_graph` skip re-reading note bodies just to extract links — turning the graph layer into an in-memory lookup like `resolve`. Bigger change; revisit if recall latency still matters after #2–#3. None of #1–#4 add dependencies or change behavior — they apply the concurrency primitive that already exists. #1 is done and gated; #2–#3 are the next cheap wins; #5 is the structural option.