ECHO Performance Test Suite

A timing harness for the operations the 1.1.0 (connection pooling + concurrent read_many + shared cache) and 1.2.0 (fuzzy resolve / alias learning) updates targeted. It reuses the live echo client module, so it measures the real pooled / concurrent code path — not a reimplementation.

This is a manual / pre-release tool, not a CI gate: it depends on the live configured endpoint (echo.BASE / $ECHO_BASE) and WAN latency. Gates are relative ratios (portable), with absolute milliseconds kept as informational context.

See PERF-TEST-PLAN.md for the design rationale behind each metric.

Run

cd eval/perf
python3 bench.py --list                 # list metrics
python3 bench.py all                     # the read-only safe set (no vault writes)
python3 bench.py read-full -n 10         # full-vault serial-vs-concurrent read
python3 bench.py subject-pull            # recall APTA / CapMetro / hubs (resolve+search+recall split)
python3 bench.py bulk-put -k 50          # write metric -> uses the _bench namespace
python3 bench.py worker-sweep            # ECHO_WORKERS = 1,2,4,8,16 scaling curve
python3 bench.py resolve                 # 1.2.0 fuzzy correctness table + timing
python3 bench.py soak --soak-seconds 120 # stability / leak watch

Configure this machine first (owner/endpoint/key): python3 ../../scripts/echo.py config init then edit the file, or export ECHO_BASE=... ECHO_KEY=....

Metrics

Metric	Plan §	Reads/Writes	What it proves
`read-full`	3.1	read-only	full-vault concurrent vs serial speedup (1.1.0 headline)
`subject-pull`	3.2	read-only	`recall` cost for APTA/CapMetro/hubs, split into resolve floor + search + expansion
`expand-graph`	3.2a	read-only	recall()'s graph layer: serial vs concurrent neighbourhood expansion + ranking-parity guard (pins the real recall bottleneck)
`bulk-get`	3.3	writes seed	concurrent vs serial read at a controlled K
`bulk-put`	3.3	writes	read-back-verified PUT throughput (puts/sec)
`bulk-append`	3.3	writes	append throughput + idempotent-skip path (no O(file) blowup)
`pool-warmup`	4.1	read-only	cold vs warm single GET — keep-alive is alive
`worker-sweep`	4.2	read-only	concurrency scaling knee; validates default workers=8
`resolve`	4.3	read-only	fuzzy-resolve timing and correctness (anti-duplicate guard)
`index`	4.4	read-only	`sweep.py` index rebuild + link symmetrize time
`lint`	4.5	read-only	`vault_lint.py` full-vault time vs 1.1.0 baseline
`lock`	4.6	writes lock	advisory-lock acquire/contend/release timing + semantics
`queue`	4.7	none (bogus base)	offline write-ahead enqueue timing (see caveat)
`cache`	4.8	read-only	`read_many` dedups a duplicated path list
`soak`	4.10	read-only	stability over a window; leak/drift detector

all runs only the read-only set. Write metrics must be requested by name.

Isolation & cleanup

Write metrics touch only _agent/_bench/<run-id>/, take the advisory lock for the duration (and abort if another session holds it), and delete the namespace at the end. Pass --keep to preserve it for debugging. Nothing is written to projects/, resources/, inbox/, or journal/.

Output

Each run writes results/<date>-<commit>.json (machine-readable, for trend diffing) and prints a human table. The JSON header captures ECHO_WORKERS, ECHO_TIMEOUT, note count, git commit, and date so two runs are comparable.

Gating

baselines.json holds relative-ratio gates (e.g. read-full.speedup_ratio >= 1.8, pool-warmup.cold_over_warm_ratio >= 1.5). Run once to capture a baseline, then tune the thresholds to the observed numbers before treating a FAIL as blocking. --no-gate reports numbers without pass/fail.

Finding: recall() bottleneck is the graph layer, not BM25

A profiled recall("operator preferences") decomposes as: load_index() ~500 ms (the BM25 index is already persisted + incrementally maintained, n_docs=119), score() 0.1 ms, and expand_graph() ~3,000 ms — the graph BFS was fetching each neighbour serially (one GET per node, ~126 nodes). The fix (shipped in echo_recall.py) makes the BFS breadth-first by hop and fetches each frontier through the existing read_many concurrency. Result: expand_graph 3.2–4.2x faster with byte-identical ranking and scores; end-to-end recall() ~1.7x (e.g. APTA 3.8 s -> 2.2 s). The expand-graph metric + gate above regression-guard this so it can't quietly revert to serial.

Caveats

resolve verdicts are PASS/INFO only. The correctness table in fixtures.py encodes expected slugs against the live vault; until confirmed, mismatches report INFO (tune-me), not FAIL. Promote to FAIL once the table is verified.
queue times the enqueue path only. A true flush-replay needs a throwaway sandbox vault so the replay doesn't write to production; that's deferred (the plan's synthetic-vault item). The metric still proves enqueue works and reports queue depth.
Absolute ms depend on where you run from. WAN latency to the configured endpoint dominates single-request times; that's why gates are ratios.
Synthetic-vault scaling (100/500/1000 notes) is deferred per decision — the suite benchmarks the real ~190-note vault today.

5.3 KiB Raw Blame History Unescape Escape