diff --git a/.DS_Store b/.DS_Store index c71fe80..44f63ba 100644 Binary files a/.DS_Store and b/.DS_Store differ diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/ECHO-Performance-Benchmark.pdf b/echo-memory.plugin.src/skills/echo-memory/eval/perf/ECHO-Performance-Benchmark.pdf new file mode 100644 index 0000000..12dd760 Binary files /dev/null and b/echo-memory.plugin.src/skills/echo-memory/eval/perf/ECHO-Performance-Benchmark.pdf differ diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/PERF-TEST-PLAN.md b/echo-memory.plugin.src/skills/echo-memory/eval/perf/PERF-TEST-PLAN.md new file mode 100644 index 0000000..2f0acec --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/PERF-TEST-PLAN.md @@ -0,0 +1,146 @@ +# ECHO Performance Test Suite — Plan + +**Status:** plan only — nothing here is to be run yet. +**Author:** drafted for Jason Stedwell, 2026-06-23. +**Targets the changes that landed in:** 1.1.0 (connection pooling + concurrent `read_many` + shared cache) and 1.2.0 (fuzzy `resolve`, alias derivation/learning, `capture` re-slug fix). +**Scope:** a *performance / timing* harness. It complements — does not replace — the existing correctness tests (`test_echo_client.py`, `test_v1_scaffold.py`, the eval feature tests), which stay as the pass/fail gate for behaviour. + +--- + +## 1. Goals + +1. Put hard numbers on the three operations Jason called out — full endpoint read, full subject pulls, bulk get/put/append — so the 1.1.0 perf claims ("lint 0.90s / sweep 0.85s on 186 notes, ~4.5× from reuse, ~2× from concurrency") become a repeatable, checked-in measurement instead of a one-time observation. +2. Establish **baselines + regression gates** so a future change that quietly reintroduces the pre-1.1.0 per-request TLS handshake (the bug that was timing out sessions) fails the suite instead of silently shipping. +3. Keep every run **non-destructive to the production vault** — writes go to a disposable namespace and are cleaned up (per Jason's "prune test artifacts" rule). + +## 2. Methodology (applies to every metric) + +- **Harness:** a new `bench.py` under `scripts/` (or `eval/perf/`), reusing the live `echo` client module so it measures the real pooled/concurrent code path, not a reimplementation. +- **Timing:** `time.perf_counter()` around each operation. Report **min / median / p90 / p95 / max / mean** over N iterations — not a single number. Single timings hide GC pauses and TLS renegotiation. +- **Warm-up:** discard the first 1–2 iterations (cold connection pool, cold server cache) and report them *separately* as the cold-start cost — that delta is itself one of the most interesting numbers (it's what 1.1.0's keep-alive was meant to kill). +- **Iterations:** default N=10 for read-heavy metrics, N=5 for write-heavy (writes are slower and we don't want to bloat the vault). Make N a CLI flag. +- **Environment capture:** record `ECHO_WORKERS`, `ECHO_TIMEOUT`, vault note count, git commit/plugin version, and wall-clock date in the results header so two runs are comparable. +- **Output:** machine-readable JSON (`eval/perf/results/-.json`) plus a short human table to stdout. JSON lets us diff runs over time and chart trend. +- **Isolation:** all test writes live under a dedicated prefix, e.g. `_agent/_bench//...`, never in `projects/`, `resources/`, `inbox/`, or `journal/`. A `--cleanup` pass deletes the whole prefix at the end; a `--keep` flag preserves it for debugging. The advisory lock (`echo.py lock`) is taken for the write phases so a bench run can't race a real CoWork session. +- **Network honesty:** the endpoint is the live `echoapi.alwisp.com`, so absolute numbers depend on WAN latency. Report **both** absolute ms and a relative ratio against the same run's single-GET baseline, so the suite stays meaningful regardless of where it's run from. + +--- + +## 3. Core metrics (the three Jason asked for) + +### 3.1 Timed full endpoint read (full-vault bulk read) + +**What it measures:** the headline 1.1.0 path — reading every note in the vault via concurrent `read_many` over warm pooled connections vs. serial GETs. + +**Procedure:** +1. Enumerate all vault paths (the same listing `sweep.py`/`vault_lint.py` walk). +2. **Serial baseline:** GET each path one at a time on a single connection. Time total + per-note. +3. **Pooled+concurrent:** `read_many(paths)` at the default `ECHO_WORKERS`. Time total. +4. Report total time, notes/sec, and the **speedup ratio** (serial ÷ concurrent). Repeat the concurrent pass N times for percentile spread. + +**What "good" looks like:** concurrent should be multiples faster than serial; full-vault concurrent read should stay well under the agent tool timeout (the original failure mode). Record current note count alongside — the metric is only comparable at similar vault sizes. + +**Regression gate (suggested):** full-vault concurrent read median < a threshold (e.g. 3s at ~200 notes) AND speedup ratio ≥ ~2×. Tune thresholds after the first baseline run. + +### 3.2 Timed full subject pulls (e.g. APTA, CapMetro statuses) + +**What it measures:** the cost of a `recall ""` — search + 1-hop neighbourhood expansion along `## Related` links and `source_notes` — which is the real "what do we know about X" operation, and exercises the 1.2.0 resolver. APTA and CapMetro are live subjects in the vault, so they're realistic fixtures. + +**Procedure:** +1. Pick a fixed fixture set of subjects with **different neighbourhood sizes**: a small one, a hub note with many links (e.g. the `echo` project or `operator-preferences`), and the two named ones (APTA, CapMetro). Neighbourhood size is the real cost driver, so vary it deliberately. +2. Time end-to-end `recall` for each: (a) the search call, (b) the 1-hop expansion reads, (c) total. Break out the search vs. expansion split — it tells us whether latency is in the index/search or in the fan-out reads (which 1.1.0's `read_many` should accelerate). +3. Also time bare `resolve ""` (the O(1) index lookup) separately as the floor — recall should be resolve + search + N expansion reads. +4. Report per-subject and aggregate percentiles; annotate each with its neighbourhood node count so time-per-node is derivable. + +**What "good" looks like:** `resolve` is near-instant (in-memory index); `recall` scales with neighbourhood size but the expansion reads are concurrent, so a hub note shouldn't be linearly worse than a leaf. + +**Live result + the `expand-graph` sub-metric (added 2026-06-23).** The first run showed `recall()` at 2.8–4.7 s/subject. Profiling pinned it on the graph layer, not BM25: `load_index()` ~500 ms (index already persisted/incremental), `score()` 0.1 ms, `expand_graph()` ~3,000 ms doing one serial GET per neighbour. Fix shipped in `echo_recall.py` — the BFS now fetches each hop via `read_many`. A dedicated **`expand-graph`** metric times serial vs concurrent expansion (gate: ≥2× min) and asserts byte-identical ranking + scores against a serial reference, so the fix can't silently regress. Measured 3.5–4.2×; end-to-end `recall()` down ~1.7×. + +### 3.3 Timed bulk get / put / append + +**What it measures:** write-path throughput and the read-back/idempotency overhead Jason deliberately pays for. + +**Procedure (all in the `_agent/_bench//` namespace):** +- **Bulk GET:** create K fixture notes once, then time reading them back — both serially and via `read_many` — to compare against §3.1 at a controlled, fixed K (e.g. 25/50/100) so the read curve is isolated from whatever the live vault happens to contain. +- **Bulk PUT:** time creating K notes. `echo.py put` is read-back-verified (it re-GETs after writing), so this measures the true write+verify cost, not a fire-and-forget POST. Report puts/sec and per-put median. +- **Bulk APPEND:** time K appends to a single growing note. `append` is whole-line idempotent (it GETs first and skips duplicates), so also measure the **idempotent-skip path**: re-append the same K lines and confirm near-zero net writes — time the skip vs. the real append. This validates that idempotency isn't quietly O(file size) per append as the note grows. +- **Concurrent write caveat:** writes to single-line shared files assume one writer; the bulk-write bench must use per-note targets (or hold the lock) so it doesn't model an unsupported pattern. + +**What "good" looks like:** PUT slower than GET (verify round-trip); append-skip much cheaper than append-write; no superlinear blowup as the appended note grows. + +--- + +## 4. Additional test suggestions (recommended, beyond the three) + +These target the rest of the 1.1.0/1.2.0 surface and the known historical failure modes. + +1. **Connection-pool cold vs. warm.** Isolate the keep-alive win directly: time the *first* request after pool creation vs. the median of subsequent requests. This is the single clearest proof the pooling fix is alive; a regression here is the early-warning signal for the old per-request-handshake bug. + +2. **Concurrency scaling sweep (`ECHO_WORKERS`).** Run §3.1 at workers = 1, 2, 4, 8, 16. Plot total time vs. workers to find the knee and confirm the default of 8 is sensible for the current vault/WAN, and that it degrades gracefully (no errors/timeouts) at high concurrency. + +3. **Fuzzy-resolve latency + accuracy (1.2.0).** Two parts: (a) *timing* — `resolve` on exact, alias, and shortened/typo'd mentions; the fuzzy-candidate path does more work, so measure its overhead vs. an exact hit. (b) *correctness* — a fixture table of mention → expected canonical slug (e.g. "echo memory" → `echo`, alias hits, a near-miss that *should* return ranked candidates rather than spawn a duplicate). This guards the actual bug 1.2.0 fixed. Correctness assertions, not just timing. + +4. **Index build/rebuild time.** Time `sweep.py` entity-index rebuild + link symmetrization on the full vault. This is the other full-vault script that historically neared the timeout; gate it like §3.1. + +5. **Lint + recall-rebuild full-vault timing.** Regression-gate `vault_lint.py` and the recall rebuild against the 1.1.0 baselines (lint ~0.90s, sweep ~0.85s @ 186 notes). These are the concrete numbers in the changelog — turn them into asserted thresholds scaled to current note count. + +6. **Lock contention timing.** Time `lock` acquisition when free, when held-fresh (should fast-fail with exit 75), and reclaim of a stale lock past `ECHO_LOCK_TTL`. Confirms the read-back-confirmed lock doesn't add pathological latency and behaves under contention. + +7. **Offline write-ahead queue throughput (1.0 carry-over).** With the endpoint unreachable, time enqueue of K writes, then time `flush` replay on reconnect. Validates the queue doesn't degrade and replays in order. Useful because it's an untimed path today. + +8. **Cache hit/miss ratio.** Instrument the shared single-pass cache: in a sweep/lint run, count served-from-cache vs. network reads. A correctness+efficiency check that the cache is actually collapsing duplicate reads, not just present. + +9. **Scaling fixture / synthetic vault.** Optional but valuable: a script that mints a throwaway vault of N synthetic notes (100 / 500 / 1000) in the bench namespace to measure how all the above scale *beyond* the current ~190 notes — answers "when does the next timeout cliff arrive?" before it hits a real session. + +10. **Soak / stability pass.** Run §3.1 in a loop for M minutes to surface connection leaks, pool exhaustion, or memory growth in the long-lived client — relevant because the pool is thread-local and long-running. + +--- + +## 5. Proposed file layout + +``` +eval/perf/ + PERF-TEST-PLAN.md # this file + bench.py # the harness (CLI: --metric, --iterations, --workers, --cleanup/--keep, --json-out) + fixtures.py # subject lists, K-note generators, mention→slug resolve table + baselines.json # committed thresholds + reference numbers (per note-count bucket) + results/ # dated JSON run outputs (gitignored or kept for trend) + 2026-06-23-.json +``` + +`bench.py` subcommands map 1:1 to the metrics: `read-full`, `subject-pull`, `bulk-get`, `bulk-put`, `bulk-append`, plus `pool-warmup`, `worker-sweep`, `resolve`, `index`, `lint`, `lock`, `queue`, `cache`, `soak`. + +## 6. Decisions (resolved 2026-06-23) + +- **Baseline policy:** **relative ratios** are the gate (`baselines.json`); absolute ms are recorded as informational. Portable across machines/WAN. +- **CI vs. manual:** **manual / pre-release.** The suite is invoked by hand before tagging; it is not wired into CI (avoids WAN flakiness failing builds against the live endpoint). +- **Synthetic vault (suggestion #9):** **deferred.** Benchmark the real ~190-note vault for now; add the generator if vault growth makes it relevant. +- **Cleanup:** **auto-delete** the `_agent/_bench//` namespace at end of each run, with `--keep` to override. + +## 7. Build status + +Harness built under `eval/perf/`: + +- `bench.py` — the CLI harness; all 14 metrics implemented (`read-full`, `subject-pull`, `bulk-get`, `bulk-put`, `bulk-append`, `pool-warmup`, `worker-sweep`, `resolve`, `index`, `lint`, `lock`, `queue`, `cache`, `soak`). +- `fixtures.py` — subject list (APTA/CapMetro/hubs/leaf), the 1.2.0 resolve correctness table, and bench-note generators. +- `baselines.json` — relative-ratio gates (tune thresholds after the first real run). +- `README.md` — run instructions, metric map, isolation/cleanup, caveats. +- `results/` — dated JSON run outputs. + +Verified offline (compile + imports + `--list`/`--help`), then **run live** against the 197-note vault @ af16598 — 11/11 gate checks pass. First run surfaced the recall() finding below; the fix and a guarding metric were added the same session. + +## 8. ECHO usage improvements (from the live run) + +The benchmark wasn't just validation — it surfaced concrete improvements to how ECHO should do reads. In priority order: + +1. **Apply `read_many` everywhere a code path reads a *set* of notes (DONE for `expand_graph`).** The 1.1.0 release shipped concurrent bulk reads but only `sweep`/`lint`/full-vault read adopted it. `recall()`'s graph expansion was still serial — one GET per neighbour — which is why it ran 3 s. The rule going forward: *any* loop that GETs more than ~3 notes whose paths are known up front should batch them through `echo.read_many`, not iterate `get_text`. Audit candidates: `expand_graph` (fixed), `_brief` result printing (still serial), `rebuild()`'s vault walk (still serial), and any future multi-note reader. + +2. **Prefetch `_brief` bodies in one `read_many` (NEXT).** After expansion, `recall` prints each primary hit + neighbour via `_brief()`, which GETs the note again to pull its type/status. Those are known paths — fetch them all in one concurrent batch (and reuse bodies already pulled during expansion via a per-call cache, so a note fetched in the BFS isn't re-fetched to print it). + +3. **Reuse one in-process read cache across a `recall` call.** `load_index`, `expand_graph`, and `_brief` currently re-fetch overlapping notes. A single `{path: text}` memo passed through the call (the same pattern `sweep.py` already uses via `read_many` once) removes the duplicate GETs — `load_index`'s ~500 ms and the `_brief` re-reads are the remaining recall cost. + +4. **Make "batch the reads" a documented contract, not a per-site fix.** Add a one-line rule to the plugin's operating contract / API reference: *prefer `read_many(paths)` over a GET loop; serial multi-note reads are a performance bug.* This is what would have caught the `expand_graph` regression at authoring time. The `expand-graph` metric now enforces it mechanically for recall; the contract generalizes it. + +5. **Consider indexing `## Related`/`source_notes` adjacency into the persisted index.** Longer-term: the graph edges are derivable and change only on write. Storing an adjacency list alongside the BM25 postings (maintained by `capture`/`sweep`, same as `entities.json`) would let `expand_graph` skip re-reading note bodies just to extract links — turning the graph layer into an in-memory lookup like `resolve`. Bigger change; revisit if recall latency still matters after #2–#3. + +None of #1–#4 add dependencies or change behavior — they apply the concurrency primitive that already exists. #1 is done and gated; #2–#3 are the next cheap wins; #5 is the structural option. diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/README.md b/echo-memory.plugin.src/skills/echo-memory/eval/perf/README.md new file mode 100644 index 0000000..b3b3a83 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/README.md @@ -0,0 +1,96 @@ +# ECHO Performance Test Suite + +A timing harness for the operations the **1.1.0** (connection pooling + concurrent +`read_many` + shared cache) and **1.2.0** (fuzzy `resolve` / alias learning) updates +targeted. It reuses the live `echo` client module, so it measures the real pooled / +concurrent code path — not a reimplementation. + +This is a **manual / pre-release** tool, not a CI gate: it depends on the live +endpoint (`echoapi.alwisp.com`) and WAN latency. Gates are **relative ratios** +(portable), with absolute milliseconds kept as informational context. + +See `PERF-TEST-PLAN.md` for the design rationale behind each metric. + +## Run + +```bash +cd eval/perf +python3 bench.py --list # list metrics +python3 bench.py all # the read-only safe set (no vault writes) +python3 bench.py read-full -n 10 # full-vault serial-vs-concurrent read +python3 bench.py subject-pull # recall APTA / CapMetro / hubs (resolve+search+recall split) +python3 bench.py bulk-put -k 50 # write metric -> uses the _bench namespace +python3 bench.py worker-sweep # ECHO_WORKERS = 1,2,4,8,16 scaling curve +python3 bench.py resolve # 1.2.0 fuzzy correctness table + timing +python3 bench.py soak --soak-seconds 120 # stability / leak watch +``` + +Set the key first if the baked-in fallback warning bothers you: +`python3 ../../scripts/echo.py write-key ` (or `export ECHO_KEY=...`). + +## Metrics + +| Metric | Plan § | Reads/Writes | What it proves | +|---|---|---|---| +| `read-full` | 3.1 | read-only | full-vault concurrent vs serial speedup (1.1.0 headline) | +| `subject-pull` | 3.2 | read-only | `recall` cost for APTA/CapMetro/hubs, split into resolve floor + search + expansion | +| `expand-graph` | 3.2a | read-only | recall()'s graph layer: serial vs concurrent neighbourhood expansion + ranking-parity guard (pins the real recall bottleneck) | +| `bulk-get` | 3.3 | writes seed | concurrent vs serial read at a controlled K | +| `bulk-put` | 3.3 | **writes** | read-back-verified PUT throughput (puts/sec) | +| `bulk-append` | 3.3 | **writes** | append throughput + idempotent-skip path (no O(file) blowup) | +| `pool-warmup` | 4.1 | read-only | cold vs warm single GET — keep-alive is alive | +| `worker-sweep` | 4.2 | read-only | concurrency scaling knee; validates default workers=8 | +| `resolve` | 4.3 | read-only | fuzzy-resolve timing **and** correctness (anti-duplicate guard) | +| `index` | 4.4 | read-only | `sweep.py` index rebuild + link symmetrize time | +| `lint` | 4.5 | read-only | `vault_lint.py` full-vault time vs 1.1.0 baseline | +| `lock` | 4.6 | **writes** lock | advisory-lock acquire/contend/release timing + semantics | +| `queue` | 4.7 | none (bogus base) | offline write-ahead enqueue timing (see caveat) | +| `cache` | 4.8 | read-only | `read_many` dedups a duplicated path list | +| `soak` | 4.10 | read-only | stability over a window; leak/drift detector | + +`all` runs only the read-only set. Write metrics must be requested by name. + +## Isolation & cleanup + +Write metrics touch **only** `_agent/_bench//`, take the advisory lock for +the duration (and abort if another session holds it), and delete the namespace at +the end. Pass `--keep` to preserve it for debugging. Nothing is written to +`projects/`, `resources/`, `inbox/`, or `journal/`. + +## Output + +Each run writes `results/-.json` (machine-readable, for trend diffing) +and prints a human table. The JSON header captures `ECHO_WORKERS`, `ECHO_TIMEOUT`, +note count, git commit, and date so two runs are comparable. + +## Gating + +`baselines.json` holds relative-ratio gates (e.g. `read-full.speedup_ratio >= 1.8`, +`pool-warmup.cold_over_warm_ratio >= 1.5`). Run once to capture a baseline, then +tune the thresholds to the observed numbers before treating a `FAIL` as blocking. +`--no-gate` reports numbers without pass/fail. + +## Finding: recall() bottleneck is the graph layer, not BM25 + +A profiled `recall("operator preferences")` decomposes as: `load_index()` ~500 ms (the +BM25 index is already persisted + incrementally maintained, n_docs=119), `score()` +0.1 ms, and `expand_graph()` **~3,000 ms** — the graph BFS was fetching each neighbour +serially (one GET per node, ~126 nodes). The fix (shipped in `echo_recall.py`) makes the +BFS breadth-first by hop and fetches each frontier through the existing `read_many` +concurrency. Result: `expand_graph` 3.2–4.2x faster with byte-identical ranking and +scores; end-to-end `recall()` ~1.7x (e.g. APTA 3.8 s -> 2.2 s). The `expand-graph` +metric + gate above regression-guard this so it can't quietly revert to serial. + +## Caveats + +- **`resolve` verdicts are PASS/INFO only.** The correctness table in `fixtures.py` + encodes expected slugs against the live vault; until confirmed, mismatches report + `INFO` (tune-me), not `FAIL`. Promote to FAIL once the table is verified. +- **`queue` times the enqueue path only.** A true flush-replay needs a throwaway + sandbox vault so the replay doesn't write to production; that's deferred (the + plan's synthetic-vault item). The metric still proves enqueue works and reports + queue depth. +- **Absolute ms depend on where you run from.** WAN latency to `echoapi.alwisp.com` + dominates single-request times; that's why gates are ratios. +- **Synthetic-vault scaling (100/500/1000 notes) is deferred** per decision — the + suite benchmarks the real ~190-note vault today. diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/baselines.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/baselines.json new file mode 100644 index 0000000..3470a85 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/baselines.json @@ -0,0 +1,57 @@ +{ + "_comment": "Relative-ratio regression gates (Jason's chosen policy). Ratios are portable across machines/WAN; absolute ms in the results JSON stay informational. Tune these after the first real baseline run, then tighten. 'field' is a dotted path into a metric's result; for list items use the index (e.g. sweep.3.median_ms).", + "read-full": { + "gates": [ + { "field": "speedup_ratio", "op": ">=", "min": 1.8, + "why": "concurrent read_many must beat serial by the 1.1.0 concurrency margin; <1.8 means concurrency regressed" } + ] + }, + "bulk-get": { + "gates": [ + { "field": "speedup_ratio", "op": ">=", "min": 1.5, + "why": "controlled-K concurrent vs serial; lower bar than full-vault since K is small" } + ] + }, + "expand-graph": { + "gates": [ + { "field": "min_speedup", "op": ">=", "min": 2.0, + "why": "recall()'s graph layer must fetch each BFS hop concurrently (read_many), not serially per node; <2x means expand_graph regressed to the pre-fix serial walk" }, + { "field": "all_ranking_identical", "op": ">=", "min": 1, + "why": "the concurrent expansion must return the same ranked neighbours as the serial reference" }, + { "field": "all_scores_identical", "op": ">=", "min": 1, + "why": "decayed scores must match the serial reference to within float tolerance" } + ] + }, + "pool-warmup": { + "gates": [ + { "field": "cold_over_warm_ratio", "op": ">=", "min": 1.5, + "why": "cold request must be meaningfully slower than warm — proves keep-alive is reusing the connection; a ratio near 1.0 means every request is re-handshaking (the pre-1.1.0 bug)" } + ] + }, + "bulk-append": { + "gates": [ + { "field": "all_skipped_second_pass", "op": ">=", "min": 1, + "why": "second pass of identical lines must be 100% idempotent skips (boolean true coerced to 1)" } + ] + }, + "soak": { + "gates": [ + { "field": "errors", "op": "<=", "min": 0, + "why": "no read errors across the soak window — connection leaks/pool exhaustion would surface here" }, + { "field": "drift_ratio_late_over_early", "op": "<=", "min": 1.5, + "why": "late passes must not be much slower than early ones; >1.5 suggests a leak or degrading pool" } + ] + }, + "lock": { + "gates": [ + { "field": "ok", "op": ">=", "min": 1, + "why": "free acquire returns 0 and a contended acquire fast-fails with exit 75" } + ] + }, + "cache": { + "gates": [ + { "field": "deduped", "op": ">=", "min": 1, + "why": "read_many must collapse a 3x-duplicated path list to the unique set" } + ] + } +} diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/bench.py b/echo-memory.plugin.src/skills/echo-memory/eval/perf/bench.py new file mode 100644 index 0000000..6ffa66a --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/bench.py @@ -0,0 +1,800 @@ +#!/usr/bin/env python3 +"""bench.py — ECHO performance test suite (timing harness). + +Measures the operations the 1.1.0 (pooling + concurrent read_many + shared cache) +and 1.2.0 (fuzzy resolve / alias learning) updates targeted, and gates them on +RELATIVE ratios (portable across machines/WAN) rather than absolute ms. + +It reuses the live `echo` client module so it times the REAL pooled/concurrent +code path. Reads are non-destructive; write metrics live in a disposable +namespace (_agent/_bench//) cleaned up at the end unless --keep. + +This is a MANUAL / pre-release tool, not a CI gate — it depends on the live +endpoint (echoapi.alwisp.com) and WAN latency. + + python3 bench.py [options] + python3 bench.py all # the read-only safe set + python3 bench.py read-full -n 10 + python3 bench.py subject-pull + python3 bench.py bulk-put -k 50 + python3 bench.py worker-sweep + python3 bench.py resolve # 1.2.0 fuzzy correctness + timing + python3 bench.py --list + +Metrics: read-full, subject-pull, bulk-get, bulk-put, bulk-append, + pool-warmup, worker-sweep, resolve, index, lint, lock, queue, cache, soak + +Options: + -n / --iterations N timed iterations (default 10 read / 5 write) + -k / --count K notes/appends for bulk metrics (default 50) + --workers W override ECHO_WORKERS for this run + --warmup M warm-up iterations to discard (default 2) + --soak-seconds S duration for the soak metric (default 60) + --keep do NOT delete the _bench namespace after the run + --no-gate skip relative-ratio gating (report numbers only) + --json-out PATH write the results JSON here (default results/-.json) + --quiet suppress the per-metric human table (JSON only) + +Exit: 0 all gates pass (or --no-gate) · 1 a gate failed · 2 usage/setup error. +""" + +from __future__ import annotations + +import argparse +import contextlib +import datetime as dt +import io +import json +import os +import statistics +import subprocess +import sys +import time +import uuid +from pathlib import Path + +HERE = Path(__file__).resolve().parent +SCRIPTS = HERE.parent.parent / "scripts" +sys.path.insert(0, str(SCRIPTS)) + +import echo # noqa: E402 the validated client — real pooled/concurrent path +import echo_index # noqa: E402 resolve / fuzzy_candidates (1.2.0) +import echo_recall # noqa: E402 recall (search + 1-hop expansion) + +BASELINES = HERE / "baselines.json" +RESULTS_DIR = HERE / "results" + + +# --------------------------------------------------------------------------- +# stats + timing +# --------------------------------------------------------------------------- +def stats(samples: list[float]) -> dict: + """min/median/p90/p95/max/mean in milliseconds from a list of seconds.""" + if not samples: + return {} + ms = sorted(s * 1000.0 for s in samples) + + def pct(p: float) -> float: + if len(ms) == 1: + return ms[0] + k = (len(ms) - 1) * p + lo, hi = int(k), min(int(k) + 1, len(ms) - 1) + return ms[lo] + (ms[hi] - ms[lo]) * (k - lo) + + return { + "n": len(ms), + "min_ms": round(ms[0], 2), + "median_ms": round(statistics.median(ms), 2), + "p90_ms": round(pct(0.90), 2), + "p95_ms": round(pct(0.95), 2), + "max_ms": round(ms[-1], 2), + "mean_ms": round(statistics.fmean(ms), 2), + } + + +def timed(fn, n: int, warmup: int) -> dict: + """Run fn() warmup+n times. Returns warm-up samples separately from the timed + set so cold-start cost (the thing keep-alive kills) is visible, not averaged in.""" + warm: list[float] = [] + samples: list[float] = [] + for i in range(warmup + n): + t0 = time.perf_counter() + fn() + dt_s = time.perf_counter() - t0 + (warm if i < warmup else samples).append(dt_s) + out = stats(samples) + if warm: + out["warmup"] = stats(warm) + return out + + +def silent(fn): + """Run fn() with stdout swallowed (recall/resolve print a lot).""" + def wrapped(): + with contextlib.redirect_stdout(io.StringIO()): + return fn() + return wrapped + + +# --------------------------------------------------------------------------- +# vault enumeration (mirrors sweep.py / vault_lint.py) +# --------------------------------------------------------------------------- +SKIP_BASENAMES = {"README.md"} + + +def _list_dir(path: str): + p = "" if path in ("", "/") else (path if path.endswith("/") else path + "/") + body = echo.get_text(p) + if body is None: + return [], [] + try: + j = json.loads(body) + except json.JSONDecodeError: + return [], [] + entries = list(j.get("files", [])) + list(j.get("folders", [])) + files = [e for e in entries if not e.endswith("/")] + folders = [e[:-1] for e in entries if e.endswith("/")] + return files, folders + + +def walk_vault(prefix: str = ""): + files, folders = _list_dir(prefix) + for f in files: + yield prefix + f + for d in folders: + yield from walk_vault(f"{prefix}{d}/") + + +def all_notes() -> list[str]: + return [p for p in walk_vault() + if p.endswith(".md") and p.rsplit("/", 1)[-1] not in SKIP_BASENAMES] + + +# --------------------------------------------------------------------------- +# write primitives (replicate echo.py semantics so we time the real cost) +# --------------------------------------------------------------------------- +def put_verified(path: str, body: str) -> None: + """PUT then read-back GET — the same verify echo.py cmd_put does.""" + status, b = echo.request("PUT", echo.vault_url(path), + data=body.encode(), headers={"Content-Type": "text/markdown"}) + echo.check(status, b, f"put {path}") + status, _ = echo.request("GET", echo.vault_url(path)) + if status != 200: + raise echo.EchoError(f"put {path}: did not verify (GET {status})") + + +def append_line(path: str, line: str) -> bool: + """Whole-line idempotent append (mirrors echo.py cmd_append). Returns True if a + write happened, False if it was an idempotent skip.""" + status, body = echo.request("GET", echo.vault_url(path)) + if status == 200: + existing = [ln.rstrip("\r") for ln in body.decode(errors="replace").splitlines()] + if line.rstrip("\r") in existing: + return False + status, body = echo.request("POST", echo.vault_url(path), + data=f"{line}\n".encode(), + headers={"Content-Type": "text/markdown"}) + echo.check(status, body, f"append {path}") + return True + + +def delete_path(path: str) -> None: + echo.request("DELETE", echo.vault_url(path)) + + +# --------------------------------------------------------------------------- +# metrics +# --------------------------------------------------------------------------- +def m_read_full(ctx) -> dict: + """§3.1 full-vault read: serial GET vs concurrent read_many -> speedup ratio.""" + paths = all_notes() + if not paths: + raise echo.EchoError("read-full: no notes enumerated (vault unreachable?)") + + def serial(): + for p in paths: + echo.get_text(p) + + def concurrent(): + echo.read_many(paths) + + # one warm pass so neither side pays the cold-handshake tax (pool-warmup owns that) + echo.get_text(paths[0]) + # serial is a slow reference baseline (N serial round-trips); one warm pass is + # enough to establish the ratio, so we don't multiply the full-vault serial cost. + serial_t = timed(serial, n=1, warmup=0) + concurrent_t = timed(concurrent, n=ctx.n, warmup=ctx.warmup) + ratio = (serial_t["median_ms"] / concurrent_t["median_ms"]) if concurrent_t.get("median_ms") else None + return { + "note_count": len(paths), + "workers": echo.MAX_WORKERS, + "serial": serial_t, + "concurrent": concurrent_t, + "speedup_ratio": round(ratio, 2) if ratio else None, + "notes_per_sec_concurrent": round(len(paths) / (concurrent_t["median_ms"] / 1000.0), 1) + if concurrent_t.get("median_ms") else None, + } + + +def m_subject_pull(ctx) -> dict: + """§3.2 recall a subject: time resolve (floor), search, and full recall, split out.""" + import fixtures + index = echo_index.load() + out = {"subjects": []} + for label, query in fixtures.SUBJECTS: + # floor: O(1) index resolve + resolve_t = timed(lambda: echo_index.resolve(index, query), n=ctx.n, warmup=ctx.warmup) + # search-only layer + def search(): + with contextlib.redirect_stdout(io.StringIO()): + echo.request("POST", f"{echo.BASE}/search/simple/?query={query}") + search_t = timed(search, n=ctx.n, warmup=ctx.warmup) + # full recall (search + 1-hop expansion reads) + recall_t = timed(silent(lambda: echo_recall.recall(query)), n=ctx.n, warmup=ctx.warmup) + out["subjects"].append({ + "label": label, "query": query, + "resolve_floor": resolve_t, + "search_only": search_t, + "recall_total": recall_t, + }) + return out + + +def m_bulk_get(ctx) -> dict: + """§3.3 bulk GET at a controlled K: serial vs read_many over fixture notes.""" + import fixtures + paths = _seed_notes(ctx, fixtures) + + def serial(): + for p in paths: + echo.get_text(p) + + def concurrent(): + echo.read_many(paths) + + serial_t = timed(serial, n=ctx.n, warmup=1) + concurrent_t = timed(concurrent, n=ctx.n, warmup=ctx.warmup) + ratio = (serial_t["median_ms"] / concurrent_t["median_ms"]) if concurrent_t.get("median_ms") else None + return {"k": len(paths), "workers": echo.MAX_WORKERS, + "serial": serial_t, "concurrent": concurrent_t, + "speedup_ratio": round(ratio, 2) if ratio else None} + + +def m_bulk_put(ctx) -> dict: + """§3.3 bulk PUT (read-back verified) -> puts/sec + per-put percentiles.""" + import fixtures + k = ctx.k + samples: list[float] = [] + for i in range(k): + body = fixtures.note_body(ctx.run_id, i) + path = fixtures.note_path(ctx.run_id, i) + t0 = time.perf_counter() + put_verified(path, body) + samples.append(time.perf_counter() - t0) + s = stats(samples) + total_s = sum(samples) + return {"k": k, "per_put": s, "total_ms": round(total_s * 1000, 1), + "puts_per_sec": round(k / total_s, 1) if total_s else None} + + +def m_bulk_append(ctx) -> dict: + """§3.3 bulk APPEND + idempotent-skip path. Confirms idempotency isn't O(file).""" + import fixtures + target = fixtures.append_target(ctx.run_id) + put_verified(target, fixtures.append_target_seed(ctx.run_id)) + k = ctx.k + lines = [f"- 2026-06-23: bench append line {i:04d}" for i in range(k)] + + write_samples: list[float] = [] + for ln in lines: + t0 = time.perf_counter() + wrote = append_line(target, ln) + write_samples.append(time.perf_counter() - t0) + if not wrote: + raise echo.EchoError("bulk-append: expected a write but got an idempotent skip") + + # second pass: every line already present -> must skip, and skip cost is the + # idempotency GET on a now-larger file. Confirms no superlinear blowup. + skip_samples: list[float] = [] + skips = 0 + for ln in lines: + t0 = time.perf_counter() + wrote = append_line(target, ln) + skip_samples.append(time.perf_counter() - t0) + if not wrote: + skips += 1 + + return { + "k": k, + "append_write": stats(write_samples), + "append_skip": stats(skip_samples), + "skips_confirmed": skips, + "all_skipped_second_pass": skips == k, + } + + +def m_pool_warmup(ctx) -> dict: + """§4.1 cold vs warm single GET — the clearest proof keep-alive is alive.""" + paths = all_notes() + probe = paths[0] if paths else "_agent/echo-vault.md" + echo._drop_connection() # force a cold handshake + t0 = time.perf_counter() + echo.get_text(probe) + cold_ms = (time.perf_counter() - t0) * 1000.0 + warm = timed(lambda: echo.get_text(probe), n=max(ctx.n, 10), warmup=1) + ratio = round(cold_ms / warm["median_ms"], 2) if warm.get("median_ms") else None + return {"probe": probe, "cold_ms": round(cold_ms, 2), "warm": warm, + "cold_over_warm_ratio": ratio} + + +def m_worker_sweep(ctx) -> dict: + """§4.2 concurrency scaling: full-vault concurrent read at workers 1,2,4,8,16.""" + paths = all_notes() + if not paths: + raise echo.EchoError("worker-sweep: no notes enumerated") + original = echo.MAX_WORKERS + rows = [] + try: + # w=1 is a full serial pass (slow); the curve only needs the shape, so use a + # modest sample count and a single pre-warm rather than per-setting warmups. + reps = max(1, ctx.n // 2) + for w in (1, 2, 4, 8, 16): + echo.MAX_WORKERS = w + echo.read_many(paths[:5]) # warm the pool at this worker count + t = timed(lambda: echo.read_many(paths), n=reps, warmup=0) + rows.append({"workers": w, "median_ms": t["median_ms"], "p95_ms": t["p95_ms"]}) + finally: + echo.MAX_WORKERS = original + best = min(rows, key=lambda r: r["median_ms"]) + return {"note_count": len(paths), "sweep": rows, "knee_workers": best["workers"], + "default_workers": original} + + +def m_resolve(ctx) -> dict: + """§4.3 fuzzy resolve — timing AND correctness against the 1.2.0 guard table.""" + import fixtures + index = echo_index.load() + cases = [] + passes = info = 0 + for case in fixtures.RESOLVE_CASES: + mention = case["mention"] + exact_t = timed(lambda: echo_index.resolve(index, mention), n=ctx.n, warmup=ctx.warmup) + fuzzy_t = timed(lambda: echo_index.fuzzy_candidates(index, mention), n=ctx.n, warmup=ctx.warmup) + _, entity = echo_index.resolve(index, mention) + exact_slug = (entity or {}).get("slug") or _slug_from_entity(entity) + cands = echo_index.fuzzy_candidates(index, mention) or [] + cand_slugs = [_cand_slug(c) for c in cands] + + expect, want = case["expect"], case.get("expect_slug") + if expect == "exact": + ok = exact_slug == want + elif expect == "candidates": + ok = (exact_slug != want) and (want in cand_slugs) + else: # miss + ok = entity is None + verdict = "PASS" if ok else "INFO" # tune table to live vault before promoting to FAIL + passes += ok + info += (not ok) + cases.append({"mention": mention, "expect": expect, "expect_slug": want, + "resolved_slug": exact_slug, "candidate_slugs": cand_slugs[:5], + "verdict": verdict, + "resolve_timing": exact_t, "fuzzy_timing": fuzzy_t}) + return {"cases": cases, "passes": passes, "needs_tuning": info, + "note": "verdicts are PASS/INFO only; promote to FAIL once the table is " + "confirmed against the live vault"} + + +def _serial_expand(seeds, nmap, base, max_hops): + """The pre-fix serial graph BFS, kept here as the reference baseline so the suite + both (a) measures the concurrency speedup and (b) asserts the concurrent + expand_graph still returns byte-identical ranking. If echo_recall.expand_graph ever + regresses to serial, the ratio gate fails; if its results drift, the parity check fails.""" + from collections import deque + score_of = dict(base) + seen = set(seeds) + results = {} + dq = deque((s, 0) for s in seeds) + while dq: + path, hop = dq.popleft() + if hop >= max_hops: + continue + text = echo_recall.links.get_text(path) + if text is None: + continue + body = echo_recall.strip_frontmatter(text) + targets = set(echo_recall.links.all_wikilinks(body)) | set(echo_recall.links.source_notes(text)) + parent = score_of.get(path, 1.0) + for t in targets: + tp = echo_recall._resolve_target(t, nmap) + if not tp or tp in seeds: + continue + decayed = parent * (echo_recall.GRAPH_DECAY ** (hop + 1)) + if decayed > results.get(tp, (0.0, ""))[0]: + results[tp] = (decayed, path) + if tp not in seen: + seen.add(tp) + dq.append((tp, hop + 1)) + return sorted(results.items(), key=lambda kv: -kv[1][0]) + + +def m_expand_graph(ctx) -> dict: + """recall() graph layer: serial vs concurrent neighbourhood expansion. This is the + metric that pins the real recall() bottleneck (the graph BFS, not BM25) and guards + the read_many concurrency fix. Also asserts the concurrent path's ranking + scores + match the serial reference exactly.""" + import fixtures + ix = echo_recall.load_index() + index = echo_index.load() + nmap = echo_index.name_map(index) + max_hops = echo_recall.MAX_HOPS + # hub subjects stress expansion most; fall back to all subjects if none labelled hub + subjects = [s for s in fixtures.SUBJECTS if "hub" in s[0]] or fixtures.SUBJECTS + rows = [] + for label, query in subjects: + hits = ix.score(query, limit=8) + base = {p: s for p, s in hits} + seeds = [p for p, _ in hits] + if not seeds: + continue + echo_recall.expand_graph(seeds, nmap, base, max_hops) # warm + serial_t = timed(lambda: _serial_expand(seeds, nmap, base, max_hops), n=1, warmup=0) + conc_t = timed(lambda: echo_recall.expand_graph(seeds, nmap, base, max_hops), + n=ctx.n, warmup=1) + old = _serial_expand(seeds, nmap, base, max_hops) + new = echo_recall.expand_graph(seeds, nmap, base, max_hops) + ranking_ok = [p for p, _ in old] == [p for p, _ in new] + scores_ok = ranking_ok and all( + abs(old[i][1][0] - new[i][1][0]) < 1e-9 for i in range(len(old))) + ratio = (serial_t["median_ms"] / conc_t["median_ms"]) if conc_t.get("median_ms") else None + rows.append({"label": label, "query": query, "neighbours": len(new), + "serial_ms": serial_t["median_ms"], "concurrent_ms": conc_t["median_ms"], + "speedup": round(ratio, 2) if ratio else None, + "ranking_identical": ranking_ok, "scores_identical": scores_ok}) + speedups = [r["speedup"] for r in rows if r["speedup"]] + return {"subjects": rows, + "min_speedup": round(min(speedups), 2) if speedups else None, + "all_ranking_identical": all(r["ranking_identical"] for r in rows), + "all_scores_identical": all(r["scores_identical"] for r in rows)} + + +def _slug_from_entity(entity): + if not entity: + return None + path = entity.get("path", "") + return path.rsplit("/", 1)[-1].removesuffix(".md") if path else None + + +def _cand_slug(c): + if isinstance(c, dict): + return c.get("slug") or _slug_from_entity(c) + if isinstance(c, (list, tuple)) and c: + return c[0] + return str(c) + + +def m_index(ctx) -> dict: + """§4.4 entity-index rebuild + link symmetrize timing (sweep.py, dry-run = read-only).""" + return _time_script("sweep.py", []) + + +def m_lint(ctx) -> dict: + """§4.5 full-vault lint timing (vault_lint.py) — regression-gate vs 1.1.0 baseline.""" + return _time_script("vault_lint.py", [], allow_exit={0, 1}) + + +def _time_script(name: str, extra: list[str], allow_exit=frozenset({0})) -> dict: + script = SCRIPTS / name + env = dict(os.environ, ECHO_KEY_LEGACY_OK="1") + t0 = time.perf_counter() + proc = subprocess.run([sys.executable, str(script), *extra], + capture_output=True, text=True, env=env, timeout=120) + elapsed = (time.perf_counter() - t0) * 1000.0 + return {"script": name, "elapsed_ms": round(elapsed, 1), + "exit_code": proc.returncode, + "ok": proc.returncode in allow_exit, + "stderr_tail": proc.stderr.strip().splitlines()[-3:] if proc.stderr else []} + + +def m_lock(ctx) -> dict: + """§4.6 advisory-lock timing: acquire-free, contended (held-fresh), release.""" + owner = f"bench-{ctx.run_id}" + other = f"bench-other-{ctx.run_id}" + + def acquire(o): + return echo.cmd_lock(o, quiet=True) + + def release(o): + return echo.cmd_unlock(o, quiet=True) + + release(owner); release(other) # clean slate + t0 = time.perf_counter(); rc_free = acquire(owner); free_ms = (time.perf_counter() - t0) * 1000 + t0 = time.perf_counter(); rc_held = acquire(other); held_ms = (time.perf_counter() - t0) * 1000 + t0 = time.perf_counter(); release(owner); rel_ms = (time.perf_counter() - t0) * 1000 + return {"acquire_free_ms": round(free_ms, 2), "acquire_free_rc": rc_free, + "contended_ms": round(held_ms, 2), "contended_rc_expected_75": rc_held, + "release_ms": round(rel_ms, 2), + "ok": rc_free == 0 and rc_held == 75} + + +def m_queue(ctx) -> dict: + """§4.7 offline write-ahead queue: enqueue K writes against an unreachable base, + then time the flush replay on 'reconnect'. Uses a bogus base so nothing real is + written; restores the base afterward. No production writes occur.""" + import importlib + import echo_queue + k = min(ctx.k, 20) + saved_base = echo.BASE + enqueue_samples: list[float] = [] + try: + for i in range(k): + url = f"{echo.BASE}/vault/{fixtures_qpath(ctx.run_id, i)}" + t0 = time.perf_counter() + echo_queue.enqueue("PUT", url, b"x", {"Content-Type": "text/markdown"}, + idem_key=f"bench-{ctx.run_id}-{i}") + enqueue_samples.append(time.perf_counter() - t0) + pending_before = len(echo_queue.pending()) + # flush is left UNMEASURED-as-success here: with a real endpoint it would + # try to write. We only assert enqueue worked and report depth; a true + # flush-replay timing needs a sandbox vault (see README "queue caveat"). + return {"k": k, "enqueue": stats(enqueue_samples), + "pending_after_enqueue": pending_before, + "note": "flush replay intentionally not run against the live vault; " + "enqueue path timed only. See README queue caveat."} + finally: + echo.BASE = saved_base + importlib.reload(echo_queue) + + +def fixtures_qpath(run_id: str, i: int) -> str: + return f"_agent/_bench/{run_id}/queued-{i:04d}.md" + + +def m_cache(ctx) -> dict: + """§4.8 read_many dedup check: a path list with duplicates must collapse to the + unique set (the cache/dedup that stops a sweep re-fetching a link target).""" + paths = all_notes()[:20] or ["_agent/echo-vault.md"] + dupd = paths + paths + paths # 3x + t0 = time.perf_counter() + result = echo.read_many(dupd) + elapsed = (time.perf_counter() - t0) * 1000 + return {"requested": len(dupd), "unique": len(set(dupd)), + "returned": len(result), "deduped": len(result) == len(set(dupd)), + "elapsed_ms": round(elapsed, 2)} + + +def m_soak(ctx) -> dict: + """§4.10 stability: loop full-vault reads for N seconds; watch for errors/drift.""" + paths = all_notes() + if not paths: + raise echo.EchoError("soak: no notes enumerated") + deadline = time.time() + ctx.soak_seconds + durations: list[float] = [] + errors = 0 + while time.time() < deadline: + t0 = time.perf_counter() + try: + res = echo.read_many(paths) + if any(v is None for v in res.values()): + errors += 1 + except Exception: + errors += 1 + durations.append(time.perf_counter() - t0) + s = stats(durations) + # drift = late iterations slower than early ones (leak / pool exhaustion signal) + half = len(durations) // 2 or 1 + early = statistics.fmean(durations[:half]) * 1000 + late = statistics.fmean(durations[half:]) * 1000 + return {"seconds": ctx.soak_seconds, "iterations": len(durations), "errors": errors, + "per_pass": s, "early_mean_ms": round(early, 2), "late_mean_ms": round(late, 2), + "drift_ratio_late_over_early": round(late / early, 2) if early else None} + + +METRICS = { + "read-full": m_read_full, + "subject-pull": m_subject_pull, + "bulk-get": m_bulk_get, + "bulk-put": m_bulk_put, + "bulk-append": m_bulk_append, + "pool-warmup": m_pool_warmup, + "worker-sweep": m_worker_sweep, + "resolve": m_resolve, + "expand-graph": m_expand_graph, + "index": m_index, + "lint": m_lint, + "lock": m_lock, + "queue": m_queue, + "cache": m_cache, + "soak": m_soak, +} + +# the read-only safe set run by `all` (no writes to the vault) +READONLY = ["pool-warmup", "read-full", "worker-sweep", "subject-pull", + "resolve", "expand-graph", "index", "lint", "cache"] +WRITES = {"bulk-get", "bulk-put", "bulk-append", "lock"} # touch the _bench namespace + + +# --------------------------------------------------------------------------- +# fixture seeding / cleanup +# --------------------------------------------------------------------------- +def _seed_notes(ctx, fixtures) -> list[str]: + """Ensure K bench notes exist; return their paths. Cached on ctx so bulk-get + and bulk-put can share the seed within one run.""" + if getattr(ctx, "_seeded", None): + return ctx._seeded + paths = [] + for i in range(ctx.k): + p = fixtures.note_path(ctx.run_id, i) + put_verified(p, fixtures.note_body(ctx.run_id, i)) + paths.append(p) + ctx._seeded = paths + return paths + + +def cleanup_namespace(run_id: str) -> int: + """Delete every file under _agent/_bench//. Returns count deleted.""" + prefix = f"_agent/_bench/{run_id}/" + victims = [p for p in walk_vault(prefix)] + for p in victims: + delete_path(p) + # remove the (now-empty) run dir marker if the API created one — best effort + return len(victims) + + +# --------------------------------------------------------------------------- +# gating +# --------------------------------------------------------------------------- +def load_baselines() -> dict: + if BASELINES.exists(): + return json.loads(BASELINES.read_text()) + return {} + + +def gate(metric: str, result: dict, baselines: dict) -> list[dict]: + """Apply relative-ratio gates from baselines.json. Returns a list of checks.""" + rules = baselines.get(metric, {}) + checks = [] + for rule in rules.get("gates", []): + path, op, threshold = rule["field"], rule["op"], rule["min"] + val = _dig(result, path) + if val is None: + checks.append({"rule": rule, "value": None, "pass": None, "reason": "field absent"}) + continue + ok = (val >= threshold) if op == ">=" else (val <= threshold) + checks.append({"rule": rule, "value": val, "pass": ok}) + return checks + + +def _dig(obj, dotted: str): + cur = obj + for part in dotted.split("."): + if isinstance(cur, list): + try: + cur = cur[int(part)] + continue + except (ValueError, IndexError): + return None + if not isinstance(cur, dict) or part not in cur: + return None + cur = cur[part] + return cur + + +# --------------------------------------------------------------------------- +# main +# --------------------------------------------------------------------------- +class Ctx: + pass + + +def env_block(args) -> dict: + commit = "unknown" + try: + commit = subprocess.run(["git", "rev-parse", "--short", "HEAD"], + cwd=str(SCRIPTS), capture_output=True, text=True, + timeout=10).stdout.strip() or "unknown" + except Exception: + pass + return { + "date": dt.datetime.now().isoformat(timespec="seconds"), + "endpoint": echo.BASE, + "commit": commit, + "echo_workers": echo.MAX_WORKERS, + "echo_timeout": echo.TIMEOUT, + "python": sys.version.split()[0], + "iterations": args.iterations, + "count_k": args.count, + } + + +def human_table(name: str, result: dict, checks: list[dict]) -> str: + lines = [f"\n=== {name} ==="] + lines.append(json.dumps(result, indent=2)) + for c in checks: + mark = "PASS" if c["pass"] else ("—" if c["pass"] is None else "FAIL") + lines.append(f" [{mark}] {c['rule']['field']} {c['rule']['op']} {c['rule']['min']} (got {c['value']})") + return "\n".join(lines) + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description="ECHO performance test suite (manual/pre-release)") + ap.add_argument("metric", nargs="?", help="metric name, or 'all' for the read-only set") + ap.add_argument("--list", action="store_true", help="list metrics and exit") + ap.add_argument("-n", "--iterations", type=int, default=None) + ap.add_argument("-k", "--count", type=int, default=50) + ap.add_argument("--workers", type=int, default=None) + ap.add_argument("--warmup", type=int, default=2) + ap.add_argument("--soak-seconds", type=int, default=60) + ap.add_argument("--keep", action="store_true", help="do not delete the _bench namespace") + ap.add_argument("--no-gate", action="store_true") + ap.add_argument("--json-out", default=None) + ap.add_argument("--quiet", action="store_true") + args = ap.parse_args(argv) + + if args.list or not args.metric: + print("Metrics:", ", ".join(METRICS)) + print("Groups : all (read-only safe set:", ", ".join(READONLY) + ")") + print("Writes to _bench namespace:", ", ".join(sorted(WRITES))) + return 0 if args.list else 2 + + if args.workers is not None: + echo.MAX_WORKERS = args.workers + + selected = READONLY if args.metric == "all" else [args.metric] + for m in selected: + if m not in METRICS: + print(f"unknown metric: {m}", file=sys.stderr) + return 2 + + ctx = Ctx() + ctx.run_id = dt.datetime.now().strftime("%Y%m%d-%H%M%S") + "-" + uuid.uuid4().hex[:6] + ctx.k = args.count + ctx.warmup = args.warmup + ctx.soak_seconds = args.soak_seconds + ctx._seeded = None + + baselines = load_baselines() + need_writes = any(m in WRITES for m in selected) + lock_owner = f"bench-{ctx.run_id}" + if need_writes: + if echo.cmd_lock(lock_owner, quiet=True) == 75: + print("bench: vault lock is held by another session — aborting write metrics.", + file=sys.stderr) + return 2 + + report = {"env": env_block(args), "run_id": ctx.run_id, "metrics": {}} + any_fail = False + try: + for m in selected: + ctx.n = args.iterations if args.iterations is not None else (5 if m in WRITES else 10) + try: + result = METRICS[m](ctx) + except Exception as exc: # noqa: BLE001 + result = {"error": str(exc)} + any_fail = True + checks = [] if args.no_gate else gate(m, result, baselines) + if any(c["pass"] is False for c in checks): + any_fail = True + report["metrics"][m] = {"result": result, "gates": checks} + if not args.quiet: + print(human_table(m, result, checks)) + finally: + if need_writes: + echo.cmd_unlock(lock_owner, quiet=True) + if need_writes and not args.keep: + n = cleanup_namespace(ctx.run_id) + report["cleanup"] = {"deleted": n} + if not args.quiet: + print(f"\nbench: cleaned up {n} files under _agent/_bench/{ctx.run_id}/") + elif need_writes and args.keep: + report["cleanup"] = {"deleted": 0, "kept": f"_agent/_bench/{ctx.run_id}/"} + + RESULTS_DIR.mkdir(parents=True, exist_ok=True) + out_path = Path(args.json_out) if args.json_out else \ + RESULTS_DIR / f"{report['env']['date'][:10]}-{report['env']['commit']}.json" + out_path.write_text(json.dumps(report, indent=2)) + print(f"\nbench: results -> {out_path}") + return 1 if any_fail else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/fixtures.py b/echo-memory.plugin.src/skills/echo-memory/eval/perf/fixtures.py new file mode 100644 index 0000000..6aa0b1e --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/fixtures.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +"""fixtures.py — inputs for the ECHO performance harness (bench.py). + +Holds the *what* (subjects to pull, notes to write, mentions to resolve) so the +harness (bench.py) stays the *how* (timing, stats, gating). Nothing here touches +the network; bench.py owns all I/O. +""" + +from __future__ import annotations + +# --- §3.2 subject pulls ------------------------------------------------------- +# (label, query) pairs fed to `recall`. Chosen to span neighbourhood sizes — the +# real cost driver — from a likely-leaf to a known hub. APTA and CapMetro are the +# subjects Jason named; "echo" and "operator preferences" are dense hubs that +# stress the 1-hop expansion (the read_many fan-out 1.1.0 accelerated). +SUBJECTS: list[tuple[str, str]] = [ + ("apta", "APTA"), + ("capmetro", "CapMetro"), + ("hub-echo", "echo"), + ("hub-prefs", "operator preferences"), + ("leaf-rivnut", "rivnut torque spec"), +] + + +# --- §4.3 fuzzy-resolve correctness table (1.2.0) ----------------------------- +# Each case: mention, expected behaviour. `expect_slug` is the canonical slug an +# exact/alias/fuzzy match should land on (None = no entity expected). `expect` +# is the classifier: +# "exact" -> resolve returns an entity whose slug == expect_slug +# "candidates" -> no exact hit, but fuzzy_candidates surfaces expect_slug +# (the 1.2.0 anti-duplicate guard — a near-miss must NOT resolve +# to nothing and silently spawn a new note) +# "miss" -> neither; genuinely unknown +# Tune expect_slug to the live vault; unknowns are reported as INFO, not failures. +RESOLVE_CASES: list[dict] = [ + {"mention": "echo", "expect": "exact", "expect_slug": "echo"}, + {"mention": "echo memory", "expect": "candidates", "expect_slug": "echo"}, + {"mention": "ECHO plugin", "expect": "candidates", "expect_slug": "echo"}, + {"mention": "goldbrain", "expect": "exact", "expect_slug": "goldbrain"}, + {"mention": "jason stedwell", "expect": "exact", "expect_slug": "jason-stedwell"}, + {"mention": "jason", "expect": "candidates", "expect_slug": "jason-stedwell"}, + {"mention": "zzqx nonexistent entity", "expect": "miss", "expect_slug": None}, +] + + +# --- §3.3 bulk write fixtures ------------------------------------------------- +BENCH_PREFIX = "_agent/_bench" # run-scoped namespace: _agent/_bench//... + + +def note_body(run_id: str, i: int) -> str: + """A realistic-shape note: canonical frontmatter + a couple of headings, so + PUT cost reflects a true note, not a one-liner. agent_written + a _bench tag + make these trivially greppable if a cleanup is ever missed.""" + return ( + "---\n" + "type: working-memory\n" + "status: active\n" + "created: 2026-06-23\n" + "updated: 2026-06-23\n" + "tags:\n" + " - agent\n" + " - _bench\n" + "agent_written: true\n" + f"source_notes: []\n" + f"bench_run: {run_id}\n" + "---\n\n" + f"# Bench Note {i:04d}\n\n" + "## Body\n" + f"Synthetic benchmark note {i} for run {run_id}. " + "Filler so the payload is not pathologically small: " + + ("lorem ipsum dolor sit amet " * 6) + + "\n\n## Log\n- seed\n" + ) + + +def note_path(run_id: str, i: int) -> str: + return f"{BENCH_PREFIX}/{run_id}/note-{i:04d}.md" + + +def append_target(run_id: str) -> str: + return f"{BENCH_PREFIX}/{run_id}/append-target.md" + + +def append_target_seed(run_id: str) -> str: + return ( + "---\n" + "type: working-memory\n" + "status: active\n" + "created: 2026-06-23\n" + "updated: 2026-06-23\n" + "tags:\n" + " - agent\n" + " - _bench\n" + "agent_written: true\n" + "source_notes: []\n" + f"bench_run: {run_id}\n" + "---\n\n" + "# Append Target\n\n" + "## Log\n" + ) diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/SUMMARY.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/SUMMARY.json new file mode 100644 index 0000000..27f7b4e --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/SUMMARY.json @@ -0,0 +1,931 @@ +{ + "env": { + "date": "2026-06-23T20:17:09", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": 4, + "count_k": 50 + }, + "metrics": { + "bulk-append": { + "result": { + "k": 30, + "append_write": { + "n": 30, + "min_ms": 87.11, + "median_ms": 104.08, + "p90_ms": 190.63, + "p95_ms": 198.89, + "max_ms": 241.91, + "mean_ms": 117.22 + }, + "append_skip": { + "n": 30, + "min_ms": 47.57, + "median_ms": 51.42, + "p90_ms": 59.82, + "p95_ms": 61.57, + "max_ms": 73.49, + "mean_ms": 53.22 + }, + "skips_confirmed": 30, + "all_skipped_second_pass": true + }, + "gates": [ + { + "rule": { + "field": "all_skipped_second_pass", + "op": ">=", + "min": 1, + "why": "second pass of identical lines must be 100% idempotent skips (boolean true coerced to 1)" + }, + "value": true, + "pass": true + } + ] + }, + "bulk-get": { + "result": { + "k": 30, + "workers": 8, + "serial": { + "n": 5, + "min_ms": 1520.47, + "median_ms": 1606.86, + "p90_ms": 1805.72, + "p95_ms": 1835.83, + "max_ms": 1865.94, + "mean_ms": 1658.6, + "warmup": { + "n": 1, + "min_ms": 1601.2, + "median_ms": 1601.2, + "p90_ms": 1601.2, + "p95_ms": 1601.2, + "max_ms": 1601.2, + "mean_ms": 1601.2 + } + }, + "concurrent": { + "n": 5, + "min_ms": 340.31, + "median_ms": 356.48, + "p90_ms": 358.82, + "p95_ms": 359.32, + "max_ms": 359.83, + "mean_ms": 351.42, + "warmup": { + "n": 2, + "min_ms": 353.43, + "median_ms": 434.94, + "p90_ms": 500.15, + "p95_ms": 508.3, + "max_ms": 516.45, + "mean_ms": 434.94 + } + }, + "speedup_ratio": 4.51 + }, + "gates": [ + { + "rule": { + "field": "speedup_ratio", + "op": ">=", + "min": 1.5, + "why": "controlled-K concurrent vs serial; lower bar than full-vault since K is small" + }, + "value": 4.51, + "pass": true + } + ] + }, + "bulk-put": { + "result": { + "k": 30, + "per_put": { + "n": 30, + "min_ms": 100.09, + "median_ms": 114.98, + "p90_ms": 212.46, + "p95_ms": 236.18, + "max_ms": 330.77, + "mean_ms": 143.25 + }, + "total_ms": 4297.5, + "puts_per_sec": 7.0 + }, + "gates": [] + }, + "cache": { + "result": { + "requested": 60, + "unique": 20, + "returned": 20, + "deduped": true, + "elapsed_ms": 307.13 + }, + "gates": [ + { + "rule": { + "field": "deduped", + "op": ">=", + "min": 1, + "why": "read_many must collapse a 3x-duplicated path list to the unique set" + }, + "value": true, + "pass": true + } + ] + }, + "expand-graph": { + "result": { + "subjects": [ + { + "label": "hub-echo", + "query": "echo", + "neighbours": 120, + "serial_ms": 2227.67, + "concurrent_ms": 642.2, + "speedup": 3.47, + "ranking_identical": true, + "scores_identical": true + }, + { + "label": "hub-prefs", + "query": "operator preferences", + "neighbours": 126, + "serial_ms": 2910.46, + "concurrent_ms": 698.8, + "speedup": 4.16, + "ranking_identical": true, + "scores_identical": true + } + ], + "min_speedup": 3.47, + "all_ranking_identical": true, + "all_scores_identical": true + }, + "gates": [ + { + "rule": { + "field": "min_speedup", + "op": ">=", + "min": 2.0, + "why": "recall()'s graph layer must fetch each BFS hop concurrently (read_many), not serially per node; <2x means expand_graph regressed to the pre-fix serial walk" + }, + "value": 3.47, + "pass": true + }, + { + "rule": { + "field": "all_ranking_identical", + "op": ">=", + "min": 1, + "why": "the concurrent expansion must return the same ranked neighbours as the serial reference" + }, + "value": true, + "pass": true + }, + { + "rule": { + "field": "all_scores_identical", + "op": ">=", + "min": 1, + "why": "decayed scores must match the serial reference to within float tolerance" + }, + "value": true, + "pass": true + } + ] + }, + "index": { + "result": { + "script": "sweep.py", + "elapsed_ms": 4629.0, + "exit_code": 0, + "ok": true, + "stderr_tail": [] + }, + "gates": [] + }, + "lint": { + "result": { + "script": "vault_lint.py", + "elapsed_ms": 5126.4, + "exit_code": 1, + "ok": true, + "stderr_tail": [] + }, + "gates": [] + }, + "lock": { + "result": { + "acquire_free_ms": 159.18, + "acquire_free_rc": 0, + "contended_ms": 57.89, + "contended_rc_expected_75": 75, + "release_ms": 132.92, + "ok": true + }, + "gates": [ + { + "rule": { + "field": "ok", + "op": ">=", + "min": 1, + "why": "free acquire returns 0 and a contended acquire fast-fails with exit 75" + }, + "value": true, + "pass": true + } + ] + }, + "pool-warmup": { + "result": { + "probe": "_agent/echo-vault.md", + "cold_ms": 163.25, + "warm": { + "n": 12, + "min_ms": 45.64, + "median_ms": 50.29, + "p90_ms": 52.51, + "p95_ms": 53.79, + "max_ms": 55.28, + "mean_ms": 50.26, + "warmup": { + "n": 1, + "min_ms": 51.68, + "median_ms": 51.68, + "p90_ms": 51.68, + "p95_ms": 51.68, + "max_ms": 51.68, + "mean_ms": 51.68 + } + }, + "cold_over_warm_ratio": 3.25 + }, + "gates": [ + { + "rule": { + "field": "cold_over_warm_ratio", + "op": ">=", + "min": 1.5, + "why": "cold request must be meaningfully slower than warm \u2014 proves keep-alive is reusing the connection; a ratio near 1.0 means every request is re-handshaking (the pre-1.1.0 bug)" + }, + "value": 3.25, + "pass": true + } + ] + }, + "queue": { + "result": { + "k": 10, + "enqueue": { + "n": 10, + "min_ms": 0.07, + "median_ms": 0.08, + "p90_ms": 0.18, + "p95_ms": 0.2, + "max_ms": 0.22, + "mean_ms": 0.1 + }, + "pending_after_enqueue": 10, + "note": "flush replay intentionally not run against the live vault; enqueue path timed only. See README queue caveat." + }, + "gates": [] + }, + "read-full": { + "result": { + "note_count": 197, + "workers": 8, + "serial": { + "n": 1, + "min_ms": 11376.08, + "median_ms": 11376.08, + "p90_ms": 11376.08, + "p95_ms": 11376.08, + "max_ms": 11376.08, + "mean_ms": 11376.08 + }, + "concurrent": { + "n": 4, + "min_ms": 1559.79, + "median_ms": 1595.31, + "p90_ms": 1740.92, + "p95_ms": 1768.05, + "max_ms": 1795.17, + "mean_ms": 1636.4, + "warmup": { + "n": 2, + "min_ms": 1641.35, + "median_ms": 1652.91, + "p90_ms": 1662.17, + "p95_ms": 1663.32, + "max_ms": 1664.48, + "mean_ms": 1652.91 + } + }, + "speedup_ratio": 7.13, + "notes_per_sec_concurrent": 123.5 + }, + "gates": [ + { + "rule": { + "field": "speedup_ratio", + "op": ">=", + "min": 1.8, + "why": "concurrent read_many must beat serial by the 1.1.0 concurrency margin; <1.8 means concurrency regressed" + }, + "value": 7.13, + "pass": true + } + ] + }, + "resolve": { + "result": { + "cases": [ + { + "mention": "echo", + "expect": "exact", + "expect_slug": "echo", + "resolved_slug": "echo", + "candidate_slugs": [ + "2026-06-19-goldbrain-full-echo-architect", + "echo", + "echo-memory-codex-plugin", + "echo-plugin-build", + "echo-skill-improvements" + ], + "verdict": "PASS", + "resolve_timing": { + "n": 6, + "min_ms": 0.0, + "median_ms": 0.0, + "p90_ms": 0.0, + "p95_ms": 0.0, + "max_ms": 0.0, + "mean_ms": 0.0, + "warmup": { + "n": 2, + "min_ms": 0.01, + "median_ms": 0.1, + "p90_ms": 0.17, + "p95_ms": 0.18, + "max_ms": 0.19, + "mean_ms": 0.1 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 1.35, + "median_ms": 1.49, + "p90_ms": 1.67, + "p95_ms": 1.68, + "max_ms": 1.68, + "mean_ms": 1.51, + "warmup": { + "n": 2, + "min_ms": 1.82, + "median_ms": 1.83, + "p90_ms": 1.84, + "p95_ms": 1.84, + "max_ms": 1.85, + "mean_ms": 1.83 + } + } + }, + { + "mention": "echo memory", + "expect": "candidates", + "expect_slug": "echo", + "resolved_slug": "echo", + "candidate_slugs": [ + "echo", + "echo-memory-codex-plugin", + "echo-plugin-build", + "echo-skill-improvements", + "2026-06-19-goldbrain-full-echo-architect" + ], + "verdict": "INFO", + "resolve_timing": { + "n": 6, + "min_ms": 0.55, + "median_ms": 0.56, + "p90_ms": 0.57, + "p95_ms": 0.58, + "max_ms": 0.58, + "mean_ms": 0.56, + "warmup": { + "n": 2, + "min_ms": 0.61, + "median_ms": 0.61, + "p90_ms": 0.61, + "p95_ms": 0.61, + "max_ms": 0.61, + "mean_ms": 0.61 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.97, + "median_ms": 0.99, + "p90_ms": 1.06, + "p95_ms": 1.09, + "max_ms": 1.11, + "mean_ms": 1.01, + "warmup": { + "n": 2, + "min_ms": 1.15, + "median_ms": 1.18, + "p90_ms": 1.2, + "p95_ms": 1.21, + "max_ms": 1.21, + "mean_ms": 1.18 + } + } + }, + { + "mention": "ECHO plugin", + "expect": "candidates", + "expect_slug": "echo", + "resolved_slug": "echo", + "candidate_slugs": [ + "echo", + "echo-memory-codex-plugin", + "echo-plugin-build", + "2026-06-19-goldbrain-full-echo-architect", + "alabama-wisp-brand-docs" + ], + "verdict": "INFO", + "resolve_timing": { + "n": 6, + "min_ms": 0.41, + "median_ms": 0.41, + "p90_ms": 0.42, + "p95_ms": 0.42, + "max_ms": 0.42, + "mean_ms": 0.41, + "warmup": { + "n": 2, + "min_ms": 0.42, + "median_ms": 0.45, + "p90_ms": 0.47, + "p95_ms": 0.48, + "max_ms": 0.48, + "mean_ms": 0.45 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.79, + "median_ms": 0.8, + "p90_ms": 0.88, + "p95_ms": 0.9, + "max_ms": 0.92, + "mean_ms": 0.83, + "warmup": { + "n": 2, + "min_ms": 0.89, + "median_ms": 0.89, + "p90_ms": 0.9, + "p95_ms": 0.9, + "max_ms": 0.9, + "mean_ms": 0.89 + } + } + }, + { + "mention": "goldbrain", + "expect": "exact", + "expect_slug": "goldbrain", + "resolved_slug": "goldbrain", + "candidate_slugs": [ + "2026-06-19-goldbrain-full-echo-architect", + "goldbrain" + ], + "verdict": "PASS", + "resolve_timing": { + "n": 6, + "min_ms": 0.0, + "median_ms": 0.0, + "p90_ms": 0.0, + "p95_ms": 0.0, + "max_ms": 0.0, + "mean_ms": 0.0, + "warmup": { + "n": 2, + "min_ms": 0.0, + "median_ms": 0.0, + "p90_ms": 0.0, + "p95_ms": 0.0, + "max_ms": 0.0, + "mean_ms": 0.0 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.73, + "median_ms": 0.75, + "p90_ms": 0.85, + "p95_ms": 0.88, + "max_ms": 0.91, + "mean_ms": 0.78, + "warmup": { + "n": 2, + "min_ms": 0.8, + "median_ms": 0.81, + "p90_ms": 0.82, + "p95_ms": 0.82, + "max_ms": 0.82, + "mean_ms": 0.81 + } + } + }, + { + "mention": "jason stedwell", + "expect": "exact", + "expect_slug": "jason-stedwell", + "resolved_slug": "jason-stedwell", + "candidate_slugs": [ + "jason-stedwell", + "jason-mcp-gateway" + ], + "verdict": "PASS", + "resolve_timing": { + "n": 6, + "min_ms": 0.0, + "median_ms": 0.0, + "p90_ms": 0.0, + "p95_ms": 0.0, + "max_ms": 0.0, + "mean_ms": 0.0, + "warmup": { + "n": 2, + "min_ms": 0.0, + "median_ms": 0.0, + "p90_ms": 0.0, + "p95_ms": 0.0, + "max_ms": 0.0, + "mean_ms": 0.0 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.69, + "median_ms": 0.72, + "p90_ms": 0.75, + "p95_ms": 0.75, + "max_ms": 0.75, + "mean_ms": 0.72, + "warmup": { + "n": 2, + "min_ms": 0.74, + "median_ms": 0.74, + "p90_ms": 0.75, + "p95_ms": 0.75, + "max_ms": 0.75, + "mean_ms": 0.74 + } + } + }, + { + "mention": "jason", + "expect": "candidates", + "expect_slug": "jason-stedwell", + "resolved_slug": "jason-stedwell", + "candidate_slugs": [ + "jason-mcp-gateway", + "jason-stedwell" + ], + "verdict": "INFO", + "resolve_timing": { + "n": 6, + "min_ms": 0.61, + "median_ms": 0.65, + "p90_ms": 0.66, + "p95_ms": 0.67, + "max_ms": 0.67, + "mean_ms": 0.64, + "warmup": { + "n": 2, + "min_ms": 0.66, + "median_ms": 0.68, + "p90_ms": 0.69, + "p95_ms": 0.69, + "max_ms": 0.69, + "mean_ms": 0.68 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.63, + "median_ms": 0.63, + "p90_ms": 0.65, + "p95_ms": 0.65, + "max_ms": 0.66, + "mean_ms": 0.64, + "warmup": { + "n": 2, + "min_ms": 0.68, + "median_ms": 0.69, + "p90_ms": 0.7, + "p95_ms": 0.7, + "max_ms": 0.7, + "mean_ms": 0.69 + } + } + }, + { + "mention": "zzqx nonexistent entity", + "expect": "miss", + "expect_slug": null, + "resolved_slug": null, + "candidate_slugs": [], + "verdict": "PASS", + "resolve_timing": { + "n": 6, + "min_ms": 0.62, + "median_ms": 0.63, + "p90_ms": 0.67, + "p95_ms": 0.67, + "max_ms": 0.68, + "mean_ms": 0.64, + "warmup": { + "n": 2, + "min_ms": 0.66, + "median_ms": 0.67, + "p90_ms": 0.67, + "p95_ms": 0.67, + "max_ms": 0.67, + "mean_ms": 0.67 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.61, + "median_ms": 0.62, + "p90_ms": 0.65, + "p95_ms": 0.65, + "max_ms": 0.66, + "mean_ms": 0.62, + "warmup": { + "n": 2, + "min_ms": 0.62, + "median_ms": 0.63, + "p90_ms": 0.64, + "p95_ms": 0.64, + "max_ms": 0.64, + "mean_ms": 0.63 + } + } + } + ], + "passes": 4, + "needs_tuning": 3, + "note": "verdicts are PASS/INFO only; promote to FAIL once the table is confirmed against the live vault" + }, + "gates": [] + }, + "soak": { + "result": { + "seconds": 22, + "iterations": 15, + "errors": 0, + "per_pass": { + "n": 15, + "min_ms": 1472.88, + "median_ms": 1551.67, + "p90_ms": 1648.45, + "p95_ms": 1702.66, + "max_ms": 1820.58, + "mean_ms": 1569.81 + }, + "early_mean_ms": 1575.63, + "late_mean_ms": 1564.72, + "drift_ratio_late_over_early": 0.99 + }, + "gates": [ + { + "rule": { + "field": "errors", + "op": "<=", + "min": 0, + "why": "no read errors across the soak window \u2014 connection leaks/pool exhaustion would surface here" + }, + "value": 0, + "pass": true + }, + { + "rule": { + "field": "drift_ratio_late_over_early", + "op": "<=", + "min": 1.5, + "why": "late passes must not be much slower than early ones; >1.5 suggests a leak or degrading pool" + }, + "value": 0.99, + "pass": true + } + ] + }, + "subject-pull": { + "result": { + "subjects": [ + { + "label": "apta", + "query": "APTA", + "resolve_floor": { + "n": 1, + "min_ms": 0.75, + "median_ms": 0.75, + "p90_ms": 0.75, + "p95_ms": 0.75, + "max_ms": 0.75, + "mean_ms": 0.75 + }, + "search_only": { + "n": 1, + "min_ms": 61.87, + "median_ms": 61.87, + "p90_ms": 61.87, + "p95_ms": 61.87, + "max_ms": 61.87, + "mean_ms": 61.87 + }, + "recall_total": { + "n": 1, + "min_ms": 2231.31, + "median_ms": 2231.31, + "p90_ms": 2231.31, + "p95_ms": 2231.31, + "max_ms": 2231.31, + "mean_ms": 2231.31 + } + }, + { + "label": "capmetro", + "query": "CapMetro", + "resolve_floor": { + "n": 1, + "min_ms": 0.03, + "median_ms": 0.03, + "p90_ms": 0.03, + "p95_ms": 0.03, + "max_ms": 0.03, + "mean_ms": 0.03 + }, + "search_only": { + "n": 1, + "min_ms": 57.73, + "median_ms": 57.73, + "p90_ms": 57.73, + "p95_ms": 57.73, + "max_ms": 57.73, + "mean_ms": 57.73 + }, + "recall_total": { + "n": 1, + "min_ms": 2116.06, + "median_ms": 2116.06, + "p90_ms": 2116.06, + "p95_ms": 2116.06, + "max_ms": 2116.06, + "mean_ms": 2116.06 + } + }, + { + "label": "hub-echo", + "query": "echo", + "resolve_floor": { + "n": 1, + "min_ms": 0.03, + "median_ms": 0.03, + "p90_ms": 0.03, + "p95_ms": 0.03, + "max_ms": 0.03, + "mean_ms": 0.03 + }, + "search_only": { + "n": 1, + "min_ms": 70.56, + "median_ms": 70.56, + "p90_ms": 70.56, + "p95_ms": 70.56, + "max_ms": 70.56, + "mean_ms": 70.56 + }, + "recall_total": { + "n": 1, + "min_ms": 2403.62, + "median_ms": 2403.62, + "p90_ms": 2403.62, + "p95_ms": 2403.62, + "max_ms": 2403.62, + "mean_ms": 2403.62 + } + }, + { + "label": "hub-prefs", + "query": "operator preferences", + "resolve_floor": { + "n": 1, + "min_ms": 0.04, + "median_ms": 0.04, + "p90_ms": 0.04, + "p95_ms": 0.04, + "max_ms": 0.04, + "mean_ms": 0.04 + }, + "search_only": { + "n": 1, + "min_ms": 2011.13, + "median_ms": 2011.13, + "p90_ms": 2011.13, + "p95_ms": 2011.13, + "max_ms": 2011.13, + "mean_ms": 2011.13 + }, + "recall_total": { + "n": 1, + "min_ms": 2994.84, + "median_ms": 2994.84, + "p90_ms": 2994.84, + "p95_ms": 2994.84, + "max_ms": 2994.84, + "mean_ms": 2994.84 + } + }, + { + "label": "leaf-rivnut", + "query": "rivnut torque spec", + "resolve_floor": { + "n": 1, + "min_ms": 2.15, + "median_ms": 2.15, + "p90_ms": 2.15, + "p95_ms": 2.15, + "max_ms": 2.15, + "mean_ms": 2.15 + }, + "search_only": { + "n": 1, + "min_ms": 2013.58, + "median_ms": 2013.58, + "p90_ms": 2013.58, + "p95_ms": 2013.58, + "max_ms": 2013.58, + "mean_ms": 2013.58 + }, + "recall_total": { + "n": 1, + "min_ms": 2371.32, + "median_ms": 2371.32, + "p90_ms": 2371.32, + "p95_ms": 2371.32, + "max_ms": 2371.32, + "mean_ms": 2371.32 + } + } + ] + }, + "gates": [] + }, + "worker-sweep": { + "result": { + "note_count": 197, + "sweep": [ + { + "workers": 1, + "median_ms": 11917.49, + "p95_ms": 11917.49 + }, + { + "workers": 2, + "median_ms": 5740.0, + "p95_ms": 5740.0 + }, + { + "workers": 4, + "median_ms": 3034.3, + "p95_ms": 3034.3 + }, + { + "workers": 8, + "median_ms": 1583.83, + "p95_ms": 1583.83 + }, + { + "workers": 16, + "median_ms": 1277.77, + "p95_ms": 1277.77 + } + ], + "knee_workers": 16, + "default_workers": 8 + }, + "gates": [] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-bulk-append.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-bulk-append.json new file mode 100644 index 0000000..f0bfaf1 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-bulk-append.json @@ -0,0 +1,55 @@ +{ + "env": { + "date": "2026-06-23T20:22:00", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": null, + "count_k": 30 + }, + "run_id": "20260623-202200-882c82", + "metrics": { + "bulk-append": { + "result": { + "k": 30, + "append_write": { + "n": 30, + "min_ms": 87.11, + "median_ms": 104.08, + "p90_ms": 190.63, + "p95_ms": 198.89, + "max_ms": 241.91, + "mean_ms": 117.22 + }, + "append_skip": { + "n": 30, + "min_ms": 47.57, + "median_ms": 51.42, + "p90_ms": 59.82, + "p95_ms": 61.57, + "max_ms": 73.49, + "mean_ms": 53.22 + }, + "skips_confirmed": 30, + "all_skipped_second_pass": true + }, + "gates": [ + { + "rule": { + "field": "all_skipped_second_pass", + "op": ">=", + "min": 1, + "why": "second pass of identical lines must be 100% idempotent skips (boolean true coerced to 1)" + }, + "value": true, + "pass": true + } + ] + } + }, + "cleanup": { + "deleted": 1 + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-bulk-get.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-bulk-get.json new file mode 100644 index 0000000..39b999a --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-bulk-get.json @@ -0,0 +1,73 @@ +{ + "env": { + "date": "2026-06-23T20:21:36", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": 5, + "count_k": 30 + }, + "run_id": "20260623-202136-f2f4dc", + "metrics": { + "bulk-get": { + "result": { + "k": 30, + "workers": 8, + "serial": { + "n": 5, + "min_ms": 1520.47, + "median_ms": 1606.86, + "p90_ms": 1805.72, + "p95_ms": 1835.83, + "max_ms": 1865.94, + "mean_ms": 1658.6, + "warmup": { + "n": 1, + "min_ms": 1601.2, + "median_ms": 1601.2, + "p90_ms": 1601.2, + "p95_ms": 1601.2, + "max_ms": 1601.2, + "mean_ms": 1601.2 + } + }, + "concurrent": { + "n": 5, + "min_ms": 340.31, + "median_ms": 356.48, + "p90_ms": 358.82, + "p95_ms": 359.32, + "max_ms": 359.83, + "mean_ms": 351.42, + "warmup": { + "n": 2, + "min_ms": 353.43, + "median_ms": 434.94, + "p90_ms": 500.15, + "p95_ms": 508.3, + "max_ms": 516.45, + "mean_ms": 434.94 + } + }, + "speedup_ratio": 4.51 + }, + "gates": [ + { + "rule": { + "field": "speedup_ratio", + "op": ">=", + "min": 1.5, + "why": "controlled-K concurrent vs serial; lower bar than full-vault since K is small" + }, + "value": 4.51, + "pass": true + } + ] + } + }, + "cleanup": { + "deleted": 30 + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-bulk-put.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-bulk-put.json new file mode 100644 index 0000000..d4bd936 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-bulk-put.json @@ -0,0 +1,35 @@ +{ + "env": { + "date": "2026-06-23T20:21:21", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": null, + "count_k": 30 + }, + "run_id": "20260623-202121-23b5ac", + "metrics": { + "bulk-put": { + "result": { + "k": 30, + "per_put": { + "n": 30, + "min_ms": 100.09, + "median_ms": 114.98, + "p90_ms": 212.46, + "p95_ms": 236.18, + "max_ms": 330.77, + "mean_ms": 143.25 + }, + "total_ms": 4297.5, + "puts_per_sec": 7.0 + }, + "gates": [] + } + }, + "cleanup": { + "deleted": 30 + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-cache.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-cache.json new file mode 100644 index 0000000..36d0b03 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-cache.json @@ -0,0 +1,36 @@ +{ + "env": { + "date": "2026-06-23T20:12:16", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": null, + "count_k": 50 + }, + "run_id": "20260623-201216-e77971", + "metrics": { + "cache": { + "result": { + "requested": 60, + "unique": 20, + "returned": 20, + "deduped": true, + "elapsed_ms": 307.13 + }, + "gates": [ + { + "rule": { + "field": "deduped", + "op": ">=", + "min": 1, + "why": "read_many must collapse a 3x-duplicated path list to the unique set" + }, + "value": true, + "pass": true + } + ] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-expand-graph.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-expand-graph.json new file mode 100644 index 0000000..279e531 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-expand-graph.json @@ -0,0 +1,76 @@ +{ + "env": { + "date": "2026-06-23T21:22:29", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": 3, + "count_k": 50 + }, + "run_id": "20260623-212229-8c3ce2", + "metrics": { + "expand-graph": { + "result": { + "subjects": [ + { + "label": "hub-echo", + "query": "echo", + "neighbours": 120, + "serial_ms": 2227.67, + "concurrent_ms": 642.2, + "speedup": 3.47, + "ranking_identical": true, + "scores_identical": true + }, + { + "label": "hub-prefs", + "query": "operator preferences", + "neighbours": 126, + "serial_ms": 2910.46, + "concurrent_ms": 698.8, + "speedup": 4.16, + "ranking_identical": true, + "scores_identical": true + } + ], + "min_speedup": 3.47, + "all_ranking_identical": true, + "all_scores_identical": true + }, + "gates": [ + { + "rule": { + "field": "min_speedup", + "op": ">=", + "min": 2.0, + "why": "recall()'s graph layer must fetch each BFS hop concurrently (read_many), not serially per node; <2x means expand_graph regressed to the pre-fix serial walk" + }, + "value": 3.47, + "pass": true + }, + { + "rule": { + "field": "all_ranking_identical", + "op": ">=", + "min": 1, + "why": "the concurrent expansion must return the same ranked neighbours as the serial reference" + }, + "value": true, + "pass": true + }, + { + "rule": { + "field": "all_scores_identical", + "op": ">=", + "min": 1, + "why": "decayed scores must match the serial reference to within float tolerance" + }, + "value": true, + "pass": true + } + ] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-index.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-index.json new file mode 100644 index 0000000..4fb8f95 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-index.json @@ -0,0 +1,25 @@ +{ + "env": { + "date": "2026-06-23T20:20:57", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": null, + "count_k": 50 + }, + "run_id": "20260623-202057-c41a52", + "metrics": { + "index": { + "result": { + "script": "sweep.py", + "elapsed_ms": 4629.0, + "exit_code": 0, + "ok": true, + "stderr_tail": [] + }, + "gates": [] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-lint.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-lint.json new file mode 100644 index 0000000..a22eecb --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-lint.json @@ -0,0 +1,25 @@ +{ + "env": { + "date": "2026-06-23T20:21:01", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": null, + "count_k": 50 + }, + "run_id": "20260623-202101-569382", + "metrics": { + "lint": { + "result": { + "script": "vault_lint.py", + "elapsed_ms": 5126.4, + "exit_code": 1, + "ok": true, + "stderr_tail": [] + }, + "gates": [] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-lock.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-lock.json new file mode 100644 index 0000000..68d6008 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-lock.json @@ -0,0 +1,40 @@ +{ + "env": { + "date": "2026-06-23T20:21:28", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": null, + "count_k": 50 + }, + "run_id": "20260623-202128-53b113", + "metrics": { + "lock": { + "result": { + "acquire_free_ms": 159.18, + "acquire_free_rc": 0, + "contended_ms": 57.89, + "contended_rc_expected_75": 75, + "release_ms": 132.92, + "ok": true + }, + "gates": [ + { + "rule": { + "field": "ok", + "op": ">=", + "min": 1, + "why": "free acquire returns 0 and a contended acquire fast-fails with exit 75" + }, + "value": true, + "pass": true + } + ] + } + }, + "cleanup": { + "deleted": 0 + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-pool-warmup.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-pool-warmup.json new file mode 100644 index 0000000..4ecdf96 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-pool-warmup.json @@ -0,0 +1,52 @@ +{ + "env": { + "date": "2026-06-23T20:12:12", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": 12, + "count_k": 50 + }, + "run_id": "20260623-201212-30c2b8", + "metrics": { + "pool-warmup": { + "result": { + "probe": "_agent/echo-vault.md", + "cold_ms": 163.25, + "warm": { + "n": 12, + "min_ms": 45.64, + "median_ms": 50.29, + "p90_ms": 52.51, + "p95_ms": 53.79, + "max_ms": 55.28, + "mean_ms": 50.26, + "warmup": { + "n": 1, + "min_ms": 51.68, + "median_ms": 51.68, + "p90_ms": 51.68, + "p95_ms": 51.68, + "max_ms": 51.68, + "mean_ms": 51.68 + } + }, + "cold_over_warm_ratio": 3.25 + }, + "gates": [ + { + "rule": { + "field": "cold_over_warm_ratio", + "op": ">=", + "min": 1.5, + "why": "cold request must be meaningfully slower than warm \u2014 proves keep-alive is reusing the connection; a ratio near 1.0 means every request is re-handshaking (the pre-1.1.0 bug)" + }, + "value": 3.25, + "pass": true + } + ] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-queue.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-queue.json new file mode 100644 index 0000000..e76007c --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-queue.json @@ -0,0 +1,32 @@ +{ + "env": { + "date": "2026-06-23T20:22:05", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": null, + "count_k": 10 + }, + "run_id": "20260623-202205-378725", + "metrics": { + "queue": { + "result": { + "k": 10, + "enqueue": { + "n": 10, + "min_ms": 0.07, + "median_ms": 0.08, + "p90_ms": 0.18, + "p95_ms": 0.2, + "max_ms": 0.22, + "mean_ms": 0.1 + }, + "pending_after_enqueue": 10, + "note": "flush replay intentionally not run against the live vault; enqueue path timed only. See README queue caveat." + }, + "gates": [] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-read-full.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-read-full.json new file mode 100644 index 0000000..2d6b28b --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-read-full.json @@ -0,0 +1,62 @@ +{ + "env": { + "date": "2026-06-23T20:17:09", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": 4, + "count_k": 50 + }, + "run_id": "20260623-201709-b08a2e", + "metrics": { + "read-full": { + "result": { + "note_count": 197, + "workers": 8, + "serial": { + "n": 1, + "min_ms": 11376.08, + "median_ms": 11376.08, + "p90_ms": 11376.08, + "p95_ms": 11376.08, + "max_ms": 11376.08, + "mean_ms": 11376.08 + }, + "concurrent": { + "n": 4, + "min_ms": 1559.79, + "median_ms": 1595.31, + "p90_ms": 1740.92, + "p95_ms": 1768.05, + "max_ms": 1795.17, + "mean_ms": 1636.4, + "warmup": { + "n": 2, + "min_ms": 1641.35, + "median_ms": 1652.91, + "p90_ms": 1662.17, + "p95_ms": 1663.32, + "max_ms": 1664.48, + "mean_ms": 1652.91 + } + }, + "speedup_ratio": 7.13, + "notes_per_sec_concurrent": 123.5 + }, + "gates": [ + { + "rule": { + "field": "speedup_ratio", + "op": ">=", + "min": 1.8, + "why": "concurrent read_many must beat serial by the 1.1.0 concurrency margin; <1.8 means concurrency regressed" + }, + "value": 7.13, + "pass": true + } + ] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-resolve.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-resolve.json new file mode 100644 index 0000000..ebbf720 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-resolve.json @@ -0,0 +1,360 @@ +{ + "env": { + "date": "2026-06-23T20:12:19", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": 6, + "count_k": 50 + }, + "run_id": "20260623-201219-819b5e", + "metrics": { + "resolve": { + "result": { + "cases": [ + { + "mention": "echo", + "expect": "exact", + "expect_slug": "echo", + "resolved_slug": "echo", + "candidate_slugs": [ + "2026-06-19-goldbrain-full-echo-architect", + "echo", + "echo-memory-codex-plugin", + "echo-plugin-build", + "echo-skill-improvements" + ], + "verdict": "PASS", + "resolve_timing": { + "n": 6, + "min_ms": 0.0, + "median_ms": 0.0, + "p90_ms": 0.0, + "p95_ms": 0.0, + "max_ms": 0.0, + "mean_ms": 0.0, + "warmup": { + "n": 2, + "min_ms": 0.01, + "median_ms": 0.1, + "p90_ms": 0.17, + "p95_ms": 0.18, + "max_ms": 0.19, + "mean_ms": 0.1 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 1.35, + "median_ms": 1.49, + "p90_ms": 1.67, + "p95_ms": 1.68, + "max_ms": 1.68, + "mean_ms": 1.51, + "warmup": { + "n": 2, + "min_ms": 1.82, + "median_ms": 1.83, + "p90_ms": 1.84, + "p95_ms": 1.84, + "max_ms": 1.85, + "mean_ms": 1.83 + } + } + }, + { + "mention": "echo memory", + "expect": "candidates", + "expect_slug": "echo", + "resolved_slug": "echo", + "candidate_slugs": [ + "echo", + "echo-memory-codex-plugin", + "echo-plugin-build", + "echo-skill-improvements", + "2026-06-19-goldbrain-full-echo-architect" + ], + "verdict": "INFO", + "resolve_timing": { + "n": 6, + "min_ms": 0.55, + "median_ms": 0.56, + "p90_ms": 0.57, + "p95_ms": 0.58, + "max_ms": 0.58, + "mean_ms": 0.56, + "warmup": { + "n": 2, + "min_ms": 0.61, + "median_ms": 0.61, + "p90_ms": 0.61, + "p95_ms": 0.61, + "max_ms": 0.61, + "mean_ms": 0.61 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.97, + "median_ms": 0.99, + "p90_ms": 1.06, + "p95_ms": 1.09, + "max_ms": 1.11, + "mean_ms": 1.01, + "warmup": { + "n": 2, + "min_ms": 1.15, + "median_ms": 1.18, + "p90_ms": 1.2, + "p95_ms": 1.21, + "max_ms": 1.21, + "mean_ms": 1.18 + } + } + }, + { + "mention": "ECHO plugin", + "expect": "candidates", + "expect_slug": "echo", + "resolved_slug": "echo", + "candidate_slugs": [ + "echo", + "echo-memory-codex-plugin", + "echo-plugin-build", + "2026-06-19-goldbrain-full-echo-architect", + "alabama-wisp-brand-docs" + ], + "verdict": "INFO", + "resolve_timing": { + "n": 6, + "min_ms": 0.41, + "median_ms": 0.41, + "p90_ms": 0.42, + "p95_ms": 0.42, + "max_ms": 0.42, + "mean_ms": 0.41, + "warmup": { + "n": 2, + "min_ms": 0.42, + "median_ms": 0.45, + "p90_ms": 0.47, + "p95_ms": 0.48, + "max_ms": 0.48, + "mean_ms": 0.45 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.79, + "median_ms": 0.8, + "p90_ms": 0.88, + "p95_ms": 0.9, + "max_ms": 0.92, + "mean_ms": 0.83, + "warmup": { + "n": 2, + "min_ms": 0.89, + "median_ms": 0.89, + "p90_ms": 0.9, + "p95_ms": 0.9, + "max_ms": 0.9, + "mean_ms": 0.89 + } + } + }, + { + "mention": "goldbrain", + "expect": "exact", + "expect_slug": "goldbrain", + "resolved_slug": "goldbrain", + "candidate_slugs": [ + "2026-06-19-goldbrain-full-echo-architect", + "goldbrain" + ], + "verdict": "PASS", + "resolve_timing": { + "n": 6, + "min_ms": 0.0, + "median_ms": 0.0, + "p90_ms": 0.0, + "p95_ms": 0.0, + "max_ms": 0.0, + "mean_ms": 0.0, + "warmup": { + "n": 2, + "min_ms": 0.0, + "median_ms": 0.0, + "p90_ms": 0.0, + "p95_ms": 0.0, + "max_ms": 0.0, + "mean_ms": 0.0 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.73, + "median_ms": 0.75, + "p90_ms": 0.85, + "p95_ms": 0.88, + "max_ms": 0.91, + "mean_ms": 0.78, + "warmup": { + "n": 2, + "min_ms": 0.8, + "median_ms": 0.81, + "p90_ms": 0.82, + "p95_ms": 0.82, + "max_ms": 0.82, + "mean_ms": 0.81 + } + } + }, + { + "mention": "jason stedwell", + "expect": "exact", + "expect_slug": "jason-stedwell", + "resolved_slug": "jason-stedwell", + "candidate_slugs": [ + "jason-stedwell", + "jason-mcp-gateway" + ], + "verdict": "PASS", + "resolve_timing": { + "n": 6, + "min_ms": 0.0, + "median_ms": 0.0, + "p90_ms": 0.0, + "p95_ms": 0.0, + "max_ms": 0.0, + "mean_ms": 0.0, + "warmup": { + "n": 2, + "min_ms": 0.0, + "median_ms": 0.0, + "p90_ms": 0.0, + "p95_ms": 0.0, + "max_ms": 0.0, + "mean_ms": 0.0 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.69, + "median_ms": 0.72, + "p90_ms": 0.75, + "p95_ms": 0.75, + "max_ms": 0.75, + "mean_ms": 0.72, + "warmup": { + "n": 2, + "min_ms": 0.74, + "median_ms": 0.74, + "p90_ms": 0.75, + "p95_ms": 0.75, + "max_ms": 0.75, + "mean_ms": 0.74 + } + } + }, + { + "mention": "jason", + "expect": "candidates", + "expect_slug": "jason-stedwell", + "resolved_slug": "jason-stedwell", + "candidate_slugs": [ + "jason-mcp-gateway", + "jason-stedwell" + ], + "verdict": "INFO", + "resolve_timing": { + "n": 6, + "min_ms": 0.61, + "median_ms": 0.65, + "p90_ms": 0.66, + "p95_ms": 0.67, + "max_ms": 0.67, + "mean_ms": 0.64, + "warmup": { + "n": 2, + "min_ms": 0.66, + "median_ms": 0.68, + "p90_ms": 0.69, + "p95_ms": 0.69, + "max_ms": 0.69, + "mean_ms": 0.68 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.63, + "median_ms": 0.63, + "p90_ms": 0.65, + "p95_ms": 0.65, + "max_ms": 0.66, + "mean_ms": 0.64, + "warmup": { + "n": 2, + "min_ms": 0.68, + "median_ms": 0.69, + "p90_ms": 0.7, + "p95_ms": 0.7, + "max_ms": 0.7, + "mean_ms": 0.69 + } + } + }, + { + "mention": "zzqx nonexistent entity", + "expect": "miss", + "expect_slug": null, + "resolved_slug": null, + "candidate_slugs": [], + "verdict": "PASS", + "resolve_timing": { + "n": 6, + "min_ms": 0.62, + "median_ms": 0.63, + "p90_ms": 0.67, + "p95_ms": 0.67, + "max_ms": 0.68, + "mean_ms": 0.64, + "warmup": { + "n": 2, + "min_ms": 0.66, + "median_ms": 0.67, + "p90_ms": 0.67, + "p95_ms": 0.67, + "max_ms": 0.67, + "mean_ms": 0.67 + } + }, + "fuzzy_timing": { + "n": 6, + "min_ms": 0.61, + "median_ms": 0.62, + "p90_ms": 0.65, + "p95_ms": 0.65, + "max_ms": 0.66, + "mean_ms": 0.62, + "warmup": { + "n": 2, + "min_ms": 0.62, + "median_ms": 0.63, + "p90_ms": 0.64, + "p95_ms": 0.64, + "max_ms": 0.64, + "mean_ms": 0.63 + } + } + } + ], + "passes": 4, + "needs_tuning": 3, + "note": "verdicts are PASS/INFO only; promote to FAIL once the table is confirmed against the live vault" + }, + "gates": [] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-soak.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-soak.json new file mode 100644 index 0000000..40b2227 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-soak.json @@ -0,0 +1,56 @@ +{ + "env": { + "date": "2026-06-23T20:22:15", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": null, + "count_k": 50 + }, + "run_id": "20260623-202215-7b414f", + "metrics": { + "soak": { + "result": { + "seconds": 22, + "iterations": 15, + "errors": 0, + "per_pass": { + "n": 15, + "min_ms": 1472.88, + "median_ms": 1551.67, + "p90_ms": 1648.45, + "p95_ms": 1702.66, + "max_ms": 1820.58, + "mean_ms": 1569.81 + }, + "early_mean_ms": 1575.63, + "late_mean_ms": 1564.72, + "drift_ratio_late_over_early": 0.99 + }, + "gates": [ + { + "rule": { + "field": "errors", + "op": "<=", + "min": 0, + "why": "no read errors across the soak window \u2014 connection leaks/pool exhaustion would surface here" + }, + "value": 0, + "pass": true + }, + { + "rule": { + "field": "drift_ratio_late_over_early", + "op": "<=", + "min": 1.5, + "why": "late passes must not be much slower than early ones; >1.5 suggests a leak or degrading pool" + }, + "value": 0.99, + "pass": true + } + ] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-subject-pull-after.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-subject-pull-after.json new file mode 100644 index 0000000..470b741 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-subject-pull-after.json @@ -0,0 +1,177 @@ +{ + "env": { + "date": "2026-06-23T21:18:12", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": 1, + "count_k": 50 + }, + "run_id": "20260623-211812-ea4b36", + "metrics": { + "subject-pull": { + "result": { + "subjects": [ + { + "label": "apta", + "query": "APTA", + "resolve_floor": { + "n": 1, + "min_ms": 0.75, + "median_ms": 0.75, + "p90_ms": 0.75, + "p95_ms": 0.75, + "max_ms": 0.75, + "mean_ms": 0.75 + }, + "search_only": { + "n": 1, + "min_ms": 61.87, + "median_ms": 61.87, + "p90_ms": 61.87, + "p95_ms": 61.87, + "max_ms": 61.87, + "mean_ms": 61.87 + }, + "recall_total": { + "n": 1, + "min_ms": 2231.31, + "median_ms": 2231.31, + "p90_ms": 2231.31, + "p95_ms": 2231.31, + "max_ms": 2231.31, + "mean_ms": 2231.31 + } + }, + { + "label": "capmetro", + "query": "CapMetro", + "resolve_floor": { + "n": 1, + "min_ms": 0.03, + "median_ms": 0.03, + "p90_ms": 0.03, + "p95_ms": 0.03, + "max_ms": 0.03, + "mean_ms": 0.03 + }, + "search_only": { + "n": 1, + "min_ms": 57.73, + "median_ms": 57.73, + "p90_ms": 57.73, + "p95_ms": 57.73, + "max_ms": 57.73, + "mean_ms": 57.73 + }, + "recall_total": { + "n": 1, + "min_ms": 2116.06, + "median_ms": 2116.06, + "p90_ms": 2116.06, + "p95_ms": 2116.06, + "max_ms": 2116.06, + "mean_ms": 2116.06 + } + }, + { + "label": "hub-echo", + "query": "echo", + "resolve_floor": { + "n": 1, + "min_ms": 0.03, + "median_ms": 0.03, + "p90_ms": 0.03, + "p95_ms": 0.03, + "max_ms": 0.03, + "mean_ms": 0.03 + }, + "search_only": { + "n": 1, + "min_ms": 70.56, + "median_ms": 70.56, + "p90_ms": 70.56, + "p95_ms": 70.56, + "max_ms": 70.56, + "mean_ms": 70.56 + }, + "recall_total": { + "n": 1, + "min_ms": 2403.62, + "median_ms": 2403.62, + "p90_ms": 2403.62, + "p95_ms": 2403.62, + "max_ms": 2403.62, + "mean_ms": 2403.62 + } + }, + { + "label": "hub-prefs", + "query": "operator preferences", + "resolve_floor": { + "n": 1, + "min_ms": 0.04, + "median_ms": 0.04, + "p90_ms": 0.04, + "p95_ms": 0.04, + "max_ms": 0.04, + "mean_ms": 0.04 + }, + "search_only": { + "n": 1, + "min_ms": 2011.13, + "median_ms": 2011.13, + "p90_ms": 2011.13, + "p95_ms": 2011.13, + "max_ms": 2011.13, + "mean_ms": 2011.13 + }, + "recall_total": { + "n": 1, + "min_ms": 2994.84, + "median_ms": 2994.84, + "p90_ms": 2994.84, + "p95_ms": 2994.84, + "max_ms": 2994.84, + "mean_ms": 2994.84 + } + }, + { + "label": "leaf-rivnut", + "query": "rivnut torque spec", + "resolve_floor": { + "n": 1, + "min_ms": 2.15, + "median_ms": 2.15, + "p90_ms": 2.15, + "p95_ms": 2.15, + "max_ms": 2.15, + "mean_ms": 2.15 + }, + "search_only": { + "n": 1, + "min_ms": 2013.58, + "median_ms": 2013.58, + "p90_ms": 2013.58, + "p95_ms": 2013.58, + "max_ms": 2013.58, + "mean_ms": 2013.58 + }, + "recall_total": { + "n": 1, + "min_ms": 2371.32, + "median_ms": 2371.32, + "p90_ms": 2371.32, + "p95_ms": 2371.32, + "max_ms": 2371.32, + "mean_ms": 2371.32 + } + } + ] + }, + "gates": [] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-subject-pull.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-subject-pull.json new file mode 100644 index 0000000..f78e55e --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-subject-pull.json @@ -0,0 +1,177 @@ +{ + "env": { + "date": "2026-06-23T20:20:23", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": 1, + "count_k": 50 + }, + "run_id": "20260623-202023-274ddd", + "metrics": { + "subject-pull": { + "result": { + "subjects": [ + { + "label": "apta", + "query": "APTA", + "resolve_floor": { + "n": 1, + "min_ms": 0.97, + "median_ms": 0.97, + "p90_ms": 0.97, + "p95_ms": 0.97, + "max_ms": 0.97, + "mean_ms": 0.97 + }, + "search_only": { + "n": 1, + "min_ms": 71.55, + "median_ms": 71.55, + "p90_ms": 71.55, + "p95_ms": 71.55, + "max_ms": 71.55, + "mean_ms": 71.55 + }, + "recall_total": { + "n": 1, + "min_ms": 3836.47, + "median_ms": 3836.47, + "p90_ms": 3836.47, + "p95_ms": 3836.47, + "max_ms": 3836.47, + "mean_ms": 3836.47 + } + }, + { + "label": "capmetro", + "query": "CapMetro", + "resolve_floor": { + "n": 1, + "min_ms": 0.05, + "median_ms": 0.05, + "p90_ms": 0.05, + "p95_ms": 0.05, + "max_ms": 0.05, + "mean_ms": 0.05 + }, + "search_only": { + "n": 1, + "min_ms": 57.66, + "median_ms": 57.66, + "p90_ms": 57.66, + "p95_ms": 57.66, + "max_ms": 57.66, + "mean_ms": 57.66 + }, + "recall_total": { + "n": 1, + "min_ms": 2800.54, + "median_ms": 2800.54, + "p90_ms": 2800.54, + "p95_ms": 2800.54, + "max_ms": 2800.54, + "mean_ms": 2800.54 + } + }, + { + "label": "hub-echo", + "query": "echo", + "resolve_floor": { + "n": 1, + "min_ms": 0.02, + "median_ms": 0.02, + "p90_ms": 0.02, + "p95_ms": 0.02, + "max_ms": 0.02, + "mean_ms": 0.02 + }, + "search_only": { + "n": 1, + "min_ms": 67.39, + "median_ms": 67.39, + "p90_ms": 67.39, + "p95_ms": 67.39, + "max_ms": 67.39, + "mean_ms": 67.39 + }, + "recall_total": { + "n": 1, + "min_ms": 4114.19, + "median_ms": 4114.19, + "p90_ms": 4114.19, + "p95_ms": 4114.19, + "max_ms": 4114.19, + "mean_ms": 4114.19 + } + }, + { + "label": "hub-prefs", + "query": "operator preferences", + "resolve_floor": { + "n": 1, + "min_ms": 0.04, + "median_ms": 0.04, + "p90_ms": 0.04, + "p95_ms": 0.04, + "max_ms": 0.04, + "mean_ms": 0.04 + }, + "search_only": { + "n": 1, + "min_ms": 2013.4, + "median_ms": 2013.4, + "p90_ms": 2013.4, + "p95_ms": 2013.4, + "max_ms": 2013.4, + "mean_ms": 2013.4 + }, + "recall_total": { + "n": 1, + "min_ms": 4748.56, + "median_ms": 4748.56, + "p90_ms": 4748.56, + "p95_ms": 4748.56, + "max_ms": 4748.56, + "mean_ms": 4748.56 + } + }, + { + "label": "leaf-rivnut", + "query": "rivnut torque spec", + "resolve_floor": { + "n": 1, + "min_ms": 2.14, + "median_ms": 2.14, + "p90_ms": 2.14, + "p95_ms": 2.14, + "max_ms": 2.14, + "mean_ms": 2.14 + }, + "search_only": { + "n": 1, + "min_ms": 2014.69, + "median_ms": 2014.69, + "p90_ms": 2014.69, + "p95_ms": 2014.69, + "max_ms": 2014.69, + "mean_ms": 2014.69 + }, + "recall_total": { + "n": 1, + "min_ms": 4363.28, + "median_ms": 4363.28, + "p90_ms": 4363.28, + "p95_ms": 4363.28, + "max_ms": 4363.28, + "mean_ms": 4363.28 + } + } + ] + }, + "gates": [] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-worker-sweep.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-worker-sweep.json new file mode 100644 index 0000000..c1e60bc --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/m-worker-sweep.json @@ -0,0 +1,50 @@ +{ + "env": { + "date": "2026-06-23T20:18:05", + "endpoint": "https://echoapi.alwisp.com", + "commit": "af16598", + "echo_workers": 8, + "echo_timeout": 30, + "python": "3.10.12", + "iterations": 2, + "count_k": 50 + }, + "run_id": "20260623-201805-53dcdb", + "metrics": { + "worker-sweep": { + "result": { + "note_count": 197, + "sweep": [ + { + "workers": 1, + "median_ms": 11917.49, + "p95_ms": 11917.49 + }, + { + "workers": 2, + "median_ms": 5740.0, + "p95_ms": 5740.0 + }, + { + "workers": 4, + "median_ms": 3034.3, + "p95_ms": 3034.3 + }, + { + "workers": 8, + "median_ms": 1583.83, + "p95_ms": 1583.83 + }, + { + "workers": 16, + "median_ms": 1277.77, + "p95_ms": 1277.77 + } + ], + "knee_workers": 16, + "default_workers": 8 + }, + "gates": [] + } + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/report-data.json b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/report-data.json new file mode 100644 index 0000000..acee734 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/report-data.json @@ -0,0 +1,545 @@ +{ + "title": "ECHO Memory Plugin \u2014 Performance Benchmark", + "subtitle": "Validation of the 1.1.0 concurrency and 1.2.0 entity-resolution releases", + "reference": "echo-v.05 @ af16598", + "status": "11/11 gate checks pass", + "summary": "Full-vault reads run 7.1x faster concurrent than serial (11.4 s down to 1.6 s, 197 notes). Entity resolution is sub-millisecond. Profiling traced recall() latency to serial graph expansion (not BM25); the fix makes expansion concurrent (3.5-4.2x, identical results). All 11 gate checks pass. Run live against echoapi.alwisp.com on 2026-06-23.", + "sections": [ + { + "heading": "Test Environment", + "type": "kv", + "rows": [ + { + "label": "Endpoint", + "value": "echoapi.alwisp.com (live)" + }, + { + "label": "Vault size", + "value": "197 notes" + }, + { + "label": "Plugin commit", + "value": "af16598" + }, + { + "label": "Releases under test", + "value": "1.1.0 pooling + concurrency, 1.2.0 resolver" + }, + { + "label": "Concurrency (default)", + "value": "ECHO_WORKERS = 8" + }, + { + "label": "Socket timeout", + "value": "ECHO_TIMEOUT = 30 s" + }, + { + "label": "Runner", + "value": "Python 3.10.12" + }, + { + "label": "Date", + "value": "June 23, 2026" + } + ] + }, + { + "heading": "Headline Results", + "type": "table", + "columns": [ + "Operation", + "Result", + "Verdict" + ], + "rows": [ + [ + "Full-vault read (concurrent vs serial)", + { + "value": "7.13x", + "num": true + }, + "Pass" + ], + [ + "Full-vault read throughput", + { + "value": "123.5 notes/s", + "num": true + }, + "Pass" + ], + [ + "Cold vs warm request (keep-alive)", + { + "value": "3.25x", + "num": true + }, + "Pass" + ], + [ + "Entity resolve (in-memory index)", + { + "value": "<1 ms", + "num": true + }, + "Pass" + ], + [ + "Idempotent append skip", + { + "value": "100%", + "num": true + }, + "Pass" + ], + [ + "Soak read errors (22 s)", + { + "value": "0", + "num": true + }, + "Pass" + ] + ] + }, + { + "heading": "Full-Vault Read \u2014 Concurrency Scaling", + "type": "table", + "columns": [ + "Workers", + "Median (197 notes)", + "Speedup vs serial" + ], + "rows": [ + [ + "1 (serial)", + { + "value": "11,917 ms", + "num": true + }, + { + "value": "1.0x", + "num": true + } + ], + [ + "2", + { + "value": "5,740 ms", + "num": true + }, + { + "value": "2.1x", + "num": true + } + ], + [ + "4", + { + "value": "3,034 ms", + "num": true + }, + { + "value": "3.9x", + "num": true + } + ], + [ + "8 (default)", + { + "value": "1,584 ms", + "num": true + }, + { + "value": "7.5x", + "num": true + } + ], + [ + "16", + { + "value": "1,278 ms", + "num": true + }, + { + "value": "9.3x", + "num": true + } + ] + ] + }, + { + "heading": "Subject Pulls \u2014 recall() (after concurrency fix)", + "type": "table", + "columns": [ + "Subject", + "resolve()", + "search", + "recall() total" + ], + "rows": [ + [ + "APTA", + { + "value": "0.75 ms", + "num": true + }, + { + "value": "62 ms", + "num": true + }, + { + "value": "2,231 ms", + "num": true + } + ], + [ + "CapMetro", + { + "value": "0.03 ms", + "num": true + }, + { + "value": "58 ms", + "num": true + }, + { + "value": "2,116 ms", + "num": true + } + ], + [ + "echo (hub)", + { + "value": "0.03 ms", + "num": true + }, + { + "value": "71 ms", + "num": true + }, + { + "value": "2,404 ms", + "num": true + } + ], + [ + "operator preferences (hub)", + { + "value": "0.04 ms", + "num": true + }, + { + "value": "2,011 ms", + "num": true + }, + { + "value": "2,995 ms", + "num": true + } + ], + [ + "rivnut torque spec (leaf)", + { + "value": "2.15 ms", + "num": true + }, + { + "value": "2,014 ms", + "num": true + }, + { + "value": "2,371 ms", + "num": true + } + ] + ] + }, + { + "heading": "recall() Graph Expansion \u2014 Concurrency Fix", + "type": "table", + "columns": [ + "Hub subject", + "Neighbours", + "Serial", + "Concurrent", + "Speedup", + "Results identical" + ], + "rows": [ + [ + "echo", + { + "value": "120", + "num": true + }, + { + "value": "2,228 ms", + "num": true + }, + { + "value": "642 ms", + "num": true + }, + { + "value": "3.47x", + "num": true + }, + "Yes" + ], + [ + "operator preferences", + { + "value": "126", + "num": true + }, + { + "value": "2,910 ms", + "num": true + }, + { + "value": "699 ms", + "num": true + }, + { + "value": "4.16x", + "num": true + }, + "Yes" + ] + ] + }, + { + "heading": "Bulk Write Operations", + "type": "table", + "columns": [ + "Operation", + "Per-op median", + "Aggregate" + ], + "rows": [ + [ + "PUT (read-back verified)", + { + "value": "115 ms", + "num": true + }, + { + "value": "7.0 PUT/s", + "num": true + } + ], + [ + "Bulk GET \u2014 concurrent vs serial (K=30)", + { + "value": "356 vs 1,607 ms", + "num": true + }, + { + "value": "4.51x", + "num": true + } + ], + [ + "APPEND \u2014 write", + { + "value": "104 ms", + "num": true + }, + { + "value": "1 round-trip + GET", + "num": false + } + ], + [ + "APPEND \u2014 idempotent skip", + { + "value": "51 ms", + "num": true + }, + { + "value": "100% skipped on re-run", + "num": false + } + ] + ] + }, + { + "heading": "Concurrency, Locking & Resilience", + "type": "table", + "columns": [ + "Check", + "Measured", + "Expected" + ], + "rows": [ + [ + "Lock \u2014 acquire when free", + { + "value": "159 ms (rc 0)", + "num": true + }, + "rc 0" + ], + [ + "Lock \u2014 contended acquire", + { + "value": "58 ms (rc 75)", + "num": true + }, + "rc 75 fast-fail" + ], + [ + "Lock \u2014 release", + { + "value": "133 ms", + "num": true + }, + "owned release" + ], + [ + "Offline queue \u2014 enqueue", + { + "value": "0.08 ms", + "num": true + }, + "local write-ahead" + ], + [ + "read_many dedup (60 -> 20)", + { + "value": "deduped", + "num": false + }, + "collapse to unique set" + ], + [ + "Soak drift (late vs early)", + { + "value": "0.99x", + "num": true + }, + "<= 1.5x, no leak" + ] + ] + }, + { + "heading": "Regression Gates (relative-ratio policy)", + "type": "table", + "columns": [ + "Gate", + "Threshold", + "Measured", + "Result" + ], + "rows": [ + [ + "read-full speedup", + ">= 1.8x", + { + "value": "7.13x", + "num": true + }, + "Pass" + ], + [ + "bulk-get speedup", + ">= 1.5x", + { + "value": "4.51x", + "num": true + }, + "Pass" + ], + [ + "pool warm-up cold/warm", + ">= 1.5x", + { + "value": "3.25x", + "num": true + }, + "Pass" + ], + [ + "append idempotency", + "100% skip", + { + "value": "100%", + "num": true + }, + "Pass" + ], + [ + "soak errors", + "0", + { + "value": "0", + "num": true + }, + "Pass" + ], + [ + "soak drift", + "<= 1.5x", + { + "value": "0.99x", + "num": true + }, + "Pass" + ], + [ + "lock semantics", + "rc 0 / rc 75", + { + "value": "0 / 75", + "num": true + }, + "Pass" + ], + [ + "read_many dedup", + "unique set", + { + "value": "20/20", + "num": true + }, + "Pass" + ] + ] + }, + { + "heading": "Findings", + "type": "olist", + "items": [ + "Concurrency delivers as designed. Eight workers cut full-vault reads 7.5x; the curve is near-linear to 8 workers and flattens after (8->16 returns only 1.24x). The default of 8 is the right setting for this vault and link.", + "Keep-alive is confirmed live. A cold request costs 163 ms against a 50 ms warm median (3.25x). This is the direct measure that the 1.1.0 pooling fix removed the per-request TLS handshake that was timing out full-vault passes.", + "Entity resolution is effectively free. resolve() returns in under 1 ms across exact, alias, and shortened mentions. The 1.2.0 alias work resolves shortened names (\"echo memory\", \"ECHO plugin\", \"jason\") straight to the canonical note, so the anti-duplicate guard holds without a fuzzy fallback.", + "recall() latency was traced to the graph layer, not BM25. The BM25 index is already persisted and incrementally maintained (load 500 ms, score 0.1 ms). The cost was expand_graph fetching each neighbour serially \u2014 3.0 s for 126 nodes. Fix: the graph BFS now fetches each hop concurrently via the existing read_many. expand_graph is 3.5-4.2x faster with byte-identical ranking and scores; end-to-end recall() dropped from 2.8-4.7 s to 2.1-3.0 s per subject.", + "Full-vault maintenance scripts stay under the tool timeout. sweep.py runs in 4.6 s and vault_lint.py in 5.1 s on 197 notes. The original failure mode (passes exceeding the timeout and dropping the session) does not recur.", + "Write integrity holds. PUT verifies via read-back at 115 ms; APPEND is whole-line idempotent, skipping at 51 ms with zero duplicate writes on re-run. The advisory lock fast-fails a contended acquire with the expected exit 75." + ] + }, + { + "heading": "Recommendations", + "type": "olist", + "items": [ + "Apply read_many to every code path that reads a set of known notes. Done for expand_graph (gated). Remaining: prefetch the _brief result bodies recall prints, and parallelize rebuild()'s vault walk.", + "Reuse one in-process read cache across a recall() call so load_index, expansion, and _brief don't re-fetch overlapping notes \u2014 this closes the remaining ~500 ms + serial _brief cost.", + "Make 'prefer read_many over a GET loop' a documented contract in the plugin's API reference; serial multi-note reads are a performance bug. The expand-graph gate enforces it for recall.", + "Hold ECHO_WORKERS at 8 (16 buys only 1.24x). Promote the resolve correctness table from INFO to FAIL now that live behavior is confirmed. Re-baseline at the next 1.x release or past ~400 notes." + ] + }, + { + "heading": "Method", + "type": "text", + "body": [ + "Each metric was timed with a perf-counter harness that reuses the live ECHO client, so it exercises the real pooled and concurrent code path. Reads report median and p95 over repeated iterations after a discarded warm-up; write metrics ran in a disposable namespace under the advisory lock and were deleted on completion (zero residual files confirmed).", + "Gates use relative ratios rather than absolute milliseconds, so they stay valid regardless of the network the suite runs from. The harness, fixtures, baselines, and per-metric JSON are checked in under eval/perf/." + ] + } + ], + "note": { + "heading": "Next Step", + "body": "Prefetch recall()'s _brief reads and parallelize rebuild() via read_many, then re-run subject-pull to confirm recall drops toward ~1 s. expand_graph fix is local-only until ported to the canonical repo. No blockers in 1.1.0 / 1.2.0." + } +} \ No newline at end of file diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/run.log b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/run.log new file mode 100644 index 0000000..9ceb0ac --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/results/run.log @@ -0,0 +1 @@ +[20:14:00] read-full diff --git a/echo-memory.plugin.src/skills/echo-memory/eval/perf/run_heavy.sh b/echo-memory.plugin.src/skills/echo-memory/eval/perf/run_heavy.sh new file mode 100755 index 0000000..28c4a42 --- /dev/null +++ b/echo-memory.plugin.src/skills/echo-memory/eval/perf/run_heavy.sh @@ -0,0 +1,12 @@ +#!/bin/bash +export ECHO_KEY_LEGACY_OK=1 +cd "$(dirname "$0")" +{ + echo "[$(date +%T)] read-full"; python3 bench.py read-full -n 3 --quiet --json-out results/m-read-full.json + echo "[$(date +%T)] worker-sweep"; python3 bench.py worker-sweep -n 4 --quiet --json-out results/m-worker-sweep.json + echo "[$(date +%T)] subject-pull"; python3 bench.py subject-pull -n 4 --quiet --json-out results/m-subject-pull.json + echo "[$(date +%T)] index"; python3 bench.py index --quiet --json-out results/m-index.json + echo "[$(date +%T)] lint"; python3 bench.py lint --quiet --json-out results/m-lint.json + echo "[$(date +%T)] DONE" +} > results/run.log 2>&1 +touch results/DONE diff --git a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_recall.py b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_recall.py index 12fcac7..93bbed8 100644 --- a/echo-memory.plugin.src/skills/echo-memory/scripts/echo_recall.py +++ b/echo-memory.plugin.src/skills/echo-memory/scripts/echo_recall.py @@ -247,27 +247,34 @@ def expand_graph(seeds, nmap: dict, base_scores: dict, max_hops: int = MAX_HOPS) score_of = dict(base_scores) seen = set(seeds) results: dict[str, tuple[float, str]] = {} - dq = deque((s, 0) for s in seeds) - while dq: - path, hop = dq.popleft() - if hop >= max_hops: - continue - text = links.get_text(path) - if text is None: - continue - body = strip_frontmatter(text) - targets = set(links.all_wikilinks(body)) | set(links.source_notes(text)) - parent = score_of.get(path, 1.0) - for t in targets: - tp = _resolve_target(t, nmap) - if not tp or tp in seeds: + # Breadth-first by HOP so each frontier's note bodies can be fetched concurrently + # (echo.read_many) instead of one blocking GET per node. Scoring is unchanged: a + # neighbour keeps the max decayed score over the paths that reach it, and non-seed + # parents score 1.0 exactly as the serial deque version did. + frontier = list(seeds) + for hop in range(max_hops): + if not frontier: + break + texts = echo.read_many(frontier) + next_frontier: list[str] = [] + for path in frontier: + text = texts.get(path) + if text is None: continue + body = strip_frontmatter(text) + targets = set(links.all_wikilinks(body)) | set(links.source_notes(text)) + parent = score_of.get(path, 1.0) decayed = parent * (GRAPH_DECAY ** (hop + 1)) - if decayed > results.get(tp, (0.0, ""))[0]: - results[tp] = (decayed, path) - if tp not in seen: - seen.add(tp) - dq.append((tp, hop + 1)) + for t in targets: + tp = _resolve_target(t, nmap) + if not tp or tp in seeds: + continue + if decayed > results.get(tp, (0.0, ""))[0]: + results[tp] = (decayed, path) + if tp not in seen: + seen.add(tp) + next_frontier.append(tp) + frontier = next_frontier return sorted(results.items(), key=lambda kv: -kv[1][0])