forked from jason/echo
testing and documentation
This commit is contained in:
Jason Stedwell
Binary file not shown.
@@ -0,0 +1,146 @@
|
|||||||
|
# ECHO Performance Test Suite — Plan
|
||||||
|
|
||||||
|
**Status:** plan only — nothing here is to be run yet.
|
||||||
|
**Author:** drafted for Jason Stedwell, 2026-06-23.
|
||||||
|
**Targets the changes that landed in:** 1.1.0 (connection pooling + concurrent `read_many` + shared cache) and 1.2.0 (fuzzy `resolve`, alias derivation/learning, `capture` re-slug fix).
|
||||||
|
**Scope:** a *performance / timing* harness. It complements — does not replace — the existing correctness tests (`test_echo_client.py`, `test_v1_scaffold.py`, the eval feature tests), which stay as the pass/fail gate for behaviour.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Goals
|
||||||
|
|
||||||
|
1. Put hard numbers on the three operations Jason called out — full endpoint read, full subject pulls, bulk get/put/append — so the 1.1.0 perf claims ("lint 0.90s / sweep 0.85s on 186 notes, ~4.5× from reuse, ~2× from concurrency") become a repeatable, checked-in measurement instead of a one-time observation.
|
||||||
|
2. Establish **baselines + regression gates** so a future change that quietly reintroduces the pre-1.1.0 per-request TLS handshake (the bug that was timing out sessions) fails the suite instead of silently shipping.
|
||||||
|
3. Keep every run **non-destructive to the production vault** — writes go to a disposable namespace and are cleaned up (per Jason's "prune test artifacts" rule).
|
||||||
|
|
||||||
|
## 2. Methodology (applies to every metric)
|
||||||
|
|
||||||
|
- **Harness:** a new `bench.py` under `scripts/` (or `eval/perf/`), reusing the live `echo` client module so it measures the real pooled/concurrent code path, not a reimplementation.
|
||||||
|
- **Timing:** `time.perf_counter()` around each operation. Report **min / median / p90 / p95 / max / mean** over N iterations — not a single number. Single timings hide GC pauses and TLS renegotiation.
|
||||||
|
- **Warm-up:** discard the first 1–2 iterations (cold connection pool, cold server cache) and report them *separately* as the cold-start cost — that delta is itself one of the most interesting numbers (it's what 1.1.0's keep-alive was meant to kill).
|
||||||
|
- **Iterations:** default N=10 for read-heavy metrics, N=5 for write-heavy (writes are slower and we don't want to bloat the vault). Make N a CLI flag.
|
||||||
|
- **Environment capture:** record `ECHO_WORKERS`, `ECHO_TIMEOUT`, vault note count, git commit/plugin version, and wall-clock date in the results header so two runs are comparable.
|
||||||
|
- **Output:** machine-readable JSON (`eval/perf/results/<date>-<commit>.json`) plus a short human table to stdout. JSON lets us diff runs over time and chart trend.
|
||||||
|
- **Isolation:** all test writes live under a dedicated prefix, e.g. `_agent/_bench/<run-id>/...`, never in `projects/`, `resources/`, `inbox/`, or `journal/`. A `--cleanup` pass deletes the whole prefix at the end; a `--keep` flag preserves it for debugging. The advisory lock (`echo.py lock`) is taken for the write phases so a bench run can't race a real CoWork session.
|
||||||
|
- **Network honesty:** the endpoint is the live `echoapi.alwisp.com`, so absolute numbers depend on WAN latency. Report **both** absolute ms and a relative ratio against the same run's single-GET baseline, so the suite stays meaningful regardless of where it's run from.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Core metrics (the three Jason asked for)
|
||||||
|
|
||||||
|
### 3.1 Timed full endpoint read (full-vault bulk read)
|
||||||
|
|
||||||
|
**What it measures:** the headline 1.1.0 path — reading every note in the vault via concurrent `read_many` over warm pooled connections vs. serial GETs.
|
||||||
|
|
||||||
|
**Procedure:**
|
||||||
|
1. Enumerate all vault paths (the same listing `sweep.py`/`vault_lint.py` walk).
|
||||||
|
2. **Serial baseline:** GET each path one at a time on a single connection. Time total + per-note.
|
||||||
|
3. **Pooled+concurrent:** `read_many(paths)` at the default `ECHO_WORKERS`. Time total.
|
||||||
|
4. Report total time, notes/sec, and the **speedup ratio** (serial ÷ concurrent). Repeat the concurrent pass N times for percentile spread.
|
||||||
|
|
||||||
|
**What "good" looks like:** concurrent should be multiples faster than serial; full-vault concurrent read should stay well under the agent tool timeout (the original failure mode). Record current note count alongside — the metric is only comparable at similar vault sizes.
|
||||||
|
|
||||||
|
**Regression gate (suggested):** full-vault concurrent read median < a threshold (e.g. 3s at ~200 notes) AND speedup ratio ≥ ~2×. Tune thresholds after the first baseline run.
|
||||||
|
|
||||||
|
### 3.2 Timed full subject pulls (e.g. APTA, CapMetro statuses)
|
||||||
|
|
||||||
|
**What it measures:** the cost of a `recall "<subject>"` — search + 1-hop neighbourhood expansion along `## Related` links and `source_notes` — which is the real "what do we know about X" operation, and exercises the 1.2.0 resolver. APTA and CapMetro are live subjects in the vault, so they're realistic fixtures.
|
||||||
|
|
||||||
|
**Procedure:**
|
||||||
|
1. Pick a fixed fixture set of subjects with **different neighbourhood sizes**: a small one, a hub note with many links (e.g. the `echo` project or `operator-preferences`), and the two named ones (APTA, CapMetro). Neighbourhood size is the real cost driver, so vary it deliberately.
|
||||||
|
2. Time end-to-end `recall` for each: (a) the search call, (b) the 1-hop expansion reads, (c) total. Break out the search vs. expansion split — it tells us whether latency is in the index/search or in the fan-out reads (which 1.1.0's `read_many` should accelerate).
|
||||||
|
3. Also time bare `resolve "<subject>"` (the O(1) index lookup) separately as the floor — recall should be resolve + search + N expansion reads.
|
||||||
|
4. Report per-subject and aggregate percentiles; annotate each with its neighbourhood node count so time-per-node is derivable.
|
||||||
|
|
||||||
|
**What "good" looks like:** `resolve` is near-instant (in-memory index); `recall` scales with neighbourhood size but the expansion reads are concurrent, so a hub note shouldn't be linearly worse than a leaf.
|
||||||
|
|
||||||
|
**Live result + the `expand-graph` sub-metric (added 2026-06-23).** The first run showed `recall()` at 2.8–4.7 s/subject. Profiling pinned it on the graph layer, not BM25: `load_index()` ~500 ms (index already persisted/incremental), `score()` 0.1 ms, `expand_graph()` ~3,000 ms doing one serial GET per neighbour. Fix shipped in `echo_recall.py` — the BFS now fetches each hop via `read_many`. A dedicated **`expand-graph`** metric times serial vs concurrent expansion (gate: ≥2× min) and asserts byte-identical ranking + scores against a serial reference, so the fix can't silently regress. Measured 3.5–4.2×; end-to-end `recall()` down ~1.7×.
|
||||||
|
|
||||||
|
### 3.3 Timed bulk get / put / append
|
||||||
|
|
||||||
|
**What it measures:** write-path throughput and the read-back/idempotency overhead Jason deliberately pays for.
|
||||||
|
|
||||||
|
**Procedure (all in the `_agent/_bench/<run-id>/` namespace):**
|
||||||
|
- **Bulk GET:** create K fixture notes once, then time reading them back — both serially and via `read_many` — to compare against §3.1 at a controlled, fixed K (e.g. 25/50/100) so the read curve is isolated from whatever the live vault happens to contain.
|
||||||
|
- **Bulk PUT:** time creating K notes. `echo.py put` is read-back-verified (it re-GETs after writing), so this measures the true write+verify cost, not a fire-and-forget POST. Report puts/sec and per-put median.
|
||||||
|
- **Bulk APPEND:** time K appends to a single growing note. `append` is whole-line idempotent (it GETs first and skips duplicates), so also measure the **idempotent-skip path**: re-append the same K lines and confirm near-zero net writes — time the skip vs. the real append. This validates that idempotency isn't quietly O(file size) per append as the note grows.
|
||||||
|
- **Concurrent write caveat:** writes to single-line shared files assume one writer; the bulk-write bench must use per-note targets (or hold the lock) so it doesn't model an unsupported pattern.
|
||||||
|
|
||||||
|
**What "good" looks like:** PUT slower than GET (verify round-trip); append-skip much cheaper than append-write; no superlinear blowup as the appended note grows.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Additional test suggestions (recommended, beyond the three)
|
||||||
|
|
||||||
|
These target the rest of the 1.1.0/1.2.0 surface and the known historical failure modes.
|
||||||
|
|
||||||
|
1. **Connection-pool cold vs. warm.** Isolate the keep-alive win directly: time the *first* request after pool creation vs. the median of subsequent requests. This is the single clearest proof the pooling fix is alive; a regression here is the early-warning signal for the old per-request-handshake bug.
|
||||||
|
|
||||||
|
2. **Concurrency scaling sweep (`ECHO_WORKERS`).** Run §3.1 at workers = 1, 2, 4, 8, 16. Plot total time vs. workers to find the knee and confirm the default of 8 is sensible for the current vault/WAN, and that it degrades gracefully (no errors/timeouts) at high concurrency.
|
||||||
|
|
||||||
|
3. **Fuzzy-resolve latency + accuracy (1.2.0).** Two parts: (a) *timing* — `resolve` on exact, alias, and shortened/typo'd mentions; the fuzzy-candidate path does more work, so measure its overhead vs. an exact hit. (b) *correctness* — a fixture table of mention → expected canonical slug (e.g. "echo memory" → `echo`, alias hits, a near-miss that *should* return ranked candidates rather than spawn a duplicate). This guards the actual bug 1.2.0 fixed. Correctness assertions, not just timing.
|
||||||
|
|
||||||
|
4. **Index build/rebuild time.** Time `sweep.py` entity-index rebuild + link symmetrization on the full vault. This is the other full-vault script that historically neared the timeout; gate it like §3.1.
|
||||||
|
|
||||||
|
5. **Lint + recall-rebuild full-vault timing.** Regression-gate `vault_lint.py` and the recall rebuild against the 1.1.0 baselines (lint ~0.90s, sweep ~0.85s @ 186 notes). These are the concrete numbers in the changelog — turn them into asserted thresholds scaled to current note count.
|
||||||
|
|
||||||
|
6. **Lock contention timing.** Time `lock` acquisition when free, when held-fresh (should fast-fail with exit 75), and reclaim of a stale lock past `ECHO_LOCK_TTL`. Confirms the read-back-confirmed lock doesn't add pathological latency and behaves under contention.
|
||||||
|
|
||||||
|
7. **Offline write-ahead queue throughput (1.0 carry-over).** With the endpoint unreachable, time enqueue of K writes, then time `flush` replay on reconnect. Validates the queue doesn't degrade and replays in order. Useful because it's an untimed path today.
|
||||||
|
|
||||||
|
8. **Cache hit/miss ratio.** Instrument the shared single-pass cache: in a sweep/lint run, count served-from-cache vs. network reads. A correctness+efficiency check that the cache is actually collapsing duplicate reads, not just present.
|
||||||
|
|
||||||
|
9. **Scaling fixture / synthetic vault.** Optional but valuable: a script that mints a throwaway vault of N synthetic notes (100 / 500 / 1000) in the bench namespace to measure how all the above scale *beyond* the current ~190 notes — answers "when does the next timeout cliff arrive?" before it hits a real session.
|
||||||
|
|
||||||
|
10. **Soak / stability pass.** Run §3.1 in a loop for M minutes to surface connection leaks, pool exhaustion, or memory growth in the long-lived client — relevant because the pool is thread-local and long-running.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Proposed file layout
|
||||||
|
|
||||||
|
```
|
||||||
|
eval/perf/
|
||||||
|
PERF-TEST-PLAN.md # this file
|
||||||
|
bench.py # the harness (CLI: --metric, --iterations, --workers, --cleanup/--keep, --json-out)
|
||||||
|
fixtures.py # subject lists, K-note generators, mention→slug resolve table
|
||||||
|
baselines.json # committed thresholds + reference numbers (per note-count bucket)
|
||||||
|
results/ # dated JSON run outputs (gitignored or kept for trend)
|
||||||
|
2026-06-23-<commit>.json
|
||||||
|
```
|
||||||
|
|
||||||
|
`bench.py` subcommands map 1:1 to the metrics: `read-full`, `subject-pull`, `bulk-get`, `bulk-put`, `bulk-append`, plus `pool-warmup`, `worker-sweep`, `resolve`, `index`, `lint`, `lock`, `queue`, `cache`, `soak`.
|
||||||
|
|
||||||
|
## 6. Decisions (resolved 2026-06-23)
|
||||||
|
|
||||||
|
- **Baseline policy:** **relative ratios** are the gate (`baselines.json`); absolute ms are recorded as informational. Portable across machines/WAN.
|
||||||
|
- **CI vs. manual:** **manual / pre-release.** The suite is invoked by hand before tagging; it is not wired into CI (avoids WAN flakiness failing builds against the live endpoint).
|
||||||
|
- **Synthetic vault (suggestion #9):** **deferred.** Benchmark the real ~190-note vault for now; add the generator if vault growth makes it relevant.
|
||||||
|
- **Cleanup:** **auto-delete** the `_agent/_bench/<run-id>/` namespace at end of each run, with `--keep` to override.
|
||||||
|
|
||||||
|
## 7. Build status
|
||||||
|
|
||||||
|
Harness built under `eval/perf/`:
|
||||||
|
|
||||||
|
- `bench.py` — the CLI harness; all 14 metrics implemented (`read-full`, `subject-pull`, `bulk-get`, `bulk-put`, `bulk-append`, `pool-warmup`, `worker-sweep`, `resolve`, `index`, `lint`, `lock`, `queue`, `cache`, `soak`).
|
||||||
|
- `fixtures.py` — subject list (APTA/CapMetro/hubs/leaf), the 1.2.0 resolve correctness table, and bench-note generators.
|
||||||
|
- `baselines.json` — relative-ratio gates (tune thresholds after the first real run).
|
||||||
|
- `README.md` — run instructions, metric map, isolation/cleanup, caveats.
|
||||||
|
- `results/` — dated JSON run outputs.
|
||||||
|
|
||||||
|
Verified offline (compile + imports + `--list`/`--help`), then **run live** against the 197-note vault @ af16598 — 11/11 gate checks pass. First run surfaced the recall() finding below; the fix and a guarding metric were added the same session.
|
||||||
|
|
||||||
|
## 8. ECHO usage improvements (from the live run)
|
||||||
|
|
||||||
|
The benchmark wasn't just validation — it surfaced concrete improvements to how ECHO should do reads. In priority order:
|
||||||
|
|
||||||
|
1. **Apply `read_many` everywhere a code path reads a *set* of notes (DONE for `expand_graph`).** The 1.1.0 release shipped concurrent bulk reads but only `sweep`/`lint`/full-vault read adopted it. `recall()`'s graph expansion was still serial — one GET per neighbour — which is why it ran 3 s. The rule going forward: *any* loop that GETs more than ~3 notes whose paths are known up front should batch them through `echo.read_many`, not iterate `get_text`. Audit candidates: `expand_graph` (fixed), `_brief` result printing (still serial), `rebuild()`'s vault walk (still serial), and any future multi-note reader.
|
||||||
|
|
||||||
|
2. **Prefetch `_brief` bodies in one `read_many` (NEXT).** After expansion, `recall` prints each primary hit + neighbour via `_brief()`, which GETs the note again to pull its type/status. Those are known paths — fetch them all in one concurrent batch (and reuse bodies already pulled during expansion via a per-call cache, so a note fetched in the BFS isn't re-fetched to print it).
|
||||||
|
|
||||||
|
3. **Reuse one in-process read cache across a `recall` call.** `load_index`, `expand_graph`, and `_brief` currently re-fetch overlapping notes. A single `{path: text}` memo passed through the call (the same pattern `sweep.py` already uses via `read_many` once) removes the duplicate GETs — `load_index`'s ~500 ms and the `_brief` re-reads are the remaining recall cost.
|
||||||
|
|
||||||
|
4. **Make "batch the reads" a documented contract, not a per-site fix.** Add a one-line rule to the plugin's operating contract / API reference: *prefer `read_many(paths)` over a GET loop; serial multi-note reads are a performance bug.* This is what would have caught the `expand_graph` regression at authoring time. The `expand-graph` metric now enforces it mechanically for recall; the contract generalizes it.
|
||||||
|
|
||||||
|
5. **Consider indexing `## Related`/`source_notes` adjacency into the persisted index.** Longer-term: the graph edges are derivable and change only on write. Storing an adjacency list alongside the BM25 postings (maintained by `capture`/`sweep`, same as `entities.json`) would let `expand_graph` skip re-reading note bodies just to extract links — turning the graph layer into an in-memory lookup like `resolve`. Bigger change; revisit if recall latency still matters after #2–#3.
|
||||||
|
|
||||||
|
None of #1–#4 add dependencies or change behavior — they apply the concurrency primitive that already exists. #1 is done and gated; #2–#3 are the next cheap wins; #5 is the structural option.
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
# ECHO Performance Test Suite
|
||||||
|
|
||||||
|
A timing harness for the operations the **1.1.0** (connection pooling + concurrent
|
||||||
|
`read_many` + shared cache) and **1.2.0** (fuzzy `resolve` / alias learning) updates
|
||||||
|
targeted. It reuses the live `echo` client module, so it measures the real pooled /
|
||||||
|
concurrent code path — not a reimplementation.
|
||||||
|
|
||||||
|
This is a **manual / pre-release** tool, not a CI gate: it depends on the live
|
||||||
|
endpoint (`echoapi.alwisp.com`) and WAN latency. Gates are **relative ratios**
|
||||||
|
(portable), with absolute milliseconds kept as informational context.
|
||||||
|
|
||||||
|
See `PERF-TEST-PLAN.md` for the design rationale behind each metric.
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd eval/perf
|
||||||
|
python3 bench.py --list # list metrics
|
||||||
|
python3 bench.py all # the read-only safe set (no vault writes)
|
||||||
|
python3 bench.py read-full -n 10 # full-vault serial-vs-concurrent read
|
||||||
|
python3 bench.py subject-pull # recall APTA / CapMetro / hubs (resolve+search+recall split)
|
||||||
|
python3 bench.py bulk-put -k 50 # write metric -> uses the _bench namespace
|
||||||
|
python3 bench.py worker-sweep # ECHO_WORKERS = 1,2,4,8,16 scaling curve
|
||||||
|
python3 bench.py resolve # 1.2.0 fuzzy correctness table + timing
|
||||||
|
python3 bench.py soak --soak-seconds 120 # stability / leak watch
|
||||||
|
```
|
||||||
|
|
||||||
|
Set the key first if the baked-in fallback warning bothers you:
|
||||||
|
`python3 ../../scripts/echo.py write-key <token>` (or `export ECHO_KEY=...`).
|
||||||
|
|
||||||
|
## Metrics
|
||||||
|
|
||||||
|
| Metric | Plan § | Reads/Writes | What it proves |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `read-full` | 3.1 | read-only | full-vault concurrent vs serial speedup (1.1.0 headline) |
|
||||||
|
| `subject-pull` | 3.2 | read-only | `recall` cost for APTA/CapMetro/hubs, split into resolve floor + search + expansion |
|
||||||
|
| `expand-graph` | 3.2a | read-only | recall()'s graph layer: serial vs concurrent neighbourhood expansion + ranking-parity guard (pins the real recall bottleneck) |
|
||||||
|
| `bulk-get` | 3.3 | writes seed | concurrent vs serial read at a controlled K |
|
||||||
|
| `bulk-put` | 3.3 | **writes** | read-back-verified PUT throughput (puts/sec) |
|
||||||
|
| `bulk-append` | 3.3 | **writes** | append throughput + idempotent-skip path (no O(file) blowup) |
|
||||||
|
| `pool-warmup` | 4.1 | read-only | cold vs warm single GET — keep-alive is alive |
|
||||||
|
| `worker-sweep` | 4.2 | read-only | concurrency scaling knee; validates default workers=8 |
|
||||||
|
| `resolve` | 4.3 | read-only | fuzzy-resolve timing **and** correctness (anti-duplicate guard) |
|
||||||
|
| `index` | 4.4 | read-only | `sweep.py` index rebuild + link symmetrize time |
|
||||||
|
| `lint` | 4.5 | read-only | `vault_lint.py` full-vault time vs 1.1.0 baseline |
|
||||||
|
| `lock` | 4.6 | **writes** lock | advisory-lock acquire/contend/release timing + semantics |
|
||||||
|
| `queue` | 4.7 | none (bogus base) | offline write-ahead enqueue timing (see caveat) |
|
||||||
|
| `cache` | 4.8 | read-only | `read_many` dedups a duplicated path list |
|
||||||
|
| `soak` | 4.10 | read-only | stability over a window; leak/drift detector |
|
||||||
|
|
||||||
|
`all` runs only the read-only set. Write metrics must be requested by name.
|
||||||
|
|
||||||
|
## Isolation & cleanup
|
||||||
|
|
||||||
|
Write metrics touch **only** `_agent/_bench/<run-id>/`, take the advisory lock for
|
||||||
|
the duration (and abort if another session holds it), and delete the namespace at
|
||||||
|
the end. Pass `--keep` to preserve it for debugging. Nothing is written to
|
||||||
|
`projects/`, `resources/`, `inbox/`, or `journal/`.
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
Each run writes `results/<date>-<commit>.json` (machine-readable, for trend diffing)
|
||||||
|
and prints a human table. The JSON header captures `ECHO_WORKERS`, `ECHO_TIMEOUT`,
|
||||||
|
note count, git commit, and date so two runs are comparable.
|
||||||
|
|
||||||
|
## Gating
|
||||||
|
|
||||||
|
`baselines.json` holds relative-ratio gates (e.g. `read-full.speedup_ratio >= 1.8`,
|
||||||
|
`pool-warmup.cold_over_warm_ratio >= 1.5`). Run once to capture a baseline, then
|
||||||
|
tune the thresholds to the observed numbers before treating a `FAIL` as blocking.
|
||||||
|
`--no-gate` reports numbers without pass/fail.
|
||||||
|
|
||||||
|
## Finding: recall() bottleneck is the graph layer, not BM25
|
||||||
|
|
||||||
|
A profiled `recall("operator preferences")` decomposes as: `load_index()` ~500 ms (the
|
||||||
|
BM25 index is already persisted + incrementally maintained, n_docs=119), `score()`
|
||||||
|
0.1 ms, and `expand_graph()` **~3,000 ms** — the graph BFS was fetching each neighbour
|
||||||
|
serially (one GET per node, ~126 nodes). The fix (shipped in `echo_recall.py`) makes the
|
||||||
|
BFS breadth-first by hop and fetches each frontier through the existing `read_many`
|
||||||
|
concurrency. Result: `expand_graph` 3.2–4.2x faster with byte-identical ranking and
|
||||||
|
scores; end-to-end `recall()` ~1.7x (e.g. APTA 3.8 s -> 2.2 s). The `expand-graph`
|
||||||
|
metric + gate above regression-guard this so it can't quietly revert to serial.
|
||||||
|
|
||||||
|
## Caveats
|
||||||
|
|
||||||
|
- **`resolve` verdicts are PASS/INFO only.** The correctness table in `fixtures.py`
|
||||||
|
encodes expected slugs against the live vault; until confirmed, mismatches report
|
||||||
|
`INFO` (tune-me), not `FAIL`. Promote to FAIL once the table is verified.
|
||||||
|
- **`queue` times the enqueue path only.** A true flush-replay needs a throwaway
|
||||||
|
sandbox vault so the replay doesn't write to production; that's deferred (the
|
||||||
|
plan's synthetic-vault item). The metric still proves enqueue works and reports
|
||||||
|
queue depth.
|
||||||
|
- **Absolute ms depend on where you run from.** WAN latency to `echoapi.alwisp.com`
|
||||||
|
dominates single-request times; that's why gates are ratios.
|
||||||
|
- **Synthetic-vault scaling (100/500/1000 notes) is deferred** per decision — the
|
||||||
|
suite benchmarks the real ~190-note vault today.
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
{
|
||||||
|
"_comment": "Relative-ratio regression gates (Jason's chosen policy). Ratios are portable across machines/WAN; absolute ms in the results JSON stay informational. Tune these after the first real baseline run, then tighten. 'field' is a dotted path into a metric's result; for list items use the index (e.g. sweep.3.median_ms).",
|
||||||
|
"read-full": {
|
||||||
|
"gates": [
|
||||||
|
{ "field": "speedup_ratio", "op": ">=", "min": 1.8,
|
||||||
|
"why": "concurrent read_many must beat serial by the 1.1.0 concurrency margin; <1.8 means concurrency regressed" }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"bulk-get": {
|
||||||
|
"gates": [
|
||||||
|
{ "field": "speedup_ratio", "op": ">=", "min": 1.5,
|
||||||
|
"why": "controlled-K concurrent vs serial; lower bar than full-vault since K is small" }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"expand-graph": {
|
||||||
|
"gates": [
|
||||||
|
{ "field": "min_speedup", "op": ">=", "min": 2.0,
|
||||||
|
"why": "recall()'s graph layer must fetch each BFS hop concurrently (read_many), not serially per node; <2x means expand_graph regressed to the pre-fix serial walk" },
|
||||||
|
{ "field": "all_ranking_identical", "op": ">=", "min": 1,
|
||||||
|
"why": "the concurrent expansion must return the same ranked neighbours as the serial reference" },
|
||||||
|
{ "field": "all_scores_identical", "op": ">=", "min": 1,
|
||||||
|
"why": "decayed scores must match the serial reference to within float tolerance" }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"pool-warmup": {
|
||||||
|
"gates": [
|
||||||
|
{ "field": "cold_over_warm_ratio", "op": ">=", "min": 1.5,
|
||||||
|
"why": "cold request must be meaningfully slower than warm — proves keep-alive is reusing the connection; a ratio near 1.0 means every request is re-handshaking (the pre-1.1.0 bug)" }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"bulk-append": {
|
||||||
|
"gates": [
|
||||||
|
{ "field": "all_skipped_second_pass", "op": ">=", "min": 1,
|
||||||
|
"why": "second pass of identical lines must be 100% idempotent skips (boolean true coerced to 1)" }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"soak": {
|
||||||
|
"gates": [
|
||||||
|
{ "field": "errors", "op": "<=", "min": 0,
|
||||||
|
"why": "no read errors across the soak window — connection leaks/pool exhaustion would surface here" },
|
||||||
|
{ "field": "drift_ratio_late_over_early", "op": "<=", "min": 1.5,
|
||||||
|
"why": "late passes must not be much slower than early ones; >1.5 suggests a leak or degrading pool" }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"lock": {
|
||||||
|
"gates": [
|
||||||
|
{ "field": "ok", "op": ">=", "min": 1,
|
||||||
|
"why": "free acquire returns 0 and a contended acquire fast-fails with exit 75" }
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"cache": {
|
||||||
|
"gates": [
|
||||||
|
{ "field": "deduped", "op": ">=", "min": 1,
|
||||||
|
"why": "read_many must collapse a 3x-duplicated path list to the unique set" }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,800 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""bench.py — ECHO performance test suite (timing harness).
|
||||||
|
|
||||||
|
Measures the operations the 1.1.0 (pooling + concurrent read_many + shared cache)
|
||||||
|
and 1.2.0 (fuzzy resolve / alias learning) updates targeted, and gates them on
|
||||||
|
RELATIVE ratios (portable across machines/WAN) rather than absolute ms.
|
||||||
|
|
||||||
|
It reuses the live `echo` client module so it times the REAL pooled/concurrent
|
||||||
|
code path. Reads are non-destructive; write metrics live in a disposable
|
||||||
|
namespace (_agent/_bench/<run-id>/) cleaned up at the end unless --keep.
|
||||||
|
|
||||||
|
This is a MANUAL / pre-release tool, not a CI gate — it depends on the live
|
||||||
|
endpoint (echoapi.alwisp.com) and WAN latency.
|
||||||
|
|
||||||
|
python3 bench.py <metric> [options]
|
||||||
|
python3 bench.py all # the read-only safe set
|
||||||
|
python3 bench.py read-full -n 10
|
||||||
|
python3 bench.py subject-pull
|
||||||
|
python3 bench.py bulk-put -k 50
|
||||||
|
python3 bench.py worker-sweep
|
||||||
|
python3 bench.py resolve # 1.2.0 fuzzy correctness + timing
|
||||||
|
python3 bench.py --list
|
||||||
|
|
||||||
|
Metrics: read-full, subject-pull, bulk-get, bulk-put, bulk-append,
|
||||||
|
pool-warmup, worker-sweep, resolve, index, lint, lock, queue, cache, soak
|
||||||
|
|
||||||
|
Options:
|
||||||
|
-n / --iterations N timed iterations (default 10 read / 5 write)
|
||||||
|
-k / --count K notes/appends for bulk metrics (default 50)
|
||||||
|
--workers W override ECHO_WORKERS for this run
|
||||||
|
--warmup M warm-up iterations to discard (default 2)
|
||||||
|
--soak-seconds S duration for the soak metric (default 60)
|
||||||
|
--keep do NOT delete the _bench namespace after the run
|
||||||
|
--no-gate skip relative-ratio gating (report numbers only)
|
||||||
|
--json-out PATH write the results JSON here (default results/<date>-<commit>.json)
|
||||||
|
--quiet suppress the per-metric human table (JSON only)
|
||||||
|
|
||||||
|
Exit: 0 all gates pass (or --no-gate) · 1 a gate failed · 2 usage/setup error.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import contextlib
|
||||||
|
import datetime as dt
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import statistics
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
HERE = Path(__file__).resolve().parent
|
||||||
|
SCRIPTS = HERE.parent.parent / "scripts"
|
||||||
|
sys.path.insert(0, str(SCRIPTS))
|
||||||
|
|
||||||
|
import echo # noqa: E402 the validated client — real pooled/concurrent path
|
||||||
|
import echo_index # noqa: E402 resolve / fuzzy_candidates (1.2.0)
|
||||||
|
import echo_recall # noqa: E402 recall (search + 1-hop expansion)
|
||||||
|
|
||||||
|
BASELINES = HERE / "baselines.json"
|
||||||
|
RESULTS_DIR = HERE / "results"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# stats + timing
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
def stats(samples: list[float]) -> dict:
|
||||||
|
"""min/median/p90/p95/max/mean in milliseconds from a list of seconds."""
|
||||||
|
if not samples:
|
||||||
|
return {}
|
||||||
|
ms = sorted(s * 1000.0 for s in samples)
|
||||||
|
|
||||||
|
def pct(p: float) -> float:
|
||||||
|
if len(ms) == 1:
|
||||||
|
return ms[0]
|
||||||
|
k = (len(ms) - 1) * p
|
||||||
|
lo, hi = int(k), min(int(k) + 1, len(ms) - 1)
|
||||||
|
return ms[lo] + (ms[hi] - ms[lo]) * (k - lo)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"n": len(ms),
|
||||||
|
"min_ms": round(ms[0], 2),
|
||||||
|
"median_ms": round(statistics.median(ms), 2),
|
||||||
|
"p90_ms": round(pct(0.90), 2),
|
||||||
|
"p95_ms": round(pct(0.95), 2),
|
||||||
|
"max_ms": round(ms[-1], 2),
|
||||||
|
"mean_ms": round(statistics.fmean(ms), 2),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def timed(fn, n: int, warmup: int) -> dict:
|
||||||
|
"""Run fn() warmup+n times. Returns warm-up samples separately from the timed
|
||||||
|
set so cold-start cost (the thing keep-alive kills) is visible, not averaged in."""
|
||||||
|
warm: list[float] = []
|
||||||
|
samples: list[float] = []
|
||||||
|
for i in range(warmup + n):
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
fn()
|
||||||
|
dt_s = time.perf_counter() - t0
|
||||||
|
(warm if i < warmup else samples).append(dt_s)
|
||||||
|
out = stats(samples)
|
||||||
|
if warm:
|
||||||
|
out["warmup"] = stats(warm)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def silent(fn):
|
||||||
|
"""Run fn() with stdout swallowed (recall/resolve print a lot)."""
|
||||||
|
def wrapped():
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
return fn()
|
||||||
|
return wrapped
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# vault enumeration (mirrors sweep.py / vault_lint.py)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
SKIP_BASENAMES = {"README.md"}
|
||||||
|
|
||||||
|
|
||||||
|
def _list_dir(path: str):
|
||||||
|
p = "" if path in ("", "/") else (path if path.endswith("/") else path + "/")
|
||||||
|
body = echo.get_text(p)
|
||||||
|
if body is None:
|
||||||
|
return [], []
|
||||||
|
try:
|
||||||
|
j = json.loads(body)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
return [], []
|
||||||
|
entries = list(j.get("files", [])) + list(j.get("folders", []))
|
||||||
|
files = [e for e in entries if not e.endswith("/")]
|
||||||
|
folders = [e[:-1] for e in entries if e.endswith("/")]
|
||||||
|
return files, folders
|
||||||
|
|
||||||
|
|
||||||
|
def walk_vault(prefix: str = ""):
|
||||||
|
files, folders = _list_dir(prefix)
|
||||||
|
for f in files:
|
||||||
|
yield prefix + f
|
||||||
|
for d in folders:
|
||||||
|
yield from walk_vault(f"{prefix}{d}/")
|
||||||
|
|
||||||
|
|
||||||
|
def all_notes() -> list[str]:
|
||||||
|
return [p for p in walk_vault()
|
||||||
|
if p.endswith(".md") and p.rsplit("/", 1)[-1] not in SKIP_BASENAMES]
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# write primitives (replicate echo.py semantics so we time the real cost)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
def put_verified(path: str, body: str) -> None:
|
||||||
|
"""PUT then read-back GET — the same verify echo.py cmd_put does."""
|
||||||
|
status, b = echo.request("PUT", echo.vault_url(path),
|
||||||
|
data=body.encode(), headers={"Content-Type": "text/markdown"})
|
||||||
|
echo.check(status, b, f"put {path}")
|
||||||
|
status, _ = echo.request("GET", echo.vault_url(path))
|
||||||
|
if status != 200:
|
||||||
|
raise echo.EchoError(f"put {path}: did not verify (GET {status})")
|
||||||
|
|
||||||
|
|
||||||
|
def append_line(path: str, line: str) -> bool:
|
||||||
|
"""Whole-line idempotent append (mirrors echo.py cmd_append). Returns True if a
|
||||||
|
write happened, False if it was an idempotent skip."""
|
||||||
|
status, body = echo.request("GET", echo.vault_url(path))
|
||||||
|
if status == 200:
|
||||||
|
existing = [ln.rstrip("\r") for ln in body.decode(errors="replace").splitlines()]
|
||||||
|
if line.rstrip("\r") in existing:
|
||||||
|
return False
|
||||||
|
status, body = echo.request("POST", echo.vault_url(path),
|
||||||
|
data=f"{line}\n".encode(),
|
||||||
|
headers={"Content-Type": "text/markdown"})
|
||||||
|
echo.check(status, body, f"append {path}")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def delete_path(path: str) -> None:
|
||||||
|
echo.request("DELETE", echo.vault_url(path))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# metrics
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
def m_read_full(ctx) -> dict:
|
||||||
|
"""§3.1 full-vault read: serial GET vs concurrent read_many -> speedup ratio."""
|
||||||
|
paths = all_notes()
|
||||||
|
if not paths:
|
||||||
|
raise echo.EchoError("read-full: no notes enumerated (vault unreachable?)")
|
||||||
|
|
||||||
|
def serial():
|
||||||
|
for p in paths:
|
||||||
|
echo.get_text(p)
|
||||||
|
|
||||||
|
def concurrent():
|
||||||
|
echo.read_many(paths)
|
||||||
|
|
||||||
|
# one warm pass so neither side pays the cold-handshake tax (pool-warmup owns that)
|
||||||
|
echo.get_text(paths[0])
|
||||||
|
# serial is a slow reference baseline (N serial round-trips); one warm pass is
|
||||||
|
# enough to establish the ratio, so we don't multiply the full-vault serial cost.
|
||||||
|
serial_t = timed(serial, n=1, warmup=0)
|
||||||
|
concurrent_t = timed(concurrent, n=ctx.n, warmup=ctx.warmup)
|
||||||
|
ratio = (serial_t["median_ms"] / concurrent_t["median_ms"]) if concurrent_t.get("median_ms") else None
|
||||||
|
return {
|
||||||
|
"note_count": len(paths),
|
||||||
|
"workers": echo.MAX_WORKERS,
|
||||||
|
"serial": serial_t,
|
||||||
|
"concurrent": concurrent_t,
|
||||||
|
"speedup_ratio": round(ratio, 2) if ratio else None,
|
||||||
|
"notes_per_sec_concurrent": round(len(paths) / (concurrent_t["median_ms"] / 1000.0), 1)
|
||||||
|
if concurrent_t.get("median_ms") else None,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def m_subject_pull(ctx) -> dict:
|
||||||
|
"""§3.2 recall a subject: time resolve (floor), search, and full recall, split out."""
|
||||||
|
import fixtures
|
||||||
|
index = echo_index.load()
|
||||||
|
out = {"subjects": []}
|
||||||
|
for label, query in fixtures.SUBJECTS:
|
||||||
|
# floor: O(1) index resolve
|
||||||
|
resolve_t = timed(lambda: echo_index.resolve(index, query), n=ctx.n, warmup=ctx.warmup)
|
||||||
|
# search-only layer
|
||||||
|
def search():
|
||||||
|
with contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
echo.request("POST", f"{echo.BASE}/search/simple/?query={query}")
|
||||||
|
search_t = timed(search, n=ctx.n, warmup=ctx.warmup)
|
||||||
|
# full recall (search + 1-hop expansion reads)
|
||||||
|
recall_t = timed(silent(lambda: echo_recall.recall(query)), n=ctx.n, warmup=ctx.warmup)
|
||||||
|
out["subjects"].append({
|
||||||
|
"label": label, "query": query,
|
||||||
|
"resolve_floor": resolve_t,
|
||||||
|
"search_only": search_t,
|
||||||
|
"recall_total": recall_t,
|
||||||
|
})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def m_bulk_get(ctx) -> dict:
|
||||||
|
"""§3.3 bulk GET at a controlled K: serial vs read_many over fixture notes."""
|
||||||
|
import fixtures
|
||||||
|
paths = _seed_notes(ctx, fixtures)
|
||||||
|
|
||||||
|
def serial():
|
||||||
|
for p in paths:
|
||||||
|
echo.get_text(p)
|
||||||
|
|
||||||
|
def concurrent():
|
||||||
|
echo.read_many(paths)
|
||||||
|
|
||||||
|
serial_t = timed(serial, n=ctx.n, warmup=1)
|
||||||
|
concurrent_t = timed(concurrent, n=ctx.n, warmup=ctx.warmup)
|
||||||
|
ratio = (serial_t["median_ms"] / concurrent_t["median_ms"]) if concurrent_t.get("median_ms") else None
|
||||||
|
return {"k": len(paths), "workers": echo.MAX_WORKERS,
|
||||||
|
"serial": serial_t, "concurrent": concurrent_t,
|
||||||
|
"speedup_ratio": round(ratio, 2) if ratio else None}
|
||||||
|
|
||||||
|
|
||||||
|
def m_bulk_put(ctx) -> dict:
|
||||||
|
"""§3.3 bulk PUT (read-back verified) -> puts/sec + per-put percentiles."""
|
||||||
|
import fixtures
|
||||||
|
k = ctx.k
|
||||||
|
samples: list[float] = []
|
||||||
|
for i in range(k):
|
||||||
|
body = fixtures.note_body(ctx.run_id, i)
|
||||||
|
path = fixtures.note_path(ctx.run_id, i)
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
put_verified(path, body)
|
||||||
|
samples.append(time.perf_counter() - t0)
|
||||||
|
s = stats(samples)
|
||||||
|
total_s = sum(samples)
|
||||||
|
return {"k": k, "per_put": s, "total_ms": round(total_s * 1000, 1),
|
||||||
|
"puts_per_sec": round(k / total_s, 1) if total_s else None}
|
||||||
|
|
||||||
|
|
||||||
|
def m_bulk_append(ctx) -> dict:
|
||||||
|
"""§3.3 bulk APPEND + idempotent-skip path. Confirms idempotency isn't O(file)."""
|
||||||
|
import fixtures
|
||||||
|
target = fixtures.append_target(ctx.run_id)
|
||||||
|
put_verified(target, fixtures.append_target_seed(ctx.run_id))
|
||||||
|
k = ctx.k
|
||||||
|
lines = [f"- 2026-06-23: bench append line {i:04d}" for i in range(k)]
|
||||||
|
|
||||||
|
write_samples: list[float] = []
|
||||||
|
for ln in lines:
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
wrote = append_line(target, ln)
|
||||||
|
write_samples.append(time.perf_counter() - t0)
|
||||||
|
if not wrote:
|
||||||
|
raise echo.EchoError("bulk-append: expected a write but got an idempotent skip")
|
||||||
|
|
||||||
|
# second pass: every line already present -> must skip, and skip cost is the
|
||||||
|
# idempotency GET on a now-larger file. Confirms no superlinear blowup.
|
||||||
|
skip_samples: list[float] = []
|
||||||
|
skips = 0
|
||||||
|
for ln in lines:
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
wrote = append_line(target, ln)
|
||||||
|
skip_samples.append(time.perf_counter() - t0)
|
||||||
|
if not wrote:
|
||||||
|
skips += 1
|
||||||
|
|
||||||
|
return {
|
||||||
|
"k": k,
|
||||||
|
"append_write": stats(write_samples),
|
||||||
|
"append_skip": stats(skip_samples),
|
||||||
|
"skips_confirmed": skips,
|
||||||
|
"all_skipped_second_pass": skips == k,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def m_pool_warmup(ctx) -> dict:
|
||||||
|
"""§4.1 cold vs warm single GET — the clearest proof keep-alive is alive."""
|
||||||
|
paths = all_notes()
|
||||||
|
probe = paths[0] if paths else "_agent/echo-vault.md"
|
||||||
|
echo._drop_connection() # force a cold handshake
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
echo.get_text(probe)
|
||||||
|
cold_ms = (time.perf_counter() - t0) * 1000.0
|
||||||
|
warm = timed(lambda: echo.get_text(probe), n=max(ctx.n, 10), warmup=1)
|
||||||
|
ratio = round(cold_ms / warm["median_ms"], 2) if warm.get("median_ms") else None
|
||||||
|
return {"probe": probe, "cold_ms": round(cold_ms, 2), "warm": warm,
|
||||||
|
"cold_over_warm_ratio": ratio}
|
||||||
|
|
||||||
|
|
||||||
|
def m_worker_sweep(ctx) -> dict:
|
||||||
|
"""§4.2 concurrency scaling: full-vault concurrent read at workers 1,2,4,8,16."""
|
||||||
|
paths = all_notes()
|
||||||
|
if not paths:
|
||||||
|
raise echo.EchoError("worker-sweep: no notes enumerated")
|
||||||
|
original = echo.MAX_WORKERS
|
||||||
|
rows = []
|
||||||
|
try:
|
||||||
|
# w=1 is a full serial pass (slow); the curve only needs the shape, so use a
|
||||||
|
# modest sample count and a single pre-warm rather than per-setting warmups.
|
||||||
|
reps = max(1, ctx.n // 2)
|
||||||
|
for w in (1, 2, 4, 8, 16):
|
||||||
|
echo.MAX_WORKERS = w
|
||||||
|
echo.read_many(paths[:5]) # warm the pool at this worker count
|
||||||
|
t = timed(lambda: echo.read_many(paths), n=reps, warmup=0)
|
||||||
|
rows.append({"workers": w, "median_ms": t["median_ms"], "p95_ms": t["p95_ms"]})
|
||||||
|
finally:
|
||||||
|
echo.MAX_WORKERS = original
|
||||||
|
best = min(rows, key=lambda r: r["median_ms"])
|
||||||
|
return {"note_count": len(paths), "sweep": rows, "knee_workers": best["workers"],
|
||||||
|
"default_workers": original}
|
||||||
|
|
||||||
|
|
||||||
|
def m_resolve(ctx) -> dict:
|
||||||
|
"""§4.3 fuzzy resolve — timing AND correctness against the 1.2.0 guard table."""
|
||||||
|
import fixtures
|
||||||
|
index = echo_index.load()
|
||||||
|
cases = []
|
||||||
|
passes = info = 0
|
||||||
|
for case in fixtures.RESOLVE_CASES:
|
||||||
|
mention = case["mention"]
|
||||||
|
exact_t = timed(lambda: echo_index.resolve(index, mention), n=ctx.n, warmup=ctx.warmup)
|
||||||
|
fuzzy_t = timed(lambda: echo_index.fuzzy_candidates(index, mention), n=ctx.n, warmup=ctx.warmup)
|
||||||
|
_, entity = echo_index.resolve(index, mention)
|
||||||
|
exact_slug = (entity or {}).get("slug") or _slug_from_entity(entity)
|
||||||
|
cands = echo_index.fuzzy_candidates(index, mention) or []
|
||||||
|
cand_slugs = [_cand_slug(c) for c in cands]
|
||||||
|
|
||||||
|
expect, want = case["expect"], case.get("expect_slug")
|
||||||
|
if expect == "exact":
|
||||||
|
ok = exact_slug == want
|
||||||
|
elif expect == "candidates":
|
||||||
|
ok = (exact_slug != want) and (want in cand_slugs)
|
||||||
|
else: # miss
|
||||||
|
ok = entity is None
|
||||||
|
verdict = "PASS" if ok else "INFO" # tune table to live vault before promoting to FAIL
|
||||||
|
passes += ok
|
||||||
|
info += (not ok)
|
||||||
|
cases.append({"mention": mention, "expect": expect, "expect_slug": want,
|
||||||
|
"resolved_slug": exact_slug, "candidate_slugs": cand_slugs[:5],
|
||||||
|
"verdict": verdict,
|
||||||
|
"resolve_timing": exact_t, "fuzzy_timing": fuzzy_t})
|
||||||
|
return {"cases": cases, "passes": passes, "needs_tuning": info,
|
||||||
|
"note": "verdicts are PASS/INFO only; promote to FAIL once the table is "
|
||||||
|
"confirmed against the live vault"}
|
||||||
|
|
||||||
|
|
||||||
|
def _serial_expand(seeds, nmap, base, max_hops):
|
||||||
|
"""The pre-fix serial graph BFS, kept here as the reference baseline so the suite
|
||||||
|
both (a) measures the concurrency speedup and (b) asserts the concurrent
|
||||||
|
expand_graph still returns byte-identical ranking. If echo_recall.expand_graph ever
|
||||||
|
regresses to serial, the ratio gate fails; if its results drift, the parity check fails."""
|
||||||
|
from collections import deque
|
||||||
|
score_of = dict(base)
|
||||||
|
seen = set(seeds)
|
||||||
|
results = {}
|
||||||
|
dq = deque((s, 0) for s in seeds)
|
||||||
|
while dq:
|
||||||
|
path, hop = dq.popleft()
|
||||||
|
if hop >= max_hops:
|
||||||
|
continue
|
||||||
|
text = echo_recall.links.get_text(path)
|
||||||
|
if text is None:
|
||||||
|
continue
|
||||||
|
body = echo_recall.strip_frontmatter(text)
|
||||||
|
targets = set(echo_recall.links.all_wikilinks(body)) | set(echo_recall.links.source_notes(text))
|
||||||
|
parent = score_of.get(path, 1.0)
|
||||||
|
for t in targets:
|
||||||
|
tp = echo_recall._resolve_target(t, nmap)
|
||||||
|
if not tp or tp in seeds:
|
||||||
|
continue
|
||||||
|
decayed = parent * (echo_recall.GRAPH_DECAY ** (hop + 1))
|
||||||
|
if decayed > results.get(tp, (0.0, ""))[0]:
|
||||||
|
results[tp] = (decayed, path)
|
||||||
|
if tp not in seen:
|
||||||
|
seen.add(tp)
|
||||||
|
dq.append((tp, hop + 1))
|
||||||
|
return sorted(results.items(), key=lambda kv: -kv[1][0])
|
||||||
|
|
||||||
|
|
||||||
|
def m_expand_graph(ctx) -> dict:
|
||||||
|
"""recall() graph layer: serial vs concurrent neighbourhood expansion. This is the
|
||||||
|
metric that pins the real recall() bottleneck (the graph BFS, not BM25) and guards
|
||||||
|
the read_many concurrency fix. Also asserts the concurrent path's ranking + scores
|
||||||
|
match the serial reference exactly."""
|
||||||
|
import fixtures
|
||||||
|
ix = echo_recall.load_index()
|
||||||
|
index = echo_index.load()
|
||||||
|
nmap = echo_index.name_map(index)
|
||||||
|
max_hops = echo_recall.MAX_HOPS
|
||||||
|
# hub subjects stress expansion most; fall back to all subjects if none labelled hub
|
||||||
|
subjects = [s for s in fixtures.SUBJECTS if "hub" in s[0]] or fixtures.SUBJECTS
|
||||||
|
rows = []
|
||||||
|
for label, query in subjects:
|
||||||
|
hits = ix.score(query, limit=8)
|
||||||
|
base = {p: s for p, s in hits}
|
||||||
|
seeds = [p for p, _ in hits]
|
||||||
|
if not seeds:
|
||||||
|
continue
|
||||||
|
echo_recall.expand_graph(seeds, nmap, base, max_hops) # warm
|
||||||
|
serial_t = timed(lambda: _serial_expand(seeds, nmap, base, max_hops), n=1, warmup=0)
|
||||||
|
conc_t = timed(lambda: echo_recall.expand_graph(seeds, nmap, base, max_hops),
|
||||||
|
n=ctx.n, warmup=1)
|
||||||
|
old = _serial_expand(seeds, nmap, base, max_hops)
|
||||||
|
new = echo_recall.expand_graph(seeds, nmap, base, max_hops)
|
||||||
|
ranking_ok = [p for p, _ in old] == [p for p, _ in new]
|
||||||
|
scores_ok = ranking_ok and all(
|
||||||
|
abs(old[i][1][0] - new[i][1][0]) < 1e-9 for i in range(len(old)))
|
||||||
|
ratio = (serial_t["median_ms"] / conc_t["median_ms"]) if conc_t.get("median_ms") else None
|
||||||
|
rows.append({"label": label, "query": query, "neighbours": len(new),
|
||||||
|
"serial_ms": serial_t["median_ms"], "concurrent_ms": conc_t["median_ms"],
|
||||||
|
"speedup": round(ratio, 2) if ratio else None,
|
||||||
|
"ranking_identical": ranking_ok, "scores_identical": scores_ok})
|
||||||
|
speedups = [r["speedup"] for r in rows if r["speedup"]]
|
||||||
|
return {"subjects": rows,
|
||||||
|
"min_speedup": round(min(speedups), 2) if speedups else None,
|
||||||
|
"all_ranking_identical": all(r["ranking_identical"] for r in rows),
|
||||||
|
"all_scores_identical": all(r["scores_identical"] for r in rows)}
|
||||||
|
|
||||||
|
|
||||||
|
def _slug_from_entity(entity):
|
||||||
|
if not entity:
|
||||||
|
return None
|
||||||
|
path = entity.get("path", "")
|
||||||
|
return path.rsplit("/", 1)[-1].removesuffix(".md") if path else None
|
||||||
|
|
||||||
|
|
||||||
|
def _cand_slug(c):
|
||||||
|
if isinstance(c, dict):
|
||||||
|
return c.get("slug") or _slug_from_entity(c)
|
||||||
|
if isinstance(c, (list, tuple)) and c:
|
||||||
|
return c[0]
|
||||||
|
return str(c)
|
||||||
|
|
||||||
|
|
||||||
|
def m_index(ctx) -> dict:
|
||||||
|
"""§4.4 entity-index rebuild + link symmetrize timing (sweep.py, dry-run = read-only)."""
|
||||||
|
return _time_script("sweep.py", [])
|
||||||
|
|
||||||
|
|
||||||
|
def m_lint(ctx) -> dict:
|
||||||
|
"""§4.5 full-vault lint timing (vault_lint.py) — regression-gate vs 1.1.0 baseline."""
|
||||||
|
return _time_script("vault_lint.py", [], allow_exit={0, 1})
|
||||||
|
|
||||||
|
|
||||||
|
def _time_script(name: str, extra: list[str], allow_exit=frozenset({0})) -> dict:
|
||||||
|
script = SCRIPTS / name
|
||||||
|
env = dict(os.environ, ECHO_KEY_LEGACY_OK="1")
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
proc = subprocess.run([sys.executable, str(script), *extra],
|
||||||
|
capture_output=True, text=True, env=env, timeout=120)
|
||||||
|
elapsed = (time.perf_counter() - t0) * 1000.0
|
||||||
|
return {"script": name, "elapsed_ms": round(elapsed, 1),
|
||||||
|
"exit_code": proc.returncode,
|
||||||
|
"ok": proc.returncode in allow_exit,
|
||||||
|
"stderr_tail": proc.stderr.strip().splitlines()[-3:] if proc.stderr else []}
|
||||||
|
|
||||||
|
|
||||||
|
def m_lock(ctx) -> dict:
|
||||||
|
"""§4.6 advisory-lock timing: acquire-free, contended (held-fresh), release."""
|
||||||
|
owner = f"bench-{ctx.run_id}"
|
||||||
|
other = f"bench-other-{ctx.run_id}"
|
||||||
|
|
||||||
|
def acquire(o):
|
||||||
|
return echo.cmd_lock(o, quiet=True)
|
||||||
|
|
||||||
|
def release(o):
|
||||||
|
return echo.cmd_unlock(o, quiet=True)
|
||||||
|
|
||||||
|
release(owner); release(other) # clean slate
|
||||||
|
t0 = time.perf_counter(); rc_free = acquire(owner); free_ms = (time.perf_counter() - t0) * 1000
|
||||||
|
t0 = time.perf_counter(); rc_held = acquire(other); held_ms = (time.perf_counter() - t0) * 1000
|
||||||
|
t0 = time.perf_counter(); release(owner); rel_ms = (time.perf_counter() - t0) * 1000
|
||||||
|
return {"acquire_free_ms": round(free_ms, 2), "acquire_free_rc": rc_free,
|
||||||
|
"contended_ms": round(held_ms, 2), "contended_rc_expected_75": rc_held,
|
||||||
|
"release_ms": round(rel_ms, 2),
|
||||||
|
"ok": rc_free == 0 and rc_held == 75}
|
||||||
|
|
||||||
|
|
||||||
|
def m_queue(ctx) -> dict:
|
||||||
|
"""§4.7 offline write-ahead queue: enqueue K writes against an unreachable base,
|
||||||
|
then time the flush replay on 'reconnect'. Uses a bogus base so nothing real is
|
||||||
|
written; restores the base afterward. No production writes occur."""
|
||||||
|
import importlib
|
||||||
|
import echo_queue
|
||||||
|
k = min(ctx.k, 20)
|
||||||
|
saved_base = echo.BASE
|
||||||
|
enqueue_samples: list[float] = []
|
||||||
|
try:
|
||||||
|
for i in range(k):
|
||||||
|
url = f"{echo.BASE}/vault/{fixtures_qpath(ctx.run_id, i)}"
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
echo_queue.enqueue("PUT", url, b"x", {"Content-Type": "text/markdown"},
|
||||||
|
idem_key=f"bench-{ctx.run_id}-{i}")
|
||||||
|
enqueue_samples.append(time.perf_counter() - t0)
|
||||||
|
pending_before = len(echo_queue.pending())
|
||||||
|
# flush is left UNMEASURED-as-success here: with a real endpoint it would
|
||||||
|
# try to write. We only assert enqueue worked and report depth; a true
|
||||||
|
# flush-replay timing needs a sandbox vault (see README "queue caveat").
|
||||||
|
return {"k": k, "enqueue": stats(enqueue_samples),
|
||||||
|
"pending_after_enqueue": pending_before,
|
||||||
|
"note": "flush replay intentionally not run against the live vault; "
|
||||||
|
"enqueue path timed only. See README queue caveat."}
|
||||||
|
finally:
|
||||||
|
echo.BASE = saved_base
|
||||||
|
importlib.reload(echo_queue)
|
||||||
|
|
||||||
|
|
||||||
|
def fixtures_qpath(run_id: str, i: int) -> str:
|
||||||
|
return f"_agent/_bench/{run_id}/queued-{i:04d}.md"
|
||||||
|
|
||||||
|
|
||||||
|
def m_cache(ctx) -> dict:
|
||||||
|
"""§4.8 read_many dedup check: a path list with duplicates must collapse to the
|
||||||
|
unique set (the cache/dedup that stops a sweep re-fetching a link target)."""
|
||||||
|
paths = all_notes()[:20] or ["_agent/echo-vault.md"]
|
||||||
|
dupd = paths + paths + paths # 3x
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
result = echo.read_many(dupd)
|
||||||
|
elapsed = (time.perf_counter() - t0) * 1000
|
||||||
|
return {"requested": len(dupd), "unique": len(set(dupd)),
|
||||||
|
"returned": len(result), "deduped": len(result) == len(set(dupd)),
|
||||||
|
"elapsed_ms": round(elapsed, 2)}
|
||||||
|
|
||||||
|
|
||||||
|
def m_soak(ctx) -> dict:
|
||||||
|
"""§4.10 stability: loop full-vault reads for N seconds; watch for errors/drift."""
|
||||||
|
paths = all_notes()
|
||||||
|
if not paths:
|
||||||
|
raise echo.EchoError("soak: no notes enumerated")
|
||||||
|
deadline = time.time() + ctx.soak_seconds
|
||||||
|
durations: list[float] = []
|
||||||
|
errors = 0
|
||||||
|
while time.time() < deadline:
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
try:
|
||||||
|
res = echo.read_many(paths)
|
||||||
|
if any(v is None for v in res.values()):
|
||||||
|
errors += 1
|
||||||
|
except Exception:
|
||||||
|
errors += 1
|
||||||
|
durations.append(time.perf_counter() - t0)
|
||||||
|
s = stats(durations)
|
||||||
|
# drift = late iterations slower than early ones (leak / pool exhaustion signal)
|
||||||
|
half = len(durations) // 2 or 1
|
||||||
|
early = statistics.fmean(durations[:half]) * 1000
|
||||||
|
late = statistics.fmean(durations[half:]) * 1000
|
||||||
|
return {"seconds": ctx.soak_seconds, "iterations": len(durations), "errors": errors,
|
||||||
|
"per_pass": s, "early_mean_ms": round(early, 2), "late_mean_ms": round(late, 2),
|
||||||
|
"drift_ratio_late_over_early": round(late / early, 2) if early else None}
|
||||||
|
|
||||||
|
|
||||||
|
METRICS = {
|
||||||
|
"read-full": m_read_full,
|
||||||
|
"subject-pull": m_subject_pull,
|
||||||
|
"bulk-get": m_bulk_get,
|
||||||
|
"bulk-put": m_bulk_put,
|
||||||
|
"bulk-append": m_bulk_append,
|
||||||
|
"pool-warmup": m_pool_warmup,
|
||||||
|
"worker-sweep": m_worker_sweep,
|
||||||
|
"resolve": m_resolve,
|
||||||
|
"expand-graph": m_expand_graph,
|
||||||
|
"index": m_index,
|
||||||
|
"lint": m_lint,
|
||||||
|
"lock": m_lock,
|
||||||
|
"queue": m_queue,
|
||||||
|
"cache": m_cache,
|
||||||
|
"soak": m_soak,
|
||||||
|
}
|
||||||
|
|
||||||
|
# the read-only safe set run by `all` (no writes to the vault)
|
||||||
|
READONLY = ["pool-warmup", "read-full", "worker-sweep", "subject-pull",
|
||||||
|
"resolve", "expand-graph", "index", "lint", "cache"]
|
||||||
|
WRITES = {"bulk-get", "bulk-put", "bulk-append", "lock"} # touch the _bench namespace
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# fixture seeding / cleanup
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
def _seed_notes(ctx, fixtures) -> list[str]:
|
||||||
|
"""Ensure K bench notes exist; return their paths. Cached on ctx so bulk-get
|
||||||
|
and bulk-put can share the seed within one run."""
|
||||||
|
if getattr(ctx, "_seeded", None):
|
||||||
|
return ctx._seeded
|
||||||
|
paths = []
|
||||||
|
for i in range(ctx.k):
|
||||||
|
p = fixtures.note_path(ctx.run_id, i)
|
||||||
|
put_verified(p, fixtures.note_body(ctx.run_id, i))
|
||||||
|
paths.append(p)
|
||||||
|
ctx._seeded = paths
|
||||||
|
return paths
|
||||||
|
|
||||||
|
|
||||||
|
def cleanup_namespace(run_id: str) -> int:
|
||||||
|
"""Delete every file under _agent/_bench/<run-id>/. Returns count deleted."""
|
||||||
|
prefix = f"_agent/_bench/{run_id}/"
|
||||||
|
victims = [p for p in walk_vault(prefix)]
|
||||||
|
for p in victims:
|
||||||
|
delete_path(p)
|
||||||
|
# remove the (now-empty) run dir marker if the API created one — best effort
|
||||||
|
return len(victims)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# gating
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
def load_baselines() -> dict:
|
||||||
|
if BASELINES.exists():
|
||||||
|
return json.loads(BASELINES.read_text())
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def gate(metric: str, result: dict, baselines: dict) -> list[dict]:
|
||||||
|
"""Apply relative-ratio gates from baselines.json. Returns a list of checks."""
|
||||||
|
rules = baselines.get(metric, {})
|
||||||
|
checks = []
|
||||||
|
for rule in rules.get("gates", []):
|
||||||
|
path, op, threshold = rule["field"], rule["op"], rule["min"]
|
||||||
|
val = _dig(result, path)
|
||||||
|
if val is None:
|
||||||
|
checks.append({"rule": rule, "value": None, "pass": None, "reason": "field absent"})
|
||||||
|
continue
|
||||||
|
ok = (val >= threshold) if op == ">=" else (val <= threshold)
|
||||||
|
checks.append({"rule": rule, "value": val, "pass": ok})
|
||||||
|
return checks
|
||||||
|
|
||||||
|
|
||||||
|
def _dig(obj, dotted: str):
|
||||||
|
cur = obj
|
||||||
|
for part in dotted.split("."):
|
||||||
|
if isinstance(cur, list):
|
||||||
|
try:
|
||||||
|
cur = cur[int(part)]
|
||||||
|
continue
|
||||||
|
except (ValueError, IndexError):
|
||||||
|
return None
|
||||||
|
if not isinstance(cur, dict) or part not in cur:
|
||||||
|
return None
|
||||||
|
cur = cur[part]
|
||||||
|
return cur
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
class Ctx:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def env_block(args) -> dict:
|
||||||
|
commit = "unknown"
|
||||||
|
try:
|
||||||
|
commit = subprocess.run(["git", "rev-parse", "--short", "HEAD"],
|
||||||
|
cwd=str(SCRIPTS), capture_output=True, text=True,
|
||||||
|
timeout=10).stdout.strip() or "unknown"
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return {
|
||||||
|
"date": dt.datetime.now().isoformat(timespec="seconds"),
|
||||||
|
"endpoint": echo.BASE,
|
||||||
|
"commit": commit,
|
||||||
|
"echo_workers": echo.MAX_WORKERS,
|
||||||
|
"echo_timeout": echo.TIMEOUT,
|
||||||
|
"python": sys.version.split()[0],
|
||||||
|
"iterations": args.iterations,
|
||||||
|
"count_k": args.count,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def human_table(name: str, result: dict, checks: list[dict]) -> str:
|
||||||
|
lines = [f"\n=== {name} ==="]
|
||||||
|
lines.append(json.dumps(result, indent=2))
|
||||||
|
for c in checks:
|
||||||
|
mark = "PASS" if c["pass"] else ("—" if c["pass"] is None else "FAIL")
|
||||||
|
lines.append(f" [{mark}] {c['rule']['field']} {c['rule']['op']} {c['rule']['min']} (got {c['value']})")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv=None) -> int:
|
||||||
|
ap = argparse.ArgumentParser(description="ECHO performance test suite (manual/pre-release)")
|
||||||
|
ap.add_argument("metric", nargs="?", help="metric name, or 'all' for the read-only set")
|
||||||
|
ap.add_argument("--list", action="store_true", help="list metrics and exit")
|
||||||
|
ap.add_argument("-n", "--iterations", type=int, default=None)
|
||||||
|
ap.add_argument("-k", "--count", type=int, default=50)
|
||||||
|
ap.add_argument("--workers", type=int, default=None)
|
||||||
|
ap.add_argument("--warmup", type=int, default=2)
|
||||||
|
ap.add_argument("--soak-seconds", type=int, default=60)
|
||||||
|
ap.add_argument("--keep", action="store_true", help="do not delete the _bench namespace")
|
||||||
|
ap.add_argument("--no-gate", action="store_true")
|
||||||
|
ap.add_argument("--json-out", default=None)
|
||||||
|
ap.add_argument("--quiet", action="store_true")
|
||||||
|
args = ap.parse_args(argv)
|
||||||
|
|
||||||
|
if args.list or not args.metric:
|
||||||
|
print("Metrics:", ", ".join(METRICS))
|
||||||
|
print("Groups : all (read-only safe set:", ", ".join(READONLY) + ")")
|
||||||
|
print("Writes to _bench namespace:", ", ".join(sorted(WRITES)))
|
||||||
|
return 0 if args.list else 2
|
||||||
|
|
||||||
|
if args.workers is not None:
|
||||||
|
echo.MAX_WORKERS = args.workers
|
||||||
|
|
||||||
|
selected = READONLY if args.metric == "all" else [args.metric]
|
||||||
|
for m in selected:
|
||||||
|
if m not in METRICS:
|
||||||
|
print(f"unknown metric: {m}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
ctx = Ctx()
|
||||||
|
ctx.run_id = dt.datetime.now().strftime("%Y%m%d-%H%M%S") + "-" + uuid.uuid4().hex[:6]
|
||||||
|
ctx.k = args.count
|
||||||
|
ctx.warmup = args.warmup
|
||||||
|
ctx.soak_seconds = args.soak_seconds
|
||||||
|
ctx._seeded = None
|
||||||
|
|
||||||
|
baselines = load_baselines()
|
||||||
|
need_writes = any(m in WRITES for m in selected)
|
||||||
|
lock_owner = f"bench-{ctx.run_id}"
|
||||||
|
if need_writes:
|
||||||
|
if echo.cmd_lock(lock_owner, quiet=True) == 75:
|
||||||
|
print("bench: vault lock is held by another session — aborting write metrics.",
|
||||||
|
file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
report = {"env": env_block(args), "run_id": ctx.run_id, "metrics": {}}
|
||||||
|
any_fail = False
|
||||||
|
try:
|
||||||
|
for m in selected:
|
||||||
|
ctx.n = args.iterations if args.iterations is not None else (5 if m in WRITES else 10)
|
||||||
|
try:
|
||||||
|
result = METRICS[m](ctx)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
result = {"error": str(exc)}
|
||||||
|
any_fail = True
|
||||||
|
checks = [] if args.no_gate else gate(m, result, baselines)
|
||||||
|
if any(c["pass"] is False for c in checks):
|
||||||
|
any_fail = True
|
||||||
|
report["metrics"][m] = {"result": result, "gates": checks}
|
||||||
|
if not args.quiet:
|
||||||
|
print(human_table(m, result, checks))
|
||||||
|
finally:
|
||||||
|
if need_writes:
|
||||||
|
echo.cmd_unlock(lock_owner, quiet=True)
|
||||||
|
if need_writes and not args.keep:
|
||||||
|
n = cleanup_namespace(ctx.run_id)
|
||||||
|
report["cleanup"] = {"deleted": n}
|
||||||
|
if not args.quiet:
|
||||||
|
print(f"\nbench: cleaned up {n} files under _agent/_bench/{ctx.run_id}/")
|
||||||
|
elif need_writes and args.keep:
|
||||||
|
report["cleanup"] = {"deleted": 0, "kept": f"_agent/_bench/{ctx.run_id}/"}
|
||||||
|
|
||||||
|
RESULTS_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
out_path = Path(args.json_out) if args.json_out else \
|
||||||
|
RESULTS_DIR / f"{report['env']['date'][:10]}-{report['env']['commit']}.json"
|
||||||
|
out_path.write_text(json.dumps(report, indent=2))
|
||||||
|
print(f"\nbench: results -> {out_path}")
|
||||||
|
return 1 if any_fail else 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""fixtures.py — inputs for the ECHO performance harness (bench.py).
|
||||||
|
|
||||||
|
Holds the *what* (subjects to pull, notes to write, mentions to resolve) so the
|
||||||
|
harness (bench.py) stays the *how* (timing, stats, gating). Nothing here touches
|
||||||
|
the network; bench.py owns all I/O.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
# --- §3.2 subject pulls -------------------------------------------------------
|
||||||
|
# (label, query) pairs fed to `recall`. Chosen to span neighbourhood sizes — the
|
||||||
|
# real cost driver — from a likely-leaf to a known hub. APTA and CapMetro are the
|
||||||
|
# subjects Jason named; "echo" and "operator preferences" are dense hubs that
|
||||||
|
# stress the 1-hop expansion (the read_many fan-out 1.1.0 accelerated).
|
||||||
|
SUBJECTS: list[tuple[str, str]] = [
|
||||||
|
("apta", "APTA"),
|
||||||
|
("capmetro", "CapMetro"),
|
||||||
|
("hub-echo", "echo"),
|
||||||
|
("hub-prefs", "operator preferences"),
|
||||||
|
("leaf-rivnut", "rivnut torque spec"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# --- §4.3 fuzzy-resolve correctness table (1.2.0) -----------------------------
|
||||||
|
# Each case: mention, expected behaviour. `expect_slug` is the canonical slug an
|
||||||
|
# exact/alias/fuzzy match should land on (None = no entity expected). `expect`
|
||||||
|
# is the classifier:
|
||||||
|
# "exact" -> resolve returns an entity whose slug == expect_slug
|
||||||
|
# "candidates" -> no exact hit, but fuzzy_candidates surfaces expect_slug
|
||||||
|
# (the 1.2.0 anti-duplicate guard — a near-miss must NOT resolve
|
||||||
|
# to nothing and silently spawn a new note)
|
||||||
|
# "miss" -> neither; genuinely unknown
|
||||||
|
# Tune expect_slug to the live vault; unknowns are reported as INFO, not failures.
|
||||||
|
RESOLVE_CASES: list[dict] = [
|
||||||
|
{"mention": "echo", "expect": "exact", "expect_slug": "echo"},
|
||||||
|
{"mention": "echo memory", "expect": "candidates", "expect_slug": "echo"},
|
||||||
|
{"mention": "ECHO plugin", "expect": "candidates", "expect_slug": "echo"},
|
||||||
|
{"mention": "goldbrain", "expect": "exact", "expect_slug": "goldbrain"},
|
||||||
|
{"mention": "jason stedwell", "expect": "exact", "expect_slug": "jason-stedwell"},
|
||||||
|
{"mention": "jason", "expect": "candidates", "expect_slug": "jason-stedwell"},
|
||||||
|
{"mention": "zzqx nonexistent entity", "expect": "miss", "expect_slug": None},
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# --- §3.3 bulk write fixtures -------------------------------------------------
|
||||||
|
BENCH_PREFIX = "_agent/_bench" # run-scoped namespace: _agent/_bench/<run-id>/...
|
||||||
|
|
||||||
|
|
||||||
|
def note_body(run_id: str, i: int) -> str:
|
||||||
|
"""A realistic-shape note: canonical frontmatter + a couple of headings, so
|
||||||
|
PUT cost reflects a true note, not a one-liner. agent_written + a _bench tag
|
||||||
|
make these trivially greppable if a cleanup is ever missed."""
|
||||||
|
return (
|
||||||
|
"---\n"
|
||||||
|
"type: working-memory\n"
|
||||||
|
"status: active\n"
|
||||||
|
"created: 2026-06-23\n"
|
||||||
|
"updated: 2026-06-23\n"
|
||||||
|
"tags:\n"
|
||||||
|
" - agent\n"
|
||||||
|
" - _bench\n"
|
||||||
|
"agent_written: true\n"
|
||||||
|
f"source_notes: []\n"
|
||||||
|
f"bench_run: {run_id}\n"
|
||||||
|
"---\n\n"
|
||||||
|
f"# Bench Note {i:04d}\n\n"
|
||||||
|
"## Body\n"
|
||||||
|
f"Synthetic benchmark note {i} for run {run_id}. "
|
||||||
|
"Filler so the payload is not pathologically small: "
|
||||||
|
+ ("lorem ipsum dolor sit amet " * 6)
|
||||||
|
+ "\n\n## Log\n- seed\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def note_path(run_id: str, i: int) -> str:
|
||||||
|
return f"{BENCH_PREFIX}/{run_id}/note-{i:04d}.md"
|
||||||
|
|
||||||
|
|
||||||
|
def append_target(run_id: str) -> str:
|
||||||
|
return f"{BENCH_PREFIX}/{run_id}/append-target.md"
|
||||||
|
|
||||||
|
|
||||||
|
def append_target_seed(run_id: str) -> str:
|
||||||
|
return (
|
||||||
|
"---\n"
|
||||||
|
"type: working-memory\n"
|
||||||
|
"status: active\n"
|
||||||
|
"created: 2026-06-23\n"
|
||||||
|
"updated: 2026-06-23\n"
|
||||||
|
"tags:\n"
|
||||||
|
" - agent\n"
|
||||||
|
" - _bench\n"
|
||||||
|
"agent_written: true\n"
|
||||||
|
"source_notes: []\n"
|
||||||
|
f"bench_run: {run_id}\n"
|
||||||
|
"---\n\n"
|
||||||
|
"# Append Target\n\n"
|
||||||
|
"## Log\n"
|
||||||
|
)
|
||||||
@@ -0,0 +1,931 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:17:09",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": 4,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"metrics": {
|
||||||
|
"bulk-append": {
|
||||||
|
"result": {
|
||||||
|
"k": 30,
|
||||||
|
"append_write": {
|
||||||
|
"n": 30,
|
||||||
|
"min_ms": 87.11,
|
||||||
|
"median_ms": 104.08,
|
||||||
|
"p90_ms": 190.63,
|
||||||
|
"p95_ms": 198.89,
|
||||||
|
"max_ms": 241.91,
|
||||||
|
"mean_ms": 117.22
|
||||||
|
},
|
||||||
|
"append_skip": {
|
||||||
|
"n": 30,
|
||||||
|
"min_ms": 47.57,
|
||||||
|
"median_ms": 51.42,
|
||||||
|
"p90_ms": 59.82,
|
||||||
|
"p95_ms": 61.57,
|
||||||
|
"max_ms": 73.49,
|
||||||
|
"mean_ms": 53.22
|
||||||
|
},
|
||||||
|
"skips_confirmed": 30,
|
||||||
|
"all_skipped_second_pass": true
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "all_skipped_second_pass",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1,
|
||||||
|
"why": "second pass of identical lines must be 100% idempotent skips (boolean true coerced to 1)"
|
||||||
|
},
|
||||||
|
"value": true,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"bulk-get": {
|
||||||
|
"result": {
|
||||||
|
"k": 30,
|
||||||
|
"workers": 8,
|
||||||
|
"serial": {
|
||||||
|
"n": 5,
|
||||||
|
"min_ms": 1520.47,
|
||||||
|
"median_ms": 1606.86,
|
||||||
|
"p90_ms": 1805.72,
|
||||||
|
"p95_ms": 1835.83,
|
||||||
|
"max_ms": 1865.94,
|
||||||
|
"mean_ms": 1658.6,
|
||||||
|
"warmup": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 1601.2,
|
||||||
|
"median_ms": 1601.2,
|
||||||
|
"p90_ms": 1601.2,
|
||||||
|
"p95_ms": 1601.2,
|
||||||
|
"max_ms": 1601.2,
|
||||||
|
"mean_ms": 1601.2
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"concurrent": {
|
||||||
|
"n": 5,
|
||||||
|
"min_ms": 340.31,
|
||||||
|
"median_ms": 356.48,
|
||||||
|
"p90_ms": 358.82,
|
||||||
|
"p95_ms": 359.32,
|
||||||
|
"max_ms": 359.83,
|
||||||
|
"mean_ms": 351.42,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 353.43,
|
||||||
|
"median_ms": 434.94,
|
||||||
|
"p90_ms": 500.15,
|
||||||
|
"p95_ms": 508.3,
|
||||||
|
"max_ms": 516.45,
|
||||||
|
"mean_ms": 434.94
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"speedup_ratio": 4.51
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "speedup_ratio",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1.5,
|
||||||
|
"why": "controlled-K concurrent vs serial; lower bar than full-vault since K is small"
|
||||||
|
},
|
||||||
|
"value": 4.51,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"bulk-put": {
|
||||||
|
"result": {
|
||||||
|
"k": 30,
|
||||||
|
"per_put": {
|
||||||
|
"n": 30,
|
||||||
|
"min_ms": 100.09,
|
||||||
|
"median_ms": 114.98,
|
||||||
|
"p90_ms": 212.46,
|
||||||
|
"p95_ms": 236.18,
|
||||||
|
"max_ms": 330.77,
|
||||||
|
"mean_ms": 143.25
|
||||||
|
},
|
||||||
|
"total_ms": 4297.5,
|
||||||
|
"puts_per_sec": 7.0
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
},
|
||||||
|
"cache": {
|
||||||
|
"result": {
|
||||||
|
"requested": 60,
|
||||||
|
"unique": 20,
|
||||||
|
"returned": 20,
|
||||||
|
"deduped": true,
|
||||||
|
"elapsed_ms": 307.13
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "deduped",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1,
|
||||||
|
"why": "read_many must collapse a 3x-duplicated path list to the unique set"
|
||||||
|
},
|
||||||
|
"value": true,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"expand-graph": {
|
||||||
|
"result": {
|
||||||
|
"subjects": [
|
||||||
|
{
|
||||||
|
"label": "hub-echo",
|
||||||
|
"query": "echo",
|
||||||
|
"neighbours": 120,
|
||||||
|
"serial_ms": 2227.67,
|
||||||
|
"concurrent_ms": 642.2,
|
||||||
|
"speedup": 3.47,
|
||||||
|
"ranking_identical": true,
|
||||||
|
"scores_identical": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "hub-prefs",
|
||||||
|
"query": "operator preferences",
|
||||||
|
"neighbours": 126,
|
||||||
|
"serial_ms": 2910.46,
|
||||||
|
"concurrent_ms": 698.8,
|
||||||
|
"speedup": 4.16,
|
||||||
|
"ranking_identical": true,
|
||||||
|
"scores_identical": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"min_speedup": 3.47,
|
||||||
|
"all_ranking_identical": true,
|
||||||
|
"all_scores_identical": true
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "min_speedup",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 2.0,
|
||||||
|
"why": "recall()'s graph layer must fetch each BFS hop concurrently (read_many), not serially per node; <2x means expand_graph regressed to the pre-fix serial walk"
|
||||||
|
},
|
||||||
|
"value": 3.47,
|
||||||
|
"pass": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "all_ranking_identical",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1,
|
||||||
|
"why": "the concurrent expansion must return the same ranked neighbours as the serial reference"
|
||||||
|
},
|
||||||
|
"value": true,
|
||||||
|
"pass": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "all_scores_identical",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1,
|
||||||
|
"why": "decayed scores must match the serial reference to within float tolerance"
|
||||||
|
},
|
||||||
|
"value": true,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"index": {
|
||||||
|
"result": {
|
||||||
|
"script": "sweep.py",
|
||||||
|
"elapsed_ms": 4629.0,
|
||||||
|
"exit_code": 0,
|
||||||
|
"ok": true,
|
||||||
|
"stderr_tail": []
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
},
|
||||||
|
"lint": {
|
||||||
|
"result": {
|
||||||
|
"script": "vault_lint.py",
|
||||||
|
"elapsed_ms": 5126.4,
|
||||||
|
"exit_code": 1,
|
||||||
|
"ok": true,
|
||||||
|
"stderr_tail": []
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
},
|
||||||
|
"lock": {
|
||||||
|
"result": {
|
||||||
|
"acquire_free_ms": 159.18,
|
||||||
|
"acquire_free_rc": 0,
|
||||||
|
"contended_ms": 57.89,
|
||||||
|
"contended_rc_expected_75": 75,
|
||||||
|
"release_ms": 132.92,
|
||||||
|
"ok": true
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "ok",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1,
|
||||||
|
"why": "free acquire returns 0 and a contended acquire fast-fails with exit 75"
|
||||||
|
},
|
||||||
|
"value": true,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"pool-warmup": {
|
||||||
|
"result": {
|
||||||
|
"probe": "_agent/echo-vault.md",
|
||||||
|
"cold_ms": 163.25,
|
||||||
|
"warm": {
|
||||||
|
"n": 12,
|
||||||
|
"min_ms": 45.64,
|
||||||
|
"median_ms": 50.29,
|
||||||
|
"p90_ms": 52.51,
|
||||||
|
"p95_ms": 53.79,
|
||||||
|
"max_ms": 55.28,
|
||||||
|
"mean_ms": 50.26,
|
||||||
|
"warmup": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 51.68,
|
||||||
|
"median_ms": 51.68,
|
||||||
|
"p90_ms": 51.68,
|
||||||
|
"p95_ms": 51.68,
|
||||||
|
"max_ms": 51.68,
|
||||||
|
"mean_ms": 51.68
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"cold_over_warm_ratio": 3.25
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "cold_over_warm_ratio",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1.5,
|
||||||
|
"why": "cold request must be meaningfully slower than warm \u2014 proves keep-alive is reusing the connection; a ratio near 1.0 means every request is re-handshaking (the pre-1.1.0 bug)"
|
||||||
|
},
|
||||||
|
"value": 3.25,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"queue": {
|
||||||
|
"result": {
|
||||||
|
"k": 10,
|
||||||
|
"enqueue": {
|
||||||
|
"n": 10,
|
||||||
|
"min_ms": 0.07,
|
||||||
|
"median_ms": 0.08,
|
||||||
|
"p90_ms": 0.18,
|
||||||
|
"p95_ms": 0.2,
|
||||||
|
"max_ms": 0.22,
|
||||||
|
"mean_ms": 0.1
|
||||||
|
},
|
||||||
|
"pending_after_enqueue": 10,
|
||||||
|
"note": "flush replay intentionally not run against the live vault; enqueue path timed only. See README queue caveat."
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
},
|
||||||
|
"read-full": {
|
||||||
|
"result": {
|
||||||
|
"note_count": 197,
|
||||||
|
"workers": 8,
|
||||||
|
"serial": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 11376.08,
|
||||||
|
"median_ms": 11376.08,
|
||||||
|
"p90_ms": 11376.08,
|
||||||
|
"p95_ms": 11376.08,
|
||||||
|
"max_ms": 11376.08,
|
||||||
|
"mean_ms": 11376.08
|
||||||
|
},
|
||||||
|
"concurrent": {
|
||||||
|
"n": 4,
|
||||||
|
"min_ms": 1559.79,
|
||||||
|
"median_ms": 1595.31,
|
||||||
|
"p90_ms": 1740.92,
|
||||||
|
"p95_ms": 1768.05,
|
||||||
|
"max_ms": 1795.17,
|
||||||
|
"mean_ms": 1636.4,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 1641.35,
|
||||||
|
"median_ms": 1652.91,
|
||||||
|
"p90_ms": 1662.17,
|
||||||
|
"p95_ms": 1663.32,
|
||||||
|
"max_ms": 1664.48,
|
||||||
|
"mean_ms": 1652.91
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"speedup_ratio": 7.13,
|
||||||
|
"notes_per_sec_concurrent": 123.5
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "speedup_ratio",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1.8,
|
||||||
|
"why": "concurrent read_many must beat serial by the 1.1.0 concurrency margin; <1.8 means concurrency regressed"
|
||||||
|
},
|
||||||
|
"value": 7.13,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"resolve": {
|
||||||
|
"result": {
|
||||||
|
"cases": [
|
||||||
|
{
|
||||||
|
"mention": "echo",
|
||||||
|
"expect": "exact",
|
||||||
|
"expect_slug": "echo",
|
||||||
|
"resolved_slug": "echo",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"2026-06-19-goldbrain-full-echo-architect",
|
||||||
|
"echo",
|
||||||
|
"echo-memory-codex-plugin",
|
||||||
|
"echo-plugin-build",
|
||||||
|
"echo-skill-improvements"
|
||||||
|
],
|
||||||
|
"verdict": "PASS",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.0,
|
||||||
|
"median_ms": 0.0,
|
||||||
|
"p90_ms": 0.0,
|
||||||
|
"p95_ms": 0.0,
|
||||||
|
"max_ms": 0.0,
|
||||||
|
"mean_ms": 0.0,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.01,
|
||||||
|
"median_ms": 0.1,
|
||||||
|
"p90_ms": 0.17,
|
||||||
|
"p95_ms": 0.18,
|
||||||
|
"max_ms": 0.19,
|
||||||
|
"mean_ms": 0.1
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 1.35,
|
||||||
|
"median_ms": 1.49,
|
||||||
|
"p90_ms": 1.67,
|
||||||
|
"p95_ms": 1.68,
|
||||||
|
"max_ms": 1.68,
|
||||||
|
"mean_ms": 1.51,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 1.82,
|
||||||
|
"median_ms": 1.83,
|
||||||
|
"p90_ms": 1.84,
|
||||||
|
"p95_ms": 1.84,
|
||||||
|
"max_ms": 1.85,
|
||||||
|
"mean_ms": 1.83
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "echo memory",
|
||||||
|
"expect": "candidates",
|
||||||
|
"expect_slug": "echo",
|
||||||
|
"resolved_slug": "echo",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"echo",
|
||||||
|
"echo-memory-codex-plugin",
|
||||||
|
"echo-plugin-build",
|
||||||
|
"echo-skill-improvements",
|
||||||
|
"2026-06-19-goldbrain-full-echo-architect"
|
||||||
|
],
|
||||||
|
"verdict": "INFO",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.55,
|
||||||
|
"median_ms": 0.56,
|
||||||
|
"p90_ms": 0.57,
|
||||||
|
"p95_ms": 0.58,
|
||||||
|
"max_ms": 0.58,
|
||||||
|
"mean_ms": 0.56,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.61,
|
||||||
|
"median_ms": 0.61,
|
||||||
|
"p90_ms": 0.61,
|
||||||
|
"p95_ms": 0.61,
|
||||||
|
"max_ms": 0.61,
|
||||||
|
"mean_ms": 0.61
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.97,
|
||||||
|
"median_ms": 0.99,
|
||||||
|
"p90_ms": 1.06,
|
||||||
|
"p95_ms": 1.09,
|
||||||
|
"max_ms": 1.11,
|
||||||
|
"mean_ms": 1.01,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 1.15,
|
||||||
|
"median_ms": 1.18,
|
||||||
|
"p90_ms": 1.2,
|
||||||
|
"p95_ms": 1.21,
|
||||||
|
"max_ms": 1.21,
|
||||||
|
"mean_ms": 1.18
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "ECHO plugin",
|
||||||
|
"expect": "candidates",
|
||||||
|
"expect_slug": "echo",
|
||||||
|
"resolved_slug": "echo",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"echo",
|
||||||
|
"echo-memory-codex-plugin",
|
||||||
|
"echo-plugin-build",
|
||||||
|
"2026-06-19-goldbrain-full-echo-architect",
|
||||||
|
"alabama-wisp-brand-docs"
|
||||||
|
],
|
||||||
|
"verdict": "INFO",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.41,
|
||||||
|
"median_ms": 0.41,
|
||||||
|
"p90_ms": 0.42,
|
||||||
|
"p95_ms": 0.42,
|
||||||
|
"max_ms": 0.42,
|
||||||
|
"mean_ms": 0.41,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.42,
|
||||||
|
"median_ms": 0.45,
|
||||||
|
"p90_ms": 0.47,
|
||||||
|
"p95_ms": 0.48,
|
||||||
|
"max_ms": 0.48,
|
||||||
|
"mean_ms": 0.45
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.79,
|
||||||
|
"median_ms": 0.8,
|
||||||
|
"p90_ms": 0.88,
|
||||||
|
"p95_ms": 0.9,
|
||||||
|
"max_ms": 0.92,
|
||||||
|
"mean_ms": 0.83,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.89,
|
||||||
|
"median_ms": 0.89,
|
||||||
|
"p90_ms": 0.9,
|
||||||
|
"p95_ms": 0.9,
|
||||||
|
"max_ms": 0.9,
|
||||||
|
"mean_ms": 0.89
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "goldbrain",
|
||||||
|
"expect": "exact",
|
||||||
|
"expect_slug": "goldbrain",
|
||||||
|
"resolved_slug": "goldbrain",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"2026-06-19-goldbrain-full-echo-architect",
|
||||||
|
"goldbrain"
|
||||||
|
],
|
||||||
|
"verdict": "PASS",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.0,
|
||||||
|
"median_ms": 0.0,
|
||||||
|
"p90_ms": 0.0,
|
||||||
|
"p95_ms": 0.0,
|
||||||
|
"max_ms": 0.0,
|
||||||
|
"mean_ms": 0.0,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.0,
|
||||||
|
"median_ms": 0.0,
|
||||||
|
"p90_ms": 0.0,
|
||||||
|
"p95_ms": 0.0,
|
||||||
|
"max_ms": 0.0,
|
||||||
|
"mean_ms": 0.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.73,
|
||||||
|
"median_ms": 0.75,
|
||||||
|
"p90_ms": 0.85,
|
||||||
|
"p95_ms": 0.88,
|
||||||
|
"max_ms": 0.91,
|
||||||
|
"mean_ms": 0.78,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.8,
|
||||||
|
"median_ms": 0.81,
|
||||||
|
"p90_ms": 0.82,
|
||||||
|
"p95_ms": 0.82,
|
||||||
|
"max_ms": 0.82,
|
||||||
|
"mean_ms": 0.81
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "jason stedwell",
|
||||||
|
"expect": "exact",
|
||||||
|
"expect_slug": "jason-stedwell",
|
||||||
|
"resolved_slug": "jason-stedwell",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"jason-stedwell",
|
||||||
|
"jason-mcp-gateway"
|
||||||
|
],
|
||||||
|
"verdict": "PASS",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.0,
|
||||||
|
"median_ms": 0.0,
|
||||||
|
"p90_ms": 0.0,
|
||||||
|
"p95_ms": 0.0,
|
||||||
|
"max_ms": 0.0,
|
||||||
|
"mean_ms": 0.0,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.0,
|
||||||
|
"median_ms": 0.0,
|
||||||
|
"p90_ms": 0.0,
|
||||||
|
"p95_ms": 0.0,
|
||||||
|
"max_ms": 0.0,
|
||||||
|
"mean_ms": 0.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.69,
|
||||||
|
"median_ms": 0.72,
|
||||||
|
"p90_ms": 0.75,
|
||||||
|
"p95_ms": 0.75,
|
||||||
|
"max_ms": 0.75,
|
||||||
|
"mean_ms": 0.72,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.74,
|
||||||
|
"median_ms": 0.74,
|
||||||
|
"p90_ms": 0.75,
|
||||||
|
"p95_ms": 0.75,
|
||||||
|
"max_ms": 0.75,
|
||||||
|
"mean_ms": 0.74
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "jason",
|
||||||
|
"expect": "candidates",
|
||||||
|
"expect_slug": "jason-stedwell",
|
||||||
|
"resolved_slug": "jason-stedwell",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"jason-mcp-gateway",
|
||||||
|
"jason-stedwell"
|
||||||
|
],
|
||||||
|
"verdict": "INFO",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.61,
|
||||||
|
"median_ms": 0.65,
|
||||||
|
"p90_ms": 0.66,
|
||||||
|
"p95_ms": 0.67,
|
||||||
|
"max_ms": 0.67,
|
||||||
|
"mean_ms": 0.64,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.66,
|
||||||
|
"median_ms": 0.68,
|
||||||
|
"p90_ms": 0.69,
|
||||||
|
"p95_ms": 0.69,
|
||||||
|
"max_ms": 0.69,
|
||||||
|
"mean_ms": 0.68
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.63,
|
||||||
|
"median_ms": 0.63,
|
||||||
|
"p90_ms": 0.65,
|
||||||
|
"p95_ms": 0.65,
|
||||||
|
"max_ms": 0.66,
|
||||||
|
"mean_ms": 0.64,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.68,
|
||||||
|
"median_ms": 0.69,
|
||||||
|
"p90_ms": 0.7,
|
||||||
|
"p95_ms": 0.7,
|
||||||
|
"max_ms": 0.7,
|
||||||
|
"mean_ms": 0.69
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "zzqx nonexistent entity",
|
||||||
|
"expect": "miss",
|
||||||
|
"expect_slug": null,
|
||||||
|
"resolved_slug": null,
|
||||||
|
"candidate_slugs": [],
|
||||||
|
"verdict": "PASS",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.62,
|
||||||
|
"median_ms": 0.63,
|
||||||
|
"p90_ms": 0.67,
|
||||||
|
"p95_ms": 0.67,
|
||||||
|
"max_ms": 0.68,
|
||||||
|
"mean_ms": 0.64,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.66,
|
||||||
|
"median_ms": 0.67,
|
||||||
|
"p90_ms": 0.67,
|
||||||
|
"p95_ms": 0.67,
|
||||||
|
"max_ms": 0.67,
|
||||||
|
"mean_ms": 0.67
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.61,
|
||||||
|
"median_ms": 0.62,
|
||||||
|
"p90_ms": 0.65,
|
||||||
|
"p95_ms": 0.65,
|
||||||
|
"max_ms": 0.66,
|
||||||
|
"mean_ms": 0.62,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.62,
|
||||||
|
"median_ms": 0.63,
|
||||||
|
"p90_ms": 0.64,
|
||||||
|
"p95_ms": 0.64,
|
||||||
|
"max_ms": 0.64,
|
||||||
|
"mean_ms": 0.63
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"passes": 4,
|
||||||
|
"needs_tuning": 3,
|
||||||
|
"note": "verdicts are PASS/INFO only; promote to FAIL once the table is confirmed against the live vault"
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
},
|
||||||
|
"soak": {
|
||||||
|
"result": {
|
||||||
|
"seconds": 22,
|
||||||
|
"iterations": 15,
|
||||||
|
"errors": 0,
|
||||||
|
"per_pass": {
|
||||||
|
"n": 15,
|
||||||
|
"min_ms": 1472.88,
|
||||||
|
"median_ms": 1551.67,
|
||||||
|
"p90_ms": 1648.45,
|
||||||
|
"p95_ms": 1702.66,
|
||||||
|
"max_ms": 1820.58,
|
||||||
|
"mean_ms": 1569.81
|
||||||
|
},
|
||||||
|
"early_mean_ms": 1575.63,
|
||||||
|
"late_mean_ms": 1564.72,
|
||||||
|
"drift_ratio_late_over_early": 0.99
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "errors",
|
||||||
|
"op": "<=",
|
||||||
|
"min": 0,
|
||||||
|
"why": "no read errors across the soak window \u2014 connection leaks/pool exhaustion would surface here"
|
||||||
|
},
|
||||||
|
"value": 0,
|
||||||
|
"pass": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "drift_ratio_late_over_early",
|
||||||
|
"op": "<=",
|
||||||
|
"min": 1.5,
|
||||||
|
"why": "late passes must not be much slower than early ones; >1.5 suggests a leak or degrading pool"
|
||||||
|
},
|
||||||
|
"value": 0.99,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"subject-pull": {
|
||||||
|
"result": {
|
||||||
|
"subjects": [
|
||||||
|
{
|
||||||
|
"label": "apta",
|
||||||
|
"query": "APTA",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.75,
|
||||||
|
"median_ms": 0.75,
|
||||||
|
"p90_ms": 0.75,
|
||||||
|
"p95_ms": 0.75,
|
||||||
|
"max_ms": 0.75,
|
||||||
|
"mean_ms": 0.75
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 61.87,
|
||||||
|
"median_ms": 61.87,
|
||||||
|
"p90_ms": 61.87,
|
||||||
|
"p95_ms": 61.87,
|
||||||
|
"max_ms": 61.87,
|
||||||
|
"mean_ms": 61.87
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2231.31,
|
||||||
|
"median_ms": 2231.31,
|
||||||
|
"p90_ms": 2231.31,
|
||||||
|
"p95_ms": 2231.31,
|
||||||
|
"max_ms": 2231.31,
|
||||||
|
"mean_ms": 2231.31
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "capmetro",
|
||||||
|
"query": "CapMetro",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.03,
|
||||||
|
"median_ms": 0.03,
|
||||||
|
"p90_ms": 0.03,
|
||||||
|
"p95_ms": 0.03,
|
||||||
|
"max_ms": 0.03,
|
||||||
|
"mean_ms": 0.03
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 57.73,
|
||||||
|
"median_ms": 57.73,
|
||||||
|
"p90_ms": 57.73,
|
||||||
|
"p95_ms": 57.73,
|
||||||
|
"max_ms": 57.73,
|
||||||
|
"mean_ms": 57.73
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2116.06,
|
||||||
|
"median_ms": 2116.06,
|
||||||
|
"p90_ms": 2116.06,
|
||||||
|
"p95_ms": 2116.06,
|
||||||
|
"max_ms": 2116.06,
|
||||||
|
"mean_ms": 2116.06
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "hub-echo",
|
||||||
|
"query": "echo",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.03,
|
||||||
|
"median_ms": 0.03,
|
||||||
|
"p90_ms": 0.03,
|
||||||
|
"p95_ms": 0.03,
|
||||||
|
"max_ms": 0.03,
|
||||||
|
"mean_ms": 0.03
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 70.56,
|
||||||
|
"median_ms": 70.56,
|
||||||
|
"p90_ms": 70.56,
|
||||||
|
"p95_ms": 70.56,
|
||||||
|
"max_ms": 70.56,
|
||||||
|
"mean_ms": 70.56
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2403.62,
|
||||||
|
"median_ms": 2403.62,
|
||||||
|
"p90_ms": 2403.62,
|
||||||
|
"p95_ms": 2403.62,
|
||||||
|
"max_ms": 2403.62,
|
||||||
|
"mean_ms": 2403.62
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "hub-prefs",
|
||||||
|
"query": "operator preferences",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.04,
|
||||||
|
"median_ms": 0.04,
|
||||||
|
"p90_ms": 0.04,
|
||||||
|
"p95_ms": 0.04,
|
||||||
|
"max_ms": 0.04,
|
||||||
|
"mean_ms": 0.04
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2011.13,
|
||||||
|
"median_ms": 2011.13,
|
||||||
|
"p90_ms": 2011.13,
|
||||||
|
"p95_ms": 2011.13,
|
||||||
|
"max_ms": 2011.13,
|
||||||
|
"mean_ms": 2011.13
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2994.84,
|
||||||
|
"median_ms": 2994.84,
|
||||||
|
"p90_ms": 2994.84,
|
||||||
|
"p95_ms": 2994.84,
|
||||||
|
"max_ms": 2994.84,
|
||||||
|
"mean_ms": 2994.84
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "leaf-rivnut",
|
||||||
|
"query": "rivnut torque spec",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2.15,
|
||||||
|
"median_ms": 2.15,
|
||||||
|
"p90_ms": 2.15,
|
||||||
|
"p95_ms": 2.15,
|
||||||
|
"max_ms": 2.15,
|
||||||
|
"mean_ms": 2.15
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2013.58,
|
||||||
|
"median_ms": 2013.58,
|
||||||
|
"p90_ms": 2013.58,
|
||||||
|
"p95_ms": 2013.58,
|
||||||
|
"max_ms": 2013.58,
|
||||||
|
"mean_ms": 2013.58
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2371.32,
|
||||||
|
"median_ms": 2371.32,
|
||||||
|
"p90_ms": 2371.32,
|
||||||
|
"p95_ms": 2371.32,
|
||||||
|
"max_ms": 2371.32,
|
||||||
|
"mean_ms": 2371.32
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
},
|
||||||
|
"worker-sweep": {
|
||||||
|
"result": {
|
||||||
|
"note_count": 197,
|
||||||
|
"sweep": [
|
||||||
|
{
|
||||||
|
"workers": 1,
|
||||||
|
"median_ms": 11917.49,
|
||||||
|
"p95_ms": 11917.49
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"workers": 2,
|
||||||
|
"median_ms": 5740.0,
|
||||||
|
"p95_ms": 5740.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"workers": 4,
|
||||||
|
"median_ms": 3034.3,
|
||||||
|
"p95_ms": 3034.3
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"workers": 8,
|
||||||
|
"median_ms": 1583.83,
|
||||||
|
"p95_ms": 1583.83
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"workers": 16,
|
||||||
|
"median_ms": 1277.77,
|
||||||
|
"p95_ms": 1277.77
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"knee_workers": 16,
|
||||||
|
"default_workers": 8
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:22:00",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": null,
|
||||||
|
"count_k": 30
|
||||||
|
},
|
||||||
|
"run_id": "20260623-202200-882c82",
|
||||||
|
"metrics": {
|
||||||
|
"bulk-append": {
|
||||||
|
"result": {
|
||||||
|
"k": 30,
|
||||||
|
"append_write": {
|
||||||
|
"n": 30,
|
||||||
|
"min_ms": 87.11,
|
||||||
|
"median_ms": 104.08,
|
||||||
|
"p90_ms": 190.63,
|
||||||
|
"p95_ms": 198.89,
|
||||||
|
"max_ms": 241.91,
|
||||||
|
"mean_ms": 117.22
|
||||||
|
},
|
||||||
|
"append_skip": {
|
||||||
|
"n": 30,
|
||||||
|
"min_ms": 47.57,
|
||||||
|
"median_ms": 51.42,
|
||||||
|
"p90_ms": 59.82,
|
||||||
|
"p95_ms": 61.57,
|
||||||
|
"max_ms": 73.49,
|
||||||
|
"mean_ms": 53.22
|
||||||
|
},
|
||||||
|
"skips_confirmed": 30,
|
||||||
|
"all_skipped_second_pass": true
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "all_skipped_second_pass",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1,
|
||||||
|
"why": "second pass of identical lines must be 100% idempotent skips (boolean true coerced to 1)"
|
||||||
|
},
|
||||||
|
"value": true,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"cleanup": {
|
||||||
|
"deleted": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:21:36",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": 5,
|
||||||
|
"count_k": 30
|
||||||
|
},
|
||||||
|
"run_id": "20260623-202136-f2f4dc",
|
||||||
|
"metrics": {
|
||||||
|
"bulk-get": {
|
||||||
|
"result": {
|
||||||
|
"k": 30,
|
||||||
|
"workers": 8,
|
||||||
|
"serial": {
|
||||||
|
"n": 5,
|
||||||
|
"min_ms": 1520.47,
|
||||||
|
"median_ms": 1606.86,
|
||||||
|
"p90_ms": 1805.72,
|
||||||
|
"p95_ms": 1835.83,
|
||||||
|
"max_ms": 1865.94,
|
||||||
|
"mean_ms": 1658.6,
|
||||||
|
"warmup": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 1601.2,
|
||||||
|
"median_ms": 1601.2,
|
||||||
|
"p90_ms": 1601.2,
|
||||||
|
"p95_ms": 1601.2,
|
||||||
|
"max_ms": 1601.2,
|
||||||
|
"mean_ms": 1601.2
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"concurrent": {
|
||||||
|
"n": 5,
|
||||||
|
"min_ms": 340.31,
|
||||||
|
"median_ms": 356.48,
|
||||||
|
"p90_ms": 358.82,
|
||||||
|
"p95_ms": 359.32,
|
||||||
|
"max_ms": 359.83,
|
||||||
|
"mean_ms": 351.42,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 353.43,
|
||||||
|
"median_ms": 434.94,
|
||||||
|
"p90_ms": 500.15,
|
||||||
|
"p95_ms": 508.3,
|
||||||
|
"max_ms": 516.45,
|
||||||
|
"mean_ms": 434.94
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"speedup_ratio": 4.51
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "speedup_ratio",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1.5,
|
||||||
|
"why": "controlled-K concurrent vs serial; lower bar than full-vault since K is small"
|
||||||
|
},
|
||||||
|
"value": 4.51,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"cleanup": {
|
||||||
|
"deleted": 30
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:21:21",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": null,
|
||||||
|
"count_k": 30
|
||||||
|
},
|
||||||
|
"run_id": "20260623-202121-23b5ac",
|
||||||
|
"metrics": {
|
||||||
|
"bulk-put": {
|
||||||
|
"result": {
|
||||||
|
"k": 30,
|
||||||
|
"per_put": {
|
||||||
|
"n": 30,
|
||||||
|
"min_ms": 100.09,
|
||||||
|
"median_ms": 114.98,
|
||||||
|
"p90_ms": 212.46,
|
||||||
|
"p95_ms": 236.18,
|
||||||
|
"max_ms": 330.77,
|
||||||
|
"mean_ms": 143.25
|
||||||
|
},
|
||||||
|
"total_ms": 4297.5,
|
||||||
|
"puts_per_sec": 7.0
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"cleanup": {
|
||||||
|
"deleted": 30
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:12:16",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": null,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-201216-e77971",
|
||||||
|
"metrics": {
|
||||||
|
"cache": {
|
||||||
|
"result": {
|
||||||
|
"requested": 60,
|
||||||
|
"unique": 20,
|
||||||
|
"returned": 20,
|
||||||
|
"deduped": true,
|
||||||
|
"elapsed_ms": 307.13
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "deduped",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1,
|
||||||
|
"why": "read_many must collapse a 3x-duplicated path list to the unique set"
|
||||||
|
},
|
||||||
|
"value": true,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T21:22:29",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": 3,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-212229-8c3ce2",
|
||||||
|
"metrics": {
|
||||||
|
"expand-graph": {
|
||||||
|
"result": {
|
||||||
|
"subjects": [
|
||||||
|
{
|
||||||
|
"label": "hub-echo",
|
||||||
|
"query": "echo",
|
||||||
|
"neighbours": 120,
|
||||||
|
"serial_ms": 2227.67,
|
||||||
|
"concurrent_ms": 642.2,
|
||||||
|
"speedup": 3.47,
|
||||||
|
"ranking_identical": true,
|
||||||
|
"scores_identical": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "hub-prefs",
|
||||||
|
"query": "operator preferences",
|
||||||
|
"neighbours": 126,
|
||||||
|
"serial_ms": 2910.46,
|
||||||
|
"concurrent_ms": 698.8,
|
||||||
|
"speedup": 4.16,
|
||||||
|
"ranking_identical": true,
|
||||||
|
"scores_identical": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"min_speedup": 3.47,
|
||||||
|
"all_ranking_identical": true,
|
||||||
|
"all_scores_identical": true
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "min_speedup",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 2.0,
|
||||||
|
"why": "recall()'s graph layer must fetch each BFS hop concurrently (read_many), not serially per node; <2x means expand_graph regressed to the pre-fix serial walk"
|
||||||
|
},
|
||||||
|
"value": 3.47,
|
||||||
|
"pass": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "all_ranking_identical",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1,
|
||||||
|
"why": "the concurrent expansion must return the same ranked neighbours as the serial reference"
|
||||||
|
},
|
||||||
|
"value": true,
|
||||||
|
"pass": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "all_scores_identical",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1,
|
||||||
|
"why": "decayed scores must match the serial reference to within float tolerance"
|
||||||
|
},
|
||||||
|
"value": true,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:20:57",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": null,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-202057-c41a52",
|
||||||
|
"metrics": {
|
||||||
|
"index": {
|
||||||
|
"result": {
|
||||||
|
"script": "sweep.py",
|
||||||
|
"elapsed_ms": 4629.0,
|
||||||
|
"exit_code": 0,
|
||||||
|
"ok": true,
|
||||||
|
"stderr_tail": []
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:21:01",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": null,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-202101-569382",
|
||||||
|
"metrics": {
|
||||||
|
"lint": {
|
||||||
|
"result": {
|
||||||
|
"script": "vault_lint.py",
|
||||||
|
"elapsed_ms": 5126.4,
|
||||||
|
"exit_code": 1,
|
||||||
|
"ok": true,
|
||||||
|
"stderr_tail": []
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:21:28",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": null,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-202128-53b113",
|
||||||
|
"metrics": {
|
||||||
|
"lock": {
|
||||||
|
"result": {
|
||||||
|
"acquire_free_ms": 159.18,
|
||||||
|
"acquire_free_rc": 0,
|
||||||
|
"contended_ms": 57.89,
|
||||||
|
"contended_rc_expected_75": 75,
|
||||||
|
"release_ms": 132.92,
|
||||||
|
"ok": true
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "ok",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1,
|
||||||
|
"why": "free acquire returns 0 and a contended acquire fast-fails with exit 75"
|
||||||
|
},
|
||||||
|
"value": true,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"cleanup": {
|
||||||
|
"deleted": 0
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:12:12",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": 12,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-201212-30c2b8",
|
||||||
|
"metrics": {
|
||||||
|
"pool-warmup": {
|
||||||
|
"result": {
|
||||||
|
"probe": "_agent/echo-vault.md",
|
||||||
|
"cold_ms": 163.25,
|
||||||
|
"warm": {
|
||||||
|
"n": 12,
|
||||||
|
"min_ms": 45.64,
|
||||||
|
"median_ms": 50.29,
|
||||||
|
"p90_ms": 52.51,
|
||||||
|
"p95_ms": 53.79,
|
||||||
|
"max_ms": 55.28,
|
||||||
|
"mean_ms": 50.26,
|
||||||
|
"warmup": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 51.68,
|
||||||
|
"median_ms": 51.68,
|
||||||
|
"p90_ms": 51.68,
|
||||||
|
"p95_ms": 51.68,
|
||||||
|
"max_ms": 51.68,
|
||||||
|
"mean_ms": 51.68
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"cold_over_warm_ratio": 3.25
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "cold_over_warm_ratio",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1.5,
|
||||||
|
"why": "cold request must be meaningfully slower than warm \u2014 proves keep-alive is reusing the connection; a ratio near 1.0 means every request is re-handshaking (the pre-1.1.0 bug)"
|
||||||
|
},
|
||||||
|
"value": 3.25,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:22:05",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": null,
|
||||||
|
"count_k": 10
|
||||||
|
},
|
||||||
|
"run_id": "20260623-202205-378725",
|
||||||
|
"metrics": {
|
||||||
|
"queue": {
|
||||||
|
"result": {
|
||||||
|
"k": 10,
|
||||||
|
"enqueue": {
|
||||||
|
"n": 10,
|
||||||
|
"min_ms": 0.07,
|
||||||
|
"median_ms": 0.08,
|
||||||
|
"p90_ms": 0.18,
|
||||||
|
"p95_ms": 0.2,
|
||||||
|
"max_ms": 0.22,
|
||||||
|
"mean_ms": 0.1
|
||||||
|
},
|
||||||
|
"pending_after_enqueue": 10,
|
||||||
|
"note": "flush replay intentionally not run against the live vault; enqueue path timed only. See README queue caveat."
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:17:09",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": 4,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-201709-b08a2e",
|
||||||
|
"metrics": {
|
||||||
|
"read-full": {
|
||||||
|
"result": {
|
||||||
|
"note_count": 197,
|
||||||
|
"workers": 8,
|
||||||
|
"serial": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 11376.08,
|
||||||
|
"median_ms": 11376.08,
|
||||||
|
"p90_ms": 11376.08,
|
||||||
|
"p95_ms": 11376.08,
|
||||||
|
"max_ms": 11376.08,
|
||||||
|
"mean_ms": 11376.08
|
||||||
|
},
|
||||||
|
"concurrent": {
|
||||||
|
"n": 4,
|
||||||
|
"min_ms": 1559.79,
|
||||||
|
"median_ms": 1595.31,
|
||||||
|
"p90_ms": 1740.92,
|
||||||
|
"p95_ms": 1768.05,
|
||||||
|
"max_ms": 1795.17,
|
||||||
|
"mean_ms": 1636.4,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 1641.35,
|
||||||
|
"median_ms": 1652.91,
|
||||||
|
"p90_ms": 1662.17,
|
||||||
|
"p95_ms": 1663.32,
|
||||||
|
"max_ms": 1664.48,
|
||||||
|
"mean_ms": 1652.91
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"speedup_ratio": 7.13,
|
||||||
|
"notes_per_sec_concurrent": 123.5
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "speedup_ratio",
|
||||||
|
"op": ">=",
|
||||||
|
"min": 1.8,
|
||||||
|
"why": "concurrent read_many must beat serial by the 1.1.0 concurrency margin; <1.8 means concurrency regressed"
|
||||||
|
},
|
||||||
|
"value": 7.13,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,360 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:12:19",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": 6,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-201219-819b5e",
|
||||||
|
"metrics": {
|
||||||
|
"resolve": {
|
||||||
|
"result": {
|
||||||
|
"cases": [
|
||||||
|
{
|
||||||
|
"mention": "echo",
|
||||||
|
"expect": "exact",
|
||||||
|
"expect_slug": "echo",
|
||||||
|
"resolved_slug": "echo",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"2026-06-19-goldbrain-full-echo-architect",
|
||||||
|
"echo",
|
||||||
|
"echo-memory-codex-plugin",
|
||||||
|
"echo-plugin-build",
|
||||||
|
"echo-skill-improvements"
|
||||||
|
],
|
||||||
|
"verdict": "PASS",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.0,
|
||||||
|
"median_ms": 0.0,
|
||||||
|
"p90_ms": 0.0,
|
||||||
|
"p95_ms": 0.0,
|
||||||
|
"max_ms": 0.0,
|
||||||
|
"mean_ms": 0.0,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.01,
|
||||||
|
"median_ms": 0.1,
|
||||||
|
"p90_ms": 0.17,
|
||||||
|
"p95_ms": 0.18,
|
||||||
|
"max_ms": 0.19,
|
||||||
|
"mean_ms": 0.1
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 1.35,
|
||||||
|
"median_ms": 1.49,
|
||||||
|
"p90_ms": 1.67,
|
||||||
|
"p95_ms": 1.68,
|
||||||
|
"max_ms": 1.68,
|
||||||
|
"mean_ms": 1.51,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 1.82,
|
||||||
|
"median_ms": 1.83,
|
||||||
|
"p90_ms": 1.84,
|
||||||
|
"p95_ms": 1.84,
|
||||||
|
"max_ms": 1.85,
|
||||||
|
"mean_ms": 1.83
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "echo memory",
|
||||||
|
"expect": "candidates",
|
||||||
|
"expect_slug": "echo",
|
||||||
|
"resolved_slug": "echo",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"echo",
|
||||||
|
"echo-memory-codex-plugin",
|
||||||
|
"echo-plugin-build",
|
||||||
|
"echo-skill-improvements",
|
||||||
|
"2026-06-19-goldbrain-full-echo-architect"
|
||||||
|
],
|
||||||
|
"verdict": "INFO",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.55,
|
||||||
|
"median_ms": 0.56,
|
||||||
|
"p90_ms": 0.57,
|
||||||
|
"p95_ms": 0.58,
|
||||||
|
"max_ms": 0.58,
|
||||||
|
"mean_ms": 0.56,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.61,
|
||||||
|
"median_ms": 0.61,
|
||||||
|
"p90_ms": 0.61,
|
||||||
|
"p95_ms": 0.61,
|
||||||
|
"max_ms": 0.61,
|
||||||
|
"mean_ms": 0.61
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.97,
|
||||||
|
"median_ms": 0.99,
|
||||||
|
"p90_ms": 1.06,
|
||||||
|
"p95_ms": 1.09,
|
||||||
|
"max_ms": 1.11,
|
||||||
|
"mean_ms": 1.01,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 1.15,
|
||||||
|
"median_ms": 1.18,
|
||||||
|
"p90_ms": 1.2,
|
||||||
|
"p95_ms": 1.21,
|
||||||
|
"max_ms": 1.21,
|
||||||
|
"mean_ms": 1.18
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "ECHO plugin",
|
||||||
|
"expect": "candidates",
|
||||||
|
"expect_slug": "echo",
|
||||||
|
"resolved_slug": "echo",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"echo",
|
||||||
|
"echo-memory-codex-plugin",
|
||||||
|
"echo-plugin-build",
|
||||||
|
"2026-06-19-goldbrain-full-echo-architect",
|
||||||
|
"alabama-wisp-brand-docs"
|
||||||
|
],
|
||||||
|
"verdict": "INFO",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.41,
|
||||||
|
"median_ms": 0.41,
|
||||||
|
"p90_ms": 0.42,
|
||||||
|
"p95_ms": 0.42,
|
||||||
|
"max_ms": 0.42,
|
||||||
|
"mean_ms": 0.41,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.42,
|
||||||
|
"median_ms": 0.45,
|
||||||
|
"p90_ms": 0.47,
|
||||||
|
"p95_ms": 0.48,
|
||||||
|
"max_ms": 0.48,
|
||||||
|
"mean_ms": 0.45
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.79,
|
||||||
|
"median_ms": 0.8,
|
||||||
|
"p90_ms": 0.88,
|
||||||
|
"p95_ms": 0.9,
|
||||||
|
"max_ms": 0.92,
|
||||||
|
"mean_ms": 0.83,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.89,
|
||||||
|
"median_ms": 0.89,
|
||||||
|
"p90_ms": 0.9,
|
||||||
|
"p95_ms": 0.9,
|
||||||
|
"max_ms": 0.9,
|
||||||
|
"mean_ms": 0.89
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "goldbrain",
|
||||||
|
"expect": "exact",
|
||||||
|
"expect_slug": "goldbrain",
|
||||||
|
"resolved_slug": "goldbrain",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"2026-06-19-goldbrain-full-echo-architect",
|
||||||
|
"goldbrain"
|
||||||
|
],
|
||||||
|
"verdict": "PASS",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.0,
|
||||||
|
"median_ms": 0.0,
|
||||||
|
"p90_ms": 0.0,
|
||||||
|
"p95_ms": 0.0,
|
||||||
|
"max_ms": 0.0,
|
||||||
|
"mean_ms": 0.0,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.0,
|
||||||
|
"median_ms": 0.0,
|
||||||
|
"p90_ms": 0.0,
|
||||||
|
"p95_ms": 0.0,
|
||||||
|
"max_ms": 0.0,
|
||||||
|
"mean_ms": 0.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.73,
|
||||||
|
"median_ms": 0.75,
|
||||||
|
"p90_ms": 0.85,
|
||||||
|
"p95_ms": 0.88,
|
||||||
|
"max_ms": 0.91,
|
||||||
|
"mean_ms": 0.78,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.8,
|
||||||
|
"median_ms": 0.81,
|
||||||
|
"p90_ms": 0.82,
|
||||||
|
"p95_ms": 0.82,
|
||||||
|
"max_ms": 0.82,
|
||||||
|
"mean_ms": 0.81
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "jason stedwell",
|
||||||
|
"expect": "exact",
|
||||||
|
"expect_slug": "jason-stedwell",
|
||||||
|
"resolved_slug": "jason-stedwell",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"jason-stedwell",
|
||||||
|
"jason-mcp-gateway"
|
||||||
|
],
|
||||||
|
"verdict": "PASS",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.0,
|
||||||
|
"median_ms": 0.0,
|
||||||
|
"p90_ms": 0.0,
|
||||||
|
"p95_ms": 0.0,
|
||||||
|
"max_ms": 0.0,
|
||||||
|
"mean_ms": 0.0,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.0,
|
||||||
|
"median_ms": 0.0,
|
||||||
|
"p90_ms": 0.0,
|
||||||
|
"p95_ms": 0.0,
|
||||||
|
"max_ms": 0.0,
|
||||||
|
"mean_ms": 0.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.69,
|
||||||
|
"median_ms": 0.72,
|
||||||
|
"p90_ms": 0.75,
|
||||||
|
"p95_ms": 0.75,
|
||||||
|
"max_ms": 0.75,
|
||||||
|
"mean_ms": 0.72,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.74,
|
||||||
|
"median_ms": 0.74,
|
||||||
|
"p90_ms": 0.75,
|
||||||
|
"p95_ms": 0.75,
|
||||||
|
"max_ms": 0.75,
|
||||||
|
"mean_ms": 0.74
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "jason",
|
||||||
|
"expect": "candidates",
|
||||||
|
"expect_slug": "jason-stedwell",
|
||||||
|
"resolved_slug": "jason-stedwell",
|
||||||
|
"candidate_slugs": [
|
||||||
|
"jason-mcp-gateway",
|
||||||
|
"jason-stedwell"
|
||||||
|
],
|
||||||
|
"verdict": "INFO",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.61,
|
||||||
|
"median_ms": 0.65,
|
||||||
|
"p90_ms": 0.66,
|
||||||
|
"p95_ms": 0.67,
|
||||||
|
"max_ms": 0.67,
|
||||||
|
"mean_ms": 0.64,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.66,
|
||||||
|
"median_ms": 0.68,
|
||||||
|
"p90_ms": 0.69,
|
||||||
|
"p95_ms": 0.69,
|
||||||
|
"max_ms": 0.69,
|
||||||
|
"mean_ms": 0.68
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.63,
|
||||||
|
"median_ms": 0.63,
|
||||||
|
"p90_ms": 0.65,
|
||||||
|
"p95_ms": 0.65,
|
||||||
|
"max_ms": 0.66,
|
||||||
|
"mean_ms": 0.64,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.68,
|
||||||
|
"median_ms": 0.69,
|
||||||
|
"p90_ms": 0.7,
|
||||||
|
"p95_ms": 0.7,
|
||||||
|
"max_ms": 0.7,
|
||||||
|
"mean_ms": 0.69
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"mention": "zzqx nonexistent entity",
|
||||||
|
"expect": "miss",
|
||||||
|
"expect_slug": null,
|
||||||
|
"resolved_slug": null,
|
||||||
|
"candidate_slugs": [],
|
||||||
|
"verdict": "PASS",
|
||||||
|
"resolve_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.62,
|
||||||
|
"median_ms": 0.63,
|
||||||
|
"p90_ms": 0.67,
|
||||||
|
"p95_ms": 0.67,
|
||||||
|
"max_ms": 0.68,
|
||||||
|
"mean_ms": 0.64,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.66,
|
||||||
|
"median_ms": 0.67,
|
||||||
|
"p90_ms": 0.67,
|
||||||
|
"p95_ms": 0.67,
|
||||||
|
"max_ms": 0.67,
|
||||||
|
"mean_ms": 0.67
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fuzzy_timing": {
|
||||||
|
"n": 6,
|
||||||
|
"min_ms": 0.61,
|
||||||
|
"median_ms": 0.62,
|
||||||
|
"p90_ms": 0.65,
|
||||||
|
"p95_ms": 0.65,
|
||||||
|
"max_ms": 0.66,
|
||||||
|
"mean_ms": 0.62,
|
||||||
|
"warmup": {
|
||||||
|
"n": 2,
|
||||||
|
"min_ms": 0.62,
|
||||||
|
"median_ms": 0.63,
|
||||||
|
"p90_ms": 0.64,
|
||||||
|
"p95_ms": 0.64,
|
||||||
|
"max_ms": 0.64,
|
||||||
|
"mean_ms": 0.63
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"passes": 4,
|
||||||
|
"needs_tuning": 3,
|
||||||
|
"note": "verdicts are PASS/INFO only; promote to FAIL once the table is confirmed against the live vault"
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:22:15",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": null,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-202215-7b414f",
|
||||||
|
"metrics": {
|
||||||
|
"soak": {
|
||||||
|
"result": {
|
||||||
|
"seconds": 22,
|
||||||
|
"iterations": 15,
|
||||||
|
"errors": 0,
|
||||||
|
"per_pass": {
|
||||||
|
"n": 15,
|
||||||
|
"min_ms": 1472.88,
|
||||||
|
"median_ms": 1551.67,
|
||||||
|
"p90_ms": 1648.45,
|
||||||
|
"p95_ms": 1702.66,
|
||||||
|
"max_ms": 1820.58,
|
||||||
|
"mean_ms": 1569.81
|
||||||
|
},
|
||||||
|
"early_mean_ms": 1575.63,
|
||||||
|
"late_mean_ms": 1564.72,
|
||||||
|
"drift_ratio_late_over_early": 0.99
|
||||||
|
},
|
||||||
|
"gates": [
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "errors",
|
||||||
|
"op": "<=",
|
||||||
|
"min": 0,
|
||||||
|
"why": "no read errors across the soak window \u2014 connection leaks/pool exhaustion would surface here"
|
||||||
|
},
|
||||||
|
"value": 0,
|
||||||
|
"pass": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"rule": {
|
||||||
|
"field": "drift_ratio_late_over_early",
|
||||||
|
"op": "<=",
|
||||||
|
"min": 1.5,
|
||||||
|
"why": "late passes must not be much slower than early ones; >1.5 suggests a leak or degrading pool"
|
||||||
|
},
|
||||||
|
"value": 0.99,
|
||||||
|
"pass": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
+177
@@ -0,0 +1,177 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T21:18:12",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": 1,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-211812-ea4b36",
|
||||||
|
"metrics": {
|
||||||
|
"subject-pull": {
|
||||||
|
"result": {
|
||||||
|
"subjects": [
|
||||||
|
{
|
||||||
|
"label": "apta",
|
||||||
|
"query": "APTA",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.75,
|
||||||
|
"median_ms": 0.75,
|
||||||
|
"p90_ms": 0.75,
|
||||||
|
"p95_ms": 0.75,
|
||||||
|
"max_ms": 0.75,
|
||||||
|
"mean_ms": 0.75
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 61.87,
|
||||||
|
"median_ms": 61.87,
|
||||||
|
"p90_ms": 61.87,
|
||||||
|
"p95_ms": 61.87,
|
||||||
|
"max_ms": 61.87,
|
||||||
|
"mean_ms": 61.87
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2231.31,
|
||||||
|
"median_ms": 2231.31,
|
||||||
|
"p90_ms": 2231.31,
|
||||||
|
"p95_ms": 2231.31,
|
||||||
|
"max_ms": 2231.31,
|
||||||
|
"mean_ms": 2231.31
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "capmetro",
|
||||||
|
"query": "CapMetro",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.03,
|
||||||
|
"median_ms": 0.03,
|
||||||
|
"p90_ms": 0.03,
|
||||||
|
"p95_ms": 0.03,
|
||||||
|
"max_ms": 0.03,
|
||||||
|
"mean_ms": 0.03
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 57.73,
|
||||||
|
"median_ms": 57.73,
|
||||||
|
"p90_ms": 57.73,
|
||||||
|
"p95_ms": 57.73,
|
||||||
|
"max_ms": 57.73,
|
||||||
|
"mean_ms": 57.73
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2116.06,
|
||||||
|
"median_ms": 2116.06,
|
||||||
|
"p90_ms": 2116.06,
|
||||||
|
"p95_ms": 2116.06,
|
||||||
|
"max_ms": 2116.06,
|
||||||
|
"mean_ms": 2116.06
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "hub-echo",
|
||||||
|
"query": "echo",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.03,
|
||||||
|
"median_ms": 0.03,
|
||||||
|
"p90_ms": 0.03,
|
||||||
|
"p95_ms": 0.03,
|
||||||
|
"max_ms": 0.03,
|
||||||
|
"mean_ms": 0.03
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 70.56,
|
||||||
|
"median_ms": 70.56,
|
||||||
|
"p90_ms": 70.56,
|
||||||
|
"p95_ms": 70.56,
|
||||||
|
"max_ms": 70.56,
|
||||||
|
"mean_ms": 70.56
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2403.62,
|
||||||
|
"median_ms": 2403.62,
|
||||||
|
"p90_ms": 2403.62,
|
||||||
|
"p95_ms": 2403.62,
|
||||||
|
"max_ms": 2403.62,
|
||||||
|
"mean_ms": 2403.62
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "hub-prefs",
|
||||||
|
"query": "operator preferences",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.04,
|
||||||
|
"median_ms": 0.04,
|
||||||
|
"p90_ms": 0.04,
|
||||||
|
"p95_ms": 0.04,
|
||||||
|
"max_ms": 0.04,
|
||||||
|
"mean_ms": 0.04
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2011.13,
|
||||||
|
"median_ms": 2011.13,
|
||||||
|
"p90_ms": 2011.13,
|
||||||
|
"p95_ms": 2011.13,
|
||||||
|
"max_ms": 2011.13,
|
||||||
|
"mean_ms": 2011.13
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2994.84,
|
||||||
|
"median_ms": 2994.84,
|
||||||
|
"p90_ms": 2994.84,
|
||||||
|
"p95_ms": 2994.84,
|
||||||
|
"max_ms": 2994.84,
|
||||||
|
"mean_ms": 2994.84
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "leaf-rivnut",
|
||||||
|
"query": "rivnut torque spec",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2.15,
|
||||||
|
"median_ms": 2.15,
|
||||||
|
"p90_ms": 2.15,
|
||||||
|
"p95_ms": 2.15,
|
||||||
|
"max_ms": 2.15,
|
||||||
|
"mean_ms": 2.15
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2013.58,
|
||||||
|
"median_ms": 2013.58,
|
||||||
|
"p90_ms": 2013.58,
|
||||||
|
"p95_ms": 2013.58,
|
||||||
|
"max_ms": 2013.58,
|
||||||
|
"mean_ms": 2013.58
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2371.32,
|
||||||
|
"median_ms": 2371.32,
|
||||||
|
"p90_ms": 2371.32,
|
||||||
|
"p95_ms": 2371.32,
|
||||||
|
"max_ms": 2371.32,
|
||||||
|
"mean_ms": 2371.32
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,177 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:20:23",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": 1,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-202023-274ddd",
|
||||||
|
"metrics": {
|
||||||
|
"subject-pull": {
|
||||||
|
"result": {
|
||||||
|
"subjects": [
|
||||||
|
{
|
||||||
|
"label": "apta",
|
||||||
|
"query": "APTA",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.97,
|
||||||
|
"median_ms": 0.97,
|
||||||
|
"p90_ms": 0.97,
|
||||||
|
"p95_ms": 0.97,
|
||||||
|
"max_ms": 0.97,
|
||||||
|
"mean_ms": 0.97
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 71.55,
|
||||||
|
"median_ms": 71.55,
|
||||||
|
"p90_ms": 71.55,
|
||||||
|
"p95_ms": 71.55,
|
||||||
|
"max_ms": 71.55,
|
||||||
|
"mean_ms": 71.55
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 3836.47,
|
||||||
|
"median_ms": 3836.47,
|
||||||
|
"p90_ms": 3836.47,
|
||||||
|
"p95_ms": 3836.47,
|
||||||
|
"max_ms": 3836.47,
|
||||||
|
"mean_ms": 3836.47
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "capmetro",
|
||||||
|
"query": "CapMetro",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.05,
|
||||||
|
"median_ms": 0.05,
|
||||||
|
"p90_ms": 0.05,
|
||||||
|
"p95_ms": 0.05,
|
||||||
|
"max_ms": 0.05,
|
||||||
|
"mean_ms": 0.05
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 57.66,
|
||||||
|
"median_ms": 57.66,
|
||||||
|
"p90_ms": 57.66,
|
||||||
|
"p95_ms": 57.66,
|
||||||
|
"max_ms": 57.66,
|
||||||
|
"mean_ms": 57.66
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2800.54,
|
||||||
|
"median_ms": 2800.54,
|
||||||
|
"p90_ms": 2800.54,
|
||||||
|
"p95_ms": 2800.54,
|
||||||
|
"max_ms": 2800.54,
|
||||||
|
"mean_ms": 2800.54
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "hub-echo",
|
||||||
|
"query": "echo",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.02,
|
||||||
|
"median_ms": 0.02,
|
||||||
|
"p90_ms": 0.02,
|
||||||
|
"p95_ms": 0.02,
|
||||||
|
"max_ms": 0.02,
|
||||||
|
"mean_ms": 0.02
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 67.39,
|
||||||
|
"median_ms": 67.39,
|
||||||
|
"p90_ms": 67.39,
|
||||||
|
"p95_ms": 67.39,
|
||||||
|
"max_ms": 67.39,
|
||||||
|
"mean_ms": 67.39
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 4114.19,
|
||||||
|
"median_ms": 4114.19,
|
||||||
|
"p90_ms": 4114.19,
|
||||||
|
"p95_ms": 4114.19,
|
||||||
|
"max_ms": 4114.19,
|
||||||
|
"mean_ms": 4114.19
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "hub-prefs",
|
||||||
|
"query": "operator preferences",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 0.04,
|
||||||
|
"median_ms": 0.04,
|
||||||
|
"p90_ms": 0.04,
|
||||||
|
"p95_ms": 0.04,
|
||||||
|
"max_ms": 0.04,
|
||||||
|
"mean_ms": 0.04
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2013.4,
|
||||||
|
"median_ms": 2013.4,
|
||||||
|
"p90_ms": 2013.4,
|
||||||
|
"p95_ms": 2013.4,
|
||||||
|
"max_ms": 2013.4,
|
||||||
|
"mean_ms": 2013.4
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 4748.56,
|
||||||
|
"median_ms": 4748.56,
|
||||||
|
"p90_ms": 4748.56,
|
||||||
|
"p95_ms": 4748.56,
|
||||||
|
"max_ms": 4748.56,
|
||||||
|
"mean_ms": 4748.56
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "leaf-rivnut",
|
||||||
|
"query": "rivnut torque spec",
|
||||||
|
"resolve_floor": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2.14,
|
||||||
|
"median_ms": 2.14,
|
||||||
|
"p90_ms": 2.14,
|
||||||
|
"p95_ms": 2.14,
|
||||||
|
"max_ms": 2.14,
|
||||||
|
"mean_ms": 2.14
|
||||||
|
},
|
||||||
|
"search_only": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 2014.69,
|
||||||
|
"median_ms": 2014.69,
|
||||||
|
"p90_ms": 2014.69,
|
||||||
|
"p95_ms": 2014.69,
|
||||||
|
"max_ms": 2014.69,
|
||||||
|
"mean_ms": 2014.69
|
||||||
|
},
|
||||||
|
"recall_total": {
|
||||||
|
"n": 1,
|
||||||
|
"min_ms": 4363.28,
|
||||||
|
"median_ms": 4363.28,
|
||||||
|
"p90_ms": 4363.28,
|
||||||
|
"p95_ms": 4363.28,
|
||||||
|
"max_ms": 4363.28,
|
||||||
|
"mean_ms": 4363.28
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
{
|
||||||
|
"env": {
|
||||||
|
"date": "2026-06-23T20:18:05",
|
||||||
|
"endpoint": "https://echoapi.alwisp.com",
|
||||||
|
"commit": "af16598",
|
||||||
|
"echo_workers": 8,
|
||||||
|
"echo_timeout": 30,
|
||||||
|
"python": "3.10.12",
|
||||||
|
"iterations": 2,
|
||||||
|
"count_k": 50
|
||||||
|
},
|
||||||
|
"run_id": "20260623-201805-53dcdb",
|
||||||
|
"metrics": {
|
||||||
|
"worker-sweep": {
|
||||||
|
"result": {
|
||||||
|
"note_count": 197,
|
||||||
|
"sweep": [
|
||||||
|
{
|
||||||
|
"workers": 1,
|
||||||
|
"median_ms": 11917.49,
|
||||||
|
"p95_ms": 11917.49
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"workers": 2,
|
||||||
|
"median_ms": 5740.0,
|
||||||
|
"p95_ms": 5740.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"workers": 4,
|
||||||
|
"median_ms": 3034.3,
|
||||||
|
"p95_ms": 3034.3
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"workers": 8,
|
||||||
|
"median_ms": 1583.83,
|
||||||
|
"p95_ms": 1583.83
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"workers": 16,
|
||||||
|
"median_ms": 1277.77,
|
||||||
|
"p95_ms": 1277.77
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"knee_workers": 16,
|
||||||
|
"default_workers": 8
|
||||||
|
},
|
||||||
|
"gates": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,545 @@
|
|||||||
|
{
|
||||||
|
"title": "ECHO Memory Plugin \u2014 Performance Benchmark",
|
||||||
|
"subtitle": "Validation of the 1.1.0 concurrency and 1.2.0 entity-resolution releases",
|
||||||
|
"reference": "echo-v.05 @ af16598",
|
||||||
|
"status": "11/11 gate checks pass",
|
||||||
|
"summary": "Full-vault reads run 7.1x faster concurrent than serial (11.4 s down to 1.6 s, 197 notes). Entity resolution is sub-millisecond. Profiling traced recall() latency to serial graph expansion (not BM25); the fix makes expansion concurrent (3.5-4.2x, identical results). All 11 gate checks pass. Run live against echoapi.alwisp.com on 2026-06-23.",
|
||||||
|
"sections": [
|
||||||
|
{
|
||||||
|
"heading": "Test Environment",
|
||||||
|
"type": "kv",
|
||||||
|
"rows": [
|
||||||
|
{
|
||||||
|
"label": "Endpoint",
|
||||||
|
"value": "echoapi.alwisp.com (live)"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "Vault size",
|
||||||
|
"value": "197 notes"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "Plugin commit",
|
||||||
|
"value": "af16598"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "Releases under test",
|
||||||
|
"value": "1.1.0 pooling + concurrency, 1.2.0 resolver"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "Concurrency (default)",
|
||||||
|
"value": "ECHO_WORKERS = 8"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "Socket timeout",
|
||||||
|
"value": "ECHO_TIMEOUT = 30 s"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "Runner",
|
||||||
|
"value": "Python 3.10.12"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"label": "Date",
|
||||||
|
"value": "June 23, 2026"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"heading": "Headline Results",
|
||||||
|
"type": "table",
|
||||||
|
"columns": [
|
||||||
|
"Operation",
|
||||||
|
"Result",
|
||||||
|
"Verdict"
|
||||||
|
],
|
||||||
|
"rows": [
|
||||||
|
[
|
||||||
|
"Full-vault read (concurrent vs serial)",
|
||||||
|
{
|
||||||
|
"value": "7.13x",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"Full-vault read throughput",
|
||||||
|
{
|
||||||
|
"value": "123.5 notes/s",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"Cold vs warm request (keep-alive)",
|
||||||
|
{
|
||||||
|
"value": "3.25x",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"Entity resolve (in-memory index)",
|
||||||
|
{
|
||||||
|
"value": "<1 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"Idempotent append skip",
|
||||||
|
{
|
||||||
|
"value": "100%",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"Soak read errors (22 s)",
|
||||||
|
{
|
||||||
|
"value": "0",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"heading": "Full-Vault Read \u2014 Concurrency Scaling",
|
||||||
|
"type": "table",
|
||||||
|
"columns": [
|
||||||
|
"Workers",
|
||||||
|
"Median (197 notes)",
|
||||||
|
"Speedup vs serial"
|
||||||
|
],
|
||||||
|
"rows": [
|
||||||
|
[
|
||||||
|
"1 (serial)",
|
||||||
|
{
|
||||||
|
"value": "11,917 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "1.0x",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"2",
|
||||||
|
{
|
||||||
|
"value": "5,740 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "2.1x",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"4",
|
||||||
|
{
|
||||||
|
"value": "3,034 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "3.9x",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"8 (default)",
|
||||||
|
{
|
||||||
|
"value": "1,584 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "7.5x",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"16",
|
||||||
|
{
|
||||||
|
"value": "1,278 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "9.3x",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"heading": "Subject Pulls \u2014 recall() (after concurrency fix)",
|
||||||
|
"type": "table",
|
||||||
|
"columns": [
|
||||||
|
"Subject",
|
||||||
|
"resolve()",
|
||||||
|
"search",
|
||||||
|
"recall() total"
|
||||||
|
],
|
||||||
|
"rows": [
|
||||||
|
[
|
||||||
|
"APTA",
|
||||||
|
{
|
||||||
|
"value": "0.75 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "62 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "2,231 ms",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"CapMetro",
|
||||||
|
{
|
||||||
|
"value": "0.03 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "58 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "2,116 ms",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"echo (hub)",
|
||||||
|
{
|
||||||
|
"value": "0.03 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "71 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "2,404 ms",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"operator preferences (hub)",
|
||||||
|
{
|
||||||
|
"value": "0.04 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "2,011 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "2,995 ms",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"rivnut torque spec (leaf)",
|
||||||
|
{
|
||||||
|
"value": "2.15 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "2,014 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "2,371 ms",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"heading": "recall() Graph Expansion \u2014 Concurrency Fix",
|
||||||
|
"type": "table",
|
||||||
|
"columns": [
|
||||||
|
"Hub subject",
|
||||||
|
"Neighbours",
|
||||||
|
"Serial",
|
||||||
|
"Concurrent",
|
||||||
|
"Speedup",
|
||||||
|
"Results identical"
|
||||||
|
],
|
||||||
|
"rows": [
|
||||||
|
[
|
||||||
|
"echo",
|
||||||
|
{
|
||||||
|
"value": "120",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "2,228 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "642 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "3.47x",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Yes"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"operator preferences",
|
||||||
|
{
|
||||||
|
"value": "126",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "2,910 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "699 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "4.16x",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Yes"
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"heading": "Bulk Write Operations",
|
||||||
|
"type": "table",
|
||||||
|
"columns": [
|
||||||
|
"Operation",
|
||||||
|
"Per-op median",
|
||||||
|
"Aggregate"
|
||||||
|
],
|
||||||
|
"rows": [
|
||||||
|
[
|
||||||
|
"PUT (read-back verified)",
|
||||||
|
{
|
||||||
|
"value": "115 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "7.0 PUT/s",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"Bulk GET \u2014 concurrent vs serial (K=30)",
|
||||||
|
{
|
||||||
|
"value": "356 vs 1,607 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "4.51x",
|
||||||
|
"num": true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"APPEND \u2014 write",
|
||||||
|
{
|
||||||
|
"value": "104 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "1 round-trip + GET",
|
||||||
|
"num": false
|
||||||
|
}
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"APPEND \u2014 idempotent skip",
|
||||||
|
{
|
||||||
|
"value": "51 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"value": "100% skipped on re-run",
|
||||||
|
"num": false
|
||||||
|
}
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"heading": "Concurrency, Locking & Resilience",
|
||||||
|
"type": "table",
|
||||||
|
"columns": [
|
||||||
|
"Check",
|
||||||
|
"Measured",
|
||||||
|
"Expected"
|
||||||
|
],
|
||||||
|
"rows": [
|
||||||
|
[
|
||||||
|
"Lock \u2014 acquire when free",
|
||||||
|
{
|
||||||
|
"value": "159 ms (rc 0)",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"rc 0"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"Lock \u2014 contended acquire",
|
||||||
|
{
|
||||||
|
"value": "58 ms (rc 75)",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"rc 75 fast-fail"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"Lock \u2014 release",
|
||||||
|
{
|
||||||
|
"value": "133 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"owned release"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"Offline queue \u2014 enqueue",
|
||||||
|
{
|
||||||
|
"value": "0.08 ms",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"local write-ahead"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"read_many dedup (60 -> 20)",
|
||||||
|
{
|
||||||
|
"value": "deduped",
|
||||||
|
"num": false
|
||||||
|
},
|
||||||
|
"collapse to unique set"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"Soak drift (late vs early)",
|
||||||
|
{
|
||||||
|
"value": "0.99x",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"<= 1.5x, no leak"
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"heading": "Regression Gates (relative-ratio policy)",
|
||||||
|
"type": "table",
|
||||||
|
"columns": [
|
||||||
|
"Gate",
|
||||||
|
"Threshold",
|
||||||
|
"Measured",
|
||||||
|
"Result"
|
||||||
|
],
|
||||||
|
"rows": [
|
||||||
|
[
|
||||||
|
"read-full speedup",
|
||||||
|
">= 1.8x",
|
||||||
|
{
|
||||||
|
"value": "7.13x",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"bulk-get speedup",
|
||||||
|
">= 1.5x",
|
||||||
|
{
|
||||||
|
"value": "4.51x",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"pool warm-up cold/warm",
|
||||||
|
">= 1.5x",
|
||||||
|
{
|
||||||
|
"value": "3.25x",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"append idempotency",
|
||||||
|
"100% skip",
|
||||||
|
{
|
||||||
|
"value": "100%",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"soak errors",
|
||||||
|
"0",
|
||||||
|
{
|
||||||
|
"value": "0",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"soak drift",
|
||||||
|
"<= 1.5x",
|
||||||
|
{
|
||||||
|
"value": "0.99x",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"lock semantics",
|
||||||
|
"rc 0 / rc 75",
|
||||||
|
{
|
||||||
|
"value": "0 / 75",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"read_many dedup",
|
||||||
|
"unique set",
|
||||||
|
{
|
||||||
|
"value": "20/20",
|
||||||
|
"num": true
|
||||||
|
},
|
||||||
|
"Pass"
|
||||||
|
]
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"heading": "Findings",
|
||||||
|
"type": "olist",
|
||||||
|
"items": [
|
||||||
|
"Concurrency delivers as designed. Eight workers cut full-vault reads 7.5x; the curve is near-linear to 8 workers and flattens after (8->16 returns only 1.24x). The default of 8 is the right setting for this vault and link.",
|
||||||
|
"Keep-alive is confirmed live. A cold request costs 163 ms against a 50 ms warm median (3.25x). This is the direct measure that the 1.1.0 pooling fix removed the per-request TLS handshake that was timing out full-vault passes.",
|
||||||
|
"Entity resolution is effectively free. resolve() returns in under 1 ms across exact, alias, and shortened mentions. The 1.2.0 alias work resolves shortened names (\"echo memory\", \"ECHO plugin\", \"jason\") straight to the canonical note, so the anti-duplicate guard holds without a fuzzy fallback.",
|
||||||
|
"recall() latency was traced to the graph layer, not BM25. The BM25 index is already persisted and incrementally maintained (load 500 ms, score 0.1 ms). The cost was expand_graph fetching each neighbour serially \u2014 3.0 s for 126 nodes. Fix: the graph BFS now fetches each hop concurrently via the existing read_many. expand_graph is 3.5-4.2x faster with byte-identical ranking and scores; end-to-end recall() dropped from 2.8-4.7 s to 2.1-3.0 s per subject.",
|
||||||
|
"Full-vault maintenance scripts stay under the tool timeout. sweep.py runs in 4.6 s and vault_lint.py in 5.1 s on 197 notes. The original failure mode (passes exceeding the timeout and dropping the session) does not recur.",
|
||||||
|
"Write integrity holds. PUT verifies via read-back at 115 ms; APPEND is whole-line idempotent, skipping at 51 ms with zero duplicate writes on re-run. The advisory lock fast-fails a contended acquire with the expected exit 75."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"heading": "Recommendations",
|
||||||
|
"type": "olist",
|
||||||
|
"items": [
|
||||||
|
"Apply read_many to every code path that reads a set of known notes. Done for expand_graph (gated). Remaining: prefetch the _brief result bodies recall prints, and parallelize rebuild()'s vault walk.",
|
||||||
|
"Reuse one in-process read cache across a recall() call so load_index, expansion, and _brief don't re-fetch overlapping notes \u2014 this closes the remaining ~500 ms + serial _brief cost.",
|
||||||
|
"Make 'prefer read_many over a GET loop' a documented contract in the plugin's API reference; serial multi-note reads are a performance bug. The expand-graph gate enforces it for recall.",
|
||||||
|
"Hold ECHO_WORKERS at 8 (16 buys only 1.24x). Promote the resolve correctness table from INFO to FAIL now that live behavior is confirmed. Re-baseline at the next 1.x release or past ~400 notes."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"heading": "Method",
|
||||||
|
"type": "text",
|
||||||
|
"body": [
|
||||||
|
"Each metric was timed with a perf-counter harness that reuses the live ECHO client, so it exercises the real pooled and concurrent code path. Reads report median and p95 over repeated iterations after a discarded warm-up; write metrics ran in a disposable namespace under the advisory lock and were deleted on completion (zero residual files confirmed).",
|
||||||
|
"Gates use relative ratios rather than absolute milliseconds, so they stay valid regardless of the network the suite runs from. The harness, fixtures, baselines, and per-metric JSON are checked in under eval/perf/."
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"note": {
|
||||||
|
"heading": "Next Step",
|
||||||
|
"body": "Prefetch recall()'s _brief reads and parallelize rebuild() via read_many, then re-run subject-pull to confirm recall drops toward ~1 s. expand_graph fix is local-only until ported to the canonical repo. No blockers in 1.1.0 / 1.2.0."
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
[20:14:00] read-full
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
export ECHO_KEY_LEGACY_OK=1
|
||||||
|
cd "$(dirname "$0")"
|
||||||
|
{
|
||||||
|
echo "[$(date +%T)] read-full"; python3 bench.py read-full -n 3 --quiet --json-out results/m-read-full.json
|
||||||
|
echo "[$(date +%T)] worker-sweep"; python3 bench.py worker-sweep -n 4 --quiet --json-out results/m-worker-sweep.json
|
||||||
|
echo "[$(date +%T)] subject-pull"; python3 bench.py subject-pull -n 4 --quiet --json-out results/m-subject-pull.json
|
||||||
|
echo "[$(date +%T)] index"; python3 bench.py index --quiet --json-out results/m-index.json
|
||||||
|
echo "[$(date +%T)] lint"; python3 bench.py lint --quiet --json-out results/m-lint.json
|
||||||
|
echo "[$(date +%T)] DONE"
|
||||||
|
} > results/run.log 2>&1
|
||||||
|
touch results/DONE
|
||||||
@@ -247,27 +247,34 @@ def expand_graph(seeds, nmap: dict, base_scores: dict, max_hops: int = MAX_HOPS)
|
|||||||
score_of = dict(base_scores)
|
score_of = dict(base_scores)
|
||||||
seen = set(seeds)
|
seen = set(seeds)
|
||||||
results: dict[str, tuple[float, str]] = {}
|
results: dict[str, tuple[float, str]] = {}
|
||||||
dq = deque((s, 0) for s in seeds)
|
# Breadth-first by HOP so each frontier's note bodies can be fetched concurrently
|
||||||
while dq:
|
# (echo.read_many) instead of one blocking GET per node. Scoring is unchanged: a
|
||||||
path, hop = dq.popleft()
|
# neighbour keeps the max decayed score over the paths that reach it, and non-seed
|
||||||
if hop >= max_hops:
|
# parents score 1.0 exactly as the serial deque version did.
|
||||||
continue
|
frontier = list(seeds)
|
||||||
text = links.get_text(path)
|
for hop in range(max_hops):
|
||||||
if text is None:
|
if not frontier:
|
||||||
continue
|
break
|
||||||
body = strip_frontmatter(text)
|
texts = echo.read_many(frontier)
|
||||||
targets = set(links.all_wikilinks(body)) | set(links.source_notes(text))
|
next_frontier: list[str] = []
|
||||||
parent = score_of.get(path, 1.0)
|
for path in frontier:
|
||||||
for t in targets:
|
text = texts.get(path)
|
||||||
tp = _resolve_target(t, nmap)
|
if text is None:
|
||||||
if not tp or tp in seeds:
|
|
||||||
continue
|
continue
|
||||||
|
body = strip_frontmatter(text)
|
||||||
|
targets = set(links.all_wikilinks(body)) | set(links.source_notes(text))
|
||||||
|
parent = score_of.get(path, 1.0)
|
||||||
decayed = parent * (GRAPH_DECAY ** (hop + 1))
|
decayed = parent * (GRAPH_DECAY ** (hop + 1))
|
||||||
if decayed > results.get(tp, (0.0, ""))[0]:
|
for t in targets:
|
||||||
results[tp] = (decayed, path)
|
tp = _resolve_target(t, nmap)
|
||||||
if tp not in seen:
|
if not tp or tp in seeds:
|
||||||
seen.add(tp)
|
continue
|
||||||
dq.append((tp, hop + 1))
|
if decayed > results.get(tp, (0.0, ""))[0]:
|
||||||
|
results[tp] = (decayed, path)
|
||||||
|
if tp not in seen:
|
||||||
|
seen.add(tp)
|
||||||
|
next_frontier.append(tp)
|
||||||
|
frontier = next_frontier
|
||||||
return sorted(results.items(), key=lambda kv: -kv[1][0])
|
return sorted(results.items(), key=lambda kv: -kv[1][0])
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user