ver 1.3 and 1.3.1

This commit is contained in:
Jason Stedwell
2026-06-23 22:17:39 -05:00
parent 1881d2b105
commit b4605a52f2
53 changed files with 549 additions and 297 deletions
@@ -1,7 +1,7 @@
# ECHO Performance Test Suite — Plan
**Status:** plan only — nothing here is to be run yet.
**Author:** drafted for Jason Stedwell, 2026-06-23.
**Author:** drafted for the vault owner, 2026-06-23.
**Targets the changes that landed in:** 1.1.0 (connection pooling + concurrent `read_many` + shared cache) and 1.2.0 (fuzzy `resolve`, alias derivation/learning, `capture` re-slug fix).
**Scope:** a *performance / timing* harness. It complements — does not replace — the existing correctness tests (`test_echo_client.py`, `test_v1_scaffold.py`, the eval feature tests), which stay as the pass/fail gate for behaviour.
@@ -9,9 +9,9 @@
## 1. Goals
1. Put hard numbers on the three operations Jason called out — full endpoint read, full subject pulls, bulk get/put/append — so the 1.1.0 perf claims ("lint 0.90s / sweep 0.85s on 186 notes, ~4.5× from reuse, ~2× from concurrency") become a repeatable, checked-in measurement instead of a one-time observation.
1. Put hard numbers on the three operations the operator called out — full endpoint read, full subject pulls, bulk get/put/append — so the 1.1.0 perf claims ("lint 0.90s / sweep 0.85s on 186 notes, ~4.5× from reuse, ~2× from concurrency") become a repeatable, checked-in measurement instead of a one-time observation.
2. Establish **baselines + regression gates** so a future change that quietly reintroduces the pre-1.1.0 per-request TLS handshake (the bug that was timing out sessions) fails the suite instead of silently shipping.
3. Keep every run **non-destructive to the production vault** — writes go to a disposable namespace and are cleaned up (per Jason's "prune test artifacts" rule).
3. Keep every run **non-destructive to the production vault** — writes go to a disposable namespace and are cleaned up (per the operator's "prune test artifacts" rule).
## 2. Methodology (applies to every metric)
@@ -22,11 +22,11 @@
- **Environment capture:** record `ECHO_WORKERS`, `ECHO_TIMEOUT`, vault note count, git commit/plugin version, and wall-clock date in the results header so two runs are comparable.
- **Output:** machine-readable JSON (`eval/perf/results/<date>-<commit>.json`) plus a short human table to stdout. JSON lets us diff runs over time and chart trend.
- **Isolation:** all test writes live under a dedicated prefix, e.g. `_agent/_bench/<run-id>/...`, never in `projects/`, `resources/`, `inbox/`, or `journal/`. A `--cleanup` pass deletes the whole prefix at the end; a `--keep` flag preserves it for debugging. The advisory lock (`echo.py lock`) is taken for the write phases so a bench run can't race a real CoWork session.
- **Network honesty:** the endpoint is the live `echoapi.alwisp.com`, so absolute numbers depend on WAN latency. Report **both** absolute ms and a relative ratio against the same run's single-GET baseline, so the suite stays meaningful regardless of where it's run from.
- **Network honesty:** the endpoint is the live configured endpoint (`echo.BASE` / `$ECHO_BASE`), so absolute numbers depend on WAN latency. Report **both** absolute ms and a relative ratio against the same run's single-GET baseline, so the suite stays meaningful regardless of where it's run from.
---
## 3. Core metrics (the three Jason asked for)
## 3. Core metrics (the three the operator asked for)
### 3.1 Timed full endpoint read (full-vault bulk read)
@@ -58,7 +58,7 @@
### 3.3 Timed bulk get / put / append
**What it measures:** write-path throughput and the read-back/idempotency overhead Jason deliberately pays for.
**What it measures:** write-path throughput and the read-back/idempotency overhead the operator deliberately pays for.
**Procedure (all in the `_agent/_bench/<run-id>/` namespace):**
- **Bulk GET:** create K fixture notes once, then time reading them back — both serially and via `read_many` — to compare against §3.1 at a controlled, fixed K (e.g. 25/50/100) so the read curve is isolated from whatever the live vault happens to contain.