ver 1.3 and 1.3.1
This commit is contained in:
Jason Stedwell
Binary file not shown.
@@ -1,7 +1,7 @@
|
||||
# ECHO Performance Test Suite — Plan
|
||||
|
||||
**Status:** plan only — nothing here is to be run yet.
|
||||
**Author:** drafted for Jason Stedwell, 2026-06-23.
|
||||
**Author:** drafted for the vault owner, 2026-06-23.
|
||||
**Targets the changes that landed in:** 1.1.0 (connection pooling + concurrent `read_many` + shared cache) and 1.2.0 (fuzzy `resolve`, alias derivation/learning, `capture` re-slug fix).
|
||||
**Scope:** a *performance / timing* harness. It complements — does not replace — the existing correctness tests (`test_echo_client.py`, `test_v1_scaffold.py`, the eval feature tests), which stay as the pass/fail gate for behaviour.
|
||||
|
||||
@@ -9,9 +9,9 @@
|
||||
|
||||
## 1. Goals
|
||||
|
||||
1. Put hard numbers on the three operations Jason called out — full endpoint read, full subject pulls, bulk get/put/append — so the 1.1.0 perf claims ("lint 0.90s / sweep 0.85s on 186 notes, ~4.5× from reuse, ~2× from concurrency") become a repeatable, checked-in measurement instead of a one-time observation.
|
||||
1. Put hard numbers on the three operations the operator called out — full endpoint read, full subject pulls, bulk get/put/append — so the 1.1.0 perf claims ("lint 0.90s / sweep 0.85s on 186 notes, ~4.5× from reuse, ~2× from concurrency") become a repeatable, checked-in measurement instead of a one-time observation.
|
||||
2. Establish **baselines + regression gates** so a future change that quietly reintroduces the pre-1.1.0 per-request TLS handshake (the bug that was timing out sessions) fails the suite instead of silently shipping.
|
||||
3. Keep every run **non-destructive to the production vault** — writes go to a disposable namespace and are cleaned up (per Jason's "prune test artifacts" rule).
|
||||
3. Keep every run **non-destructive to the production vault** — writes go to a disposable namespace and are cleaned up (per the operator's "prune test artifacts" rule).
|
||||
|
||||
## 2. Methodology (applies to every metric)
|
||||
|
||||
@@ -22,11 +22,11 @@
|
||||
- **Environment capture:** record `ECHO_WORKERS`, `ECHO_TIMEOUT`, vault note count, git commit/plugin version, and wall-clock date in the results header so two runs are comparable.
|
||||
- **Output:** machine-readable JSON (`eval/perf/results/<date>-<commit>.json`) plus a short human table to stdout. JSON lets us diff runs over time and chart trend.
|
||||
- **Isolation:** all test writes live under a dedicated prefix, e.g. `_agent/_bench/<run-id>/...`, never in `projects/`, `resources/`, `inbox/`, or `journal/`. A `--cleanup` pass deletes the whole prefix at the end; a `--keep` flag preserves it for debugging. The advisory lock (`echo.py lock`) is taken for the write phases so a bench run can't race a real CoWork session.
|
||||
- **Network honesty:** the endpoint is the live `echoapi.alwisp.com`, so absolute numbers depend on WAN latency. Report **both** absolute ms and a relative ratio against the same run's single-GET baseline, so the suite stays meaningful regardless of where it's run from.
|
||||
- **Network honesty:** the endpoint is the live configured endpoint (`echo.BASE` / `$ECHO_BASE`), so absolute numbers depend on WAN latency. Report **both** absolute ms and a relative ratio against the same run's single-GET baseline, so the suite stays meaningful regardless of where it's run from.
|
||||
|
||||
---
|
||||
|
||||
## 3. Core metrics (the three Jason asked for)
|
||||
## 3. Core metrics (the three the operator asked for)
|
||||
|
||||
### 3.1 Timed full endpoint read (full-vault bulk read)
|
||||
|
||||
@@ -58,7 +58,7 @@
|
||||
|
||||
### 3.3 Timed bulk get / put / append
|
||||
|
||||
**What it measures:** write-path throughput and the read-back/idempotency overhead Jason deliberately pays for.
|
||||
**What it measures:** write-path throughput and the read-back/idempotency overhead the operator deliberately pays for.
|
||||
|
||||
**Procedure (all in the `_agent/_bench/<run-id>/` namespace):**
|
||||
- **Bulk GET:** create K fixture notes once, then time reading them back — both serially and via `read_many` — to compare against §3.1 at a controlled, fixed K (e.g. 25/50/100) so the read curve is isolated from whatever the live vault happens to contain.
|
||||
|
||||
@@ -6,7 +6,7 @@ targeted. It reuses the live `echo` client module, so it measures the real poole
|
||||
concurrent code path — not a reimplementation.
|
||||
|
||||
This is a **manual / pre-release** tool, not a CI gate: it depends on the live
|
||||
endpoint (`echoapi.alwisp.com`) and WAN latency. Gates are **relative ratios**
|
||||
configured endpoint (`echo.BASE` / `$ECHO_BASE`) and WAN latency. Gates are **relative ratios**
|
||||
(portable), with absolute milliseconds kept as informational context.
|
||||
|
||||
See `PERF-TEST-PLAN.md` for the design rationale behind each metric.
|
||||
@@ -25,8 +25,8 @@ python3 bench.py resolve # 1.2.0 fuzzy correctness table + timin
|
||||
python3 bench.py soak --soak-seconds 120 # stability / leak watch
|
||||
```
|
||||
|
||||
Set the key first if the baked-in fallback warning bothers you:
|
||||
`python3 ../../scripts/echo.py write-key <token>` (or `export ECHO_KEY=...`).
|
||||
Configure this machine first (owner/endpoint/key):
|
||||
`python3 ../../scripts/echo.py config init` then edit the file, or `export ECHO_BASE=... ECHO_KEY=...`.
|
||||
|
||||
## Metrics
|
||||
|
||||
@@ -90,7 +90,7 @@ metric + gate above regression-guard this so it can't quietly revert to serial.
|
||||
sandbox vault so the replay doesn't write to production; that's deferred (the
|
||||
plan's synthetic-vault item). The metric still proves enqueue works and reports
|
||||
queue depth.
|
||||
- **Absolute ms depend on where you run from.** WAN latency to `echoapi.alwisp.com`
|
||||
dominates single-request times; that's why gates are ratios.
|
||||
- **Absolute ms depend on where you run from.** WAN latency to the configured
|
||||
endpoint dominates single-request times; that's why gates are ratios.
|
||||
- **Synthetic-vault scaling (100/500/1000 notes) is deferred** per decision — the
|
||||
suite benchmarks the real ~190-note vault today.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"_comment": "Relative-ratio regression gates (Jason's chosen policy). Ratios are portable across machines/WAN; absolute ms in the results JSON stay informational. Tune these after the first real baseline run, then tighten. 'field' is a dotted path into a metric's result; for list items use the index (e.g. sweep.3.median_ms).",
|
||||
"_comment": "Relative-ratio regression gates (the maintainer's chosen policy). Ratios are portable across machines/WAN; absolute ms in the results JSON stay informational. Tune these after the first real baseline run, then tighten. 'field' is a dotted path into a metric's result; for list items use the index (e.g. sweep.3.median_ms).",
|
||||
"read-full": {
|
||||
"gates": [
|
||||
{ "field": "speedup_ratio", "op": ">=", "min": 1.8,
|
||||
|
||||
@@ -10,7 +10,7 @@ code path. Reads are non-destructive; write metrics live in a disposable
|
||||
namespace (_agent/_bench/<run-id>/) cleaned up at the end unless --keep.
|
||||
|
||||
This is a MANUAL / pre-release tool, not a CI gate — it depends on the live
|
||||
endpoint (echoapi.alwisp.com) and WAN latency.
|
||||
configured endpoint (echo.BASE / $ECHO_BASE) and WAN latency.
|
||||
|
||||
python3 bench.py <metric> [options]
|
||||
python3 bench.py all # the read-only safe set
|
||||
|
||||
@@ -11,8 +11,8 @@ from __future__ import annotations
|
||||
# --- §3.2 subject pulls -------------------------------------------------------
|
||||
# (label, query) pairs fed to `recall`. Chosen to span neighbourhood sizes — the
|
||||
# real cost driver — from a likely-leaf to a known hub. APTA and CapMetro are the
|
||||
# subjects Jason named; "echo" and "operator preferences" are dense hubs that
|
||||
# stress the 1-hop expansion (the read_many fan-out 1.1.0 accelerated).
|
||||
# subjects the maintainer configured; "echo" and "operator preferences" are dense hubs
|
||||
# that stress the 1-hop expansion (the read_many fan-out 1.1.0 accelerated).
|
||||
SUBJECTS: list[tuple[str, str]] = [
|
||||
("apta", "APTA"),
|
||||
("capmetro", "CapMetro"),
|
||||
@@ -32,13 +32,16 @@ SUBJECTS: list[tuple[str, str]] = [
|
||||
# to nothing and silently spawn a new note)
|
||||
# "miss" -> neither; genuinely unknown
|
||||
# Tune expect_slug to the live vault; unknowns are reported as INFO, not failures.
|
||||
# NOTE: the subject/slug values below are neutral placeholders. The maintainer
|
||||
# should point these fixtures at dense hubs that actually exist in their own
|
||||
# vault, otherwise the resolve cases will report INFO misses rather than hits.
|
||||
RESOLVE_CASES: list[dict] = [
|
||||
{"mention": "echo", "expect": "exact", "expect_slug": "echo"},
|
||||
{"mention": "echo memory", "expect": "candidates", "expect_slug": "echo"},
|
||||
{"mention": "ECHO plugin", "expect": "candidates", "expect_slug": "echo"},
|
||||
{"mention": "goldbrain", "expect": "exact", "expect_slug": "goldbrain"},
|
||||
{"mention": "jason stedwell", "expect": "exact", "expect_slug": "jason-stedwell"},
|
||||
{"mention": "jason", "expect": "candidates", "expect_slug": "jason-stedwell"},
|
||||
{"mention": "other-vault", "expect": "exact", "expect_slug": "other-vault"},
|
||||
{"mention": "example subject", "expect": "exact", "expect_slug": "example-subject"},
|
||||
{"mention": "operator", "expect": "candidates", "expect_slug": "example-subject"},
|
||||
{"mention": "zzqx nonexistent entity", "expect": "miss", "expect_slug": None},
|
||||
]
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:17:09",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
@@ -353,7 +353,7 @@
|
||||
"expect_slug": "echo",
|
||||
"resolved_slug": "echo",
|
||||
"candidate_slugs": [
|
||||
"2026-06-19-goldbrain-full-echo-architect",
|
||||
"2026-06-19-other-vault-full-echo-architect",
|
||||
"echo",
|
||||
"echo-memory-codex-plugin",
|
||||
"echo-plugin-build",
|
||||
@@ -407,7 +407,7 @@
|
||||
"echo-memory-codex-plugin",
|
||||
"echo-plugin-build",
|
||||
"echo-skill-improvements",
|
||||
"2026-06-19-goldbrain-full-echo-architect"
|
||||
"2026-06-19-other-vault-full-echo-architect"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
@@ -456,8 +456,8 @@
|
||||
"echo",
|
||||
"echo-memory-codex-plugin",
|
||||
"echo-plugin-build",
|
||||
"2026-06-19-goldbrain-full-echo-architect",
|
||||
"alabama-wisp-brand-docs"
|
||||
"2026-06-19-other-vault-full-echo-architect",
|
||||
"example-isp-brand-docs"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
@@ -498,13 +498,13 @@
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "goldbrain",
|
||||
"mention": "other-vault",
|
||||
"expect": "exact",
|
||||
"expect_slug": "goldbrain",
|
||||
"resolved_slug": "goldbrain",
|
||||
"expect_slug": "other-vault",
|
||||
"resolved_slug": "other-vault",
|
||||
"candidate_slugs": [
|
||||
"2026-06-19-goldbrain-full-echo-architect",
|
||||
"goldbrain"
|
||||
"2026-06-19-other-vault-full-echo-architect",
|
||||
"other-vault"
|
||||
],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
@@ -545,13 +545,13 @@
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "jason stedwell",
|
||||
"mention": "example subject",
|
||||
"expect": "exact",
|
||||
"expect_slug": "jason-stedwell",
|
||||
"resolved_slug": "jason-stedwell",
|
||||
"expect_slug": "example-subject",
|
||||
"resolved_slug": "example-subject",
|
||||
"candidate_slugs": [
|
||||
"jason-stedwell",
|
||||
"jason-mcp-gateway"
|
||||
"example-subject",
|
||||
"operator-mcp-gateway"
|
||||
],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
@@ -592,13 +592,13 @@
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "jason",
|
||||
"mention": "operator",
|
||||
"expect": "candidates",
|
||||
"expect_slug": "jason-stedwell",
|
||||
"resolved_slug": "jason-stedwell",
|
||||
"expect_slug": "example-subject",
|
||||
"resolved_slug": "example-subject",
|
||||
"candidate_slugs": [
|
||||
"jason-mcp-gateway",
|
||||
"jason-stedwell"
|
||||
"operator-mcp-gateway",
|
||||
"example-subject"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:22:00",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:21:36",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:21:21",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:12:16",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T21:22:29",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:20:57",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:21:01",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:21:28",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:12:12",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:22:05",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:17:09",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:12:19",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
@@ -20,7 +20,7 @@
|
||||
"expect_slug": "echo",
|
||||
"resolved_slug": "echo",
|
||||
"candidate_slugs": [
|
||||
"2026-06-19-goldbrain-full-echo-architect",
|
||||
"2026-06-19-other-vault-full-echo-architect",
|
||||
"echo",
|
||||
"echo-memory-codex-plugin",
|
||||
"echo-plugin-build",
|
||||
@@ -74,7 +74,7 @@
|
||||
"echo-memory-codex-plugin",
|
||||
"echo-plugin-build",
|
||||
"echo-skill-improvements",
|
||||
"2026-06-19-goldbrain-full-echo-architect"
|
||||
"2026-06-19-other-vault-full-echo-architect"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
@@ -123,8 +123,8 @@
|
||||
"echo",
|
||||
"echo-memory-codex-plugin",
|
||||
"echo-plugin-build",
|
||||
"2026-06-19-goldbrain-full-echo-architect",
|
||||
"alabama-wisp-brand-docs"
|
||||
"2026-06-19-other-vault-full-echo-architect",
|
||||
"example-isp-brand-docs"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
@@ -165,13 +165,13 @@
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "goldbrain",
|
||||
"mention": "other-vault",
|
||||
"expect": "exact",
|
||||
"expect_slug": "goldbrain",
|
||||
"resolved_slug": "goldbrain",
|
||||
"expect_slug": "other-vault",
|
||||
"resolved_slug": "other-vault",
|
||||
"candidate_slugs": [
|
||||
"2026-06-19-goldbrain-full-echo-architect",
|
||||
"goldbrain"
|
||||
"2026-06-19-other-vault-full-echo-architect",
|
||||
"other-vault"
|
||||
],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
@@ -212,13 +212,13 @@
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "jason stedwell",
|
||||
"mention": "example subject",
|
||||
"expect": "exact",
|
||||
"expect_slug": "jason-stedwell",
|
||||
"resolved_slug": "jason-stedwell",
|
||||
"expect_slug": "example-subject",
|
||||
"resolved_slug": "example-subject",
|
||||
"candidate_slugs": [
|
||||
"jason-stedwell",
|
||||
"jason-mcp-gateway"
|
||||
"example-subject",
|
||||
"operator-mcp-gateway"
|
||||
],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
@@ -259,13 +259,13 @@
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "jason",
|
||||
"mention": "operator",
|
||||
"expect": "candidates",
|
||||
"expect_slug": "jason-stedwell",
|
||||
"resolved_slug": "jason-stedwell",
|
||||
"expect_slug": "example-subject",
|
||||
"resolved_slug": "example-subject",
|
||||
"candidate_slugs": [
|
||||
"jason-mcp-gateway",
|
||||
"jason-stedwell"
|
||||
"operator-mcp-gateway",
|
||||
"example-subject"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:22:15",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T21:18:12",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:20:23",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:18:05",
|
||||
"endpoint": "https://echoapi.alwisp.com",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"echo_workers": 8,
|
||||
"echo_timeout": 30,
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"subtitle": "Validation of the 1.1.0 concurrency and 1.2.0 entity-resolution releases",
|
||||
"reference": "echo-v.05 @ af16598",
|
||||
"status": "11/11 gate checks pass",
|
||||
"summary": "Full-vault reads run 7.1x faster concurrent than serial (11.4 s down to 1.6 s, 197 notes). Entity resolution is sub-millisecond. Profiling traced recall() latency to serial graph expansion (not BM25); the fix makes expansion concurrent (3.5-4.2x, identical results). All 11 gate checks pass. Run live against echoapi.alwisp.com on 2026-06-23.",
|
||||
"summary": "Full-vault reads run 7.1x faster concurrent than serial (11.4 s down to 1.6 s, 197 notes). Entity resolution is sub-millisecond. Profiling traced recall() latency to serial graph expansion (not BM25); the fix makes expansion concurrent (3.5-4.2x, identical results). All 11 gate checks pass. Run live against obsidian.example.com on 2026-06-23.",
|
||||
"sections": [
|
||||
{
|
||||
"heading": "Test Environment",
|
||||
@@ -11,7 +11,7 @@
|
||||
"rows": [
|
||||
{
|
||||
"label": "Endpoint",
|
||||
"value": "echoapi.alwisp.com (live)"
|
||||
"value": "obsidian.example.com (live)"
|
||||
},
|
||||
{
|
||||
"label": "Vault size",
|
||||
@@ -513,7 +513,7 @@
|
||||
"items": [
|
||||
"Concurrency delivers as designed. Eight workers cut full-vault reads 7.5x; the curve is near-linear to 8 workers and flattens after (8->16 returns only 1.24x). The default of 8 is the right setting for this vault and link.",
|
||||
"Keep-alive is confirmed live. A cold request costs 163 ms against a 50 ms warm median (3.25x). This is the direct measure that the 1.1.0 pooling fix removed the per-request TLS handshake that was timing out full-vault passes.",
|
||||
"Entity resolution is effectively free. resolve() returns in under 1 ms across exact, alias, and shortened mentions. The 1.2.0 alias work resolves shortened names (\"echo memory\", \"ECHO plugin\", \"jason\") straight to the canonical note, so the anti-duplicate guard holds without a fuzzy fallback.",
|
||||
"Entity resolution is effectively free. resolve() returns in under 1 ms across exact, alias, and shortened mentions. The 1.2.0 alias work resolves shortened names (\"echo memory\", \"ECHO plugin\", \"operator\") straight to the canonical note, so the anti-duplicate guard holds without a fuzzy fallback.",
|
||||
"recall() latency was traced to the graph layer, not BM25. The BM25 index is already persisted and incrementally maintained (load 500 ms, score 0.1 ms). The cost was expand_graph fetching each neighbour serially \u2014 3.0 s for 126 nodes. Fix: the graph BFS now fetches each hop concurrently via the existing read_many. expand_graph is 3.5-4.2x faster with byte-identical ranking and scores; end-to-end recall() dropped from 2.8-4.7 s to 2.1-3.0 s per subject.",
|
||||
"Full-vault maintenance scripts stay under the tool timeout. sweep.py runs in 4.6 s and vault_lint.py in 5.1 s on 197 notes. The original failure mode (passes exceeding the timeout and dropping the session) does not recur.",
|
||||
"Write integrity holds. PUT verifies via read-back at 115 ms; APPEND is whole-line idempotent, skipping at 51 ms with zero duplicate writes on re-run. The advisory lock fast-fails a contended acquire with the expected exit 75."
|
||||
|
||||
Reference in New Issue
Block a user