{
  "title": "ECHO Memory Plugin \u2014 Performance Benchmark",
  "subtitle": "Validation of the 1.1.0 concurrency and 1.2.0 entity-resolution releases",
  "reference": "echo-v.05 @ af16598",
  "status": "11/11 gate checks pass",
  "summary": "Full-vault reads run 7.1x faster concurrent than serial (11.4 s down to 1.6 s, 197 notes). Entity resolution is sub-millisecond. Profiling traced recall() latency to serial graph expansion (not BM25); the fix makes expansion concurrent (3.5-4.2x, identical results). All 11 gate checks pass. Run live against obsidian.example.com on 2026-06-23.",
  "sections": [
    {
      "heading": "Test Environment",
      "type": "kv",
      "rows": [
        {
          "label": "Endpoint",
          "value": "obsidian.example.com (live)"
        },
        {
          "label": "Vault size",
          "value": "197 notes"
        },
        {
          "label": "Plugin commit",
          "value": "af16598"
        },
        {
          "label": "Releases under test",
          "value": "1.1.0 pooling + concurrency, 1.2.0 resolver"
        },
        {
          "label": "Concurrency (default)",
          "value": "ECHO_WORKERS = 8"
        },
        {
          "label": "Socket timeout",
          "value": "ECHO_TIMEOUT = 30 s"
        },
        {
          "label": "Runner",
          "value": "Python 3.10.12"
        },
        {
          "label": "Date",
          "value": "June 23, 2026"
        }
      ]
    },
    {
      "heading": "Headline Results",
      "type": "table",
      "columns": [
        "Operation",
        "Result",
        "Verdict"
      ],
      "rows": [
        [
          "Full-vault read (concurrent vs serial)",
          {
            "value": "7.13x",
            "num": true
          },
          "Pass"
        ],
        [
          "Full-vault read throughput",
          {
            "value": "123.5 notes/s",
            "num": true
          },
          "Pass"
        ],
        [
          "Cold vs warm request (keep-alive)",
          {
            "value": "3.25x",
            "num": true
          },
          "Pass"
        ],
        [
          "Entity resolve (in-memory index)",
          {
            "value": "<1 ms",
            "num": true
          },
          "Pass"
        ],
        [
          "Idempotent append skip",
          {
            "value": "100%",
            "num": true
          },
          "Pass"
        ],
        [
          "Soak read errors (22 s)",
          {
            "value": "0",
            "num": true
          },
          "Pass"
        ]
      ]
    },
    {
      "heading": "Full-Vault Read \u2014 Concurrency Scaling",
      "type": "table",
      "columns": [
        "Workers",
        "Median (197 notes)",
        "Speedup vs serial"
      ],
      "rows": [
        [
          "1 (serial)",
          {
            "value": "11,917 ms",
            "num": true
          },
          {
            "value": "1.0x",
            "num": true
          }
        ],
        [
          "2",
          {
            "value": "5,740 ms",
            "num": true
          },
          {
            "value": "2.1x",
            "num": true
          }
        ],
        [
          "4",
          {
            "value": "3,034 ms",
            "num": true
          },
          {
            "value": "3.9x",
            "num": true
          }
        ],
        [
          "8 (default)",
          {
            "value": "1,584 ms",
            "num": true
          },
          {
            "value": "7.5x",
            "num": true
          }
        ],
        [
          "16",
          {
            "value": "1,278 ms",
            "num": true
          },
          {
            "value": "9.3x",
            "num": true
          }
        ]
      ]
    },
    {
      "heading": "Subject Pulls \u2014 recall() (after concurrency fix)",
      "type": "table",
      "columns": [
        "Subject",
        "resolve()",
        "search",
        "recall() total"
      ],
      "rows": [
        [
          "APTA",
          {
            "value": "0.75 ms",
            "num": true
          },
          {
            "value": "62 ms",
            "num": true
          },
          {
            "value": "2,231 ms",
            "num": true
          }
        ],
        [
          "CapMetro",
          {
            "value": "0.03 ms",
            "num": true
          },
          {
            "value": "58 ms",
            "num": true
          },
          {
            "value": "2,116 ms",
            "num": true
          }
        ],
        [
          "echo (hub)",
          {
            "value": "0.03 ms",
            "num": true
          },
          {
            "value": "71 ms",
            "num": true
          },
          {
            "value": "2,404 ms",
            "num": true
          }
        ],
        [
          "operator preferences (hub)",
          {
            "value": "0.04 ms",
            "num": true
          },
          {
            "value": "2,011 ms",
            "num": true
          },
          {
            "value": "2,995 ms",
            "num": true
          }
        ],
        [
          "rivnut torque spec (leaf)",
          {
            "value": "2.15 ms",
            "num": true
          },
          {
            "value": "2,014 ms",
            "num": true
          },
          {
            "value": "2,371 ms",
            "num": true
          }
        ]
      ]
    },
    {
      "heading": "recall() Graph Expansion \u2014 Concurrency Fix",
      "type": "table",
      "columns": [
        "Hub subject",
        "Neighbours",
        "Serial",
        "Concurrent",
        "Speedup",
        "Results identical"
      ],
      "rows": [
        [
          "echo",
          {
            "value": "120",
            "num": true
          },
          {
            "value": "2,228 ms",
            "num": true
          },
          {
            "value": "642 ms",
            "num": true
          },
          {
            "value": "3.47x",
            "num": true
          },
          "Yes"
        ],
        [
          "operator preferences",
          {
            "value": "126",
            "num": true
          },
          {
            "value": "2,910 ms",
            "num": true
          },
          {
            "value": "699 ms",
            "num": true
          },
          {
            "value": "4.16x",
            "num": true
          },
          "Yes"
        ]
      ]
    },
    {
      "heading": "Bulk Write Operations",
      "type": "table",
      "columns": [
        "Operation",
        "Per-op median",
        "Aggregate"
      ],
      "rows": [
        [
          "PUT (read-back verified)",
          {
            "value": "115 ms",
            "num": true
          },
          {
            "value": "7.0 PUT/s",
            "num": true
          }
        ],
        [
          "Bulk GET \u2014 concurrent vs serial (K=30)",
          {
            "value": "356 vs 1,607 ms",
            "num": true
          },
          {
            "value": "4.51x",
            "num": true
          }
        ],
        [
          "APPEND \u2014 write",
          {
            "value": "104 ms",
            "num": true
          },
          {
            "value": "1 round-trip + GET",
            "num": false
          }
        ],
        [
          "APPEND \u2014 idempotent skip",
          {
            "value": "51 ms",
            "num": true
          },
          {
            "value": "100% skipped on re-run",
            "num": false
          }
        ]
      ]
    },
    {
      "heading": "Concurrency, Locking & Resilience",
      "type": "table",
      "columns": [
        "Check",
        "Measured",
        "Expected"
      ],
      "rows": [
        [
          "Lock \u2014 acquire when free",
          {
            "value": "159 ms (rc 0)",
            "num": true
          },
          "rc 0"
        ],
        [
          "Lock \u2014 contended acquire",
          {
            "value": "58 ms (rc 75)",
            "num": true
          },
          "rc 75 fast-fail"
        ],
        [
          "Lock \u2014 release",
          {
            "value": "133 ms",
            "num": true
          },
          "owned release"
        ],
        [
          "Offline queue \u2014 enqueue",
          {
            "value": "0.08 ms",
            "num": true
          },
          "local write-ahead"
        ],
        [
          "read_many dedup (60 -> 20)",
          {
            "value": "deduped",
            "num": false
          },
          "collapse to unique set"
        ],
        [
          "Soak drift (late vs early)",
          {
            "value": "0.99x",
            "num": true
          },
          "<= 1.5x, no leak"
        ]
      ]
    },
    {
      "heading": "Regression Gates (relative-ratio policy)",
      "type": "table",
      "columns": [
        "Gate",
        "Threshold",
        "Measured",
        "Result"
      ],
      "rows": [
        [
          "read-full speedup",
          ">= 1.8x",
          {
            "value": "7.13x",
            "num": true
          },
          "Pass"
        ],
        [
          "bulk-get speedup",
          ">= 1.5x",
          {
            "value": "4.51x",
            "num": true
          },
          "Pass"
        ],
        [
          "pool warm-up cold/warm",
          ">= 1.5x",
          {
            "value": "3.25x",
            "num": true
          },
          "Pass"
        ],
        [
          "append idempotency",
          "100% skip",
          {
            "value": "100%",
            "num": true
          },
          "Pass"
        ],
        [
          "soak errors",
          "0",
          {
            "value": "0",
            "num": true
          },
          "Pass"
        ],
        [
          "soak drift",
          "<= 1.5x",
          {
            "value": "0.99x",
            "num": true
          },
          "Pass"
        ],
        [
          "lock semantics",
          "rc 0 / rc 75",
          {
            "value": "0 / 75",
            "num": true
          },
          "Pass"
        ],
        [
          "read_many dedup",
          "unique set",
          {
            "value": "20/20",
            "num": true
          },
          "Pass"
        ]
      ]
    },
    {
      "heading": "Findings",
      "type": "olist",
      "items": [
        "Concurrency delivers as designed. Eight workers cut full-vault reads 7.5x; the curve is near-linear to 8 workers and flattens after (8->16 returns only 1.24x). The default of 8 is the right setting for this vault and link.",
        "Keep-alive is confirmed live. A cold request costs 163 ms against a 50 ms warm median (3.25x). This is the direct measure that the 1.1.0 pooling fix removed the per-request TLS handshake that was timing out full-vault passes.",
        "Entity resolution is effectively free. resolve() returns in under 1 ms across exact, alias, and shortened mentions. The 1.2.0 alias work resolves shortened names (\"echo memory\", \"ECHO plugin\", \"operator\") straight to the canonical note, so the anti-duplicate guard holds without a fuzzy fallback.",
        "recall() latency was traced to the graph layer, not BM25. The BM25 index is already persisted and incrementally maintained (load 500 ms, score 0.1 ms). The cost was expand_graph fetching each neighbour serially \u2014 3.0 s for 126 nodes. Fix: the graph BFS now fetches each hop concurrently via the existing read_many. expand_graph is 3.5-4.2x faster with byte-identical ranking and scores; end-to-end recall() dropped from 2.8-4.7 s to 2.1-3.0 s per subject.",
        "Full-vault maintenance scripts stay under the tool timeout. sweep.py runs in 4.6 s and vault_lint.py in 5.1 s on 197 notes. The original failure mode (passes exceeding the timeout and dropping the session) does not recur.",
        "Write integrity holds. PUT verifies via read-back at 115 ms; APPEND is whole-line idempotent, skipping at 51 ms with zero duplicate writes on re-run. The advisory lock fast-fails a contended acquire with the expected exit 75."
      ]
    },
    {
      "heading": "Recommendations",
      "type": "olist",
      "items": [
        "Apply read_many to every code path that reads a set of known notes. Done for expand_graph (gated). Remaining: prefetch the _brief result bodies recall prints, and parallelize rebuild()'s vault walk.",
        "Reuse one in-process read cache across a recall() call so load_index, expansion, and _brief don't re-fetch overlapping notes \u2014 this closes the remaining ~500 ms + serial _brief cost.",
        "Make 'prefer read_many over a GET loop' a documented contract in the plugin's API reference; serial multi-note reads are a performance bug. The expand-graph gate enforces it for recall.",
        "Hold ECHO_WORKERS at 8 (16 buys only 1.24x). Promote the resolve correctness table from INFO to FAIL now that live behavior is confirmed. Re-baseline at the next 1.x release or past ~400 notes."
      ]
    },
    {
      "heading": "Method",
      "type": "text",
      "body": [
        "Each metric was timed with a perf-counter harness that reuses the live ECHO client, so it exercises the real pooled and concurrent code path. Reads report median and p95 over repeated iterations after a discarded warm-up; write metrics ran in a disposable namespace under the advisory lock and were deleted on completion (zero residual files confirmed).",
        "Gates use relative ratios rather than absolute milliseconds, so they stay valid regardless of the network the suite runs from. The harness, fixtures, baselines, and per-metric JSON are checked in under eval/perf/."
      ]
    }
  ],
  "note": {
    "heading": "Next Step",
    "body": "Prefetch recall()'s _brief reads and parallelize rebuild() via read_many, then re-run subject-pull to confirm recall drops toward ~1 s. expand_graph fix is local-only until ported to the canonical repo. No blockers in 1.1.0 / 1.2.0."
  }
}