1
0
forked from jason/echo

Phase 1: mechanical echo→chorus rename with back-compat shims

Identity rename, no behavior change (CHORUS-PLAN.md Phase 1):
- Plugin echo-memory → chorus-memory: manifest (v2.0.0-alpha.1), skill dir,
  16 scripts (chorus.py, chorus_config.py, …), EchoError → ChorusError,
  /chorus-* commands, hook paths, docs, scaffold seeds, eval harness,
  build.py. Docs endpoint → chorusapi.mpm.to.
- Env ECHO_* → CHORUS_*; config → ~/.claude/chorus-memory/config.json;
  state dir → ~/.chorus-memory/; marker → _agent/chorus-vault.md.

Back-compat shims (one major version):
- chorus_config aliases legacy ECHO_* env at import; reads a legacy
  echo-memory config.json when no CHORUS config exists (writes never
  land there); doctor/config report the legacy source.
- State dir honors an existing ~/.echo-memory when the new dir is absent
  (offline queue not stranded).
- Marker dual-probe in load/doctor/bootstrap/lint/sweep/migrate: a
  pre-fork _agent/echo-vault.md counts as bootstrapped; bootstrap repair
  won't write a second marker; routing gains agent-marker-legacy (GET).

Verified: 25/25 unit tests, scaffold + routing-sync checks, 4 mock e2e
suites, run_eval metrics unchanged, +6 shim smoke tests green.
Rebuilt chorus-memory.plugin (79 entries).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-21 13:35:51 -05:00
parent ff2f2efd0b
commit d366ca4032
98 changed files with 1312 additions and 1164 deletions
+5 -5
View File
@@ -1,4 +1,4 @@
# echo-memory eval — current-version metrics harness
# chorus-memory eval — current-version metrics harness
A reproducible, credential-free eval of the shipped plugin against the deterministic
mock REST API. `run_eval.py` measures the four things the plugin claims (results print
@@ -9,7 +9,7 @@ as a table and land in `results/latest.json`):
(per-word substring ranking over entity notes only). Precision@5 / recall@5 / MRR,
session/journal answerability, freshness top-1.
2. **Dedup** — duplicate notes created replaying a capture stream with the pre-write
gate ON (default) vs OFF (`ECHO_DUP_GATE=99` ≈ pre-1.5 warn-only), plus legitimate
gate ON (default) vs OFF (`CHORUS_DUP_GATE=99` ≈ pre-1.5 warn-only), plus legitimate
captures wrongly blocked (must be 0).
3. **Write safety** — silent write failures under fault injection (transient 503,
phantom PUT, PATCH to a missing heading, replayed append). Must be 0; must be loud.
@@ -29,7 +29,7 @@ python3 test_features.py # end-to-end features vs the mock (capture/reca
python3 test_reflect.py # reflection pipeline
python3 test_offline_queue.py # H2 outage queue + read cache
python3 test_patch_semantics.py # real PATCH semantics vs mock_olrapi_hifi.py (incl. fm create-or-replace)
# plus, in the plugin tree: scripts/test_echo_client.py (offline unit + routing-sync guard)
# plus, in the plugin tree: scripts/test_chorus_client.py (offline unit + routing-sync guard)
```
## Run it (historical A/B)
@@ -56,7 +56,7 @@ Results table prints to stdout and a machine-readable copy lands in `results/lat
- a `PATCH` to a missing heading → `400` (the silent-write-loss trigger).
- **`run_eval.py`** — for each scenario, runs both methods against a freshly reset +
re-seeded server (so faults are identical for both), then reads ground truth back
**independently** from the mock. The 0.7 side executes the *actual shipped `echo.py`*;
**independently** from the mock. The 0.7 side executes the *actual shipped `chorus.py`*;
the 0.6 side faithfully models the documented recipe (real HTTP, but no status check,
no retry, no verify, no dedupe).
@@ -106,7 +106,7 @@ effective tokens (assumed) 6723 -> 334
To measure real model behavior and true token counts:
1. Define the same six scenarios as natural-language tasks (e.g. "log a session note for X").
2. Run each twice — once with the 0.6 skill files, once with 0.7 — through the Agent SDK
against the **mock** server (point `ECHO_BASE` at it) so faults stay deterministic.
against the **mock** server (point `CHORUS_BASE` at it) so faults stay deterministic.
3. Record `usage.output_tokens` per task from the API and whether the vault ended correct
(same ground-truth read used here).