forked from jason/echo
Phase 1: mechanical echo→chorus rename with back-compat shims
Identity rename, no behavior change (CHORUS-PLAN.md Phase 1): - Plugin echo-memory → chorus-memory: manifest (v2.0.0-alpha.1), skill dir, 16 scripts (chorus.py, chorus_config.py, …), EchoError → ChorusError, /chorus-* commands, hook paths, docs, scaffold seeds, eval harness, build.py. Docs endpoint → chorusapi.mpm.to. - Env ECHO_* → CHORUS_*; config → ~/.claude/chorus-memory/config.json; state dir → ~/.chorus-memory/; marker → _agent/chorus-vault.md. Back-compat shims (one major version): - chorus_config aliases legacy ECHO_* env at import; reads a legacy echo-memory config.json when no CHORUS config exists (writes never land there); doctor/config report the legacy source. - State dir honors an existing ~/.echo-memory when the new dir is absent (offline queue not stranded). - Marker dual-probe in load/doctor/bootstrap/lint/sweep/migrate: a pre-fork _agent/echo-vault.md counts as bootstrapped; bootstrap repair won't write a second marker; routing gains agent-marker-legacy (GET). Verified: 25/25 unit tests, scaffold + routing-sync checks, 4 mock e2e suites, run_eval metrics unchanged, +6 shim smoke tests green. Rebuilt chorus-memory.plugin (79 entries). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+5
-5
@@ -1,4 +1,4 @@
|
||||
# echo-memory eval — current-version metrics harness
|
||||
# chorus-memory eval — current-version metrics harness
|
||||
|
||||
A reproducible, credential-free eval of the shipped plugin against the deterministic
|
||||
mock REST API. `run_eval.py` measures the four things the plugin claims (results print
|
||||
@@ -9,7 +9,7 @@ as a table and land in `results/latest.json`):
|
||||
(per-word substring ranking over entity notes only). Precision@5 / recall@5 / MRR,
|
||||
session/journal answerability, freshness top-1.
|
||||
2. **Dedup** — duplicate notes created replaying a capture stream with the pre-write
|
||||
gate ON (default) vs OFF (`ECHO_DUP_GATE=99` ≈ pre-1.5 warn-only), plus legitimate
|
||||
gate ON (default) vs OFF (`CHORUS_DUP_GATE=99` ≈ pre-1.5 warn-only), plus legitimate
|
||||
captures wrongly blocked (must be 0).
|
||||
3. **Write safety** — silent write failures under fault injection (transient 503,
|
||||
phantom PUT, PATCH to a missing heading, replayed append). Must be 0; must be loud.
|
||||
@@ -29,7 +29,7 @@ python3 test_features.py # end-to-end features vs the mock (capture/reca
|
||||
python3 test_reflect.py # reflection pipeline
|
||||
python3 test_offline_queue.py # H2 outage queue + read cache
|
||||
python3 test_patch_semantics.py # real PATCH semantics vs mock_olrapi_hifi.py (incl. fm create-or-replace)
|
||||
# plus, in the plugin tree: scripts/test_echo_client.py (offline unit + routing-sync guard)
|
||||
# plus, in the plugin tree: scripts/test_chorus_client.py (offline unit + routing-sync guard)
|
||||
```
|
||||
|
||||
## Run it (historical A/B)
|
||||
@@ -56,7 +56,7 @@ Results table prints to stdout and a machine-readable copy lands in `results/lat
|
||||
- a `PATCH` to a missing heading → `400` (the silent-write-loss trigger).
|
||||
- **`run_eval.py`** — for each scenario, runs both methods against a freshly reset +
|
||||
re-seeded server (so faults are identical for both), then reads ground truth back
|
||||
**independently** from the mock. The 0.7 side executes the *actual shipped `echo.py`*;
|
||||
**independently** from the mock. The 0.7 side executes the *actual shipped `chorus.py`*;
|
||||
the 0.6 side faithfully models the documented recipe (real HTTP, but no status check,
|
||||
no retry, no verify, no dedupe).
|
||||
|
||||
@@ -106,7 +106,7 @@ effective tokens (assumed) 6723 -> 334
|
||||
To measure real model behavior and true token counts:
|
||||
1. Define the same six scenarios as natural-language tasks (e.g. "log a session note for X").
|
||||
2. Run each twice — once with the 0.6 skill files, once with 0.7 — through the Agent SDK
|
||||
against the **mock** server (point `ECHO_BASE` at it) so faults stay deterministic.
|
||||
against the **mock** server (point `CHORUS_BASE` at it) so faults stay deterministic.
|
||||
3. Record `usage.output_tokens` per task from the API and whether the vault ended correct
|
||||
(same ground-truth read used here).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user