1
0
forked from jason/echo
This commit is contained in:
jason
2026-06-21 11:46:54 -05:00
parent 88210a4e84
commit d404f6e96f
38 changed files with 2561 additions and 1046 deletions
+3 -3
View File
@@ -1,7 +1,7 @@
# echo-memory eval — 0.6 vs 0.7 A/B harness
A reproducible, credential-free A/B comparison of the plugin **before** (0.6: raw-curl
recipes from `SKILL.md`) and **after** (0.7: the shipped `scripts/echo.sh` client) the
recipes from `SKILL.md`) and **after** (the shipped `scripts/echo.py` client) the
hardening work. It quantifies the claims in the comparative analysis: token cost of the
I/O layer, and the rate of **silent write failures**.
@@ -14,7 +14,7 @@ python3 run_eval.py --recovery 2500 # sensitivity-test the recovery assumption
python3 run_eval.py --cpt 3.5 # different chars/token proxy
```
No network, no API key, no live vault. Pure stdlib (Python 3 + bash for `echo.sh`).
No network, no API key, no live vault. Pure stdlib Python 3 (the shipped client is now Python — no bash required).
Results table prints to stdout and a machine-readable copy lands in `results/latest.json`.
## How it works
@@ -29,7 +29,7 @@ Results table prints to stdout and a machine-readable copy lands in `results/lat
- a `PATCH` to a missing heading → `400` (the silent-write-loss trigger).
- **`run_eval.py`** — for each scenario, runs both methods against a freshly reset +
re-seeded server (so faults are identical for both), then reads ground truth back
**independently** from the mock. The 0.7 side executes the *actual shipped `echo.sh`*;
**independently** from the mock. The 0.7 side executes the *actual shipped `echo.py`*;
the 0.6 side faithfully models the documented recipe (real HTTP, but no status check,
no retry, no verify, no dedupe).