forked from jason/echo
ver 0.9
This commit is contained in:
+3
-3
@@ -1,7 +1,7 @@
|
||||
# echo-memory eval — 0.6 vs 0.7 A/B harness
|
||||
|
||||
A reproducible, credential-free A/B comparison of the plugin **before** (0.6: raw-curl
|
||||
recipes from `SKILL.md`) and **after** (0.7: the shipped `scripts/echo.sh` client) the
|
||||
recipes from `SKILL.md`) and **after** (the shipped `scripts/echo.py` client) the
|
||||
hardening work. It quantifies the claims in the comparative analysis: token cost of the
|
||||
I/O layer, and the rate of **silent write failures**.
|
||||
|
||||
@@ -14,7 +14,7 @@ python3 run_eval.py --recovery 2500 # sensitivity-test the recovery assumption
|
||||
python3 run_eval.py --cpt 3.5 # different chars/token proxy
|
||||
```
|
||||
|
||||
No network, no API key, no live vault. Pure stdlib (Python 3 + bash for `echo.sh`).
|
||||
No network, no API key, no live vault. Pure stdlib Python 3 (the shipped client is now Python — no bash required).
|
||||
Results table prints to stdout and a machine-readable copy lands in `results/latest.json`.
|
||||
|
||||
## How it works
|
||||
@@ -29,7 +29,7 @@ Results table prints to stdout and a machine-readable copy lands in `results/lat
|
||||
- a `PATCH` to a missing heading → `400` (the silent-write-loss trigger).
|
||||
- **`run_eval.py`** — for each scenario, runs both methods against a freshly reset +
|
||||
re-seeded server (so faults are identical for both), then reads ground truth back
|
||||
**independently** from the mock. The 0.7 side executes the *actual shipped `echo.sh`*;
|
||||
**independently** from the mock. The 0.7 side executes the *actual shipped `echo.py`*;
|
||||
the 0.6 side faithfully models the documented recipe (real HTTP, but no status check,
|
||||
no retry, no verify, no dedupe).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user