jason/chorus
1
0
Fork 0
forked from jason/echo
Code Pull Requests Activity

Phase 1: mechanical echo→chorus rename with back-compat shims

Browse Source 
Identity rename, no behavior change (CHORUS-PLAN.md Phase 1):
- Plugin echo-memory → chorus-memory: manifest (v2.0.0-alpha.1), skill dir,
  16 scripts (chorus.py, chorus_config.py, …), EchoError → ChorusError,
  /chorus-* commands, hook paths, docs, scaffold seeds, eval harness,
  build.py. Docs endpoint → chorusapi.mpm.to.
- Env ECHO_* → CHORUS_*; config → ~/.claude/chorus-memory/config.json;
  state dir → ~/.chorus-memory/; marker → _agent/chorus-vault.md.

Back-compat shims (one major version):
- chorus_config aliases legacy ECHO_* env at import; reads a legacy
  echo-memory config.json when no CHORUS config exists (writes never
  land there); doctor/config report the legacy source.
- State dir honors an existing ~/.echo-memory when the new dir is absent
  (offline queue not stranded).
- Marker dual-probe in load/doctor/bootstrap/lint/sweep/migrate: a
  pre-fork _agent/echo-vault.md counts as bootstrapped; bootstrap repair
  won't write a second marker; routing gains agent-marker-legacy (GET).

Verified: 25/25 unit tests, scaffold + routing-sync checks, 4 mock e2e
suites, run_eval metrics unchanged, +6 shim smoke tests green.
Rebuilt chorus-memory.plugin (79 entries).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jason Stedwell
2026-07-21 13:35:51 -05:00
parent ff2f2efd0b
commit d366ca4032
98 changed files with 1312 additions and 1164 deletions
Download Patch File Download Diff File Expand all files Collapse all files
eval/README.md
+5 -5
View File
@@ -1,4 +1,4 @@
# echo-memory eval — current-version metrics harness
# chorus-memory eval — current-version metrics harness
A reproducible, credential-free eval of the shipped plugin against the deterministic
mock REST API. `run_eval.py` measures the four things the plugin claims (results print
@@ -9,7 +9,7 @@ as a table and land in `results/latest.json`):
(per-word substring ranking over entity notes only). Precision@5 / recall@5 / MRR,
session/journal answerability, freshness top-1.
2. **Dedup** — duplicate notes created replaying a capture stream with the pre-write
gate ON (default) vs OFF (`ECHO_DUP_GATE=99` ≈ pre-1.5 warn-only), plus legitimate
gate ON (default) vs OFF (`CHORUS_DUP_GATE=99` ≈ pre-1.5 warn-only), plus legitimate
captures wrongly blocked (must be 0).
3. **Write safety** — silent write failures under fault injection (transient 503,
phantom PUT, PATCH to a missing heading, replayed append). Must be 0; must be loud.
@@ -29,7 +29,7 @@ python3 test_features.py # end-to-end features vs the mock (capture/reca
python3 test_reflect.py # reflection pipeline
python3 test_offline_queue.py # H2 outage queue + read cache
python3 test_patch_semantics.py # real PATCH semantics vs mock_olrapi_hifi.py (incl. fm create-or-replace)
# plus, in the plugin tree: scripts/test_echo_client.py (offline unit + routing-sync guard)
# plus, in the plugin tree: scripts/test_chorus_client.py (offline unit + routing-sync guard)
```
## Run it (historical A/B)
@@ -56,7 +56,7 @@ Results table prints to stdout and a machine-readable copy lands in `results/lat
- a `PATCH` to a missing heading → `400` (the silent-write-loss trigger).
- **`run_eval.py`** — for each scenario, runs both methods against a freshly reset +
re-seeded server (so faults are identical for both), then reads ground truth back
**independently** from the mock. The 0.7 side executes the *actual shipped `echo.py`*;
**independently** from the mock. The 0.7 side executes the *actual shipped `chorus.py`*;
the 0.6 side faithfully models the documented recipe (real HTTP, but no status check,
no retry, no verify, no dedupe).
@@ -106,7 +106,7 @@ effective tokens (assumed) 6723 -> 334
To measure real model behavior and true token counts:
1. Define the same six scenarios as natural-language tasks (e.g. "log a session note for X").
2. Run each twice — once with the 0.6 skill files, once with 0.7 — through the Agent SDK
against the **mock** server (point `ECHO_BASE` at it) so faults stay deterministic.
against the **mock** server (point `CHORUS_BASE` at it) so faults stay deterministic.
3. Record `usage.output_tokens` per task from the API and whether the vault ended correct
(same ground-truth read used here).
eval/mock_olrapi.py
+1 -1
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
"""mock_olrapi.py — a deterministic mock of the Obsidian Local REST API surface the
echo-memory plugin uses, with controllable fault injection.
chorus-memory plugin uses, with controllable fault injection.
It reproduces the real API's observed behavior and quirks so the A/B eval can run
without credentials and without touching the live vault:
eval/run_eval.py
+29 -29
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""run_eval.py — current-version metrics for echo-memory (v1.5+).
"""run_eval.py — current-version metrics for chorus-memory (v1.5+).
Replaces the historical 0.6-vs-0.7 A/B harness (see git history for that version).
Measures the four things the plugin now claims, against the deterministic mock
@@ -11,7 +11,7 @@ REST API (mock_olrapi.py) — no credentials, no live vault:
Metrics: precision@5, recall@5, MRR, session/journal queries
answered, freshness top-1 correctness.
2. DEDUP — duplicate notes created replaying a capture stream with the
pre-write gate ON (default) vs OFF (ECHO_DUP_GATE=99, i.e. the
pre-write gate ON (default) vs OFF (CHORUS_DUP_GATE=99, i.e. the
pre-1.5 warn-only behavior). Also counts legitimate captures
wrongly blocked (must be 0).
3. WRITE SAFETY— silent write failures (claimed success but ground truth wrong)
@@ -40,8 +40,8 @@ import urllib.request
from pathlib import Path
HERE = Path(__file__).resolve().parent
SCRIPTS = HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts"
ECHO = SCRIPTS / "echo.py"
SCRIPTS = HERE.parent / "chorus-memory.plugin.src" / "skills" / "chorus-memory" / "scripts"
CHORUS = SCRIPTS / "chorus.py"
SWEEP = SCRIPTS / "sweep.py"
KEY = "test-key-not-a-real-secret"
TODAY = "2026-07-03"
@@ -76,8 +76,8 @@ class Harness:
return None if body == "<<MISSING>>" else body
def run(self, script, *args, env_extra=None, base=None):
env = dict(os.environ, ECHO_BASE=base or self.base, ECHO_KEY=KEY,
ECHO_VERIFY="1", ECHO_TODAY=TODAY, ECHO_STATE_DIR=self.state_dir,
env = dict(os.environ, CHORUS_BASE=base or self.base, CHORUS_KEY=KEY,
CHORUS_VERIFY="1", CHORUS_TODAY=TODAY, CHORUS_STATE_DIR=self.state_dir,
**(env_extra or {}))
return subprocess.run([sys.executable, str(script), *args],
capture_output=True, text=True, env=env)
@@ -179,7 +179,7 @@ def metrics_for(ranked_by_query):
def eval_retrieval(h):
h.reset()
h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n")
h.seed("_agent/chorus-vault.md", "---\nschema_version: 4\n---\n# marker\n")
for path, text in {**ENTITY_CORPUS, **TIME_SERIES_CORPUS}.items():
h.seed(path, text)
r = h.run(SWEEP, "--apply")
@@ -188,7 +188,7 @@ def eval_retrieval(h):
current_ranked, baseline_ranked = [], []
fresh_top1_current = fresh_top1_baseline = None
for q, gold, ts_only in QUERIES:
r = h.run(ECHO, "recall", q, "--json", "--limit", "5")
r = h.run(CHORUS, "recall", q, "--json", "--limit", "5")
try:
primary = [hit["path"] for hit in json.loads(r.stdout).get("primary", [])]
except Exception: # noqa: BLE001
@@ -212,28 +212,28 @@ def eval_retrieval(h):
# Attempts referring to an EXISTING entity under another name; a create == duplicate.
DUP_ATTEMPTS = [
("Robert Smith", "person", "resources/people/robert-smith.md"),
("Echo Memory", "project", "projects/active/echo-memory.md"),
("Chorus Memory", "project", "projects/active/chorus-memory.md"),
("Acme Consulting", "company", "resources/companies/acme-consulting.md"),
]
# Legitimate NEW entities; a block == false positive (must create in both modes).
LEGIT_ATTEMPTS = [
("Echo Review Meeting", "meeting",
f"resources/meetings/{TODAY}-echo-review-meeting.md"), # cross-kind lookalike
("Chorus Review Meeting", "meeting",
f"resources/meetings/{TODAY}-chorus-review-meeting.md"), # cross-kind lookalike
("Zephyr", "concept", "resources/concepts/zephyr.md"), # unrelated
]
def _dedup_pass(h, gate_env):
h.reset()
h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n")
for title, kind in [("Bob Smith", "person"), ("Echo", "project"), ("Acme", "company")]:
h.run(ECHO, "capture", title, "--kind", kind, env_extra=gate_env)
h.seed("_agent/chorus-vault.md", "---\nschema_version: 4\n---\n# marker\n")
for title, kind in [("Bob Smith", "person"), ("Chorus", "project"), ("Acme", "company")]:
h.run(CHORUS, "capture", title, "--kind", kind, env_extra=gate_env)
dupes = blocked_legit = 0
for title, kind, would_be in DUP_ATTEMPTS:
h.run(ECHO, "capture", title, "--kind", kind, env_extra=gate_env)
h.run(CHORUS, "capture", title, "--kind", kind, env_extra=gate_env)
dupes += int(h.ground(would_be) is not None)
for title, kind, path in LEGIT_ATTEMPTS:
h.run(ECHO, "capture", title, "--kind", kind, env_extra=gate_env)
h.run(CHORUS, "capture", title, "--kind", kind, env_extra=gate_env)
blocked_legit += int(h.ground(path) is None)
return {"duplicate_notes_created": dupes, "legitimate_captures_blocked": blocked_legit,
"referring_attempts": len(DUP_ATTEMPTS), "legit_attempts": len(LEGIT_ATTEMPTS)}
@@ -241,13 +241,13 @@ def _dedup_pass(h, gate_env):
def eval_dedup(h):
return {"gate ON (v1.5.1 default)": _dedup_pass(h, {}),
"gate OFF (pre-1.5, warn-only)": _dedup_pass(h, {"ECHO_DUP_GATE": "99"})}
"gate OFF (pre-1.5, warn-only)": _dedup_pass(h, {"CHORUS_DUP_GATE": "99"})}
# ------------------------------------------------------------- 3. WRITE SAFETY
def eval_write_safety(h):
h.reset()
h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n")
h.seed("_agent/chorus-vault.md", "---\nschema_version: 4\n---\n# marker\n")
silent = loud = 0
ops = 0
@@ -259,7 +259,7 @@ def eval_write_safety(h):
# transient 503 on first write ("flaky" marker) -> client must retry and land it
ops += 1
r = h.run(ECHO, "put", "resources/concepts/flaky-note.md", bodyfile("# Flaky\nok\n"))
r = h.run(CHORUS, "put", "resources/concepts/flaky-note.md", bodyfile("# Flaky\nok\n"))
landed = h.ground("resources/concepts/flaky-note.md") is not None
if r.returncode == 0 and not landed:
silent += 1
@@ -268,7 +268,7 @@ def eval_write_safety(h):
# phantom PUT ("phantom" marker: 200 but not persisted) -> verify must fail LOUD
ops += 1
r = h.run(ECHO, "put", "resources/concepts/phantom-note.md", bodyfile("# Phantom\n"))
r = h.run(CHORUS, "put", "resources/concepts/phantom-note.md", bodyfile("# Phantom\n"))
landed = h.ground("resources/concepts/phantom-note.md") is not None
if r.returncode == 0 and not landed:
silent += 1
@@ -278,7 +278,7 @@ def eval_write_safety(h):
# PATCH to a missing heading -> 400 must be surfaced, nothing written
ops += 1
h.seed("resources/concepts/target.md", "# Target\n\n## Real\nx\n")
r = h.run(ECHO, "patch", "resources/concepts/target.md", "append", "heading",
r = h.run(CHORUS, "patch", "resources/concepts/target.md", "append", "heading",
"Target::Nope", bodyfile("lost?"))
if r.returncode == 0 and "lost?" not in (h.ground("resources/concepts/target.md") or ""):
silent += 1
@@ -287,8 +287,8 @@ def eval_write_safety(h):
# replayed append -> whole-line idempotency, no duplicate
ops += 1
h.run(ECHO, "append", "inbox/captures/inbox.md", f"- {TODAY}: replay me")
h.run(ECHO, "append", "inbox/captures/inbox.md", f"- {TODAY}: replay me")
h.run(CHORUS, "append", "inbox/captures/inbox.md", f"- {TODAY}: replay me")
h.run(CHORUS, "append", "inbox/captures/inbox.md", f"- {TODAY}: replay me")
n = (h.ground("inbox/captures/inbox.md") or "").count("replay me")
if n != 1:
silent += 1
@@ -300,15 +300,15 @@ def eval_write_safety(h):
# --------------------------------------------------------------- 4. DURABILITY
def eval_durability(h, dead_base):
h.reset()
h.seed("_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n")
h.seed("_agent/chorus-vault.md", "---\nschema_version: 4\n---\n# marker\n")
queued_ok = 0
for i in range(3):
r = h.run(ECHO, "append", "inbox/captures/inbox.md",
r = h.run(CHORUS, "append", "inbox/captures/inbox.md",
f"- {TODAY}: offline capture {i}", base=dead_base)
queued_ok += int(r.returncode == 0 and "queued" in r.stdout)
# vault returns: flush replays; a second flush must not duplicate
h.run(ECHO, "flush")
h.run(ECHO, "flush")
h.run(CHORUS, "flush")
h.run(CHORUS, "flush")
inbox = h.ground("inbox/captures/inbox.md") or ""
landed = sum(1 for i in range(3) if f"offline capture {i}" in inbox)
dupes = sum(inbox.count(f"offline capture {i}") - 1
@@ -325,7 +325,7 @@ def main():
base = f"http://127.0.0.1:{a.port}"
srv = subprocess.Popen([sys.executable, str(HERE / "mock_olrapi.py"), "--port", str(a.port)],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
state = tempfile.mkdtemp(prefix="echo-eval-state-")
state = tempfile.mkdtemp(prefix="chorus-eval-state-")
try:
for _ in range(50):
try:
@@ -345,7 +345,7 @@ def main():
out.parent.mkdir(exist_ok=True)
out.write_text(json.dumps(results, indent=2), encoding="utf-8")
print(f"\n== echo-memory eval — v{results['version']} ({results['date']}) ==\n")
print(f"\n== chorus-memory eval — v{results['version']} ({results['date']}) ==\n")
for section, data in results.items():
if section in ("version", "date"):
continue
eval/test_features.py
+45 -45
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
"""test_features.py — end-to-end test of the v0.9 features against the mock OLRAPI.
Exercises the real shipped scripts (echo.py capture/resolve/recall/link + sweep.py)
Exercises the real shipped scripts (chorus.py capture/resolve/recall/link + sweep.py)
through subprocess, then reads ground truth back from the mock. No network, no creds,
no live vault.
@@ -11,8 +11,8 @@ import argparse, json, os, subprocess, sys, time, urllib.request, urllib.error
from pathlib import Path
HERE = Path(__file__).resolve().parent
SCRIPTS = HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts"
ECHO = SCRIPTS / "echo.py"
SCRIPTS = HERE.parent / "chorus-memory.plugin.src" / "skills" / "chorus-memory" / "scripts"
CHORUS = SCRIPTS / "chorus.py"
SWEEP = SCRIPTS / "sweep.py"
KEY = "test-key-not-a-real-secret"
@@ -51,9 +51,9 @@ class Harness:
_, body = self.http("GET", f"{self.base}/__debug__?path={path}")
return None if body == "<<MISSING>>" else body
def echo(self, *args):
env = dict(os.environ, ECHO_BASE=self.base, ECHO_KEY=KEY, ECHO_VERIFY="1",
ECHO_TODAY="2026-06-21")
def chorus(self, *args):
env = dict(os.environ, CHORUS_BASE=self.base, CHORUS_KEY=KEY, CHORUS_VERIFY="1",
CHORUS_TODAY="2026-06-21")
return subprocess.run([sys.executable, str(args[0]), *args[1:]],
capture_output=True, text=True, env=env)
@@ -73,10 +73,10 @@ def main():
time.sleep(0.1)
h = Harness(base)
h.reset()
h.seed("_agent/echo-vault.md", "---\nschema_version: 2\n---\n# marker\n")
h.seed("_agent/chorus-vault.md", "---\nschema_version: 2\n---\n# marker\n")
# 1. capture a person (with an alias)
r = h.echo(ECHO, "capture", "Bob Smith", "--kind", "person", "--aliases", "bob,rs")
r = h.chorus(CHORUS, "capture", "Bob Smith", "--kind", "person", "--aliases", "bob,rs")
note = h.ground("resources/people/bob-smith.md")
check("capture person creates note", note is not None and "type: person" in note, r.stderr)
check("capture stamps agent_written", note and "agent_written: true" in note)
@@ -84,14 +84,14 @@ def main():
check("index records entity", idx and "bob-smith" in idx and '"bob"' in idx)
# 2. resolve by alias
r = h.echo(ECHO, "resolve", "bob")
r = h.chorus(CHORUS, "resolve", "bob")
out = json.loads(r.stdout) if r.stdout.strip().startswith("{") else {}
check("resolve alias -> path", out.get("match") and out.get("path") == "resources/people/bob-smith.md", r.stdout)
# 3. capture a company whose body mentions Bob Smith -> auto bidirectional link
r = subprocess.run([sys.executable, str(ECHO), "capture", "MPM", "--kind", "company", "-"],
r = subprocess.run([sys.executable, str(CHORUS), "capture", "MPM", "--kind", "company", "-"],
input="Bob Smith is the principal here.\n", capture_output=True, text=True,
env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_TODAY="2026-06-21"))
env=dict(os.environ, CHORUS_BASE=base, CHORUS_KEY=KEY, CHORUS_TODAY="2026-06-21"))
mpm = h.ground("resources/companies/mpm.md")
bob = h.ground("resources/people/bob-smith.md")
check("auto-link forward (mpm -> bob)", mpm and "[[resources/people/bob-smith]]" in mpm, r.stdout + r.stderr)
@@ -99,32 +99,32 @@ def main():
# 4. recall an entity surfaces its linked neighborhood (search is mocked empty,
# so this exercises the index-resolve + 1-hop expansion path)
r = h.echo(ECHO, "recall", "MPM")
r = h.chorus(CHORUS, "recall", "MPM")
check("recall surfaces linked person", "resources/people/bob-smith.md" in r.stdout, r.stdout)
# 4b. BM25 finds a note by a BODY term. /search/simple is mocked empty, so a hit
# here can only come from the local BM25 recall index (proves H1 lexical layer).
r = h.echo(ECHO, "recall", "principal")
r = h.chorus(CHORUS, "recall", "principal")
check("recall finds note by body term (BM25)",
"resources/companies/mpm.md" in r.stdout, r.stdout)
# 5. explicit link between two fresh notes
h.echo(ECHO, "capture", "Alpha", "--kind", "concept")
h.echo(ECHO, "capture", "Beta", "--kind", "concept")
h.echo(ECHO, "link", "resources/concepts/alpha.md", "resources/concepts/beta.md")
h.chorus(CHORUS, "capture", "Alpha", "--kind", "concept")
h.chorus(CHORUS, "capture", "Beta", "--kind", "concept")
h.chorus(CHORUS, "link", "resources/concepts/alpha.md", "resources/concepts/beta.md")
alpha = h.ground("resources/concepts/alpha.md")
beta = h.ground("resources/concepts/beta.md")
check("link adds A->B", alpha and "[[resources/concepts/beta]]" in alpha)
check("link adds B->A", beta and "[[resources/concepts/alpha]]" in beta)
# 6. inbox capture (unknown home)
h.echo(ECHO, "capture", "stray thought", "--inbox")
h.chorus(CHORUS, "capture", "stray thought", "--inbox")
inbox = h.ground("inbox/captures/inbox.md")
check("inbox capture lands", inbox and "stray thought" in inbox)
# 7. sweep --apply rebuilds the index, symmetrizes, stamps schema 3
r = h.echo(SWEEP, "--apply")
marker = h.ground("_agent/echo-vault.md")
r = h.chorus(SWEEP, "--apply")
marker = h.ground("_agent/chorus-vault.md")
check("sweep runs clean", r.returncode == 0, r.stdout + r.stderr)
check("sweep stamps schema 4", marker and "schema_version=4" in marker.replace(": ", "="), marker)
idx2 = h.ground("_agent/index/entities.json")
@@ -140,8 +140,8 @@ def main():
'{"schema":1,"updated":"2026-06-21","entities":{"ext-entity":'
'{"path":"resources/concepts/ext-entity.md","kind":"concept","title":"Ext",'
'"aliases":[],"last_seen":"2026-06-21"}}}')
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_TODAY="2026-06-21", PYTHONPATH=str(SCRIPTS))
snippet = ("import echo_index as ix, echo_concurrency as c; "
env = dict(os.environ, CHORUS_BASE=base, CHORUS_KEY=KEY, CHORUS_TODAY="2026-06-21", PYTHONPATH=str(SCRIPTS))
snippet = ("import chorus_index as ix, chorus_concurrency as c; "
"c.atomic_index_update(lambda d: ix.upsert(d,'mine',"
"'resources/concepts/mine.md','concept','Mine'))")
subprocess.run([sys.executable, "-c", snippet], env=env, capture_output=True, text=True)
@@ -152,7 +152,7 @@ def main():
h.ground("_agent/locks/vault.lock") is None)
# 9. M4 — --dry-run previews and writes nothing; --json emits a parseable envelope.
r = h.echo(ECHO, "capture", "DryRun Co", "--kind", "company", "--dry-run", "--json")
r = h.chorus(CHORUS, "capture", "DryRun Co", "--kind", "company", "--dry-run", "--json")
try:
env = json.loads(r.stdout)
except Exception:
@@ -160,7 +160,7 @@ def main():
check("M4 --dry-run emits a dry-run JSON envelope",
env.get("action", "").startswith("dry-run") and env.get("path") == "resources/companies/dryrun-co.md", r.stdout)
check("M4 --dry-run writes nothing", h.ground("resources/companies/dryrun-co.md") is None)
r = h.echo(ECHO, "capture", "Json Co", "--kind", "company", "--json")
r = h.chorus(CHORUS, "capture", "Json Co", "--kind", "company", "--json")
try:
env = json.loads(r.stdout)
except Exception:
@@ -173,7 +173,7 @@ def main():
# ---------------- v1.5 features ----------------------------------------
# step 8 clobbered entities.json on purpose; rebuild the index from the notes
# so the v1.5 tests run against a coherent vault.
h.echo(SWEEP, "--apply")
h.chorus(SWEEP, "--apply")
# 10. frontmatter completeness at write time: entity notes are born classified.
bob2 = h.ground("resources/people/bob-smith.md") or ""
@@ -181,25 +181,25 @@ def main():
check("v1.5 capture seeds tags with the kind", "tags: [person]" in bob2, bob2)
# 11. update-capture preserves the FULL body (old code kept only line 1).
r = subprocess.run([sys.executable, str(ECHO), "capture", "Bob Smith", "-", "--kind", "person"],
r = subprocess.run([sys.executable, str(CHORUS), "capture", "Bob Smith", "-", "--kind", "person"],
input="met at expo\nsecond line survives\nthird line too\n",
capture_output=True, text=True,
env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_TODAY="2026-06-21"))
env=dict(os.environ, CHORUS_BASE=base, CHORUS_KEY=KEY, CHORUS_TODAY="2026-06-21"))
bob3 = h.ground("resources/people/bob-smith.md") or ""
check("v1.5 update keeps the dated bullet", "- 2026-06-21: met at expo" in bob3, r.stdout + r.stderr)
check("v1.5 update keeps EVERY body line",
" second line survives" in bob3 and " third line too" in bob3, bob3)
# 12. duplicate gate: same-kind + distinctive-token candidate STOPS creation...
r = h.echo(ECHO, "capture", "Robert Smith", "--kind", "person")
r = h.chorus(CHORUS, "capture", "Robert Smith", "--kind", "person")
check("v1.5 duplicate gate stops the create (exit 76)", r.returncode == 76, str(r.returncode))
check("v1.5 gate wrote nothing", h.ground("resources/people/robert-smith.md") is None)
# ... --force overrides ...
r = h.echo(ECHO, "capture", "Robert Smith", "--kind", "person", "--force")
r = h.chorus(CHORUS, "capture", "Robert Smith", "--kind", "person", "--force")
check("v1.5 --force creates past the gate",
h.ground("resources/people/robert-smith.md") is not None, r.stdout + r.stderr)
# 12b (1.5.1). cross-kind lookalike does NOT gate — it creates with a warning.
r = h.echo(ECHO, "capture", "Smith Consulting", "--kind", "company")
r = h.chorus(CHORUS, "capture", "Smith Consulting", "--kind", "company")
check("v1.5.1 cross-kind lookalike is not gated",
r.returncode == 0 and h.ground("resources/companies/smith-consulting.md") is not None,
str(r.returncode) + r.stdout + r.stderr)
@@ -207,14 +207,14 @@ def main():
"similar existing" in (r.stdout + r.stderr), r.stdout + r.stderr)
# 12c (1.5.1). a single vault-common token does NOT gate: once "acme" appears in
# two entities, "Acme Tools" creates (with a warning) instead of blocking.
h.echo(ECHO, "capture", "Acme", "--kind", "company")
h.echo(ECHO, "capture", "Acme Group", "--kind", "company", "--force")
r = h.echo(ECHO, "capture", "Acme Tools", "--kind", "company")
h.chorus(CHORUS, "capture", "Acme", "--kind", "company")
h.chorus(CHORUS, "capture", "Acme Group", "--kind", "company", "--force")
r = h.chorus(CHORUS, "capture", "Acme Tools", "--kind", "company")
check("v1.5.1 single common token does not gate",
r.returncode == 0 and h.ground("resources/companies/acme-tools.md") is not None,
str(r.returncode) + r.stdout + r.stderr)
# ... and --merge-into routes the capture as an update to the canonical entity.
r = h.echo(ECHO, "capture", "Bobby The Builder", "--kind", "person", "--merge-into", "bob-smith")
r = h.chorus(CHORUS, "capture", "Bobby The Builder", "--kind", "person", "--merge-into", "bob-smith")
idx4 = h.ground("_agent/index/entities.json") or ""
check("v1.5 --merge-into updates the existing entity (mention learned as alias)",
r.returncode == 0 and "Bobby The Builder" in idx4, r.stdout + r.stderr)
@@ -225,8 +225,8 @@ def main():
h.seed("_agent/sessions/2026-06-15-1200-mpm-review.md",
"---\ntype: session-log\ncreated: 2026-06-15\nupdated: 2026-06-15\n---\n"
"# MPM review\n\nDiscussed the kerfuffle about invoicing.\n")
h.echo(SWEEP, "--apply")
r = h.echo(ECHO, "recall", "kerfuffle")
h.chorus(SWEEP, "--apply")
r = h.chorus(CHORUS, "recall", "kerfuffle")
check("v1.5 recall finds a session log by body term",
"_agent/sessions/2026-06-15-1200-mpm-review.md" in r.stdout, r.stdout)
rix2 = h.ground("_agent/index/recall-index.json") or ""
@@ -240,8 +240,8 @@ def main():
h.seed("resources/concepts/new-idea.md",
"---\ntype: concept\nstatus: active\ncreated: 2026-06-20\nupdated: 2026-06-20\n"
"tags: [concept]\n---\n# New Idea\n\nzeppelin research notes\n")
h.echo(SWEEP, "--apply")
r = h.echo(ECHO, "recall", "zeppelin", "--json")
h.chorus(SWEEP, "--apply")
r = h.chorus(CHORUS, "recall", "zeppelin", "--json")
try:
env2 = json.loads(r.stdout)
except Exception:
@@ -259,7 +259,7 @@ def main():
# 15. one-tap triage: structured list, then route with an audit trail.
h.seed("inbox/captures/inbox.md",
"- 2026-06-01: prefers uv over pip\n- 2026-06-20: try the new espresso place\n")
r = h.echo(ECHO, "triage", "--list", "--json")
r = h.chorus(CHORUS, "triage", "--list", "--json")
try:
tenv = json.loads(r.stdout)
except Exception:
@@ -272,7 +272,7 @@ def main():
"line": "- 2026-06-01: prefers uv over pip", "confidence": 0.9}]
pfile = HERE / "_triage_props.json"
pfile.write_text(json.dumps(props), encoding="utf-8")
r = h.echo(ECHO, "triage", str(pfile), "--apply")
r = h.chorus(CHORUS, "triage", str(pfile), "--apply")
pfile.unlink()
routed = h.ground("_agent/memory/semantic/prefers-uv-over-pip.md")
plog = h.ground("inbox/processing-log/2026-06-21.md") or ""
@@ -286,10 +286,10 @@ def main():
h.seed("resources/companies/driftco.md",
"---\ntype: company\ncreated: 2026-06-01\nupdated: 2026-06-01\ntags: []\n---\n# DriftCo\n")
LINT = SCRIPTS / "vault_lint.py"
r = h.echo(LINT)
r = h.chorus(LINT)
check("v1.5 lint flags missing status", "driftco.md: missing status" in r.stdout, r.stdout[-800:])
check("v1.5 lint flags empty tags", "driftco.md: empty tags" in r.stdout, r.stdout[-800:])
h.echo(SWEEP, "--apply")
h.chorus(SWEEP, "--apply")
drift = h.ground("resources/companies/driftco.md") or ""
# (the naive mock records frontmatter PATCHes as <fm:...> markers instead of
# editing the YAML; the hi-fi suite proves the real fm semantics)
@@ -304,10 +304,10 @@ def main():
transcript = tdir / "t.jsonl"
turns = [json.dumps({"type": "user", "message": {"content": f"user turn {i}"}}) for i in range(6)]
transcript.write_text("\n".join(turns), encoding="utf-8")
hook_env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_STATE_DIR=str(tdir))
hook_env = dict(os.environ, CHORUS_BASE=base, CHORUS_KEY=KEY, CHORUS_STATE_DIR=str(tdir))
payload = json.dumps({"session_id": "s1", "transcript_path": str(transcript),
"stop_hook_active": False})
HOOK_STOP = SCRIPTS / "echo_hook_stop.py"
HOOK_STOP = SCRIPTS / "chorus_hook_stop.py"
r = subprocess.run([sys.executable, str(HOOK_STOP)], input=payload,
capture_output=True, text=True, env=hook_env)
try:
@@ -320,7 +320,7 @@ def main():
capture_output=True, text=True, env=hook_env)
check("v1.5 stop hook fires only once per session",
r.returncode == 0 and not r.stdout.strip(), r.stdout)
HOOK_START = SCRIPTS / "echo_hook_session_start.py"
HOOK_START = SCRIPTS / "chorus_hook_session_start.py"
r = subprocess.run([sys.executable, str(HOOK_START)],
input=json.dumps({"source": "startup"}),
capture_output=True, text=True, env=hook_env)
@@ -330,7 +330,7 @@ def main():
senv = {}
ctx = (senv.get("hookSpecificOutput") or {}).get("additionalContext", "")
check("v1.5 session-start hook injects the load output",
r.returncode == 0 and "marker" in ctx and "echo-vault.md" in ctx, r.stdout[:300])
r.returncode == 0 and "marker" in ctx and "chorus-vault.md" in ctx, r.stdout[:300])
r = subprocess.run([sys.executable, str(HOOK_START)],
input=json.dumps({"source": "resume"}),
capture_output=True, text=True, env=hook_env)
eval/test_offline_queue.py
+14 -14
View File
@@ -6,7 +6,7 @@ Phase 2 — vault UP: `flush` replays them idempotently; the writes land.
Phase 3 — vault UP: `load` caches the orientation reads.
Phase 4 — vault DOWN: `load` serves the last-known-good cache and says OFFLINE.
Drives the real echo.py against eval/mock_olrapi.py. No creds, no live vault.
Drives the real chorus.py against eval/mock_olrapi.py. No creds, no live vault.
Run: python test_offline_queue.py [--port 8840]
"""
from __future__ import annotations
@@ -21,7 +21,7 @@ import urllib.request
from pathlib import Path
HERE = Path(__file__).resolve().parent
ECHO = HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts" / "echo.py"
CHORUS = HERE.parent / "chorus-memory.plugin.src" / "skills" / "chorus-memory" / "scripts" / "chorus.py"
KEY = "test-key-not-a-real-secret"
DEAD = "http://127.0.0.1:59998" # nothing listens here -> connection refused
@@ -58,10 +58,10 @@ def main():
mock_base = f"http://127.0.0.1:{a.port}"
state = tempfile.mkdtemp()
def echo(base, *args):
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_STATE_DIR=state,
ECHO_VERIFY="0", ECHO_TODAY="2026-06-22")
return subprocess.run([sys.executable, str(ECHO), *args], capture_output=True, text=True, env=env)
def chorus(base, *args):
env = dict(os.environ, CHORUS_BASE=base, CHORUS_KEY=KEY, CHORUS_STATE_DIR=state,
CHORUS_VERIFY="0", CHORUS_TODAY="2026-06-22")
return subprocess.run([sys.executable, str(CHORUS), *args], capture_output=True, text=True, env=env)
def start_mock():
p = subprocess.Popen([sys.executable, str(HERE / "mock_olrapi.py"), "--port", str(a.port)],
@@ -80,10 +80,10 @@ def main():
srv = None
try:
# --- Phase 1: vault DOWN -> writes are queued, not lost ------------------
r = echo(DEAD, "put", "_agent/sessions/2026-06-22-1200-x.md", tmp("EVALMARK-session\n"))
r = chorus(DEAD, "put", "_agent/sessions/2026-06-22-1200-x.md", tmp("EVALMARK-session\n"))
check("offline put exits 0 (queued)", r.returncode == 0, r.stderr)
check("offline put reports queued", "queued (offline)" in r.stdout, r.stdout)
r = echo(DEAD, "append", "inbox/captures/inbox.md", "- 2026-06-22: EVALMARK-inbox")
r = chorus(DEAD, "append", "inbox/captures/inbox.md", "- 2026-06-22: EVALMARK-inbox")
check("offline append reports queued", "queued (offline)" in r.stdout, r.stdout)
outbox = Path(state) / "outbox.ndjson"
check("two writes are persisted to the outbox",
@@ -91,27 +91,27 @@ def main():
# --- Phase 2: vault UP -> flush replays them -----------------------------
srv = start_mock()
r = echo(mock_base, "flush")
r = chorus(mock_base, "flush")
check("flush replays the queue", "flushed 2" in r.stdout, r.stdout + r.stderr)
check("queued PUT landed in the vault", (ground("_agent/sessions/2026-06-22-1200-x.md") or "").find("EVALMARK-session") >= 0)
check("queued APPEND landed in the vault", "EVALMARK-inbox" in (ground("inbox/captures/inbox.md") or ""))
check("outbox is empty after a clean flush", not outbox.exists() or not outbox.read_text(encoding="utf-8").strip())
# idempotent replay: a second flush of the same content must not duplicate.
echo(DEAD, "append", "inbox/captures/inbox.md", "- 2026-06-22: EVALMARK-inbox") # re-queue same line
echo(mock_base, "flush")
chorus(DEAD, "append", "inbox/captures/inbox.md", "- 2026-06-22: EVALMARK-inbox") # re-queue same line
chorus(mock_base, "flush")
inbox = ground("inbox/captures/inbox.md") or ""
check("replay is idempotent (no duplicate line)", inbox.count("EVALMARK-inbox") == 1, inbox)
# --- Phase 3: vault UP -> load caches the orientation reads ---------------
http("PUT", f"{mock_base}/vault/_agent/echo-vault.md", "schema_version: 4\nEVALMARK-marker\n")
r = echo(mock_base, "load")
http("PUT", f"{mock_base}/vault/_agent/chorus-vault.md", "schema_version: 4\nEVALMARK-marker\n")
r = chorus(mock_base, "load")
check("online load shows marker", "EVALMARK-marker" in r.stdout, r.stdout)
check("load wrote a cache dir", (Path(state) / "cache").exists())
# --- Phase 4: vault DOWN -> load serves the cache ------------------------
srv.terminate(); srv.wait(timeout=5); srv = None
r = echo(DEAD, "load")
r = chorus(DEAD, "load")
check("offline load flags OFFLINE", "OFFLINE" in r.stdout, r.stdout)
check("offline load serves cached marker", "EVALMARK-marker" in r.stdout, r.stdout)
eval/test_patch_semantics.py
+13 -13
View File
@@ -3,7 +3,7 @@
These are exactly the behaviors the shipped naive mock cannot reproduce (it appends
everything at EOF), so without this the section-replace / section-append / missing-
heading / frontmatter-replace paths are untested. Drives the real echo.py against
heading / frontmatter-replace paths are untested. Drives the real chorus.py against
mock_olrapi_hifi.py.
Run: python test_patch_semantics.py [--port 8820]
@@ -20,7 +20,7 @@ import urllib.request
from pathlib import Path
HERE = Path(__file__).resolve().parent
ECHO = HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts" / "echo.py"
CHORUS = HERE.parent / "chorus-memory.plugin.src" / "skills" / "chorus-memory" / "scripts" / "chorus.py"
KEY = "test-key-not-a-real-secret"
failures = []
@@ -58,9 +58,9 @@ def main():
srv = subprocess.Popen([sys.executable, str(HERE / "mock_olrapi_hifi.py"), "--port", str(a.port)],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
def echo(*args):
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="0")
return subprocess.run([sys.executable, str(ECHO), *args], capture_output=True, text=True, env=env)
def chorus(*args):
env = dict(os.environ, CHORUS_BASE=base, CHORUS_KEY=KEY, CHORUS_VERIFY="0")
return subprocess.run([sys.executable, str(CHORUS), *args], capture_output=True, text=True, env=env)
def ground(path):
_, body = http("GET", f"{base}/__debug__?path={path}")
@@ -80,30 +80,30 @@ def main():
# replace: Scope body becomes new; Other untouched.
http("PUT", f"{base}/vault/{doc}", seed)
echo("patch", doc, "replace", "heading", "Doc::Scope", tmp("new scope"))
chorus("patch", doc, "replace", "heading", "Doc::Scope", tmp("new scope"))
g = ground(doc) or ""
check("heading replace swaps section body", "new scope" in g and "old scope" not in g, g)
check("heading replace leaves sibling section intact", "## Other\nkeep me" in g, g)
# append: lands INSIDE the section (before '## Other'), not at EOF.
echo("patch", doc, "append", "heading", "Doc::Scope", tmp("appended line"))
chorus("patch", doc, "append", "heading", "Doc::Scope", tmp("appended line"))
g = ground(doc) or ""
check("heading append inserts within the section (not EOF)",
"appended line" in g and g.index("appended line") < g.index("## Other"), g)
# prepend: lands at the TOP of the section body.
echo("patch", doc, "prepend", "heading", "Doc::Scope", tmp("first line"))
chorus("patch", doc, "prepend", "heading", "Doc::Scope", tmp("first line"))
g = ground(doc) or ""
check("heading prepend inserts at section top",
"first line" in g and g.index("first line") < g.index("new scope"), g)
# missing heading -> 400 -> echo.py exits non-zero (the silent-loss guard).
r = echo("patch", doc, "append", "heading", "Doc::Nope", tmp("lost?"))
# missing heading -> 400 -> chorus.py exits non-zero (the silent-loss guard).
r = chorus("patch", doc, "append", "heading", "Doc::Nope", tmp("lost?"))
check("missing heading fails loud (non-zero exit)", r.returncode != 0, r.stderr)
check("missing heading does not write", "lost?" not in (ground(doc) or ""))
# frontmatter replace on an existing field updates it.
echo("fm", doc, "updated", "2026-06-22")
chorus("fm", doc, "updated", "2026-06-22")
g = ground(doc) or ""
check("frontmatter replace updates existing field", "updated: 2026-06-22" in g, g)
@@ -112,7 +112,7 @@ def main():
# every other line is preserved. (The old contract failed loud on exactly the
# notes that needed repair.)
before = ground(doc) or ""
r = echo("fm", doc, "nonexistent_field", "x")
r = chorus("fm", doc, "nonexistent_field", "x")
check("fm creates a missing field (create-or-replace)", r.returncode == 0,
r.stderr or r.stdout)
g = ground(doc) or ""
@@ -121,7 +121,7 @@ def main():
all(line in g for line in before.splitlines() if line.strip()), g)
# raw `patch replace frontmatter` (no fallback) still fails loud on a missing key.
r = echo("patch", doc, "replace", "frontmatter", "still_missing", tmp('"x"'))
r = chorus("patch", doc, "replace", "frontmatter", "still_missing", tmp('"x"'))
check("raw frontmatter PATCH of a missing field fails loud", r.returncode != 0, r.stderr)
print(f"\n{len(failures)} failure(s)" if failures else "\nall PATCH-semantics tests passed")
eval/test_reflect.py
+10 -10
View File
@@ -3,7 +3,7 @@
A dry-run previews and writes NOTHING; --apply routes each proposal through capture
(creating notes, the inbox line, and skipping low-confidence items). Drives the real
echo.py against eval/mock_olrapi.py. No creds, no live vault.
chorus.py against eval/mock_olrapi.py. No creds, no live vault.
Run: python test_reflect.py [--port 8850]
"""
@@ -20,7 +20,7 @@ import urllib.request
from pathlib import Path
HERE = Path(__file__).resolve().parent
ECHO = HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts" / "echo.py"
CHORUS = HERE.parent / "chorus-memory.plugin.src" / "skills" / "chorus-memory" / "scripts" / "chorus.py"
KEY = "test-key-not-a-real-secret"
failures = []
@@ -50,9 +50,9 @@ def main():
except Exception as e: # noqa: BLE001
return getattr(e, "code", 0), ""
def echo(*args, stdin=None):
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1", ECHO_TODAY="2026-06-22")
return subprocess.run([sys.executable, str(ECHO), *args], input=stdin,
def chorus(*args, stdin=None):
env = dict(os.environ, CHORUS_BASE=base, CHORUS_KEY=KEY, CHORUS_VERIFY="1", CHORUS_TODAY="2026-06-22")
return subprocess.run([sys.executable, str(CHORUS), *args], input=stdin,
capture_output=True, text=True, env=env)
def ground(path):
@@ -65,10 +65,10 @@ def main():
urllib.request.urlopen(f"{base}/__debug__reset", data=b"", timeout=1); break
except Exception:
time.sleep(0.1)
http("PUT", f"{base}/vault/_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n")
http("PUT", f"{base}/vault/_agent/chorus-vault.md", "---\nschema_version: 4\n---\n# marker\n")
proposals = [
{"title": "Acme Corp", "kind": "company", "body": "A vendor ECHO integrates with.", "confidence": 0.9},
{"title": "Acme Corp", "kind": "company", "body": "A vendor CHORUS integrates with.", "confidence": 0.9},
{"title": "Use uv not pip", "kind": "semantic", "body": "Jason standardizes on uv.", "confidence": 0.95},
{"title": "half-formed idea", "inbox": True, "confidence": 0.9},
{"title": "Maybe relevant", "kind": "concept", "confidence": 0.2}, # below floor -> skipped
@@ -77,13 +77,13 @@ def main():
json.dump(proposals, pfile); pfile.close()
# dry-run: previews, writes nothing
r = echo("reflect", pfile.name)
r = chorus("reflect", pfile.name)
check("dry-run previews", "dry-run" in r.stdout and "Acme Corp" in r.stdout, r.stdout)
check("dry-run drops the low-confidence proposal", "Maybe relevant" not in r.stdout, r.stdout)
check("dry-run writes nothing", ground("resources/companies/acme-corp.md") is None)
# --apply: routes each through capture
r = echo("reflect", pfile.name, "--apply")
r = chorus("reflect", pfile.name, "--apply")
check("apply reports applied count", "applied" in r.stdout, r.stdout + r.stderr)
check("apply creates the company note",
(ground("resources/companies/acme-corp.md") or "").find("type: company") >= 0)
@@ -94,7 +94,7 @@ def main():
ground("resources/concepts/maybe-relevant.md") is None)
# stdin path also works (proposals piped, not a file)
r = echo("reflect", "-", stdin=json.dumps([{"title": "Piped Co", "kind": "company", "confidence": 0.9}]))
r = chorus("reflect", "-", stdin=json.dumps([{"title": "Piped Co", "kind": "company", "confidence": 0.9}]))
check("reflect reads proposals from stdin", "Piped Co" in r.stdout, r.stdout + r.stderr)
print(f"\n{len(failures)} failure(s)" if failures else "\nall reflect tests passed")