From 7f76e3521d8c290a9883d537a4e1a02ef5063cf9 Mon Sep 17 00:00:00 2001 From: Jason Stedwell Date: Tue, 21 Jul 2026 13:19:45 -0500 Subject: [PATCH] =?UTF-8?q?Add=20CHORUS-PLAN.md=20=E2=80=94=20ECHO=20to=20?= =?UTF-8?q?group-based=20memory=20conversion=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full review of the ECHO v1.5.1 architecture plus the phased plan to convert the plugin to multi-member (group) memory: identity model, schema-5 member namespacing, concurrency posture, rename map, and the five settled design decisions (shared rollups, group-visible prefs, shared-key first, /chorus-* commands, per-member + group inboxes). Deployment endpoint: chorusapi.mpm.to. Co-Authored-By: Claude Fable 5 --- CHORUS-PLAN.md | 248 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 CHORUS-PLAN.md diff --git a/CHORUS-PLAN.md b/CHORUS-PLAN.md new file mode 100644 index 0000000..ed4b226 --- /dev/null +++ b/CHORUS-PLAN.md @@ -0,0 +1,248 @@ +# CHORUS — Group-Based Memory: Conversion Plan + +**Fork of:** `jason/echo` (ECHO v1.5.1, schema 4) → `jason/chorus` +**Goal:** convert the single-operator ECHO memory plugin into CHORUS — one shared +Obsidian vault serving a **group of members**, with per-member identity, +attribution, and isolation of the churn-prone per-person state, while keeping the +shared knowledge graph (projects, areas, resources, decisions) communal. + +**Naming metaphor:** ECHO is one voice reflected back; CHORUS is many voices in +one performance. + +--- + +## 1. What the review found + +ECHO is ~4,800 LOC of pure-stdlib Python (18 scripts) + a large doc surface +(SKILL.md ~456 lines, 6 references, 8 commands, 15 scaffold seeds). The +architecture is already favorable for this conversion: + +- **Identity is barely wired in.** `echo.py` loads `OWNER` from config but never + actually consumes it — attribution today is purely doc-driven ("write in third + person about Jason"). Adding a `member` identity breaks nothing. +- **Routing is data-driven.** `routing.json` is the single source of truth for + "what may be written where"; `vault_lint.py` and `check_routing.py` enforce it. + Per-member paths are new route rows, not new code paths. +- **The vault is self-describing.** Bootstrap/migrate/sweep + `schema_version` + in the marker give us a clean upgrade lever (schema 4 → 5). + +The hard part is **not** identity plumbing — it's the five shared single-writer +artifacts that make no sense (or clobber) with multiple concurrent users: + +| Artifact | Today | Problem in a group | +|---|---|---| +| `_agent/memory/semantic/operator-preferences.md` | One profile | Whose preferences? | +| `_agent/context/current-context.md` | One active scope for the vault | Members work on different things simultaneously | +| `_agent/heartbeat/last-session.md` | One "last session" pointer (PUT overwrite) | Race: lost updates; meaningless shared | +| `inbox/captures/inbox.md` | One shared inbox (POST append) | Append races; triage can't tell whose capture | +| `journal/daily/YYYY-MM-DD.md` `## Agent Log` | One shared heading | Duplicate-heading races; unattributed lines | + +Also single-user by design: session filenames carry no author +(`YYYY-MM-DD-HHMM-.md` — two members can collide on the same minute), +frontmatter has no `author:` field anywhere, the offline queue is per-machine +with no member attribution, and the advisory lock owner is a session id, not a +person. Concurrency is advisory-only — `vault_lock()` warns and **proceeds** if +it can't acquire (echo_concurrency.py:78–81), which is acceptable for one person +on two machines but not for N people. + +--- + +## 2. Target design + +### 2.1 Identity model + +- Config gains a `member` field: + `{ "group": "...", "member": "jason", "endpoint": "https://chorusapi.mpm.to", "key": "..." }`. + **Deployment endpoint: `https://chorusapi.mpm.to`** — the CHORUS vault gets + its own Obsidian + Local REST API instance exposed there (separate from + ECHO's `echoapi.alwisp.com`; the personal ECHO vault is untouched). The + endpoint stays machine-local/baked config, never in source, same as ECHO. + `owner` → `group` (with `owner` accepted as a legacy alias). Resolution order + unchanged: baked > env (`CHORUS_MEMBER` etc.) > config file. Baked per-user + builds (`build.py --bake-key --label alice`) already exist and become the + primary **onboarding** mechanism: one artifact per member, member id baked in. +- Every agent-written note gets `author: ` stamped in frontmatter + (`echo_ops._build_note`), added to the canonical frontmatter block, to + `KIND_REQUIRED_FM` for agent-written kinds, linted by `vault_lint`, and + backfilled by `sweep --apply` (existing notes → the migrating member). +- Lock owner and offline-queue entries carry the member id + (`auto_owner()` → `f"{member}-{client}-{pid}"`), so contention and replay are + attributable. + +**Auth reality check:** the Obsidian Local REST API has **one bearer key per +vault**. Phase 1–4 ship with a shared key and honor-system attribution (fine for +a trusted team). Real per-member credentials/revocation would need a thin auth +proxy in front of `chorusapi.mpm.to` that maps per-member tokens → the vault +key and can inject/verify the member id. That's deliberately **out of scope** +here but the config shape (`member` + per-member baked artifacts) is designed so +the proxy slots in later without another migration. + +### 2.2 Vault layout changes (schema 5) + +Per-member state moves under a member namespace; shared knowledge stays where it +is. + +``` +_agent/ +├── chorus-vault.md ← renamed marker (schema_version: 5, managed_by: chorus-memory-plugin) +├── members/ +│ └── / +│ ├── preferences.md ← was operator-preferences.md (one per member) +│ ├── current-context.md ← per-member scope + Scope History +│ ├── heartbeat.md ← per-member last-session pointer +│ └── inbox.md ← per-member quick-capture inbox +├── memory/ +│ └── semantic/ +│ └── group-profile.md ← NEW: shared facts about the group itself +├── sessions/ +│ └── /YYYY-MM-DD-HHMM-.md ← member-namespaced session logs +└── (context/, memory/, health/, index/, locks/, templates/, skills/ unchanged — shared) +``` + +- **Shared and unchanged:** `projects/`, `areas/`, `resources/`, `decisions/`, + `journal/`, semantic/episodic/working memory (now with `author:`), the entity + index, recall index, health notes. +- **Daily Agent Log stays one shared heading** (it's the group's timeline) but + every line is member-prefixed: `- HH:MM [alice] did X`. `ensure_daily_log` + keeps its GET→POST-heading→PATCH flow; idempotent whole-line append plus the + member prefix makes interleaving safe and attributed. (Per-member daily notes + were considered and rejected — the whole point of a group vault is one shared + timeline.) +- **Inbox:** per-member capture files (`_agent/members//inbox.md`) kill the + append race and make triage member-aware, **plus** one shared + `inbox/captures/group.md` for unowned "someone should look at this" items + (member-prefixed lines; routed by whoever triages first). Triage covers both: + accepted items route to shared homes; preference-type items route to *that + member's* `preferences.md`. Processing-log lines carry the member id. +- `resources/people/.md` continues to exist for every member — the + public "who is this person" note, distinct from their private-ish + `preferences.md` (how the agent should behave *for them*). + +### 2.3 Cold-start load (per member) + +`chorus.py load` reads: marker → **group-profile** → **my** preferences → **my** +current-context → **my** heartbeat/last-session (fallback: my sessions dir) → +today's shared daily note → **my** inbox depth → **group inbox** depth. Optional `--all-members` flag +for a group-wide orientation (e.g., "what has the team been doing"). Reconcile +(inbox aging, scope drift) runs against the member's own state only. + +### 2.4 Concurrency posture + +Namespacing removes ~all of the contention (each member's churn-prone files are +theirs alone). What remains shared-write: + +- **Entity + recall indexes** — keep `vault_lock` + `atomic_index_update`, but + flip the lock to **required** (`required=True`) on index writes: block/retry + briefly instead of warn-and-proceed. TTL reclaim unchanged. +- **Daily note / shared notes** — idempotent member-prefixed appends; PATCH to + headings is last-writer-wins per line, which is acceptable for log lines. +- **Duplicate gate** — unchanged and global (dedup across the group is the + *feature*: two members shouldn't create parallel notes for the same client). + Gate messages name the `author` of the existing candidate. + +### 2.5 Recall / reflection + +- Corpus unchanged (shared graph is the point). `recall` gains an optional + `--author ` filter and surfaces `author:` in hits. +- `reflect` proposals route preference items to the current member's + `preferences.md`; everything else routes to shared homes with `author:` + stamped. The Stop-hook nudge state stays per-machine (already is). + +--- + +## 3. Renaming map (echo → chorus) + +| Category | From | To | +|---|---|---| +| Plugin | `echo-memory` (plugin.json, skill dir, SKILL.md name) | `chorus-memory` | +| Commands | `/echo-{load,save,recall,reflect,triage,sweep,health,doctor}` | `/chorus-*` | +| Scripts | `echo*.py` (16 modules + imports everywhere) | `chorus*.py` | +| Exception | `EchoError` | `ChorusError` | +| Env vars | `ECHO_OWNER/BASE/KEY/CONFIG/TODAY/VERIFY/LOCK_TTL/TIMEOUT/WORKERS/STATE_DIR/CLIENT/DUP_GATE/REFLECT_MIN_TURNS/FRESH_HALF_LIFE` | `CHORUS_*` (+ new `CHORUS_MEMBER`); `ECHO_*` read as fallback aliases in `chorus_config` for one major version | +| Config file | `~/.claude/echo-memory/config.json` | `~/.claude/chorus-memory/config.json` (import path: `config import` accepts an old echo config) | +| State dir | `~/.echo-memory/` | `~/.chorus-memory/` | +| Marker | `_agent/echo-vault.md` (hardcoded in 8 files + routing.json) | `_agent/chorus-vault.md`; probe checks new name **then** old (adoption path) | +| Hooks | `hooks.json` paths ×2 incl. CoWork fallback glob | updated paths | +| Docs/branding | README, SKILL.md, references, scaffold seeds, icons | rewrite pass (cosmetic but large) | + +`operator-preferences.md` is renamed as part of the **layout** change +(§2.2), not the branding pass. + +--- + +## 4. Phased plan + +### Phase 0 — Baseline (½ day) +Repo hygiene on the fork: strip stale versioned `.plugin` artifacts from the +tree (move to Gitea releases per ROADMAP-2.0), confirm `eval/` suites green +as-is, tag the fork point. + +### Phase 1 — Mechanical rename (1–2 days) +Everything in §3, with back-compat shims (env aliases, marker dual-probe, +config import from echo paths). All existing tests pass under the new names. +No behavior change. **Deliverable:** installable `chorus-memory` plugin that is +ECHO-with-a-new-name. + +### Phase 2 — Identity plumbing (1–2 days) +`member` in config (+ `CHORUS_MEMBER`, baked-tier support, `config set --member`), +`MEMBER` global in `chorus.py`, `author:` stamping in `_build_note`, canonical +frontmatter + `KIND_REQUIRED_FM` + lint + sweep backfill, member-aware lock +owner and queue entries, `doctor` reports resolved member. Vault layout +untouched. **Deliverable:** every new write is attributed; single-user behavior +otherwise identical. + +### Phase 3 — Per-member namespacing (3–5 days — the core) +Schema 5. New `LEAVES` + seeds (`members//…`, `group-profile.md`), +member-parameterized bootstrap (bootstraps *your* member subtree on first load +if the vault exists but your namespace doesn't — new members self-onboard), +`cmd_load`/`cmd_scope`/heartbeat/inbox/triage/reflect retargeted to member +paths, session logs under `sessions//`, member-prefixed Agent Log +lines, `routing.json` route rows (with `` pattern segment) + retired +old paths, lint updates, `migrate.py` 4→5 (moves Jason's existing +operator-preferences/current-context/heartbeat/inbox/sessions into +`members/jason/…`, stamps `author: jason` vault-wide via sweep). Docs: +vault-layout.md, routing-map.md, bootstrap.md rewritten. +**Deliverable:** a real multi-member vault; the live ECHO vault migrates in +place. + +### Phase 4 — Shared-write hardening (1–2 days) +Required lock on index writes with bounded retry/backoff, gate messages carry +author, `recall --author`, `load --all-members`, concurrency docs +(operating-contract.md) rewritten for N writers. + +### Phase 5 — Docs, onboarding & eval (2–3 days) +SKILL.md full rewrite (operator → member voice; drop the "single owner" +safety rule, replace with group rules: don't edit another member's +`preferences.md` except via their session; shared notes are fair game), +8 command docs, README. Onboarding runbook: per-member baked builds. Extend +`eval/`: two-member mock scenarios (concurrent capture, interleaved daily-log +appends, index race, cross-member dedup gate, member-scoped load) + rerun the +retrieval/durability metrics. New icon. + +**Total: roughly 9–15 working days**, shippable at every phase boundary +(1 = rename, 2 = attribution, 3 = groups). + +--- + +## 5. Decisions (settled with Jason, 2026-07-21) + +1. **Journal rollups — shared.** One group rollup per period + (`journal/weekly/YYYY-Www.md` etc.) with member-attributed bullets inside. + No per-member rollup trees. +2. **Preference privacy — group-visible, documented.** Member `preferences.md` + files live in the shared vault and are readable by every member (one vault, + one key — no read ACLs possible). SKILL.md documents this explicitly; + anything a member considers private belongs in their own personal ECHO + vault, not CHORUS. +3. **Auth proxy — out of scope; shared key first.** Phases 1–5 ship with the + shared vault key and honor-system attribution. The per-member token proxy + (namespace write ACL, server-side audit, per-member revocation) is a later + standalone project; the config shape (`member` + per-member baked + artifacts) already accommodates it with zero plugin changes. +4. **Commands — `/chorus-*`.** Full-prefix names + (`/chorus-load`, `/chorus-save`, …), consistent with the ECHO pattern and + collision-free if both plugins are installed. +5. **Inboxes — per-member + shared group inbox.** Each member gets + `_agent/members//inbox.md`; a shared `inbox/captures/group.md` holds + unowned items (member-prefixed lines, idempotent appends), routed by + whoever triages first. Both are probed at load and covered by triage.