forked from jason/echo
Phase 1: mechanical echo→chorus rename with back-compat shims
Identity rename, no behavior change (CHORUS-PLAN.md Phase 1): - Plugin echo-memory → chorus-memory: manifest (v2.0.0-alpha.1), skill dir, 16 scripts (chorus.py, chorus_config.py, …), EchoError → ChorusError, /chorus-* commands, hook paths, docs, scaffold seeds, eval harness, build.py. Docs endpoint → chorusapi.mpm.to. - Env ECHO_* → CHORUS_*; config → ~/.claude/chorus-memory/config.json; state dir → ~/.chorus-memory/; marker → _agent/chorus-vault.md. Back-compat shims (one major version): - chorus_config aliases legacy ECHO_* env at import; reads a legacy echo-memory config.json when no CHORUS config exists (writes never land there); doctor/config report the legacy source. - State dir honors an existing ~/.echo-memory when the new dir is absent (offline queue not stranded). - Marker dual-probe in load/doctor/bootstrap/lint/sweep/migrate: a pre-fork _agent/echo-vault.md counts as bootstrapped; bootstrap repair won't write a second marker; routing gains agent-marker-legacy (GET). Verified: 25/25 unit tests, scaffold + routing-sync checks, 4 mock e2e suites, run_eval metrics unchanged, +6 shim smoke tests green. Rebuilt chorus-memory.plugin (79 entries). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"name": "chorus-memory",
|
||||
"version": "2.0.0-alpha.1",
|
||||
"description": "Group-based persistent memory via the shared CHORUS Obsidian vault over the Obsidian Local REST API. Cross-platform Python client: one-call capture/resolve/recall/link/triage over an entity index, hybrid BM25 + graph recall spanning entities + sessions/journal (recency/status-aware), a pre-write duplicate gate, complete-frontmatter capture, session hooks that self-fire load/reflect, offline write-ahead queue, lock-guarded concurrency, linter-enforced routing, and /chorus-* commands.",
|
||||
"author": {
|
||||
"name": "Jason Stedwell"
|
||||
},
|
||||
"license": "UNLICENSED",
|
||||
"keywords": [
|
||||
"memory",
|
||||
"obsidian",
|
||||
"notes",
|
||||
"persistence",
|
||||
"chorus"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
# chorus-memory
|
||||
|
||||
Persistent memory for Claude via the **CHORUS** Obsidian vault, using the [Obsidian Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api).
|
||||
|
||||
Reads and writes notes across Claude/CoWork sessions through direct REST calls, routed through a bundled validated client (`scripts/chorus.py`) that status-checks every write. The toolchain is **pure Python**, so it runs identically on Windows, macOS, and Linux (only a Python 3 interpreter is needed — no bash). The plugin is **user-agnostic**: the vault owner, endpoint, and API key are not shipped in it — each machine supplies them through a local config file (see **Configuration**).
|
||||
|
||||
Architected by **Jason Stedwell**.
|
||||
|
||||
**The plugin is the single source of truth.** All control logic — bootstrap/repair, operating contract, taxonomy, frontmatter conventions, and the canonical note templates — ships inside this plugin under `skills/chorus-memory/`. The vault itself holds **data only**: there are no `CLAUDE.md` / `BOOTSTRAP.md` / `STRUCTURE.md` / `index.md` control docs in it. This makes CHORUS self-bootstrapping (point it at an empty Obsidian vault and it stands up the full structure), easy to update (update the plugin, not the vault), and portable to any other vault.
|
||||
|
||||
## What it does
|
||||
|
||||
- Loads operator preferences, current context, and relevant project notes at the start of substantive conversations
|
||||
- **One-call `capture`** routes a memory to its canonical home, stamps **complete** frontmatter (kind-default `status`, kind-seeded `tags`), indexes it, auto-links any entity it mentions, and logs it — driven by a machine-maintained **entity index** so routing is an alias-aware lookup, not a fuzzy search. Updates keep the **whole body**; a title that strongly resembles an existing entity **stops at a duplicate gate** (exit 76 — `--merge-into <slug>` / `--force`) instead of spawning a near-duplicate
|
||||
- **`recall`** returns a topic's matching notes *and* their one-hop linked neighbourhood (Related links + `source_notes`) — the corpus spans the entity graph **plus session logs and journal notes** (down-weighted), ranked by BM25 × freshness × status so live notes outrank stale ones; **`resolve`** maps any mention to its canonical path; **`link`** adds reciprocal cross-links; **`triage`** routes aging inbox captures with an automatic processing-log audit trail
|
||||
- Logs working sessions so future conversations can pick up where they left off
|
||||
- **Bootstraps an empty vault from the bundled `scaffold/`** (folders, templates, anchor seeds, marker), repairs/migrates an existing one via `scripts/bootstrap.py` / `scripts/migrate.py`, and brings an upgraded vault up to spec (index + cross-links) via `scripts/sweep.py` — see `skills/chorus-memory/references/bootstrap.md`
|
||||
- Exposes `/chorus-load`, `/chorus-save`, `/chorus-recall`, `/chorus-triage`, `/chorus-health`, `/chorus-sweep`, `/chorus-reflect`, `/chorus-doctor` slash commands as explicit entry points, and coordinates concurrent Claude/CoWork sessions via a cooperative advisory lock
|
||||
- **Ships session hooks** (`hooks/hooks.json`): SessionStart auto-runs the cold-start load into context, and Stop nudges `/chorus-reflect` once per substantive session — the load/reflect discipline fires deterministically instead of depending on the model remembering (reflection still previews before writing)
|
||||
|
||||
## Configuration
|
||||
|
||||
The plugin ships **no** owner, endpoint, or key. On each machine it installs on, provide a machine-local JSON config:
|
||||
|
||||
```
|
||||
~/.claude/chorus-memory/config.json (honors $CLAUDE_CONFIG_DIR; file path overridable with $CHORUS_CONFIG)
|
||||
{
|
||||
"owner": "Full Name — how the agent refers to the vault owner (third person)",
|
||||
"endpoint": "https://your-obsidian-rest-endpoint.example.com",
|
||||
"key": "<obsidian-local-rest-api-bearer-token>"
|
||||
}
|
||||
```
|
||||
|
||||
Set it up on a fresh install by copying the bundled template and filling it in, or via the client:
|
||||
|
||||
```bash
|
||||
python3 skills/chorus-memory/scripts/chorus.py config init # writes the template if absent → edit it
|
||||
python3 skills/chorus-memory/scripts/chorus.py config set --owner "…" --endpoint "https://…" --key "…"
|
||||
python3 skills/chorus-memory/scripts/chorus.py config # show resolved config (key redacted) + source
|
||||
```
|
||||
|
||||
Per field, an environment variable overrides the file: `CHORUS_OWNER`, `CHORUS_BASE` (endpoint), `CHORUS_KEY`. `config init` writes the template (shown above) when the file is absent; a copy also ships at the repo root (`chorus-memory.config.template.json`). The config is **never committed** and **never written into a vault note**; the bearer key stays out of the plugin tree entirely.
|
||||
|
||||
## Vault layout (root-addressed)
|
||||
|
||||
```
|
||||
/vault/
|
||||
├── README.md ← thin human signpost (not read for routing)
|
||||
├── inbox/ (captures, imports, processing-log)
|
||||
├── journal/ (daily, weekly, monthly, quarterly, annual, templates) — one time-series stream; rollups live here, not in a separate reviews/ tree
|
||||
├── projects/ (active, incubating, on-hold, archived)
|
||||
├── areas/ (business, personal, learning, systems)
|
||||
├── resources/ (companies, concepts, references, people, meetings)
|
||||
├── decisions/ (by-date)
|
||||
└── _agent/
|
||||
├── chorus-vault.md ← bootstrap marker (schema_version + date)
|
||||
├── context/ ← task-scoped context bundles
|
||||
├── memory/ ← working / episodic / semantic
|
||||
├── sessions/ ← YYYY-MM-DD-HHMM-<slug>.md
|
||||
├── health/ ← YYYY-MM-vault-health.md (monthly self-maintenance audit)
|
||||
├── templates/ ← canonical note templates (seeded from the plugin's scaffold/)
|
||||
├── skills/ ← active / archived
|
||||
├── heartbeat/ ← last-session.md orientation pointer (read at load, written at session end)
|
||||
├── index/ ← entities.json — machine-maintained slug→{path,kind,aliases} registry
|
||||
└── locks/ ← vault.lock — cooperative advisory multi-writer lock
|
||||
```
|
||||
|
||||
Control logic and the master scaffold live in the plugin, not the vault:
|
||||
|
||||
```
|
||||
commands/ ← slash commands: chorus-load, chorus-save, chorus-recall, chorus-triage, chorus-health, chorus-sweep, chorus-reflect, chorus-doctor
|
||||
skills/chorus-memory/
|
||||
├── SKILL.md ← operating procedure (authoritative)
|
||||
├── references/
|
||||
│ ├── operating-contract.md ← durable principles + safety + concurrency
|
||||
│ ├── bootstrap.md ← bootstrap / repair / migrate an empty vault
|
||||
│ ├── vault-layout.md ← canonical layout + frontmatter
|
||||
│ ├── routing-map.md ← complete endpoint→logic map (human authority)
|
||||
│ ├── api-reference.md ← REST endpoint patterns + routing map
|
||||
│ └── session-log-template.md
|
||||
├── scripts/ ← pure Python; run with python3 (Windows: python / py -3)
|
||||
│ ├── chorus.py ← validated client + high-level CLI (capture/resolve/recall/link/load/scope/lock/config)
|
||||
│ ├── chorus_config.py ← resolves owner/endpoint/key from the machine-local config (env-overridable)
|
||||
│ ├── chorus_index.py ← entity index (registry, resolve, name→path map)
|
||||
│ ├── chorus_links.py ← cross-link primitives (Related parsing, bidirectional linking)
|
||||
│ ├── chorus_ops.py ← high-level ops (capture, recall, resolve, link, agent-log)
|
||||
│ ├── routing.json ← canonical machine-readable route manifest (linter enforces it)
|
||||
│ ├── vault_lint.py ← read-only invariant + graph-health checker (Vault Health)
|
||||
│ ├── check_routing.py ← verifies routing docs stay in sync with routing.json (offline)
|
||||
│ ├── test_chorus_client.py ← offline regression tests + the routing-sync guard
|
||||
│ ├── bootstrap.py ← deterministic, idempotent vault setup/repair
|
||||
│ ├── migrate.py ← deterministic schema migration (dry-run by default)
|
||||
│ └── sweep.py ← bring an upgraded vault up to spec (build index + symmetrize links)
|
||||
└── scaffold/ ← verbatim files the bootstrap writes into the vault
|
||||
├── README.vault.md chorus-vault.md
|
||||
├── templates/ (8 note templates)
|
||||
└── anchors/ (operator-preferences, current-context, inbox seeds)
|
||||
```
|
||||
|
||||
## Skills & commands
|
||||
|
||||
| Skill | Triggers |
|
||||
|-------|----------|
|
||||
| `chorus-memory` | "remember that", "save to memory", "what do you know about me", "load my profile", "check my notes", "log this decision", "add to my inbox" — and proactively at the start of substantive conversations |
|
||||
|
||||
| Command | Does |
|
||||
|---------|------|
|
||||
| `/chorus-load` | Cold-start context read (profile, scope, latest session, today, inbox) |
|
||||
| `/chorus-save <text>` | Route + persist via one-call `capture` (index-resolved, auto-linked) |
|
||||
| `/chorus-recall <query>` | Recall a topic's matching notes plus their linked neighbourhood |
|
||||
| `/chorus-triage` | Drain aging inbox captures to their homes, logging each move |
|
||||
| `/chorus-health` | Run `vault_lint.py` and summarize invariant + graph-health violations |
|
||||
| `/chorus-sweep` | Bring the vault up to current spec (build index + symmetrize links) |
|
||||
| `/chorus-reflect` | Extract durable items from the session, preview, and apply on approval |
|
||||
| `/chorus-doctor` | Readiness check: config, endpoint reachability, auth, bootstrap/schema |
|
||||
|
||||
## Requirements
|
||||
|
||||
- **Python 3** available in the session environment (the scripts use only the standard library; invoke as `python3`, or `python` / `py -3` on Windows)
|
||||
- Obsidian running on the backend with the [Local REST API plugin](https://github.com/coddingtonbear/obsidian-local-rest-api) enabled
|
||||
- HTTPS access from the Claude/CoWork session environment to the endpoint configured in `~/.claude/chorus-memory/config.json`
|
||||
- A machine-local config (`~/.claude/chorus-memory/config.json`) supplying the vault owner, endpoint, and API key — see **Configuration**
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
description: Check CHORUS readiness — Python, vault reachability, auth, bootstrap/schema, and key source
|
||||
---
|
||||
|
||||
Use the **chorus-memory** skill to run a one-call readiness check before relying on memory.
|
||||
|
||||
```bash
|
||||
CHORUS="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts/chorus.py"
|
||||
[ -f "$CHORUS" ] || CHORUS=$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/chorus.py 2>/dev/null | head -1) # CoWork sandbox fallback
|
||||
python3 "$CHORUS" doctor
|
||||
# Windows: use `python` or `py -3` if `python3` is not on PATH.
|
||||
```
|
||||
|
||||
It prints green/red for: Python version, vault reachability, auth accepted, vault
|
||||
bootstrapped (+ `schema_version`), and the **config source** for owner/endpoint/key
|
||||
(per-field: env override `CHORUS_OWNER`/`CHORUS_BASE`/`CHORUS_KEY`, then the machine-local
|
||||
`~/.claude/chorus-memory/config.json` — or `baked-in (plugin)` on a per-user baked build,
|
||||
which is authoritative and reported as such). Exits non-zero if anything is red — if the config is
|
||||
missing (or still the placeholder template), ask the operator for their chorus-memory config file and install it with `chorus.py config import <path>` (or `config set --owner … --endpoint … --key …`). For the full vault-invariant lint, run
|
||||
`/chorus-health`.
|
||||
@@ -0,0 +1,17 @@
|
||||
---
|
||||
description: Run the CHORUS vault-health linter and summarize any invariant violations
|
||||
allowed-tools: Bash(python3 *vault_lint.py*), Bash(python *vault_lint.py*), Bash(ls /sessions/*), Bash(dirname *)
|
||||
---
|
||||
|
||||
Run the bundled, read-only vault linter and report findings. Pass the conversation's current date (`CHORUS_TODAY`) so stale/aging math uses the same clock the agent writes with:
|
||||
|
||||
```bash
|
||||
SDIR="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts"
|
||||
[ -d "$SDIR" ] || SDIR=$(dirname "$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/vault_lint.py 2>/dev/null | head -1)") # CoWork sandbox fallback
|
||||
CHORUS_TODAY=<currentDate> python3 "$SDIR/vault_lint.py"
|
||||
# Windows: use `python` or `py -3` if `python3` is not on PATH.
|
||||
```
|
||||
|
||||
Exit codes: `0` clean · `1` violations (printed, grouped by check) · `2` vault unreachable · `3` vault not bootstrapped (run `bootstrap.py`).
|
||||
|
||||
Summarize the violations grouped by category and propose fixes, but **do not auto-fix** without the operator's go-ahead. If this is the first substantive session of a calendar month, offer to write the findings to `_agent/health/YYYY-MM-vault-health.md` (per the skill's **Vault Health** section).
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
description: Load CHORUS memory — cold-start context read (profile, scope, latest session, today, inbox)
|
||||
---
|
||||
|
||||
Use the **chorus-memory** skill to load memory now. Run the cold-start **Loading procedure** from `SKILL.md`: the six orientation reads (marker, operator-preferences, current-context, heartbeat, today's daily note, inbox) in **one call**, then do the load-time **reconcile** (inbox-depth + scope-drift) and surface it in a single line.
|
||||
|
||||
```bash
|
||||
CHORUS="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts/chorus.py"
|
||||
[ -f "$CHORUS" ] || CHORUS=$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/chorus.py 2>/dev/null | head -1) # CoWork sandbox fallback
|
||||
python3 "$CHORUS" load
|
||||
# Windows: use `python` or `py -3` if `python3` is not on PATH.
|
||||
```
|
||||
|
||||
`load` prints all six sections (404s on today's note / inbox are shown as absent, not errors) and flags an un-bootstrapped vault. Follow up with `chorus.py scope show` if you need the sessions-since-switch count, and a project search if a specific project is in play.
|
||||
|
||||
**If `load` prints `NOT CONFIGURED` (exit 78)**, this machine has no usable key file yet. Don't proceed with memory — follow **First run** in `SKILL.md`: tell the operator CHORUS isn't configured, ask for their chorus-memory config file (owner/endpoint/key), install it with `chorus.py config import <path>` (or `config set`), then re-run `load`.
|
||||
|
||||
Do not narrate the reads. End with a one-line orientation: who/what/where the active scope is, and any reconcile prompt.
|
||||
@@ -0,0 +1,21 @@
|
||||
---
|
||||
description: Recall a topic from CHORUS memory — matching notes plus their linked neighbourhood
|
||||
argument-hint: "[topic, person, or project]"
|
||||
---
|
||||
|
||||
Use the **chorus-memory** skill to recall context for:
|
||||
|
||||
> $ARGUMENTS
|
||||
|
||||
Run the one-call recall — it resolves the term against the entity index, searches, and expands one hop along `## Related` links and `source_notes`, so you get the connected web (linked decisions, people, prior sessions), not an isolated note:
|
||||
|
||||
```bash
|
||||
CHORUS="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts/chorus.py"
|
||||
[ -f "$CHORUS" ] || CHORUS=$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/chorus.py 2>/dev/null | head -1) # CoWork sandbox fallback
|
||||
python3 "$CHORUS" recall "$ARGUMENTS"
|
||||
# Windows: use `python` or `py -3` if `python3` is not on PATH.
|
||||
```
|
||||
|
||||
The corpus spans the entity graph **plus session logs and journal notes**, ranked by relevance × freshness × status — each hit shows its `updated:`/`status:`, so prefer fresh/active hits and treat stale ones as history. Add `--json` for structured hits (`path/score/type/updated/status/excerpt`) when you need to act on the results programmatically.
|
||||
|
||||
Then answer from the assembled context. If you only need the canonical path for one entity, use `chorus.py resolve "<title>"` instead. Don't narrate the retrieval.
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
description: Reflect on this session — extract durable memories and propose them for one-tap capture
|
||||
argument-hint: "[optional focus, e.g. 'just decisions']"
|
||||
---
|
||||
|
||||
Use the **chorus-memory** skill to run **session reflection** (H5). Scan this conversation for
|
||||
things worth remembering across sessions, then propose them — don't write blindly.
|
||||
|
||||
1. **Extract** durable items from the conversation: new facts, preferences, decisions,
|
||||
commitments, people/companies/projects introduced, and anything the operator said to remember.
|
||||
Skip the ephemeral. Anchor relative dates on the conversation's `currentDate`.
|
||||
2. **Emit a JSON array** of proposals (one per item), each:
|
||||
`{"title","kind","body","aliases","tags","sources","confidence"}` — `kind` ∈
|
||||
`person, company, concept, reference, meeting, project, area, semantic, episodic,
|
||||
working, skill, decision`; set `"inbox": true` when the home is genuinely unknown;
|
||||
`confidence` 0–1 (items below 0.6 are dropped — send those to the inbox instead).
|
||||
Write the array to a file with the Write tool (cross-platform; no heredoc).
|
||||
3. **Preview, then apply.** Dry-run first (writes nothing) and show the operator the plan; only
|
||||
apply after they confirm.
|
||||
|
||||
```bash
|
||||
CHORUS="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts/chorus.py" # python3 (Windows: python / py -3)
|
||||
[ -f "$CHORUS" ] || CHORUS=$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/chorus.py 2>/dev/null | head -1) # CoWork sandbox fallback
|
||||
python3 "$CHORUS" reflect proposals.json # dry-run: validate + classify + preview
|
||||
python3 "$CHORUS" reflect proposals.json --apply # write — routes each via capture (indexed, linked, logged)
|
||||
```
|
||||
|
||||
`reflect` dedups every proposal against the entity index (so it shows create-vs-update),
|
||||
skips below-confidence items, and routes `inbox` proposals to the capture inbox. Each
|
||||
applied item goes through the normal `capture` path — canonical frontmatter, auto-linking,
|
||||
and the recall index — and the writes are lock-guarded. $ARGUMENTS
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
description: Save to CHORUS memory — route content to its canonical home (search-first, idempotent)
|
||||
argument-hint: "[what to remember]"
|
||||
---
|
||||
|
||||
Use the **chorus-memory** skill to persist this to the CHORUS vault:
|
||||
|
||||
> $ARGUMENTS
|
||||
|
||||
Prefer the one-call **`capture`** — it routes (via the entity index), derives the canonical path, stamps complete frontmatter (kind-default `status`, the kind seeded into `tags`), indexes, auto-links mentioned entities, and writes the Agent-Log line. Pick the `--kind` from the content (`person, company, concept, reference, meeting, project, area, semantic, episodic, working, skill, decision`); add `--tags` when obvious; use `--inbox` only when the home is genuinely unknown. Write multi-line bodies to a file with the Write tool (cross-platform, no heredoc) — updates to an existing entity preserve the whole body, so don't pre-trim it.
|
||||
|
||||
```bash
|
||||
CHORUS="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts/chorus.py" # run with: python3 "$CHORUS" ... (Windows: python / py -3)
|
||||
[ -f "$CHORUS" ] || CHORUS=$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/chorus.py 2>/dev/null | head -1) # CoWork sandbox fallback
|
||||
python3 "$CHORUS" capture "<title>" <bodyfile> --kind <kind> [--aliases a,b] [--tags t1,t2] [--source p1,p2]
|
||||
python3 "$CHORUS" capture "<title>" --inbox # unknown home -> idempotent inbox line
|
||||
```
|
||||
|
||||
**If capture exits `76` (duplicate gate):** the title strongly resembles an existing entity — do NOT retry blindly. Show the operator the printed candidates and either update the existing note (`capture ... --merge-into <slug>`) or, only after they confirm it's genuinely distinct, re-run with `--force`.
|
||||
|
||||
Drop to the low-level verbs only for shapes `capture` doesn't model (a project `## Status` replace, an ADR mirror, a daily-note edit):
|
||||
|
||||
```bash
|
||||
python3 "$CHORUS" put <routed/path>.md <bodyfile> # create/overwrite (verifies)
|
||||
python3 "$CHORUS" patch <path>.md append heading "<H1::Sub>" <bodyfile> # targeted append
|
||||
```
|
||||
|
||||
Write in third person about the vault owner; never put `[[wikilinks]]` in frontmatter. `capture` handles `agent_written`, `source_notes`, indexing, and linking for you; if you write by hand, `resolve "<title>"` first to avoid duplicates and `bump` `updated:` only on substantive changes.
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
description: Bring the CHORUS vault up to the current feature spec (entity index + cross-links)
|
||||
allowed-tools: Bash(python3 *sweep.py*), Bash(python *sweep.py*), Bash(ls /sessions/*), Bash(dirname *)
|
||||
---
|
||||
|
||||
Use the **chorus-memory** skill to bring the vault up to the current spec. Run the sweep **dry-run first**, show the operator the plan, and only `--apply` on their go-ahead:
|
||||
|
||||
```bash
|
||||
SDIR="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts"
|
||||
[ -d "$SDIR" ] || SDIR=$(dirname "$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/sweep.py 2>/dev/null | head -1)") # CoWork sandbox fallback
|
||||
python3 "$SDIR/sweep.py" # plan (read-only)
|
||||
python3 "$SDIR/sweep.py" --apply # write
|
||||
# Windows: use `python` or `py -3` if `python3` is not on PATH.
|
||||
```
|
||||
|
||||
The sweep (1) creates `_agent/index/`, (2) rebuilds the entity index from existing notes, (3) symmetrizes `## Related` cross-links (adds only the missing reciprocal direction — never invents new links), and (4) stamps the marker `schema_version`. It is idempotent and read-only without `--apply`. If `schema_version` is behind, run `migrate.py` first. After applying, run `/chorus-health` to confirm invariants.
|
||||
@@ -0,0 +1,23 @@
|
||||
---
|
||||
description: Triage the CHORUS inbox — route aging captures to their canonical homes and log the moves
|
||||
---
|
||||
|
||||
Use the **chorus-memory** skill to run **one-tap Inbox Triage** via the `triage` verb (it reuses the reflect pipeline: classify against the entity index → preview → apply via `capture`, and writes the audit log for you).
|
||||
|
||||
```bash
|
||||
CHORUS="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts/chorus.py"
|
||||
[ -f "$CHORUS" ] || CHORUS=$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/chorus.py 2>/dev/null | head -1) # CoWork sandbox fallback
|
||||
python3 "$CHORUS" triage --list --json
|
||||
# Windows: use `python` or `py -3` if `python3` is not on PATH.
|
||||
```
|
||||
|
||||
1. **List** — the JSON gives each capture as `{line, date, text, age_days}`. Surface the aging ones (≥ ~7 days) to the operator and ask which to route.
|
||||
2. **Propose** — for accepted items, write a proposals JSON with the Write tool (reflect schema: `{"title","kind","body",...}` **plus `"line"`: the original inbox line**, echoed into the audit log). Route per the map: preference/pattern → `semantic` (or a direct PATCH to `operator-preferences.md::Observations`); project idea → `project` with `--status incubating`; durable fact → `semantic`; person fact → `person`.
|
||||
3. **Preview, then apply** with the operator's go-ahead:
|
||||
|
||||
```bash
|
||||
python3 "$CHORUS" triage proposals.json # dry-run: classify + preview, writes nothing
|
||||
python3 "$CHORUS" triage proposals.json --apply # route via capture + log each move
|
||||
```
|
||||
|
||||
`--apply` records every move in `inbox/processing-log/YYYY-MM-DD.md` (`- <original> → <destination>`) automatically. Do **not** delete the original captures unless the operator explicitly asks — the processing log is the audit trail. A proposal that stops at the duplicate gate (exit note in the summary) should be re-proposed with the existing entity's title.
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"hooks": {
|
||||
"SessionStart": [
|
||||
{
|
||||
"matcher": "startup|clear",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "S=\"${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts/chorus_hook_session_start.py\"; [ -f \"$S\" ] || S=$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/chorus_hook_session_start.py 2>/dev/null | head -1); [ -n \"$S\" ] || exit 0; python3 \"$S\" || python \"$S\"",
|
||||
"timeout": 60
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"Stop": [
|
||||
{
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "S=\"${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts/chorus_hook_stop.py\"; [ -f \"$S\" ] || S=$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/chorus_hook_stop.py 2>/dev/null | head -1); [ -n \"$S\" ] || exit 0; python3 \"$S\" || python \"$S\"",
|
||||
"timeout": 30
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,455 @@
|
||||
---
|
||||
name: chorus-memory
|
||||
description: Use the CHORUS Obsidian vault as the operator's persistent memory across Claude/CoWork sessions. Use whenever the operator asks to remember, save, note, log, or capture anything durable — facts, preferences, decisions, schedule changes, commitments — or asks what Claude knows about them, what was discussed or decided before, to check their notes, load their memory/profile/context, add to their inbox, or to track a new project. Trigger even without memory phrasing when the request implies recalling or persisting state across sessions ("pick up where we left off", "anything on X before my meeting?"). Also use proactively at the start of substantive work sessions to load context and at the end to log outcomes. Do NOT use for a different owner's vault, Obsidian/REST-API development questions, "memory" meaning RAM, timed reminders, email or local-file lookups, generic second-brain advice, or notes the operator routes to a specific other file or app.
|
||||
---
|
||||
|
||||
# CHORUS Memory
|
||||
|
||||
Use the **CHORUS** Obsidian vault as persistent memory. Read context accumulated across sessions; write things the operator asks to be remembered.
|
||||
|
||||
The vault belongs to a single **owner**, configured per machine (the `owner` field in `~/.claude/chorus-memory/config.json`) — this is that person's personal memory substrate. Write memory in third person about the owner ("the operator prefers X", not "I prefer X") so the vault stays readable by humans and other agents.
|
||||
|
||||
## API Configuration
|
||||
|
||||
The owner, endpoint, and API key are **machine-local** — the committed plugin ships none of them. `chorus.py` (via `chorus_config`) resolves each field from `~/.claude/chorus-memory/config.json`, with an environment override per field (`CHORUS_OWNER`, `CHORUS_BASE`, `CHORUS_KEY`). Never paste the key into a vault note or a doc.
|
||||
|
||||
Exception — **per-user baked builds:** a `build.py --bake-key` artifact carries the owner/endpoint/key inside the plugin. When a *complete* baked set is present it is **authoritative** — it wins over env vars and the config file and can't be shadowed (so a baked install just works, even on a machine with a stale or placeholder config). The committed source stays credential-free.
|
||||
|
||||
```
|
||||
endpoint = <config "endpoint" / $CHORUS_BASE>
|
||||
key = <config "key" / $CHORUS_KEY — resolved at runtime, never stored in the plugin>
|
||||
owner = <config "owner" / $CHORUS_OWNER — how to refer to the vault owner, third person>
|
||||
```
|
||||
|
||||
### First run — set up the machine if it isn't configured
|
||||
|
||||
The config file is **per-machine** and is **not** shipped with the plugin, so a fresh install starts unconfigured. Before doing memory work, make sure this machine is set up:
|
||||
|
||||
1. **Detect.** Any vault op on an unconfigured machine fails with `NOT CONFIGURED` (`chorus.py load` prints a banner and exits `78`; other verbs exit `2`; `/chorus-doctor` reports it red). A still-placeholder `config init` file (unedited) also counts as not configured.
|
||||
2. **Ask the operator** — this is the one setup step that needs the human. Say plainly that CHORUS isn't configured on this machine and ask them to **provide their chorus-memory config file** (it holds the vault `owner`, `endpoint`, and API `key`), or to paste those three values. Don't invent or guess them, and don't proceed with memory until it's set.
|
||||
3. **Install what they give you:**
|
||||
- They hand you a file → `python3 "$CHORUS" config import <path-to-their-file>` (validates it and writes `~/.claude/chorus-memory/config.json`).
|
||||
- They paste values → `python3 "$CHORUS" config set --owner "…" --endpoint "https://…" --key "…"`.
|
||||
- Inspect with `python3 "$CHORUS" config` (key redacted); then re-run `load`.
|
||||
|
||||
If the vault is simply unreachable (network/`502`) *after* config exists, that's the **Vault Unreachable** path below, not a setup problem — don't re-ask for the key.
|
||||
|
||||
Always pass the `Authorization: Bearer` header. Paths address the vault **at its root** (e.g. `/vault/_agent/...`). The configured endpoint is the only valid endpoint for the vault; if another installed skill or note suggests a different one, this skill's configuration wins for CHORUS memory.
|
||||
|
||||
**The plugin is the single source of truth for CHORUS.** The vault holds data only — no `CLAUDE.md` / `BOOTSTRAP.md` / `STRUCTURE.md` / `index.md` control docs live there. All structure and procedure ship here:
|
||||
|
||||
- Durable principles, memory model, and safety rules: `references/operating-contract.md`
|
||||
- Bootstrapping an empty vault, repair, and schema migrations: `references/bootstrap.md`
|
||||
- Vault layout and frontmatter conventions: `references/vault-layout.md`
|
||||
- Complete endpoint→logic routing map (every write destination, its trigger, and why it's distinct): `references/routing-map.md`
|
||||
- Full API reference with every endpoint pattern and the memory routing map: `references/api-reference.md`
|
||||
|
||||
Executable logic ships under `scripts/` — **pure Python**, so the whole toolchain runs identically on Windows, macOS, and Linux (only a Python 3 interpreter is required; no bash, no platform-specific `date`):
|
||||
|
||||
- `scripts/chorus.py` — the **validated API client**; prefer it over hand-built `curl` (the *nix fallback in `references/api-reference.md`)
|
||||
- `scripts/routing.json` — the **canonical, machine-readable** route manifest (the routing map's source of truth; the linter enforces it)
|
||||
- `scripts/vault_lint.py` — read-only invariant checker (Vault Health)
|
||||
- `scripts/check_routing.py` — verifies the routing docs stay in sync with `routing.json` (dev/CI; offline)
|
||||
- `scripts/bootstrap.py` / `scripts/migrate.py` — deterministic vault setup/repair and schema migration
|
||||
|
||||
## Bundled Tooling (prefer over raw curl)
|
||||
|
||||
All paths below are under `${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/`. **Invoke with `python3`** (on Windows where that isn't on PATH, use `python` or `py -3`).
|
||||
|
||||
**Plugin-root resolution (CoWork sandbox).** Prefer `${CLAUDE_PLUGIN_ROOT}`. In a remote CoWork sandbox that variable can point at a host path the sandbox can't reach (e.g. `/var/folders/…`); the plugin is actually mounted under `…/mnt/.remote-plugins/…`. So resolve the scripts dir once with a fallback and reuse it — the `$CHORUS`/`$LINT`/`$SWEEP` snippets below already do this. The same fallback applies to `vault_lint.py`, `sweep.py`, `bootstrap.py`, and `migrate.py`.
|
||||
|
||||
**`scripts/chorus.py` — use this for every read/write.** It centralizes auth, **HTTP-status checking** (a failed write exits non-zero instead of looking like success), one bounded retry on 5xx/connection errors, whole-line idempotent append, correct `::` heading targets, frontmatter patches, a read-back-confirmed advisory lock, and a one-call cold-start `load`. The raw `curl` recipes in `references/api-reference.md` are the underlying mechanics / *nix fallback — reach for them only if `chorus.py` is unavailable, and if you do, **check the HTTP status yourself** (the PATCH-heading `400 invalid-target` failure silently loses writes otherwise).
|
||||
|
||||
```bash
|
||||
CHORUS="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts/chorus.py" # call as: python3 "$CHORUS" <cmd> ...
|
||||
[ -f "$CHORUS" ] || CHORUS=$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/chorus.py 2>/dev/null | head -1) # CoWork sandbox fallback
|
||||
python3 "$CHORUS" load # cold-start: the 6 orientation reads in one call
|
||||
python3 "$CHORUS" get <path> # 404 -> exit 44
|
||||
python3 "$CHORUS" ls <dir> ; python3 "$CHORUS" map <path> # listing / document-map
|
||||
python3 "$CHORUS" search <terms...>
|
||||
python3 "$CHORUS" put <path> <file> # create/overwrite (read-back verified)
|
||||
python3 "$CHORUS" append <path> "<line>" # idempotent: skips only on an exact whole-line match
|
||||
python3 "$CHORUS" patch <path> append heading "<H1::Sub>" <file>
|
||||
python3 "$CHORUS" fm <path> updated '"2026-06-19"' ; python3 "$CHORUS" bump <path> # frontmatter
|
||||
python3 "$CHORUS" scope show ; python3 "$CHORUS" scope set "<text>"
|
||||
python3 "$CHORUS" lock <session-id> ; python3 "$CHORUS" unlock <session-id>
|
||||
```
|
||||
|
||||
### High-level memory ops (prefer these — they do the routing for you)
|
||||
|
||||
These collapse the multi-step write/recall discipline into one call each, backed by the **entity index** (`_agent/index/entities.json`, a slug→{path,kind,title,aliases} registry). **Default to them** — they are how this skill reduces the input needed to route and cross-link memory:
|
||||
|
||||
```bash
|
||||
python3 "$CHORUS" capture "<title>" <bodyfile> --kind <kind> [--aliases a,b] [--tags t1,t2] [--source p1,p2] [--status s] [--date YYYY-MM-DD] [--domain business]
|
||||
python3 "$CHORUS" capture "<title>" --inbox # unknown home -> idempotent inbox line
|
||||
python3 "$CHORUS" resolve "<mention>" # mention -> canonical path (or where to create)
|
||||
python3 "$CHORUS" recall "<query>" [--json] # ranked search + 1-hop expansion -> connected context
|
||||
python3 "$CHORUS" link <pathA> <pathB> [--json] # add reciprocal [[Related]] links A<->B
|
||||
python3 "$CHORUS" triage --list --json # structured inbox listing (see Inbox Triage)
|
||||
```
|
||||
|
||||
- **`capture` is the default write.** Given a title + `--kind`, it: resolves the entity via the index (create-vs-update, no manual search-first), derives the canonical path, stamps **complete** canonical frontmatter (`type/status/created/updated/tags/aliases/agent_written/source_notes` — a kind-appropriate default `status` and the kind seeded as a baseline tag, enriched by `--tags`), updates the index, **auto-links** any known entity mentioned in the body (bidirectionally), and appends today's Agent Log line — one call instead of search→put→bump→log. It also **auto-derives aliases** from the title and **learns the mention as an alias** when updating an entity under a different name. Updating an existing entity appends the **whole body** as a dated block (first line on the bullet, the rest indented under it) — nothing is truncated. `--kind` ∈ `person, company, concept, reference, meeting, project, area, semantic, episodic, working, skill, decision`. Use `--inbox` (or omit `--kind`) only when the home is genuinely unknown.
|
||||
- **Duplicate gate (create path).** When a new title strongly resembles an existing entity (fuzzy score ≥ 0.6, env `CHORUS_DUP_GATE`), `capture` **stops before writing** and exits `76` with the candidate list, instead of creating a near-duplicate and warning afterwards. Two precision rules (1.5.1) keep it from over-firing: the gate only blocks on a **same-kind** candidate (a decision or meeting named after a project/person is intentional naming — those surface as warnings only), and a **single shared token blocks only when it's unique to that entity** (a vault-common word like a project family name can't gate everything that mentions it; multi-token overlaps still block). Resolve a gate stop deliberately: `--merge-into <slug>` routes the capture as an update to the existing entity (the new name is learned as an alias), or `--force` creates anyway after you've confirmed they're genuinely distinct. Weak resemblances still create + warn as before.
|
||||
- **`resolve` before constructing any path by hand** — it returns the canonical note (matching slug/title/alias, so *Bob/Robert/RS* converge) or, when nothing matches exactly, a ranked list of **`candidates`** (entities sharing a distinctive token — e.g. "chorus memory" surfaces the project `chorus`). Always check candidates before creating a new note; a shortened name not matching exactly is the classic way a duplicate gets made. This replaces the two-search "search-first" dance for slug-addressed notes.
|
||||
- **`recall` for "what do we know about X"** — it returns the matching notes *and* their one-hop neighbourhood (Related links, `source_notes`), so recall surfaces the web around a topic, not an isolated note. The corpus covers the entity graph **plus session logs and journal notes** (down-weighted, so entity notes win ties) — past discussions and decisions are findable even when nobody promoted them into an entity note. Ranking fuses BM25 with **freshness** (`updated:` half-life decay) and **status** (`active` boosted, `archived` demoted), and each hit prints its `updated:`/`status:` so staleness is visible. `--json` emits structured hits (`path/score/type/updated/status/excerpt`) instead of prose.
|
||||
- **`link`** when two existing notes are related but not yet connected; `capture` already auto-links what it detects.
|
||||
|
||||
The index is maintained automatically by `capture`; `sweep.py` rebuilds it and back-fills links for an existing vault (see **Vault Maintenance**).
|
||||
|
||||
**Bootstrap / migrate** are scripts, not hand-run curl loops: `python3 scripts/bootstrap.py [--dry-run]` (idempotent, probe-before-write, never overwrites) and `python3 scripts/migrate.py [--apply]` (reads the marker's `schema_version` and applies migrations; dry-run by default). See `references/bootstrap.md`.
|
||||
|
||||
### Concurrency — the vault is shared, so coordinate writes
|
||||
|
||||
CHORUS is read/written by multiple clients (Claude Code **and** CoWork sessions). The single-line files (`heartbeat/last-session.md`, `current-context.md::Scope`, `inbox.md`) assume a single writer at a time. Before a burst of writes in a session that may overlap another, take the **advisory lock**, and release it at session end:
|
||||
|
||||
```bash
|
||||
python3 "$CHORUS" lock "cc-<session-id>" # exit 75 if another session holds a fresh lock, or you lost the race
|
||||
# ... do the writes ...
|
||||
python3 "$CHORUS" unlock "cc-<session-id>"
|
||||
```
|
||||
|
||||
Use a stable `<session-id>` for both calls (any unique string for this session). The lock is read-back-confirmed (after PUT, `chorus.py` re-reads and backs off if another writer won the race) and cooperative (a stale lock past `CHORUS_LOCK_TTL`, default 15 min, is reclaimable); it lives at `_agent/locks/vault.lock`. It is a courtesy, not a hard mutex — if you can't take it, tell the operator another session may be active rather than racing it.
|
||||
|
||||
### Slash commands
|
||||
|
||||
`/chorus-load` (cold-start read), `/chorus-save <text>` (route + persist via `capture`), `/chorus-recall <query>` (recall a topic's neighbourhood), `/chorus-triage` (drain the inbox), `/chorus-health` (run the linter), `/chorus-sweep` (bring the vault up to current spec), `/chorus-reflect` (extract→preview→apply durable items from the session), `/chorus-doctor` (readiness check: config, reachability, auth, bootstrap/schema). These are explicit entry points to the procedures below.
|
||||
|
||||
### Session hooks (self-firing load & reflect)
|
||||
|
||||
The plugin ships `hooks/hooks.json`, so the two most drift-prone behaviors no longer depend on remembering them:
|
||||
|
||||
- **SessionStart** runs `chorus.py load` and injects the output as context — cold-start loading is deterministic. Still do the **reconcile** (inbox depth, scope confirm) on that injected context before working.
|
||||
- **Stop** fires **once per substantive session** when nothing has been reflected or session-logged yet: it blocks with a reminder to run the `/chorus-reflect` flow and write the session log + heartbeat. Reflection still previews and asks the operator before writing — the hook only ensures the question gets asked. If nothing durable emerged, just finish; never invent memories to satisfy the nudge.
|
||||
|
||||
If the hooks are disabled or unavailable, the manual procedures below are unchanged and still authoritative.
|
||||
|
||||
## Operating Contract & Safety
|
||||
|
||||
The vault is the **system of record** for long-term memory, not a scratchpad. Default to **additive updates, explicit status changes, and traceable summaries**. Keep agent-managed content (`agent_written: true` + `source_notes`) separable from human-authored content. Non-negotiable safety rules:
|
||||
|
||||
- Do not fabricate facts, relationships, or prior decisions.
|
||||
- Do not mass-restructure the vault unless explicitly asked.
|
||||
- Do not delete notes unless deletion is explicitly requested and clearly safe.
|
||||
- Never store secrets or API keys inside a vault note.
|
||||
|
||||
Full contract (principles, agent role, memory model): `references/operating-contract.md`.
|
||||
|
||||
## When to Load Memory
|
||||
|
||||
Load at the start of any substantive conversation — anything beyond a single quick factual question. The signal: the operator is starting work, planning, asking for help with something that has state, or referencing prior discussions.
|
||||
|
||||
### Loading procedure
|
||||
|
||||
**Run `python3 "$CHORUS" load`** — one call that performs the six orientation reads below and prints each labeled section (404s on today's note / inbox show as absent, not errors; an absent marker is flagged). This replaces issuing six separate GETs by hand. The table documents what `load` reads and why:
|
||||
|
||||
| # | GET | Notes |
|
||||
|---|-----|-------|
|
||||
| 1 | `/vault/_agent/chorus-vault.md` | The bootstrap marker. 404 → vault not set up; follow `references/bootstrap.md`. 200 → check its `schema_version` and migrate if older. |
|
||||
| 2 | `/vault/_agent/memory/semantic/operator-preferences.md` | the operator's profile |
|
||||
| 3 | `/vault/_agent/context/current-context.md` | Active scope + Scope History |
|
||||
| 4 | `/vault/_agent/heartbeat/last-session.md` → then `/vault/_agent/sessions/` | **Read the heartbeat first** — a one-line pointer (`<session-log-path> @ <ISO-timestamp>`) written at the end of the previous session. It's an O(1) jump to the latest log, so you can skip or shortcut the full listing. Fall back to the `sessions/` listing only if the pointer is missing or looks stale; then pick the ~5 most recent by reverse lex sort (filenames `YYYY-MM-DD-HHMM-<slug>.md`, so lex == chrono) and read only the relevant ones. |
|
||||
| 5 | `/vault/journal/daily/YYYY-MM-DD.md` | Today's note; 404 is fine — it's created on first agent activity |
|
||||
| 6 | `/vault/inbox/captures/inbox.md` | Inbox depth probe — feeds the load-time reconcile below. 404 is fine (empty inbox). |
|
||||
|
||||
Do not read every session log — older sessions are reachable via `POST /search/simple/?query=...` when needed.
|
||||
|
||||
**Reconcile at load (do this every cold start, after the batch returns).** The batch already fetched everything needed for a cheap self-check — run it before diving into the work so memory maintains itself instead of drifting:
|
||||
|
||||
1. **Inbox depth (Inbox Triage).** If `inbox/captures/inbox.md` (GET #6) holds dated capture lines older than ~7 days that were never routed, surface the count once and offer to triage — see **Inbox Triage** below. This is the load-time trigger that makes triage self-firing rather than something you only run when asked.
|
||||
2. **Scope drift (state it, don't just check it).** Scope is the most churn-prone state — the operator runs several sessions a day across different topics, so the recorded `## Scope` is frequently stale at load. **Silently working under a stale scope is the default failure mode.** To prevent it, at load read the active scope and its freshness in one call — `python3 "$CHORUS" scope show` (prints `## Scope`, `scope_updated`, and how many sessions have been logged since) — and form a one-line judgment: *does this session's request match the recorded scope?* If it diverges, switch **before** doing the work via `python3 "$CHORUS" scope set "<new scope>"` (see **Scope Switching**). If `scope show` reports several sessions logged since the last switch, treat the recorded scope as suspect and confirm with the operator rather than trusting it.
|
||||
|
||||
Keep the reconcile to a single short line to the operator (e.g. "3 inbox captures from last week are still un-routed — triage now?"); don't let it crowd out the actual request.
|
||||
|
||||
**If a specific topic/project/person is in play**, pull its connected context in one call:
|
||||
|
||||
```bash
|
||||
python3 "$CHORUS" recall "<topic or title>" # the matching notes AND their one-hop neighbourhood
|
||||
```
|
||||
|
||||
`recall` resolves the term against the entity index, searches, and expands one hop along `## Related` links and `source_notes` — so you get the project note *plus* its linked decisions, people, and prior sessions, not an isolated file. (If you only need the canonical path, `resolve "<title>"` is the O(1) lookup.)
|
||||
|
||||
Do NOT narrate this loading. Reading memory is expected behavior.
|
||||
|
||||
## Inbox Triage (one-tap: `chorus.py triage`)
|
||||
|
||||
`inbox/captures/inbox.md` is the catch-all for "save this somewhere — figure out where later." Without triage it grows forever and durable facts get stranded there.
|
||||
|
||||
At the start of a substantive session, check the inbox. If it holds lines older than ~7 days that haven't been routed elsewhere, surface them once:
|
||||
|
||||
> "Three captures from last week are still in the inbox — want to route them now or leave them?"
|
||||
|
||||
**The triage verb does the mechanical work** — it reuses the reflect pipeline (classify against the entity index → preview → apply via `capture`) and writes the processing-log audit line for every move:
|
||||
|
||||
```bash
|
||||
python3 "$CHORUS" triage --list --json # structured captures: {line, date, text, age_days}
|
||||
# decide a kind/title per accepted item, write a proposals JSON (reflect schema +
|
||||
# optional "line": the original inbox line, echoed into the audit log), then:
|
||||
python3 "$CHORUS" triage proposals.json # dry-run: classify + preview, writes nothing
|
||||
python3 "$CHORUS" triage proposals.json --apply # route via capture + log each move
|
||||
```
|
||||
|
||||
Routing targets are the usual homes (preference/pattern → `operator-preferences.md::Observations` via a `semantic` proposal or a direct PATCH; project idea → `project` with `--status incubating`; durable fact → `semantic`; person fact → `person`). `--apply` records each move in `inbox/processing-log/YYYY-MM-DD.md` (`- <original line> → <destination path>`) automatically. Originals are **never deleted** unless the operator explicitly asks — the processing log is the audit trail.
|
||||
|
||||
## Project Lifecycle
|
||||
|
||||
Projects move through four folders under `projects/`. The folder name and the `status:` frontmatter field MUST agree — they are two views of the same state.
|
||||
|
||||
| Folder | `status:` | Meaning |
|
||||
|--------|-----------|---------|
|
||||
| `projects/incubating/` | `incubating` | Idea captured; not actively worked. Promote when work starts. |
|
||||
| `projects/active/` | `active` | Current work. The default state for anything in motion. |
|
||||
| `projects/on-hold/` | `on-hold` | Paused but still tracked. Resumable. |
|
||||
| `projects/archived/` | `archived` | Done, abandoned, or rolled into something else. Not deleted. |
|
||||
|
||||
**Promotion / transition rule:** move the file to the new folder AND update `status:` in the same change. A file in `projects/active/` with `status: on-hold` is broken state — fix it when you see it.
|
||||
|
||||
**Searching:** `POST /search/simple/?query=<slug>` covers all four subfolders in one call. Always search before creating a new note (see the pre-write rule below).
|
||||
|
||||
## When to Write Memory
|
||||
|
||||
Write when the operator:
|
||||
|
||||
- States a fact, preference, or commitment worth keeping ("I prefer X", "we use uv not pip", "standup is Tuesday at 10")
|
||||
- Makes a non-obvious decision worth recording
|
||||
- Says "remember that", "save this", "log this", "add to memory", "note that"
|
||||
- Finishes a meaningful working session future sessions should pick up
|
||||
|
||||
Write in third person about the operator. Every note carries the canonical frontmatter (see below). Agent-generated notes set `agent_written: true`.
|
||||
|
||||
**Default write path: `capture`.** For a routed memory (a fact about a person/company/project/concept/etc.), prefer `python3 "$CHORUS" capture "<title>" <bodyfile> --kind <kind>` — it does the search-first, path derivation, frontmatter, indexing, auto-linking, and Agent-Log line in one call (see **High-level memory ops**). Drop to the lower-level `put`/`patch`/`append` verbs below only for shapes `capture` doesn't model (a project `## Status` replacement, an ADR mirror, a daily-note edit) or to inspect/repair.
|
||||
|
||||
### Before you write — search first (MANDATORY for new notes)
|
||||
|
||||
`capture` and `resolve` enforce this automatically via the entity index. If you write a slug-addressed note **by hand**, first **`resolve` the title/slug** (it matches aliases and returns the canonical path or confirms none exists); if `resolve` is unavailable, `search` the whole vault for the slug. Listing a single folder (e.g. `projects/active/`) is NOT sufficient — a note with the same slug may exist in `projects/on-hold/`, `projects/incubating/`, `projects/archived/`, or under a different folder entirely.
|
||||
|
||||
```bash
|
||||
python3 "$CHORUS" resolve "<title>" # preferred: alias-aware, O(1)
|
||||
python3 "$CHORUS" search <slug> # fallback when the index isn't trusted
|
||||
```
|
||||
|
||||
If a match is found:
|
||||
- In a non-active project subfolder (`on-hold/`, `incubating/`, `archived/`): **promote/merge** — PUT the merged content to `projects/active/<slug>.md` with `status: active`, then DELETE the old location. Preserve the earliest `created:` date.
|
||||
- In the same folder you intended to write: **update in place** (PATCH or merged PUT). Never silently overwrite — fold the existing content in first.
|
||||
- Elsewhere (e.g. a stale duplicate under `resources/`): tell the operator and ask which should be canonical before writing.
|
||||
|
||||
**Search both the slug AND any human title** the operator used (e.g. slug `chorus-memory` and title `CHORUS plugin`). Slug-only searches miss notes filed under a different naming scheme. Two cheap `POST /search/simple/?query=...` calls beat one expensive cleanup pass later.
|
||||
|
||||
Only after the search comes back empty (or you've decided to merge) is it safe to create a new note. This rule prevents the most common duplication bug: a note exists in `on-hold/` but the agent only checked `active/` and created a parallel record.
|
||||
|
||||
### Before you append — read first (idempotency)
|
||||
|
||||
`chorus.py append` is **whole-line idempotent**: it GETs the file and skips the POST when the exact line already exists, so a retry, replay, or overlapping session can't grow duplicate lines. Use it for single-line additions to `inbox/captures/inbox.md`, a daily note's `## Agent Log`, or a `## Fact / Pattern` / `## Observations` / `## Log` heading. (A raw POST is **not** idempotent — if you must use curl, GET first and whole-line-match the entry yourself; see `references/api-reference.md`.) This does **not** apply to PUT or PATCH `replace`, which overwrite by design.
|
||||
|
||||
### Append, patch, put — via the client
|
||||
|
||||
Write multi-line bodies to a file first (use the Write tool — cross-platform, no heredoc) and pass the path; or pipe via stdin (`-`).
|
||||
|
||||
```bash
|
||||
# additive line (idempotent). Anchor the date on the conversation's currentDate.
|
||||
python3 "$CHORUS" append inbox/captures/inbox.md "- <currentDate>: <entry>"
|
||||
|
||||
# targeted heading append/replace. The heading Target is the FULL `::`-delimited path
|
||||
# from the H1 — a bare subheading returns 400 invalid-target and the write is LOST.
|
||||
# If unsure of the exact path, GET the document map first and copy the heading verbatim:
|
||||
python3 "$CHORUS" map _agent/memory/semantic/operator-preferences.md
|
||||
python3 "$CHORUS" patch _agent/memory/semantic/operator-preferences.md \
|
||||
append heading "Operator Preferences::Fact / Pattern" <bodyfile>
|
||||
# `replace` (instead of `append`) overwrites a section entirely (e.g. a project's "Name::Status").
|
||||
|
||||
# create/overwrite a whole file (read-back verified; intermediate dirs auto-created)
|
||||
python3 "$CHORUS" put projects/active/<slug>.md <bodyfile>
|
||||
|
||||
# frontmatter freshness after a SUBSTANTIVE change (status/decision/scope) — NOT routine log appends
|
||||
CHORUS_TODAY=<currentDate> python3 "$CHORUS" bump projects/active/<slug>.md # sets updated: to today
|
||||
# fm is CREATE-or-replace: a key the note lacks is inserted surgically (nothing else
|
||||
# in the file is touched), so repairing a note missing `status:` is one call:
|
||||
python3 "$CHORUS" fm <path> status active
|
||||
|
||||
# search (run before creating any slug-addressed note — see the search-first rule above)
|
||||
python3 "$CHORUS" search <terms...>
|
||||
```
|
||||
|
||||
Every PUT body needs the canonical YAML frontmatter (see `references/vault-layout.md`); set `agent_written: true` and `source_notes`. Bump `updated:` on substance, not on heartbeat — adding an Agent Log line, an inbox capture, or a timestamped Observations bullet is not a meaningful content change. Raw curl mechanics for every verb live in `references/api-reference.md`.
|
||||
|
||||
## Scope Switching (`current-context.md`)
|
||||
|
||||
`_agent/context/current-context.md` tracks a single active scope. The operator routinely shifts scope within a day (one project → another → docs), so this is high-churn — switch deliberately, every time the work changes topic.
|
||||
|
||||
**Preferred — one command:**
|
||||
|
||||
```bash
|
||||
python3 "$CHORUS" scope set "<new scope summary>"
|
||||
```
|
||||
|
||||
`scope set` does the whole switch atomically and correctly: it archives the **prior** scope to `## Scope History` (dated, truncated), replaces `## Scope` with the new text, and stamps the `scope_updated:` frontmatter timestamp. That timestamp is the **freshness signal** — it's what `chorus.py scope show` and the `vault_lint.py` drift check read to tell whether the recorded scope still reflects current work. Always switch through `scope set` so `scope_updated` stays honest; a hand-edited `## Scope` that skips the stamp reintroduces silent drift.
|
||||
|
||||
**Manual fallback** (only if `chorus.py` is unavailable): PATCH `prepend` the prior scope to `## Scope History`, PATCH `replace` `## Scope`, then PATCH the frontmatter `scope_updated:` (and `updated:`) to today (see `references/api-reference.md` for the raw curl). Note `scope_updated` must already exist in frontmatter — a `PATCH replace` on a missing field returns `400 invalid-target`; run `bootstrap.py` to add it.
|
||||
|
||||
This keeps a rolling trail of recent scopes in one file instead of spawning separate stash notes. Trim Scope History to the last ~10 entries when it grows past that.
|
||||
|
||||
**Drift backstop:** `vault_lint.py` flags when ≥ `SCOPE_STALE_SESSIONS` (default 3) session logs are dated after `scope_updated` — i.e. work happened without a scope switch. It's advisory (surfaced in Vault Health / `/chorus-health`), the mechanical safety net under the load-time judgment above.
|
||||
|
||||
## Journal Rollups (the journal is one continuum)
|
||||
|
||||
The journal is a single append-only chronological stream. Rollups are just coarser-grained journal entries over the same timeline, so they **all live under `journal/`** — there is no separate `reviews/` tree. One place to read the whole time-series story, daily through annual.
|
||||
|
||||
| Cadence | Path | Trigger | Type |
|
||||
|---------|------|---------|------|
|
||||
| Daily | `journal/daily/YYYY-MM-DD.md` | first agent activity that day | `daily-note` |
|
||||
| Weekly | `journal/weekly/YYYY-Www.md` (e.g. `2026-W24.md`) | first substantive session of a new ISO week — **opt-in**, offer first | `weekly-note` |
|
||||
| Monthly | `journal/monthly/YYYY-MM.md` | first substantive session of a new calendar month — offer alongside the Vault Health pass | `monthly-note` |
|
||||
| Quarterly | `journal/quarterly/YYYY-Qn.md` | **manual / on request only** | `review` |
|
||||
| Annual | `journal/annual/YYYY.md` | **manual / on request only** | `review` |
|
||||
|
||||
All filenames lex-sort chronologically. Derive the ISO week (`YYYY-Www`) and month (`YYYY-MM`) from the conversation's `currentDate` and check whether that period's note already exists before offering a rollup.
|
||||
|
||||
A weekly/monthly rollup is a **light digest** — open threads across `projects/active/`, inbox items aging past ~7 days, and the period's `## Scope History` changes from `current-context.md`. It is *not* a vault-health audit (that's an agent-maintenance artifact — see below).
|
||||
|
||||
## Vault Health (monthly)
|
||||
|
||||
On the first substantive session of a calendar month, run a quick health pass and write findings to `_agent/health/YYYY-MM-vault-health.md`. This is **agent self-maintenance, not a journal entry** — it lives under `_agent/` because it's about the vault's mechanical integrity, not the operator's work narrative. Don't auto-fix without asking.
|
||||
|
||||
Run the bundled linter first — it mechanically checks the invariants below so you don't eyeball them. **Pass `CHORUS_TODAY` = the conversation's `currentDate`** so stale/aging math uses the same clock you write with (not the runner's machine date):
|
||||
|
||||
```bash
|
||||
LINT="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts/vault_lint.py"
|
||||
[ -f "$LINT" ] || LINT=$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/vault_lint.py 2>/dev/null | head -1) # CoWork sandbox fallback
|
||||
CHORUS_TODAY=<currentDate> python3 "$LINT"
|
||||
```
|
||||
|
||||
Exit codes: `0` clean · `1` violations · `2` unreachable · `3` not bootstrapped. Checks (the linter asserts each and prints violations):
|
||||
|
||||
1. **Stale active projects** — for each note in `projects/active/`, check `updated:` >30 days. Likely belongs in `on-hold/`.
|
||||
2. **Unprocessed inbox** — GET `inbox/captures/inbox.md`. List items older than 14 days that never moved through the triage protocol.
|
||||
3. **Duplicate slugs across lifecycle folders** — any slug appearing in more than one of `active/`, `incubating/`, `on-hold/`, `archived/` is broken state.
|
||||
4. **Folder ↔ `status:` mismatch** — any `projects/<lifecycle>/` note whose `status:` frontmatter disagrees with its folder.
|
||||
5. **Wikilinks in frontmatter** — any `[[...]]` inside a YAML frontmatter block (breaks Obsidian reading view), swept across all folders.
|
||||
6. **Duplicate `## Agent Log` headings** — any daily note with more than one.
|
||||
7. **Unknown / retired paths** — any vault file that matches no route in `scripts/routing.json` (or sits at a retired path).
|
||||
8. **Frontmatter integrity** — missing required fields, `updated` < `created`, future `updated`, and `[[wikilinks]]` leaking into `source_notes`.
|
||||
9. **Graph health** — **broken wikilinks** (`[[X]]` resolving to no note), **orphans** (an entity note with no inbound links and an empty `## Related`), and **index drift** (entity-index entries whose file is gone, or indexable notes missing from `entities.json` — fix with `sweep.py`).
|
||||
10. **Incomplete entity frontmatter** — entity notes missing or empty `status:`/`tags:` per the kind-scoped requirement map (`KIND_REQUIRED_FM` in `chorus_index.py`; journal/session notes are exempt). `capture` now stamps these at write time; `sweep.py --apply` backfills older notes (kind-default status + the kind as a baseline tag).
|
||||
|
||||
## Vault Maintenance — `sweep.py` (bring the vault up to spec)
|
||||
|
||||
After a plugin upgrade, run the sweep once to back-fill what older vaults lack — it (1) creates `_agent/index/`, (2) **rebuilds the entity index** from the notes already present (and the recall index, now spanning entities + sessions/journal), (3) **backfills incomplete entity frontmatter** (kind-default `status`, the kind as a baseline tag — what `/chorus-health` flags as `incomplete-frontmatter`), (4) **symmetrizes cross-links** (for every `## Related` A→B, adds the missing B→A; it never invents new links from body text), and (5) stamps the marker `schema_version` to current. Dry-run by default:
|
||||
|
||||
```bash
|
||||
SWEEP="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts/sweep.py"
|
||||
[ -f "$SWEEP" ] || SWEEP=$(ls /sessions/*/mnt/.remote-plugins/*/skills/chorus-memory/scripts/sweep.py 2>/dev/null | head -1) # CoWork sandbox fallback
|
||||
python3 "$SWEEP" # plan (read-only)
|
||||
python3 "$SWEEP" --apply # write
|
||||
```
|
||||
|
||||
Run it after `migrate.py` (which handles structural/schema changes) — or any time `/chorus-health` reports index drift. It is idempotent; re-running is safe.
|
||||
|
||||
The pass is cheap and pays for itself by catching drift before it requires a reorg. Write the findings as a digest; act on them only with the operator's go-ahead.
|
||||
|
||||
**Performance.** Full-vault scripts (`sweep.py`, `vault_lint.py`, recall rebuild) are connection-pooled (keep-alive) and read the whole vault concurrently via `chorus.read_many`, so they finish in about a second on a few-hundred-note vault instead of timing out on hundreds of serial TLS handshakes. Tune with `CHORUS_WORKERS` (default 8, bulk-read concurrency) and `CHORUS_TIMEOUT` (default 30s, per-request). For a vault large enough that even the concurrent pass nears the tool timeout, run the script with the harness's background-execution option rather than blocking the turn.
|
||||
|
||||
## Daily Note — Agent Log
|
||||
|
||||
After substantive activity, write a one-line entry to today's daily note's `## Agent Log` heading. The daily-note **template** (`journal/templates/daily-note-template.md`) defines this heading, but ad-hoc daily notes created without the template don't have it — so PATCH can fail with `invalid-target` if the note exists but lacks the heading.
|
||||
|
||||
**Procedure (resilient)** — all dates are the conversation's `currentDate`, not the machine clock:
|
||||
|
||||
1. `python3 "$CHORUS" get journal/daily/<currentDate>.md`.
|
||||
2. **404** — fetch `journal/templates/daily-note-template.md`, substitute its `{{date:YYYY-MM-DD}}` tokens to `<currentDate>`, write the result to a file (Write tool), and `put` it to the daily path; then go to step 4.
|
||||
3. **200 but no `## Agent Log`** — detect by matching an anchored `^## Agent Log` line in the **raw markdown** you just fetched. Do **not** check the document map, whose headings are full `::`-delimited paths (e.g. `2026-06-07::Agent Log`), so a bare `Agent Log` never matches and you'd append a duplicate heading. If missing, `post` a body of exactly `\n\n## Agent Log\n` to add the heading.
|
||||
4. PATCH-append the entry under `<currentDate>::Agent Log` (the daily note's H1 is the date, so that is the full target path):
|
||||
|
||||
```bash
|
||||
python3 "$CHORUS" patch journal/daily/<currentDate>.md append heading "<currentDate>::Agent Log" <bodyfile>
|
||||
```
|
||||
|
||||
## Where to Write
|
||||
|
||||
| Situation | File / path | Method |
|
||||
|-----------|-------------|--------|
|
||||
| Unsure where it goes / quick capture | `inbox/captures/inbox.md` (date-prefixed line) | POST |
|
||||
| Operator preference or durable fact | `_agent/memory/semantic/operator-preferences.md` (append under the right heading) | PATCH |
|
||||
| Other durable facts / patterns | `_agent/memory/semantic/<slug>.md` | PUT |
|
||||
| What happened (event record) | `_agent/memory/episodic/<slug>.md` | PUT |
|
||||
| Short-lived working state | `_agent/memory/working/<slug>.md` | PUT |
|
||||
| Task-scoped context / focus | `_agent/context/current-context.md` | PATCH / PUT |
|
||||
| Working session ended with substance | `_agent/sessions/YYYY-MM-DD-HHMM-<slug>.md` | PUT |
|
||||
| Long-running project state | `projects/<lifecycle>/<slug>.md` (see Project Lifecycle) | PUT + PATCH |
|
||||
| Ongoing area of responsibility (standing domain, no end state) | `areas/<domain>/<slug>.md` (domain: `business`/`personal`/`learning`/`systems`) | PUT |
|
||||
| Non-obvious decision (ADR) | `decisions/by-date/YYYY-MM-DD-<slug>.md` (see mirror note below) | PUT |
|
||||
| Person context | `resources/people/<name>.md` | PUT / PATCH |
|
||||
| Company / organization context | `resources/companies/<slug>.md` | PUT / PATCH |
|
||||
| Concept / reference note | `resources/concepts/` or `resources/references/` | PUT |
|
||||
| Meeting notes / call recap | `resources/meetings/YYYY-MM-DD-<slug>.md` | PUT |
|
||||
| Skill / plugin capability entry (catalog, not the build work) | `_agent/skills/active/<slug>.md` (→ `_agent/skills/archived/` when retired) | PUT |
|
||||
| Daily activity / Agent Log | `journal/daily/YYYY-MM-DD.md` — see **Daily Note — Agent Log** above | PATCH (with auto-create) |
|
||||
| Weekly / monthly rollup | `journal/weekly/YYYY-Www.md` · `journal/monthly/YYYY-MM.md` — see **Journal Rollups** | PUT |
|
||||
| Quarterly / annual review | `journal/quarterly/YYYY-Qn.md` · `journal/annual/YYYY.md` (manual / on request) | PUT |
|
||||
| Vault-health audit (agent self-maintenance) | `_agent/health/YYYY-MM-vault-health.md` — see **Vault Health** | PUT |
|
||||
| Session-end orientation pointer | `_agent/heartbeat/last-session.md` (one line: `<session-log-path> @ <ISO-timestamp>`) | PUT |
|
||||
|
||||
> **The complete, audited routing map lives in `references/routing-map.md`** — every write destination with its trigger, what lands there, and why it's distinct from its neighbors. This table is the quick-reference; the map is the authority. If a path isn't in the map, it shouldn't be written to.
|
||||
|
||||
**Decision mirrors:** if the decision belongs to an existing note in `projects/active/`, add a `[[wikilink]]` to the ADR under that project's `## Key Decisions` heading (PATCH). If no matching project note exists, skip the mirror — the by-date ADR is sufficient; do not invent topical mirror files in `decisions/by-project/`.
|
||||
|
||||
**Routing triggers — areas / meetings / skills:**
|
||||
|
||||
- **Area** (`areas/`): a standing responsibility the operator maintains with no finish line. Decision rule: has an end state → `projects/`; never "done" → `areas/`. Subfolder is the domain (`business`/`personal`/`learning`/`systems`); create it on first write. `type: area`.
|
||||
- **Meeting** (`resources/meetings/`): notes, recaps, or action items tied to a specific meeting or call. Mirror decisions to `decisions/by-date/` and commitments to the relevant project. `type: meeting`.
|
||||
- **Skill** (`_agent/skills/`): the operator authors, installs, or retires a skill/plugin and wants it tracked as a capability — distinct from `projects/`, which tracks the build effort. Active entries in `active/`, retired ones in `archived/`. `type: skill`.
|
||||
|
||||
Never delete files unless the operator explicitly asks. Memory is append-friendly; deletion is destructive.
|
||||
|
||||
## Session Logging
|
||||
|
||||
At the end of substantive conversations (ones that produced decisions, artifacts, or commitments), create a session log. Ask once if unsure: "Want me to log a session note for this?"
|
||||
|
||||
**Filename format is canonical: `_agent/sessions/YYYY-MM-DD-HHMM-<slug>.md`.** Always include the four-digit local-time HHMM component — it makes filenames lexically sort in true chronological order, which is what Step 3 of loading relies on. Older session logs without the HHMM part exist; leave them alone, but every new one must use the full form.
|
||||
|
||||
Write the body (see `references/session-log-template.md`) to a file with the Write tool, then:
|
||||
|
||||
```bash
|
||||
python3 "$CHORUS" put _agent/sessions/<currentDate>-HHMM-<slug>.md <bodyfile>
|
||||
```
|
||||
|
||||
Then add a one-line entry to today's daily note via the **Daily Note — Agent Log** procedure above.
|
||||
|
||||
Finally, update the heartbeat pointer so the next session can orient in one read (this closes the loop with loading-procedure Step 4 — a pointer nobody writes is a pointer nobody can read). It is a single line, `<session-log-path> @ <ISO-timestamp>`; write it to a file and PUT it:
|
||||
|
||||
```bash
|
||||
python3 "$CHORUS" put _agent/heartbeat/last-session.md <bodyfile>
|
||||
```
|
||||
|
||||
`last-session.md` is overwritten (PUT) each session end — never appended, so it can't grow or duplicate.
|
||||
|
||||
## Vault Unreachable
|
||||
|
||||
If the API returns a connection error, timeout, or `502`, tell the operator once that the memory vault is unreachable (a `502` usually means Obsidian/the REST plugin is not running on the backend), then proceed without memory. Do not retry repeatedly.
|
||||
|
||||
## Style Rules
|
||||
|
||||
- Write in third person about the operator: "the operator prefers X", not "I prefer X".
|
||||
- This vault holds only its own owner's memory. Do not cross-write between distinct vaults/owners — another person's vault and preferences belong in their own vault, not here.
|
||||
- **Anchor relative dates on the conversation's `currentDate`** before writing. "Today" → `currentDate`. "Thursday" / "next week" → resolve to an absolute `YYYY-MM-DD`. Never guess from training-data knowledge of the current year.
|
||||
- Every memory file has canonical YAML frontmatter — see `references/vault-layout.md`.
|
||||
- Set `agent_written: true` on agent-generated notes and list `source_notes` (plain relative paths, not links).
|
||||
- **`created:` is the earliest known date the entity was tracked anywhere in the vault, not "today".** When merging notes (e.g. promoting `on-hold/` → `active/`), preserve the earliest `created:` and only bump `updated:`.
|
||||
- **`source_notes` lists the note(s) that *triggered* or *supplied content for* this one** — e.g. the session log that produced a project update, or the daily note where a captured fact originated. It is a *backward* link to inputs. Forward links (this note → other notes it references) belong in the `## Related` section in the note body, never in frontmatter.
|
||||
- **Never put `[[wikilinks]]` in frontmatter** — YAML parses them as nested lists and the links break in Obsidian's reading view. Put all cross-references in a `## Related` section in the note **body** as a bulleted list of `[[links]]`.
|
||||
- Use Obsidian wiki links (`[[note name]]`) freely in the note **body** for cross-references.
|
||||
- Keep entries short and focused. Fewer, sharper entries beat many noisy ones — the operator explicitly prefers concision.
|
||||
- About to write something large or sensitive? Show the operator the content first and confirm.
|
||||
|
||||
## operator-preferences.md — Rules vs Observations
|
||||
|
||||
`_agent/memory/semantic/operator-preferences.md` separates two kinds of content:
|
||||
|
||||
- `## Fact / Pattern` — **promoted, deduped rules.** No date prefix. These are timeless: "the operator prefers concise communication." Append here only when a rule is stable.
|
||||
- `## Observations` — **timestamped raw observations.** Date-prefixed: `- 2026-06-06: the operator chose X over Y because Z.` This is where new evidence goes by default.
|
||||
|
||||
During monthly Vault Health or when an observation stabilizes, promote it from `## Observations` into `## Fact / Pattern` (drop the date) and remove the duplicate from Observations. Trim Observations to the last ~30 entries when it grows past that — the rest live in session logs.
|
||||
|
||||
## What This Skill Does Not Do
|
||||
|
||||
- Does not replace reasoning. The vault is reference material — apply judgment.
|
||||
- Does not auto-summarize the whole vault. Read targeted files, not everything.
|
||||
- Does not store passwords or secrets, and never writes the API key into a vault note. If asked to "remember my password is X", decline and suggest a password manager.
|
||||
@@ -0,0 +1,146 @@
|
||||
# CHORUS Performance Test Suite — Plan
|
||||
|
||||
**Status:** plan only — nothing here is to be run yet.
|
||||
**Author:** drafted for the vault owner, 2026-06-23.
|
||||
**Targets the changes that landed in:** 1.1.0 (connection pooling + concurrent `read_many` + shared cache) and 1.2.0 (fuzzy `resolve`, alias derivation/learning, `capture` re-slug fix).
|
||||
**Scope:** a *performance / timing* harness. It complements — does not replace — the existing correctness tests (`test_chorus_client.py`, `test_v1_scaffold.py`, the eval feature tests), which stay as the pass/fail gate for behaviour.
|
||||
|
||||
---
|
||||
|
||||
## 1. Goals
|
||||
|
||||
1. Put hard numbers on the three operations the operator called out — full endpoint read, full subject pulls, bulk get/put/append — so the 1.1.0 perf claims ("lint 0.90s / sweep 0.85s on 186 notes, ~4.5× from reuse, ~2× from concurrency") become a repeatable, checked-in measurement instead of a one-time observation.
|
||||
2. Establish **baselines + regression gates** so a future change that quietly reintroduces the pre-1.1.0 per-request TLS handshake (the bug that was timing out sessions) fails the suite instead of silently shipping.
|
||||
3. Keep every run **non-destructive to the production vault** — writes go to a disposable namespace and are cleaned up (per the operator's "prune test artifacts" rule).
|
||||
|
||||
## 2. Methodology (applies to every metric)
|
||||
|
||||
- **Harness:** a new `bench.py` under `scripts/` (or `eval/perf/`), reusing the live `chorus` client module so it measures the real pooled/concurrent code path, not a reimplementation.
|
||||
- **Timing:** `time.perf_counter()` around each operation. Report **min / median / p90 / p95 / max / mean** over N iterations — not a single number. Single timings hide GC pauses and TLS renegotiation.
|
||||
- **Warm-up:** discard the first 1–2 iterations (cold connection pool, cold server cache) and report them *separately* as the cold-start cost — that delta is itself one of the most interesting numbers (it's what 1.1.0's keep-alive was meant to kill).
|
||||
- **Iterations:** default N=10 for read-heavy metrics, N=5 for write-heavy (writes are slower and we don't want to bloat the vault). Make N a CLI flag.
|
||||
- **Environment capture:** record `CHORUS_WORKERS`, `CHORUS_TIMEOUT`, vault note count, git commit/plugin version, and wall-clock date in the results header so two runs are comparable.
|
||||
- **Output:** machine-readable JSON (`eval/perf/results/<date>-<commit>.json`) plus a short human table to stdout. JSON lets us diff runs over time and chart trend.
|
||||
- **Isolation:** all test writes live under a dedicated prefix, e.g. `_agent/_bench/<run-id>/...`, never in `projects/`, `resources/`, `inbox/`, or `journal/`. A `--cleanup` pass deletes the whole prefix at the end; a `--keep` flag preserves it for debugging. The advisory lock (`chorus.py lock`) is taken for the write phases so a bench run can't race a real CoWork session.
|
||||
- **Network honesty:** the endpoint is the live configured endpoint (`chorus.BASE` / `$CHORUS_BASE`), so absolute numbers depend on WAN latency. Report **both** absolute ms and a relative ratio against the same run's single-GET baseline, so the suite stays meaningful regardless of where it's run from.
|
||||
|
||||
---
|
||||
|
||||
## 3. Core metrics (the three the operator asked for)
|
||||
|
||||
### 3.1 Timed full endpoint read (full-vault bulk read)
|
||||
|
||||
**What it measures:** the headline 1.1.0 path — reading every note in the vault via concurrent `read_many` over warm pooled connections vs. serial GETs.
|
||||
|
||||
**Procedure:**
|
||||
1. Enumerate all vault paths (the same listing `sweep.py`/`vault_lint.py` walk).
|
||||
2. **Serial baseline:** GET each path one at a time on a single connection. Time total + per-note.
|
||||
3. **Pooled+concurrent:** `read_many(paths)` at the default `CHORUS_WORKERS`. Time total.
|
||||
4. Report total time, notes/sec, and the **speedup ratio** (serial ÷ concurrent). Repeat the concurrent pass N times for percentile spread.
|
||||
|
||||
**What "good" looks like:** concurrent should be multiples faster than serial; full-vault concurrent read should stay well under the agent tool timeout (the original failure mode). Record current note count alongside — the metric is only comparable at similar vault sizes.
|
||||
|
||||
**Regression gate (suggested):** full-vault concurrent read median < a threshold (e.g. 3s at ~200 notes) AND speedup ratio ≥ ~2×. Tune thresholds after the first baseline run.
|
||||
|
||||
### 3.2 Timed full subject pulls (e.g. APTA, CapMetro statuses)
|
||||
|
||||
**What it measures:** the cost of a `recall "<subject>"` — search + 1-hop neighbourhood expansion along `## Related` links and `source_notes` — which is the real "what do we know about X" operation, and exercises the 1.2.0 resolver. APTA and CapMetro are live subjects in the vault, so they're realistic fixtures.
|
||||
|
||||
**Procedure:**
|
||||
1. Pick a fixed fixture set of subjects with **different neighbourhood sizes**: a small one, a hub note with many links (e.g. the `chorus` project or `operator-preferences`), and the two named ones (APTA, CapMetro). Neighbourhood size is the real cost driver, so vary it deliberately.
|
||||
2. Time end-to-end `recall` for each: (a) the search call, (b) the 1-hop expansion reads, (c) total. Break out the search vs. expansion split — it tells us whether latency is in the index/search or in the fan-out reads (which 1.1.0's `read_many` should accelerate).
|
||||
3. Also time bare `resolve "<subject>"` (the O(1) index lookup) separately as the floor — recall should be resolve + search + N expansion reads.
|
||||
4. Report per-subject and aggregate percentiles; annotate each with its neighbourhood node count so time-per-node is derivable.
|
||||
|
||||
**What "good" looks like:** `resolve` is near-instant (in-memory index); `recall` scales with neighbourhood size but the expansion reads are concurrent, so a hub note shouldn't be linearly worse than a leaf.
|
||||
|
||||
**Live result + the `expand-graph` sub-metric (added 2026-06-23).** The first run showed `recall()` at 2.8–4.7 s/subject. Profiling pinned it on the graph layer, not BM25: `load_index()` ~500 ms (index already persisted/incremental), `score()` 0.1 ms, `expand_graph()` ~3,000 ms doing one serial GET per neighbour. Fix shipped in `chorus_recall.py` — the BFS now fetches each hop via `read_many`. A dedicated **`expand-graph`** metric times serial vs concurrent expansion (gate: ≥2× min) and asserts byte-identical ranking + scores against a serial reference, so the fix can't silently regress. Measured 3.5–4.2×; end-to-end `recall()` down ~1.7×.
|
||||
|
||||
### 3.3 Timed bulk get / put / append
|
||||
|
||||
**What it measures:** write-path throughput and the read-back/idempotency overhead the operator deliberately pays for.
|
||||
|
||||
**Procedure (all in the `_agent/_bench/<run-id>/` namespace):**
|
||||
- **Bulk GET:** create K fixture notes once, then time reading them back — both serially and via `read_many` — to compare against §3.1 at a controlled, fixed K (e.g. 25/50/100) so the read curve is isolated from whatever the live vault happens to contain.
|
||||
- **Bulk PUT:** time creating K notes. `chorus.py put` is read-back-verified (it re-GETs after writing), so this measures the true write+verify cost, not a fire-and-forget POST. Report puts/sec and per-put median.
|
||||
- **Bulk APPEND:** time K appends to a single growing note. `append` is whole-line idempotent (it GETs first and skips duplicates), so also measure the **idempotent-skip path**: re-append the same K lines and confirm near-zero net writes — time the skip vs. the real append. This validates that idempotency isn't quietly O(file size) per append as the note grows.
|
||||
- **Concurrent write caveat:** writes to single-line shared files assume one writer; the bulk-write bench must use per-note targets (or hold the lock) so it doesn't model an unsupported pattern.
|
||||
|
||||
**What "good" looks like:** PUT slower than GET (verify round-trip); append-skip much cheaper than append-write; no superlinear blowup as the appended note grows.
|
||||
|
||||
---
|
||||
|
||||
## 4. Additional test suggestions (recommended, beyond the three)
|
||||
|
||||
These target the rest of the 1.1.0/1.2.0 surface and the known historical failure modes.
|
||||
|
||||
1. **Connection-pool cold vs. warm.** Isolate the keep-alive win directly: time the *first* request after pool creation vs. the median of subsequent requests. This is the single clearest proof the pooling fix is alive; a regression here is the early-warning signal for the old per-request-handshake bug.
|
||||
|
||||
2. **Concurrency scaling sweep (`CHORUS_WORKERS`).** Run §3.1 at workers = 1, 2, 4, 8, 16. Plot total time vs. workers to find the knee and confirm the default of 8 is sensible for the current vault/WAN, and that it degrades gracefully (no errors/timeouts) at high concurrency.
|
||||
|
||||
3. **Fuzzy-resolve latency + accuracy (1.2.0).** Two parts: (a) *timing* — `resolve` on exact, alias, and shortened/typo'd mentions; the fuzzy-candidate path does more work, so measure its overhead vs. an exact hit. (b) *correctness* — a fixture table of mention → expected canonical slug (e.g. "chorus memory" → `chorus`, alias hits, a near-miss that *should* return ranked candidates rather than spawn a duplicate). This guards the actual bug 1.2.0 fixed. Correctness assertions, not just timing.
|
||||
|
||||
4. **Index build/rebuild time.** Time `sweep.py` entity-index rebuild + link symmetrization on the full vault. This is the other full-vault script that historically neared the timeout; gate it like §3.1.
|
||||
|
||||
5. **Lint + recall-rebuild full-vault timing.** Regression-gate `vault_lint.py` and the recall rebuild against the 1.1.0 baselines (lint ~0.90s, sweep ~0.85s @ 186 notes). These are the concrete numbers in the changelog — turn them into asserted thresholds scaled to current note count.
|
||||
|
||||
6. **Lock contention timing.** Time `lock` acquisition when free, when held-fresh (should fast-fail with exit 75), and reclaim of a stale lock past `CHORUS_LOCK_TTL`. Confirms the read-back-confirmed lock doesn't add pathological latency and behaves under contention.
|
||||
|
||||
7. **Offline write-ahead queue throughput (1.0 carry-over).** With the endpoint unreachable, time enqueue of K writes, then time `flush` replay on reconnect. Validates the queue doesn't degrade and replays in order. Useful because it's an untimed path today.
|
||||
|
||||
8. **Cache hit/miss ratio.** Instrument the shared single-pass cache: in a sweep/lint run, count served-from-cache vs. network reads. A correctness+efficiency check that the cache is actually collapsing duplicate reads, not just present.
|
||||
|
||||
9. **Scaling fixture / synthetic vault.** Optional but valuable: a script that mints a throwaway vault of N synthetic notes (100 / 500 / 1000) in the bench namespace to measure how all the above scale *beyond* the current ~190 notes — answers "when does the next timeout cliff arrive?" before it hits a real session.
|
||||
|
||||
10. **Soak / stability pass.** Run §3.1 in a loop for M minutes to surface connection leaks, pool exhaustion, or memory growth in the long-lived client — relevant because the pool is thread-local and long-running.
|
||||
|
||||
---
|
||||
|
||||
## 5. Proposed file layout
|
||||
|
||||
```
|
||||
eval/perf/
|
||||
PERF-TEST-PLAN.md # this file
|
||||
bench.py # the harness (CLI: --metric, --iterations, --workers, --cleanup/--keep, --json-out)
|
||||
fixtures.py # subject lists, K-note generators, mention→slug resolve table
|
||||
baselines.json # committed thresholds + reference numbers (per note-count bucket)
|
||||
results/ # dated JSON run outputs (gitignored or kept for trend)
|
||||
2026-06-23-<commit>.json
|
||||
```
|
||||
|
||||
`bench.py` subcommands map 1:1 to the metrics: `read-full`, `subject-pull`, `bulk-get`, `bulk-put`, `bulk-append`, plus `pool-warmup`, `worker-sweep`, `resolve`, `index`, `lint`, `lock`, `queue`, `cache`, `soak`.
|
||||
|
||||
## 6. Decisions (resolved 2026-06-23)
|
||||
|
||||
- **Baseline policy:** **relative ratios** are the gate (`baselines.json`); absolute ms are recorded as informational. Portable across machines/WAN.
|
||||
- **CI vs. manual:** **manual / pre-release.** The suite is invoked by hand before tagging; it is not wired into CI (avoids WAN flakiness failing builds against the live endpoint).
|
||||
- **Synthetic vault (suggestion #9):** **deferred.** Benchmark the real ~190-note vault for now; add the generator if vault growth makes it relevant.
|
||||
- **Cleanup:** **auto-delete** the `_agent/_bench/<run-id>/` namespace at end of each run, with `--keep` to override.
|
||||
|
||||
## 7. Build status
|
||||
|
||||
Harness built under `eval/perf/`:
|
||||
|
||||
- `bench.py` — the CLI harness; all 14 metrics implemented (`read-full`, `subject-pull`, `bulk-get`, `bulk-put`, `bulk-append`, `pool-warmup`, `worker-sweep`, `resolve`, `index`, `lint`, `lock`, `queue`, `cache`, `soak`).
|
||||
- `fixtures.py` — subject list (APTA/CapMetro/hubs/leaf), the 1.2.0 resolve correctness table, and bench-note generators.
|
||||
- `baselines.json` — relative-ratio gates (tune thresholds after the first real run).
|
||||
- `README.md` — run instructions, metric map, isolation/cleanup, caveats.
|
||||
- `results/` — dated JSON run outputs.
|
||||
|
||||
Verified offline (compile + imports + `--list`/`--help`), then **run live** against the 197-note vault @ af16598 — 11/11 gate checks pass. First run surfaced the recall() finding below; the fix and a guarding metric were added the same session.
|
||||
|
||||
## 8. CHORUS usage improvements (from the live run)
|
||||
|
||||
The benchmark wasn't just validation — it surfaced concrete improvements to how CHORUS should do reads. In priority order:
|
||||
|
||||
1. **Apply `read_many` everywhere a code path reads a *set* of notes (DONE for `expand_graph`).** The 1.1.0 release shipped concurrent bulk reads but only `sweep`/`lint`/full-vault read adopted it. `recall()`'s graph expansion was still serial — one GET per neighbour — which is why it ran 3 s. The rule going forward: *any* loop that GETs more than ~3 notes whose paths are known up front should batch them through `chorus.read_many`, not iterate `get_text`. Audit candidates: `expand_graph` (fixed), `_brief` result printing (still serial), `rebuild()`'s vault walk (still serial), and any future multi-note reader.
|
||||
|
||||
2. **Prefetch `_brief` bodies in one `read_many` (NEXT).** After expansion, `recall` prints each primary hit + neighbour via `_brief()`, which GETs the note again to pull its type/status. Those are known paths — fetch them all in one concurrent batch (and reuse bodies already pulled during expansion via a per-call cache, so a note fetched in the BFS isn't re-fetched to print it).
|
||||
|
||||
3. **Reuse one in-process read cache across a `recall` call.** `load_index`, `expand_graph`, and `_brief` currently re-fetch overlapping notes. A single `{path: text}` memo passed through the call (the same pattern `sweep.py` already uses via `read_many` once) removes the duplicate GETs — `load_index`'s ~500 ms and the `_brief` re-reads are the remaining recall cost.
|
||||
|
||||
4. **Make "batch the reads" a documented contract, not a per-site fix.** Add a one-line rule to the plugin's operating contract / API reference: *prefer `read_many(paths)` over a GET loop; serial multi-note reads are a performance bug.* This is what would have caught the `expand_graph` regression at authoring time. The `expand-graph` metric now enforces it mechanically for recall; the contract generalizes it.
|
||||
|
||||
5. **Consider indexing `## Related`/`source_notes` adjacency into the persisted index.** Longer-term: the graph edges are derivable and change only on write. Storing an adjacency list alongside the BM25 postings (maintained by `capture`/`sweep`, same as `entities.json`) would let `expand_graph` skip re-reading note bodies just to extract links — turning the graph layer into an in-memory lookup like `resolve`. Bigger change; revisit if recall latency still matters after #2–#3.
|
||||
|
||||
None of #1–#4 add dependencies or change behavior — they apply the concurrency primitive that already exists. #1 is done and gated; #2–#3 are the next cheap wins; #5 is the structural option.
|
||||
@@ -0,0 +1,96 @@
|
||||
# CHORUS Performance Test Suite
|
||||
|
||||
A timing harness for the operations the **1.1.0** (connection pooling + concurrent
|
||||
`read_many` + shared cache) and **1.2.0** (fuzzy `resolve` / alias learning) updates
|
||||
targeted. It reuses the live `chorus` client module, so it measures the real pooled /
|
||||
concurrent code path — not a reimplementation.
|
||||
|
||||
This is a **manual / pre-release** tool, not a CI gate: it depends on the live
|
||||
configured endpoint (`chorus.BASE` / `$CHORUS_BASE`) and WAN latency. Gates are **relative ratios**
|
||||
(portable), with absolute milliseconds kept as informational context.
|
||||
|
||||
See `PERF-TEST-PLAN.md` for the design rationale behind each metric.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
cd eval/perf
|
||||
python3 bench.py --list # list metrics
|
||||
python3 bench.py all # the read-only safe set (no vault writes)
|
||||
python3 bench.py read-full -n 10 # full-vault serial-vs-concurrent read
|
||||
python3 bench.py subject-pull # recall APTA / CapMetro / hubs (resolve+search+recall split)
|
||||
python3 bench.py bulk-put -k 50 # write metric -> uses the _bench namespace
|
||||
python3 bench.py worker-sweep # CHORUS_WORKERS = 1,2,4,8,16 scaling curve
|
||||
python3 bench.py resolve # 1.2.0 fuzzy correctness table + timing
|
||||
python3 bench.py soak --soak-seconds 120 # stability / leak watch
|
||||
```
|
||||
|
||||
Configure this machine first (owner/endpoint/key):
|
||||
`python3 ../../scripts/chorus.py config init` then edit the file, or `export CHORUS_BASE=... CHORUS_KEY=...`.
|
||||
|
||||
## Metrics
|
||||
|
||||
| Metric | Plan § | Reads/Writes | What it proves |
|
||||
|---|---|---|---|
|
||||
| `read-full` | 3.1 | read-only | full-vault concurrent vs serial speedup (1.1.0 headline) |
|
||||
| `subject-pull` | 3.2 | read-only | `recall` cost for APTA/CapMetro/hubs, split into resolve floor + search + expansion |
|
||||
| `expand-graph` | 3.2a | read-only | recall()'s graph layer: serial vs concurrent neighbourhood expansion + ranking-parity guard (pins the real recall bottleneck) |
|
||||
| `bulk-get` | 3.3 | writes seed | concurrent vs serial read at a controlled K |
|
||||
| `bulk-put` | 3.3 | **writes** | read-back-verified PUT throughput (puts/sec) |
|
||||
| `bulk-append` | 3.3 | **writes** | append throughput + idempotent-skip path (no O(file) blowup) |
|
||||
| `pool-warmup` | 4.1 | read-only | cold vs warm single GET — keep-alive is alive |
|
||||
| `worker-sweep` | 4.2 | read-only | concurrency scaling knee; validates default workers=8 |
|
||||
| `resolve` | 4.3 | read-only | fuzzy-resolve timing **and** correctness (anti-duplicate guard) |
|
||||
| `index` | 4.4 | read-only | `sweep.py` index rebuild + link symmetrize time |
|
||||
| `lint` | 4.5 | read-only | `vault_lint.py` full-vault time vs 1.1.0 baseline |
|
||||
| `lock` | 4.6 | **writes** lock | advisory-lock acquire/contend/release timing + semantics |
|
||||
| `queue` | 4.7 | none (bogus base) | offline write-ahead enqueue timing (see caveat) |
|
||||
| `cache` | 4.8 | read-only | `read_many` dedups a duplicated path list |
|
||||
| `soak` | 4.10 | read-only | stability over a window; leak/drift detector |
|
||||
|
||||
`all` runs only the read-only set. Write metrics must be requested by name.
|
||||
|
||||
## Isolation & cleanup
|
||||
|
||||
Write metrics touch **only** `_agent/_bench/<run-id>/`, take the advisory lock for
|
||||
the duration (and abort if another session holds it), and delete the namespace at
|
||||
the end. Pass `--keep` to preserve it for debugging. Nothing is written to
|
||||
`projects/`, `resources/`, `inbox/`, or `journal/`.
|
||||
|
||||
## Output
|
||||
|
||||
Each run writes `results/<date>-<commit>.json` (machine-readable, for trend diffing)
|
||||
and prints a human table. The JSON header captures `CHORUS_WORKERS`, `CHORUS_TIMEOUT`,
|
||||
note count, git commit, and date so two runs are comparable.
|
||||
|
||||
## Gating
|
||||
|
||||
`baselines.json` holds relative-ratio gates (e.g. `read-full.speedup_ratio >= 1.8`,
|
||||
`pool-warmup.cold_over_warm_ratio >= 1.5`). Run once to capture a baseline, then
|
||||
tune the thresholds to the observed numbers before treating a `FAIL` as blocking.
|
||||
`--no-gate` reports numbers without pass/fail.
|
||||
|
||||
## Finding: recall() bottleneck is the graph layer, not BM25
|
||||
|
||||
A profiled `recall("operator preferences")` decomposes as: `load_index()` ~500 ms (the
|
||||
BM25 index is already persisted + incrementally maintained, n_docs=119), `score()`
|
||||
0.1 ms, and `expand_graph()` **~3,000 ms** — the graph BFS was fetching each neighbour
|
||||
serially (one GET per node, ~126 nodes). The fix (shipped in `chorus_recall.py`) makes the
|
||||
BFS breadth-first by hop and fetches each frontier through the existing `read_many`
|
||||
concurrency. Result: `expand_graph` 3.2–4.2x faster with byte-identical ranking and
|
||||
scores; end-to-end `recall()` ~1.7x (e.g. APTA 3.8 s -> 2.2 s). The `expand-graph`
|
||||
metric + gate above regression-guard this so it can't quietly revert to serial.
|
||||
|
||||
## Caveats
|
||||
|
||||
- **`resolve` verdicts are PASS/INFO only.** The correctness table in `fixtures.py`
|
||||
encodes expected slugs against the live vault; until confirmed, mismatches report
|
||||
`INFO` (tune-me), not `FAIL`. Promote to FAIL once the table is verified.
|
||||
- **`queue` times the enqueue path only.** A true flush-replay needs a throwaway
|
||||
sandbox vault so the replay doesn't write to production; that's deferred (the
|
||||
plan's synthetic-vault item). The metric still proves enqueue works and reports
|
||||
queue depth.
|
||||
- **Absolute ms depend on where you run from.** WAN latency to the configured
|
||||
endpoint dominates single-request times; that's why gates are ratios.
|
||||
- **Synthetic-vault scaling (100/500/1000 notes) is deferred** per decision — the
|
||||
suite benchmarks the real ~190-note vault today.
|
||||
@@ -0,0 +1,57 @@
|
||||
{
|
||||
"_comment": "Relative-ratio regression gates (the maintainer's chosen policy). Ratios are portable across machines/WAN; absolute ms in the results JSON stay informational. Tune these after the first real baseline run, then tighten. 'field' is a dotted path into a metric's result; for list items use the index (e.g. sweep.3.median_ms).",
|
||||
"read-full": {
|
||||
"gates": [
|
||||
{ "field": "speedup_ratio", "op": ">=", "min": 1.8,
|
||||
"why": "concurrent read_many must beat serial by the 1.1.0 concurrency margin; <1.8 means concurrency regressed" }
|
||||
]
|
||||
},
|
||||
"bulk-get": {
|
||||
"gates": [
|
||||
{ "field": "speedup_ratio", "op": ">=", "min": 1.5,
|
||||
"why": "controlled-K concurrent vs serial; lower bar than full-vault since K is small" }
|
||||
]
|
||||
},
|
||||
"expand-graph": {
|
||||
"gates": [
|
||||
{ "field": "min_speedup", "op": ">=", "min": 2.0,
|
||||
"why": "recall()'s graph layer must fetch each BFS hop concurrently (read_many), not serially per node; <2x means expand_graph regressed to the pre-fix serial walk" },
|
||||
{ "field": "all_ranking_identical", "op": ">=", "min": 1,
|
||||
"why": "the concurrent expansion must return the same ranked neighbours as the serial reference" },
|
||||
{ "field": "all_scores_identical", "op": ">=", "min": 1,
|
||||
"why": "decayed scores must match the serial reference to within float tolerance" }
|
||||
]
|
||||
},
|
||||
"pool-warmup": {
|
||||
"gates": [
|
||||
{ "field": "cold_over_warm_ratio", "op": ">=", "min": 1.5,
|
||||
"why": "cold request must be meaningfully slower than warm — proves keep-alive is reusing the connection; a ratio near 1.0 means every request is re-handshaking (the pre-1.1.0 bug)" }
|
||||
]
|
||||
},
|
||||
"bulk-append": {
|
||||
"gates": [
|
||||
{ "field": "all_skipped_second_pass", "op": ">=", "min": 1,
|
||||
"why": "second pass of identical lines must be 100% idempotent skips (boolean true coerced to 1)" }
|
||||
]
|
||||
},
|
||||
"soak": {
|
||||
"gates": [
|
||||
{ "field": "errors", "op": "<=", "min": 0,
|
||||
"why": "no read errors across the soak window — connection leaks/pool exhaustion would surface here" },
|
||||
{ "field": "drift_ratio_late_over_early", "op": "<=", "min": 1.5,
|
||||
"why": "late passes must not be much slower than early ones; >1.5 suggests a leak or degrading pool" }
|
||||
]
|
||||
},
|
||||
"lock": {
|
||||
"gates": [
|
||||
{ "field": "ok", "op": ">=", "min": 1,
|
||||
"why": "free acquire returns 0 and a contended acquire fast-fails with exit 75" }
|
||||
]
|
||||
},
|
||||
"cache": {
|
||||
"gates": [
|
||||
{ "field": "deduped", "op": ">=", "min": 1,
|
||||
"why": "read_many must collapse a 3x-duplicated path list to the unique set" }
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,800 @@
|
||||
#!/usr/bin/env python3
|
||||
"""bench.py — CHORUS performance test suite (timing harness).
|
||||
|
||||
Measures the operations the 1.1.0 (pooling + concurrent read_many + shared cache)
|
||||
and 1.2.0 (fuzzy resolve / alias learning) updates targeted, and gates them on
|
||||
RELATIVE ratios (portable across machines/WAN) rather than absolute ms.
|
||||
|
||||
It reuses the live `chorus` client module so it times the REAL pooled/concurrent
|
||||
code path. Reads are non-destructive; write metrics live in a disposable
|
||||
namespace (_agent/_bench/<run-id>/) cleaned up at the end unless --keep.
|
||||
|
||||
This is a MANUAL / pre-release tool, not a CI gate — it depends on the live
|
||||
configured endpoint (chorus.BASE / $CHORUS_BASE) and WAN latency.
|
||||
|
||||
python3 bench.py <metric> [options]
|
||||
python3 bench.py all # the read-only safe set
|
||||
python3 bench.py read-full -n 10
|
||||
python3 bench.py subject-pull
|
||||
python3 bench.py bulk-put -k 50
|
||||
python3 bench.py worker-sweep
|
||||
python3 bench.py resolve # 1.2.0 fuzzy correctness + timing
|
||||
python3 bench.py --list
|
||||
|
||||
Metrics: read-full, subject-pull, bulk-get, bulk-put, bulk-append,
|
||||
pool-warmup, worker-sweep, resolve, index, lint, lock, queue, cache, soak
|
||||
|
||||
Options:
|
||||
-n / --iterations N timed iterations (default 10 read / 5 write)
|
||||
-k / --count K notes/appends for bulk metrics (default 50)
|
||||
--workers W override CHORUS_WORKERS for this run
|
||||
--warmup M warm-up iterations to discard (default 2)
|
||||
--soak-seconds S duration for the soak metric (default 60)
|
||||
--keep do NOT delete the _bench namespace after the run
|
||||
--no-gate skip relative-ratio gating (report numbers only)
|
||||
--json-out PATH write the results JSON here (default results/<date>-<commit>.json)
|
||||
--quiet suppress the per-metric human table (JSON only)
|
||||
|
||||
Exit: 0 all gates pass (or --no-gate) · 1 a gate failed · 2 usage/setup error.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import datetime as dt
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
SCRIPTS = HERE.parent.parent / "scripts"
|
||||
sys.path.insert(0, str(SCRIPTS))
|
||||
|
||||
import chorus # noqa: E402 the validated client — real pooled/concurrent path
|
||||
import chorus_index # noqa: E402 resolve / fuzzy_candidates (1.2.0)
|
||||
import chorus_recall # noqa: E402 recall (search + 1-hop expansion)
|
||||
|
||||
BASELINES = HERE / "baselines.json"
|
||||
RESULTS_DIR = HERE / "results"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# stats + timing
|
||||
# ---------------------------------------------------------------------------
|
||||
def stats(samples: list[float]) -> dict:
|
||||
"""min/median/p90/p95/max/mean in milliseconds from a list of seconds."""
|
||||
if not samples:
|
||||
return {}
|
||||
ms = sorted(s * 1000.0 for s in samples)
|
||||
|
||||
def pct(p: float) -> float:
|
||||
if len(ms) == 1:
|
||||
return ms[0]
|
||||
k = (len(ms) - 1) * p
|
||||
lo, hi = int(k), min(int(k) + 1, len(ms) - 1)
|
||||
return ms[lo] + (ms[hi] - ms[lo]) * (k - lo)
|
||||
|
||||
return {
|
||||
"n": len(ms),
|
||||
"min_ms": round(ms[0], 2),
|
||||
"median_ms": round(statistics.median(ms), 2),
|
||||
"p90_ms": round(pct(0.90), 2),
|
||||
"p95_ms": round(pct(0.95), 2),
|
||||
"max_ms": round(ms[-1], 2),
|
||||
"mean_ms": round(statistics.fmean(ms), 2),
|
||||
}
|
||||
|
||||
|
||||
def timed(fn, n: int, warmup: int) -> dict:
|
||||
"""Run fn() warmup+n times. Returns warm-up samples separately from the timed
|
||||
set so cold-start cost (the thing keep-alive kills) is visible, not averaged in."""
|
||||
warm: list[float] = []
|
||||
samples: list[float] = []
|
||||
for i in range(warmup + n):
|
||||
t0 = time.perf_counter()
|
||||
fn()
|
||||
dt_s = time.perf_counter() - t0
|
||||
(warm if i < warmup else samples).append(dt_s)
|
||||
out = stats(samples)
|
||||
if warm:
|
||||
out["warmup"] = stats(warm)
|
||||
return out
|
||||
|
||||
|
||||
def silent(fn):
|
||||
"""Run fn() with stdout swallowed (recall/resolve print a lot)."""
|
||||
def wrapped():
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
return fn()
|
||||
return wrapped
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# vault enumeration (mirrors sweep.py / vault_lint.py)
|
||||
# ---------------------------------------------------------------------------
|
||||
SKIP_BASENAMES = {"README.md"}
|
||||
|
||||
|
||||
def _list_dir(path: str):
|
||||
p = "" if path in ("", "/") else (path if path.endswith("/") else path + "/")
|
||||
body = chorus.get_text(p)
|
||||
if body is None:
|
||||
return [], []
|
||||
try:
|
||||
j = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
return [], []
|
||||
entries = list(j.get("files", [])) + list(j.get("folders", []))
|
||||
files = [e for e in entries if not e.endswith("/")]
|
||||
folders = [e[:-1] for e in entries if e.endswith("/")]
|
||||
return files, folders
|
||||
|
||||
|
||||
def walk_vault(prefix: str = ""):
|
||||
files, folders = _list_dir(prefix)
|
||||
for f in files:
|
||||
yield prefix + f
|
||||
for d in folders:
|
||||
yield from walk_vault(f"{prefix}{d}/")
|
||||
|
||||
|
||||
def all_notes() -> list[str]:
|
||||
return [p for p in walk_vault()
|
||||
if p.endswith(".md") and p.rsplit("/", 1)[-1] not in SKIP_BASENAMES]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# write primitives (replicate chorus.py semantics so we time the real cost)
|
||||
# ---------------------------------------------------------------------------
|
||||
def put_verified(path: str, body: str) -> None:
|
||||
"""PUT then read-back GET — the same verify chorus.py cmd_put does."""
|
||||
status, b = chorus.request("PUT", chorus.vault_url(path),
|
||||
data=body.encode(), headers={"Content-Type": "text/markdown"})
|
||||
chorus.check(status, b, f"put {path}")
|
||||
status, _ = chorus.request("GET", chorus.vault_url(path))
|
||||
if status != 200:
|
||||
raise chorus.ChorusError(f"put {path}: did not verify (GET {status})")
|
||||
|
||||
|
||||
def append_line(path: str, line: str) -> bool:
|
||||
"""Whole-line idempotent append (mirrors chorus.py cmd_append). Returns True if a
|
||||
write happened, False if it was an idempotent skip."""
|
||||
status, body = chorus.request("GET", chorus.vault_url(path))
|
||||
if status == 200:
|
||||
existing = [ln.rstrip("\r") for ln in body.decode(errors="replace").splitlines()]
|
||||
if line.rstrip("\r") in existing:
|
||||
return False
|
||||
status, body = chorus.request("POST", chorus.vault_url(path),
|
||||
data=f"{line}\n".encode(),
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
chorus.check(status, body, f"append {path}")
|
||||
return True
|
||||
|
||||
|
||||
def delete_path(path: str) -> None:
|
||||
chorus.request("DELETE", chorus.vault_url(path))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# metrics
|
||||
# ---------------------------------------------------------------------------
|
||||
def m_read_full(ctx) -> dict:
|
||||
"""§3.1 full-vault read: serial GET vs concurrent read_many -> speedup ratio."""
|
||||
paths = all_notes()
|
||||
if not paths:
|
||||
raise chorus.ChorusError("read-full: no notes enumerated (vault unreachable?)")
|
||||
|
||||
def serial():
|
||||
for p in paths:
|
||||
chorus.get_text(p)
|
||||
|
||||
def concurrent():
|
||||
chorus.read_many(paths)
|
||||
|
||||
# one warm pass so neither side pays the cold-handshake tax (pool-warmup owns that)
|
||||
chorus.get_text(paths[0])
|
||||
# serial is a slow reference baseline (N serial round-trips); one warm pass is
|
||||
# enough to establish the ratio, so we don't multiply the full-vault serial cost.
|
||||
serial_t = timed(serial, n=1, warmup=0)
|
||||
concurrent_t = timed(concurrent, n=ctx.n, warmup=ctx.warmup)
|
||||
ratio = (serial_t["median_ms"] / concurrent_t["median_ms"]) if concurrent_t.get("median_ms") else None
|
||||
return {
|
||||
"note_count": len(paths),
|
||||
"workers": chorus.MAX_WORKERS,
|
||||
"serial": serial_t,
|
||||
"concurrent": concurrent_t,
|
||||
"speedup_ratio": round(ratio, 2) if ratio else None,
|
||||
"notes_per_sec_concurrent": round(len(paths) / (concurrent_t["median_ms"] / 1000.0), 1)
|
||||
if concurrent_t.get("median_ms") else None,
|
||||
}
|
||||
|
||||
|
||||
def m_subject_pull(ctx) -> dict:
|
||||
"""§3.2 recall a subject: time resolve (floor), search, and full recall, split out."""
|
||||
import fixtures
|
||||
index = chorus_index.load()
|
||||
out = {"subjects": []}
|
||||
for label, query in fixtures.SUBJECTS:
|
||||
# floor: O(1) index resolve
|
||||
resolve_t = timed(lambda: chorus_index.resolve(index, query), n=ctx.n, warmup=ctx.warmup)
|
||||
# search-only layer
|
||||
def search():
|
||||
with contextlib.redirect_stdout(io.StringIO()):
|
||||
chorus.request("POST", f"{chorus.BASE}/search/simple/?query={query}")
|
||||
search_t = timed(search, n=ctx.n, warmup=ctx.warmup)
|
||||
# full recall (search + 1-hop expansion reads)
|
||||
recall_t = timed(silent(lambda: chorus_recall.recall(query)), n=ctx.n, warmup=ctx.warmup)
|
||||
out["subjects"].append({
|
||||
"label": label, "query": query,
|
||||
"resolve_floor": resolve_t,
|
||||
"search_only": search_t,
|
||||
"recall_total": recall_t,
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
def m_bulk_get(ctx) -> dict:
|
||||
"""§3.3 bulk GET at a controlled K: serial vs read_many over fixture notes."""
|
||||
import fixtures
|
||||
paths = _seed_notes(ctx, fixtures)
|
||||
|
||||
def serial():
|
||||
for p in paths:
|
||||
chorus.get_text(p)
|
||||
|
||||
def concurrent():
|
||||
chorus.read_many(paths)
|
||||
|
||||
serial_t = timed(serial, n=ctx.n, warmup=1)
|
||||
concurrent_t = timed(concurrent, n=ctx.n, warmup=ctx.warmup)
|
||||
ratio = (serial_t["median_ms"] / concurrent_t["median_ms"]) if concurrent_t.get("median_ms") else None
|
||||
return {"k": len(paths), "workers": chorus.MAX_WORKERS,
|
||||
"serial": serial_t, "concurrent": concurrent_t,
|
||||
"speedup_ratio": round(ratio, 2) if ratio else None}
|
||||
|
||||
|
||||
def m_bulk_put(ctx) -> dict:
|
||||
"""§3.3 bulk PUT (read-back verified) -> puts/sec + per-put percentiles."""
|
||||
import fixtures
|
||||
k = ctx.k
|
||||
samples: list[float] = []
|
||||
for i in range(k):
|
||||
body = fixtures.note_body(ctx.run_id, i)
|
||||
path = fixtures.note_path(ctx.run_id, i)
|
||||
t0 = time.perf_counter()
|
||||
put_verified(path, body)
|
||||
samples.append(time.perf_counter() - t0)
|
||||
s = stats(samples)
|
||||
total_s = sum(samples)
|
||||
return {"k": k, "per_put": s, "total_ms": round(total_s * 1000, 1),
|
||||
"puts_per_sec": round(k / total_s, 1) if total_s else None}
|
||||
|
||||
|
||||
def m_bulk_append(ctx) -> dict:
|
||||
"""§3.3 bulk APPEND + idempotent-skip path. Confirms idempotency isn't O(file)."""
|
||||
import fixtures
|
||||
target = fixtures.append_target(ctx.run_id)
|
||||
put_verified(target, fixtures.append_target_seed(ctx.run_id))
|
||||
k = ctx.k
|
||||
lines = [f"- 2026-06-23: bench append line {i:04d}" for i in range(k)]
|
||||
|
||||
write_samples: list[float] = []
|
||||
for ln in lines:
|
||||
t0 = time.perf_counter()
|
||||
wrote = append_line(target, ln)
|
||||
write_samples.append(time.perf_counter() - t0)
|
||||
if not wrote:
|
||||
raise chorus.ChorusError("bulk-append: expected a write but got an idempotent skip")
|
||||
|
||||
# second pass: every line already present -> must skip, and skip cost is the
|
||||
# idempotency GET on a now-larger file. Confirms no superlinear blowup.
|
||||
skip_samples: list[float] = []
|
||||
skips = 0
|
||||
for ln in lines:
|
||||
t0 = time.perf_counter()
|
||||
wrote = append_line(target, ln)
|
||||
skip_samples.append(time.perf_counter() - t0)
|
||||
if not wrote:
|
||||
skips += 1
|
||||
|
||||
return {
|
||||
"k": k,
|
||||
"append_write": stats(write_samples),
|
||||
"append_skip": stats(skip_samples),
|
||||
"skips_confirmed": skips,
|
||||
"all_skipped_second_pass": skips == k,
|
||||
}
|
||||
|
||||
|
||||
def m_pool_warmup(ctx) -> dict:
|
||||
"""§4.1 cold vs warm single GET — the clearest proof keep-alive is alive."""
|
||||
paths = all_notes()
|
||||
probe = paths[0] if paths else "_agent/chorus-vault.md"
|
||||
chorus._drop_connection() # force a cold handshake
|
||||
t0 = time.perf_counter()
|
||||
chorus.get_text(probe)
|
||||
cold_ms = (time.perf_counter() - t0) * 1000.0
|
||||
warm = timed(lambda: chorus.get_text(probe), n=max(ctx.n, 10), warmup=1)
|
||||
ratio = round(cold_ms / warm["median_ms"], 2) if warm.get("median_ms") else None
|
||||
return {"probe": probe, "cold_ms": round(cold_ms, 2), "warm": warm,
|
||||
"cold_over_warm_ratio": ratio}
|
||||
|
||||
|
||||
def m_worker_sweep(ctx) -> dict:
|
||||
"""§4.2 concurrency scaling: full-vault concurrent read at workers 1,2,4,8,16."""
|
||||
paths = all_notes()
|
||||
if not paths:
|
||||
raise chorus.ChorusError("worker-sweep: no notes enumerated")
|
||||
original = chorus.MAX_WORKERS
|
||||
rows = []
|
||||
try:
|
||||
# w=1 is a full serial pass (slow); the curve only needs the shape, so use a
|
||||
# modest sample count and a single pre-warm rather than per-setting warmups.
|
||||
reps = max(1, ctx.n // 2)
|
||||
for w in (1, 2, 4, 8, 16):
|
||||
chorus.MAX_WORKERS = w
|
||||
chorus.read_many(paths[:5]) # warm the pool at this worker count
|
||||
t = timed(lambda: chorus.read_many(paths), n=reps, warmup=0)
|
||||
rows.append({"workers": w, "median_ms": t["median_ms"], "p95_ms": t["p95_ms"]})
|
||||
finally:
|
||||
chorus.MAX_WORKERS = original
|
||||
best = min(rows, key=lambda r: r["median_ms"])
|
||||
return {"note_count": len(paths), "sweep": rows, "knee_workers": best["workers"],
|
||||
"default_workers": original}
|
||||
|
||||
|
||||
def m_resolve(ctx) -> dict:
|
||||
"""§4.3 fuzzy resolve — timing AND correctness against the 1.2.0 guard table."""
|
||||
import fixtures
|
||||
index = chorus_index.load()
|
||||
cases = []
|
||||
passes = info = 0
|
||||
for case in fixtures.RESOLVE_CASES:
|
||||
mention = case["mention"]
|
||||
exact_t = timed(lambda: chorus_index.resolve(index, mention), n=ctx.n, warmup=ctx.warmup)
|
||||
fuzzy_t = timed(lambda: chorus_index.fuzzy_candidates(index, mention), n=ctx.n, warmup=ctx.warmup)
|
||||
_, entity = chorus_index.resolve(index, mention)
|
||||
exact_slug = (entity or {}).get("slug") or _slug_from_entity(entity)
|
||||
cands = chorus_index.fuzzy_candidates(index, mention) or []
|
||||
cand_slugs = [_cand_slug(c) for c in cands]
|
||||
|
||||
expect, want = case["expect"], case.get("expect_slug")
|
||||
if expect == "exact":
|
||||
ok = exact_slug == want
|
||||
elif expect == "candidates":
|
||||
ok = (exact_slug != want) and (want in cand_slugs)
|
||||
else: # miss
|
||||
ok = entity is None
|
||||
verdict = "PASS" if ok else "INFO" # tune table to live vault before promoting to FAIL
|
||||
passes += ok
|
||||
info += (not ok)
|
||||
cases.append({"mention": mention, "expect": expect, "expect_slug": want,
|
||||
"resolved_slug": exact_slug, "candidate_slugs": cand_slugs[:5],
|
||||
"verdict": verdict,
|
||||
"resolve_timing": exact_t, "fuzzy_timing": fuzzy_t})
|
||||
return {"cases": cases, "passes": passes, "needs_tuning": info,
|
||||
"note": "verdicts are PASS/INFO only; promote to FAIL once the table is "
|
||||
"confirmed against the live vault"}
|
||||
|
||||
|
||||
def _serial_expand(seeds, nmap, base, max_hops):
|
||||
"""The pre-fix serial graph BFS, kept here as the reference baseline so the suite
|
||||
both (a) measures the concurrency speedup and (b) asserts the concurrent
|
||||
expand_graph still returns byte-identical ranking. If chorus_recall.expand_graph ever
|
||||
regresses to serial, the ratio gate fails; if its results drift, the parity check fails."""
|
||||
from collections import deque
|
||||
score_of = dict(base)
|
||||
seen = set(seeds)
|
||||
results = {}
|
||||
dq = deque((s, 0) for s in seeds)
|
||||
while dq:
|
||||
path, hop = dq.popleft()
|
||||
if hop >= max_hops:
|
||||
continue
|
||||
text = chorus_recall.links.get_text(path)
|
||||
if text is None:
|
||||
continue
|
||||
body = chorus_recall.strip_frontmatter(text)
|
||||
targets = set(chorus_recall.links.all_wikilinks(body)) | set(chorus_recall.links.source_notes(text))
|
||||
parent = score_of.get(path, 1.0)
|
||||
for t in targets:
|
||||
tp = chorus_recall._resolve_target(t, nmap)
|
||||
if not tp or tp in seeds:
|
||||
continue
|
||||
decayed = parent * (chorus_recall.GRAPH_DECAY ** (hop + 1))
|
||||
if decayed > results.get(tp, (0.0, ""))[0]:
|
||||
results[tp] = (decayed, path)
|
||||
if tp not in seen:
|
||||
seen.add(tp)
|
||||
dq.append((tp, hop + 1))
|
||||
return sorted(results.items(), key=lambda kv: -kv[1][0])
|
||||
|
||||
|
||||
def m_expand_graph(ctx) -> dict:
|
||||
"""recall() graph layer: serial vs concurrent neighbourhood expansion. This is the
|
||||
metric that pins the real recall() bottleneck (the graph BFS, not BM25) and guards
|
||||
the read_many concurrency fix. Also asserts the concurrent path's ranking + scores
|
||||
match the serial reference exactly."""
|
||||
import fixtures
|
||||
ix = chorus_recall.load_index()
|
||||
index = chorus_index.load()
|
||||
nmap = chorus_index.name_map(index)
|
||||
max_hops = chorus_recall.MAX_HOPS
|
||||
# hub subjects stress expansion most; fall back to all subjects if none labelled hub
|
||||
subjects = [s for s in fixtures.SUBJECTS if "hub" in s[0]] or fixtures.SUBJECTS
|
||||
rows = []
|
||||
for label, query in subjects:
|
||||
hits = ix.score(query, limit=8)
|
||||
base = {p: s for p, s in hits}
|
||||
seeds = [p for p, _ in hits]
|
||||
if not seeds:
|
||||
continue
|
||||
chorus_recall.expand_graph(seeds, nmap, base, max_hops) # warm
|
||||
serial_t = timed(lambda: _serial_expand(seeds, nmap, base, max_hops), n=1, warmup=0)
|
||||
conc_t = timed(lambda: chorus_recall.expand_graph(seeds, nmap, base, max_hops),
|
||||
n=ctx.n, warmup=1)
|
||||
old = _serial_expand(seeds, nmap, base, max_hops)
|
||||
new = chorus_recall.expand_graph(seeds, nmap, base, max_hops)
|
||||
ranking_ok = [p for p, _ in old] == [p for p, _ in new]
|
||||
scores_ok = ranking_ok and all(
|
||||
abs(old[i][1][0] - new[i][1][0]) < 1e-9 for i in range(len(old)))
|
||||
ratio = (serial_t["median_ms"] / conc_t["median_ms"]) if conc_t.get("median_ms") else None
|
||||
rows.append({"label": label, "query": query, "neighbours": len(new),
|
||||
"serial_ms": serial_t["median_ms"], "concurrent_ms": conc_t["median_ms"],
|
||||
"speedup": round(ratio, 2) if ratio else None,
|
||||
"ranking_identical": ranking_ok, "scores_identical": scores_ok})
|
||||
speedups = [r["speedup"] for r in rows if r["speedup"]]
|
||||
return {"subjects": rows,
|
||||
"min_speedup": round(min(speedups), 2) if speedups else None,
|
||||
"all_ranking_identical": all(r["ranking_identical"] for r in rows),
|
||||
"all_scores_identical": all(r["scores_identical"] for r in rows)}
|
||||
|
||||
|
||||
def _slug_from_entity(entity):
|
||||
if not entity:
|
||||
return None
|
||||
path = entity.get("path", "")
|
||||
return path.rsplit("/", 1)[-1].removesuffix(".md") if path else None
|
||||
|
||||
|
||||
def _cand_slug(c):
|
||||
if isinstance(c, dict):
|
||||
return c.get("slug") or _slug_from_entity(c)
|
||||
if isinstance(c, (list, tuple)) and c:
|
||||
return c[0]
|
||||
return str(c)
|
||||
|
||||
|
||||
def m_index(ctx) -> dict:
|
||||
"""§4.4 entity-index rebuild + link symmetrize timing (sweep.py, dry-run = read-only)."""
|
||||
return _time_script("sweep.py", [])
|
||||
|
||||
|
||||
def m_lint(ctx) -> dict:
|
||||
"""§4.5 full-vault lint timing (vault_lint.py) — regression-gate vs 1.1.0 baseline."""
|
||||
return _time_script("vault_lint.py", [], allow_exit={0, 1})
|
||||
|
||||
|
||||
def _time_script(name: str, extra: list[str], allow_exit=frozenset({0})) -> dict:
|
||||
script = SCRIPTS / name
|
||||
env = dict(os.environ, CHORUS_KEY_LEGACY_OK="1")
|
||||
t0 = time.perf_counter()
|
||||
proc = subprocess.run([sys.executable, str(script), *extra],
|
||||
capture_output=True, text=True, env=env, timeout=120)
|
||||
elapsed = (time.perf_counter() - t0) * 1000.0
|
||||
return {"script": name, "elapsed_ms": round(elapsed, 1),
|
||||
"exit_code": proc.returncode,
|
||||
"ok": proc.returncode in allow_exit,
|
||||
"stderr_tail": proc.stderr.strip().splitlines()[-3:] if proc.stderr else []}
|
||||
|
||||
|
||||
def m_lock(ctx) -> dict:
|
||||
"""§4.6 advisory-lock timing: acquire-free, contended (held-fresh), release."""
|
||||
owner = f"bench-{ctx.run_id}"
|
||||
other = f"bench-other-{ctx.run_id}"
|
||||
|
||||
def acquire(o):
|
||||
return chorus.cmd_lock(o, quiet=True)
|
||||
|
||||
def release(o):
|
||||
return chorus.cmd_unlock(o, quiet=True)
|
||||
|
||||
release(owner); release(other) # clean slate
|
||||
t0 = time.perf_counter(); rc_free = acquire(owner); free_ms = (time.perf_counter() - t0) * 1000
|
||||
t0 = time.perf_counter(); rc_held = acquire(other); held_ms = (time.perf_counter() - t0) * 1000
|
||||
t0 = time.perf_counter(); release(owner); rel_ms = (time.perf_counter() - t0) * 1000
|
||||
return {"acquire_free_ms": round(free_ms, 2), "acquire_free_rc": rc_free,
|
||||
"contended_ms": round(held_ms, 2), "contended_rc_expected_75": rc_held,
|
||||
"release_ms": round(rel_ms, 2),
|
||||
"ok": rc_free == 0 and rc_held == 75}
|
||||
|
||||
|
||||
def m_queue(ctx) -> dict:
|
||||
"""§4.7 offline write-ahead queue: enqueue K writes against an unreachable base,
|
||||
then time the flush replay on 'reconnect'. Uses a bogus base so nothing real is
|
||||
written; restores the base afterward. No production writes occur."""
|
||||
import importlib
|
||||
import chorus_queue
|
||||
k = min(ctx.k, 20)
|
||||
saved_base = chorus.BASE
|
||||
enqueue_samples: list[float] = []
|
||||
try:
|
||||
for i in range(k):
|
||||
url = f"{chorus.BASE}/vault/{fixtures_qpath(ctx.run_id, i)}"
|
||||
t0 = time.perf_counter()
|
||||
chorus_queue.enqueue("PUT", url, b"x", {"Content-Type": "text/markdown"},
|
||||
idem_key=f"bench-{ctx.run_id}-{i}")
|
||||
enqueue_samples.append(time.perf_counter() - t0)
|
||||
pending_before = len(chorus_queue.pending())
|
||||
# flush is left UNMEASURED-as-success here: with a real endpoint it would
|
||||
# try to write. We only assert enqueue worked and report depth; a true
|
||||
# flush-replay timing needs a sandbox vault (see README "queue caveat").
|
||||
return {"k": k, "enqueue": stats(enqueue_samples),
|
||||
"pending_after_enqueue": pending_before,
|
||||
"note": "flush replay intentionally not run against the live vault; "
|
||||
"enqueue path timed only. See README queue caveat."}
|
||||
finally:
|
||||
chorus.BASE = saved_base
|
||||
importlib.reload(chorus_queue)
|
||||
|
||||
|
||||
def fixtures_qpath(run_id: str, i: int) -> str:
|
||||
return f"_agent/_bench/{run_id}/queued-{i:04d}.md"
|
||||
|
||||
|
||||
def m_cache(ctx) -> dict:
|
||||
"""§4.8 read_many dedup check: a path list with duplicates must collapse to the
|
||||
unique set (the cache/dedup that stops a sweep re-fetching a link target)."""
|
||||
paths = all_notes()[:20] or ["_agent/chorus-vault.md"]
|
||||
dupd = paths + paths + paths # 3x
|
||||
t0 = time.perf_counter()
|
||||
result = chorus.read_many(dupd)
|
||||
elapsed = (time.perf_counter() - t0) * 1000
|
||||
return {"requested": len(dupd), "unique": len(set(dupd)),
|
||||
"returned": len(result), "deduped": len(result) == len(set(dupd)),
|
||||
"elapsed_ms": round(elapsed, 2)}
|
||||
|
||||
|
||||
def m_soak(ctx) -> dict:
|
||||
"""§4.10 stability: loop full-vault reads for N seconds; watch for errors/drift."""
|
||||
paths = all_notes()
|
||||
if not paths:
|
||||
raise chorus.ChorusError("soak: no notes enumerated")
|
||||
deadline = time.time() + ctx.soak_seconds
|
||||
durations: list[float] = []
|
||||
errors = 0
|
||||
while time.time() < deadline:
|
||||
t0 = time.perf_counter()
|
||||
try:
|
||||
res = chorus.read_many(paths)
|
||||
if any(v is None for v in res.values()):
|
||||
errors += 1
|
||||
except Exception:
|
||||
errors += 1
|
||||
durations.append(time.perf_counter() - t0)
|
||||
s = stats(durations)
|
||||
# drift = late iterations slower than early ones (leak / pool exhaustion signal)
|
||||
half = len(durations) // 2 or 1
|
||||
early = statistics.fmean(durations[:half]) * 1000
|
||||
late = statistics.fmean(durations[half:]) * 1000
|
||||
return {"seconds": ctx.soak_seconds, "iterations": len(durations), "errors": errors,
|
||||
"per_pass": s, "early_mean_ms": round(early, 2), "late_mean_ms": round(late, 2),
|
||||
"drift_ratio_late_over_early": round(late / early, 2) if early else None}
|
||||
|
||||
|
||||
METRICS = {
|
||||
"read-full": m_read_full,
|
||||
"subject-pull": m_subject_pull,
|
||||
"bulk-get": m_bulk_get,
|
||||
"bulk-put": m_bulk_put,
|
||||
"bulk-append": m_bulk_append,
|
||||
"pool-warmup": m_pool_warmup,
|
||||
"worker-sweep": m_worker_sweep,
|
||||
"resolve": m_resolve,
|
||||
"expand-graph": m_expand_graph,
|
||||
"index": m_index,
|
||||
"lint": m_lint,
|
||||
"lock": m_lock,
|
||||
"queue": m_queue,
|
||||
"cache": m_cache,
|
||||
"soak": m_soak,
|
||||
}
|
||||
|
||||
# the read-only safe set run by `all` (no writes to the vault)
|
||||
READONLY = ["pool-warmup", "read-full", "worker-sweep", "subject-pull",
|
||||
"resolve", "expand-graph", "index", "lint", "cache"]
|
||||
WRITES = {"bulk-get", "bulk-put", "bulk-append", "lock"} # touch the _bench namespace
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# fixture seeding / cleanup
|
||||
# ---------------------------------------------------------------------------
|
||||
def _seed_notes(ctx, fixtures) -> list[str]:
|
||||
"""Ensure K bench notes exist; return their paths. Cached on ctx so bulk-get
|
||||
and bulk-put can share the seed within one run."""
|
||||
if getattr(ctx, "_seeded", None):
|
||||
return ctx._seeded
|
||||
paths = []
|
||||
for i in range(ctx.k):
|
||||
p = fixtures.note_path(ctx.run_id, i)
|
||||
put_verified(p, fixtures.note_body(ctx.run_id, i))
|
||||
paths.append(p)
|
||||
ctx._seeded = paths
|
||||
return paths
|
||||
|
||||
|
||||
def cleanup_namespace(run_id: str) -> int:
|
||||
"""Delete every file under _agent/_bench/<run-id>/. Returns count deleted."""
|
||||
prefix = f"_agent/_bench/{run_id}/"
|
||||
victims = [p for p in walk_vault(prefix)]
|
||||
for p in victims:
|
||||
delete_path(p)
|
||||
# remove the (now-empty) run dir marker if the API created one — best effort
|
||||
return len(victims)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# gating
|
||||
# ---------------------------------------------------------------------------
|
||||
def load_baselines() -> dict:
|
||||
if BASELINES.exists():
|
||||
return json.loads(BASELINES.read_text())
|
||||
return {}
|
||||
|
||||
|
||||
def gate(metric: str, result: dict, baselines: dict) -> list[dict]:
|
||||
"""Apply relative-ratio gates from baselines.json. Returns a list of checks."""
|
||||
rules = baselines.get(metric, {})
|
||||
checks = []
|
||||
for rule in rules.get("gates", []):
|
||||
path, op, threshold = rule["field"], rule["op"], rule["min"]
|
||||
val = _dig(result, path)
|
||||
if val is None:
|
||||
checks.append({"rule": rule, "value": None, "pass": None, "reason": "field absent"})
|
||||
continue
|
||||
ok = (val >= threshold) if op == ">=" else (val <= threshold)
|
||||
checks.append({"rule": rule, "value": val, "pass": ok})
|
||||
return checks
|
||||
|
||||
|
||||
def _dig(obj, dotted: str):
|
||||
cur = obj
|
||||
for part in dotted.split("."):
|
||||
if isinstance(cur, list):
|
||||
try:
|
||||
cur = cur[int(part)]
|
||||
continue
|
||||
except (ValueError, IndexError):
|
||||
return None
|
||||
if not isinstance(cur, dict) or part not in cur:
|
||||
return None
|
||||
cur = cur[part]
|
||||
return cur
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# main
|
||||
# ---------------------------------------------------------------------------
|
||||
class Ctx:
|
||||
pass
|
||||
|
||||
|
||||
def env_block(args) -> dict:
|
||||
commit = "unknown"
|
||||
try:
|
||||
commit = subprocess.run(["git", "rev-parse", "--short", "HEAD"],
|
||||
cwd=str(SCRIPTS), capture_output=True, text=True,
|
||||
timeout=10).stdout.strip() or "unknown"
|
||||
except Exception:
|
||||
pass
|
||||
return {
|
||||
"date": dt.datetime.now().isoformat(timespec="seconds"),
|
||||
"endpoint": chorus.BASE,
|
||||
"commit": commit,
|
||||
"chorus_workers": chorus.MAX_WORKERS,
|
||||
"chorus_timeout": chorus.TIMEOUT,
|
||||
"python": sys.version.split()[0],
|
||||
"iterations": args.iterations,
|
||||
"count_k": args.count,
|
||||
}
|
||||
|
||||
|
||||
def human_table(name: str, result: dict, checks: list[dict]) -> str:
|
||||
lines = [f"\n=== {name} ==="]
|
||||
lines.append(json.dumps(result, indent=2))
|
||||
for c in checks:
|
||||
mark = "PASS" if c["pass"] else ("—" if c["pass"] is None else "FAIL")
|
||||
lines.append(f" [{mark}] {c['rule']['field']} {c['rule']['op']} {c['rule']['min']} (got {c['value']})")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(description="CHORUS performance test suite (manual/pre-release)")
|
||||
ap.add_argument("metric", nargs="?", help="metric name, or 'all' for the read-only set")
|
||||
ap.add_argument("--list", action="store_true", help="list metrics and exit")
|
||||
ap.add_argument("-n", "--iterations", type=int, default=None)
|
||||
ap.add_argument("-k", "--count", type=int, default=50)
|
||||
ap.add_argument("--workers", type=int, default=None)
|
||||
ap.add_argument("--warmup", type=int, default=2)
|
||||
ap.add_argument("--soak-seconds", type=int, default=60)
|
||||
ap.add_argument("--keep", action="store_true", help="do not delete the _bench namespace")
|
||||
ap.add_argument("--no-gate", action="store_true")
|
||||
ap.add_argument("--json-out", default=None)
|
||||
ap.add_argument("--quiet", action="store_true")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
if args.list or not args.metric:
|
||||
print("Metrics:", ", ".join(METRICS))
|
||||
print("Groups : all (read-only safe set:", ", ".join(READONLY) + ")")
|
||||
print("Writes to _bench namespace:", ", ".join(sorted(WRITES)))
|
||||
return 0 if args.list else 2
|
||||
|
||||
if args.workers is not None:
|
||||
chorus.MAX_WORKERS = args.workers
|
||||
|
||||
selected = READONLY if args.metric == "all" else [args.metric]
|
||||
for m in selected:
|
||||
if m not in METRICS:
|
||||
print(f"unknown metric: {m}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
ctx = Ctx()
|
||||
ctx.run_id = dt.datetime.now().strftime("%Y%m%d-%H%M%S") + "-" + uuid.uuid4().hex[:6]
|
||||
ctx.k = args.count
|
||||
ctx.warmup = args.warmup
|
||||
ctx.soak_seconds = args.soak_seconds
|
||||
ctx._seeded = None
|
||||
|
||||
baselines = load_baselines()
|
||||
need_writes = any(m in WRITES for m in selected)
|
||||
lock_owner = f"bench-{ctx.run_id}"
|
||||
if need_writes:
|
||||
if chorus.cmd_lock(lock_owner, quiet=True) == 75:
|
||||
print("bench: vault lock is held by another session — aborting write metrics.",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
|
||||
report = {"env": env_block(args), "run_id": ctx.run_id, "metrics": {}}
|
||||
any_fail = False
|
||||
try:
|
||||
for m in selected:
|
||||
ctx.n = args.iterations if args.iterations is not None else (5 if m in WRITES else 10)
|
||||
try:
|
||||
result = METRICS[m](ctx)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
result = {"error": str(exc)}
|
||||
any_fail = True
|
||||
checks = [] if args.no_gate else gate(m, result, baselines)
|
||||
if any(c["pass"] is False for c in checks):
|
||||
any_fail = True
|
||||
report["metrics"][m] = {"result": result, "gates": checks}
|
||||
if not args.quiet:
|
||||
print(human_table(m, result, checks))
|
||||
finally:
|
||||
if need_writes:
|
||||
chorus.cmd_unlock(lock_owner, quiet=True)
|
||||
if need_writes and not args.keep:
|
||||
n = cleanup_namespace(ctx.run_id)
|
||||
report["cleanup"] = {"deleted": n}
|
||||
if not args.quiet:
|
||||
print(f"\nbench: cleaned up {n} files under _agent/_bench/{ctx.run_id}/")
|
||||
elif need_writes and args.keep:
|
||||
report["cleanup"] = {"deleted": 0, "kept": f"_agent/_bench/{ctx.run_id}/"}
|
||||
|
||||
RESULTS_DIR.mkdir(parents=True, exist_ok=True)
|
||||
out_path = Path(args.json_out) if args.json_out else \
|
||||
RESULTS_DIR / f"{report['env']['date'][:10]}-{report['env']['commit']}.json"
|
||||
out_path.write_text(json.dumps(report, indent=2))
|
||||
print(f"\nbench: results -> {out_path}")
|
||||
return 1 if any_fail else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,103 @@
|
||||
#!/usr/bin/env python3
|
||||
"""fixtures.py — inputs for the CHORUS performance harness (bench.py).
|
||||
|
||||
Holds the *what* (subjects to pull, notes to write, mentions to resolve) so the
|
||||
harness (bench.py) stays the *how* (timing, stats, gating). Nothing here touches
|
||||
the network; bench.py owns all I/O.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# --- §3.2 subject pulls -------------------------------------------------------
|
||||
# (label, query) pairs fed to `recall`. Chosen to span neighbourhood sizes — the
|
||||
# real cost driver — from a likely-leaf to a known hub. APTA and CapMetro are the
|
||||
# subjects the maintainer configured; "chorus" and "operator preferences" are dense hubs
|
||||
# that stress the 1-hop expansion (the read_many fan-out 1.1.0 accelerated).
|
||||
SUBJECTS: list[tuple[str, str]] = [
|
||||
("apta", "APTA"),
|
||||
("capmetro", "CapMetro"),
|
||||
("hub-chorus", "chorus"),
|
||||
("hub-prefs", "operator preferences"),
|
||||
("leaf-rivnut", "rivnut torque spec"),
|
||||
]
|
||||
|
||||
|
||||
# --- §4.3 fuzzy-resolve correctness table (1.2.0) -----------------------------
|
||||
# Each case: mention, expected behaviour. `expect_slug` is the canonical slug an
|
||||
# exact/alias/fuzzy match should land on (None = no entity expected). `expect`
|
||||
# is the classifier:
|
||||
# "exact" -> resolve returns an entity whose slug == expect_slug
|
||||
# "candidates" -> no exact hit, but fuzzy_candidates surfaces expect_slug
|
||||
# (the 1.2.0 anti-duplicate guard — a near-miss must NOT resolve
|
||||
# to nothing and silently spawn a new note)
|
||||
# "miss" -> neither; genuinely unknown
|
||||
# Tune expect_slug to the live vault; unknowns are reported as INFO, not failures.
|
||||
# NOTE: the subject/slug values below are neutral placeholders. The maintainer
|
||||
# should point these fixtures at dense hubs that actually exist in their own
|
||||
# vault, otherwise the resolve cases will report INFO misses rather than hits.
|
||||
RESOLVE_CASES: list[dict] = [
|
||||
{"mention": "chorus", "expect": "exact", "expect_slug": "chorus"},
|
||||
{"mention": "chorus memory", "expect": "candidates", "expect_slug": "chorus"},
|
||||
{"mention": "CHORUS plugin", "expect": "candidates", "expect_slug": "chorus"},
|
||||
{"mention": "other-vault", "expect": "exact", "expect_slug": "other-vault"},
|
||||
{"mention": "example subject", "expect": "exact", "expect_slug": "example-subject"},
|
||||
{"mention": "operator", "expect": "candidates", "expect_slug": "example-subject"},
|
||||
{"mention": "zzqx nonexistent entity", "expect": "miss", "expect_slug": None},
|
||||
]
|
||||
|
||||
|
||||
# --- §3.3 bulk write fixtures -------------------------------------------------
|
||||
BENCH_PREFIX = "_agent/_bench" # run-scoped namespace: _agent/_bench/<run-id>/...
|
||||
|
||||
|
||||
def note_body(run_id: str, i: int) -> str:
|
||||
"""A realistic-shape note: canonical frontmatter + a couple of headings, so
|
||||
PUT cost reflects a true note, not a one-liner. agent_written + a _bench tag
|
||||
make these trivially greppable if a cleanup is ever missed."""
|
||||
return (
|
||||
"---\n"
|
||||
"type: working-memory\n"
|
||||
"status: active\n"
|
||||
"created: 2026-06-23\n"
|
||||
"updated: 2026-06-23\n"
|
||||
"tags:\n"
|
||||
" - agent\n"
|
||||
" - _bench\n"
|
||||
"agent_written: true\n"
|
||||
f"source_notes: []\n"
|
||||
f"bench_run: {run_id}\n"
|
||||
"---\n\n"
|
||||
f"# Bench Note {i:04d}\n\n"
|
||||
"## Body\n"
|
||||
f"Synthetic benchmark note {i} for run {run_id}. "
|
||||
"Filler so the payload is not pathologically small: "
|
||||
+ ("lorem ipsum dolor sit amet " * 6)
|
||||
+ "\n\n## Log\n- seed\n"
|
||||
)
|
||||
|
||||
|
||||
def note_path(run_id: str, i: int) -> str:
|
||||
return f"{BENCH_PREFIX}/{run_id}/note-{i:04d}.md"
|
||||
|
||||
|
||||
def append_target(run_id: str) -> str:
|
||||
return f"{BENCH_PREFIX}/{run_id}/append-target.md"
|
||||
|
||||
|
||||
def append_target_seed(run_id: str) -> str:
|
||||
return (
|
||||
"---\n"
|
||||
"type: working-memory\n"
|
||||
"status: active\n"
|
||||
"created: 2026-06-23\n"
|
||||
"updated: 2026-06-23\n"
|
||||
"tags:\n"
|
||||
" - agent\n"
|
||||
" - _bench\n"
|
||||
"agent_written: true\n"
|
||||
"source_notes: []\n"
|
||||
f"bench_run: {run_id}\n"
|
||||
"---\n\n"
|
||||
"# Append Target\n\n"
|
||||
"## Log\n"
|
||||
)
|
||||
@@ -0,0 +1,931 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:17:09",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": 4,
|
||||
"count_k": 50
|
||||
},
|
||||
"metrics": {
|
||||
"bulk-append": {
|
||||
"result": {
|
||||
"k": 30,
|
||||
"append_write": {
|
||||
"n": 30,
|
||||
"min_ms": 87.11,
|
||||
"median_ms": 104.08,
|
||||
"p90_ms": 190.63,
|
||||
"p95_ms": 198.89,
|
||||
"max_ms": 241.91,
|
||||
"mean_ms": 117.22
|
||||
},
|
||||
"append_skip": {
|
||||
"n": 30,
|
||||
"min_ms": 47.57,
|
||||
"median_ms": 51.42,
|
||||
"p90_ms": 59.82,
|
||||
"p95_ms": 61.57,
|
||||
"max_ms": 73.49,
|
||||
"mean_ms": 53.22
|
||||
},
|
||||
"skips_confirmed": 30,
|
||||
"all_skipped_second_pass": true
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "all_skipped_second_pass",
|
||||
"op": ">=",
|
||||
"min": 1,
|
||||
"why": "second pass of identical lines must be 100% idempotent skips (boolean true coerced to 1)"
|
||||
},
|
||||
"value": true,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"bulk-get": {
|
||||
"result": {
|
||||
"k": 30,
|
||||
"workers": 8,
|
||||
"serial": {
|
||||
"n": 5,
|
||||
"min_ms": 1520.47,
|
||||
"median_ms": 1606.86,
|
||||
"p90_ms": 1805.72,
|
||||
"p95_ms": 1835.83,
|
||||
"max_ms": 1865.94,
|
||||
"mean_ms": 1658.6,
|
||||
"warmup": {
|
||||
"n": 1,
|
||||
"min_ms": 1601.2,
|
||||
"median_ms": 1601.2,
|
||||
"p90_ms": 1601.2,
|
||||
"p95_ms": 1601.2,
|
||||
"max_ms": 1601.2,
|
||||
"mean_ms": 1601.2
|
||||
}
|
||||
},
|
||||
"concurrent": {
|
||||
"n": 5,
|
||||
"min_ms": 340.31,
|
||||
"median_ms": 356.48,
|
||||
"p90_ms": 358.82,
|
||||
"p95_ms": 359.32,
|
||||
"max_ms": 359.83,
|
||||
"mean_ms": 351.42,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 353.43,
|
||||
"median_ms": 434.94,
|
||||
"p90_ms": 500.15,
|
||||
"p95_ms": 508.3,
|
||||
"max_ms": 516.45,
|
||||
"mean_ms": 434.94
|
||||
}
|
||||
},
|
||||
"speedup_ratio": 4.51
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "speedup_ratio",
|
||||
"op": ">=",
|
||||
"min": 1.5,
|
||||
"why": "controlled-K concurrent vs serial; lower bar than full-vault since K is small"
|
||||
},
|
||||
"value": 4.51,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"bulk-put": {
|
||||
"result": {
|
||||
"k": 30,
|
||||
"per_put": {
|
||||
"n": 30,
|
||||
"min_ms": 100.09,
|
||||
"median_ms": 114.98,
|
||||
"p90_ms": 212.46,
|
||||
"p95_ms": 236.18,
|
||||
"max_ms": 330.77,
|
||||
"mean_ms": 143.25
|
||||
},
|
||||
"total_ms": 4297.5,
|
||||
"puts_per_sec": 7.0
|
||||
},
|
||||
"gates": []
|
||||
},
|
||||
"cache": {
|
||||
"result": {
|
||||
"requested": 60,
|
||||
"unique": 20,
|
||||
"returned": 20,
|
||||
"deduped": true,
|
||||
"elapsed_ms": 307.13
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "deduped",
|
||||
"op": ">=",
|
||||
"min": 1,
|
||||
"why": "read_many must collapse a 3x-duplicated path list to the unique set"
|
||||
},
|
||||
"value": true,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"expand-graph": {
|
||||
"result": {
|
||||
"subjects": [
|
||||
{
|
||||
"label": "hub-chorus",
|
||||
"query": "chorus",
|
||||
"neighbours": 120,
|
||||
"serial_ms": 2227.67,
|
||||
"concurrent_ms": 642.2,
|
||||
"speedup": 3.47,
|
||||
"ranking_identical": true,
|
||||
"scores_identical": true
|
||||
},
|
||||
{
|
||||
"label": "hub-prefs",
|
||||
"query": "operator preferences",
|
||||
"neighbours": 126,
|
||||
"serial_ms": 2910.46,
|
||||
"concurrent_ms": 698.8,
|
||||
"speedup": 4.16,
|
||||
"ranking_identical": true,
|
||||
"scores_identical": true
|
||||
}
|
||||
],
|
||||
"min_speedup": 3.47,
|
||||
"all_ranking_identical": true,
|
||||
"all_scores_identical": true
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "min_speedup",
|
||||
"op": ">=",
|
||||
"min": 2.0,
|
||||
"why": "recall()'s graph layer must fetch each BFS hop concurrently (read_many), not serially per node; <2x means expand_graph regressed to the pre-fix serial walk"
|
||||
},
|
||||
"value": 3.47,
|
||||
"pass": true
|
||||
},
|
||||
{
|
||||
"rule": {
|
||||
"field": "all_ranking_identical",
|
||||
"op": ">=",
|
||||
"min": 1,
|
||||
"why": "the concurrent expansion must return the same ranked neighbours as the serial reference"
|
||||
},
|
||||
"value": true,
|
||||
"pass": true
|
||||
},
|
||||
{
|
||||
"rule": {
|
||||
"field": "all_scores_identical",
|
||||
"op": ">=",
|
||||
"min": 1,
|
||||
"why": "decayed scores must match the serial reference to within float tolerance"
|
||||
},
|
||||
"value": true,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"index": {
|
||||
"result": {
|
||||
"script": "sweep.py",
|
||||
"elapsed_ms": 4629.0,
|
||||
"exit_code": 0,
|
||||
"ok": true,
|
||||
"stderr_tail": []
|
||||
},
|
||||
"gates": []
|
||||
},
|
||||
"lint": {
|
||||
"result": {
|
||||
"script": "vault_lint.py",
|
||||
"elapsed_ms": 5126.4,
|
||||
"exit_code": 1,
|
||||
"ok": true,
|
||||
"stderr_tail": []
|
||||
},
|
||||
"gates": []
|
||||
},
|
||||
"lock": {
|
||||
"result": {
|
||||
"acquire_free_ms": 159.18,
|
||||
"acquire_free_rc": 0,
|
||||
"contended_ms": 57.89,
|
||||
"contended_rc_expected_75": 75,
|
||||
"release_ms": 132.92,
|
||||
"ok": true
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "ok",
|
||||
"op": ">=",
|
||||
"min": 1,
|
||||
"why": "free acquire returns 0 and a contended acquire fast-fails with exit 75"
|
||||
},
|
||||
"value": true,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"pool-warmup": {
|
||||
"result": {
|
||||
"probe": "_agent/chorus-vault.md",
|
||||
"cold_ms": 163.25,
|
||||
"warm": {
|
||||
"n": 12,
|
||||
"min_ms": 45.64,
|
||||
"median_ms": 50.29,
|
||||
"p90_ms": 52.51,
|
||||
"p95_ms": 53.79,
|
||||
"max_ms": 55.28,
|
||||
"mean_ms": 50.26,
|
||||
"warmup": {
|
||||
"n": 1,
|
||||
"min_ms": 51.68,
|
||||
"median_ms": 51.68,
|
||||
"p90_ms": 51.68,
|
||||
"p95_ms": 51.68,
|
||||
"max_ms": 51.68,
|
||||
"mean_ms": 51.68
|
||||
}
|
||||
},
|
||||
"cold_over_warm_ratio": 3.25
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "cold_over_warm_ratio",
|
||||
"op": ">=",
|
||||
"min": 1.5,
|
||||
"why": "cold request must be meaningfully slower than warm \u2014 proves keep-alive is reusing the connection; a ratio near 1.0 means every request is re-handshaking (the pre-1.1.0 bug)"
|
||||
},
|
||||
"value": 3.25,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"queue": {
|
||||
"result": {
|
||||
"k": 10,
|
||||
"enqueue": {
|
||||
"n": 10,
|
||||
"min_ms": 0.07,
|
||||
"median_ms": 0.08,
|
||||
"p90_ms": 0.18,
|
||||
"p95_ms": 0.2,
|
||||
"max_ms": 0.22,
|
||||
"mean_ms": 0.1
|
||||
},
|
||||
"pending_after_enqueue": 10,
|
||||
"note": "flush replay intentionally not run against the live vault; enqueue path timed only. See README queue caveat."
|
||||
},
|
||||
"gates": []
|
||||
},
|
||||
"read-full": {
|
||||
"result": {
|
||||
"note_count": 197,
|
||||
"workers": 8,
|
||||
"serial": {
|
||||
"n": 1,
|
||||
"min_ms": 11376.08,
|
||||
"median_ms": 11376.08,
|
||||
"p90_ms": 11376.08,
|
||||
"p95_ms": 11376.08,
|
||||
"max_ms": 11376.08,
|
||||
"mean_ms": 11376.08
|
||||
},
|
||||
"concurrent": {
|
||||
"n": 4,
|
||||
"min_ms": 1559.79,
|
||||
"median_ms": 1595.31,
|
||||
"p90_ms": 1740.92,
|
||||
"p95_ms": 1768.05,
|
||||
"max_ms": 1795.17,
|
||||
"mean_ms": 1636.4,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 1641.35,
|
||||
"median_ms": 1652.91,
|
||||
"p90_ms": 1662.17,
|
||||
"p95_ms": 1663.32,
|
||||
"max_ms": 1664.48,
|
||||
"mean_ms": 1652.91
|
||||
}
|
||||
},
|
||||
"speedup_ratio": 7.13,
|
||||
"notes_per_sec_concurrent": 123.5
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "speedup_ratio",
|
||||
"op": ">=",
|
||||
"min": 1.8,
|
||||
"why": "concurrent read_many must beat serial by the 1.1.0 concurrency margin; <1.8 means concurrency regressed"
|
||||
},
|
||||
"value": 7.13,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"resolve": {
|
||||
"result": {
|
||||
"cases": [
|
||||
{
|
||||
"mention": "chorus",
|
||||
"expect": "exact",
|
||||
"expect_slug": "chorus",
|
||||
"resolved_slug": "chorus",
|
||||
"candidate_slugs": [
|
||||
"2026-06-19-other-vault-full-chorus-architect",
|
||||
"chorus",
|
||||
"chorus-memory-codex-plugin",
|
||||
"chorus-plugin-build",
|
||||
"chorus-skill-improvements"
|
||||
],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.0,
|
||||
"median_ms": 0.0,
|
||||
"p90_ms": 0.0,
|
||||
"p95_ms": 0.0,
|
||||
"max_ms": 0.0,
|
||||
"mean_ms": 0.0,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.01,
|
||||
"median_ms": 0.1,
|
||||
"p90_ms": 0.17,
|
||||
"p95_ms": 0.18,
|
||||
"max_ms": 0.19,
|
||||
"mean_ms": 0.1
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 1.35,
|
||||
"median_ms": 1.49,
|
||||
"p90_ms": 1.67,
|
||||
"p95_ms": 1.68,
|
||||
"max_ms": 1.68,
|
||||
"mean_ms": 1.51,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 1.82,
|
||||
"median_ms": 1.83,
|
||||
"p90_ms": 1.84,
|
||||
"p95_ms": 1.84,
|
||||
"max_ms": 1.85,
|
||||
"mean_ms": 1.83
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "chorus memory",
|
||||
"expect": "candidates",
|
||||
"expect_slug": "chorus",
|
||||
"resolved_slug": "chorus",
|
||||
"candidate_slugs": [
|
||||
"chorus",
|
||||
"chorus-memory-codex-plugin",
|
||||
"chorus-plugin-build",
|
||||
"chorus-skill-improvements",
|
||||
"2026-06-19-other-vault-full-chorus-architect"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.55,
|
||||
"median_ms": 0.56,
|
||||
"p90_ms": 0.57,
|
||||
"p95_ms": 0.58,
|
||||
"max_ms": 0.58,
|
||||
"mean_ms": 0.56,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.61,
|
||||
"median_ms": 0.61,
|
||||
"p90_ms": 0.61,
|
||||
"p95_ms": 0.61,
|
||||
"max_ms": 0.61,
|
||||
"mean_ms": 0.61
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.97,
|
||||
"median_ms": 0.99,
|
||||
"p90_ms": 1.06,
|
||||
"p95_ms": 1.09,
|
||||
"max_ms": 1.11,
|
||||
"mean_ms": 1.01,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 1.15,
|
||||
"median_ms": 1.18,
|
||||
"p90_ms": 1.2,
|
||||
"p95_ms": 1.21,
|
||||
"max_ms": 1.21,
|
||||
"mean_ms": 1.18
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "CHORUS plugin",
|
||||
"expect": "candidates",
|
||||
"expect_slug": "chorus",
|
||||
"resolved_slug": "chorus",
|
||||
"candidate_slugs": [
|
||||
"chorus",
|
||||
"chorus-memory-codex-plugin",
|
||||
"chorus-plugin-build",
|
||||
"2026-06-19-other-vault-full-chorus-architect",
|
||||
"example-isp-brand-docs"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.41,
|
||||
"median_ms": 0.41,
|
||||
"p90_ms": 0.42,
|
||||
"p95_ms": 0.42,
|
||||
"max_ms": 0.42,
|
||||
"mean_ms": 0.41,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.42,
|
||||
"median_ms": 0.45,
|
||||
"p90_ms": 0.47,
|
||||
"p95_ms": 0.48,
|
||||
"max_ms": 0.48,
|
||||
"mean_ms": 0.45
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.79,
|
||||
"median_ms": 0.8,
|
||||
"p90_ms": 0.88,
|
||||
"p95_ms": 0.9,
|
||||
"max_ms": 0.92,
|
||||
"mean_ms": 0.83,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.89,
|
||||
"median_ms": 0.89,
|
||||
"p90_ms": 0.9,
|
||||
"p95_ms": 0.9,
|
||||
"max_ms": 0.9,
|
||||
"mean_ms": 0.89
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "other-vault",
|
||||
"expect": "exact",
|
||||
"expect_slug": "other-vault",
|
||||
"resolved_slug": "other-vault",
|
||||
"candidate_slugs": [
|
||||
"2026-06-19-other-vault-full-chorus-architect",
|
||||
"other-vault"
|
||||
],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.0,
|
||||
"median_ms": 0.0,
|
||||
"p90_ms": 0.0,
|
||||
"p95_ms": 0.0,
|
||||
"max_ms": 0.0,
|
||||
"mean_ms": 0.0,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.0,
|
||||
"median_ms": 0.0,
|
||||
"p90_ms": 0.0,
|
||||
"p95_ms": 0.0,
|
||||
"max_ms": 0.0,
|
||||
"mean_ms": 0.0
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.73,
|
||||
"median_ms": 0.75,
|
||||
"p90_ms": 0.85,
|
||||
"p95_ms": 0.88,
|
||||
"max_ms": 0.91,
|
||||
"mean_ms": 0.78,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.8,
|
||||
"median_ms": 0.81,
|
||||
"p90_ms": 0.82,
|
||||
"p95_ms": 0.82,
|
||||
"max_ms": 0.82,
|
||||
"mean_ms": 0.81
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "example subject",
|
||||
"expect": "exact",
|
||||
"expect_slug": "example-subject",
|
||||
"resolved_slug": "example-subject",
|
||||
"candidate_slugs": [
|
||||
"example-subject",
|
||||
"operator-mcp-gateway"
|
||||
],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.0,
|
||||
"median_ms": 0.0,
|
||||
"p90_ms": 0.0,
|
||||
"p95_ms": 0.0,
|
||||
"max_ms": 0.0,
|
||||
"mean_ms": 0.0,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.0,
|
||||
"median_ms": 0.0,
|
||||
"p90_ms": 0.0,
|
||||
"p95_ms": 0.0,
|
||||
"max_ms": 0.0,
|
||||
"mean_ms": 0.0
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.69,
|
||||
"median_ms": 0.72,
|
||||
"p90_ms": 0.75,
|
||||
"p95_ms": 0.75,
|
||||
"max_ms": 0.75,
|
||||
"mean_ms": 0.72,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.74,
|
||||
"median_ms": 0.74,
|
||||
"p90_ms": 0.75,
|
||||
"p95_ms": 0.75,
|
||||
"max_ms": 0.75,
|
||||
"mean_ms": 0.74
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "operator",
|
||||
"expect": "candidates",
|
||||
"expect_slug": "example-subject",
|
||||
"resolved_slug": "example-subject",
|
||||
"candidate_slugs": [
|
||||
"operator-mcp-gateway",
|
||||
"example-subject"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.61,
|
||||
"median_ms": 0.65,
|
||||
"p90_ms": 0.66,
|
||||
"p95_ms": 0.67,
|
||||
"max_ms": 0.67,
|
||||
"mean_ms": 0.64,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.66,
|
||||
"median_ms": 0.68,
|
||||
"p90_ms": 0.69,
|
||||
"p95_ms": 0.69,
|
||||
"max_ms": 0.69,
|
||||
"mean_ms": 0.68
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.63,
|
||||
"median_ms": 0.63,
|
||||
"p90_ms": 0.65,
|
||||
"p95_ms": 0.65,
|
||||
"max_ms": 0.66,
|
||||
"mean_ms": 0.64,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.68,
|
||||
"median_ms": 0.69,
|
||||
"p90_ms": 0.7,
|
||||
"p95_ms": 0.7,
|
||||
"max_ms": 0.7,
|
||||
"mean_ms": 0.69
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "zzqx nonexistent entity",
|
||||
"expect": "miss",
|
||||
"expect_slug": null,
|
||||
"resolved_slug": null,
|
||||
"candidate_slugs": [],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.62,
|
||||
"median_ms": 0.63,
|
||||
"p90_ms": 0.67,
|
||||
"p95_ms": 0.67,
|
||||
"max_ms": 0.68,
|
||||
"mean_ms": 0.64,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.66,
|
||||
"median_ms": 0.67,
|
||||
"p90_ms": 0.67,
|
||||
"p95_ms": 0.67,
|
||||
"max_ms": 0.67,
|
||||
"mean_ms": 0.67
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.61,
|
||||
"median_ms": 0.62,
|
||||
"p90_ms": 0.65,
|
||||
"p95_ms": 0.65,
|
||||
"max_ms": 0.66,
|
||||
"mean_ms": 0.62,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.62,
|
||||
"median_ms": 0.63,
|
||||
"p90_ms": 0.64,
|
||||
"p95_ms": 0.64,
|
||||
"max_ms": 0.64,
|
||||
"mean_ms": 0.63
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"passes": 4,
|
||||
"needs_tuning": 3,
|
||||
"note": "verdicts are PASS/INFO only; promote to FAIL once the table is confirmed against the live vault"
|
||||
},
|
||||
"gates": []
|
||||
},
|
||||
"soak": {
|
||||
"result": {
|
||||
"seconds": 22,
|
||||
"iterations": 15,
|
||||
"errors": 0,
|
||||
"per_pass": {
|
||||
"n": 15,
|
||||
"min_ms": 1472.88,
|
||||
"median_ms": 1551.67,
|
||||
"p90_ms": 1648.45,
|
||||
"p95_ms": 1702.66,
|
||||
"max_ms": 1820.58,
|
||||
"mean_ms": 1569.81
|
||||
},
|
||||
"early_mean_ms": 1575.63,
|
||||
"late_mean_ms": 1564.72,
|
||||
"drift_ratio_late_over_early": 0.99
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "errors",
|
||||
"op": "<=",
|
||||
"min": 0,
|
||||
"why": "no read errors across the soak window \u2014 connection leaks/pool exhaustion would surface here"
|
||||
},
|
||||
"value": 0,
|
||||
"pass": true
|
||||
},
|
||||
{
|
||||
"rule": {
|
||||
"field": "drift_ratio_late_over_early",
|
||||
"op": "<=",
|
||||
"min": 1.5,
|
||||
"why": "late passes must not be much slower than early ones; >1.5 suggests a leak or degrading pool"
|
||||
},
|
||||
"value": 0.99,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"subject-pull": {
|
||||
"result": {
|
||||
"subjects": [
|
||||
{
|
||||
"label": "apta",
|
||||
"query": "APTA",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.75,
|
||||
"median_ms": 0.75,
|
||||
"p90_ms": 0.75,
|
||||
"p95_ms": 0.75,
|
||||
"max_ms": 0.75,
|
||||
"mean_ms": 0.75
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 61.87,
|
||||
"median_ms": 61.87,
|
||||
"p90_ms": 61.87,
|
||||
"p95_ms": 61.87,
|
||||
"max_ms": 61.87,
|
||||
"mean_ms": 61.87
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 2231.31,
|
||||
"median_ms": 2231.31,
|
||||
"p90_ms": 2231.31,
|
||||
"p95_ms": 2231.31,
|
||||
"max_ms": 2231.31,
|
||||
"mean_ms": 2231.31
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "capmetro",
|
||||
"query": "CapMetro",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.03,
|
||||
"median_ms": 0.03,
|
||||
"p90_ms": 0.03,
|
||||
"p95_ms": 0.03,
|
||||
"max_ms": 0.03,
|
||||
"mean_ms": 0.03
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 57.73,
|
||||
"median_ms": 57.73,
|
||||
"p90_ms": 57.73,
|
||||
"p95_ms": 57.73,
|
||||
"max_ms": 57.73,
|
||||
"mean_ms": 57.73
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 2116.06,
|
||||
"median_ms": 2116.06,
|
||||
"p90_ms": 2116.06,
|
||||
"p95_ms": 2116.06,
|
||||
"max_ms": 2116.06,
|
||||
"mean_ms": 2116.06
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "hub-chorus",
|
||||
"query": "chorus",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.03,
|
||||
"median_ms": 0.03,
|
||||
"p90_ms": 0.03,
|
||||
"p95_ms": 0.03,
|
||||
"max_ms": 0.03,
|
||||
"mean_ms": 0.03
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 70.56,
|
||||
"median_ms": 70.56,
|
||||
"p90_ms": 70.56,
|
||||
"p95_ms": 70.56,
|
||||
"max_ms": 70.56,
|
||||
"mean_ms": 70.56
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 2403.62,
|
||||
"median_ms": 2403.62,
|
||||
"p90_ms": 2403.62,
|
||||
"p95_ms": 2403.62,
|
||||
"max_ms": 2403.62,
|
||||
"mean_ms": 2403.62
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "hub-prefs",
|
||||
"query": "operator preferences",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.04,
|
||||
"median_ms": 0.04,
|
||||
"p90_ms": 0.04,
|
||||
"p95_ms": 0.04,
|
||||
"max_ms": 0.04,
|
||||
"mean_ms": 0.04
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 2011.13,
|
||||
"median_ms": 2011.13,
|
||||
"p90_ms": 2011.13,
|
||||
"p95_ms": 2011.13,
|
||||
"max_ms": 2011.13,
|
||||
"mean_ms": 2011.13
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 2994.84,
|
||||
"median_ms": 2994.84,
|
||||
"p90_ms": 2994.84,
|
||||
"p95_ms": 2994.84,
|
||||
"max_ms": 2994.84,
|
||||
"mean_ms": 2994.84
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "leaf-rivnut",
|
||||
"query": "rivnut torque spec",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 2.15,
|
||||
"median_ms": 2.15,
|
||||
"p90_ms": 2.15,
|
||||
"p95_ms": 2.15,
|
||||
"max_ms": 2.15,
|
||||
"mean_ms": 2.15
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 2013.58,
|
||||
"median_ms": 2013.58,
|
||||
"p90_ms": 2013.58,
|
||||
"p95_ms": 2013.58,
|
||||
"max_ms": 2013.58,
|
||||
"mean_ms": 2013.58
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 2371.32,
|
||||
"median_ms": 2371.32,
|
||||
"p90_ms": 2371.32,
|
||||
"p95_ms": 2371.32,
|
||||
"max_ms": 2371.32,
|
||||
"mean_ms": 2371.32
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"gates": []
|
||||
},
|
||||
"worker-sweep": {
|
||||
"result": {
|
||||
"note_count": 197,
|
||||
"sweep": [
|
||||
{
|
||||
"workers": 1,
|
||||
"median_ms": 11917.49,
|
||||
"p95_ms": 11917.49
|
||||
},
|
||||
{
|
||||
"workers": 2,
|
||||
"median_ms": 5740.0,
|
||||
"p95_ms": 5740.0
|
||||
},
|
||||
{
|
||||
"workers": 4,
|
||||
"median_ms": 3034.3,
|
||||
"p95_ms": 3034.3
|
||||
},
|
||||
{
|
||||
"workers": 8,
|
||||
"median_ms": 1583.83,
|
||||
"p95_ms": 1583.83
|
||||
},
|
||||
{
|
||||
"workers": 16,
|
||||
"median_ms": 1277.77,
|
||||
"p95_ms": 1277.77
|
||||
}
|
||||
],
|
||||
"knee_workers": 16,
|
||||
"default_workers": 8
|
||||
},
|
||||
"gates": []
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:22:00",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": null,
|
||||
"count_k": 30
|
||||
},
|
||||
"run_id": "20260623-202200-882c82",
|
||||
"metrics": {
|
||||
"bulk-append": {
|
||||
"result": {
|
||||
"k": 30,
|
||||
"append_write": {
|
||||
"n": 30,
|
||||
"min_ms": 87.11,
|
||||
"median_ms": 104.08,
|
||||
"p90_ms": 190.63,
|
||||
"p95_ms": 198.89,
|
||||
"max_ms": 241.91,
|
||||
"mean_ms": 117.22
|
||||
},
|
||||
"append_skip": {
|
||||
"n": 30,
|
||||
"min_ms": 47.57,
|
||||
"median_ms": 51.42,
|
||||
"p90_ms": 59.82,
|
||||
"p95_ms": 61.57,
|
||||
"max_ms": 73.49,
|
||||
"mean_ms": 53.22
|
||||
},
|
||||
"skips_confirmed": 30,
|
||||
"all_skipped_second_pass": true
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "all_skipped_second_pass",
|
||||
"op": ">=",
|
||||
"min": 1,
|
||||
"why": "second pass of identical lines must be 100% idempotent skips (boolean true coerced to 1)"
|
||||
},
|
||||
"value": true,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"cleanup": {
|
||||
"deleted": 1
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:21:36",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": 5,
|
||||
"count_k": 30
|
||||
},
|
||||
"run_id": "20260623-202136-f2f4dc",
|
||||
"metrics": {
|
||||
"bulk-get": {
|
||||
"result": {
|
||||
"k": 30,
|
||||
"workers": 8,
|
||||
"serial": {
|
||||
"n": 5,
|
||||
"min_ms": 1520.47,
|
||||
"median_ms": 1606.86,
|
||||
"p90_ms": 1805.72,
|
||||
"p95_ms": 1835.83,
|
||||
"max_ms": 1865.94,
|
||||
"mean_ms": 1658.6,
|
||||
"warmup": {
|
||||
"n": 1,
|
||||
"min_ms": 1601.2,
|
||||
"median_ms": 1601.2,
|
||||
"p90_ms": 1601.2,
|
||||
"p95_ms": 1601.2,
|
||||
"max_ms": 1601.2,
|
||||
"mean_ms": 1601.2
|
||||
}
|
||||
},
|
||||
"concurrent": {
|
||||
"n": 5,
|
||||
"min_ms": 340.31,
|
||||
"median_ms": 356.48,
|
||||
"p90_ms": 358.82,
|
||||
"p95_ms": 359.32,
|
||||
"max_ms": 359.83,
|
||||
"mean_ms": 351.42,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 353.43,
|
||||
"median_ms": 434.94,
|
||||
"p90_ms": 500.15,
|
||||
"p95_ms": 508.3,
|
||||
"max_ms": 516.45,
|
||||
"mean_ms": 434.94
|
||||
}
|
||||
},
|
||||
"speedup_ratio": 4.51
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "speedup_ratio",
|
||||
"op": ">=",
|
||||
"min": 1.5,
|
||||
"why": "controlled-K concurrent vs serial; lower bar than full-vault since K is small"
|
||||
},
|
||||
"value": 4.51,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"cleanup": {
|
||||
"deleted": 30
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:21:21",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": null,
|
||||
"count_k": 30
|
||||
},
|
||||
"run_id": "20260623-202121-23b5ac",
|
||||
"metrics": {
|
||||
"bulk-put": {
|
||||
"result": {
|
||||
"k": 30,
|
||||
"per_put": {
|
||||
"n": 30,
|
||||
"min_ms": 100.09,
|
||||
"median_ms": 114.98,
|
||||
"p90_ms": 212.46,
|
||||
"p95_ms": 236.18,
|
||||
"max_ms": 330.77,
|
||||
"mean_ms": 143.25
|
||||
},
|
||||
"total_ms": 4297.5,
|
||||
"puts_per_sec": 7.0
|
||||
},
|
||||
"gates": []
|
||||
}
|
||||
},
|
||||
"cleanup": {
|
||||
"deleted": 30
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:12:16",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": null,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-201216-e77971",
|
||||
"metrics": {
|
||||
"cache": {
|
||||
"result": {
|
||||
"requested": 60,
|
||||
"unique": 20,
|
||||
"returned": 20,
|
||||
"deduped": true,
|
||||
"elapsed_ms": 307.13
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "deduped",
|
||||
"op": ">=",
|
||||
"min": 1,
|
||||
"why": "read_many must collapse a 3x-duplicated path list to the unique set"
|
||||
},
|
||||
"value": true,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T21:22:29",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": 3,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-212229-8c3ce2",
|
||||
"metrics": {
|
||||
"expand-graph": {
|
||||
"result": {
|
||||
"subjects": [
|
||||
{
|
||||
"label": "hub-chorus",
|
||||
"query": "chorus",
|
||||
"neighbours": 120,
|
||||
"serial_ms": 2227.67,
|
||||
"concurrent_ms": 642.2,
|
||||
"speedup": 3.47,
|
||||
"ranking_identical": true,
|
||||
"scores_identical": true
|
||||
},
|
||||
{
|
||||
"label": "hub-prefs",
|
||||
"query": "operator preferences",
|
||||
"neighbours": 126,
|
||||
"serial_ms": 2910.46,
|
||||
"concurrent_ms": 698.8,
|
||||
"speedup": 4.16,
|
||||
"ranking_identical": true,
|
||||
"scores_identical": true
|
||||
}
|
||||
],
|
||||
"min_speedup": 3.47,
|
||||
"all_ranking_identical": true,
|
||||
"all_scores_identical": true
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "min_speedup",
|
||||
"op": ">=",
|
||||
"min": 2.0,
|
||||
"why": "recall()'s graph layer must fetch each BFS hop concurrently (read_many), not serially per node; <2x means expand_graph regressed to the pre-fix serial walk"
|
||||
},
|
||||
"value": 3.47,
|
||||
"pass": true
|
||||
},
|
||||
{
|
||||
"rule": {
|
||||
"field": "all_ranking_identical",
|
||||
"op": ">=",
|
||||
"min": 1,
|
||||
"why": "the concurrent expansion must return the same ranked neighbours as the serial reference"
|
||||
},
|
||||
"value": true,
|
||||
"pass": true
|
||||
},
|
||||
{
|
||||
"rule": {
|
||||
"field": "all_scores_identical",
|
||||
"op": ">=",
|
||||
"min": 1,
|
||||
"why": "decayed scores must match the serial reference to within float tolerance"
|
||||
},
|
||||
"value": true,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:20:57",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": null,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-202057-c41a52",
|
||||
"metrics": {
|
||||
"index": {
|
||||
"result": {
|
||||
"script": "sweep.py",
|
||||
"elapsed_ms": 4629.0,
|
||||
"exit_code": 0,
|
||||
"ok": true,
|
||||
"stderr_tail": []
|
||||
},
|
||||
"gates": []
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:21:01",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": null,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-202101-569382",
|
||||
"metrics": {
|
||||
"lint": {
|
||||
"result": {
|
||||
"script": "vault_lint.py",
|
||||
"elapsed_ms": 5126.4,
|
||||
"exit_code": 1,
|
||||
"ok": true,
|
||||
"stderr_tail": []
|
||||
},
|
||||
"gates": []
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:21:28",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": null,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-202128-53b113",
|
||||
"metrics": {
|
||||
"lock": {
|
||||
"result": {
|
||||
"acquire_free_ms": 159.18,
|
||||
"acquire_free_rc": 0,
|
||||
"contended_ms": 57.89,
|
||||
"contended_rc_expected_75": 75,
|
||||
"release_ms": 132.92,
|
||||
"ok": true
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "ok",
|
||||
"op": ">=",
|
||||
"min": 1,
|
||||
"why": "free acquire returns 0 and a contended acquire fast-fails with exit 75"
|
||||
},
|
||||
"value": true,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"cleanup": {
|
||||
"deleted": 0
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:12:12",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": 12,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-201212-30c2b8",
|
||||
"metrics": {
|
||||
"pool-warmup": {
|
||||
"result": {
|
||||
"probe": "_agent/chorus-vault.md",
|
||||
"cold_ms": 163.25,
|
||||
"warm": {
|
||||
"n": 12,
|
||||
"min_ms": 45.64,
|
||||
"median_ms": 50.29,
|
||||
"p90_ms": 52.51,
|
||||
"p95_ms": 53.79,
|
||||
"max_ms": 55.28,
|
||||
"mean_ms": 50.26,
|
||||
"warmup": {
|
||||
"n": 1,
|
||||
"min_ms": 51.68,
|
||||
"median_ms": 51.68,
|
||||
"p90_ms": 51.68,
|
||||
"p95_ms": 51.68,
|
||||
"max_ms": 51.68,
|
||||
"mean_ms": 51.68
|
||||
}
|
||||
},
|
||||
"cold_over_warm_ratio": 3.25
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "cold_over_warm_ratio",
|
||||
"op": ">=",
|
||||
"min": 1.5,
|
||||
"why": "cold request must be meaningfully slower than warm \u2014 proves keep-alive is reusing the connection; a ratio near 1.0 means every request is re-handshaking (the pre-1.1.0 bug)"
|
||||
},
|
||||
"value": 3.25,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:22:05",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": null,
|
||||
"count_k": 10
|
||||
},
|
||||
"run_id": "20260623-202205-378725",
|
||||
"metrics": {
|
||||
"queue": {
|
||||
"result": {
|
||||
"k": 10,
|
||||
"enqueue": {
|
||||
"n": 10,
|
||||
"min_ms": 0.07,
|
||||
"median_ms": 0.08,
|
||||
"p90_ms": 0.18,
|
||||
"p95_ms": 0.2,
|
||||
"max_ms": 0.22,
|
||||
"mean_ms": 0.1
|
||||
},
|
||||
"pending_after_enqueue": 10,
|
||||
"note": "flush replay intentionally not run against the live vault; enqueue path timed only. See README queue caveat."
|
||||
},
|
||||
"gates": []
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:17:09",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": 4,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-201709-b08a2e",
|
||||
"metrics": {
|
||||
"read-full": {
|
||||
"result": {
|
||||
"note_count": 197,
|
||||
"workers": 8,
|
||||
"serial": {
|
||||
"n": 1,
|
||||
"min_ms": 11376.08,
|
||||
"median_ms": 11376.08,
|
||||
"p90_ms": 11376.08,
|
||||
"p95_ms": 11376.08,
|
||||
"max_ms": 11376.08,
|
||||
"mean_ms": 11376.08
|
||||
},
|
||||
"concurrent": {
|
||||
"n": 4,
|
||||
"min_ms": 1559.79,
|
||||
"median_ms": 1595.31,
|
||||
"p90_ms": 1740.92,
|
||||
"p95_ms": 1768.05,
|
||||
"max_ms": 1795.17,
|
||||
"mean_ms": 1636.4,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 1641.35,
|
||||
"median_ms": 1652.91,
|
||||
"p90_ms": 1662.17,
|
||||
"p95_ms": 1663.32,
|
||||
"max_ms": 1664.48,
|
||||
"mean_ms": 1652.91
|
||||
}
|
||||
},
|
||||
"speedup_ratio": 7.13,
|
||||
"notes_per_sec_concurrent": 123.5
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "speedup_ratio",
|
||||
"op": ">=",
|
||||
"min": 1.8,
|
||||
"why": "concurrent read_many must beat serial by the 1.1.0 concurrency margin; <1.8 means concurrency regressed"
|
||||
},
|
||||
"value": 7.13,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,360 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:12:19",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": 6,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-201219-819b5e",
|
||||
"metrics": {
|
||||
"resolve": {
|
||||
"result": {
|
||||
"cases": [
|
||||
{
|
||||
"mention": "chorus",
|
||||
"expect": "exact",
|
||||
"expect_slug": "chorus",
|
||||
"resolved_slug": "chorus",
|
||||
"candidate_slugs": [
|
||||
"2026-06-19-other-vault-full-chorus-architect",
|
||||
"chorus",
|
||||
"chorus-memory-codex-plugin",
|
||||
"chorus-plugin-build",
|
||||
"chorus-skill-improvements"
|
||||
],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.0,
|
||||
"median_ms": 0.0,
|
||||
"p90_ms": 0.0,
|
||||
"p95_ms": 0.0,
|
||||
"max_ms": 0.0,
|
||||
"mean_ms": 0.0,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.01,
|
||||
"median_ms": 0.1,
|
||||
"p90_ms": 0.17,
|
||||
"p95_ms": 0.18,
|
||||
"max_ms": 0.19,
|
||||
"mean_ms": 0.1
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 1.35,
|
||||
"median_ms": 1.49,
|
||||
"p90_ms": 1.67,
|
||||
"p95_ms": 1.68,
|
||||
"max_ms": 1.68,
|
||||
"mean_ms": 1.51,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 1.82,
|
||||
"median_ms": 1.83,
|
||||
"p90_ms": 1.84,
|
||||
"p95_ms": 1.84,
|
||||
"max_ms": 1.85,
|
||||
"mean_ms": 1.83
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "chorus memory",
|
||||
"expect": "candidates",
|
||||
"expect_slug": "chorus",
|
||||
"resolved_slug": "chorus",
|
||||
"candidate_slugs": [
|
||||
"chorus",
|
||||
"chorus-memory-codex-plugin",
|
||||
"chorus-plugin-build",
|
||||
"chorus-skill-improvements",
|
||||
"2026-06-19-other-vault-full-chorus-architect"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.55,
|
||||
"median_ms": 0.56,
|
||||
"p90_ms": 0.57,
|
||||
"p95_ms": 0.58,
|
||||
"max_ms": 0.58,
|
||||
"mean_ms": 0.56,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.61,
|
||||
"median_ms": 0.61,
|
||||
"p90_ms": 0.61,
|
||||
"p95_ms": 0.61,
|
||||
"max_ms": 0.61,
|
||||
"mean_ms": 0.61
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.97,
|
||||
"median_ms": 0.99,
|
||||
"p90_ms": 1.06,
|
||||
"p95_ms": 1.09,
|
||||
"max_ms": 1.11,
|
||||
"mean_ms": 1.01,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 1.15,
|
||||
"median_ms": 1.18,
|
||||
"p90_ms": 1.2,
|
||||
"p95_ms": 1.21,
|
||||
"max_ms": 1.21,
|
||||
"mean_ms": 1.18
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "CHORUS plugin",
|
||||
"expect": "candidates",
|
||||
"expect_slug": "chorus",
|
||||
"resolved_slug": "chorus",
|
||||
"candidate_slugs": [
|
||||
"chorus",
|
||||
"chorus-memory-codex-plugin",
|
||||
"chorus-plugin-build",
|
||||
"2026-06-19-other-vault-full-chorus-architect",
|
||||
"example-isp-brand-docs"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.41,
|
||||
"median_ms": 0.41,
|
||||
"p90_ms": 0.42,
|
||||
"p95_ms": 0.42,
|
||||
"max_ms": 0.42,
|
||||
"mean_ms": 0.41,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.42,
|
||||
"median_ms": 0.45,
|
||||
"p90_ms": 0.47,
|
||||
"p95_ms": 0.48,
|
||||
"max_ms": 0.48,
|
||||
"mean_ms": 0.45
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.79,
|
||||
"median_ms": 0.8,
|
||||
"p90_ms": 0.88,
|
||||
"p95_ms": 0.9,
|
||||
"max_ms": 0.92,
|
||||
"mean_ms": 0.83,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.89,
|
||||
"median_ms": 0.89,
|
||||
"p90_ms": 0.9,
|
||||
"p95_ms": 0.9,
|
||||
"max_ms": 0.9,
|
||||
"mean_ms": 0.89
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "other-vault",
|
||||
"expect": "exact",
|
||||
"expect_slug": "other-vault",
|
||||
"resolved_slug": "other-vault",
|
||||
"candidate_slugs": [
|
||||
"2026-06-19-other-vault-full-chorus-architect",
|
||||
"other-vault"
|
||||
],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.0,
|
||||
"median_ms": 0.0,
|
||||
"p90_ms": 0.0,
|
||||
"p95_ms": 0.0,
|
||||
"max_ms": 0.0,
|
||||
"mean_ms": 0.0,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.0,
|
||||
"median_ms": 0.0,
|
||||
"p90_ms": 0.0,
|
||||
"p95_ms": 0.0,
|
||||
"max_ms": 0.0,
|
||||
"mean_ms": 0.0
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.73,
|
||||
"median_ms": 0.75,
|
||||
"p90_ms": 0.85,
|
||||
"p95_ms": 0.88,
|
||||
"max_ms": 0.91,
|
||||
"mean_ms": 0.78,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.8,
|
||||
"median_ms": 0.81,
|
||||
"p90_ms": 0.82,
|
||||
"p95_ms": 0.82,
|
||||
"max_ms": 0.82,
|
||||
"mean_ms": 0.81
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "example subject",
|
||||
"expect": "exact",
|
||||
"expect_slug": "example-subject",
|
||||
"resolved_slug": "example-subject",
|
||||
"candidate_slugs": [
|
||||
"example-subject",
|
||||
"operator-mcp-gateway"
|
||||
],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.0,
|
||||
"median_ms": 0.0,
|
||||
"p90_ms": 0.0,
|
||||
"p95_ms": 0.0,
|
||||
"max_ms": 0.0,
|
||||
"mean_ms": 0.0,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.0,
|
||||
"median_ms": 0.0,
|
||||
"p90_ms": 0.0,
|
||||
"p95_ms": 0.0,
|
||||
"max_ms": 0.0,
|
||||
"mean_ms": 0.0
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.69,
|
||||
"median_ms": 0.72,
|
||||
"p90_ms": 0.75,
|
||||
"p95_ms": 0.75,
|
||||
"max_ms": 0.75,
|
||||
"mean_ms": 0.72,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.74,
|
||||
"median_ms": 0.74,
|
||||
"p90_ms": 0.75,
|
||||
"p95_ms": 0.75,
|
||||
"max_ms": 0.75,
|
||||
"mean_ms": 0.74
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "operator",
|
||||
"expect": "candidates",
|
||||
"expect_slug": "example-subject",
|
||||
"resolved_slug": "example-subject",
|
||||
"candidate_slugs": [
|
||||
"operator-mcp-gateway",
|
||||
"example-subject"
|
||||
],
|
||||
"verdict": "INFO",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.61,
|
||||
"median_ms": 0.65,
|
||||
"p90_ms": 0.66,
|
||||
"p95_ms": 0.67,
|
||||
"max_ms": 0.67,
|
||||
"mean_ms": 0.64,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.66,
|
||||
"median_ms": 0.68,
|
||||
"p90_ms": 0.69,
|
||||
"p95_ms": 0.69,
|
||||
"max_ms": 0.69,
|
||||
"mean_ms": 0.68
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.63,
|
||||
"median_ms": 0.63,
|
||||
"p90_ms": 0.65,
|
||||
"p95_ms": 0.65,
|
||||
"max_ms": 0.66,
|
||||
"mean_ms": 0.64,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.68,
|
||||
"median_ms": 0.69,
|
||||
"p90_ms": 0.7,
|
||||
"p95_ms": 0.7,
|
||||
"max_ms": 0.7,
|
||||
"mean_ms": 0.69
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"mention": "zzqx nonexistent entity",
|
||||
"expect": "miss",
|
||||
"expect_slug": null,
|
||||
"resolved_slug": null,
|
||||
"candidate_slugs": [],
|
||||
"verdict": "PASS",
|
||||
"resolve_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.62,
|
||||
"median_ms": 0.63,
|
||||
"p90_ms": 0.67,
|
||||
"p95_ms": 0.67,
|
||||
"max_ms": 0.68,
|
||||
"mean_ms": 0.64,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.66,
|
||||
"median_ms": 0.67,
|
||||
"p90_ms": 0.67,
|
||||
"p95_ms": 0.67,
|
||||
"max_ms": 0.67,
|
||||
"mean_ms": 0.67
|
||||
}
|
||||
},
|
||||
"fuzzy_timing": {
|
||||
"n": 6,
|
||||
"min_ms": 0.61,
|
||||
"median_ms": 0.62,
|
||||
"p90_ms": 0.65,
|
||||
"p95_ms": 0.65,
|
||||
"max_ms": 0.66,
|
||||
"mean_ms": 0.62,
|
||||
"warmup": {
|
||||
"n": 2,
|
||||
"min_ms": 0.62,
|
||||
"median_ms": 0.63,
|
||||
"p90_ms": 0.64,
|
||||
"p95_ms": 0.64,
|
||||
"max_ms": 0.64,
|
||||
"mean_ms": 0.63
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"passes": 4,
|
||||
"needs_tuning": 3,
|
||||
"note": "verdicts are PASS/INFO only; promote to FAIL once the table is confirmed against the live vault"
|
||||
},
|
||||
"gates": []
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:22:15",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": null,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-202215-7b414f",
|
||||
"metrics": {
|
||||
"soak": {
|
||||
"result": {
|
||||
"seconds": 22,
|
||||
"iterations": 15,
|
||||
"errors": 0,
|
||||
"per_pass": {
|
||||
"n": 15,
|
||||
"min_ms": 1472.88,
|
||||
"median_ms": 1551.67,
|
||||
"p90_ms": 1648.45,
|
||||
"p95_ms": 1702.66,
|
||||
"max_ms": 1820.58,
|
||||
"mean_ms": 1569.81
|
||||
},
|
||||
"early_mean_ms": 1575.63,
|
||||
"late_mean_ms": 1564.72,
|
||||
"drift_ratio_late_over_early": 0.99
|
||||
},
|
||||
"gates": [
|
||||
{
|
||||
"rule": {
|
||||
"field": "errors",
|
||||
"op": "<=",
|
||||
"min": 0,
|
||||
"why": "no read errors across the soak window \u2014 connection leaks/pool exhaustion would surface here"
|
||||
},
|
||||
"value": 0,
|
||||
"pass": true
|
||||
},
|
||||
{
|
||||
"rule": {
|
||||
"field": "drift_ratio_late_over_early",
|
||||
"op": "<=",
|
||||
"min": 1.5,
|
||||
"why": "late passes must not be much slower than early ones; >1.5 suggests a leak or degrading pool"
|
||||
},
|
||||
"value": 0.99,
|
||||
"pass": true
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
+177
@@ -0,0 +1,177 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T21:18:12",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": 1,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-211812-ea4b36",
|
||||
"metrics": {
|
||||
"subject-pull": {
|
||||
"result": {
|
||||
"subjects": [
|
||||
{
|
||||
"label": "apta",
|
||||
"query": "APTA",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.75,
|
||||
"median_ms": 0.75,
|
||||
"p90_ms": 0.75,
|
||||
"p95_ms": 0.75,
|
||||
"max_ms": 0.75,
|
||||
"mean_ms": 0.75
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 61.87,
|
||||
"median_ms": 61.87,
|
||||
"p90_ms": 61.87,
|
||||
"p95_ms": 61.87,
|
||||
"max_ms": 61.87,
|
||||
"mean_ms": 61.87
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 2231.31,
|
||||
"median_ms": 2231.31,
|
||||
"p90_ms": 2231.31,
|
||||
"p95_ms": 2231.31,
|
||||
"max_ms": 2231.31,
|
||||
"mean_ms": 2231.31
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "capmetro",
|
||||
"query": "CapMetro",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.03,
|
||||
"median_ms": 0.03,
|
||||
"p90_ms": 0.03,
|
||||
"p95_ms": 0.03,
|
||||
"max_ms": 0.03,
|
||||
"mean_ms": 0.03
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 57.73,
|
||||
"median_ms": 57.73,
|
||||
"p90_ms": 57.73,
|
||||
"p95_ms": 57.73,
|
||||
"max_ms": 57.73,
|
||||
"mean_ms": 57.73
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 2116.06,
|
||||
"median_ms": 2116.06,
|
||||
"p90_ms": 2116.06,
|
||||
"p95_ms": 2116.06,
|
||||
"max_ms": 2116.06,
|
||||
"mean_ms": 2116.06
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "hub-chorus",
|
||||
"query": "chorus",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.03,
|
||||
"median_ms": 0.03,
|
||||
"p90_ms": 0.03,
|
||||
"p95_ms": 0.03,
|
||||
"max_ms": 0.03,
|
||||
"mean_ms": 0.03
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 70.56,
|
||||
"median_ms": 70.56,
|
||||
"p90_ms": 70.56,
|
||||
"p95_ms": 70.56,
|
||||
"max_ms": 70.56,
|
||||
"mean_ms": 70.56
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 2403.62,
|
||||
"median_ms": 2403.62,
|
||||
"p90_ms": 2403.62,
|
||||
"p95_ms": 2403.62,
|
||||
"max_ms": 2403.62,
|
||||
"mean_ms": 2403.62
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "hub-prefs",
|
||||
"query": "operator preferences",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.04,
|
||||
"median_ms": 0.04,
|
||||
"p90_ms": 0.04,
|
||||
"p95_ms": 0.04,
|
||||
"max_ms": 0.04,
|
||||
"mean_ms": 0.04
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 2011.13,
|
||||
"median_ms": 2011.13,
|
||||
"p90_ms": 2011.13,
|
||||
"p95_ms": 2011.13,
|
||||
"max_ms": 2011.13,
|
||||
"mean_ms": 2011.13
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 2994.84,
|
||||
"median_ms": 2994.84,
|
||||
"p90_ms": 2994.84,
|
||||
"p95_ms": 2994.84,
|
||||
"max_ms": 2994.84,
|
||||
"mean_ms": 2994.84
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "leaf-rivnut",
|
||||
"query": "rivnut torque spec",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 2.15,
|
||||
"median_ms": 2.15,
|
||||
"p90_ms": 2.15,
|
||||
"p95_ms": 2.15,
|
||||
"max_ms": 2.15,
|
||||
"mean_ms": 2.15
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 2013.58,
|
||||
"median_ms": 2013.58,
|
||||
"p90_ms": 2013.58,
|
||||
"p95_ms": 2013.58,
|
||||
"max_ms": 2013.58,
|
||||
"mean_ms": 2013.58
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 2371.32,
|
||||
"median_ms": 2371.32,
|
||||
"p90_ms": 2371.32,
|
||||
"p95_ms": 2371.32,
|
||||
"max_ms": 2371.32,
|
||||
"mean_ms": 2371.32
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"gates": []
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,177 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:20:23",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": 1,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-202023-274ddd",
|
||||
"metrics": {
|
||||
"subject-pull": {
|
||||
"result": {
|
||||
"subjects": [
|
||||
{
|
||||
"label": "apta",
|
||||
"query": "APTA",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.97,
|
||||
"median_ms": 0.97,
|
||||
"p90_ms": 0.97,
|
||||
"p95_ms": 0.97,
|
||||
"max_ms": 0.97,
|
||||
"mean_ms": 0.97
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 71.55,
|
||||
"median_ms": 71.55,
|
||||
"p90_ms": 71.55,
|
||||
"p95_ms": 71.55,
|
||||
"max_ms": 71.55,
|
||||
"mean_ms": 71.55
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 3836.47,
|
||||
"median_ms": 3836.47,
|
||||
"p90_ms": 3836.47,
|
||||
"p95_ms": 3836.47,
|
||||
"max_ms": 3836.47,
|
||||
"mean_ms": 3836.47
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "capmetro",
|
||||
"query": "CapMetro",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.05,
|
||||
"median_ms": 0.05,
|
||||
"p90_ms": 0.05,
|
||||
"p95_ms": 0.05,
|
||||
"max_ms": 0.05,
|
||||
"mean_ms": 0.05
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 57.66,
|
||||
"median_ms": 57.66,
|
||||
"p90_ms": 57.66,
|
||||
"p95_ms": 57.66,
|
||||
"max_ms": 57.66,
|
||||
"mean_ms": 57.66
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 2800.54,
|
||||
"median_ms": 2800.54,
|
||||
"p90_ms": 2800.54,
|
||||
"p95_ms": 2800.54,
|
||||
"max_ms": 2800.54,
|
||||
"mean_ms": 2800.54
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "hub-chorus",
|
||||
"query": "chorus",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.02,
|
||||
"median_ms": 0.02,
|
||||
"p90_ms": 0.02,
|
||||
"p95_ms": 0.02,
|
||||
"max_ms": 0.02,
|
||||
"mean_ms": 0.02
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 67.39,
|
||||
"median_ms": 67.39,
|
||||
"p90_ms": 67.39,
|
||||
"p95_ms": 67.39,
|
||||
"max_ms": 67.39,
|
||||
"mean_ms": 67.39
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 4114.19,
|
||||
"median_ms": 4114.19,
|
||||
"p90_ms": 4114.19,
|
||||
"p95_ms": 4114.19,
|
||||
"max_ms": 4114.19,
|
||||
"mean_ms": 4114.19
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "hub-prefs",
|
||||
"query": "operator preferences",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 0.04,
|
||||
"median_ms": 0.04,
|
||||
"p90_ms": 0.04,
|
||||
"p95_ms": 0.04,
|
||||
"max_ms": 0.04,
|
||||
"mean_ms": 0.04
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 2013.4,
|
||||
"median_ms": 2013.4,
|
||||
"p90_ms": 2013.4,
|
||||
"p95_ms": 2013.4,
|
||||
"max_ms": 2013.4,
|
||||
"mean_ms": 2013.4
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 4748.56,
|
||||
"median_ms": 4748.56,
|
||||
"p90_ms": 4748.56,
|
||||
"p95_ms": 4748.56,
|
||||
"max_ms": 4748.56,
|
||||
"mean_ms": 4748.56
|
||||
}
|
||||
},
|
||||
{
|
||||
"label": "leaf-rivnut",
|
||||
"query": "rivnut torque spec",
|
||||
"resolve_floor": {
|
||||
"n": 1,
|
||||
"min_ms": 2.14,
|
||||
"median_ms": 2.14,
|
||||
"p90_ms": 2.14,
|
||||
"p95_ms": 2.14,
|
||||
"max_ms": 2.14,
|
||||
"mean_ms": 2.14
|
||||
},
|
||||
"search_only": {
|
||||
"n": 1,
|
||||
"min_ms": 2014.69,
|
||||
"median_ms": 2014.69,
|
||||
"p90_ms": 2014.69,
|
||||
"p95_ms": 2014.69,
|
||||
"max_ms": 2014.69,
|
||||
"mean_ms": 2014.69
|
||||
},
|
||||
"recall_total": {
|
||||
"n": 1,
|
||||
"min_ms": 4363.28,
|
||||
"median_ms": 4363.28,
|
||||
"p90_ms": 4363.28,
|
||||
"p95_ms": 4363.28,
|
||||
"max_ms": 4363.28,
|
||||
"mean_ms": 4363.28
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"gates": []
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"env": {
|
||||
"date": "2026-06-23T20:18:05",
|
||||
"endpoint": "https://obsidian.example.com",
|
||||
"commit": "af16598",
|
||||
"chorus_workers": 8,
|
||||
"chorus_timeout": 30,
|
||||
"python": "3.10.12",
|
||||
"iterations": 2,
|
||||
"count_k": 50
|
||||
},
|
||||
"run_id": "20260623-201805-53dcdb",
|
||||
"metrics": {
|
||||
"worker-sweep": {
|
||||
"result": {
|
||||
"note_count": 197,
|
||||
"sweep": [
|
||||
{
|
||||
"workers": 1,
|
||||
"median_ms": 11917.49,
|
||||
"p95_ms": 11917.49
|
||||
},
|
||||
{
|
||||
"workers": 2,
|
||||
"median_ms": 5740.0,
|
||||
"p95_ms": 5740.0
|
||||
},
|
||||
{
|
||||
"workers": 4,
|
||||
"median_ms": 3034.3,
|
||||
"p95_ms": 3034.3
|
||||
},
|
||||
{
|
||||
"workers": 8,
|
||||
"median_ms": 1583.83,
|
||||
"p95_ms": 1583.83
|
||||
},
|
||||
{
|
||||
"workers": 16,
|
||||
"median_ms": 1277.77,
|
||||
"p95_ms": 1277.77
|
||||
}
|
||||
],
|
||||
"knee_workers": 16,
|
||||
"default_workers": 8
|
||||
},
|
||||
"gates": []
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,545 @@
|
||||
{
|
||||
"title": "CHORUS Memory Plugin \u2014 Performance Benchmark",
|
||||
"subtitle": "Validation of the 1.1.0 concurrency and 1.2.0 entity-resolution releases",
|
||||
"reference": "chorus-v.05 @ af16598",
|
||||
"status": "11/11 gate checks pass",
|
||||
"summary": "Full-vault reads run 7.1x faster concurrent than serial (11.4 s down to 1.6 s, 197 notes). Entity resolution is sub-millisecond. Profiling traced recall() latency to serial graph expansion (not BM25); the fix makes expansion concurrent (3.5-4.2x, identical results). All 11 gate checks pass. Run live against obsidian.example.com on 2026-06-23.",
|
||||
"sections": [
|
||||
{
|
||||
"heading": "Test Environment",
|
||||
"type": "kv",
|
||||
"rows": [
|
||||
{
|
||||
"label": "Endpoint",
|
||||
"value": "obsidian.example.com (live)"
|
||||
},
|
||||
{
|
||||
"label": "Vault size",
|
||||
"value": "197 notes"
|
||||
},
|
||||
{
|
||||
"label": "Plugin commit",
|
||||
"value": "af16598"
|
||||
},
|
||||
{
|
||||
"label": "Releases under test",
|
||||
"value": "1.1.0 pooling + concurrency, 1.2.0 resolver"
|
||||
},
|
||||
{
|
||||
"label": "Concurrency (default)",
|
||||
"value": "CHORUS_WORKERS = 8"
|
||||
},
|
||||
{
|
||||
"label": "Socket timeout",
|
||||
"value": "CHORUS_TIMEOUT = 30 s"
|
||||
},
|
||||
{
|
||||
"label": "Runner",
|
||||
"value": "Python 3.10.12"
|
||||
},
|
||||
{
|
||||
"label": "Date",
|
||||
"value": "June 23, 2026"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"heading": "Headline Results",
|
||||
"type": "table",
|
||||
"columns": [
|
||||
"Operation",
|
||||
"Result",
|
||||
"Verdict"
|
||||
],
|
||||
"rows": [
|
||||
[
|
||||
"Full-vault read (concurrent vs serial)",
|
||||
{
|
||||
"value": "7.13x",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"Full-vault read throughput",
|
||||
{
|
||||
"value": "123.5 notes/s",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"Cold vs warm request (keep-alive)",
|
||||
{
|
||||
"value": "3.25x",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"Entity resolve (in-memory index)",
|
||||
{
|
||||
"value": "<1 ms",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"Idempotent append skip",
|
||||
{
|
||||
"value": "100%",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"Soak read errors (22 s)",
|
||||
{
|
||||
"value": "0",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"heading": "Full-Vault Read \u2014 Concurrency Scaling",
|
||||
"type": "table",
|
||||
"columns": [
|
||||
"Workers",
|
||||
"Median (197 notes)",
|
||||
"Speedup vs serial"
|
||||
],
|
||||
"rows": [
|
||||
[
|
||||
"1 (serial)",
|
||||
{
|
||||
"value": "11,917 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "1.0x",
|
||||
"num": true
|
||||
}
|
||||
],
|
||||
[
|
||||
"2",
|
||||
{
|
||||
"value": "5,740 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "2.1x",
|
||||
"num": true
|
||||
}
|
||||
],
|
||||
[
|
||||
"4",
|
||||
{
|
||||
"value": "3,034 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "3.9x",
|
||||
"num": true
|
||||
}
|
||||
],
|
||||
[
|
||||
"8 (default)",
|
||||
{
|
||||
"value": "1,584 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "7.5x",
|
||||
"num": true
|
||||
}
|
||||
],
|
||||
[
|
||||
"16",
|
||||
{
|
||||
"value": "1,278 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "9.3x",
|
||||
"num": true
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"heading": "Subject Pulls \u2014 recall() (after concurrency fix)",
|
||||
"type": "table",
|
||||
"columns": [
|
||||
"Subject",
|
||||
"resolve()",
|
||||
"search",
|
||||
"recall() total"
|
||||
],
|
||||
"rows": [
|
||||
[
|
||||
"APTA",
|
||||
{
|
||||
"value": "0.75 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "62 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "2,231 ms",
|
||||
"num": true
|
||||
}
|
||||
],
|
||||
[
|
||||
"CapMetro",
|
||||
{
|
||||
"value": "0.03 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "58 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "2,116 ms",
|
||||
"num": true
|
||||
}
|
||||
],
|
||||
[
|
||||
"chorus (hub)",
|
||||
{
|
||||
"value": "0.03 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "71 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "2,404 ms",
|
||||
"num": true
|
||||
}
|
||||
],
|
||||
[
|
||||
"operator preferences (hub)",
|
||||
{
|
||||
"value": "0.04 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "2,011 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "2,995 ms",
|
||||
"num": true
|
||||
}
|
||||
],
|
||||
[
|
||||
"rivnut torque spec (leaf)",
|
||||
{
|
||||
"value": "2.15 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "2,014 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "2,371 ms",
|
||||
"num": true
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"heading": "recall() Graph Expansion \u2014 Concurrency Fix",
|
||||
"type": "table",
|
||||
"columns": [
|
||||
"Hub subject",
|
||||
"Neighbours",
|
||||
"Serial",
|
||||
"Concurrent",
|
||||
"Speedup",
|
||||
"Results identical"
|
||||
],
|
||||
"rows": [
|
||||
[
|
||||
"chorus",
|
||||
{
|
||||
"value": "120",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "2,228 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "642 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "3.47x",
|
||||
"num": true
|
||||
},
|
||||
"Yes"
|
||||
],
|
||||
[
|
||||
"operator preferences",
|
||||
{
|
||||
"value": "126",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "2,910 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "699 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "4.16x",
|
||||
"num": true
|
||||
},
|
||||
"Yes"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"heading": "Bulk Write Operations",
|
||||
"type": "table",
|
||||
"columns": [
|
||||
"Operation",
|
||||
"Per-op median",
|
||||
"Aggregate"
|
||||
],
|
||||
"rows": [
|
||||
[
|
||||
"PUT (read-back verified)",
|
||||
{
|
||||
"value": "115 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "7.0 PUT/s",
|
||||
"num": true
|
||||
}
|
||||
],
|
||||
[
|
||||
"Bulk GET \u2014 concurrent vs serial (K=30)",
|
||||
{
|
||||
"value": "356 vs 1,607 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "4.51x",
|
||||
"num": true
|
||||
}
|
||||
],
|
||||
[
|
||||
"APPEND \u2014 write",
|
||||
{
|
||||
"value": "104 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "1 round-trip + GET",
|
||||
"num": false
|
||||
}
|
||||
],
|
||||
[
|
||||
"APPEND \u2014 idempotent skip",
|
||||
{
|
||||
"value": "51 ms",
|
||||
"num": true
|
||||
},
|
||||
{
|
||||
"value": "100% skipped on re-run",
|
||||
"num": false
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"heading": "Concurrency, Locking & Resilience",
|
||||
"type": "table",
|
||||
"columns": [
|
||||
"Check",
|
||||
"Measured",
|
||||
"Expected"
|
||||
],
|
||||
"rows": [
|
||||
[
|
||||
"Lock \u2014 acquire when free",
|
||||
{
|
||||
"value": "159 ms (rc 0)",
|
||||
"num": true
|
||||
},
|
||||
"rc 0"
|
||||
],
|
||||
[
|
||||
"Lock \u2014 contended acquire",
|
||||
{
|
||||
"value": "58 ms (rc 75)",
|
||||
"num": true
|
||||
},
|
||||
"rc 75 fast-fail"
|
||||
],
|
||||
[
|
||||
"Lock \u2014 release",
|
||||
{
|
||||
"value": "133 ms",
|
||||
"num": true
|
||||
},
|
||||
"owned release"
|
||||
],
|
||||
[
|
||||
"Offline queue \u2014 enqueue",
|
||||
{
|
||||
"value": "0.08 ms",
|
||||
"num": true
|
||||
},
|
||||
"local write-ahead"
|
||||
],
|
||||
[
|
||||
"read_many dedup (60 -> 20)",
|
||||
{
|
||||
"value": "deduped",
|
||||
"num": false
|
||||
},
|
||||
"collapse to unique set"
|
||||
],
|
||||
[
|
||||
"Soak drift (late vs early)",
|
||||
{
|
||||
"value": "0.99x",
|
||||
"num": true
|
||||
},
|
||||
"<= 1.5x, no leak"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"heading": "Regression Gates (relative-ratio policy)",
|
||||
"type": "table",
|
||||
"columns": [
|
||||
"Gate",
|
||||
"Threshold",
|
||||
"Measured",
|
||||
"Result"
|
||||
],
|
||||
"rows": [
|
||||
[
|
||||
"read-full speedup",
|
||||
">= 1.8x",
|
||||
{
|
||||
"value": "7.13x",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"bulk-get speedup",
|
||||
">= 1.5x",
|
||||
{
|
||||
"value": "4.51x",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"pool warm-up cold/warm",
|
||||
">= 1.5x",
|
||||
{
|
||||
"value": "3.25x",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"append idempotency",
|
||||
"100% skip",
|
||||
{
|
||||
"value": "100%",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"soak errors",
|
||||
"0",
|
||||
{
|
||||
"value": "0",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"soak drift",
|
||||
"<= 1.5x",
|
||||
{
|
||||
"value": "0.99x",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"lock semantics",
|
||||
"rc 0 / rc 75",
|
||||
{
|
||||
"value": "0 / 75",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
],
|
||||
[
|
||||
"read_many dedup",
|
||||
"unique set",
|
||||
{
|
||||
"value": "20/20",
|
||||
"num": true
|
||||
},
|
||||
"Pass"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"heading": "Findings",
|
||||
"type": "olist",
|
||||
"items": [
|
||||
"Concurrency delivers as designed. Eight workers cut full-vault reads 7.5x; the curve is near-linear to 8 workers and flattens after (8->16 returns only 1.24x). The default of 8 is the right setting for this vault and link.",
|
||||
"Keep-alive is confirmed live. A cold request costs 163 ms against a 50 ms warm median (3.25x). This is the direct measure that the 1.1.0 pooling fix removed the per-request TLS handshake that was timing out full-vault passes.",
|
||||
"Entity resolution is effectively free. resolve() returns in under 1 ms across exact, alias, and shortened mentions. The 1.2.0 alias work resolves shortened names (\"chorus memory\", \"CHORUS plugin\", \"operator\") straight to the canonical note, so the anti-duplicate guard holds without a fuzzy fallback.",
|
||||
"recall() latency was traced to the graph layer, not BM25. The BM25 index is already persisted and incrementally maintained (load 500 ms, score 0.1 ms). The cost was expand_graph fetching each neighbour serially \u2014 3.0 s for 126 nodes. Fix: the graph BFS now fetches each hop concurrently via the existing read_many. expand_graph is 3.5-4.2x faster with byte-identical ranking and scores; end-to-end recall() dropped from 2.8-4.7 s to 2.1-3.0 s per subject.",
|
||||
"Full-vault maintenance scripts stay under the tool timeout. sweep.py runs in 4.6 s and vault_lint.py in 5.1 s on 197 notes. The original failure mode (passes exceeding the timeout and dropping the session) does not recur.",
|
||||
"Write integrity holds. PUT verifies via read-back at 115 ms; APPEND is whole-line idempotent, skipping at 51 ms with zero duplicate writes on re-run. The advisory lock fast-fails a contended acquire with the expected exit 75."
|
||||
]
|
||||
},
|
||||
{
|
||||
"heading": "Recommendations",
|
||||
"type": "olist",
|
||||
"items": [
|
||||
"Apply read_many to every code path that reads a set of known notes. Done for expand_graph (gated). Remaining: prefetch the _brief result bodies recall prints, and parallelize rebuild()'s vault walk.",
|
||||
"Reuse one in-process read cache across a recall() call so load_index, expansion, and _brief don't re-fetch overlapping notes \u2014 this closes the remaining ~500 ms + serial _brief cost.",
|
||||
"Make 'prefer read_many over a GET loop' a documented contract in the plugin's API reference; serial multi-note reads are a performance bug. The expand-graph gate enforces it for recall.",
|
||||
"Hold CHORUS_WORKERS at 8 (16 buys only 1.24x). Promote the resolve correctness table from INFO to FAIL now that live behavior is confirmed. Re-baseline at the next 1.x release or past ~400 notes."
|
||||
]
|
||||
},
|
||||
{
|
||||
"heading": "Method",
|
||||
"type": "text",
|
||||
"body": [
|
||||
"Each metric was timed with a perf-counter harness that reuses the live CHORUS client, so it exercises the real pooled and concurrent code path. Reads report median and p95 over repeated iterations after a discarded warm-up; write metrics ran in a disposable namespace under the advisory lock and were deleted on completion (zero residual files confirmed).",
|
||||
"Gates use relative ratios rather than absolute milliseconds, so they stay valid regardless of the network the suite runs from. The harness, fixtures, baselines, and per-metric JSON are checked in under eval/perf/."
|
||||
]
|
||||
}
|
||||
],
|
||||
"note": {
|
||||
"heading": "Next Step",
|
||||
"body": "Prefetch recall()'s _brief reads and parallelize rebuild() via read_many, then re-run subject-pull to confirm recall drops toward ~1 s. expand_graph fix is local-only until ported to the canonical repo. No blockers in 1.1.0 / 1.2.0."
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
[20:14:00] read-full
|
||||
@@ -0,0 +1,12 @@
|
||||
#!/bin/bash
|
||||
export CHORUS_KEY_LEGACY_OK=1
|
||||
cd "$(dirname "$0")"
|
||||
{
|
||||
chorus "[$(date +%T)] read-full"; python3 bench.py read-full -n 3 --quiet --json-out results/m-read-full.json
|
||||
chorus "[$(date +%T)] worker-sweep"; python3 bench.py worker-sweep -n 4 --quiet --json-out results/m-worker-sweep.json
|
||||
chorus "[$(date +%T)] subject-pull"; python3 bench.py subject-pull -n 4 --quiet --json-out results/m-subject-pull.json
|
||||
chorus "[$(date +%T)] index"; python3 bench.py index --quiet --json-out results/m-index.json
|
||||
chorus "[$(date +%T)] lint"; python3 bench.py lint --quiet --json-out results/m-lint.json
|
||||
chorus "[$(date +%T)] DONE"
|
||||
} > results/run.log 2>&1
|
||||
touch results/DONE
|
||||
@@ -0,0 +1,244 @@
|
||||
# CHORUS — Obsidian Local REST API Reference
|
||||
|
||||
Server: the configured endpoint — the `endpoint` from `~/.claude/chorus-memory/config.json` (overridable with `CHORUS_BASE`), a reverse proxy → backend Obsidian Local REST API. Examples below use `$CHORUS_BASE` (or the neutral placeholder `https://obsidian.example.com`).
|
||||
Auth header: `Authorization: Bearer $CHORUS_KEY` — `export CHORUS_KEY=<token>` before running these recipes, or let `chorus.py` resolve it automatically (per-field, first wins: env override `CHORUS_KEY` → the `key` in `~/.claude/chorus-memory/config.json`, resolved via the `chorus_config` module). The literal key is never stored in docs.
|
||||
A configured endpoint should present a **valid TLS certificate** — `-k` is not required. Paths address the vault at its **root**.
|
||||
|
||||
> **Prefer `scripts/chorus.py` over the raw recipes below.** It wraps every verb with auth, status checking, retry, idempotent append, and frontmatter patches, and runs on any platform with a Python interpreter. The `curl` recipes here are the underlying mechanics and a **\*nix/bash fallback** (they use heredocs and `--data-binary`); on Windows, prefer `chorus.py` or an equivalent. **If you call `curl` directly, check the HTTP status** — add `-o /dev/null -w "%{http_code}"` and branch on it. A `PATCH` to a non-existent heading returns `400 invalid-target` (errorCode 40080) and the write is *silently lost*; a bare `curl` that ignores status will report success anyway. `GET` returns `404` for a missing file. Treat any `>= 400` as a failed operation, surface it, and do not continue as if it succeeded.
|
||||
|
||||
---
|
||||
|
||||
## Client performance (chorus.py) — pooling & concurrency
|
||||
|
||||
`chorus.request()` keeps **one persistent keep-alive connection per thread** and reuses it across calls, instead of opening a fresh TCP+TLS connection per request. Against a remote HTTPS endpoint the repeated handshake was the dominant cost, so a full-vault pass used to make hundreds of serial handshakes and exceed the agent/tool timeout (which kills the run mid-flight). A stale pooled connection reconnects transparently; the `(status, body)` contract is unchanged (status `0` still means transport failure, so the offline write-queue still triggers).
|
||||
|
||||
For bulk reads, **`chorus.read_many(paths)`** fans GETs across a thread pool and returns `{path: text_or_None}` — resilient (one unreadable file becomes `None` rather than aborting the batch). `sweep.py` and `vault_lint.py` prefetch the whole vault once with it and share that single cache across every pass, so no note is fetched more than once. Net effect: a several-minute full-vault sweep becomes sub-second on a few-hundred-note vault.
|
||||
|
||||
Tuning knobs (env overrides):
|
||||
|
||||
- `CHORUS_TIMEOUT` (default `30`) — per-request socket timeout in seconds. With concurrency, one slow file only blocks its own worker, not the run.
|
||||
- `CHORUS_WORKERS` (default `8`) — `read_many` thread-pool size. Raise it for very large vaults; lower it to be gentler on the backend.
|
||||
|
||||
For a vault large enough that even the concurrent pass approaches the tool timeout, run the script with the harness's background-execution option rather than blocking the turn.
|
||||
|
||||
---
|
||||
|
||||
## Reading Files
|
||||
|
||||
```bash
|
||||
# Read any file by vault path
|
||||
curl -s \
|
||||
-H "Authorization: Bearer $CHORUS_KEY" \
|
||||
"$CHORUS_BASE/vault/_agent/context/current-context.md"
|
||||
```
|
||||
|
||||
Returns raw file content (text/markdown). On 404, the file does not exist.
|
||||
|
||||
```bash
|
||||
# Read a specific heading's content only
|
||||
curl -s \
|
||||
-H "Authorization: Bearer $CHORUS_KEY" \
|
||||
"$CHORUS_BASE/vault/_agent/memory/semantic/operator-preferences.md/heading/Operator"
|
||||
```
|
||||
|
||||
Nested headings: separate levels with `::` (URL-encode spaces as `%20`):
|
||||
```
|
||||
/vault/path/to/note.md/heading/Work%3A%3AMeetings
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Listing Directories
|
||||
|
||||
```bash
|
||||
# List contents of a directory (trailing slash required)
|
||||
curl -s \
|
||||
-H "Authorization: Bearer $CHORUS_KEY" \
|
||||
"$CHORUS_BASE/vault/_agent/sessions/"
|
||||
```
|
||||
|
||||
Returns JSON: `{ "files": [...], "folders": [...] }`.
|
||||
|
||||
---
|
||||
|
||||
## Appending Content (POST)
|
||||
|
||||
`POST` appends to the **end** of an existing file. Creates the file if it doesn't exist.
|
||||
|
||||
```bash
|
||||
cat > /tmp/obs_entry.md << 'OBSEOF'
|
||||
- 2026-06-05: your entry here
|
||||
OBSEOF
|
||||
|
||||
curl -s -X POST \
|
||||
-H "Authorization: Bearer $CHORUS_KEY" \
|
||||
-H "Content-Type: text/markdown" \
|
||||
--data-binary @/tmp/obs_entry.md \
|
||||
"$CHORUS_BASE/vault/inbox/captures/inbox.md"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Creating or Overwriting Files (PUT)
|
||||
|
||||
`PUT` creates a new file or fully overwrites an existing one. Intermediate directories are created automatically.
|
||||
|
||||
```bash
|
||||
cat > /tmp/obs_file.md << 'OBSEOF'
|
||||
---
|
||||
type: session-log
|
||||
status: complete
|
||||
created: 2026-06-05
|
||||
updated: 2026-06-05
|
||||
tags: [agent, session]
|
||||
agent_written: true
|
||||
source_notes: []
|
||||
---
|
||||
|
||||
# content here
|
||||
|
||||
## Related
|
||||
- [[projects/active/some-project]]
|
||||
OBSEOF
|
||||
|
||||
curl -s -X PUT \
|
||||
-H "Authorization: Bearer $CHORUS_KEY" \
|
||||
-H "Content-Type: text/markdown" \
|
||||
--data-binary @/tmp/obs_file.md \
|
||||
"$CHORUS_BASE/vault/_agent/sessions/2026-06-05-1430-my-session.md"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Patching a Specific Section (PATCH)
|
||||
|
||||
`PATCH` edits inside a file without rewriting it. Target and operation are set via headers.
|
||||
|
||||
### Append under a heading
|
||||
|
||||
**Heading targets must be the FULL heading path, `::`-delimited from the top-level heading.** A bare subheading name returns `400 invalid-target` (errorCode 40080). Example: `## Fact / Pattern` nested under `# Operator Preferences` → `Target: Operator Preferences::Fact / Pattern`. Percent-encode non-ASCII characters (e.g. `H%C3%A9llo`); spaces are fine.
|
||||
|
||||
```bash
|
||||
cat > /tmp/obs_patch.md << 'OBSEOF'
|
||||
The operator prefers concise status updates — lead with the decision.
|
||||
OBSEOF
|
||||
|
||||
curl -s -X PATCH \
|
||||
-H "Authorization: Bearer $CHORUS_KEY" \
|
||||
-H "Operation: append" \
|
||||
-H "Target-Type: heading" \
|
||||
-H "Target: Operator Preferences::Fact / Pattern" \
|
||||
-H "Content-Type: text/markdown" \
|
||||
--data-binary @/tmp/obs_patch.md \
|
||||
"$CHORUS_BASE/vault/_agent/memory/semantic/operator-preferences.md"
|
||||
```
|
||||
|
||||
### Discover heading / block / frontmatter targets
|
||||
|
||||
When unsure of the exact heading path, GET the note with the document-map Accept header:
|
||||
|
||||
```bash
|
||||
curl -s \
|
||||
-H "Authorization: Bearer $CHORUS_KEY" \
|
||||
-H "Accept: application/vnd.olrapi.document-map+json" \
|
||||
"$CHORUS_BASE/vault/_agent/memory/semantic/operator-preferences.md"
|
||||
```
|
||||
|
||||
Returns `{ "headings": [...], "blocks": [...], "frontmatterFields": [...] }`. Copy the heading string verbatim into `Target`.
|
||||
|
||||
### Replace a heading's content entirely
|
||||
|
||||
Same call with `Operation: replace` — e.g. to refresh a project's `Project Name::Status`.
|
||||
|
||||
### Prepend under a heading
|
||||
|
||||
Same call with `Operation: prepend`.
|
||||
|
||||
### Patch a frontmatter field
|
||||
|
||||
> A `PATCH` `Target-Type: frontmatter` **replace** on a key the note does not have returns
|
||||
> `400 invalid-target` — the raw API cannot create a key. `chorus.py fm` (v1.5+) handles this:
|
||||
> on that 400 it surgically inserts the one `field: value` line into the frontmatter block
|
||||
> and re-PUTs (verified), leaving every other line untouched. Raw-curl users must GET-edit-PUT
|
||||
> themselves — carefully (a naive rewrite is how frontmatter gets clobbered).
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH \
|
||||
-H "Authorization: Bearer $CHORUS_KEY" \
|
||||
-H "Operation: replace" \
|
||||
-H "Target-Type: frontmatter" \
|
||||
-H "Target: updated" \
|
||||
-H "Content-Type: application/json" \
|
||||
--data '"2026-06-05"' \
|
||||
"$CHORUS_BASE/vault/projects/active/vault-foundation.md"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Searching
|
||||
|
||||
```bash
|
||||
curl -s -X POST \
|
||||
-H "Authorization: Bearer $CHORUS_KEY" \
|
||||
"$CHORUS_BASE/search/simple/?query=weekly+review"
|
||||
```
|
||||
|
||||
Returns an array of `{ filename, score, matches: [{ context, match }] }`.
|
||||
|
||||
---
|
||||
|
||||
## Deleting Files
|
||||
|
||||
```bash
|
||||
curl -s -X DELETE \
|
||||
-H "Authorization: Bearer $CHORUS_KEY" \
|
||||
"$CHORUS_BASE/vault/inbox/imports/old-note.md"
|
||||
```
|
||||
|
||||
Only on explicit operator request. Deletion is destructive.
|
||||
|
||||
> **DELETE removes files only — never directories.** Deleting the last file in a folder
|
||||
> leaves the empty folder orphaned on disk; the REST listing then 404s (nothing left to
|
||||
> list), which falsely reads as "the tree is gone", but Obsidian's file explorer still
|
||||
> shows it. After removing a folder's last file, tell the operator the empty folder
|
||||
> needs manual deletion in Obsidian — do not claim the tree is gone from a 404 listing.
|
||||
> (Learned 2026-07-03 removing the retired `archive/` and `_agent/outputs/` trees.)
|
||||
|
||||
---
|
||||
|
||||
## URL-Encoding Notes
|
||||
|
||||
- Path separators (`/`) in the vault path are **literal** — do not encode them.
|
||||
- Spaces in filenames or heading targets in a URL: use `%20`.
|
||||
- Nested heading levels in a URL path: use `%3A%3A` for `::`.
|
||||
- Heading text in the `Target:` header: the **full heading path** joined by `::` (e.g. `Operator Preferences::Fact / Pattern`), no `#`; spaces are fine; percent-encode non-ASCII.
|
||||
|
||||
---
|
||||
|
||||
## Memory Routing Map
|
||||
|
||||
| Situation | Vault path | Method |
|
||||
|-----------|-----------|--------|
|
||||
| Quick capture / unsorted | `inbox/captures/inbox.md` (date-prefixed line) | POST |
|
||||
| Raw imported material | `inbox/imports/` | PUT |
|
||||
| Operator preference / durable fact | `_agent/memory/semantic/operator-preferences.md` | PATCH |
|
||||
| Other durable facts, patterns | `_agent/memory/semantic/<slug>.md` | PUT |
|
||||
| Event record (what happened) | `_agent/memory/episodic/<slug>.md` | PUT |
|
||||
| Short-lived, time-boxed state | `_agent/memory/working/<slug>.md` | PUT |
|
||||
| Task-scoped context / focus | `_agent/context/current-context.md` | PATCH / PUT |
|
||||
| Working-session log | `_agent/sessions/YYYY-MM-DD-HHMM-<slug>.md` | PUT |
|
||||
| Long-running project state | `projects/<lifecycle>/<slug>.md` (lifecycle: `incubating` → `active` → `on-hold`/`archived`; folder and `status:` MUST agree) | PUT + PATCH |
|
||||
| Ongoing area of responsibility (standing domain, no end state) | `areas/<domain>/<slug>.md` (`<domain>`: `business`/`personal`/`learning`/`systems`) | PUT |
|
||||
| Non-obvious decision (ADR) | `decisions/by-date/YYYY-MM-DD-<slug>.md` (mirror only into an existing project note's `## Key Decisions`; otherwise skip) | PUT |
|
||||
| Person context | `resources/people/<name>.md` | PUT / PATCH |
|
||||
| Company / organization context | `resources/companies/<slug>.md` | PUT / PATCH |
|
||||
| Concept / reference note | `resources/concepts/` or `resources/references/` | PUT |
|
||||
| Meeting notes / call recap | `resources/meetings/YYYY-MM-DD-<slug>.md` | PUT |
|
||||
| Skill / plugin capability entry (catalog, not build work) | `_agent/skills/active/<slug>.md` (→ `archived/` when retired) | PUT |
|
||||
| Daily activity / Agent Log | `journal/daily/YYYY-MM-DD.md` | POST / PATCH |
|
||||
| Journal rollup | `journal/{weekly/YYYY-Www,monthly/YYYY-MM,quarterly/YYYY-Qn,annual/YYYY}.md` (weekly = opt-in on first session of a new ISO week; monthly = offered with Vault Health; quarterly/annual = manual) | PUT |
|
||||
| Vault-health audit (agent self-maintenance) | `_agent/health/YYYY-MM-vault-health.md` (monthly; NOT a journal entry) | PUT |
|
||||
| Session-end orientation pointer | `_agent/heartbeat/last-session.md` (one line, overwritten each session end) | PUT |
|
||||
| Bootstrap marker (plugin-owned) | `_agent/chorus-vault.md` (`schema_version`, bootstrap date) — the "is this vault set up?" probe | GET / PUT |
|
||||
|
||||
**Slug rules:** kebab-case, ASCII, ~40 chars max. Every file carries canonical frontmatter (see `vault-layout.md`).
|
||||
@@ -0,0 +1,142 @@
|
||||
# Bootstrap & Repair
|
||||
|
||||
The **plugin is the single source of truth** for CHORUS's structure. Everything needed to stand up a vault ships in this skill under `scaffold/` — there is no dependency on any in-vault control doc and no external/local re-seed path. This makes the vault portable: point the REST API at any empty Obsidian vault, run this procedure, and it becomes a working CHORUS vault.
|
||||
|
||||
The vault holds **data only**. It carries no `CLAUDE.md` / `BOOTSTRAP.md` / `STRUCTURE.md` / `index.md`. The "is this vault set up?" signal is a small marker file, `_agent/chorus-vault.md`.
|
||||
|
||||
## Quick path — run the scripts
|
||||
|
||||
Bootstrap, repair, and migration are deterministic scripts; prefer them over running the curl steps by hand. They resolve the scaffold relative to their own location, so they work regardless of the caller's CWD:
|
||||
|
||||
```bash
|
||||
SCRIPTS="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scripts"
|
||||
python3 "$SCRIPTS/bootstrap.py" --dry-run # preview what would be seeded
|
||||
python3 "$SCRIPTS/bootstrap.py" # idempotent, additive — fills only what is missing (also the repair path)
|
||||
python3 "$SCRIPTS/migrate.py" # plan a schema migration (dry-run)
|
||||
python3 "$SCRIPTS/migrate.py" --apply # perform the migration (moves/deletes, after review)
|
||||
# Windows: use `python` or `py -3` if `python3` is not on PATH.
|
||||
```
|
||||
|
||||
`bootstrap.py` writes through `chorus.py`, so every step is status-checked and the marker is written last. The scripts are pure Python (only an interpreter required — no bash, no platform-specific `date`), so they run identically on Windows, macOS, and Linux. The manual steps below document what the script does (and serve as a fallback if the script can't run).
|
||||
|
||||
```
|
||||
AUTH="Authorization: Bearer $CHORUS_KEY" # export CHORUS_KEY first; chorus.py resolves it automatically via chorus_config
|
||||
BASE="$CHORUS_BASE" # the endpoint from ~/.claude/chorus-memory/config.json (override: CHORUS_BASE)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Probe — is the vault bootstrapped?
|
||||
|
||||
At session start, GET the marker:
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w "%{http_code}" -H "$AUTH" "$BASE/vault/_agent/chorus-vault.md"
|
||||
```
|
||||
|
||||
- **200** → bootstrapped. Read the marker's `schema_version`; if it is **less than** the plugin's current schema (4), run a migration pass (see *Migrations* below), otherwise proceed straight to the loading procedure in `SKILL.md`.
|
||||
- **404** → empty/unconfigured vault. Run **Fresh bootstrap** below. (If you expected an existing vault, confirm with the operator once that the REST API is pointed at the right vault before seeding.)
|
||||
|
||||
---
|
||||
|
||||
## Fresh bootstrap (empty vault)
|
||||
|
||||
Idempotent and additive. **Never overwrite an existing file** — GET-probe each path and skip any that already returns `200`. The marker is written **last**, so the vault is only flagged "set up" after every piece is in place.
|
||||
|
||||
Throughout, `{{DATE}}` means today's date (`YYYY-MM-DD`), resolved from the conversation's `currentDate`. Substitute it before PUTting any scaffold file or anchor seed.
|
||||
|
||||
### 1. Folder tree
|
||||
|
||||
The REST API auto-creates parent directories on PUT, so creating the tree = seeding a file into each leaf. Obsidian and git both ignore empty dirs, so drop a one-line `README.md` into any leaf that wouldn't otherwise receive a file. Required tree (leaves marked `→ README`):
|
||||
|
||||
```
|
||||
inbox/captures inbox/imports inbox/processing-log
|
||||
journal/daily journal/weekly journal/monthly journal/quarterly journal/annual journal/templates
|
||||
projects/active projects/incubating projects/on-hold projects/archived
|
||||
areas/business areas/personal areas/learning areas/systems
|
||||
resources/concepts resources/references resources/people resources/companies resources/meetings
|
||||
decisions/by-date
|
||||
_agent/context _agent/memory/working _agent/memory/episodic _agent/memory/semantic
|
||||
_agent/sessions _agent/health _agent/templates _agent/heartbeat
|
||||
_agent/skills/active _agent/skills/archived
|
||||
```
|
||||
|
||||
> `decisions/by-project/` is intentionally **not** created — it is retired. A project-relevant decision is mirrored as a `[[wikilink]]` under that project's `## Key Decisions` heading instead.
|
||||
>
|
||||
> `reviews/` is **not** created — it is retired. Journal rollups (weekly/monthly/quarterly/annual) live under `journal/`; the monthly vault-health audit lives under `_agent/health/`.
|
||||
|
||||
A leaf README is just a one-liner, e.g.:
|
||||
|
||||
```bash
|
||||
printf '# %s\n\nMemory vault folder. See the chorus-memory plugin for conventions.\n' "captures" \
|
||||
| curl -s -X PUT -H "$AUTH" -H "Content-Type: text/markdown" --data-binary @- \
|
||||
"$BASE/vault/inbox/captures/README.md"
|
||||
```
|
||||
|
||||
### 2. Templates
|
||||
|
||||
PUT every file under this skill's `scaffold/templates/` to its mirrored vault path. The tree mirrors 1:1:
|
||||
|
||||
| Scaffold file | Vault path |
|
||||
|---|---|
|
||||
| `scaffold/templates/_agent/templates/session-log-template.md` | `_agent/templates/session-log-template.md` |
|
||||
| `scaffold/templates/_agent/templates/context-bundle-template.md` | `_agent/templates/context-bundle-template.md` |
|
||||
| `scaffold/templates/_agent/templates/working-memory-template.md` | `_agent/templates/working-memory-template.md` |
|
||||
| `scaffold/templates/_agent/templates/semantic-memory-template.md` | `_agent/templates/semantic-memory-template.md` |
|
||||
| `scaffold/templates/journal/templates/daily-note-template.md` | `journal/templates/daily-note-template.md` |
|
||||
| `scaffold/templates/journal/templates/weekly-review-template.md` | `journal/templates/weekly-review-template.md` |
|
||||
| `scaffold/templates/projects/project-template.md` | `projects/project-template.md` |
|
||||
| `scaffold/templates/decisions/decision-template.md` | `decisions/decision-template.md` |
|
||||
|
||||
Templates keep their Obsidian Templater tokens (`{{date:YYYY-MM-DD}}` etc.) verbatim — those are resolved by Templater / by the skill's daily-note routine, not at seed time.
|
||||
|
||||
Resolve scaffold paths against the skill directory — **never a bare relative `@scaffold/...`**, which assumes the caller's CWD is the skill dir and silently sends an empty body otherwise:
|
||||
|
||||
```bash
|
||||
SCAFFOLD="${CLAUDE_PLUGIN_ROOT}/skills/chorus-memory/scaffold"
|
||||
curl -s -X PUT -H "$AUTH" -H "Content-Type: text/markdown" \
|
||||
--data-binary @"$SCAFFOLD/templates/journal/templates/daily-note-template.md" \
|
||||
"$BASE/vault/journal/templates/daily-note-template.md"
|
||||
```
|
||||
|
||||
### 3. Anchor seeds (only if absent — no fabricated facts)
|
||||
|
||||
Substitute `{{DATE}}` → today, then PUT each:
|
||||
|
||||
| Scaffold file | Vault path |
|
||||
|---|---|
|
||||
| `scaffold/anchors/operator-preferences.seed.md` | `_agent/memory/semantic/operator-preferences.md` |
|
||||
| `scaffold/anchors/current-context.seed.md` | `_agent/context/current-context.md` |
|
||||
| `scaffold/anchors/inbox.seed.md` | `inbox/captures/inbox.md` |
|
||||
|
||||
The operator-preferences seed is deliberately empty — **do not invent preferences.** Real facts accrue through use.
|
||||
|
||||
### 4. Vault README (human signpost)
|
||||
|
||||
Substitute `{{DATE}}` if present, then PUT `scaffold/README.vault.md` → `/vault/README.md`. This is the only human-facing control doc in the vault and is **not** read by the agent for routing.
|
||||
|
||||
### 5. Marker (write last)
|
||||
|
||||
Substitute `{{DATE}}`, then PUT `scaffold/chorus-vault.md` → `/vault/_agent/chorus-vault.md`. Once this returns `200`, the vault is bootstrapped.
|
||||
|
||||
### 6. First-run trace
|
||||
|
||||
Create today's daily note from `journal/templates/daily-note-template.md` (resolve the `{{date:…}}` tokens to today), append a one-line `## Agent Log` entry noting the bootstrap, and write a session log in `_agent/sessions/YYYY-MM-DD-HHMM-bootstrap.md`. Tell the operator briefly what was created.
|
||||
|
||||
---
|
||||
|
||||
## Repair (existing vault, something missing)
|
||||
|
||||
Run the same steps 1–5, but GET-probe each path first and **only create what is missing**. Never overwrite. If a file exists but looks malformed, flag it in the session log rather than replacing it. Repair is safe to run any time and is the right response if the loading procedure hits an unexpected `404` on a structural path.
|
||||
|
||||
---
|
||||
|
||||
## Migrations (`schema_version` mismatch)
|
||||
|
||||
`scripts/migrate.py` automates this: it reads the marker's `schema_version`, applies each intervening migration (idempotent, additive; destructive steps gated behind `--apply` and printed first), and stamps the marker at the end. Run it dry-run, review the plan, then `--apply`. The steps below are what it encodes — and the manual fallback.
|
||||
|
||||
When the marker's `schema_version` is older than the plugin's, apply the migration steps for each intervening version, then PATCH the marker's `schema_version` frontmatter to the new value.
|
||||
|
||||
- **0 → 1** (control-docs-in-plugin): the vault previously carried root control docs (`CLAUDE.md`, `BOOTSTRAP.md`, `STRUCTURE.md`, `index.md`). Back them up outside the vault, DELETE them, PUT the thin `scaffold/README.vault.md` over the old verbose `README.md`, write the `_agent/chorus-vault.md` marker, and scrub now-dangling `[[CLAUDE]]`/`[[BOOTSTRAP]]`/`[[STRUCTURE]]`/`[[index]]` links from the `## Related` sections of `operator-preferences.md` and `current-context.md` (leave historical session logs alone). Confirm with the operator before deleting.
|
||||
- **1 → 2** (reviews-folded-into-journal): the `reviews/` tree is retired. (a) For each note under `reviews/weekly/` and `reviews/monthly/`, MOVE it into `journal/weekly/` (rename `YYYY-Www-review.md` → `YYYY-Www.md`) and `journal/monthly/` respectively, preserving the earliest `created:`. (b) Move any `reviews/monthly/YYYY-MM-vault-health.md` to `_agent/health/`. (c) Move `reviews/quarterly|annual/` artifacts to `journal/quarterly|annual/`. (d) Update inbound `[[reviews/...]]` wikilinks in `## Related` sections to the new paths. (e) DELETE the now-empty `reviews/` tree. Confirm with the operator before deleting; leave historical session logs alone. *(This step covers any vault bootstrapped under schema 1.)*
|
||||
- **2 → 3** (entity index + cross-linking): adds the `_agent/index/` folder for the machine-maintained entity registry. `migrate.py` creates the folder; the **data back-fill is `sweep.py`** — after migrating, run `python3 scripts/sweep.py --apply` to build `_agent/index/entities.json` from the notes already in the vault and to symmetrize existing `## Related` cross-links (it adds only the missing reciprocal direction, never invents links). `sweep.py` is idempotent and dry-run by default; re-run it any time `/chorus-health` reports index drift.
|
||||
@@ -0,0 +1,45 @@
|
||||
# CHORUS — Operating Contract
|
||||
|
||||
The durable, client-independent contract for any agent operating against the CHORUS vault. These principles and safety rules formerly lived in the vault's `CLAUDE.md`; they now live in the plugin so they survive regardless of what is (or isn't) in the vault. Day-to-day *procedure* — loading order, search-first, triage, scope switching, PATCH/append rules — is owned by `SKILL.md`. This file holds the things that don't change between sessions or clients.
|
||||
|
||||
## What this agent is
|
||||
|
||||
You are an agent operating against an Obsidian vault that functions as a shared, long-term memory substrate for human work, Claude Code / CoWork sessions, and other REST API clients. Your role is to read context, synthesize across notes, produce structured outputs, update memory carefully, and leave durable traces in the vault rather than keeping important state only in the conversation. The vault is the **system of record**, not a scratchpad.
|
||||
|
||||
## Core principles
|
||||
|
||||
- Treat the vault as the system of record for long-term memory.
|
||||
- Prefer Markdown, YAML frontmatter, wiki-links, and stable folder conventions over proprietary structures.
|
||||
- Write notes so both humans and agents can understand them without hidden context.
|
||||
- Preserve history instead of silently overwriting important decisions, summaries, or inferred preferences.
|
||||
- Keep agent-managed content (`agent_written: true` + `source_notes`) clearly separated from human-authored content.
|
||||
- Default to **additive updates, explicit status changes, and traceable summaries.**
|
||||
- Optimize for portability: the same vault should work across Claude Code, CoWork, and any REST API client.
|
||||
|
||||
## Memory model (where things live)
|
||||
|
||||
- **Working** → `_agent/memory/working/` — transient, time-boxed.
|
||||
- **Episodic** → `_agent/memory/episodic/` — what happened, when.
|
||||
- **Semantic** → `_agent/memory/semantic/` — durable facts, patterns, preferences (`operator-preferences.md`).
|
||||
- **Context bundles** → `_agent/context/` — task-focused reading lists and active state.
|
||||
|
||||
## Safety rules
|
||||
|
||||
- Do not fabricate facts, relationships, or prior decisions.
|
||||
- Do not mass-restructure the vault unless explicitly asked.
|
||||
- Do not delete notes unless deletion is explicitly requested and clearly safe.
|
||||
- Do not store secrets or API keys inside the vault, and never write the API key into a vault note.
|
||||
- Default to additive updates and explicit status changes over destructive edits.
|
||||
|
||||
## REST/API readiness
|
||||
|
||||
Assume clients may operate without filesystem access, through the Obsidian Local REST API. Keep paths predictable, frontmatter parseable, titles stable, and all state stored in notes rather than hidden conversation memory. Structure must stay portable: a fresh, empty vault is brought fully online by the plugin's bootstrap (`references/bootstrap.md`), so nothing essential should depend on files existing in the vault ahead of time.
|
||||
|
||||
## Concurrency (shared vault)
|
||||
|
||||
The vault is a **shared** substrate — Claude Code, CoWork, and other REST API clients may operate on it concurrently. The REST API offers no transactions, so writers coordinate cooperatively:
|
||||
|
||||
- Single-line, overwrite-style files (`_agent/heartbeat/last-session.md`, `current-context.md::Scope`) and append targets (`inbox.md`, `## Agent Log`) assume **one writer at a time**. Two sessions writing at once can clobber or duplicate.
|
||||
- Before a burst of writes in a session that may overlap another, take the advisory lock (`chorus.py lock <id>` → `_agent/locks/vault.lock`) and release it at session end. The lock is read-back-confirmed and cooperative with a TTL (stale locks are reclaimable); it is a courtesy, not a hard mutex.
|
||||
- Idempotent append (read-before-POST, whole-line match, via `chorus.py append`) is the second line of defense against duplicate lines from retries or overlapping sessions.
|
||||
- Status-check every write. A write that returns `>= 400` did **not** land — surface it rather than assuming success.
|
||||
@@ -0,0 +1,128 @@
|
||||
# CHORUS Routing Map
|
||||
|
||||
**This document is canonical and complete.** Every write destination in the vault appears here exactly once, with the condition that routes content to it, what lands there, and why it is distinct from its neighbours. The rule for the whole system: **if a path is not in this map, nothing is written to it.** A path that cannot justify its separateness from a neighbour is a merge candidate, not a valid destination.
|
||||
|
||||
Views of the same truth: `scripts/routing.json` is the **machine-readable canonical source** (one route per destination, as a regex pattern + trigger + reason); `vault_lint.py` loads it and flags any vault path that matches no route (and any write to a retired path). This prose map is the human-readable authority and must stay in sync with `routing.json` — `scripts/check_routing.py` enforces that mechanically (every path here matches a route, every route appears here). The `SKILL.md` *Where to Write* table is the quick-reference and `vault-layout.md` is the physical tree. When they disagree, `routing.json` + this map win and the others are fixed to match.
|
||||
|
||||
## How to read a row
|
||||
|
||||
- **Trigger** — the observable condition that sends content here. If two rows could both fire, the more specific trigger wins.
|
||||
- **What lands** — the unit of content written.
|
||||
- **Distinct because** — the one reason this path is not merged into its nearest neighbour. This is the load-bearing column; a row without it is a bug.
|
||||
- **Method** — the dominant REST verb (see `api-reference.md` for mechanics).
|
||||
|
||||
---
|
||||
|
||||
## inbox/ — unsorted intake
|
||||
|
||||
| Path | Trigger | What lands | Distinct because | Method |
|
||||
|------|---------|------------|------------------|--------|
|
||||
| `inbox/captures/inbox.md` | "Save this, sort later" — destination genuinely unknown at capture time | One date-prefixed line | The only path whose contract is *deferred routing*; everything else here has a known home | POST |
|
||||
| `inbox/imports/<slug>.md` | Raw external material dropped in wholesale (export, paste, dump) | The raw artifact, unedited | Holds un-triaged *bulk*, vs `captures` which holds single lines | PUT |
|
||||
| `inbox/processing-log/YYYY-MM-DD.md` | An inbox item is routed to its real home | One line: `<original> → <destination path>` | Audit trail of moves — never holds memory itself, only the record of routing | POST |
|
||||
|
||||
Captures and imports are temporary by contract. Triage drains them into the homes below and logs the move; the original is left until the operator okays deletion.
|
||||
|
||||
## journal/ — the one append-only time-series stream
|
||||
|
||||
Rollups are coarser-grained journal entries over the same timeline, so they live in the same tree. There is no separate `reviews/` tree.
|
||||
|
||||
| Path | Trigger | What lands | Distinct because | Method |
|
||||
|------|---------|------------|------------------|--------|
|
||||
| `journal/daily/YYYY-MM-DD.md` | First agent activity on a given day | `## Agent Log` line(s) + day notes | Finest grain; the only journal note PATCHed repeatedly within its period | PATCH (auto-create) |
|
||||
| `journal/weekly/YYYY-Www.md` | First substantive session of a new ISO week — **opt-in**, offer first | Light digest: open `active/` threads, aging inbox, week's scope changes | Coarser than daily, finer than monthly; ISO-week grain | PUT |
|
||||
| `journal/monthly/YYYY-MM.md` | First substantive session of a new month — offered alongside Vault Health | Month digest, same shape as weekly | Month grain; separate cadence and trigger from weekly | PUT |
|
||||
| `journal/quarterly/YYYY-Qn.md` | **Manual / on request only** | Quarter-scale narrative review | Strategic grain; never auto-fires | PUT |
|
||||
| `journal/annual/YYYY.md` | **Manual / on request only** | Year-scale narrative review | Coarsest grain; never auto-fires | PUT |
|
||||
| `journal/templates/` | Bootstrap only (seeded from plugin masters) | Note templates | Holds templates, not journal content; never a runtime routing target | PUT (seed) |
|
||||
|
||||
## projects/ — work with an end state
|
||||
|
||||
Lifecycle folders; `status:` frontmatter MUST equal the folder name (the linter checks this).
|
||||
|
||||
| Path | Trigger | What lands | Distinct because | Method |
|
||||
|------|---------|------------|------------------|--------|
|
||||
| `projects/active/<slug>.md` | Work in motion now | Project note (Status PATCHed fresh) | Default state for anything being worked | PUT + PATCH |
|
||||
| `projects/incubating/<slug>.md` | Idea captured, work not started | Project note | Pre-work; promote to `active/` when work begins | PUT |
|
||||
| `projects/on-hold/<slug>.md` | Paused but still tracked | Project note | Resumable; distinct from `archived` (which is terminal) | PUT |
|
||||
| `projects/archived/<slug>.md` | Done, abandoned, or rolled up | Project note | Terminal; kept for history, never deleted | PUT |
|
||||
|
||||
**Project vs area:** has an end state → `projects/`. Never "done" → `areas/`.
|
||||
|
||||
## areas/ — standing responsibilities, no finish line
|
||||
|
||||
| Path | Trigger | What lands | Distinct because | Method |
|
||||
|------|---------|------------|------------------|--------|
|
||||
| `areas/<domain>/<slug>.md` | Ongoing domain the operator maintains indefinitely (`<domain>`: business/personal/learning/systems) | Area note | No end state — the one thing that disqualifies it from `projects/` | PUT |
|
||||
|
||||
## resources/ — reference material about the world
|
||||
|
||||
| Path | Trigger | What lands | Distinct because | Method |
|
||||
|------|---------|------------|------------------|--------|
|
||||
| `resources/people/<name>.md` | A fact about a specific person | Person note (kebab-case slug) | Keyed to a person, not a topic or event | PUT / PATCH |
|
||||
| `resources/companies/<slug>.md` | A fact about an organization (client, vendor, partner, employer) | Company note (kebab-case slug) | Keyed to an organization, not an individual (`people/`) or an external source (`references/`) | PUT / PATCH |
|
||||
| `resources/concepts/<slug>.md` | A reusable concept/idea the operator wants on file | Concept note | An idea, vs a `reference` which is an external source | PUT |
|
||||
| `resources/references/<slug>.md` | An external source/link worth keeping | Reference note | Points outward (a source), vs `concepts` (an idea) | PUT |
|
||||
| `resources/meetings/YYYY-MM-DD-<slug>.md` | Notes/recap tied to a specific meeting or call | Meeting note; mirror decisions to `decisions/by-date/`, commitments to the project | Event-anchored to a meeting, vs a project's ongoing thread | PUT |
|
||||
|
||||
## decisions/ — non-obvious decisions (ADR)
|
||||
|
||||
| Path | Trigger | What lands | Distinct because | Method |
|
||||
|------|---------|------------|------------------|--------|
|
||||
| `decisions/by-date/YYYY-MM-DD-<slug>.md` | A non-obvious decision worth recording | ADR: Context → Decision → Consequences | The chronological system of record for decisions | PUT |
|
||||
| `decisions/decision-template.md` | Bootstrap seed only | ADR template | Template, not a decision; never a runtime routing target | PUT (seed) |
|
||||
|
||||
**Mirror, don't duplicate:** if the decision belongs to an existing `projects/active/` note, PATCH a `[[wikilink]]` to the ADR under that project's `## Key Decisions`. No matching project → skip the mirror; the by-date ADR stands alone.
|
||||
|
||||
## _agent/ — the agent's own working substrate
|
||||
|
||||
| Path | Trigger | What lands | Distinct because | Method |
|
||||
|------|---------|------------|------------------|--------|
|
||||
| `_agent/chorus-vault.md` | Bootstrap / schema migration only | Marker: `schema_version`, bootstrap date | Plugin-owned probe; never hand-edited | GET / PUT |
|
||||
| `_agent/echo-vault.md` | Pre-fork ECHO vault adoption (read-only probe) | Legacy ECHO marker, honored until the rename migration | CHORUS is a fork of ECHO; adopted vaults still carry the old marker | GET |
|
||||
| `_agent/context/current-context.md` | Active scope changes; task focus shifts | `## Scope`, `## Scope History`, priorities | Single *live* scope pointer, vs episodic which is a *past* record | PATCH / PUT |
|
||||
| `_agent/memory/semantic/operator-preferences.md` | A preference/pattern about the operator | Append under `## Observations`; promote to `## Fact / Pattern` when stable | The one curated profile; distinct from ad-hoc semantic notes | PATCH |
|
||||
| `_agent/memory/semantic/<slug>.md` | A durable fact/pattern that isn't an operator-preference | Semantic note | Timeless fact, vs episodic (time-stamped event) and working (transient) | PUT |
|
||||
| `_agent/memory/episodic/<slug>.md` | A record of *what happened, when* | Episodic note | Anchored to an event in time; not a standing fact | PUT |
|
||||
| `_agent/memory/working/<slug>.md` | Short-lived state needed only for the current effort | Working note | Explicitly transient/time-boxed; safe to go stale | PUT |
|
||||
| `_agent/sessions/YYYY-MM-DD-HHMM-<slug>.md` | A substantive session ends (decisions/artifacts/commitments) | Session log (see template) | Per-session record; the unit loading Step 4 scans | PUT |
|
||||
| `_agent/health/YYYY-MM-vault-health.md` | First substantive session of a month (Vault Health pass) | Health-audit findings | Agent self-maintenance about vault integrity — NOT the operator's work narrative, so not in `journal/` | PUT |
|
||||
| `_agent/heartbeat/last-session.md` | End of every session, after the session log is written | One line: `<session-log-path> @ <ISO-timestamp>` | O(1) orientation pointer read first at load (Step 4); overwritten, never grows | PUT |
|
||||
| `_agent/templates/` | Bootstrap only (seeded from plugin masters) | Canonical note templates | Holds templates, not memory; never a runtime routing target | PUT (seed) |
|
||||
| `_agent/skills/active/<slug>.md` | The operator authors/installs a skill and wants it catalogued | Skill capability entry | Catalogs a *capability*, vs `projects/` which tracks the *build effort* | PUT |
|
||||
| `_agent/skills/archived/<slug>.md` | A catalogued skill is retired | Skill entry (moved from `active/`) | Terminal state of the skill catalog | PUT |
|
||||
| `_agent/locks/vault.lock` | Advisory multi-writer lock acquire/release | One line: `<owner> @ <ISO-timestamp>` | Concurrency coordination, not memory; managed only by `chorus.py lock/unlock` | PUT / DELETE |
|
||||
| `_agent/index/entities.json` | Rebuilt on every `capture` and on `sweep` | Slug→{path, kind, title, aliases, last_seen} registry | Machine-maintained routing/recall index, not a note; the one JSON in the vault | PUT |
|
||||
|
||||
---
|
||||
|
||||
## Structural signposts (not memory)
|
||||
|
||||
Created at bootstrap; never a runtime memory destination, but valid vault paths so the linter must recognise them.
|
||||
|
||||
| Path | Trigger | What lands | Distinct because | Method |
|
||||
|------|---------|------------|------------------|--------|
|
||||
| `<folder>/README.md` | Bootstrap leaf signpost (and the vault-root README) | One-line human note | Human signpost; never read for routing | PUT (seed) |
|
||||
| `projects/project-template.md` · `decisions/decision-template.md` · `journal/templates/` · `_agent/templates/` | Bootstrap seed only | Note templates | Hold templates, not content | PUT (seed) |
|
||||
|
||||
---
|
||||
|
||||
## Retired paths — explicitly never written
|
||||
|
||||
Listed so they are recognised as dead and never recreated. Any one of these appearing in a live vault is a migration miss (see `bootstrap.md` Migrations).
|
||||
|
||||
| Path | Status | Where it went instead |
|
||||
|------|--------|-----------------------|
|
||||
| `reviews/` (weekly/monthly/quarterly/annual) | Retired in schema 2 | Journal rollups → `journal/{weekly,monthly,quarterly,annual}/`; vault-health → `_agent/health/` |
|
||||
| `decisions/by-project/` | Retired in schema 1 | ADR mirrored as a `[[wikilink]]` under the project's `## Key Decisions` |
|
||||
| `archive/` (top-level) | Never existed | Project archival → `projects/archived/`; skill archival → `_agent/skills/archived/` |
|
||||
| `CLAUDE.md` / `BOOTSTRAP.md` / `STRUCTURE.md` / `index.md` (in-vault) | Retired in schema 1 | All control logic lives in the plugin (`references/`), not the vault |
|
||||
|
||||
## Routing decision tree (the calls that get mis-made)
|
||||
|
||||
1. **Destination unknown right now?** → `inbox/captures/`. Known? → route directly; never park a known fact in the inbox.
|
||||
2. **Is it about the operator's work over time?** → `journal/` (pick the grain by cadence). **About the vault's mechanical health?** → `_agent/health/`. These two look similar monthly but answer different questions.
|
||||
3. **Does the effort have an end state?** → `projects/` (folder = `status:`). **No finish line?** → `areas/`.
|
||||
4. **A fact about the world:** timeless → `semantic/`; a thing that happened → `episodic/`; needed only for now → `working/`. A fact about a *person* → `resources/people/`; a fact about an *organization* → `resources/companies/`.
|
||||
5. **A decision:** always `decisions/by-date/`; mirror into a project only if one already exists.
|
||||
6. **A capability (skill/plugin):** `_agent/skills/` catalogs the capability; `projects/` tracks building it. Both can exist for the same skill.
|
||||
@@ -0,0 +1,110 @@
|
||||
# Session Log Template
|
||||
|
||||
Session logs go in: `_agent/sessions/YYYY-MM-DD-HHMM-<slug>.md`
|
||||
|
||||
**Filename format is canonical and not optional.** The four-digit local-time HHMM component is what makes session filenames lex-sort in true chronological order — the loading procedure depends on it. Before PUT-ing a new session log, validate the filename matches `^\d{4}-\d{2}-\d{2}-\d{4}-[a-z0-9-]+\.md$`. Legacy session logs without HHMM exist in the vault; do not edit their names, but every new write must use the full form.
|
||||
|
||||
The slug describes what the session was about in 2–5 words, kebab-case.
|
||||
Examples: `2026-06-05-1430-chorus-plugin-build.md`, `2026-05-14-0900-q1-review-prep.md`.
|
||||
|
||||
Keep logs focused. Capture the goal, what was read/done, decisions, outputs, open threads, and the next step. This matches the vault's `_agent/templates/session-log-template.md`.
|
||||
|
||||
---
|
||||
|
||||
## Template
|
||||
|
||||
```markdown
|
||||
---
|
||||
type: session-log
|
||||
status: complete
|
||||
created: 2026-06-05T14:30
|
||||
updated: 2026-06-05T14:30
|
||||
tags: [agent, session]
|
||||
agent_written: true
|
||||
source_notes: []
|
||||
session_date: 2026-06-05
|
||||
client: claude-code
|
||||
---
|
||||
|
||||
# Session Log
|
||||
|
||||
## Goal
|
||||
One line — what this session set out to do.
|
||||
|
||||
## Notes Read
|
||||
- [[note]] — why it was relevant
|
||||
(omit if none)
|
||||
|
||||
## Actions Taken
|
||||
Brief narrative or bullets of what was done.
|
||||
|
||||
## Decisions Made
|
||||
- Decision — why
|
||||
(omit section if none)
|
||||
|
||||
## Outputs Created
|
||||
- `path/or/filename` — what it is
|
||||
(omit section if none)
|
||||
|
||||
## Open Threads
|
||||
- [ ] unresolved question or next step
|
||||
(omit section if none)
|
||||
|
||||
## Suggested Next Step
|
||||
One sentence: what to do first next time on this topic.
|
||||
|
||||
## Related
|
||||
- [[wikilinks go here, in the body — never in frontmatter]]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example
|
||||
|
||||
```markdown
|
||||
---
|
||||
type: session-log
|
||||
status: complete
|
||||
created: 2026-06-05T14:30
|
||||
updated: 2026-06-05T14:30
|
||||
tags: [agent, session, plugin]
|
||||
agent_written: true
|
||||
source_notes: ["resources/references/obsidian-local-rest-api.md"]
|
||||
session_date: 2026-06-05
|
||||
client: claude-code
|
||||
---
|
||||
|
||||
# Session Log
|
||||
|
||||
## Goal
|
||||
Build and package the chorus-memory CoWork plugin against the live vault.
|
||||
|
||||
## Notes Read
|
||||
- [[BOOTSTRAP]], [[STRUCTURE]], [[_agent/memory/semantic/operator-preferences]]
|
||||
|
||||
## Actions Taken
|
||||
Verified the REST API end-to-end, confirmed the scaffold copied into the live vault,
|
||||
created missing empty folders, and built the plugin (SKILL + 4 reference files).
|
||||
|
||||
## Decisions Made
|
||||
- Vault addressed at root — CHORUS is a dedicated vault.
|
||||
- Key hardcoded in the plugin (not in the vault) — personal plugin, per the reference pattern.
|
||||
|
||||
## Outputs Created
|
||||
- `chorus-memory.plugin` — installable CoWork plugin
|
||||
|
||||
## Open Threads
|
||||
- [ ] Validate Claude Code direct filesystem access to the vault host.
|
||||
|
||||
## Suggested Next Step
|
||||
Install the plugin and run a live load/write through it to confirm the skill triggers.
|
||||
|
||||
## Related
|
||||
- [[projects/active/vault-foundation]]
|
||||
- [[resources/references/obsidian-local-rest-api]]
|
||||
```
|
||||
|
||||
After writing the log, append a one-line entry to the daily note's **Agent Log** section.
|
||||
|
||||
> **Reminder:** wiki links go in the body (e.g. this `## Related` section), never in YAML
|
||||
> frontmatter. `source_notes` in frontmatter holds plain relative path strings, not `[[links]]`.
|
||||
@@ -0,0 +1,187 @@
|
||||
# Vault Layout & Frontmatter Conventions
|
||||
|
||||
**This document is canonical.** The CHORUS vault holds data only — there are no `CLAUDE.md` / `STRUCTURE.md` / `BOOTSTRAP.md` / `index.md` control docs in it. Layout, taxonomy, and frontmatter conventions live here in the plugin; the bootstrap procedure (`references/bootstrap.md`) builds the tree below into any empty vault.
|
||||
|
||||
## Folder Map (root-addressed)
|
||||
|
||||
```
|
||||
/vault/
|
||||
├── README.md ← thin human signpost (NOT read for routing)
|
||||
├── inbox/
|
||||
│ ├── captures/ ← quick captures (inbox.md), date-prefixed lines
|
||||
│ ├── imports/ ← raw imported material
|
||||
│ └── processing-log/
|
||||
├── journal/ ← one append-only time-series stream; rollups are coarser journal entries, NOT a separate reviews/ tree
|
||||
│ ├── daily/ ← YYYY-MM-DD.md (has an "Agent Log" section)
|
||||
│ ├── weekly/ ← YYYY-Www.md (ISO week, opt-in rollup)
|
||||
│ ├── monthly/ ← YYYY-MM.md (monthly rollup)
|
||||
│ ├── quarterly/ ← YYYY-Qn.md (manual / on request)
|
||||
│ ├── annual/ ← YYYY.md (manual / on request)
|
||||
│ └── templates/
|
||||
├── projects/ ← lifecycle: incubating → active → on-hold/archived
|
||||
│ ├── active/ ← current work (status: active)
|
||||
│ ├── incubating/ ← idea captured, not yet started (status: incubating)
|
||||
│ ├── on-hold/ ← paused but kept (status: on-hold)
|
||||
│ ├── archived/ ← done / abandoned (status: archived)
|
||||
│ └── project-template.md
|
||||
├── areas/ ← business / personal / learning / systems
|
||||
├── resources/
|
||||
│ ├── companies/ ← <slug>.md — organizations (clients, vendors, partners, employers)
|
||||
│ ├── concepts/ references/ meetings/
|
||||
│ └── people/ ← <name>.md
|
||||
├── decisions/
|
||||
│ ├── by-date/ ← YYYY-MM-DD-<slug>.md (ADR-style) — the canonical home
|
||||
│ └── decision-template.md
|
||||
│ (decisions/by-project/ is retired and not created —
|
||||
│ mirror an ADR into a project's `## Key Decisions` heading instead)
|
||||
│ (reviews/ is retired — journal rollups live under journal/; vault-health audits under _agent/health/)
|
||||
└── _agent/
|
||||
├── chorus-vault.md ← bootstrap marker: schema_version + bootstrap date (plugin-owned; the "is this vault set up?" probe)
|
||||
├── context/ ← current-context.md and task bundles
|
||||
├── memory/
|
||||
│ ├── working/ ← transient, time-boxed
|
||||
│ ├── episodic/ ← what happened, when
|
||||
│ └── semantic/ ← durable facts/patterns (operator-preferences.md)
|
||||
├── sessions/ ← YYYY-MM-DD-HHMM-<slug>.md
|
||||
├── health/ ← YYYY-MM-vault-health.md (monthly self-maintenance audit; NOT a journal entry)
|
||||
├── templates/ ← canonical note templates
|
||||
├── skills/ ← active / archived
|
||||
├── heartbeat/ ← single-line pointers (e.g. last-session.md → most-recent session log path)
|
||||
├── index/ ← entities.json — machine-maintained slug→{path,kind,title,aliases} registry
|
||||
└── locks/ ← vault.lock — cooperative advisory multi-writer lock
|
||||
```
|
||||
|
||||
**Entity index:** `_agent/index/entities.json` is a machine-maintained registry mapping each entity slug to its `{path, kind, title, aliases, last_seen}`. It makes routing/resolve an O(1), alias-aware lookup and supplies the name→path map for cross-linking and recall. `resolve` matches slug/title/alias exactly; when nothing matches exactly it falls back to **fuzzy candidates** — entities sharing a distinctive token with the mention — so a shortened name ("chorus memory" → the project `chorus`) surfaces the existing note instead of spawning a duplicate. Aliases are auto-derived from titles, learned from mentions on update, and re-folded from note frontmatter on `sweep`. It is rebuilt automatically by `chorus.py capture` and by `sweep.py`; do not hand-edit. The only JSON file in the vault.
|
||||
|
||||
**Heartbeat:** `_agent/heartbeat/last-session.md` is a one-line pointer (`<session-log-path> @ <ISO-timestamp>`) the **session-logging procedure writes (PUT, overwrite) at session end** and the **loading procedure reads first (Step 4)** as an O(1) shortcut to the latest session log. It is a hint, not a source of truth — fall back to the `sessions/` directory listing if it's missing or stale. Because it's PUT-overwritten, it never grows or duplicates.
|
||||
|
||||
**Slug rules:** kebab-case, ASCII only, truncate to ~40 chars.
|
||||
|
||||
---
|
||||
|
||||
## Canonical Frontmatter
|
||||
|
||||
Every note starts with this block. Fill what applies; leave the rest empty rather than guessing — **except `status:` and `tags:` on entity notes** (person/company/concept/reference/project/area/skill, plus `tags` on meeting/decision), which must be populated: `capture` stamps a kind-appropriate default `status` and seeds `tags` with the note's kind, `/chorus-health` flags missing/empty ones (`incomplete-frontmatter`), and `sweep.py --apply` backfills older notes. The per-kind requirement map is `KIND_REQUIRED_FM` / `KIND_STATUS` in `scripts/chorus_index.py`.
|
||||
|
||||
```yaml
|
||||
---
|
||||
type: # see Note Types below
|
||||
status: # active | draft | done | archived | complete — REQUIRED on entity notes (kind default stamped by capture)
|
||||
created: # YYYY-MM-DD (or YYYY-MM-DDTHH:mm for sessions)
|
||||
updated: # YYYY-MM-DD
|
||||
tags: [] # REQUIRED (non-empty) on entity notes — capture seeds the kind (e.g. [company]); enrich via --tags
|
||||
aliases: [] # optional — other names this entity is called (Obsidian-native). Folded
|
||||
# into the entity index so a shortened/expanded name resolves to this note.
|
||||
agent_written: false
|
||||
source_notes: [] # plain relative paths as strings — NEVER [[wikilinks]]
|
||||
---
|
||||
```
|
||||
|
||||
`aliases:` is the **durable, Obsidian-native home for alternate names** of an entity (e.g. the project `chorus` is also "chorus-memory" / "chorus plugin"). `capture` auto-derives the obvious case/punctuation variants of the title and writes them here; `sweep` folds frontmatter aliases back into the entity index, so they survive an index rebuild. List plain strings, never `[[wikilinks]]`.
|
||||
|
||||
`agent_written: true` + a populated `source_notes` is the key signal separating
|
||||
agent-managed content from human-authored content. When appending with POST, do
|
||||
not rewrite frontmatter — the append goes after existing content. To change
|
||||
`updated:` or `status:`, use PATCH with `Target-Type: frontmatter`.
|
||||
|
||||
**Frontmatter field semantics:**
|
||||
|
||||
- `created:` is the **earliest known date** the entity was tracked in the vault — *not* "today". When merging notes (e.g. promoting an `on-hold/` project into `active/`), preserve the earliest `created:` from any merged source and only update `updated:`.
|
||||
- `source_notes` is a **backward link** — the note(s) that triggered or supplied content for this one (e.g. the session log a project update came from). Forward links go in the `## Related` body section, never here.
|
||||
- `status:` for a project MUST match its folder under `projects/` (`active`, `incubating`, `on-hold`, `archived`). Moving the file and updating `status:` are the same operation.
|
||||
|
||||
> **No `[[wikilinks]]` in frontmatter.** YAML parses `[[...]]` as nested lists, so wiki
|
||||
> links there break and never render as clickable links in reading view. Put all
|
||||
> cross-references in a **`## Related`** section in the note **body** (bulleted `[[links]]`).
|
||||
> Frontmatter holds scalar/string metadata only; `source_notes` values are plain relative
|
||||
> paths, not links.
|
||||
|
||||
## Note Types
|
||||
|
||||
`daily-note`, `weekly-note`, `monthly-note`, `project`, `project-update`, `area`,
|
||||
`concept`, `reference`, `person`, `company`, `meeting`, `decision`, `review`, `session-log`,
|
||||
`working-memory`, `episodic-memory`, `semantic-memory`, `context-bundle`, `skill`,
|
||||
`draft`, `inbox-item`.
|
||||
|
||||
---
|
||||
|
||||
## File-Specific Conventions
|
||||
|
||||
### operator-preferences.md (`_agent/memory/semantic/`)
|
||||
|
||||
The profile analog. Canonical headings:
|
||||
|
||||
- `## Operator` — who the vault owner is (one paragraph)
|
||||
- `## Fact / Pattern` — **promoted, deduped rules.** No date prefix. Timeless.
|
||||
- `## Observations` — **timestamped raw observations.** Date-prefixed lines (`- 2026-06-06: ...`). Default landing zone for new evidence.
|
||||
- `## Evidence` — citations/links supporting the rules
|
||||
- `## Recommendation or Implication` — how the rules should shape behavior
|
||||
- `## Review Notes` — confidence / last review date
|
||||
|
||||
Append observed facts under `## Observations` by default. Promote to `## Fact / Pattern` (dropping the date) once a pattern stabilizes. "The operator" is the vault owner — refer to them generically and in the third person (the display name is the runtime `owner` config value).
|
||||
|
||||
### projects/active/\<slug\>.md
|
||||
|
||||
Mirrors `scaffold/templates/projects/project-template.md` — keep this block in sync with that template.
|
||||
|
||||
```markdown
|
||||
---
|
||||
type: project
|
||||
status: active
|
||||
created: {{date:YYYY-MM-DD}}
|
||||
updated: {{date:YYYY-MM-DD}}
|
||||
tags: [project]
|
||||
agent_written: false
|
||||
source_notes: []
|
||||
owner:
|
||||
review_cycle: weekly
|
||||
---
|
||||
|
||||
# Project Name
|
||||
|
||||
## Purpose
|
||||
|
||||
## Status
|
||||
One paragraph, kept fresh via PATCH replace (target `Project Name::Status`).
|
||||
|
||||
## Goals
|
||||
|
||||
## Current Context
|
||||
|
||||
## Open Loops
|
||||
- [ ] unresolved thing
|
||||
|
||||
## Key Decisions
|
||||
- [[YYYY-MM-DD-decision-slug]] — one-line summary (ADR mirror target)
|
||||
|
||||
## Related Notes
|
||||
|
||||
## Session History
|
||||
- 2026-06-05: observation or update
|
||||
|
||||
## Related
|
||||
- [[areas/business/business-ops]]
|
||||
```
|
||||
|
||||
### sessions/YYYY-MM-DD-HHMM-\<slug\>.md
|
||||
|
||||
See `session-log-template.md`. CHORUS uses an **HHMM time component** in the filename — this is **canonical, not optional**. The four-digit local-time component makes filenames lex-sort in true chronological order, which the loading procedure relies on. Older session logs without HHMM exist; leave them alone, but every new one must use the full `YYYY-MM-DD-HHMM-<slug>.md` form.
|
||||
|
||||
### decisions/by-date/YYYY-MM-DD-\<slug\>.md
|
||||
|
||||
ADR-style: Context → Decision → Consequences. If the decision belongs to an existing project, PATCH-append the wikilink into that project's `## Key Decisions` heading. Don't use `decisions/by-project/` — it's legacy scaffolding that's intentionally left empty.
|
||||
|
||||
### people/\<name\>.md
|
||||
|
||||
`type: person`. Use lowercase kebab-case for the slug (e.g. `robert-smith.md`).
|
||||
|
||||
### companies/\<slug\>.md
|
||||
|
||||
`type: company`. An organization the vault owner works with — client, vendor, partner, or employer (e.g. `acme.md`). Distinct from `people/` (individuals, who *belong to* companies) and `references/` (external sources). Lowercase kebab-case slug.
|
||||
|
||||
---
|
||||
|
||||
## Cross-References
|
||||
|
||||
Use Obsidian wiki links freely: `[[note-name]]` or `[[folder/note]]`. The REST API
|
||||
doesn't resolve them, but Obsidian does when the operator browses the vault.
|
||||
@@ -0,0 +1,12 @@
|
||||
# CHORUS Memory Vault
|
||||
|
||||
This Obsidian vault is the persistent memory substrate ("second brain") for its operator. It is read and written across Claude / CoWork sessions through the Obsidian Local REST API.
|
||||
|
||||
**This vault holds data, not logic.** All operating procedure — how the vault is bootstrapped, how notes are routed, the taxonomy, frontmatter conventions, and safety rules — lives in the **`chorus-memory` plugin**, which is the single source of truth. There are intentionally no `CLAUDE.md` / `BOOTSTRAP.md` / `STRUCTURE.md` control docs in this vault; updating or porting CHORUS means updating or installing the plugin, not editing files here.
|
||||
|
||||
- **Layout:** see the plugin's `references/vault-layout.md`.
|
||||
- **Operating contract & safety:** see the plugin's `references/operating-contract.md`.
|
||||
- **Bootstrap / repair:** see the plugin's `references/bootstrap.md`.
|
||||
- **Version marker:** `_agent/chorus-vault.md` records the schema version and bootstrap date.
|
||||
|
||||
Folders: `inbox/`, `journal/` (daily + weekly/monthly/quarterly/annual rollups), `projects/`, `areas/`, `resources/`, `decisions/`, and the agent subtree `_agent/`.
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
---
|
||||
type: context-bundle
|
||||
status: active
|
||||
created: {{DATE}}
|
||||
updated: {{DATE}}
|
||||
tags: [agent, context]
|
||||
agent_written: true
|
||||
source_notes: []
|
||||
scope:
|
||||
scope_updated: {{DATE}}
|
||||
refresh_strategy: on-demand
|
||||
---
|
||||
|
||||
# Current Context
|
||||
|
||||
## Scope
|
||||
<!-- The single active scope. Replaced (PATCH replace) on each scope switch. -->
|
||||
|
||||
## Scope History
|
||||
<!-- Dated bullets of prior scopes, newest first: `- {{DATE}}: <prior scope summary>`. Trim to ~10. -->
|
||||
|
||||
## Active Priorities
|
||||
|
||||
## Open Questions
|
||||
|
||||
## Related
|
||||
@@ -0,0 +1,3 @@
|
||||
# Inbox — Captures
|
||||
|
||||
Quick captures land here as date-prefixed lines (`- {{DATE}}: <thing>`), one per line, via POST. Triage routes durable items to their proper home (see the chorus-memory skill's Inbox Triage). Don't delete originals on triage — the processing log is the audit trail.
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
---
|
||||
type: semantic-memory
|
||||
status: active
|
||||
created: {{DATE}}
|
||||
updated: {{DATE}}
|
||||
tags: [agent, semantic-memory, operator]
|
||||
agent_written: true
|
||||
source_notes: []
|
||||
confidence: low
|
||||
last_reviewed: {{DATE}}
|
||||
---
|
||||
|
||||
# Operator Preferences
|
||||
|
||||
## Operator
|
||||
<!-- One paragraph: who the operator is. Leave for the operator to confirm; do not fabricate. -->
|
||||
|
||||
## Fact / Pattern
|
||||
<!-- Promoted, deduped, timeless rules. No date prefix. Add only when a rule is stable. -->
|
||||
|
||||
## Observations
|
||||
<!-- Timestamped raw observations. Default landing zone for new evidence: `- {{DATE}}: ...` -->
|
||||
|
||||
## Evidence
|
||||
|
||||
## Recommendation or Implication
|
||||
|
||||
## Review Notes
|
||||
Seeded empty at bootstrap ({{DATE}}). Raise confidence once preferences are confirmed through day-to-day use.
|
||||
|
||||
## Related
|
||||
@@ -0,0 +1,19 @@
|
||||
---
|
||||
type: reference
|
||||
status: active
|
||||
created: {{DATE}}
|
||||
updated: {{DATE}}
|
||||
tags: [agent, system, marker]
|
||||
agent_written: true
|
||||
source_notes: []
|
||||
schema_version: 4
|
||||
bootstrapped: {{DATE}}
|
||||
managed_by: chorus-memory-plugin
|
||||
---
|
||||
|
||||
# CHORUS Vault Marker
|
||||
|
||||
This file marks the vault as bootstrapped by the **chorus-memory plugin** and records its schema version. The plugin's loading procedure GETs this file as its "is this vault set up?" probe; a `404` triggers a fresh bootstrap.
|
||||
|
||||
- `schema_version` — bumped by the plugin when the vault layout changes; a mismatch is the hook for a migration pass (see `references/bootstrap.md`).
|
||||
- Do not hand-edit. The plugin owns this file.
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
---
|
||||
type: context-bundle
|
||||
status: active
|
||||
created: {{date:YYYY-MM-DD}}
|
||||
updated: {{date:YYYY-MM-DD}}
|
||||
tags: [agent, context]
|
||||
agent_written: true
|
||||
source_notes: []
|
||||
scope:
|
||||
refresh_strategy: on-demand
|
||||
---
|
||||
|
||||
# Context Bundle Title
|
||||
|
||||
## Scope
|
||||
|
||||
## Active Priorities
|
||||
|
||||
## Relevant Entities
|
||||
|
||||
## Key Decisions
|
||||
|
||||
## Open Questions
|
||||
|
||||
## Source Notes
|
||||
|
||||
## Related
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
---
|
||||
type: semantic-memory
|
||||
status: active
|
||||
created: {{date:YYYY-MM-DD}}
|
||||
updated: {{date:YYYY-MM-DD}}
|
||||
tags: [agent, semantic-memory]
|
||||
agent_written: true
|
||||
source_notes: []
|
||||
confidence: high
|
||||
last_reviewed: {{date:YYYY-MM-DD}}
|
||||
---
|
||||
|
||||
# Semantic Memory Title
|
||||
|
||||
## Fact / Pattern
|
||||
|
||||
## Evidence
|
||||
|
||||
## Recommendation or Implication
|
||||
|
||||
## Review Notes
|
||||
|
||||
## Related
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
---
|
||||
type: session-log
|
||||
status: complete
|
||||
created: {{date:YYYY-MM-DDTHH:mm}}
|
||||
updated: {{date:YYYY-MM-DDTHH:mm}}
|
||||
tags: [agent, session]
|
||||
agent_written: true
|
||||
source_notes: []
|
||||
session_date: {{date:YYYY-MM-DD}}
|
||||
client: claude-code
|
||||
---
|
||||
|
||||
# Session Log
|
||||
|
||||
## Goal
|
||||
|
||||
## Notes Read
|
||||
|
||||
## Actions Taken
|
||||
|
||||
## Decisions Made
|
||||
|
||||
## Outputs Created
|
||||
|
||||
## Open Threads
|
||||
|
||||
## Suggested Next Step
|
||||
|
||||
## Related
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
---
|
||||
type: working-memory
|
||||
status: active
|
||||
created: {{date:YYYY-MM-DDTHH:mm}}
|
||||
updated: {{date:YYYY-MM-DDTHH:mm}}
|
||||
tags: [agent, working-memory]
|
||||
agent_written: true
|
||||
source_notes: []
|
||||
expires_after: 48h
|
||||
---
|
||||
|
||||
# Working Context
|
||||
|
||||
## Active Focus
|
||||
|
||||
## Recent Decisions
|
||||
|
||||
## Open Threads
|
||||
|
||||
## Relevant Links
|
||||
|
||||
## Related
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
---
|
||||
type: decision
|
||||
status: complete
|
||||
created: {{date:YYYY-MM-DD}}
|
||||
updated: {{date:YYYY-MM-DD}}
|
||||
tags: [decision]
|
||||
agent_written: true
|
||||
source_notes: []
|
||||
decision_date: {{date:YYYY-MM-DD}}
|
||||
impact: medium
|
||||
---
|
||||
|
||||
# Decision Title
|
||||
|
||||
## Context
|
||||
|
||||
## Decision
|
||||
|
||||
## Rationale
|
||||
|
||||
## Consequences
|
||||
|
||||
## Follow-Up
|
||||
|
||||
## Related
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
---
|
||||
type: daily-note
|
||||
status: active
|
||||
created: {{date:YYYY-MM-DD}}
|
||||
updated: {{date:YYYY-MM-DD}}
|
||||
tags: [daily, journal]
|
||||
agent_written: false
|
||||
source_notes: []
|
||||
---
|
||||
|
||||
# {{date:YYYY-MM-DD}}
|
||||
|
||||
## Top Priorities
|
||||
|
||||
## Schedule / Commitments
|
||||
|
||||
## Open Loops
|
||||
|
||||
## Notes
|
||||
|
||||
## Context for AI
|
||||
|
||||
## Agent Log
|
||||
|
||||
## Related
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
---
|
||||
type: review
|
||||
status: active
|
||||
created: {{date:gggg-[W]WW}}
|
||||
updated: {{date:gggg-[W]WW}}
|
||||
tags: [review, weekly]
|
||||
agent_written: true
|
||||
source_notes: []
|
||||
period: weekly
|
||||
---
|
||||
|
||||
# Weekly Review
|
||||
|
||||
## Completed Work
|
||||
|
||||
## Open Loops
|
||||
|
||||
## Stale Projects
|
||||
|
||||
## Decisions
|
||||
|
||||
## Priorities Next Week
|
||||
|
||||
## Agent Notes
|
||||
|
||||
## Related
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
---
|
||||
type: project
|
||||
status: active
|
||||
created: {{date:YYYY-MM-DD}}
|
||||
updated: {{date:YYYY-MM-DD}}
|
||||
tags: [project]
|
||||
agent_written: false
|
||||
source_notes: []
|
||||
owner:
|
||||
review_cycle: weekly
|
||||
---
|
||||
|
||||
# Project Name
|
||||
|
||||
## Purpose
|
||||
|
||||
## Status
|
||||
|
||||
## Goals
|
||||
|
||||
## Current Context
|
||||
|
||||
## Open Loops
|
||||
|
||||
## Key Decisions
|
||||
|
||||
## Related Notes
|
||||
|
||||
## Session History
|
||||
|
||||
## Related
|
||||
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env python3
|
||||
"""bootstrap.py — stand up (or repair) an CHORUS vault deterministically.
|
||||
|
||||
Idempotent and additive: every write is probe-before-write and NEVER overwrites
|
||||
an existing file. The marker (_agent/chorus-vault.md) is written LAST, so the vault
|
||||
is only flagged "set up" once every piece is in place. Re-running is the repair
|
||||
path (it fills in only what is missing). Cross-platform: pure Python via chorus.py.
|
||||
|
||||
Usage:
|
||||
bootstrap.py [--dry-run]
|
||||
|
||||
Env: CHORUS_BASE, CHORUS_KEY (via chorus.py), CHORUS_TODAY (YYYY-MM-DD for {{DATE}}).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
SKILL_DIR = SCRIPT_DIR.parent
|
||||
SCAFFOLD = SKILL_DIR / "scaffold"
|
||||
sys.path.insert(0, str(SCRIPT_DIR))
|
||||
|
||||
import chorus # noqa: E402
|
||||
|
||||
LEAVES = [
|
||||
"inbox/captures", "inbox/imports", "inbox/processing-log",
|
||||
"journal/daily", "journal/weekly", "journal/monthly", "journal/quarterly",
|
||||
"journal/annual", "journal/templates",
|
||||
"projects/active", "projects/incubating", "projects/on-hold", "projects/archived",
|
||||
"areas/business", "areas/personal", "areas/learning", "areas/systems",
|
||||
"resources/concepts", "resources/references", "resources/people",
|
||||
"resources/companies", "resources/meetings",
|
||||
"decisions/by-date",
|
||||
"_agent/context", "_agent/memory/working", "_agent/memory/episodic",
|
||||
"_agent/memory/semantic", "_agent/sessions", "_agent/health", "_agent/templates",
|
||||
"_agent/heartbeat", "_agent/skills/active", "_agent/skills/archived", "_agent/locks",
|
||||
"_agent/index",
|
||||
]
|
||||
|
||||
|
||||
def exists(path: str) -> bool:
|
||||
status, _ = chorus.request("GET", chorus.vault_url(path))
|
||||
return status == 200
|
||||
|
||||
|
||||
def put_text(path: str, text: str) -> None:
|
||||
status, body = chorus.request("PUT", chorus.vault_url(path), data=text.encode(),
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
chorus.check(status, body, f"bootstrap put {path}")
|
||||
|
||||
|
||||
def seed(path: str, source: Path, dry_run: bool) -> None:
|
||||
if exists(path):
|
||||
print(f"bootstrap: skip (exists) {path}")
|
||||
return
|
||||
if dry_run:
|
||||
print(f"bootstrap: would seed {path} <- {source.relative_to(SKILL_DIR)}")
|
||||
return
|
||||
text = source.read_text(encoding="utf-8").replace("{{DATE}}", chorus.today())
|
||||
put_text(path, text)
|
||||
print(f"bootstrap: seeded {path}")
|
||||
|
||||
|
||||
def leaf_readme(folder: str, dry_run: bool) -> None:
|
||||
path = f"{folder}/README.md"
|
||||
if exists(path):
|
||||
return
|
||||
if dry_run:
|
||||
print(f"bootstrap: would readme {path}")
|
||||
return
|
||||
name = folder.rsplit("/", 1)[-1]
|
||||
put_text(path, f"# {name}\n\nMemory vault folder. See the chorus-memory plugin for conventions.\n")
|
||||
print(f"bootstrap: readme {path}")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Bootstrap or repair an CHORUS vault")
|
||||
parser.add_argument("--dry-run", "-n", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
if not SCAFFOLD.is_dir():
|
||||
print(f"bootstrap: scaffold not found at {SCAFFOLD}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
legacy_marker = (not exists(chorus.MARKER_PATH)) and exists(chorus.LEGACY_MARKER_PATH)
|
||||
if exists(chorus.MARKER_PATH) or legacy_marker:
|
||||
probe = chorus.LEGACY_MARKER_PATH if legacy_marker else chorus.MARKER_PATH
|
||||
status, body = chorus.request("GET", chorus.vault_url(probe))
|
||||
version = "unknown"
|
||||
if status == 200:
|
||||
version = next((ln.split(":", 1)[1].strip()
|
||||
for ln in body.decode(errors="replace").splitlines()
|
||||
if ln.startswith("schema_version:")), "unknown")
|
||||
note = " [legacy ECHO marker]" if legacy_marker else ""
|
||||
print(f"bootstrap: marker present{note} (schema_version={version}). Running repair pass (fills only missing files).")
|
||||
|
||||
for folder in LEAVES:
|
||||
leaf_readme(folder, args.dry_run)
|
||||
|
||||
templates = SCAFFOLD / "templates"
|
||||
if templates.is_dir():
|
||||
for source in sorted(templates.rglob("*.md")):
|
||||
seed(source.relative_to(templates).as_posix(), source, args.dry_run)
|
||||
|
||||
seed("_agent/memory/semantic/operator-preferences.md", SCAFFOLD / "anchors" / "operator-preferences.seed.md", args.dry_run)
|
||||
seed("_agent/context/current-context.md", SCAFFOLD / "anchors" / "current-context.seed.md", args.dry_run)
|
||||
seed("inbox/captures/inbox.md", SCAFFOLD / "anchors" / "inbox.seed.md", args.dry_run)
|
||||
seed("README.md", SCAFFOLD / "README.vault.md", args.dry_run)
|
||||
if legacy_marker:
|
||||
# Adopted pre-fork ECHO vault: leave the legacy marker as the single
|
||||
# authoritative probe — the schema migration renames it. Two markers
|
||||
# would make "which is authoritative" ambiguous.
|
||||
print("bootstrap: legacy ECHO marker present — not writing the CHORUS marker (migrate.py renames it).")
|
||||
else:
|
||||
seed(chorus.MARKER_PATH, SCAFFOLD / "chorus-vault.md", args.dry_run) # marker LAST
|
||||
|
||||
print(f"bootstrap: done ({'DRY-RUN ' if args.dry_run else ''}{chorus.today()}).")
|
||||
print("bootstrap: next: create today's daily note + a bootstrap session log + heartbeat (see SKILL.md).")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,196 @@
|
||||
#!/usr/bin/env python3
|
||||
"""check_routing.py — keep the routing VIEWS in sync with routing.json.
|
||||
|
||||
routing.json is the canonical route manifest. SKILL.md ("Where to Write"),
|
||||
references/routing-map.md, and references/api-reference.md are hand-maintained
|
||||
human views of it, guarded until now only by a prose "routing.json wins"
|
||||
convention. This check makes that convention mechanically true:
|
||||
|
||||
CHECK A (validity, all three docs): every concrete vault path mentioned in the
|
||||
docs must match some route OR retired pattern in routing.json. Catches
|
||||
a doc pointing at a path the manifest does not define (typo, stale row).
|
||||
|
||||
CHECK B (coverage, routing-map.md): every route in routing.json must be named
|
||||
by at least one path in routing-map.md, which claims to be the complete
|
||||
canonical map. Catches a route added to the manifest but never documented.
|
||||
|
||||
Exit: 0 = in sync, 1 = drift found, 2 = could not read inputs. No network, no vault.
|
||||
Run it in CI / from the eval harness; it is pure local file parsing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
for _stream in (sys.stdout, sys.stderr):
|
||||
try:
|
||||
_stream.reconfigure(encoding="utf-8", errors="replace") # type: ignore[union-attr]
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
SKILL_DIR = SCRIPT_DIR.parent
|
||||
ROUTING_JSON = SCRIPT_DIR / "routing.json"
|
||||
DOCS = {
|
||||
"SKILL.md": SKILL_DIR / "SKILL.md",
|
||||
"routing-map.md": SKILL_DIR / "references" / "routing-map.md",
|
||||
"api-reference.md": SKILL_DIR / "references" / "api-reference.md",
|
||||
}
|
||||
CANONICAL_MAP = "routing-map.md"
|
||||
|
||||
ROOTS = ("_agent", "inbox", "journal", "projects", "areas", "resources", "decisions")
|
||||
FILE_SUFFIXES = (".md", ".lock", ".json")
|
||||
|
||||
# Order matters: substitute the most specific placeholder first.
|
||||
DATE_SUBS = [
|
||||
("YYYY-MM-DD", "2026-01-02"),
|
||||
("YYYY-Www", "2026-W01"),
|
||||
("YYYY-MM", "2026-01"),
|
||||
("YYYY-Qn", "2026-Q1"),
|
||||
("HHMM", "0900"),
|
||||
("YYYY", "2026"),
|
||||
]
|
||||
|
||||
# Named placeholders whose route constrains the value to a fixed vocabulary, so a
|
||||
# generic "sample" would not match the pattern.
|
||||
NAMED_SUBS = {
|
||||
"<lifecycle>": "active",
|
||||
"<domain>": "business",
|
||||
}
|
||||
|
||||
|
||||
def load_routing():
|
||||
import json
|
||||
data = json.loads(ROUTING_JSON.read_text(encoding="utf-8"))
|
||||
routes = [(r["id"], re.compile(r["pattern"]), stem(r["pattern"])) for r in data.get("routes", [])]
|
||||
retired = [re.compile(r["pattern"]) for r in data.get("retired", [])]
|
||||
return routes, retired
|
||||
|
||||
|
||||
def stem(pattern: str) -> str:
|
||||
"""Literal directory prefix of a regex pattern, e.g. ^projects/active/[^/]+\\.md$ -> projects/active/."""
|
||||
body = pattern[1:] if pattern.startswith("^") else pattern
|
||||
literal = []
|
||||
for ch in body:
|
||||
if ch in "\\([{+*?.|^$":
|
||||
break
|
||||
literal.append(ch)
|
||||
text = "".join(literal)
|
||||
return text[: text.rfind("/") + 1] if "/" in text else ""
|
||||
|
||||
|
||||
def strip_fenced_blocks(text: str) -> str:
|
||||
out, in_fence = [], False
|
||||
for line in text.splitlines():
|
||||
if line.lstrip().startswith("```"):
|
||||
in_fence = not in_fence
|
||||
continue
|
||||
if not in_fence:
|
||||
out.append(line)
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def expand_braces(token: str) -> list[str]:
|
||||
"""Expand a single brace group: a/{x,y}/b -> [a/x/b, a/y/b]. Recurses for nested groups."""
|
||||
match = re.search(r"\{([^{}]*)\}", token)
|
||||
if not match:
|
||||
return [token]
|
||||
pre, post = token[: match.start()], token[match.end():]
|
||||
results = []
|
||||
for part in match.group(1).split(","):
|
||||
results.extend(expand_braces(pre + part.strip() + post))
|
||||
return results
|
||||
|
||||
|
||||
def concretize(token: str) -> str:
|
||||
for placeholder, sample in DATE_SUBS:
|
||||
token = token.replace(placeholder, sample)
|
||||
for placeholder, sample in NAMED_SUBS.items():
|
||||
token = token.replace(placeholder, sample)
|
||||
return re.sub(r"<[^>]+>", "sample", token)
|
||||
|
||||
|
||||
def looks_like_path(token: str) -> bool:
|
||||
if " " in token or "::" in token or token.count("`"):
|
||||
return False
|
||||
# README signposts can live under any folder, including placeholder folders.
|
||||
if token == "README.md" or token.endswith("/README.md"):
|
||||
return True
|
||||
# Otherwise must sit under a known vault root directory (root + "/"), which
|
||||
# excludes shorthand like `inbox.md` and non-paths like `application/vnd...`.
|
||||
if not any(token.startswith(root + "/") for root in ROOTS):
|
||||
return False
|
||||
return token.endswith("/") or token.endswith(FILE_SUFFIXES) or "<" in token or "{" in token
|
||||
|
||||
|
||||
def extract_tokens(text: str) -> set[str]:
|
||||
text = strip_fenced_blocks(text)
|
||||
tokens: set[str] = set()
|
||||
for raw in re.findall(r"`([^`]+)`", text):
|
||||
raw = raw.strip()
|
||||
for piece in expand_braces(raw):
|
||||
if looks_like_path(piece):
|
||||
tokens.add(piece)
|
||||
return tokens
|
||||
|
||||
|
||||
def classify(tokens: set[str]):
|
||||
files, dirs = set(), set()
|
||||
for token in tokens:
|
||||
if token.endswith("/"):
|
||||
dirs.add(token)
|
||||
else:
|
||||
files.add(concretize(token))
|
||||
return files, dirs
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
routes, retired = load_routing()
|
||||
except Exception as exc:
|
||||
print(f"check-routing: cannot read routing.json ({exc})", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
problems: list[str] = []
|
||||
doc_tokens: dict[str, tuple[set[str], set[str]]] = {}
|
||||
for name, path in DOCS.items():
|
||||
try:
|
||||
files, dirs = classify(extract_tokens(path.read_text(encoding="utf-8")))
|
||||
except Exception as exc:
|
||||
print(f"check-routing: cannot read {name} ({exc})", file=sys.stderr)
|
||||
return 2
|
||||
doc_tokens[name] = (files, dirs)
|
||||
|
||||
route_stems = {st for _, _, st in routes if st}
|
||||
|
||||
# CHECK A — every documented path is valid under the manifest.
|
||||
for name, (files, dirs) in doc_tokens.items():
|
||||
for f in sorted(files):
|
||||
if not (any(rx.match(f) for _, rx, _ in routes) or any(rx.match(f) for rx in retired)):
|
||||
problems.append(f"[A] {name}: path `{f}` matches no route or retired pattern in routing.json")
|
||||
for d in sorted(dirs):
|
||||
ok = (any(rx.match(d) for rx in retired)
|
||||
or any(st == d or st.startswith(d) or d.startswith(st) for st in route_stems))
|
||||
if not ok:
|
||||
problems.append(f"[A] {name}: folder `{d}` matches no route stem or retired pattern")
|
||||
|
||||
# CHECK B — every route appears in the canonical map.
|
||||
map_files, map_dirs = doc_tokens[CANONICAL_MAP]
|
||||
for rid, rx, st in routes:
|
||||
covered = any(rx.match(f) for f in map_files) or (st and st in map_dirs)
|
||||
if not covered:
|
||||
problems.append(f"[B] route '{rid}' ({rx.pattern}) is not documented in {CANONICAL_MAP}")
|
||||
|
||||
if problems:
|
||||
print(f"check-routing: {len(problems)} drift issue(s) found\n")
|
||||
for p in problems:
|
||||
print(f" - {p}")
|
||||
return 1
|
||||
print(f"check-routing: in sync — {len(routes)} routes documented and all doc paths are valid.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,910 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus.py — the single validated, cross-platform client for the CHORUS Obsidian
|
||||
Local REST API.
|
||||
|
||||
Replaces the original chorus.sh so the toolchain runs identically on Windows,
|
||||
macOS, and Linux with no bash / GNU-vs-BSD `date` dependency — only a Python 3
|
||||
interpreter (which the linter already requires).
|
||||
|
||||
Every read/write to the vault should go through this script. It centralizes:
|
||||
auth injection, HTTP-status checking (non-zero exit on >=400 so a failed write
|
||||
can never look like success), one bounded retry on 5xx/connection errors,
|
||||
WHOLE-LINE idempotent append (a new line that is merely a substring of an
|
||||
existing one is no longer wrongly skipped), correct `::` heading-target handling,
|
||||
frontmatter field patches, a read-back-confirmed advisory multi-writer lock, a
|
||||
one-call cold-start `load`, and scope show/set.
|
||||
|
||||
Network is keep-alive and connection-pooled (one persistent connection per thread,
|
||||
reused across requests), and `read_many` fans bulk GETs across a thread pool — so
|
||||
full-vault scripts (sweep, lint, recall rebuild) make a handful of warm round-trips
|
||||
instead of hundreds of fresh TLS handshakes, and no longer blow past tool timeouts.
|
||||
|
||||
Config (owner/endpoint/key are machine-local — see chorus_config; the plugin ships none):
|
||||
owner/endpoint/key resolved by chorus_config from ~/.claude/chorus-memory/config.json
|
||||
(env CHORUS_OWNER / CHORUS_BASE / CHORUS_KEY override per field).
|
||||
Set up a machine with `chorus.py config init` then edit, or
|
||||
`chorus.py config set --owner ... --endpoint ... --key ...`.
|
||||
CHORUS_VERIFY default 1 — read-back verify after a PUT
|
||||
CHORUS_LOCK_TTL default 900 — seconds before an advisory lock is considered stale
|
||||
CHORUS_TIMEOUT default 30 — per-request socket timeout (seconds)
|
||||
CHORUS_WORKERS default 8 — concurrency for read_many bulk reads
|
||||
CHORUS_TODAY YYYY-MM-DD to use as "today" (pass the conversation's currentDate)
|
||||
|
||||
Usage:
|
||||
chorus.py load # cold-start: the 6 orientation reads in one call
|
||||
chorus.py get <path> # print file contents (404 -> exit 44)
|
||||
chorus.py map <path> # document-map JSON (headings/blocks/frontmatter)
|
||||
chorus.py ls <dir> # directory listing JSON
|
||||
chorus.py search <query...> # /search/simple
|
||||
chorus.py put <path> [file] # create/overwrite (body from file or stdin)
|
||||
chorus.py post <path> [file] # raw append (NON-idempotent; prefer `append`)
|
||||
chorus.py append <path> <line> # idempotent append: skips only on an exact whole-line match
|
||||
chorus.py patch <path> <append|prepend|replace> <heading|frontmatter|block> <target> [file]
|
||||
chorus.py fm <path> <field> <json-value> # create-or-replace a frontmatter scalar
|
||||
chorus.py bump <path> [YYYY-MM-DD] # set frontmatter updated: to today (or given date)
|
||||
chorus.py triage [--list --json | <proposals.json> [--apply]] # one-tap inbox triage + audit log
|
||||
chorus.py delete <path> # DELETE (destructive; explicit use only)
|
||||
chorus.py lock <owner-id> # acquire advisory lock (exit 75 if held & fresh, or lost the race)
|
||||
chorus.py unlock <owner-id> # release advisory lock if owned by <owner-id>
|
||||
chorus.py scope show # print active scope, its freshness, and sessions-since
|
||||
chorus.py scope set "<text>" # switch scope atomically (history + replace + stamp scope_updated)
|
||||
chorus.py config [show|init|set|import] # machine-local owner/endpoint/key (config import <file> adopts a provided file)
|
||||
|
||||
Exit codes: 0 ok · 44 not-found(404) · 75 lock-held/lost · 76 duplicate-gate (capture) ·
|
||||
78 config-required · 2 usage · 1 other HTTP/transport error.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import http.client
|
||||
import json
|
||||
import locale
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import urllib.parse
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
|
||||
# Never let a non-ASCII byte (em-dash in our own output, or Unicode in a fetched
|
||||
# note) raise UnicodeEncodeError on a legacy Windows console.
|
||||
for _stream in (sys.stdout, sys.stderr):
|
||||
try:
|
||||
_stream.reconfigure(encoding="utf-8", errors="replace") # type: ignore[union-attr]
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Owner / endpoint / key are machine-local and resolved by chorus_config from
|
||||
# ~/.claude/chorus-memory/config.json (env CHORUS_OWNER / CHORUS_BASE / CHORUS_KEY override).
|
||||
# The plugin ships none of them; load() returns "" for anything unset so --help,
|
||||
# `config`, and `doctor` stay usable. Network ops raise a clear error if endpoint/key
|
||||
# are missing (see _require_endpoint / request).
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import chorus_config # noqa: E402
|
||||
_CFG = chorus_config.load()
|
||||
BASE = _CFG["endpoint"]
|
||||
KEY = _CFG["key"]
|
||||
OWNER = _CFG["owner"]
|
||||
VERIFY = os.environ.get("CHORUS_VERIFY", "1") != "0"
|
||||
LOCK_TTL = int(os.environ.get("CHORUS_LOCK_TTL", "900"))
|
||||
# Per-request socket timeout. A single stuck file fails fast instead of stalling a
|
||||
# full-vault pass; with read_many concurrency it only blocks one worker, not the run.
|
||||
TIMEOUT = int(os.environ.get("CHORUS_TIMEOUT", "30"))
|
||||
# Concurrency for bulk reads (read_many). I/O-bound, so threads bypass the GIL fine.
|
||||
MAX_WORKERS = max(1, int(os.environ.get("CHORUS_WORKERS", "8")))
|
||||
|
||||
LOCK_PATH = "_agent/locks/vault.lock"
|
||||
|
||||
MARKER_PATH = "_agent/chorus-vault.md"
|
||||
# Pre-fork ECHO vaults carry the old marker name. It is honored read-only as a
|
||||
# bootstrap probe (vault-adoption path); the schema migration renames it.
|
||||
LEGACY_MARKER_PATH = "_agent/echo-vault.md"
|
||||
|
||||
|
||||
def marker_get():
|
||||
"""GET the bootstrap marker: the CHORUS name first, then the legacy ECHO
|
||||
name. Returns (path, status, body) — on a double miss, the CHORUS path with
|
||||
its 404 (or error) result."""
|
||||
status, body = request("GET", vault_url(MARKER_PATH))
|
||||
if status == 404:
|
||||
l_status, l_body = request("GET", vault_url(LEGACY_MARKER_PATH))
|
||||
if l_status == 200:
|
||||
return LEGACY_MARKER_PATH, l_status, l_body
|
||||
return MARKER_PATH, status, body
|
||||
|
||||
|
||||
class ChorusError(RuntimeError):
|
||||
def __init__(self, message: str, code: int = 1) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
|
||||
|
||||
def today() -> str:
|
||||
return os.environ.get("CHORUS_TODAY") or dt.date.today().isoformat()
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return dt.datetime.now(dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def vault_url(path: str) -> str:
|
||||
safe = "/".join(urllib.parse.quote(part) for part in path.split("/"))
|
||||
if path.endswith("/") and not safe.endswith("/"):
|
||||
safe += "/"
|
||||
return f"{BASE}/vault/{safe}"
|
||||
|
||||
|
||||
# --- persistent connection pool (keep-alive) ---------------------------------
|
||||
# urllib.urlopen opened a fresh TCP+TLS connection per call; against a remote HTTPS
|
||||
# endpoint the repeated handshake dominated, so a full-vault pass made hundreds of
|
||||
# serial handshakes and blew past the tool/sandbox timeout. We keep ONE connection
|
||||
# per thread (http.client is not thread-safe, so each thread — main + every pool
|
||||
# worker — owns its own) and reuse it across requests. read_many fans out over a
|
||||
# thread pool, so the pool's connections stay warm for the duration of a sweep.
|
||||
_local = threading.local()
|
||||
MAX_ATTEMPTS = 3
|
||||
|
||||
|
||||
def _require_config() -> None:
|
||||
"""Fail loudly (not with an opaque DNS/connection error) when this machine has no
|
||||
usable key file yet — the most common fresh-install state. Also catches an
|
||||
untouched `config init` template (placeholder endpoint/key)."""
|
||||
if not chorus_config.is_configured(_CFG):
|
||||
raise ChorusError(
|
||||
"chorus: NOT CONFIGURED — no usable CHORUS key file on this machine. Ask the "
|
||||
f"operator for their config (owner/endpoint/key) and install it at "
|
||||
f"{chorus_config.config_path()}: `chorus.py config import <file>`, or "
|
||||
"`chorus.py config set --owner ... --endpoint ... --key ...` "
|
||||
"(or set CHORUS_BASE / CHORUS_KEY).", 2)
|
||||
|
||||
|
||||
def _new_connection() -> http.client.HTTPConnection:
|
||||
parts = urllib.parse.urlsplit(BASE)
|
||||
host = parts.hostname or parts.path
|
||||
if parts.scheme == "http":
|
||||
return http.client.HTTPConnection(host, parts.port or 80, timeout=TIMEOUT)
|
||||
return http.client.HTTPSConnection(host, parts.port or 443, timeout=TIMEOUT)
|
||||
|
||||
|
||||
def _connection() -> http.client.HTTPConnection:
|
||||
conn = getattr(_local, "conn", None)
|
||||
if conn is None:
|
||||
conn = _new_connection()
|
||||
_local.conn = conn
|
||||
return conn
|
||||
|
||||
|
||||
def _drop_connection() -> None:
|
||||
conn = getattr(_local, "conn", None)
|
||||
if conn is not None:
|
||||
try:
|
||||
conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
_local.conn = None
|
||||
|
||||
|
||||
def request(method: str, url: str, data: bytes | None = None,
|
||||
headers: dict[str, str] | None = None) -> tuple[int, bytes]:
|
||||
"""Issue one request over this thread's reused connection. Returns (status, body);
|
||||
status 0 == transport failure (chorus_queue relies on this to trigger offline queueing).
|
||||
Retries: a stale pooled connection reconnects immediately; 5xx and other transport
|
||||
errors get one bounded sleep-retry, same as before."""
|
||||
_require_config()
|
||||
merged = {"Authorization": f"Bearer {KEY}", **(headers or {})}
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
target = parts.path or "/"
|
||||
if parts.query:
|
||||
target += "?" + parts.query
|
||||
last_status, last_body = 0, b""
|
||||
for attempt in range(MAX_ATTEMPTS):
|
||||
try:
|
||||
conn = _connection()
|
||||
conn.request(method, target, body=data, headers=merged)
|
||||
resp = conn.getresponse()
|
||||
body = resp.read() # drain fully so the connection can be reused
|
||||
status = resp.status
|
||||
if status >= 500 and attempt < MAX_ATTEMPTS - 1:
|
||||
_drop_connection()
|
||||
time.sleep(1)
|
||||
continue
|
||||
return status, body
|
||||
except (http.client.RemoteDisconnected, http.client.BadStatusLine,
|
||||
ConnectionResetError, BrokenPipeError) as exc:
|
||||
# The server closed an idle keep-alive connection between requests.
|
||||
# Expected, not a failure: reconnect and retry without backing off.
|
||||
_drop_connection()
|
||||
last_status, last_body = 0, str(exc).encode()
|
||||
continue
|
||||
except Exception as exc: # timeout, DNS, refused, TLS, ...
|
||||
_drop_connection()
|
||||
last_status, last_body = 0, str(exc).encode()
|
||||
if attempt < MAX_ATTEMPTS - 1:
|
||||
time.sleep(1)
|
||||
continue
|
||||
return last_status, last_body
|
||||
|
||||
|
||||
def get_text(path: str) -> str | None:
|
||||
"""GET a vault path -> decoded text, or None on 404/any error. Never raises, so a
|
||||
single unreadable note can't abort a bulk pass (resilience for sweep/lint)."""
|
||||
try:
|
||||
status, body = request("GET", vault_url(path))
|
||||
except Exception:
|
||||
return None
|
||||
return body.decode(errors="replace") if status == 200 else None
|
||||
|
||||
|
||||
def read_many(paths) -> dict[str, str | None]:
|
||||
"""Concurrently GET many vault paths -> {path: text_or_None}. The single biggest
|
||||
win for full-vault scripts: it turns hundreds of serial round-trips into MAX_WORKERS
|
||||
parallel ones over warm keep-alive connections. Resilient — a bad file maps to None."""
|
||||
unique = list(dict.fromkeys(paths))
|
||||
if not unique:
|
||||
return {}
|
||||
workers = min(MAX_WORKERS, len(unique))
|
||||
with ThreadPoolExecutor(max_workers=workers) as pool:
|
||||
return dict(zip(unique, pool.map(get_text, unique)))
|
||||
|
||||
|
||||
def check(status: int, body: bytes, context: str) -> None:
|
||||
if status == 404:
|
||||
raise ChorusError(f"{context}: not found", 44)
|
||||
if status == 0:
|
||||
raise ChorusError(f"{context}: vault unreachable ({body.decode(errors='replace')}) [{BASE}]", 1)
|
||||
if status >= 400:
|
||||
raise ChorusError(f"{context}: HTTP {status} - {body.decode(errors='replace')}", 1)
|
||||
|
||||
|
||||
def read_body(arg: str | None) -> bytes:
|
||||
if arg in (None, "-"):
|
||||
raw = sys.stdin.buffer.read()
|
||||
try:
|
||||
text = raw.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
text = raw.decode(locale.getpreferredencoding(False) or "cp1252", errors="replace")
|
||||
return text.encode("utf-8")
|
||||
return Path(arg).read_bytes()
|
||||
|
||||
|
||||
def normalize_patch_body(data: bytes, op: str, target_type: str) -> bytes:
|
||||
"""Heading PATCH bodies need a leading newline on replace and a trailing
|
||||
newline on append/prepend so the API does not concatenate onto adjacent text."""
|
||||
if target_type != "heading":
|
||||
return data
|
||||
text = data.decode("utf-8", errors="replace")
|
||||
if op == "replace":
|
||||
text = text.strip("\r\n")
|
||||
text = f"\n{text}\n" if text else "\n"
|
||||
elif text and not text.endswith("\n"):
|
||||
text += "\n"
|
||||
return text.encode("utf-8")
|
||||
|
||||
|
||||
def normalize_json_scalar(value: str) -> bytes:
|
||||
"""Accept either a raw scalar ('Fabrication Lead') or pre-formed JSON ('"2026-06-20"')."""
|
||||
value = value.strip()
|
||||
try:
|
||||
json.loads(value)
|
||||
return value.encode("utf-8")
|
||||
except json.JSONDecodeError:
|
||||
return json.dumps(value).encode("utf-8")
|
||||
|
||||
|
||||
def temp_file(data: bytes) -> str:
|
||||
fh = tempfile.NamedTemporaryFile(delete=False)
|
||||
try:
|
||||
fh.write(data)
|
||||
return fh.name
|
||||
finally:
|
||||
fh.close()
|
||||
|
||||
|
||||
# ---- commands ----------------------------------------------------------------
|
||||
|
||||
def cmd_get(path: str) -> int:
|
||||
status, body = request("GET", vault_url(path))
|
||||
check(status, body, f"get {path}")
|
||||
sys.stdout.buffer.write(body)
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_map(path: str) -> int:
|
||||
status, body = request("GET", vault_url(path),
|
||||
headers={"Accept": "application/vnd.olrapi.document-map+json"})
|
||||
check(status, body, f"map {path}")
|
||||
sys.stdout.buffer.write(body)
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_ls(path: str) -> int:
|
||||
if not path.endswith("/"):
|
||||
path += "/"
|
||||
status, body = request("GET", vault_url(path))
|
||||
check(status, body, f"ls {path}")
|
||||
sys.stdout.buffer.write(body)
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_search(query: list[str]) -> int:
|
||||
q = urllib.parse.quote_plus(" ".join(query))
|
||||
status, body = request("POST", f"{BASE}/search/simple/?query={q}")
|
||||
check(status, body, "search")
|
||||
sys.stdout.buffer.write(body)
|
||||
return 0
|
||||
|
||||
|
||||
def _qwrite(method: str, url: str, data: bytes | None = None, headers: dict | None = None):
|
||||
"""A mutating request that survives an outage: routes through chorus_queue.safe_request,
|
||||
which enqueues the write and returns (202, ...) when the vault is unreachable (H2)."""
|
||||
import chorus_queue
|
||||
return chorus_queue.safe_request(method, url, data=data, headers=headers)
|
||||
|
||||
|
||||
def cmd_put(path: str, file_arg: str | None) -> int:
|
||||
data = read_body(file_arg)
|
||||
status, body = _qwrite("PUT", vault_url(path), data=data,
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
if status == 202:
|
||||
print(f"queued (offline): PUT {path} — will sync on the next reachable session")
|
||||
return 0
|
||||
check(status, body, f"put {path}")
|
||||
if VERIFY:
|
||||
status, _ = request("GET", vault_url(path))
|
||||
if status != 200:
|
||||
raise ChorusError(f"put {path}: write did not verify (GET returned {status})")
|
||||
print(f"ok: PUT {path}")
|
||||
# Keep the recall (BM25) index current when a corpus note is written directly —
|
||||
# session logs, journal notes, hand-authored entity notes. update_note early-returns
|
||||
# for non-corpus paths and never raises, so this can't fail or slow a normal put.
|
||||
try:
|
||||
import chorus_recall
|
||||
chorus_recall.update_note(path, data.decode("utf-8", errors="replace"))
|
||||
except Exception as exc: # noqa: BLE001 — index upkeep must never fail a put
|
||||
print(f"chorus.py: recall-index update skipped ({exc})", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_post(path: str, file_arg: str | None) -> int:
|
||||
data = read_body(file_arg)
|
||||
status, body = _qwrite("POST", vault_url(path), data=data,
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
if status == 202:
|
||||
print(f"queued (offline): POST {path} — will sync on the next reachable session")
|
||||
return 0
|
||||
check(status, body, f"post {path}")
|
||||
print(f"ok: POST {path}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_append(path: str, line: str) -> int:
|
||||
"""Idempotent append. Skips ONLY when the exact whole line already exists —
|
||||
a new line that is a substring of an existing one is still appended."""
|
||||
status, body = request("GET", vault_url(path))
|
||||
if status == 200:
|
||||
existing = [ln.rstrip("\r") for ln in body.decode(errors="replace").splitlines()]
|
||||
if line.rstrip("\r") in existing:
|
||||
print(f"skip: line already present in {path}")
|
||||
return 0
|
||||
elif status == 0:
|
||||
pass # offline: skip the idempotency read; queue the POST (flush dedups by content)
|
||||
elif status != 404:
|
||||
check(status, body, f"append(read) {path}")
|
||||
status, body = _qwrite("POST", vault_url(path), data=f"{line}\n".encode(),
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
if status == 202:
|
||||
print(f"queued (offline): APPEND {path} — will sync on the next reachable session")
|
||||
return 0
|
||||
check(status, body, f"append {path}")
|
||||
print(f"ok: APPEND {path}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_patch(path: str, op: str, target_type: str, target: str, file_arg: str | None) -> int:
|
||||
if op not in {"append", "prepend", "replace"}:
|
||||
raise ChorusError("patch: op must be append, prepend, or replace", 2)
|
||||
if target_type not in {"heading", "frontmatter", "block"}:
|
||||
raise ChorusError("patch: target-type must be heading, frontmatter, or block", 2)
|
||||
data = normalize_patch_body(read_body(file_arg), op, target_type)
|
||||
status, body = _qwrite(
|
||||
"PATCH",
|
||||
vault_url(path),
|
||||
data=data,
|
||||
headers={
|
||||
"Operation": op,
|
||||
"Target-Type": target_type,
|
||||
"Target": target,
|
||||
"Content-Type": "application/json" if target_type == "frontmatter" else "text/markdown",
|
||||
},
|
||||
)
|
||||
if status == 202:
|
||||
print(f"queued (offline): PATCH {op} '{target}' -> {path} — will sync on the next reachable session")
|
||||
return 0
|
||||
check(status, body, f"patch {path} ({target})")
|
||||
print(f"ok: PATCH {op} {target_type} '{target}' -> {path}")
|
||||
return 0
|
||||
|
||||
|
||||
def _yaml_flow(value) -> str:
|
||||
"""Render a JSON-decoded value as a YAML flow scalar/list for one frontmatter line."""
|
||||
if isinstance(value, list):
|
||||
return "[" + ", ".join(_yaml_flow(v) for v in value) + "]"
|
||||
if isinstance(value, bool):
|
||||
return "true" if value else "false"
|
||||
if value is None:
|
||||
return ""
|
||||
s = str(value)
|
||||
return s if re.match(r"^[A-Za-z0-9][A-Za-z0-9 ._/-]*$", s) else json.dumps(s)
|
||||
|
||||
|
||||
def _fm_upsert_line(text: str, field: str, rendered: str) -> str:
|
||||
"""Surgically set `field: rendered` inside the YAML frontmatter block, creating the
|
||||
key (or the whole block) if absent. Every other line is preserved verbatim — this is
|
||||
the safety property that makes it OK to fall back from a failed PATCH."""
|
||||
line = f"{field}: {rendered}".rstrip()
|
||||
lines = text.splitlines()
|
||||
if lines and lines[0].strip() == "---":
|
||||
for end in range(1, len(lines)):
|
||||
if lines[end].strip() == "---":
|
||||
for i in range(1, end): # key exists -> replace that one line
|
||||
if re.match(rf"^{re.escape(field)}\s*:", lines[i]):
|
||||
lines[i] = line
|
||||
break
|
||||
else:
|
||||
lines.insert(end, line) # else append just before the closing ---
|
||||
return "\n".join(lines) + ("\n" if text.endswith("\n") else "")
|
||||
return f"---\n{line}\n---\n{text}" # no frontmatter at all -> prepend a minimal block
|
||||
|
||||
|
||||
def cmd_fm(path: str, field: str, value: str) -> int:
|
||||
"""Create-or-replace a frontmatter scalar. PATCH replace handles the common case;
|
||||
a missing key returns 400 invalid-target (PATCH cannot create), which used to make
|
||||
`fm` fail on exactly the notes that needed repair — now it falls back to a surgical
|
||||
GET -> insert-one-line -> verified PUT."""
|
||||
data = normalize_json_scalar(value)
|
||||
try:
|
||||
return cmd_patch(path, "replace", "frontmatter", field, temp_file(data))
|
||||
except ChorusError as exc:
|
||||
if "HTTP 400" not in str(exc):
|
||||
raise
|
||||
status, body = request("GET", vault_url(path))
|
||||
check(status, body, f"fm {path}")
|
||||
new = _fm_upsert_line(body.decode(errors="replace"), field,
|
||||
_yaml_flow(json.loads(data.decode("utf-8"))))
|
||||
status, body = _qwrite("PUT", vault_url(path), data=new.encode("utf-8"),
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
if status == 202:
|
||||
print(f"queued (offline): FM {field} -> {path} — will sync on the next reachable session")
|
||||
return 0
|
||||
check(status, body, f"fm {path}")
|
||||
if VERIFY:
|
||||
status, body = request("GET", vault_url(path))
|
||||
if status != 200 or not re.search(rf"(?m)^{re.escape(field)}\s*:", body.decode(errors="replace")):
|
||||
raise ChorusError(f"fm {path}: created key '{field}' did not verify on read-back")
|
||||
print(f"ok: FM created '{field}' in {path}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_delete(path: str) -> int:
|
||||
status, body = _qwrite("DELETE", vault_url(path))
|
||||
if status == 202:
|
||||
print(f"queued (offline): DELETE {path} — will sync on the next reachable session")
|
||||
return 0
|
||||
check(status, body, f"delete {path}")
|
||||
print(f"ok: DELETE {path}")
|
||||
return 0
|
||||
|
||||
|
||||
def parse_lock_time(value: str) -> int:
|
||||
try:
|
||||
return int(dt.datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ")
|
||||
.replace(tzinfo=dt.timezone.utc).timestamp())
|
||||
except Exception:
|
||||
# Unparseable timestamp -> treat the lock as FRESH (return "now"), so a
|
||||
# garbled lock is respected rather than silently stomped.
|
||||
return int(time.time())
|
||||
|
||||
|
||||
def cmd_lock(owner: str, quiet: bool = False) -> int:
|
||||
status, body = request("GET", vault_url(LOCK_PATH))
|
||||
if status == 200 and body.strip():
|
||||
current = body.decode(errors="replace").strip()
|
||||
held_owner, _, held_iso = current.partition(" @ ")
|
||||
if held_owner != owner and int(time.time()) - parse_lock_time(held_iso) < LOCK_TTL:
|
||||
print(f"lock held by '{held_owner}' since {held_iso} (fresh)", file=sys.stderr)
|
||||
return 75
|
||||
status, body = request("PUT", vault_url(LOCK_PATH), data=f"{owner} @ {now_iso()}\n".encode(),
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
check(status, body, "lock")
|
||||
# Read-back confirm we actually hold it. The REST API has no compare-and-swap,
|
||||
# so two racers can both PUT; last writer wins. Whoever does not own the file
|
||||
# on read-back backs off instead of proceeding under a false lock.
|
||||
status, body = request("GET", vault_url(LOCK_PATH))
|
||||
if status == 200:
|
||||
held_owner = body.decode(errors="replace").split(" @ ", 1)[0].strip()
|
||||
if held_owner != owner:
|
||||
print(f"lock race: now held by '{held_owner}', not '{owner}' — backing off", file=sys.stderr)
|
||||
return 75
|
||||
if not quiet:
|
||||
print(f"ok: locked by {owner}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_unlock(owner: str, quiet: bool = False) -> int:
|
||||
status, body = request("GET", vault_url(LOCK_PATH))
|
||||
if status == 200:
|
||||
held_owner = body.decode(errors="replace").split(" @ ", 1)[0].strip()
|
||||
if held_owner != owner:
|
||||
print(f"lock owned by '{held_owner}', not '{owner}' — not releasing", file=sys.stderr)
|
||||
return 75
|
||||
status, body = request("DELETE", vault_url(LOCK_PATH))
|
||||
if status != 404:
|
||||
check(status, body, "unlock")
|
||||
if not quiet:
|
||||
print("ok: unlocked")
|
||||
return 0
|
||||
|
||||
|
||||
def extract_heading(markdown: str, heading: str) -> str:
|
||||
"""Return the body under a `## <heading>` up to the next `## ` heading."""
|
||||
out: list[str] = []
|
||||
capture = False
|
||||
marker = f"## {heading}"
|
||||
for line in markdown.splitlines():
|
||||
if line.strip() == marker:
|
||||
capture = True
|
||||
continue
|
||||
if capture and line.startswith("## "):
|
||||
break
|
||||
if capture:
|
||||
out.append(line)
|
||||
return "\n".join(out).strip()
|
||||
|
||||
|
||||
def cmd_scope(subcommand: str, text: str | None = None, as_json: bool = False) -> int:
|
||||
path = "_agent/context/current-context.md"
|
||||
status, body = request("GET", vault_url(path))
|
||||
check(status, body, f"scope {subcommand}")
|
||||
current = body.decode(errors="replace")
|
||||
|
||||
if subcommand == "show":
|
||||
scope_text = extract_heading(current, "Scope")
|
||||
scope_updated = ""
|
||||
for line in current.splitlines():
|
||||
if line.startswith("scope_updated:"):
|
||||
scope_updated = line.split(":", 1)[1].strip().strip('"').strip("'")
|
||||
break
|
||||
sessions_since = None
|
||||
# Count session logs dated after scope_updated (the drift signal).
|
||||
if scope_updated:
|
||||
status, body = request("GET", vault_url("_agent/sessions/"))
|
||||
if status == 200:
|
||||
try:
|
||||
files = json.loads(body).get("files", [])
|
||||
sessions_since = sum(1 for f in files if f.endswith(".md") and f[:10] > scope_updated)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
if as_json:
|
||||
print(json.dumps({"ok": True, "action": "scope-show", "scope": scope_text,
|
||||
"scope_updated": scope_updated or None,
|
||||
"sessions_since": sessions_since}, ensure_ascii=False))
|
||||
return 0
|
||||
print("-- Active scope --")
|
||||
print(scope_text)
|
||||
print(f"scope_updated: {scope_updated or '<missing — drift cannot be detected; run scope set or repair>'}")
|
||||
if sessions_since is not None:
|
||||
print(f"sessions logged since: {sessions_since}")
|
||||
return 0
|
||||
|
||||
if subcommand != "set":
|
||||
raise ChorusError("scope: use 'show' or 'set \"<text>\"'", 2)
|
||||
if not text:
|
||||
raise ChorusError("scope set needs the new scope text", 2)
|
||||
prior = extract_heading(current, "Scope").replace("\n", " ").strip()[:140] or "(prior scope)"
|
||||
cmd_patch(path, "prepend", "heading", "Current Context::Scope History",
|
||||
temp_file(f"- {today()}: {prior}\n".encode()))
|
||||
cmd_patch(path, "replace", "heading", "Current Context::Scope",
|
||||
temp_file(f"{text}\n".encode()))
|
||||
try:
|
||||
cmd_patch(path, "replace", "frontmatter", "scope_updated",
|
||||
temp_file(json.dumps(today()).encode()))
|
||||
except ChorusError as exc:
|
||||
raise ChorusError(
|
||||
"scope set: body switched, but scope_updated frontmatter is missing "
|
||||
f"(run bootstrap.py to add it) [{exc}]"
|
||||
)
|
||||
print(f"ok: scope switched (prior archived to Scope History; scope_updated={today()})")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_load() -> int:
|
||||
"""Cold-start orientation: the canonical 6 reads in one call. 404s on today's
|
||||
daily note and the inbox are normal (printed as absent, not errors)."""
|
||||
if not chorus_config.is_configured(_CFG):
|
||||
print("chorus: NOT CONFIGURED — this machine has no usable CHORUS key file yet.")
|
||||
print(f"Expected at: {chorus_config.config_path()}")
|
||||
print("ACTION: ask the operator for their chorus-memory config file (it holds the")
|
||||
print(" vault owner, endpoint, and API key), then install it with ONE of:")
|
||||
print(" python3 chorus.py config import <path-to-their-file>")
|
||||
print(" python3 chorus.py config set --owner \"…\" --endpoint \"https://…\" --key \"…\"")
|
||||
print("Then re-run load. (Memory is unavailable until configured.)")
|
||||
return 78 # distinct: configuration required
|
||||
targets = [
|
||||
("marker", MARKER_PATH),
|
||||
("preferences", "_agent/memory/semantic/operator-preferences.md"),
|
||||
("context", "_agent/context/current-context.md"),
|
||||
("heartbeat", "_agent/heartbeat/last-session.md"),
|
||||
("today", f"journal/daily/{today()}.md"),
|
||||
("inbox", "inbox/captures/inbox.md"),
|
||||
]
|
||||
import chorus_queue
|
||||
# H2: sync any writes queued during a prior outage, best-effort and quiet on empty.
|
||||
try:
|
||||
synced = chorus_queue.flush()
|
||||
if synced:
|
||||
print(f"(synced {synced} write(s) queued during a prior offline session)\n")
|
||||
except Exception as exc: # noqa: BLE001 — never let queue upkeep block a load
|
||||
print(f"(queue flush skipped: {exc})\n", file=sys.stderr)
|
||||
|
||||
marker_missing = False
|
||||
heartbeat_absent = False
|
||||
offline = False
|
||||
for label, path in targets:
|
||||
if label == "marker":
|
||||
path, status, body = marker_get() # falls back to a legacy ECHO marker
|
||||
else:
|
||||
status, body = request("GET", vault_url(path))
|
||||
if status == 200:
|
||||
chorus_queue.cache_put(path, body) # refresh last-known-good for offline fallback
|
||||
print(f"===== {label}: {path} (HTTP 200) =====")
|
||||
text = body.decode(errors="replace")
|
||||
sys.stdout.write(text)
|
||||
if not text.endswith("\n"):
|
||||
sys.stdout.write("\n")
|
||||
elif status == 404:
|
||||
print(f"===== {label}: {path} (HTTP 404) =====")
|
||||
print("(absent — fine)")
|
||||
if label == "marker":
|
||||
marker_missing = True
|
||||
if label == "heartbeat":
|
||||
heartbeat_absent = True
|
||||
elif status == 0: # vault unreachable -> degrade to last-known-good cache
|
||||
offline = True
|
||||
cached = chorus_queue.cache_get(path)
|
||||
if cached is not None:
|
||||
print(f"===== {label}: {path} (OFFLINE — serving stale cache) =====")
|
||||
sys.stdout.write(cached.decode(errors="replace"))
|
||||
if not cached.endswith(b"\n"):
|
||||
sys.stdout.write("\n")
|
||||
else:
|
||||
print(f"===== {label}: {path} (OFFLINE — no cache) =====")
|
||||
print("(unreachable and not cached)")
|
||||
else:
|
||||
print(f"===== {label}: {path} (HTTP {status}) =====")
|
||||
print(f"(error HTTP {status}: {body.decode(errors='replace')[:200]})")
|
||||
print()
|
||||
# M3: heartbeat pointer missing/stale -> fall back to the recent sessions listing
|
||||
# (matches the documented loading procedure) so orientation works without the pointer.
|
||||
if heartbeat_absent and not offline:
|
||||
st, body = request("GET", vault_url("_agent/sessions/"))
|
||||
if st == 200:
|
||||
try:
|
||||
files = sorted((f for f in json.loads(body).get("files", []) if f.endswith(".md")),
|
||||
reverse=True)
|
||||
except json.JSONDecodeError:
|
||||
files = []
|
||||
if files:
|
||||
print("===== recent sessions (heartbeat absent — fallback) =====")
|
||||
for f in files[:5]:
|
||||
print(f" _agent/sessions/{f}")
|
||||
print()
|
||||
if offline:
|
||||
print("NOTE: vault unreachable — context above is last-known-good cache; writes this "
|
||||
"session will be queued and synced when the vault returns.")
|
||||
if marker_missing:
|
||||
print("NOTE: marker absent — vault not bootstrapped. Run bootstrap.py before relying on memory.")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_config(args) -> int:
|
||||
"""show | init | set the machine-local owner/endpoint/key config."""
|
||||
path = chorus_config.config_path()
|
||||
if args.action == "init":
|
||||
p, created = chorus_config.init_template()
|
||||
print(f"ok: wrote config template to {p} — edit it with your owner/endpoint/key"
|
||||
if created else f"config already exists at {p} (left unchanged)")
|
||||
return 0
|
||||
if args.action == "import":
|
||||
if not args.path:
|
||||
print("config import: pass the path to a config file "
|
||||
"(JSON with owner/endpoint/key)", file=sys.stderr)
|
||||
return 2
|
||||
try:
|
||||
p = chorus_config.import_file(args.path)
|
||||
except RuntimeError as exc:
|
||||
print(exc, file=sys.stderr)
|
||||
return 2
|
||||
print(f"ok: imported config to {p} (chmod 600 best-effort)")
|
||||
return 0
|
||||
if args.action == "set":
|
||||
if args.owner is None and args.endpoint is None and args.key is None:
|
||||
print("config set: pass at least one of --owner / --endpoint / --key", file=sys.stderr)
|
||||
return 2
|
||||
p = chorus_config.write_config(args.owner, args.endpoint, args.key)
|
||||
print(f"ok: updated {p} (chmod 600 best-effort)")
|
||||
return 0
|
||||
# show — never prints the key in the clear
|
||||
cfg = chorus_config.load()
|
||||
key = cfg["key"]
|
||||
redacted = (key[:4] + "…" + key[-4:]) if len(key) >= 12 else ("set" if key else "")
|
||||
print(f"config file: {path}" + ("" if path.exists() else " (absent)"))
|
||||
for field, shown in (("owner", cfg["owner"]), ("endpoint", cfg["endpoint"]), ("key", redacted)):
|
||||
src = chorus_config.source(field)
|
||||
print(f" {field:9} {shown or '(unset)'} [{src}]")
|
||||
return 0
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description="Validated cross-platform CHORUS vault client")
|
||||
sub = parser.add_subparsers(dest="cmd", required=True)
|
||||
sub.add_parser("load")
|
||||
sub.add_parser("flush") # H2: replay writes queued during a prior outage
|
||||
sub.add_parser("doctor") # M3: one-call readiness check
|
||||
sub.add_parser("get").add_argument("path")
|
||||
sub.add_parser("map").add_argument("path")
|
||||
sub.add_parser("ls").add_argument("path")
|
||||
sub.add_parser("search").add_argument("query", nargs="+")
|
||||
p = sub.add_parser("put"); p.add_argument("path"); p.add_argument("file", nargs="?")
|
||||
p = sub.add_parser("post"); p.add_argument("path"); p.add_argument("file", nargs="?")
|
||||
p = sub.add_parser("append"); p.add_argument("path"); p.add_argument("line")
|
||||
p = sub.add_parser("patch")
|
||||
p.add_argument("path"); p.add_argument("op"); p.add_argument("target_type")
|
||||
p.add_argument("target"); p.add_argument("file", nargs="?")
|
||||
p = sub.add_parser("fm"); p.add_argument("path"); p.add_argument("field"); p.add_argument("value")
|
||||
p = sub.add_parser("bump"); p.add_argument("path"); p.add_argument("date", nargs="?")
|
||||
sub.add_parser("delete").add_argument("path")
|
||||
sub.add_parser("lock").add_argument("owner")
|
||||
sub.add_parser("unlock").add_argument("owner")
|
||||
p = sub.add_parser("config") # machine-local owner/endpoint/key (see chorus_config)
|
||||
p.add_argument("action", nargs="?", default="show", choices=["show", "init", "set", "import"])
|
||||
p.add_argument("path", nargs="?") # for `config import <path>`
|
||||
p.add_argument("--owner"); p.add_argument("--endpoint"); p.add_argument("--key")
|
||||
p = sub.add_parser("scope")
|
||||
p.add_argument("subcommand", nargs="?", default="show", choices=["show", "set"])
|
||||
p.add_argument("text", nargs="?")
|
||||
p.add_argument("--json", action="store_true") # structured scope + freshness
|
||||
p = sub.add_parser("reflect") # H5: apply a JSON proposal set (dry-run unless --apply)
|
||||
p.add_argument("file", nargs="?")
|
||||
p.add_argument("--apply", action="store_true")
|
||||
p = sub.add_parser("triage") # one-tap inbox triage (reflect pipeline + audit log)
|
||||
p.add_argument("file", nargs="?") # proposals JSON; omit to list
|
||||
p.add_argument("--list", action="store_true", dest="list_inbox")
|
||||
p.add_argument("--json", action="store_true") # structured inbox listing
|
||||
p.add_argument("--apply", action="store_true") # route + write processing log
|
||||
|
||||
# High-level operations (entity index + cross-links + recall). See chorus_ops.py.
|
||||
sub.add_parser("resolve").add_argument("mention", nargs="+")
|
||||
p = sub.add_parser("recall"); p.add_argument("query", nargs="+"); p.add_argument("--limit", type=int, default=6)
|
||||
p.add_argument("--json", action="store_true") # structured hits + neighbourhood
|
||||
p = sub.add_parser("link"); p.add_argument("a"); p.add_argument("b")
|
||||
p.add_argument("--json", action="store_true")
|
||||
p = sub.add_parser("capture")
|
||||
p.add_argument("title")
|
||||
p.add_argument("file", nargs="?")
|
||||
p.add_argument("--kind")
|
||||
p.add_argument("--inbox", action="store_true")
|
||||
p.add_argument("--status", default="")
|
||||
p.add_argument("--aliases", default="")
|
||||
p.add_argument("--source", default="")
|
||||
p.add_argument("--tags", default="") # extra tags; the kind is always seeded
|
||||
p.add_argument("--date")
|
||||
p.add_argument("--domain", default="business")
|
||||
p.add_argument("--no-log", action="store_true")
|
||||
p.add_argument("--json", action="store_true") # M4: emit a JSON result envelope
|
||||
p.add_argument("--dry-run", action="store_true") # M4: preview the plan, write nothing
|
||||
p.add_argument("--force", action="store_true") # create despite a duplicate-gate hit
|
||||
p.add_argument("--merge-into", metavar="SLUG") # route as an update to this entity
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = build_parser().parse_args(argv)
|
||||
try:
|
||||
if args.cmd == "load":
|
||||
return cmd_load()
|
||||
if args.cmd == "doctor":
|
||||
import chorus_doctor
|
||||
return chorus_doctor.run()
|
||||
if args.cmd == "flush":
|
||||
import chorus_queue
|
||||
n = chorus_queue.flush()
|
||||
remaining = len(chorus_queue.pending())
|
||||
if n:
|
||||
print(f"ok: flushed {n} queued write(s)"
|
||||
+ (f"; {remaining} still queued (vault unreachable)" if remaining else ""))
|
||||
elif remaining:
|
||||
print(f"ok: {remaining} write(s) still queued (vault unreachable)")
|
||||
else:
|
||||
print("ok: queue empty")
|
||||
return 0
|
||||
if args.cmd == "config":
|
||||
return cmd_config(args)
|
||||
if args.cmd == "get":
|
||||
return cmd_get(args.path)
|
||||
if args.cmd == "map":
|
||||
return cmd_map(args.path)
|
||||
if args.cmd == "ls":
|
||||
return cmd_ls(args.path)
|
||||
if args.cmd == "search":
|
||||
return cmd_search(args.query)
|
||||
if args.cmd == "put":
|
||||
return cmd_put(args.path, args.file)
|
||||
if args.cmd == "post":
|
||||
return cmd_post(args.path, args.file)
|
||||
if args.cmd == "append":
|
||||
return cmd_append(args.path, args.line)
|
||||
if args.cmd == "patch":
|
||||
return cmd_patch(args.path, args.op, args.target_type, args.target, args.file)
|
||||
if args.cmd == "fm":
|
||||
return cmd_fm(args.path, args.field, args.value)
|
||||
if args.cmd == "bump":
|
||||
return cmd_fm(args.path, "updated", json.dumps(args.date or today()))
|
||||
if args.cmd == "delete":
|
||||
return cmd_delete(args.path)
|
||||
if args.cmd == "lock":
|
||||
return cmd_lock(args.owner)
|
||||
if args.cmd == "unlock":
|
||||
return cmd_unlock(args.owner)
|
||||
if args.cmd == "scope":
|
||||
return cmd_scope(args.subcommand, args.text, as_json=args.json)
|
||||
if args.cmd == "reflect":
|
||||
import chorus_reflect
|
||||
raw = read_body(args.file).decode("utf-8", errors="replace")
|
||||
try:
|
||||
proposals = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ChorusError(f"reflect: proposals must be a JSON array ({exc})", 2)
|
||||
if not isinstance(proposals, list):
|
||||
raise ChorusError("reflect: proposals must be a JSON array of objects", 2)
|
||||
return chorus_reflect.apply(proposals, confirm=args.apply)
|
||||
if args.cmd == "triage":
|
||||
import chorus_triage
|
||||
if args.list_inbox or not args.file:
|
||||
return chorus_triage.list_inbox(as_json=args.json)
|
||||
raw = read_body(args.file).decode("utf-8", errors="replace")
|
||||
try:
|
||||
proposals = json.loads(raw)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ChorusError(f"triage: proposals must be a JSON array ({exc})", 2)
|
||||
if not isinstance(proposals, list):
|
||||
raise ChorusError("triage: proposals must be a JSON array of objects", 2)
|
||||
return chorus_triage.apply(proposals, confirm=args.apply)
|
||||
if args.cmd in ("resolve", "recall", "link", "capture"):
|
||||
import chorus_ops # lazy: only the high-level ops pull in the index/link modules
|
||||
if args.cmd == "resolve":
|
||||
return chorus_ops.resolve(" ".join(args.mention))
|
||||
if args.cmd == "recall":
|
||||
return chorus_ops.recall(args.query, limit=args.limit, as_json=args.json)
|
||||
if args.cmd == "link":
|
||||
return chorus_ops.link(args.a, args.b, as_json=args.json)
|
||||
return chorus_ops.capture(
|
||||
args.kind, args.title, args.file, status_v=args.status,
|
||||
aliases=args.aliases.split(",") if args.aliases else [],
|
||||
sources=args.source.split(",") if args.source else [],
|
||||
tags=args.tags.split(",") if args.tags else [],
|
||||
date=args.date, domain=args.domain, inbox=args.inbox, no_log=args.no_log,
|
||||
as_json=args.json, dry_run=args.dry_run,
|
||||
force=args.force, merge_into=args.merge_into)
|
||||
except ChorusError as exc:
|
||||
print(f"chorus.py: {exc}", file=sys.stderr)
|
||||
return exc.code
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,82 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_concurrency.py — H3: make the multi-writer story actually safe. [v1.0 SCAFFOLD — not yet wired]
|
||||
|
||||
The plugin advertises a shared vault (Claude Code + CoWork), but two real races exist:
|
||||
1. The entity index is a full-file load -> mutate -> PUT (chorus_index.load/save) with
|
||||
NO lock. Two concurrent `capture` calls both read the old index; last PUT wins;
|
||||
the other entity is silently dropped from the index (the note file survives, but
|
||||
resolve/recall miss it until the next sweep).
|
||||
2. The advisory lock is MANUAL (chorus.py cmd_lock) — capture / scope set never take it;
|
||||
it relies on the model remembering to.
|
||||
|
||||
This module provides (a) a context manager that auto-acquires/releases the existing
|
||||
advisory lock around a write burst, with crash-safe release, and (b) an atomic
|
||||
index-update that re-reads UNDER the lock before saving, so concurrent captures merge
|
||||
instead of clobber.
|
||||
|
||||
ACCEPTANCE (v1.0 roadmap H3, shipped; roadmap retired — see git history):
|
||||
- two concurrent captures both land in the index (no lost entries)
|
||||
- lock auto-released on normal AND error exit
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import chorus # noqa: E402
|
||||
import chorus_index as idx_mod # noqa: E402
|
||||
|
||||
|
||||
def auto_owner() -> str:
|
||||
"""A stable-per-process owner id: client tag + pid. Avoids Math.random/time-based
|
||||
ids; pid is unique enough for cooperative locking and is reproducible in tests."""
|
||||
tag = os.environ.get("CHORUS_CLIENT", "cc")
|
||||
return f"{tag}-{os.getpid()}"
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def vault_lock(owner: str | None = None, required: bool = False):
|
||||
"""Hold the advisory lock for a write burst; release on exit (incl. exceptions).
|
||||
|
||||
Yields `held: bool`. The lock is cooperative (read-back-confirmed, TTL-reclaimable),
|
||||
so if another session holds it we do NOT block: with `required=False` we yield
|
||||
`held=False` and let the caller proceed (advisory) or warn; with `required=True` we
|
||||
raise. Release is best-effort and never masks the body's own exception.
|
||||
"""
|
||||
owner = owner or auto_owner()
|
||||
rc = chorus.cmd_lock(owner, quiet=True)
|
||||
held = rc == 0
|
||||
if not held and required:
|
||||
raise chorus.ChorusError(f"vault_lock: could not acquire lock for '{owner}' (held elsewhere)", 75)
|
||||
try:
|
||||
yield held
|
||||
finally:
|
||||
if held:
|
||||
try:
|
||||
chorus.cmd_unlock(owner, quiet=True)
|
||||
except Exception: # noqa: BLE001 — release is best-effort; TTL reclaims a leak
|
||||
pass
|
||||
|
||||
|
||||
def atomic_index_update(mutate, owner: str | None = None):
|
||||
"""Apply `mutate(index)` to the entity index without losing a concurrent writer's
|
||||
entry: take the lock, **re-read the index fresh** (not a stale in-memory copy),
|
||||
run the mutator, save, release. Returns the freshly-read-and-saved index.
|
||||
|
||||
The fresh re-read inside the critical section is the core of the fix — even if the
|
||||
advisory lock can't be taken (another session active), reading immediately before
|
||||
the save collapses the read-modify-write window to two calls instead of spanning the
|
||||
whole capture, so an unlocked update is still far safer than the prior code.
|
||||
"""
|
||||
with vault_lock(owner) as held:
|
||||
index = idx_mod.load()
|
||||
mutate(index)
|
||||
idx_mod.save(index)
|
||||
if not held:
|
||||
print("chorus_concurrency: entity index updated WITHOUT the advisory lock "
|
||||
"(another session may be active); the fresh re-read minimizes the race.",
|
||||
file=sys.stderr)
|
||||
return index
|
||||
@@ -0,0 +1,266 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_config.py — resolve the vault owner, endpoint, and API key for CHORUS.
|
||||
|
||||
The plugin ships NO owner, endpoint, or key — it is fully user-agnostic. On each
|
||||
machine, a machine-local JSON config supplies them:
|
||||
|
||||
~/.claude/chorus-memory/config.json
|
||||
{
|
||||
"owner": "Full Name (how the agent refers to the vault owner, third person)",
|
||||
"endpoint": "https://your-obsidian-rest-endpoint.example.com",
|
||||
"key": "<obsidian-local-rest-api-bearer-token>"
|
||||
}
|
||||
|
||||
This file is per-machine: it must be created on each fresh install, is never
|
||||
committed, and is never written into a vault note. To create it, run
|
||||
`python3 chorus.py config init` (writes a placeholder template when absent) and edit
|
||||
it, or `python3 chorus.py config set --owner ... --endpoint ... --key ...`.
|
||||
|
||||
Resolution — a COMPLETE baked credential set wins unconditionally:
|
||||
|
||||
If the artifact carries both a baked endpoint AND key (a per-user build), those
|
||||
baked values are used as-is and CANNOT be shadowed by env vars or a config file.
|
||||
This is the default path: a delivered per-user plugin "just works" with no setup,
|
||||
and a stale CHORUS_* env var or a leftover/placeholder ~/.claude config.json can
|
||||
never override or break it.
|
||||
|
||||
Otherwise (nothing baked — the generic published plugin), fall back per-field,
|
||||
first hit wins:
|
||||
owner env CHORUS_OWNER -> config "owner"
|
||||
endpoint env CHORUS_BASE -> config "endpoint"
|
||||
key env CHORUS_KEY -> config "key"
|
||||
|
||||
The config directory honors $CLAUDE_CONFIG_DIR (defaults to ~/.claude); the config
|
||||
file path itself can be overridden with $CHORUS_CONFIG. Pure stdlib only.
|
||||
|
||||
BAKED CREDENTIALS (the default tier): the DEFAULT_* constants below are EMPTY in
|
||||
source — the committed plugin ships zero credentials. `build.py --bake-key`
|
||||
substitutes a specific user's owner/endpoint/key into them at package time,
|
||||
producing a per-user artifact that needs no config file. This is how the plugin
|
||||
works in the CoWork sandbox, where the user's ~/.claude is not bridged: the plugin
|
||||
itself (mounted read-only every session) carries the values. A baked artifact is
|
||||
secret-bearing — deliver it directly to that one user, NEVER commit or publish it.
|
||||
On a normal host the empty defaults mean the env/config-file path takes over.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# Build-time injection targets — MUST stay empty string literals in source so the
|
||||
# committed tree ships no secret. `build.py --bake-key` rewrites these lines.
|
||||
DEFAULT_OWNER = ""
|
||||
DEFAULT_BASE = ""
|
||||
DEFAULT_KEY = ""
|
||||
|
||||
# ---------------------------------------------------------------- legacy shim
|
||||
# CHORUS is a fork of ECHO. For one major version, legacy ECHO_* environment
|
||||
# variables keep working: at import time each ECHO_<X> is adopted as CHORUS_<X>
|
||||
# whenever CHORUS_<X> is not itself set. Scripts only ever read CHORUS_*.
|
||||
_LEGACY_ENV_SUFFIXES = (
|
||||
"OWNER", "BASE", "KEY", "CONFIG", "TODAY", "VERIFY", "LOCK_TTL", "TIMEOUT",
|
||||
"WORKERS", "STATE_DIR", "CLIENT", "DUP_GATE", "REFLECT_MIN_TURNS",
|
||||
"FRESH_HALF_LIFE",
|
||||
)
|
||||
|
||||
|
||||
def _alias_legacy_env() -> None:
|
||||
for suffix in _LEGACY_ENV_SUFFIXES:
|
||||
new, old = f"CHORUS_{suffix}", f"ECHO_{suffix}"
|
||||
if not os.environ.get(new) and os.environ.get(old):
|
||||
os.environ[new] = os.environ[old]
|
||||
|
||||
|
||||
_alias_legacy_env()
|
||||
|
||||
TEMPLATE = {
|
||||
"owner": "Full Name — how the agent should refer to the vault owner (third person)",
|
||||
"endpoint": "https://your-obsidian-rest-endpoint.example.com",
|
||||
"key": "<obsidian-local-rest-api-bearer-token>",
|
||||
}
|
||||
|
||||
|
||||
def claude_dir() -> Path:
|
||||
"""The Claude config directory ($CLAUDE_CONFIG_DIR, else ~/.claude)."""
|
||||
return Path(os.environ.get("CLAUDE_CONFIG_DIR") or (Path.home() / ".claude"))
|
||||
|
||||
|
||||
def config_path() -> Path:
|
||||
"""Absolute path to the machine-local config file (the CHORUS path — this is
|
||||
where writes always land)."""
|
||||
override = os.environ.get("CHORUS_CONFIG")
|
||||
if override:
|
||||
return Path(override).expanduser()
|
||||
return claude_dir() / "chorus-memory" / "config.json"
|
||||
|
||||
|
||||
def legacy_config_path() -> Path:
|
||||
"""The pre-fork ECHO config location, honored read-only as a fallback."""
|
||||
return claude_dir() / "echo-memory" / "config.json"
|
||||
|
||||
|
||||
def read_config_path() -> Path:
|
||||
"""The config file to READ: the CHORUS path when overridden or present,
|
||||
else a legacy ECHO config if one exists (fork adoption path). Writes never
|
||||
go to the legacy path — `config set`/`import` land at config_path()."""
|
||||
p = config_path()
|
||||
if os.environ.get("CHORUS_CONFIG") or p.exists():
|
||||
return p
|
||||
legacy = legacy_config_path()
|
||||
return legacy if legacy.exists() else p
|
||||
|
||||
|
||||
def _from_file() -> dict:
|
||||
p = read_config_path()
|
||||
if not p.exists():
|
||||
return {}
|
||||
try:
|
||||
data = json.loads(p.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return {}
|
||||
return data if isinstance(data, dict) else {}
|
||||
|
||||
|
||||
def baked_complete() -> bool:
|
||||
"""True when the artifact carries a usable baked endpoint AND key — i.e. a
|
||||
per-user `--bake-key` build. When True, the baked values win unconditionally."""
|
||||
return bool(DEFAULT_BASE and DEFAULT_KEY)
|
||||
|
||||
|
||||
def load() -> dict:
|
||||
"""Return {owner, endpoint, key}. A complete baked set wins unconditionally;
|
||||
otherwise env overrides the file, missing -> "".
|
||||
|
||||
Never raises — callers that REQUIRE a field use resolve_endpoint()/resolve_key(),
|
||||
which raise a helpful error. This keeps `--help`, `config`, and `doctor` usable
|
||||
even when nothing is configured yet."""
|
||||
f = _from_file()
|
||||
if baked_complete():
|
||||
# Per-user artifact: baked creds are authoritative and must not be shadowed
|
||||
# by a stale env var or a leftover/placeholder config file. Owner is purely
|
||||
# descriptive, so it may still fall back when not baked.
|
||||
return {
|
||||
"owner": DEFAULT_OWNER or os.environ.get("CHORUS_OWNER") or f.get("owner") or "",
|
||||
"endpoint": DEFAULT_BASE.rstrip("/"),
|
||||
"key": DEFAULT_KEY,
|
||||
}
|
||||
endpoint = (os.environ.get("CHORUS_BASE") or f.get("endpoint") or "").rstrip("/")
|
||||
return {
|
||||
"owner": os.environ.get("CHORUS_OWNER") or f.get("owner") or "",
|
||||
"endpoint": endpoint,
|
||||
"key": os.environ.get("CHORUS_KEY") or f.get("key") or "",
|
||||
}
|
||||
|
||||
|
||||
def _missing(field: str, env: str) -> RuntimeError:
|
||||
return RuntimeError(
|
||||
f"chorus: no {field} configured. Set {env}, or create {config_path()} "
|
||||
f"(run `chorus.py config init` to scaffold it, then edit). "
|
||||
f'Expected JSON keys: "owner", "endpoint", "key".'
|
||||
)
|
||||
|
||||
|
||||
def resolve_endpoint() -> str:
|
||||
ep = load()["endpoint"]
|
||||
if not ep:
|
||||
raise _missing("endpoint", "CHORUS_BASE")
|
||||
return ep
|
||||
|
||||
|
||||
def resolve_key() -> str:
|
||||
k = load()["key"]
|
||||
if not k:
|
||||
raise _missing("key", "CHORUS_KEY")
|
||||
return k
|
||||
|
||||
|
||||
def resolve_owner() -> str:
|
||||
"""The owner display name. Descriptive only, so an empty value is tolerated."""
|
||||
return load()["owner"]
|
||||
|
||||
|
||||
def is_configured(cfg: dict | None = None) -> bool:
|
||||
"""True only when endpoint AND key are present and NOT the untouched template
|
||||
placeholders — so a freshly `config init`-ed (but unedited) file reads as not yet
|
||||
configured, which is what triggers the first-run "ask for the key file" prompt."""
|
||||
c = cfg if cfg is not None else load()
|
||||
ep, key = c.get("endpoint", ""), c.get("key", "")
|
||||
if not ep or not key:
|
||||
return False
|
||||
if "your-obsidian-rest-endpoint" in ep or key.startswith("<"):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def source(field: str) -> str:
|
||||
"""Where a field is currently coming from — for `config`/`doctor` reporting."""
|
||||
baked = {"owner": DEFAULT_OWNER, "endpoint": DEFAULT_BASE, "key": DEFAULT_KEY}[field]
|
||||
# A complete baked set wins unconditionally (see load()): endpoint/key always
|
||||
# come from the artifact, and owner too whenever it was baked.
|
||||
if baked_complete() and (field in ("endpoint", "key") or baked):
|
||||
return "baked-in (plugin)"
|
||||
env = {"owner": "CHORUS_OWNER", "endpoint": "CHORUS_BASE", "key": "CHORUS_KEY"}[field]
|
||||
if os.environ.get(env):
|
||||
return f"{env} env"
|
||||
rp = read_config_path()
|
||||
if rp.exists() and _from_file().get(field):
|
||||
return "config file" if rp == config_path() else "config file (legacy echo-memory path)"
|
||||
if baked:
|
||||
return "baked-in (plugin)"
|
||||
return "unset"
|
||||
|
||||
|
||||
def template_text() -> str:
|
||||
return json.dumps(TEMPLATE, indent=2, ensure_ascii=False) + "\n"
|
||||
|
||||
|
||||
def _secure(p: Path) -> None:
|
||||
try:
|
||||
os.chmod(p, 0o600) # advisory on filesystems without POSIX mode bits (e.g. Windows)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def write_config(owner: str, endpoint: str, key: str) -> Path:
|
||||
"""Persist a filled-in config (merging over any existing values), chmod 0600."""
|
||||
cur = _from_file()
|
||||
merged = {
|
||||
"owner": owner if owner is not None else cur.get("owner", ""),
|
||||
"endpoint": (endpoint if endpoint is not None else cur.get("endpoint", "")).rstrip("/"),
|
||||
"key": key if key is not None else cur.get("key", ""),
|
||||
}
|
||||
p = config_path()
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(json.dumps(merged, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
|
||||
_secure(p)
|
||||
return p
|
||||
|
||||
|
||||
def import_file(src) -> Path:
|
||||
"""Adopt a config file someone was handed: validate it has a usable endpoint+key,
|
||||
then copy it into config_path(). The 'provide the file -> plugin installs it' path."""
|
||||
p = Path(src).expanduser()
|
||||
if not p.exists():
|
||||
raise RuntimeError(f"config import: file not found: {p}")
|
||||
try:
|
||||
data = json.loads(p.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError) as exc:
|
||||
raise RuntimeError(f"config import: {p} is not valid JSON ({exc})")
|
||||
if not isinstance(data, dict) or not is_configured(data):
|
||||
raise RuntimeError(
|
||||
'config import: file must be JSON with a real "endpoint" and "key" '
|
||||
'(and ideally "owner") — not the template placeholders.')
|
||||
return write_config(data.get("owner", ""), data["endpoint"], data["key"])
|
||||
|
||||
|
||||
def init_template() -> tuple[Path, bool]:
|
||||
"""Write the placeholder template to config_path() if absent.
|
||||
Returns (path, created) — created is False when a config already exists."""
|
||||
p = config_path()
|
||||
if p.exists():
|
||||
return p, False
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(template_text(), encoding="utf-8")
|
||||
_secure(p)
|
||||
return p, True
|
||||
@@ -0,0 +1,88 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_doctor.py — M3: one-call readiness check. [v1.0 SCAFFOLD — run() implemented]
|
||||
|
||||
There is no single "is everything OK" command. doctor checks, in order, the things
|
||||
that make CHORUS usable this session and prints a green/red report. Intended to back an
|
||||
`chorus.py doctor` subcommand and the start of /chorus-load when something looks wrong.
|
||||
|
||||
Also tracked here (TODO): complete the `load` fallback — chorus.cmd_load reads the
|
||||
heartbeat pointer but never lists _agent/sessions/ when it's missing/stale, though
|
||||
SKILL.md:122 documents that fallback. See patch_load_fallback() below.
|
||||
|
||||
Exit: 0 all green · 1 a check is red.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import chorus # noqa: E402
|
||||
|
||||
MIN_PY = (3, 9)
|
||||
|
||||
|
||||
def run() -> int:
|
||||
reds = 0
|
||||
|
||||
def line(ok: bool, label: str, detail: str = "") -> None:
|
||||
nonlocal reds
|
||||
reds += 0 if ok else 1
|
||||
print(f" [{'OK ' if ok else 'RED'}] {label}" + (f" — {detail}" if detail else ""))
|
||||
|
||||
import chorus_config
|
||||
cfg = chorus_config.load()
|
||||
print(f"chorus doctor — endpoint {cfg['endpoint'] or '(not configured)'}")
|
||||
|
||||
# 1. interpreter
|
||||
line(sys.version_info >= MIN_PY, f"python >= {MIN_PY[0]}.{MIN_PY[1]}",
|
||||
f"running {sys.version_info.major}.{sys.version_info.minor}")
|
||||
|
||||
# 2. machine-local config (owner/endpoint/key). Without a usable endpoint+key
|
||||
# nothing else can run, so report it clearly before touching the network.
|
||||
# A still-placeholder `config init` template counts as not configured.
|
||||
ep_real = bool(cfg["endpoint"]) and "your-obsidian-rest-endpoint" not in cfg["endpoint"]
|
||||
key_real = bool(cfg["key"]) and not cfg["key"].startswith("<")
|
||||
line(ep_real, "endpoint configured",
|
||||
f"[{chorus_config.source('endpoint')}]" if ep_real
|
||||
else (f"placeholder — edit {chorus_config.config_path()}" if cfg["endpoint"]
|
||||
else f"create {chorus_config.config_path()} (run `chorus.py config init`)"))
|
||||
line(key_real, "API key configured",
|
||||
f"[{chorus_config.source('key')}]" if key_real
|
||||
else ("placeholder — not yet filled in" if cfg["key"] else "missing — see `chorus.py config`"))
|
||||
line(bool(cfg["owner"]), "vault owner set",
|
||||
cfg["owner"] if cfg["owner"] else "optional, but recommended for third-person writes")
|
||||
if not chorus_config.is_configured(cfg):
|
||||
print("\ndoctor: not configured — ask the operator for their key file and install it "
|
||||
"(`chorus.py config import <file>`, or `config set --owner … --endpoint … --key …`), "
|
||||
"then re-run.")
|
||||
return 1
|
||||
|
||||
# 3. reachability + auth + marker, in one GET of the bootstrap marker
|
||||
# (marker_get falls back to a pre-fork ECHO marker on adopted vaults)
|
||||
marker_path, status, body = chorus.marker_get()
|
||||
if status == 0:
|
||||
line(False, "vault reachable", body.decode(errors="replace")[:120])
|
||||
print("\ndoctor: endpoint unreachable — is Obsidian + the Local REST API running?")
|
||||
return 1
|
||||
line(status not in (401, 403), "auth accepted", f"HTTP {status}" if status in (401, 403) else "")
|
||||
if status == 404:
|
||||
line(False, "vault bootstrapped", "marker absent — run bootstrap.py")
|
||||
elif status == 200:
|
||||
text = body.decode(errors="replace")
|
||||
ver = next((ln.split(":", 1)[1].strip() for ln in text.splitlines()
|
||||
if ln.startswith("schema_version:")), "?")
|
||||
legacy = " (legacy ECHO marker — run migrate.py to adopt)" if marker_path == chorus.LEGACY_MARKER_PATH else ""
|
||||
line(True, "vault bootstrapped", f"schema_version={ver}{legacy}")
|
||||
else:
|
||||
line(status < 400, "marker fetch", f"HTTP {status}")
|
||||
|
||||
# 4. invariants pointer.
|
||||
print(" [i] invariants — run `/chorus-health` (vault_lint.py) for the full check")
|
||||
|
||||
print(f"\ndoctor: {'all green' if reds == 0 else f'{reds} issue(s) — see RED above'}")
|
||||
return 1 if reds else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(run())
|
||||
@@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_hook_session_start.py — SessionStart hook: self-firing cold-start load.
|
||||
|
||||
The two behaviours the skill cares most about (load at session start, reflect at
|
||||
session end) used to depend entirely on the model remembering to do them — scope
|
||||
drift and the inbox backlog are the residue of that discipline failing. This hook
|
||||
makes loading deterministic: on session start it runs the canonical `chorus.py load`
|
||||
and hands the output to the model as additionalContext.
|
||||
|
||||
Contract (a hook must NEVER break a session):
|
||||
* always exits 0 — any failure degrades to a one-line note or to silence;
|
||||
* fires only on `startup` / `clear` (resume & compact already carry context);
|
||||
* NOT CONFIGURED (exit 78) becomes a short instruction to run the first-run flow,
|
||||
not an error;
|
||||
* output is capped so a huge vault can't blow up the context window.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
MAX_CONTEXT = 16000 # chars of load output forwarded into the session context
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
payload = json.load(sys.stdin)
|
||||
except Exception: # noqa: BLE001
|
||||
payload = {}
|
||||
if payload.get("source", "startup") not in ("startup", "clear"):
|
||||
return 0
|
||||
|
||||
header = ("[chorus-memory] Cold-start memory load (SessionStart hook). Reconcile per "
|
||||
"the chorus-memory skill: check inbox depth and confirm the scope still "
|
||||
"matches this session's request before working.\n\n")
|
||||
buf = io.StringIO()
|
||||
try:
|
||||
import chorus
|
||||
with contextlib.redirect_stdout(buf):
|
||||
rc = chorus.cmd_load()
|
||||
text = buf.getvalue()
|
||||
if rc == 78:
|
||||
context = ("[chorus-memory] CHORUS is NOT CONFIGURED on this machine. Before "
|
||||
"doing memory work, ask the operator for their chorus-memory config "
|
||||
"(owner/endpoint/key) and install it via `chorus.py config import "
|
||||
"<file>` — see the skill's First-run section.")
|
||||
elif rc == 0 and text.strip():
|
||||
if len(text) > MAX_CONTEXT:
|
||||
text = text[:MAX_CONTEXT] + "\n[... load output truncated by the hook ...]"
|
||||
context = header + text
|
||||
else:
|
||||
context = ("[chorus-memory] `chorus.py load` returned nothing usable "
|
||||
f"(exit {rc}) — run /chorus-doctor if memory seems unavailable.")
|
||||
except Exception as exc: # noqa: BLE001 — never break session start
|
||||
context = (f"[chorus-memory] SessionStart load skipped ({exc}). The vault may be "
|
||||
"unreachable; proceed without memory and mention it once if relevant.")
|
||||
|
||||
print(json.dumps({"hookSpecificOutput": {
|
||||
"hookEventName": "SessionStart",
|
||||
"additionalContext": context,
|
||||
}}, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,128 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_hook_stop.py — Stop hook: one-shot session-end reflection nudge.
|
||||
|
||||
Reflection (/chorus-reflect) only captures memory if someone remembers to run it.
|
||||
This hook watches the first Stop of a *substantive* session and, when nothing has
|
||||
been reflected or session-logged yet, blocks once with a reminder so the model
|
||||
runs the reflect flow (which still previews and asks the operator before writing
|
||||
— the safety gate lives in reflect, not here).
|
||||
|
||||
Guards (a hook must NEVER trap a session in a loop or nag):
|
||||
* `stop_hook_active` -> exit immediately (loop guard);
|
||||
* fires at most ONCE per session (marker file under CHORUS_STATE_DIR/hook-state);
|
||||
* skips quiet sessions (< CHORUS_REFLECT_MIN_TURNS user turns, default 5);
|
||||
* skips when the transcript shows reflection / a session log already happened;
|
||||
* skips when CHORUS isn't configured on this machine;
|
||||
* always exits 0.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
MIN_TURNS = int(os.environ.get("CHORUS_REFLECT_MIN_TURNS", "5"))
|
||||
|
||||
REASON = (
|
||||
"[chorus-memory] Session-end reflection has not run yet. If this session produced "
|
||||
"durable facts, decisions, commitments, or artifacts worth remembering: run the "
|
||||
"/chorus-reflect flow (extract -> preview -> apply only with the operator's confirm) "
|
||||
"and write the session log + heartbeat per the chorus-memory skill. If nothing "
|
||||
"durable emerged, simply finish — do not invent memories. "
|
||||
"(This reminder fires once per session.)"
|
||||
)
|
||||
|
||||
|
||||
def state_marker(session_id: str) -> Path:
|
||||
# Mirrors chorus_queue.state_dir() (this hook runs standalone, so the
|
||||
# ECHO->CHORUS env aliasing in chorus_config may not have run): env first
|
||||
# (either name), else ~/.chorus-memory, honoring a pre-fork ~/.echo-memory
|
||||
# when the new dir doesn't exist yet.
|
||||
override = os.environ.get("CHORUS_STATE_DIR") or os.environ.get("ECHO_STATE_DIR")
|
||||
if override:
|
||||
base = Path(override)
|
||||
else:
|
||||
new = Path.home() / ".chorus-memory"
|
||||
legacy = Path.home() / ".echo-memory"
|
||||
base = legacy if (not new.exists() and legacy.exists()) else new
|
||||
return base / "hook-state" / f"{session_id}.reflect-nudged"
|
||||
|
||||
|
||||
def count_user_turns(raw: str) -> int:
|
||||
turns = 0
|
||||
for line in raw.splitlines():
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except Exception: # noqa: BLE001
|
||||
continue
|
||||
if rec.get("type") != "user":
|
||||
continue
|
||||
content = (rec.get("message") or {}).get("content")
|
||||
if isinstance(content, str) and content.strip():
|
||||
turns += 1
|
||||
elif isinstance(content, list) and any(
|
||||
isinstance(b, dict) and b.get("type") == "text" for b in content):
|
||||
turns += 1
|
||||
return turns
|
||||
|
||||
|
||||
def already_reflected(raw: str) -> bool:
|
||||
"""Transcript evidence that reflection or session logging already happened."""
|
||||
return ("/chorus-reflect" in raw
|
||||
or re.search(r'chorus\.py"?\s+reflect\b', raw) is not None
|
||||
or "put _agent/sessions/" in raw
|
||||
or "put _agent/heartbeat/last-session.md" in raw)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
payload = json.load(sys.stdin)
|
||||
except Exception: # noqa: BLE001
|
||||
return 0
|
||||
if payload.get("stop_hook_active"):
|
||||
return 0
|
||||
session_id = str(payload.get("session_id") or "unknown")
|
||||
marker = state_marker(session_id)
|
||||
if marker.exists():
|
||||
return 0
|
||||
|
||||
try:
|
||||
import chorus_config
|
||||
if not chorus_config.is_configured(chorus_config.load()):
|
||||
return 0 # nothing to reflect into — stay silent
|
||||
except Exception: # noqa: BLE001
|
||||
return 0
|
||||
|
||||
raw = ""
|
||||
tp = payload.get("transcript_path")
|
||||
if tp:
|
||||
try:
|
||||
raw = Path(tp).read_text(encoding="utf-8", errors="replace")
|
||||
except Exception: # noqa: BLE001
|
||||
raw = ""
|
||||
|
||||
if already_reflected(raw):
|
||||
try: # done for this session — never fire
|
||||
marker.parent.mkdir(parents=True, exist_ok=True)
|
||||
marker.write_text("reflected\n", encoding="utf-8")
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
return 0
|
||||
if count_user_turns(raw) < MIN_TURNS:
|
||||
return 0 # quiet so far; a later stop may qualify (no marker written)
|
||||
|
||||
try:
|
||||
marker.parent.mkdir(parents=True, exist_ok=True)
|
||||
marker.write_text("nudged\n", encoding="utf-8")
|
||||
except Exception: # noqa: BLE001
|
||||
pass # if the marker can't be written, still nudge; stop_hook_active guards loops
|
||||
print(json.dumps({"decision": "block", "reason": REASON}, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,323 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_index.py — the CHORUS entity index.
|
||||
|
||||
A small machine-readable registry at `_agent/index/entities.json` that turns
|
||||
routing/resolve into an O(1) lookup (no fuzzy search per write) and supplies the
|
||||
name->path map used for cross-linking and recall.
|
||||
|
||||
{
|
||||
"schema": 1,
|
||||
"updated": "YYYY-MM-DD",
|
||||
"entities": {
|
||||
"<slug>": {"path": "...", "kind": "...", "title": "...",
|
||||
"aliases": [...], "last_seen": "YYYY-MM-DD"}
|
||||
}
|
||||
}
|
||||
|
||||
Imported by chorus_ops.py, sweep.py, and vault_lint.py. Network goes through chorus.py.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import chorus # noqa: E402
|
||||
import chorus_quality # noqa: E402 — distinctive-token guards for fuzzy candidate matching
|
||||
|
||||
INDEX_PATH = "_agent/index/entities.json"
|
||||
|
||||
# kind -> destination folder (the canonical home for that entity kind)
|
||||
KIND_FOLDER = {
|
||||
"person": "resources/people",
|
||||
"company": "resources/companies",
|
||||
"concept": "resources/concepts",
|
||||
"reference": "resources/references",
|
||||
"meeting": "resources/meetings",
|
||||
"project": "projects/active", # new projects default to active
|
||||
"area": "areas", # needs a domain segment
|
||||
"semantic": "_agent/memory/semantic",
|
||||
"episodic": "_agent/memory/episodic",
|
||||
"working": "_agent/memory/working",
|
||||
"skill": "_agent/skills/active",
|
||||
"decision": "decisions/by-date",
|
||||
}
|
||||
# kind -> frontmatter `type:` value
|
||||
KIND_TYPE = {
|
||||
"person": "person", "company": "company", "concept": "concept",
|
||||
"reference": "reference", "meeting": "meeting", "project": "project",
|
||||
"area": "area", "semantic": "semantic-memory", "episodic": "episodic-memory",
|
||||
"working": "working-memory", "skill": "skill", "decision": "decision",
|
||||
}
|
||||
DATE_KINDS = {"meeting", "decision"} # filename carries a YYYY-MM-DD prefix
|
||||
AREA_DOMAINS = ("business", "personal", "learning", "systems")
|
||||
|
||||
# kind -> default `status:` stamped by capture when none is given. Every entity note is
|
||||
# born with a status — a missing key is the #1 source of frontmatter drift (18/71 notes
|
||||
# in the field audit). Records of past events default to done; living notes to active.
|
||||
KIND_STATUS = {
|
||||
"person": "active", "company": "active", "concept": "active",
|
||||
"reference": "active", "meeting": "done", "project": "active",
|
||||
"area": "active", "semantic": "active", "episodic": "done",
|
||||
"working": "active", "skill": "active", "decision": "done",
|
||||
}
|
||||
|
||||
# kind -> frontmatter fields that must be present AND non-empty on that kind's notes.
|
||||
# Data-driven so vault_lint's incomplete-frontmatter check and sweep's backfill stay in
|
||||
# lockstep with what capture stamps; add a (kind, field) pair here and all three follow.
|
||||
KIND_REQUIRED_FM = {
|
||||
"person": ("status", "tags"),
|
||||
"company": ("status", "tags"),
|
||||
"concept": ("status", "tags"),
|
||||
"reference": ("status", "tags"),
|
||||
"project": ("status", "tags"),
|
||||
"area": ("status", "tags"),
|
||||
"skill": ("status", "tags"),
|
||||
"meeting": ("tags",),
|
||||
"decision": ("tags",),
|
||||
}
|
||||
|
||||
|
||||
def slugify(text: str) -> str:
|
||||
text = str(text).strip().lower()
|
||||
text = re.sub(r"[^a-z0-9]+", "-", text).strip("-")
|
||||
return text[:40] or "untitled"
|
||||
|
||||
|
||||
def derive_aliases(title: str) -> list[str]:
|
||||
"""Safe, lossless name variants of a title — case/punctuation normalizations that are
|
||||
unambiguously the SAME name (hyphen<->space, lowercased). Deliberately NOT acronyms or
|
||||
token subsets: those collide across entities and would make resolve() match too much.
|
||||
Lets `chorus-memory` and `chorus memory` both resolve to a title written either way."""
|
||||
t = str(title or "").strip()
|
||||
if not t:
|
||||
return []
|
||||
low = t.lower()
|
||||
variants = {low, slugify(t), slugify(t).replace("-", " "),
|
||||
low.replace("-", " "), low.replace(" ", "-")}
|
||||
return sorted(v for v in variants if v)
|
||||
|
||||
|
||||
_WORD = re.compile(r"[a-z0-9]+")
|
||||
|
||||
|
||||
def _sig_tokens(s) -> set[str]:
|
||||
"""Distinctive tokens of a string: long enough and not a common word (reuses the
|
||||
auto-link guards), so 'data'/'api'/'the' can't drive a fuzzy match on their own."""
|
||||
return {t for t in _WORD.findall(str(s or "").lower())
|
||||
if len(t) >= chorus_quality.MIN_NAME_LEN and t not in chorus_quality._COMMON}
|
||||
|
||||
|
||||
def aliases_in_frontmatter(text: str) -> list[str]:
|
||||
"""Read an `aliases:` list (flow `[a, b]` or block `- a`) from a note's YAML frontmatter.
|
||||
Frontmatter is the durable, Obsidian-native home for aliases; sweep folds these into the
|
||||
index so they survive an index rebuild."""
|
||||
if not str(text).startswith("---"):
|
||||
return []
|
||||
end = text.find("\n---", 3)
|
||||
fm = text[:end] if end != -1 else text
|
||||
flow = re.search(r"(?m)^aliases:\s*\[(.*?)\]", fm)
|
||||
if flow:
|
||||
return [a.strip().strip('"').strip("'") for a in flow.group(1).split(",") if a.strip()]
|
||||
block = re.search(r"(?m)^aliases:\s*\n((?:[ \t]*-[ \t]*.+\n?)+)", fm)
|
||||
if block:
|
||||
return [ln.strip().lstrip("-").strip().strip('"').strip("'")
|
||||
for ln in block.group(1).splitlines() if ln.strip()]
|
||||
return []
|
||||
|
||||
|
||||
def derive_path(kind: str, slug: str, date: str | None = None, domain: str = "business") -> str:
|
||||
folder = KIND_FOLDER.get(kind)
|
||||
if folder is None:
|
||||
raise chorus.ChorusError(f"unknown kind '{kind}' (one of: {', '.join(sorted(KIND_FOLDER))})", 2)
|
||||
if kind == "area":
|
||||
if domain not in AREA_DOMAINS:
|
||||
raise chorus.ChorusError(f"area domain must be one of {AREA_DOMAINS}", 2)
|
||||
return f"areas/{domain}/{slug}.md"
|
||||
if kind in DATE_KINDS:
|
||||
return f"{folder}/{date or chorus.today()}-{slug}.md"
|
||||
return f"{folder}/{slug}.md"
|
||||
|
||||
|
||||
_KIND_RULES = [
|
||||
(r"^resources/people/", "person"),
|
||||
(r"^resources/companies/", "company"),
|
||||
(r"^resources/concepts/", "concept"),
|
||||
(r"^resources/references/", "reference"),
|
||||
(r"^resources/meetings/", "meeting"),
|
||||
(r"^projects/(active|incubating|on-hold|archived)/", "project"),
|
||||
(r"^areas/[^/]+/", "area"),
|
||||
(r"^decisions/by-date/", "decision"),
|
||||
(r"^_agent/memory/semantic/", "semantic"),
|
||||
(r"^_agent/memory/episodic/", "episodic"),
|
||||
(r"^_agent/skills/(active|archived)/", "skill"),
|
||||
]
|
||||
|
||||
|
||||
def kind_for_path(path: str) -> str | None:
|
||||
"""The entity kind a vault path belongs to (the inverse of derive_path), or None
|
||||
if the path is not an indexable entity note (journal, sessions, inbox, ...)."""
|
||||
for rx, kind in _KIND_RULES:
|
||||
if re.match(rx, path):
|
||||
return kind
|
||||
return None
|
||||
|
||||
|
||||
def empty_index() -> dict:
|
||||
return {"schema": 1, "updated": chorus.today(), "entities": {}}
|
||||
|
||||
|
||||
def load() -> dict:
|
||||
status, body = chorus.request("GET", chorus.vault_url(INDEX_PATH))
|
||||
if status == 404:
|
||||
return empty_index()
|
||||
chorus.check(status, body, "index load")
|
||||
try:
|
||||
d = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
return empty_index()
|
||||
d.setdefault("entities", {})
|
||||
return d
|
||||
|
||||
|
||||
def save(index: dict) -> None:
|
||||
index["updated"] = chorus.today()
|
||||
body = json.dumps(index, indent=2, ensure_ascii=False).encode("utf-8")
|
||||
status, b = chorus.request("PUT", chorus.vault_url(INDEX_PATH), data=body,
|
||||
headers={"Content-Type": "application/json"})
|
||||
chorus.check(status, b, "index save")
|
||||
|
||||
|
||||
def _norm(s) -> str:
|
||||
return slugify(s)
|
||||
|
||||
|
||||
def resolve(index: dict, mention: str):
|
||||
"""Return (slug, entry) for a mention EXACTLY matched by slug, title, or alias — else
|
||||
(None, None). Exact-only by design: this drives capture's create-vs-update decision, so a
|
||||
loose match here would silently merge distinct entities. Fuzzy lookup is fuzzy_candidates()."""
|
||||
key = _norm(mention)
|
||||
ents = index.get("entities", {})
|
||||
if key in ents:
|
||||
return key, ents[key]
|
||||
plain = str(mention).strip().lower()
|
||||
for slug, e in ents.items():
|
||||
cands = {slug, _norm(e.get("title", "")), str(e.get("title", "")).strip().lower()}
|
||||
cands |= {_norm(a) for a in e.get("aliases", [])}
|
||||
cands |= {str(a).strip().lower() for a in e.get("aliases", [])}
|
||||
if key in cands or plain in cands:
|
||||
return slug, e
|
||||
return None, None
|
||||
|
||||
|
||||
def fuzzy_candidates(index: dict, mention: str, limit: int = 5):
|
||||
"""Advisory near-matches for when resolve() finds no exact hit. Ranks entities sharing
|
||||
>=1 *distinctive* token (>= MIN_NAME_LEN, not a common word) with the mention, so a
|
||||
shortened or expanded name — 'chorus memory' for the project 'chorus' — surfaces the existing
|
||||
note instead of vanishing. Returns [(slug, entry, score)] sorted best-first. NEVER used to
|
||||
auto-merge (that's resolve's job); only to SURFACE 'did you mean' candidates."""
|
||||
mtoks = _sig_tokens(mention)
|
||||
if not mtoks:
|
||||
return []
|
||||
scored = []
|
||||
for slug, e in index.get("entities", {}).items():
|
||||
etoks = set()
|
||||
for name in (slug, e.get("title", ""), *e.get("aliases", [])):
|
||||
etoks |= _sig_tokens(name)
|
||||
overlap = mtoks & etoks
|
||||
if not overlap or not etoks:
|
||||
continue
|
||||
score = round(len(overlap) / min(len(mtoks), len(etoks)), 3)
|
||||
scored.append((score, len(overlap), slug, e))
|
||||
scored.sort(key=lambda x: (-x[0], -x[1], x[2]))
|
||||
return [(slug, e, score) for score, _, slug, e in scored[:limit]]
|
||||
|
||||
|
||||
def _entity_tokens(slug: str, e: dict) -> set[str]:
|
||||
toks: set[str] = set()
|
||||
for name in (slug, e.get("title", ""), *e.get("aliases", [])):
|
||||
toks |= _sig_tokens(name)
|
||||
return toks
|
||||
|
||||
|
||||
def token_df(index: dict) -> Counter:
|
||||
"""Vault-wide token document frequency: token -> how many entities carry it in their
|
||||
slug/title/aliases. A token shared by several entities ("chorus" in an chorus-centric
|
||||
vault) is a weak duplicate signal on its own; df lets the gate discount it."""
|
||||
df: Counter = Counter()
|
||||
for slug, e in index.get("entities", {}).items():
|
||||
for t in _entity_tokens(slug, e):
|
||||
df[t] += 1
|
||||
return df
|
||||
|
||||
|
||||
def gate_candidates(index: dict, mention: str, kind: str | None = None,
|
||||
threshold: float = 0.6):
|
||||
"""The subset of fuzzy_candidates strong enough to BLOCK a create (capture's
|
||||
duplicate gate). Stricter than the advisory warning list, by two rules learned
|
||||
from live use (1.5.1):
|
||||
|
||||
* same-kind only — a cross-kind name collision (a decision titled after its
|
||||
project, a meeting named for a company) is usually intentional naming, not a
|
||||
duplicate. Cross-kind lookalikes still surface as warnings, never as a block.
|
||||
* no single-common-token blocks — score = overlap/min(|tokens|) means an entity
|
||||
whose ONE distinctive token appears in the mention scores 1.0, so any title
|
||||
containing a vault-common word ("chorus") would be blocked by every such entity.
|
||||
A lone shared token only blocks when it is unique to that entity (df == 1).
|
||||
|
||||
Returns [(slug, entry, score)] like fuzzy_candidates."""
|
||||
mtoks = _sig_tokens(mention)
|
||||
if not mtoks:
|
||||
return []
|
||||
df = token_df(index)
|
||||
out = []
|
||||
for slug, e, score in fuzzy_candidates(index, mention):
|
||||
if score < threshold:
|
||||
continue
|
||||
if kind and e.get("kind") and e["kind"] != kind:
|
||||
continue
|
||||
overlap = mtoks & _entity_tokens(slug, e)
|
||||
if len(overlap) == 1 and df[next(iter(overlap))] > 1:
|
||||
continue
|
||||
out.append((slug, e, score))
|
||||
return out
|
||||
|
||||
|
||||
def upsert(index: dict, slug: str, path: str, kind: str, title: str | None = None,
|
||||
aliases=None) -> dict:
|
||||
ents = index.setdefault("entities", {})
|
||||
e = ents.get(slug, {})
|
||||
e["path"] = path
|
||||
e["kind"] = kind
|
||||
e["title"] = title or e.get("title") or slug
|
||||
# Keep any prior + explicit aliases, and ALWAYS fold in the safe title variants so a
|
||||
# hyphen/space/case form of the name resolves even if no one passed --aliases. Drop any
|
||||
# alias that just equals the slug (resolve already checks the slug).
|
||||
merged = set(e.get("aliases", [])) | set(aliases or []) | set(derive_aliases(e["title"]))
|
||||
merged.discard(slug)
|
||||
e["aliases"] = sorted(a for a in merged if a)
|
||||
e["last_seen"] = chorus.today()
|
||||
ents[slug] = e
|
||||
return e
|
||||
|
||||
|
||||
def name_map(index: dict) -> dict:
|
||||
"""Map every resolvable name -> path: slug, normalized title, basename, aliases.
|
||||
Used to resolve `[[wikilinks]]` and entity mentions back to vault paths."""
|
||||
m: dict[str, str] = {}
|
||||
for slug, e in index.get("entities", {}).items():
|
||||
path = e.get("path")
|
||||
if not path:
|
||||
continue
|
||||
base = path.rsplit("/", 1)[-1]
|
||||
base = base[:-3] if base.endswith(".md") else base
|
||||
keys = {slug, _norm(e.get("title", "")), _norm(base)}
|
||||
keys |= {_norm(a) for a in e.get("aliases", [])}
|
||||
for k in keys:
|
||||
if k:
|
||||
m.setdefault(k, path)
|
||||
return m
|
||||
@@ -0,0 +1,116 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_links.py — cross-link primitives.
|
||||
|
||||
Parse and add `## Related` wikilinks, and create **bidirectional** links between
|
||||
notes (read-modify-write, idempotent). Links use path-style targets without the
|
||||
`.md` extension (e.g. `[[resources/people/bob-smith]]`), which are unambiguous
|
||||
and resolve in Obsidian. Network goes through chorus.py.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import chorus # noqa: E402
|
||||
|
||||
WIKILINK = re.compile(r"\[\[([^\]|#]+)(?:[#|][^\]]*)?\]\]")
|
||||
|
||||
|
||||
def first_h1(text: str) -> str:
|
||||
for line in text.splitlines():
|
||||
if line.startswith("# "):
|
||||
return line[2:].strip()
|
||||
return ""
|
||||
|
||||
|
||||
def all_wikilinks(text: str) -> list[str]:
|
||||
return [m.group(1).strip() for m in WIKILINK.finditer(text)]
|
||||
|
||||
|
||||
def related_targets(text: str) -> list[str]:
|
||||
"""Wikilink targets under the `## Related` heading only."""
|
||||
out, cap = [], False
|
||||
for line in text.splitlines():
|
||||
if line.strip().lower() == "## related":
|
||||
cap = True
|
||||
continue
|
||||
if cap and line.startswith("## "):
|
||||
break
|
||||
if cap:
|
||||
out += [m.group(1).strip() for m in WIKILINK.finditer(line)]
|
||||
return out
|
||||
|
||||
|
||||
def source_notes(text: str) -> list[str]:
|
||||
"""Plain relative paths listed in the frontmatter `source_notes:` field."""
|
||||
if not text.startswith("---"):
|
||||
return []
|
||||
end = text.find("\n---", 3)
|
||||
fm = text[:end] if end != -1 else text
|
||||
m = re.search(r"(?m)^source_notes:\s*\[(.*?)\]", fm)
|
||||
if not m:
|
||||
return []
|
||||
return [p.strip().strip('"').strip("'") for p in m.group(1).split(",") if p.strip()]
|
||||
|
||||
|
||||
def link_token(path: str) -> str:
|
||||
"""Path-style wikilink target without the .md extension."""
|
||||
return path[:-3] if path.endswith(".md") else path
|
||||
|
||||
|
||||
def add_related(text: str, target_path: str):
|
||||
"""Return (new_text, changed): ensure `- [[<target>]]` exists under ## Related."""
|
||||
token = link_token(target_path)
|
||||
if token in related_targets(text):
|
||||
return text, False
|
||||
bullet = f"- [[{token}]]"
|
||||
trailing_nl = text.endswith("\n")
|
||||
lines = text.splitlines()
|
||||
idx = next((i for i, l in enumerate(lines) if l.strip().lower() == "## related"), None)
|
||||
if idx is None:
|
||||
sep = "" if trailing_nl else "\n"
|
||||
return text + f"{sep}\n## Related\n{bullet}\n", True
|
||||
# insert after the section's existing content (before the next heading / EOF)
|
||||
j = idx + 1
|
||||
while j < len(lines) and not lines[j].startswith("## "):
|
||||
j += 1
|
||||
k = j
|
||||
while k > idx + 1 and not lines[k - 1].strip():
|
||||
k -= 1
|
||||
lines.insert(k, bullet)
|
||||
return "\n".join(lines) + ("\n" if trailing_nl else ""), True
|
||||
|
||||
|
||||
def get_text(path: str) -> str | None:
|
||||
status, body = chorus.request("GET", chorus.vault_url(path))
|
||||
if status == 404:
|
||||
return None
|
||||
chorus.check(status, body, f"get {path}")
|
||||
return body.decode(errors="replace")
|
||||
|
||||
|
||||
def put_text(path: str, text: str) -> None:
|
||||
status, body = chorus.request("PUT", chorus.vault_url(path), data=text.encode("utf-8"),
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
chorus.check(status, body, f"put {path}")
|
||||
|
||||
|
||||
def add_one(path: str, target_path: str) -> bool:
|
||||
"""Add [[target]] under path's ## Related. Returns True if the file changed."""
|
||||
text = get_text(path)
|
||||
if text is None:
|
||||
return False
|
||||
new, changed = add_related(text, target_path)
|
||||
if changed:
|
||||
put_text(path, new)
|
||||
return changed
|
||||
|
||||
|
||||
def link_bidirectional(a_path: str, b_path: str):
|
||||
"""Add A<->B reciprocal links. Returns (a_changed, b_changed)."""
|
||||
if a_path == b_path:
|
||||
return (False, False)
|
||||
return (add_one(a_path, b_path), add_one(b_path, a_path))
|
||||
@@ -0,0 +1,347 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_ops.py — high-level CHORUS operations layered on the client + index + links.
|
||||
|
||||
resolve — map a mention to its canonical note path (or where to create it)
|
||||
recall — search, then expand one hop along links/source_notes for connected context
|
||||
link — create bidirectional `## Related` links between two notes
|
||||
capture — one-call write: route + canonical frontmatter + index + auto-link + agent-log
|
||||
|
||||
Network goes through chorus.py; routing/aliases through chorus_index; links through chorus_links.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import chorus # noqa: E402
|
||||
import chorus_index as idx_mod # noqa: E402
|
||||
import chorus_links as links # noqa: E402
|
||||
|
||||
# A fuzzy candidate scoring at/above this blocks `capture` from creating what is very
|
||||
# likely a duplicate (require --force to create anyway, or --merge-into to update the
|
||||
# existing entity). Below the gate, candidates are surfaced as a warning only.
|
||||
DUP_GATE = float(os.environ.get("CHORUS_DUP_GATE", "0.6"))
|
||||
|
||||
# Existing-entity capture appends a dated bullet under this per-kind heading.
|
||||
LOG_HEADING = {
|
||||
"project": "Session History", "person": "Log", "company": "Log",
|
||||
"semantic": "Observations", "area": "Log", "concept": "Notes",
|
||||
"reference": "Notes", "meeting": "Notes", "decision": "Notes",
|
||||
"skill": "Notes", "episodic": "Notes", "working": "Notes",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- resolve -----
|
||||
def resolve(mention: str) -> int:
|
||||
index = idx_mod.load()
|
||||
slug, e = idx_mod.resolve(index, mention)
|
||||
if e:
|
||||
print(json.dumps({"match": True, "slug": slug, **e}, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
# No exact match — surface fuzzy candidates so a shortened/expanded name (e.g. "chorus
|
||||
# memory" for the project "chorus") reveals the existing note instead of looking absent.
|
||||
cands = idx_mod.fuzzy_candidates(index, mention)
|
||||
out = {"match": False, "mention": mention, "suggest_slug": idx_mod.slugify(mention)}
|
||||
if cands:
|
||||
out["candidates"] = [{"slug": s, "path": c.get("path"), "title": c.get("title"),
|
||||
"kind": c.get("kind"), "score": sc} for s, c, sc in cands]
|
||||
out["note"] = ("no exact match, but similar entities exist (see candidates) — check "
|
||||
"these before creating a new note; reuse the right path or `chorus.py link`.")
|
||||
else:
|
||||
out["note"] = "no index entry — derive a path with the right --kind and create it"
|
||||
print(json.dumps(out, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- link -----
|
||||
def link(a_path: str, b_path: str, as_json: bool = False) -> int:
|
||||
a_changed, b_changed = links.link_bidirectional(a_path, b_path)
|
||||
if as_json:
|
||||
import chorus_output
|
||||
env = chorus_output.envelope("link", {"a": a_path, "b": b_path,
|
||||
"a_changed": a_changed, "b_changed": b_changed})
|
||||
print(json.dumps(env, ensure_ascii=False))
|
||||
return 0
|
||||
print(f"ok: linked {a_path} <-> {b_path} "
|
||||
f"(added: {'A' if a_changed else '-'}{'B' if b_changed else '-'})")
|
||||
return 0
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- recall -----
|
||||
def recall(query, limit: int = 8, as_json: bool = False) -> int:
|
||||
"""Hybrid lexical (BM25) + graph recall — implemented in chorus_recall. Kept here as
|
||||
the stable public entrypoint (chorus.py and /chorus-recall call chorus_ops.recall)."""
|
||||
import chorus_recall # lazy: only pulls the BM25 modules when recall is actually run
|
||||
return chorus_recall.recall(query, limit=limit, as_json=as_json)
|
||||
|
||||
|
||||
# ----------------------------------------------------------- agent log -------
|
||||
def ensure_daily_log(line: str) -> None:
|
||||
"""Resilient: ensure today's daily note + its `## Agent Log` heading exist, then
|
||||
idempotently append the line. Best-effort — never raises into the caller."""
|
||||
try:
|
||||
date = chorus.today()
|
||||
path = f"journal/daily/{date}.md"
|
||||
status, body = chorus.request("GET", chorus.vault_url(path))
|
||||
if status == 404:
|
||||
ts, tb = chorus.request("GET", chorus.vault_url("journal/templates/daily-note-template.md"))
|
||||
tmpl = tb.decode(errors="replace") if ts == 200 else f"# {date}\n\n## Agent Log\n\n## Related\n"
|
||||
tmpl = tmpl.replace("{{date:YYYY-MM-DD}}", date).replace("{{DATE}}", date)
|
||||
chorus.request("PUT", chorus.vault_url(path), data=tmpl.encode(),
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
text = tmpl
|
||||
else:
|
||||
text = body.decode(errors="replace")
|
||||
if not re.search(r"(?m)^## Agent Log\s*$", text):
|
||||
chorus.request("POST", chorus.vault_url(path), data=b"\n\n## Agent Log\n",
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
status, body = chorus.request("GET", chorus.vault_url(path))
|
||||
if status == 200 and line in body.decode(errors="replace"):
|
||||
return
|
||||
chorus.request("PATCH", chorus.vault_url(path),
|
||||
data=chorus.normalize_patch_body((line + "\n").encode(), "append", "heading"),
|
||||
headers={"Operation": "append", "Target-Type": "heading",
|
||||
"Target": f"{date}::Agent Log", "Content-Type": "text/markdown"})
|
||||
except Exception as exc:
|
||||
print(f"chorus_ops: agent-log skipped ({exc})", file=sys.stderr)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- capture ---
|
||||
def _build_note(fm_type: str, status_v: str, today_s: str, title: str,
|
||||
body_text: str, sources: list[str], aliases: list[str] | None = None,
|
||||
tags: list[str] | None = None) -> str:
|
||||
src = "[" + ", ".join(json.dumps(s) for s in sources) + "]"
|
||||
fm = ["---", f"type: {fm_type}"]
|
||||
if status_v:
|
||||
fm.append(f"status: {status_v}")
|
||||
# Never scaffold an empty `tags: []` — an entity note is born complete (the caller
|
||||
# seeds the kind as a baseline tag), so the incomplete-frontmatter lint stays quiet.
|
||||
tags_v = "[" + ", ".join(tags or []) + "]"
|
||||
fm += [f"created: {today_s}", f"updated: {today_s}", f"tags: {tags_v}"]
|
||||
if aliases:
|
||||
# Obsidian-native, durable home for aliases; sweep folds these back into the index.
|
||||
fm.append("aliases: [" + ", ".join(json.dumps(a) for a in aliases) + "]")
|
||||
fm += ["agent_written: true", f"source_notes: {src}", "---", "", f"# {title}", ""]
|
||||
out = "\n".join(fm)
|
||||
if body_text.strip():
|
||||
out += body_text.strip() + "\n"
|
||||
out += "\n## Related\n"
|
||||
return out
|
||||
|
||||
|
||||
def _dated_block(today_s: str, body_text: str) -> tuple[str, str]:
|
||||
"""Render a capture body as a dated log entry that PRESERVES the whole body:
|
||||
the first line rides on the bullet, every further line is indented under it as
|
||||
a markdown continuation. Returns (bullet, full_block) — the bullet alone is the
|
||||
idempotency key (same first-line-per-day semantics as before)."""
|
||||
lines = [ln.rstrip() for ln in body_text.strip().splitlines()]
|
||||
bullet = f"- {today_s}: {lines[0] if lines else '(update)'}"
|
||||
block = bullet
|
||||
for ln in lines[1:]:
|
||||
block += "\n" + (f" {ln}" if ln else "")
|
||||
return bullet, block
|
||||
|
||||
|
||||
def _append_to_existing(path: str, kind: str, today_s: str, body_text: str) -> None:
|
||||
text = links.get_text(path) or ""
|
||||
h1 = links.first_h1(text) or path.rsplit("/", 1)[-1][:-3]
|
||||
heading = LOG_HEADING.get(kind, "Notes")
|
||||
if not re.search(rf"(?m)^##\s+{re.escape(heading)}\s*$", text):
|
||||
chorus.request("POST", chorus.vault_url(path), data=f"\n\n## {heading}\n".encode(),
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
bullet, block = _dated_block(today_s, body_text)
|
||||
status, body = chorus.request("GET", chorus.vault_url(path))
|
||||
if status == 200 and bullet in body.decode(errors="replace"):
|
||||
return
|
||||
chorus.request("PATCH", chorus.vault_url(path),
|
||||
data=chorus.normalize_patch_body((block + "\n").encode(), "append", "heading"),
|
||||
headers={"Operation": "append", "Target-Type": "heading",
|
||||
"Target": f"{h1}::{heading}", "Content-Type": "text/markdown"})
|
||||
chorus.cmd_fm(path, "updated", json.dumps(today_s))
|
||||
|
||||
|
||||
def capture(kind: str | None, title: str, file_arg: str | None, status_v: str = "",
|
||||
aliases=None, sources=None, tags=None, date: str | None = None,
|
||||
domain: str = "business", inbox: bool = False, no_log: bool = False,
|
||||
as_json: bool = False, dry_run: bool = False, force: bool = False,
|
||||
merge_into: str | None = None) -> int:
|
||||
import contextlib
|
||||
import io
|
||||
import chorus_output
|
||||
import chorus_quality
|
||||
|
||||
real_stdout = sys.stdout
|
||||
|
||||
def quiet():
|
||||
# M4: in --json mode, swallow the helper "ok:" chatter so stdout is clean JSON.
|
||||
return contextlib.redirect_stdout(io.StringIO()) if as_json else contextlib.nullcontext()
|
||||
|
||||
def done(action: str, path: str, links: int = 0, ok: bool = True, dry: bool = False,
|
||||
near=None) -> int:
|
||||
if as_json:
|
||||
act = f"dry-run:{action}" if dry else action
|
||||
data = {"kind": kind, "path": path, "title": title, "links_added": links}
|
||||
if near:
|
||||
data["near_duplicates"] = near
|
||||
env = chorus_output.envelope(act, data, ok=ok)
|
||||
print(json.dumps(env, ensure_ascii=False), file=real_stdout)
|
||||
elif dry:
|
||||
print(f"would {action} {kind or '-'} -> {path}")
|
||||
# non-dry human output is the helper "ok:" lines + the summary printed by the caller.
|
||||
return 0 if ok else 1
|
||||
|
||||
body_text = chorus.read_body(file_arg).decode("utf-8", errors="replace")
|
||||
today_s = chorus.today()
|
||||
aliases = [a.strip() for a in (aliases or []) if a.strip()]
|
||||
sources = [s.strip() for s in (sources or []) if s.strip()]
|
||||
tags = [t.strip() for t in (tags or []) if t.strip()]
|
||||
|
||||
# Unknown home -> defer to the inbox (a single idempotent capture line).
|
||||
if inbox or not kind:
|
||||
if dry_run:
|
||||
return done("inbox", "inbox/captures/inbox.md", dry=True)
|
||||
line = f"- {today_s}: {title}"
|
||||
if body_text.strip():
|
||||
line += f" — {body_text.strip().splitlines()[0]}"
|
||||
with quiet():
|
||||
rc = chorus.cmd_append("inbox/captures/inbox.md", line)
|
||||
return done("inbox", "inbox/captures/inbox.md", ok=rc == 0)
|
||||
|
||||
slug = idx_mod.slugify(title)
|
||||
index = idx_mod.load()
|
||||
match_slug, existing = idx_mod.resolve(index, title)
|
||||
# --merge-into: the operator has already identified the canonical entity (e.g. after
|
||||
# a duplicate-gate stop) — route this capture as an UPDATE to it, whatever the title.
|
||||
if merge_into and not existing:
|
||||
match_slug, existing = idx_mod.resolve(index, merge_into)
|
||||
if not existing:
|
||||
print(f"chorus_ops: --merge-into '{merge_into}' matches no entity in the index "
|
||||
f"(try `resolve` first)", file=sys.stderr)
|
||||
return 2
|
||||
existing_reachable = bool(existing and chorus.request("GET", chorus.vault_url(existing["path"]))[0] == 200)
|
||||
|
||||
# Pre-write duplicate gate: a strong fuzzy candidate means this title is very likely
|
||||
# an existing entity under a different name. STOP before creating the duplicate —
|
||||
# after the fact, the warning arrives too late (the parallel note already exists).
|
||||
# gate_candidates applies the 1.5.1 precision rules (same-kind only; a lone shared
|
||||
# token blocks only when unique to that entity) so vault-common words can't gate.
|
||||
gate_hits = []
|
||||
if not existing_reachable and not force and not dry_run:
|
||||
gate_hits = [{"slug": s, "path": c.get("path"), "title": c.get("title"),
|
||||
"kind": c.get("kind"), "score": sc}
|
||||
for s, c, sc in idx_mod.gate_candidates(index, title, kind=kind,
|
||||
threshold=DUP_GATE)]
|
||||
if gate_hits:
|
||||
if as_json:
|
||||
env = chorus_output.envelope("duplicate-gate", {
|
||||
"kind": kind, "title": title, "candidates": gate_hits,
|
||||
"note": "likely duplicate — re-run with --merge-into <slug> to update the "
|
||||
"existing entity, or --force to create anyway"}, ok=False)
|
||||
print(json.dumps(env, ensure_ascii=False), file=real_stdout)
|
||||
else:
|
||||
print(f"STOP: '{title}' likely already exists — not creating a duplicate.",
|
||||
file=real_stdout)
|
||||
for c in gate_hits:
|
||||
print(f" candidate: {c['slug']} -> {c['path']} (score {c['score']})",
|
||||
file=real_stdout)
|
||||
print(" re-run with --merge-into <slug> to update the existing entity, "
|
||||
"or --force to create anyway.", file=real_stdout)
|
||||
return 76 # distinct exit: duplicate gate (cf. 75 lock, 78 config)
|
||||
|
||||
if dry_run:
|
||||
if existing_reachable:
|
||||
return done("update", existing["path"], dry=True)
|
||||
s2 = chorus_quality.safe_slug(set(index.get("entities", {}).keys()), slug)
|
||||
near = [c.get("path") for _, c, _ in idx_mod.fuzzy_candidates(index, title)][:3]
|
||||
plan = done("create", idx_mod.derive_path(kind, s2, date=date, domain=domain),
|
||||
dry=True, near=near)
|
||||
if (not as_json and not force
|
||||
and idx_mod.gate_candidates(index, title, kind=kind, threshold=DUP_GATE)):
|
||||
# (in --json mode the near_duplicates field carries this; keep stdout clean)
|
||||
print("note: a real run would STOP at the duplicate gate — use --merge-into "
|
||||
"or --force.", file=real_stdout)
|
||||
return plan
|
||||
|
||||
near_dupes: list[str] = []
|
||||
index_title = title # title to record in the index entry
|
||||
index_aliases = list(aliases)
|
||||
with quiet():
|
||||
if existing_reachable:
|
||||
# Reuse the matched entity's CANONICAL slug — never re-slug from the mention, or
|
||||
# two index slugs end up pointing at one note. Keep its title; learn the mention
|
||||
# as an alias when it's a new distinctive form, so this name resolves next time.
|
||||
slug = match_slug or slug
|
||||
path = existing["path"]
|
||||
kind = existing.get("kind", kind)
|
||||
index_title = existing.get("title") or title
|
||||
known = {idx_mod._norm(a) for a in existing.get("aliases", [])} | {match_slug}
|
||||
if idx_mod.slugify(title) not in known and len(title.strip()) >= chorus_quality.MIN_NAME_LEN:
|
||||
index_aliases.append(title)
|
||||
_append_to_existing(path, kind, today_s, body_text)
|
||||
action = "updated"
|
||||
else:
|
||||
if not status_v:
|
||||
# Every entity note is born with a status — a kind-appropriate default
|
||||
# (KIND_STATUS) instead of the old project-only special case. Missing
|
||||
# status was the root cause of the frontmatter-completeness drift.
|
||||
status_v = idx_mod.KIND_STATUS.get(kind, "active")
|
||||
# Surface (don't silently create alongside) an existing entity this resembles —
|
||||
# the duplication trap when a shortened name didn't resolve exactly. (Strong
|
||||
# candidates already stopped at the duplicate gate above; these are the weak
|
||||
# ones, surfaced as a warning.)
|
||||
near_dupes = [c.get("path") for _, c, _ in idx_mod.fuzzy_candidates(index, title)][:3]
|
||||
# M2: don't let a 40-char-truncated slug silently collide with a different
|
||||
# entity already in the index — disambiguate to slug-2, slug-3, ...
|
||||
slug = chorus_quality.safe_slug(set(index.get("entities", {}).keys()), slug)
|
||||
path = idx_mod.derive_path(kind, slug, date=date, domain=domain)
|
||||
note_aliases = sorted((set(index_aliases) | set(idx_mod.derive_aliases(title))) - {slug})
|
||||
note_tags = sorted({kind, *tags}) # baseline `- <kind>` tag; --tags enriches
|
||||
note = _build_note(idx_mod.KIND_TYPE.get(kind, kind), status_v, today_s, title,
|
||||
body_text, sources, note_aliases, tags=note_tags)
|
||||
chorus.cmd_put(path, chorus.temp_file(note.encode()))
|
||||
action = "created"
|
||||
|
||||
# H3: the entity-index write is the race the review flagged — a bare load->save lets
|
||||
# two concurrent captures clobber each other's entry. Route it through a lock-guarded,
|
||||
# fresh-re-read transaction. Returns the fresh index (used by auto-link below).
|
||||
import chorus_concurrency
|
||||
index = chorus_concurrency.atomic_index_update(
|
||||
lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases))
|
||||
|
||||
# auto-link: any other known entity CONFIDENTLY named in the body (M2: is_confident_link
|
||||
# rejects short/common tokens so a 3-char alias or a word like "API" can't link everywhere).
|
||||
linked = 0
|
||||
for s2, e2 in index.get("entities", {}).items():
|
||||
if e2.get("path") in (None, path):
|
||||
continue
|
||||
names = [n for n in [e2.get("title", ""), s2, *e2.get("aliases", [])] if n]
|
||||
if any(chorus_quality.is_confident_link(n, body_text) for n in names):
|
||||
ca, cb = links.link_bidirectional(path, e2["path"])
|
||||
linked += int(ca or cb)
|
||||
|
||||
# Keep the BM25 recall index current for this note (best-effort; lock-guarded inside
|
||||
# update_note, so it can't clobber a concurrent writer either).
|
||||
try:
|
||||
import chorus_recall
|
||||
chorus_recall.update_note(path, links.get_text(path) or "")
|
||||
except Exception as exc: # never let recall-index upkeep fail a capture
|
||||
print(f"chorus_ops: recall-index update skipped ({exc})", file=sys.stderr)
|
||||
|
||||
if not no_log:
|
||||
ensure_daily_log(f"- {today_s}: {action} {kind} [[{links.link_token(path)}]]")
|
||||
|
||||
if as_json:
|
||||
done(action, path, links=linked, near=near_dupes)
|
||||
else:
|
||||
print(f"ok: {action} {kind} -> {path}" + (f"; auto-linked {linked}" if linked else ""))
|
||||
if near_dupes:
|
||||
kind_word = "entity" if len(near_dupes) == 1 else "entities"
|
||||
print(f"WARNING: similar existing {kind_word} ({', '.join(near_dupes)}) — if this is "
|
||||
f"the same thing, merge or `chorus.py link` instead of keeping a duplicate.",
|
||||
file=real_stdout)
|
||||
return 0
|
||||
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_output.py — M4: machine-readable output + dry-run plumbing. [v1.0 SCAFFOLD — implemented]
|
||||
|
||||
The high-level ops (capture/recall/resolve/scope) print prose, forcing the agent to
|
||||
re-parse free text to act on a result. This provides a single JSON envelope and a
|
||||
small dry-run helper so every op can emit a structured result and preview writes.
|
||||
|
||||
INTEGRATION (merge step): thread a global `--json` / `--dry-run` flag through chorus.py's
|
||||
parser and have each cmd_* return its data dict; emit() it. In --dry-run, the write
|
||||
verbs print the envelope with action="dry-run" and perform no network mutation.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
|
||||
|
||||
def envelope(action: str, data: dict | None = None, ok: bool = True,
|
||||
error: str | None = None) -> dict:
|
||||
"""Canonical result shape for every high-level op."""
|
||||
env = {"ok": ok, "action": action}
|
||||
if data:
|
||||
env.update(data)
|
||||
if error:
|
||||
env["error"] = error
|
||||
return env
|
||||
|
||||
|
||||
def emit(env: dict, as_json: bool) -> None:
|
||||
"""Print either the JSON envelope (machine mode) or a one-line human summary."""
|
||||
if as_json:
|
||||
print(json.dumps(env, ensure_ascii=False))
|
||||
return
|
||||
if not env.get("ok"):
|
||||
print(f"error: {env.get('error', 'failed')}", file=sys.stderr)
|
||||
return
|
||||
bits = [env.get("action", "ok")]
|
||||
if env.get("path"):
|
||||
bits.append(str(env["path"]))
|
||||
if env.get("links_added"):
|
||||
bits.append(f"+{env['links_added']} links")
|
||||
print("ok: " + " ".join(bits))
|
||||
|
||||
|
||||
def dry_run_envelope(action: str, data: dict | None = None) -> dict:
|
||||
"""Result for a previewed-but-not-executed write."""
|
||||
return envelope(f"dry-run:{action}", data, ok=True)
|
||||
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_quality.py — M2: keep the graph (which recall depends on) clean.
|
||||
[v1.0 SCAFFOLD — helpers implemented; integration into chorus_links/chorus_index is the merge step]
|
||||
|
||||
Two graph-polluting bugs today:
|
||||
1. Auto-link fires on any indexed entity whose name/alias is >=3 chars and word-matches
|
||||
the body (chorus_ops.py:236). A concept titled "API" or a 2-3 char alias links to every
|
||||
note that happens to mention the word -> noisy, wrong edges.
|
||||
2. slugify truncates to 40 chars (chorus_index.py:58) with NO collision check, so two
|
||||
distinct long titles can resolve to the same slug -> same path -> silent merge.
|
||||
|
||||
This module supplies the two guards. Integration:
|
||||
* chorus_ops auto-link loop -> gate each candidate on is_confident_link(name, body)
|
||||
* chorus_index.derive_path -> route the slug through safe_slug(existing_slugs, slug)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
# Tokens too short or too common to be a confident entity reference on their own.
|
||||
MIN_NAME_LEN = 4
|
||||
_COMMON = frozenset(
|
||||
"api app data note task user team work code test main repo file plan goal item "
|
||||
"the and for with from".split()
|
||||
)
|
||||
|
||||
|
||||
def is_confident_link(name: str, body: str) -> bool:
|
||||
"""True only if `name` is a strong enough signal to auto-link on its appearance in
|
||||
`body`. Rejects short names, common words, and non-matches. Multi-word names are
|
||||
trusted at lower length (a two-word phrase is rarely incidental)."""
|
||||
n = (name or "").strip()
|
||||
if not n:
|
||||
return False
|
||||
multiword = len(n.split()) > 1
|
||||
if not multiword:
|
||||
if len(n) < MIN_NAME_LEN or n.lower() in _COMMON:
|
||||
return False
|
||||
return re.search(rf"\b{re.escape(n)}\b", body or "", re.IGNORECASE) is not None
|
||||
|
||||
|
||||
def safe_slug(existing_slugs, slug: str, max_len: int = 40) -> str:
|
||||
"""Return `slug` if free, else a disambiguated `slug-2`, `slug-3`, ... that still
|
||||
fits `max_len` and does not collide with anything in `existing_slugs`."""
|
||||
existing = set(existing_slugs or ())
|
||||
if slug not in existing:
|
||||
return slug
|
||||
n = 2
|
||||
while True:
|
||||
suffix = f"-{n}"
|
||||
base = slug[: max_len - len(suffix)] if len(slug) + len(suffix) > max_len else slug
|
||||
cand = f"{base}{suffix}"
|
||||
if cand not in existing:
|
||||
return cand
|
||||
n += 1
|
||||
@@ -0,0 +1,200 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_queue.py — H2: offline write-ahead queue + read cache. [v1.0 — wired]
|
||||
|
||||
Today vault-unreachable => "proceed without memory" (SKILL.md), so a 502 (Obsidian not
|
||||
running — the most common real failure) loses everything said that session. This module
|
||||
makes explicit writes durable across an outage and lets cold-start `load` degrade to
|
||||
last-known context instead of nothing.
|
||||
|
||||
DESIGN:
|
||||
* Outbox : append-only NDJSON of intended writes. Replayed in order on the next
|
||||
reachable session. Replay is idempotent by construction — PUT/DELETE/PATCH-
|
||||
replace overwrite, and POST / PATCH-append are content-deduped (skip if the
|
||||
line/block is already present), the same strategy chorus.py append uses. Each
|
||||
record carries an idem_key so the SAME write is never queued twice.
|
||||
* Cache : last-known-good GET bodies, so `load` has a fallback. Written by cmd_load.
|
||||
* Location: a LOCAL state dir — it CANNOT live in the vault, because the vault is the
|
||||
thing that's down. Default ~/.chorus-memory/, override CHORUS_STATE_DIR.
|
||||
|
||||
Integration seam: `safe_request()` — the write verbs (cmd_put/post/append/patch/delete)
|
||||
call it instead of chorus.request; on an outage it enqueues and returns a synthesized
|
||||
(202, b"queued ...") so the caller reports "queued, will sync" instead of failing.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import urllib.parse
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import chorus # noqa: E402
|
||||
|
||||
MUTATING = {"PUT", "POST", "PATCH", "DELETE"}
|
||||
|
||||
|
||||
def state_dir() -> Path:
|
||||
# Env first (chorus_config aliases a legacy ECHO_STATE_DIR at import); the
|
||||
# default is ~/.chorus-memory, but an existing pre-fork ~/.echo-memory is
|
||||
# honored when the new dir doesn't exist yet — so queued offline writes
|
||||
# from an ECHO install aren't stranded by the rename.
|
||||
override = os.environ.get("CHORUS_STATE_DIR") or os.environ.get("ECHO_STATE_DIR")
|
||||
if override:
|
||||
return Path(override)
|
||||
new = Path.home() / ".chorus-memory"
|
||||
legacy = Path.home() / ".echo-memory"
|
||||
return legacy if (not new.exists() and legacy.exists()) else new
|
||||
|
||||
|
||||
def outbox_path() -> Path:
|
||||
return state_dir() / "outbox.ndjson"
|
||||
|
||||
|
||||
def cache_dir() -> Path:
|
||||
return state_dir() / "cache"
|
||||
|
||||
|
||||
def _ensure_dirs() -> None:
|
||||
state_dir().mkdir(parents=True, exist_ok=True)
|
||||
cache_dir().mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def _key(method: str, url: str, data: bytes | None) -> str:
|
||||
h = hashlib.sha1()
|
||||
h.update(method.encode())
|
||||
h.update(b"\0")
|
||||
h.update(url.encode())
|
||||
h.update(b"\0")
|
||||
h.update(data or b"")
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- outbox
|
||||
def enqueue(method: str, url: str, data: bytes | None, headers: dict, idem_key: str | None = None) -> None:
|
||||
"""Append one intended write to the outbox (NDJSON; base64 body for binary safety).
|
||||
Dedups: if a record with the same idem_key is already pending, do nothing."""
|
||||
idem_key = idem_key or _key(method, url, data)
|
||||
if any(rec.get("idem_key") == idem_key for rec in pending()):
|
||||
return
|
||||
_ensure_dirs()
|
||||
rec = {
|
||||
"method": method.upper(), "url": url,
|
||||
"body_b64": base64.b64encode(data).decode() if data else None,
|
||||
"headers": headers or {}, "idem_key": idem_key,
|
||||
# No wall-clock: replay order is file order, not a timestamp (keeps this testable).
|
||||
}
|
||||
with outbox_path().open("a", encoding="utf-8") as fh:
|
||||
fh.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
def pending() -> list[dict]:
|
||||
p = outbox_path()
|
||||
if not p.exists():
|
||||
return []
|
||||
return [json.loads(ln) for ln in p.read_text(encoding="utf-8").splitlines() if ln.strip()]
|
||||
|
||||
|
||||
def _rewrite(records: list[dict]) -> None:
|
||||
p = outbox_path()
|
||||
if not records:
|
||||
if p.exists():
|
||||
p.unlink()
|
||||
return
|
||||
_ensure_dirs()
|
||||
tmp = p.with_suffix(".ndjson.tmp")
|
||||
tmp.write_text("".join(json.dumps(r, ensure_ascii=False) + "\n" for r in records), encoding="utf-8")
|
||||
os.replace(tmp, p) # atomic swap so a crash mid-flush can't corrupt the queue
|
||||
|
||||
|
||||
def _rebase(url: str) -> str:
|
||||
"""Re-point a queued URL at the CURRENT chorus.BASE. Writes are queued with the base
|
||||
that was active when they failed; on replay the vault may be back at a different base
|
||||
(e.g. an endpoint migration), so we always reconstruct the origin from chorus.BASE."""
|
||||
parts = urllib.parse.urlsplit(url)
|
||||
return chorus.BASE + parts.path + (("?" + parts.query) if parts.query else "")
|
||||
|
||||
|
||||
def _replay(rec: dict) -> str:
|
||||
"""Re-issue one queued write idempotently. Returns 'ok' (landed/already-present),
|
||||
'drop' (permanent 4xx — can never succeed), or 'retry' (still offline / 5xx)."""
|
||||
method = rec["method"]
|
||||
url = _rebase(rec["url"])
|
||||
data = base64.b64decode(rec["body_b64"]) if rec.get("body_b64") else None
|
||||
headers = rec.get("headers") or {}
|
||||
|
||||
# Append-style writes: skip if the content is already present (idempotency on replay).
|
||||
is_append = method == "POST" or (method == "PATCH" and headers.get("Operation") == "append")
|
||||
if is_append and data:
|
||||
st, cur = chorus.request("GET", url)
|
||||
if st == 0:
|
||||
return "retry" # still offline
|
||||
if st == 200 and data.decode("utf-8", "replace").strip("\n") in cur.decode("utf-8", "replace"):
|
||||
return "ok" # already landed (or a duplicate we must not re-add)
|
||||
|
||||
st, body = chorus.request(method, url, data=data, headers=headers)
|
||||
if st == 0 or 500 <= st < 600:
|
||||
return "retry"
|
||||
if 400 <= st < 500:
|
||||
print(f"chorus_queue: dropping a queued {method} {url} — permanent HTTP {st} on replay",
|
||||
file=sys.stderr)
|
||||
return "drop"
|
||||
return "ok"
|
||||
|
||||
|
||||
def flush() -> int:
|
||||
"""Replay queued writes in order; stop at the first still-unreachable one (keeping it
|
||||
and the rest). Returns the number actually replayed. Safe to call when empty."""
|
||||
items = pending()
|
||||
if not items:
|
||||
return 0
|
||||
replayed = 0
|
||||
remaining: list[dict] = []
|
||||
for i, rec in enumerate(items):
|
||||
result = _replay(rec)
|
||||
if result == "ok":
|
||||
replayed += 1
|
||||
elif result == "drop":
|
||||
continue # warned in _replay; remove from queue
|
||||
else: # retry -> still offline; keep this and everything after, in order
|
||||
remaining = items[i:]
|
||||
break
|
||||
_rewrite(remaining)
|
||||
return replayed
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- cache
|
||||
def cache_key(path: str) -> Path:
|
||||
return cache_dir() / (path.replace("/", "__") + ".cache")
|
||||
|
||||
|
||||
def cache_put(path: str, body: bytes) -> None:
|
||||
_ensure_dirs()
|
||||
cache_key(path).write_bytes(body)
|
||||
|
||||
|
||||
def cache_get(path: str) -> bytes | None:
|
||||
p = cache_key(path)
|
||||
return p.read_bytes() if p.exists() else None
|
||||
|
||||
|
||||
# ----------------------------------------------------------- integration seam
|
||||
def safe_request(method: str, url: str, data=None, headers=None, idem_key: str | None = None):
|
||||
"""Network-first wrapper around chorus.request with offline queueing for writes.
|
||||
|
||||
- success or a client error (4xx): return (status, body) as-is — 4xx is a bug, fail loud.
|
||||
- transport failure (status 0) or 5xx (after chorus.request's own retry) on a MUTATING
|
||||
verb: enqueue the write and return (202, b"queued (offline)") so the caller reports it
|
||||
as durably queued rather than failed.
|
||||
- otherwise (e.g. a failing GET): return (status, body); the read-cache fallback lives
|
||||
in cmd_load, which knows which reads are worth serving stale.
|
||||
"""
|
||||
method = method.upper()
|
||||
status, body = chorus.request(method, url, data=data, headers=headers)
|
||||
offline = status == 0 or 500 <= status < 600
|
||||
if offline and method in MUTATING:
|
||||
enqueue(method, url, data, headers or {}, idem_key)
|
||||
return 202, b"queued (offline)"
|
||||
return status, body
|
||||
@@ -0,0 +1,494 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_recall.py — H1: hybrid lexical (BM25) + graph recall. [v1.0 — wired]
|
||||
|
||||
Replaces the keyword-only recall (search/simple + fixed one-hop) with a ranked, local
|
||||
retriever that finds a note even when the query is phrased differently than the note
|
||||
was stored, then expands along the graph with a per-hop decay so the *connected*
|
||||
context is surfaced in relevance order.
|
||||
|
||||
DESIGN (bounded by the project's pure-stdlib, no-deps constraint):
|
||||
* Lexical layer : BM25 over note bodies. No embeddings / external API — BM25 is a
|
||||
strong, dependency-free baseline that handles vocabulary mismatch
|
||||
far better than substring search.
|
||||
* Graph layer : from the top lexical hits, walk body wikilinks + source_notes up
|
||||
to MAX_HOPS, scoring each node `parent * GRAPH_DECAY**hop` (max over
|
||||
paths) so near, strongly-cued neighbours rank above distant ones.
|
||||
* Persistence : the BM25 postings + doc stats are a DERIVED artifact, so they live
|
||||
in the vault's machine index dir (`_agent/index/recall-index.json`),
|
||||
maintained on `capture` (update_note) and fully rebuilt by `sweep.py`
|
||||
— exactly how `entities.json` is maintained. Rebuildable => portable.
|
||||
* Resilience : if the index can't be built (vault unreachable), recall falls back
|
||||
to the server's /search/simple so it degrades instead of dying.
|
||||
|
||||
Corpus = the indexable ENTITY notes (idx_mod.kind_for_path is not None) — people,
|
||||
companies, concepts, references, meetings, projects, areas, decisions, semantic/
|
||||
episodic memory, skills — PLUS the time-series notes where conversation-derived
|
||||
context actually lives: session logs (`_agent/sessions/`) and journal notes
|
||||
(`journal/daily|weekly|monthly|quarterly|annual/`). Time-series docs carry a
|
||||
per-kind DOWN-WEIGHT so entity notes still rank first for the same lexical match.
|
||||
|
||||
RANKING (v1.5): final score = BM25 * kind_weight * freshness * status_factor —
|
||||
* kind_weight : 1.0 entity · 0.5 session · 0.4 journal (KIND_WEIGHT)
|
||||
* freshness : half-life decay on `updated:` (or the filename date), floored at
|
||||
FRESH_FLOOR so old notes are demoted, never buried
|
||||
* status : `active` boosted, `archived` demoted (STATUS_FACTOR)
|
||||
so a stale archived project no longer outranks the live one on raw term frequency.
|
||||
Doc metadata rides in the index (schema 2); a schema-1 index is discarded and
|
||||
rebuilt on the next recall/sweep.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as _dt
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.parse
|
||||
from collections import Counter, deque
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import chorus # noqa: E402
|
||||
import chorus_index as idx_mod # noqa: E402
|
||||
import chorus_links as links # noqa: E402
|
||||
|
||||
RECALL_INDEX_PATH = "_agent/index/recall-index.json"
|
||||
INDEX_SCHEMA = 2 # bumped: schema 2 adds per-doc meta (weight/updated/status)
|
||||
|
||||
# Tuning knobs. Defaults are standard BM25 (k1/b) + a 0.6 hop decay over <=2 hops.
|
||||
K1 = 1.5
|
||||
B = 0.75
|
||||
MAX_HOPS = 2
|
||||
GRAPH_DECAY = 0.6
|
||||
MIN_SCORE = 0.0 # relevance floor; raise to trim weak lexical hits
|
||||
|
||||
# --- rank-fusion priors -------------------------------------------------------
|
||||
# Time-series notes are in the corpus but down-weighted: they are where "what did we
|
||||
# decide three weeks ago" lives, yet the entity note should win a tie on the same terms.
|
||||
_TIME_SERIES = [
|
||||
(re.compile(r"^_agent/sessions/"), "session"),
|
||||
(re.compile(r"^journal/daily/"), "daily"),
|
||||
(re.compile(r"^journal/(weekly|monthly|quarterly|annual)/"), "rollup"),
|
||||
]
|
||||
KIND_WEIGHT = {"session": 0.5, "daily": 0.4, "rollup": 0.4}
|
||||
STATUS_FACTOR = {"active": 1.1, "on-hold": 0.9, "archived": 0.75}
|
||||
FRESH_HALF_LIFE = float(os.environ.get("CHORUS_FRESH_HALF_LIFE", "90")) # days
|
||||
FRESH_FLOOR = 0.6 # freshness multiplier never drops below this — demote, don't bury
|
||||
|
||||
_FM_UPDATED = re.compile(r"(?m)^updated:\s*[\"']?(\d{4}-\d{2}-\d{2})")
|
||||
_FM_STATUS = re.compile(r"(?m)^status:\s*[\"']?([A-Za-z-]+)")
|
||||
_PATH_DATE = re.compile(r"(\d{4}-\d{2}-\d{2})")
|
||||
|
||||
|
||||
def _series_kind(path: str) -> str | None:
|
||||
for rx, kind in _TIME_SERIES:
|
||||
if rx.match(path):
|
||||
return kind
|
||||
return None
|
||||
|
||||
|
||||
def doc_meta(path: str, raw_text: str) -> dict:
|
||||
"""Per-doc ranking priors, extracted once at index time: kind weight, last-updated
|
||||
date (frontmatter `updated:`, else the date in the filename), and `status:`."""
|
||||
kind = _series_kind(path)
|
||||
w = KIND_WEIGHT.get(kind, 1.0) if kind else 1.0
|
||||
head = raw_text[:2000]
|
||||
mu = _FM_UPDATED.search(head)
|
||||
updated = mu.group(1) if mu else None
|
||||
if updated is None:
|
||||
mp = _PATH_DATE.search(path.rsplit("/", 1)[-1])
|
||||
updated = mp.group(1) if mp else None
|
||||
ms = _FM_STATUS.search(head)
|
||||
meta = {"w": w}
|
||||
if updated:
|
||||
meta["u"] = updated
|
||||
if ms:
|
||||
meta["s"] = ms.group(1)
|
||||
return meta
|
||||
|
||||
|
||||
def freshness(updated: str | None, today_s: str) -> float:
|
||||
"""Half-life decay on document age, floored. Unknown age is neutral (1.0)."""
|
||||
if not updated:
|
||||
return 1.0
|
||||
try:
|
||||
age = (_dt.date.fromisoformat(today_s) - _dt.date.fromisoformat(updated)).days
|
||||
except ValueError:
|
||||
return 1.0
|
||||
if age <= 0:
|
||||
return 1.0
|
||||
return FRESH_FLOOR + (1.0 - FRESH_FLOOR) * (0.5 ** (age / FRESH_HALF_LIFE))
|
||||
|
||||
_WORD = re.compile(r"[a-z0-9]+")
|
||||
# Minimal English stoplist — keeps BM25 idf from being dominated by glue words.
|
||||
_STOP = frozenset(
|
||||
"the a an and or of to in is are was were be been for on at by with as it this that "
|
||||
"from into over under not no yes do does did has have had will would can could".split()
|
||||
)
|
||||
_TEMPLATE_RE = re.compile(r"(^|/)(templates/|.*-template\.md$)")
|
||||
_SKIP_BASENAMES = {"README.md"}
|
||||
|
||||
|
||||
def tokenize(text: str) -> list[str]:
|
||||
"""Lowercase word tokens, stopworded. Frontmatter should be stripped by the caller."""
|
||||
return [t for t in _WORD.findall(text.lower()) if t not in _STOP and len(t) > 1]
|
||||
|
||||
|
||||
def strip_frontmatter(text: str) -> str:
|
||||
if not text.startswith("---"):
|
||||
return text
|
||||
end = text.find("\n---", 3)
|
||||
return text[end + 4:] if end != -1 else text
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- BM25 core
|
||||
class Bm25Index:
|
||||
"""A compact, serializable BM25 index with per-doc ranking meta.
|
||||
add()/remove() are idempotent per path. add() takes the RAW note text — it strips
|
||||
frontmatter itself and extracts the doc meta (weight/updated/status) in one pass."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.df: Counter[str] = Counter() # term -> #docs containing it
|
||||
self.postings: dict[str, dict[str, int]] = {} # term -> {doc_path: tf}
|
||||
self.length: dict[str, int] = {} # doc_path -> token count
|
||||
self.meta: dict[str, dict] = {} # doc_path -> {w, u, s} priors
|
||||
self.n_docs = 0
|
||||
self.avg_len = 0.0
|
||||
|
||||
def _recompute(self) -> None:
|
||||
self.n_docs = len(self.length)
|
||||
self.avg_len = (sum(self.length.values()) / self.n_docs) if self.n_docs else 0.0
|
||||
|
||||
def add(self, path: str, raw_text: str) -> None:
|
||||
if path in self.length: # re-index in place (idempotent)
|
||||
self.remove(path)
|
||||
toks = tokenize(strip_frontmatter(raw_text))
|
||||
self.length[path] = len(toks)
|
||||
self.meta[path] = doc_meta(path, raw_text)
|
||||
for term, tf in Counter(toks).items():
|
||||
self.postings.setdefault(term, {})[path] = tf
|
||||
self.df[term] += 1
|
||||
self._recompute()
|
||||
|
||||
def remove(self, path: str) -> None:
|
||||
if path not in self.length:
|
||||
return
|
||||
for term in list(self.postings.keys()):
|
||||
if path in self.postings[term]:
|
||||
del self.postings[term][path]
|
||||
self.df[term] -= 1
|
||||
if not self.postings[term]:
|
||||
del self.postings[term]
|
||||
del self.df[term]
|
||||
del self.length[path]
|
||||
self.meta.pop(path, None)
|
||||
self._recompute()
|
||||
|
||||
def _doc_factor(self, path: str, today_s: str) -> float:
|
||||
"""The rank-fusion prior: kind weight x freshness x status factor."""
|
||||
m = self.meta.get(path) or {}
|
||||
return (float(m.get("w", 1.0))
|
||||
* freshness(m.get("u"), today_s)
|
||||
* STATUS_FACTOR.get(m.get("s", ""), 1.0))
|
||||
|
||||
def score(self, query: str, limit: int = 10,
|
||||
today_s: str | None = None) -> list[tuple[str, float]]:
|
||||
today_s = today_s or chorus.today()
|
||||
scores: Counter[str] = Counter()
|
||||
for term in set(tokenize(query)):
|
||||
posting = self.postings.get(term)
|
||||
if not posting:
|
||||
continue
|
||||
idf = math.log(1 + (self.n_docs - self.df[term] + 0.5) / (self.df[term] + 0.5))
|
||||
for path, tf in posting.items():
|
||||
dl = self.length.get(path, 0)
|
||||
denom = tf + K1 * (1 - B + B * (dl / self.avg_len if self.avg_len else 1))
|
||||
scores[path] += idf * (tf * (K1 + 1) / denom if denom else 0)
|
||||
fused = {p: s * self._doc_factor(p, today_s) for p, s in scores.items()}
|
||||
ranked = sorted(fused.items(), key=lambda kv: -kv[1])[:limit]
|
||||
return [(p, s) for p, s in ranked if s > MIN_SCORE]
|
||||
|
||||
# ---- serialization (compact; rebuildable, so the format can change freely) ----
|
||||
def to_json(self) -> dict:
|
||||
return {"schema": INDEX_SCHEMA, "n_docs": self.n_docs, "avg_len": self.avg_len,
|
||||
"length": self.length, "postings": self.postings, "meta": self.meta}
|
||||
|
||||
@classmethod
|
||||
def from_json(cls, d: dict) -> "Bm25Index":
|
||||
if d.get("schema") != INDEX_SCHEMA:
|
||||
return cls() # older/newer format -> empty; recall's cold path rebuilds
|
||||
ix = cls()
|
||||
ix.length = d.get("length", {})
|
||||
ix.postings = d.get("postings", {})
|
||||
ix.meta = d.get("meta", {})
|
||||
ix.n_docs = d.get("n_docs", len(ix.length))
|
||||
ix.avg_len = d.get("avg_len", 0.0)
|
||||
ix.df = Counter({term: len(posting) for term, posting in ix.postings.items()})
|
||||
return ix
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- vault I/O
|
||||
def _get(path: str) -> str | None:
|
||||
status, body = chorus.request("GET", chorus.vault_url(path))
|
||||
if status == 404:
|
||||
return None
|
||||
chorus.check(status, body, f"recall get {path}")
|
||||
return body.decode(errors="replace")
|
||||
|
||||
|
||||
def _list_dir(path: str):
|
||||
p = "" if path in ("", "/") else (path if path.endswith("/") else path + "/")
|
||||
body = _get(p)
|
||||
if body is None:
|
||||
return [], []
|
||||
try:
|
||||
j = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
return [], []
|
||||
entries = list(j.get("files", [])) + list(j.get("folders", []))
|
||||
files = [e for e in entries if not e.endswith("/")]
|
||||
folders = [e[:-1] for e in entries if e.endswith("/")]
|
||||
return files, folders
|
||||
|
||||
|
||||
def _walk(prefix: str = ""):
|
||||
files, folders = _list_dir(prefix)
|
||||
for f in files:
|
||||
yield prefix + f
|
||||
for d in folders:
|
||||
yield from _walk(f"{prefix}{d}/")
|
||||
|
||||
|
||||
def _indexable(path: str) -> bool:
|
||||
base = path.rsplit("/", 1)[-1]
|
||||
if (not path.endswith(".md") or base in _SKIP_BASENAMES
|
||||
or _TEMPLATE_RE.search(path)):
|
||||
return False
|
||||
# Entity notes (the durable graph) + time-series notes (sessions, journal) — the
|
||||
# latter join the corpus down-weighted so past decisions/discussions are findable.
|
||||
return idx_mod.kind_for_path(path) is not None or _series_kind(path) is not None
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- persistence
|
||||
def load_index() -> Bm25Index:
|
||||
status, body = chorus.request("GET", chorus.vault_url(RECALL_INDEX_PATH))
|
||||
if status == 404:
|
||||
return Bm25Index()
|
||||
chorus.check(status, body, "recall-index load")
|
||||
try:
|
||||
return Bm25Index.from_json(json.loads(body))
|
||||
except (json.JSONDecodeError, KeyError, TypeError):
|
||||
return Bm25Index()
|
||||
|
||||
|
||||
def save_index(ix: Bm25Index) -> None:
|
||||
body = json.dumps(ix.to_json(), ensure_ascii=False).encode("utf-8")
|
||||
status, b = chorus.request("PUT", chorus.vault_url(RECALL_INDEX_PATH), data=body,
|
||||
headers={"Content-Type": "application/json"})
|
||||
chorus.check(status, b, "recall-index save")
|
||||
|
||||
|
||||
def rebuild(prefetched: dict | None = None) -> Bm25Index:
|
||||
"""Full rebuild from every indexable entity note. Called by sweep.py and as the
|
||||
cold-cache path inside recall(). Saves and returns the index.
|
||||
|
||||
`prefetched` (a {path: text} map, e.g. from chorus.read_many) lets sweep.py reuse the
|
||||
content it already pulled instead of walking and re-fetching the whole vault again."""
|
||||
ix = Bm25Index()
|
||||
if prefetched is not None:
|
||||
items = ((p, t) for p, t in prefetched.items() if _indexable(p))
|
||||
else:
|
||||
items = ((p, _get(p)) for p in _walk() if _indexable(p))
|
||||
for path, text in items:
|
||||
if text is not None:
|
||||
ix.add(path, text) # add() strips frontmatter + extracts doc meta itself
|
||||
save_index(ix)
|
||||
return ix
|
||||
|
||||
|
||||
def update_note(path: str, text: str) -> None:
|
||||
"""Best-effort incremental upkeep for a single note (mirrors how entities.json is
|
||||
maintained on capture). Never raises into the caller. The load->save is wrapped in the
|
||||
advisory lock with a fresh re-read (H3), so concurrent captures can't clobber the
|
||||
recall index either."""
|
||||
try:
|
||||
if not _indexable(path):
|
||||
return # only entity notes are part of the recall corpus
|
||||
import chorus_concurrency
|
||||
with chorus_concurrency.vault_lock():
|
||||
ix = load_index() # fresh read inside the lock
|
||||
ix.add(path, text) # raw text: add() strips + extracts meta
|
||||
save_index(ix)
|
||||
except Exception as exc: # noqa: BLE001 — upkeep must never fail a capture
|
||||
print(f"chorus_recall: index update skipped ({exc})", file=sys.stderr)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- graph fusion
|
||||
def _resolve_target(target: str, nmap: dict) -> str | None:
|
||||
target = target.strip()
|
||||
if not target:
|
||||
return None
|
||||
if "/" in target:
|
||||
return target if target.endswith(".md") else target + ".md"
|
||||
return nmap.get(idx_mod.slugify(target))
|
||||
|
||||
|
||||
def expand_graph(seeds, nmap: dict, base_scores: dict, max_hops: int = MAX_HOPS):
|
||||
"""BFS from seeds over body wikilinks + source_notes, scoring each neighbour
|
||||
`parent_score * GRAPH_DECAY**hop` (kept as the max over reaching paths). Returns a
|
||||
score-sorted list of (path, (score, via)) excluding the seeds themselves."""
|
||||
score_of = dict(base_scores)
|
||||
seen = set(seeds)
|
||||
results: dict[str, tuple[float, str]] = {}
|
||||
# Breadth-first by HOP so each frontier's note bodies can be fetched concurrently
|
||||
# (chorus.read_many) instead of one blocking GET per node. Scoring is unchanged: a
|
||||
# neighbour keeps the max decayed score over the paths that reach it, and non-seed
|
||||
# parents score 1.0 exactly as the serial deque version did.
|
||||
frontier = list(seeds)
|
||||
for hop in range(max_hops):
|
||||
if not frontier:
|
||||
break
|
||||
texts = chorus.read_many(frontier)
|
||||
next_frontier: list[str] = []
|
||||
for path in frontier:
|
||||
text = texts.get(path)
|
||||
if text is None:
|
||||
continue
|
||||
body = strip_frontmatter(text)
|
||||
targets = set(links.all_wikilinks(body)) | set(links.source_notes(text))
|
||||
parent = score_of.get(path, 1.0)
|
||||
decayed = parent * (GRAPH_DECAY ** (hop + 1))
|
||||
for t in targets:
|
||||
tp = _resolve_target(t, nmap)
|
||||
if not tp or tp in seeds:
|
||||
continue
|
||||
if decayed > results.get(tp, (0.0, ""))[0]:
|
||||
results[tp] = (decayed, path)
|
||||
if tp not in seen:
|
||||
seen.add(tp)
|
||||
next_frontier.append(tp)
|
||||
frontier = next_frontier
|
||||
return sorted(results.items(), key=lambda kv: -kv[1][0])
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- presentation
|
||||
def _doc_summary(path: str, text: str) -> dict:
|
||||
"""Structured per-note summary shared by the human brief and --json output."""
|
||||
out: dict = {"path": path}
|
||||
mtype = re.search(r"(?m)^type:\s*(.+)$", text)
|
||||
if mtype:
|
||||
out["type"] = mtype.group(1).strip()
|
||||
mu = _FM_UPDATED.search(text[:2000])
|
||||
if mu:
|
||||
out["updated"] = mu.group(1)
|
||||
ms = _FM_STATUS.search(text[:2000])
|
||||
if ms:
|
||||
out["status"] = ms.group(1)
|
||||
body = strip_frontmatter(text)
|
||||
status = re.search(r"(?ms)^##\s+Status\s*$(.*?)(?=^##\s|\Z)", body)
|
||||
if status and status.group(1).strip():
|
||||
out["excerpt"] = status.group(1).strip()[:400]
|
||||
else:
|
||||
kept = [ln[:200] for ln in body.splitlines()
|
||||
if ln.strip() and not ln.startswith("# ")][:6]
|
||||
out["excerpt"] = "\n".join(kept)
|
||||
return out
|
||||
|
||||
|
||||
def _brief(path: str, score: float | None = None, via: str | None = None) -> None:
|
||||
text = links.get_text(path)
|
||||
if text is None:
|
||||
return
|
||||
info = _doc_summary(path, text)
|
||||
head = f"\n### {path}"
|
||||
if score is not None:
|
||||
head += f" (score {score:.2f})"
|
||||
if via:
|
||||
head += f" (via {via})"
|
||||
print(head)
|
||||
stamps = []
|
||||
if info.get("type"):
|
||||
stamps.append(f"type: {info['type']}")
|
||||
if info.get("updated"):
|
||||
stamps.append(f"updated: {info['updated']}")
|
||||
if info.get("status"):
|
||||
stamps.append(f"status: {info['status']}")
|
||||
if stamps:
|
||||
print("_" + " · ".join(stamps) + "_") # staleness is visible, not guessed
|
||||
if info.get("excerpt"):
|
||||
print(info["excerpt"])
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- entrypoint
|
||||
def recall(query, limit: int = 8, as_json: bool = False) -> int:
|
||||
q = " ".join(query) if isinstance(query, list) else query
|
||||
today_s = chorus.today()
|
||||
index = idx_mod.load()
|
||||
nmap = idx_mod.name_map(index)
|
||||
|
||||
# --- lexical layer (BM25); rebuild the cache on a cold miss --------------
|
||||
lexical: list[tuple[str, float]] = []
|
||||
try:
|
||||
ix = load_index()
|
||||
if ix.n_docs == 0:
|
||||
ix = rebuild()
|
||||
lexical = ix.score(q, limit=limit, today_s=today_s)
|
||||
except chorus.ChorusError as exc:
|
||||
print(f"recall: BM25 index unavailable ({exc}); falling back to server search",
|
||||
file=sys.stderr)
|
||||
|
||||
base: dict[str, float] = {p: s for p, s in lexical}
|
||||
hits: list[str] = [p for p, _ in lexical]
|
||||
|
||||
# --- entity-index seed (alias-aware exact match) -------------------------
|
||||
_, e = idx_mod.resolve(index, q)
|
||||
if e and e.get("path") and e["path"] not in base:
|
||||
hits.insert(0, e["path"])
|
||||
base[e["path"]] = max(base.values(), default=1.0)
|
||||
|
||||
# --- resilience fallback: server search when lexical found nothing -------
|
||||
if not hits:
|
||||
try:
|
||||
status, body = chorus.request(
|
||||
"POST", f"{chorus.BASE}/search/simple/?query={urllib.parse.quote_plus(q)}")
|
||||
if status == 200:
|
||||
for r in (json.loads(body) or [])[:limit]:
|
||||
fn = r.get("filename") or r.get("path")
|
||||
if fn and fn not in base:
|
||||
hits.append(fn)
|
||||
base[fn] = 0.0
|
||||
except Exception: # noqa: BLE001 — fallback only; stay quiet
|
||||
pass
|
||||
|
||||
# --- graph layer ----------------------------------------------------------
|
||||
neighbours = expand_graph(hits, nmap, base, max_hops=MAX_HOPS)
|
||||
|
||||
if as_json:
|
||||
import chorus_output
|
||||
texts = chorus.read_many(hits + [p for p, _ in neighbours[: 2 * limit]])
|
||||
primary = []
|
||||
for p in hits:
|
||||
if texts.get(p) is None:
|
||||
continue
|
||||
primary.append({**_doc_summary(p, texts[p]),
|
||||
"score": round(base.get(p, 0.0), 3)})
|
||||
linked = []
|
||||
for p, (sc, via) in neighbours[: 2 * limit]:
|
||||
if texts.get(p) is None:
|
||||
continue
|
||||
linked.append({**_doc_summary(p, texts[p]),
|
||||
"score": round(sc, 3), "via": via})
|
||||
env = chorus_output.envelope("recall", {"query": q, "primary": primary,
|
||||
"linked": linked})
|
||||
print(json.dumps(env, ensure_ascii=False))
|
||||
return 0
|
||||
|
||||
print(f"== recall: {q} ==")
|
||||
print(f"\n# Primary hits ({len(hits)})")
|
||||
for p in hits:
|
||||
_brief(p, score=base.get(p))
|
||||
print(f"\n# Linked context ({len(neighbours)})")
|
||||
for p, (sc, via) in neighbours[: 2 * limit]:
|
||||
_brief(p, score=sc, via=via)
|
||||
return 0
|
||||
@@ -0,0 +1,141 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_reflect.py — H5: automatic session-reflection capture. [v1.0 — wired]
|
||||
|
||||
Today memory only accrues when the agent decides to write. This makes accrual the
|
||||
default: at session end the model extracts durable items from the conversation and
|
||||
emits a PROPOSAL set; this module dedups them against the entity index, previews them,
|
||||
and applies the accepted ones via the normal `capture` path — so nothing is written
|
||||
without a go-ahead (honors the "show before large/sensitive writes" safety rule).
|
||||
|
||||
DIVISION OF LABOR:
|
||||
* Extraction is MODEL-side (only the LLM has the conversation). The agent produces a
|
||||
JSON array of proposals matching PROPOSAL_SCHEMA.
|
||||
* This script is the deterministic spine: validate -> classify (new/update/inbox vs the
|
||||
entity index) -> preview -> apply on confirm. No NL understanding lives here, so it's
|
||||
testable offline.
|
||||
|
||||
PROPOSAL_SCHEMA (one per durable item the agent wants to remember):
|
||||
{
|
||||
"title": "Bob Smith", # required
|
||||
"kind": "person", # required unless "inbox": true
|
||||
"body": "Principal at Acme; ...", # markdown body (optional)
|
||||
"aliases": ["bob", "rs"], # optional
|
||||
"tags": ["client"], # optional; the kind is always seeded as a tag
|
||||
"sources": ["_agent/sessions/..."], # optional backward links
|
||||
"date": "YYYY-MM-DD", # optional (meeting/decision kinds)
|
||||
"domain": "business", # optional (area kind)
|
||||
"inbox": false, # route to inbox when the home is unknown
|
||||
"confidence": 0.0-1.0 # agent's own confidence; gates auto-suggest
|
||||
}
|
||||
|
||||
CLI (chorus.py reflect): dry-run previews; --apply writes — matching bootstrap/migrate/sweep.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import chorus # noqa: E402
|
||||
import chorus_index as idx_mod # noqa: E402
|
||||
|
||||
MIN_CONFIDENCE = 0.6
|
||||
|
||||
|
||||
def validate(proposals: list[dict]) -> tuple[list[dict], list[str]]:
|
||||
"""Return (valid, errors). title always required; kind required unless inbox; a
|
||||
confidence below the floor is routed away (the agent should send it to the inbox)."""
|
||||
valid, errors = [], []
|
||||
for i, p in enumerate(proposals or []):
|
||||
if not str(p.get("title", "")).strip():
|
||||
errors.append(f"proposal[{i}]: missing title")
|
||||
continue
|
||||
is_inbox = bool(p.get("inbox"))
|
||||
if not is_inbox:
|
||||
kind = str(p.get("kind", "")).strip()
|
||||
if not kind:
|
||||
errors.append(f"proposal[{i}] '{p['title']}': missing kind (or set inbox:true)")
|
||||
continue
|
||||
if kind not in idx_mod.KIND_FOLDER:
|
||||
errors.append(f"proposal[{i}] '{p['title']}': unknown kind '{kind}'")
|
||||
continue
|
||||
if float(p.get("confidence", 1.0)) < MIN_CONFIDENCE:
|
||||
errors.append(f"proposal[{i}] '{p['title']}': confidence < {MIN_CONFIDENCE} — send to inbox instead")
|
||||
continue
|
||||
valid.append(p)
|
||||
return valid, errors
|
||||
|
||||
|
||||
def classify(proposals: list[dict]) -> list[dict]:
|
||||
"""Annotate each proposal with `_action` (create / update / inbox) and `_path`, by
|
||||
resolving its title against the entity index (alias-aware), so the preview shows
|
||||
create-vs-append and dedups against what's already in memory."""
|
||||
index = idx_mod.load()
|
||||
for p in proposals:
|
||||
if p.get("inbox") or not p.get("kind"):
|
||||
p["_action"], p["_path"] = "inbox", "inbox/captures/inbox.md"
|
||||
continue
|
||||
_, e = idx_mod.resolve(index, p["title"])
|
||||
if e and e.get("path"):
|
||||
p["_action"], p["_path"] = "update", e["path"]
|
||||
else:
|
||||
try:
|
||||
p["_path"] = idx_mod.derive_path(
|
||||
p["kind"], idx_mod.slugify(p["title"]),
|
||||
date=p.get("date"), domain=p.get("domain", "business"))
|
||||
p["_action"] = "create"
|
||||
except chorus.ChorusError as exc:
|
||||
p["_action"], p["_path"] = "error", str(exc)
|
||||
return proposals
|
||||
|
||||
|
||||
def preview(proposals: list[dict]) -> str:
|
||||
"""One confirmable line per proposal: action | kind | title -> target path."""
|
||||
rows = []
|
||||
for p in proposals:
|
||||
act = p.get("_action", "?")
|
||||
kind = "-" if act == "inbox" else p.get("kind", "?")
|
||||
rows.append(f" {act:7} | {kind:9} | {p['title']} -> {p.get('_path', '?')}")
|
||||
return "\n".join(rows)
|
||||
|
||||
|
||||
def apply(proposals: list[dict], confirm: bool = False) -> int:
|
||||
"""Validate -> classify -> preview; with confirm=True, apply each via chorus_ops.capture
|
||||
(which routes/indexes/links/logs, each self-lock-guarded). Without confirm it is a
|
||||
dry-run that writes nothing — the preview IS the confirmation step."""
|
||||
valid, errors = validate(proposals)
|
||||
for e in errors:
|
||||
print(f"skip: {e}", file=sys.stderr)
|
||||
if not valid:
|
||||
print("reflect: no valid proposals to apply.")
|
||||
return 0
|
||||
|
||||
classify(valid)
|
||||
counts = {a: sum(1 for p in valid if p.get("_action") == a) for a in ("create", "update", "inbox", "error")}
|
||||
print(f"reflect: {len(valid)} proposal(s) — "
|
||||
f"{counts['create']} new, {counts['update']} update, {counts['inbox']} inbox"
|
||||
+ (f", {counts['error']} error" if counts["error"] else ""))
|
||||
print(preview(valid))
|
||||
|
||||
if not confirm:
|
||||
print("\nreflect: dry-run — re-run with --apply to write these to memory.")
|
||||
return 0
|
||||
|
||||
import chorus_ops # lazy: the apply path pulls in the capture/index/link stack
|
||||
applied = gated = 0
|
||||
for p in valid:
|
||||
if p.get("_action") == "error":
|
||||
continue
|
||||
bodyfile = chorus.temp_file((p.get("body") or "").encode()) if p.get("body") else None
|
||||
rc = chorus_ops.capture(
|
||||
p.get("kind"), p["title"], bodyfile,
|
||||
aliases=p.get("aliases") or [], sources=p.get("sources") or [],
|
||||
tags=p.get("tags") or [],
|
||||
date=p.get("date"), domain=p.get("domain", "business"),
|
||||
inbox=bool(p.get("inbox")) or not p.get("kind"))
|
||||
applied += 1 if rc == 0 else 0
|
||||
gated += 1 if rc == 76 else 0
|
||||
print(f"reflect: applied {applied}/{len(valid)} proposal(s)."
|
||||
+ (f" {gated} stopped at the duplicate gate — re-propose with the existing "
|
||||
f"entity's title (or capture --merge-into <slug>)." if gated else ""))
|
||||
return 0
|
||||
@@ -0,0 +1,133 @@
|
||||
#!/usr/bin/env python3
|
||||
"""chorus_triage.py — one-tap inbox triage, built on the reflect pipeline.
|
||||
|
||||
Triage used to be pure prose: the agent hand-routed each inbox line with individual
|
||||
PATCH/PUT calls and hand-wrote the processing log — which is why the inbox rots.
|
||||
But the machinery it needs already exists in chorus_reflect (validate -> classify
|
||||
against the entity index -> preview -> apply via capture). This module reuses it
|
||||
and adds the two triage-specific pieces:
|
||||
|
||||
* `list_inbox` — parse `inbox/captures/inbox.md` into structured capture lines
|
||||
(date, text, age in days) so the model builds proposals from
|
||||
data, not by re-reading prose;
|
||||
* `apply` — same contract as reflect (dry-run unless confirm), plus an
|
||||
audit line in `inbox/processing-log/YYYY-MM-DD.md` for every
|
||||
routed item. Originals are NEVER deleted (operating contract:
|
||||
the processing log is the audit trail, deletion is explicit).
|
||||
|
||||
Proposals are reflect's PROPOSAL_SCHEMA plus an optional `"line"` — the original
|
||||
inbox line, echoed into the processing log so the audit trail maps 1:1.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import chorus # noqa: E402
|
||||
|
||||
INBOX_PATH = "inbox/captures/inbox.md"
|
||||
LOG_DIR = "inbox/processing-log"
|
||||
|
||||
_CAPTURE_LINE = re.compile(r"^\s*-\s*(\d{4}-\d{2}-\d{2})\s*:?\s*(.*\S)\s*$")
|
||||
|
||||
|
||||
def parse_inbox(text: str) -> list[dict]:
|
||||
"""Dated capture bullets -> [{line, date, text, age_days}] (age from CHORUS_TODAY)."""
|
||||
import datetime as dt
|
||||
try:
|
||||
today = dt.date.fromisoformat(chorus.today())
|
||||
except ValueError:
|
||||
today = dt.date.today()
|
||||
items = []
|
||||
for raw_line in text.splitlines():
|
||||
m = _CAPTURE_LINE.match(raw_line)
|
||||
if not m:
|
||||
continue
|
||||
try:
|
||||
age = (today - dt.date.fromisoformat(m.group(1))).days
|
||||
except ValueError:
|
||||
age = None
|
||||
items.append({"line": raw_line.strip(), "date": m.group(1),
|
||||
"text": m.group(2), "age_days": age})
|
||||
return items
|
||||
|
||||
|
||||
def list_inbox(as_json: bool = False) -> int:
|
||||
status, body = chorus.request("GET", chorus.vault_url(INBOX_PATH))
|
||||
if status == 404:
|
||||
items = []
|
||||
else:
|
||||
chorus.check(status, body, f"triage list {INBOX_PATH}")
|
||||
items = parse_inbox(body.decode(errors="replace"))
|
||||
if as_json:
|
||||
import chorus_output
|
||||
env = chorus_output.envelope("triage-list", {"path": INBOX_PATH, "items": items,
|
||||
"count": len(items)})
|
||||
print(json.dumps(env, ensure_ascii=False))
|
||||
return 0
|
||||
if not items:
|
||||
print("triage: inbox is empty — nothing to route.")
|
||||
return 0
|
||||
print(f"triage: {len(items)} capture(s) in {INBOX_PATH}")
|
||||
for it in items:
|
||||
age = f"{it['age_days']}d" if it["age_days"] is not None else "?"
|
||||
print(f" [{age:>4}] {it['date']}: {it['text']}")
|
||||
print("\nBuild a proposals JSON (reflect schema + optional \"line\") and run "
|
||||
"`chorus.py triage <file>` to preview, `--apply` to route.")
|
||||
return 0
|
||||
|
||||
|
||||
def apply(proposals: list[dict], confirm: bool = False) -> int:
|
||||
"""Validate -> classify -> preview; with confirm, route each via capture AND write
|
||||
the processing-log audit line. Mirrors chorus_reflect.apply's contract exactly."""
|
||||
import chorus_reflect
|
||||
valid, errors = chorus_reflect.validate(proposals)
|
||||
for e in errors:
|
||||
print(f"skip: {e}", file=sys.stderr)
|
||||
if not valid:
|
||||
print("triage: no valid proposals to route.")
|
||||
return 0
|
||||
|
||||
chorus_reflect.classify(valid)
|
||||
counts = {a: sum(1 for p in valid if p.get("_action") == a)
|
||||
for a in ("create", "update", "inbox", "error")}
|
||||
print(f"triage: {len(valid)} proposal(s) — "
|
||||
f"{counts['create']} new, {counts['update']} update, {counts['inbox']} stay-in-inbox"
|
||||
+ (f", {counts['error']} error" if counts["error"] else ""))
|
||||
print(chorus_reflect.preview(valid))
|
||||
|
||||
if not confirm:
|
||||
print("\ntriage: dry-run — re-run with --apply to route these and log the moves.")
|
||||
return 0
|
||||
|
||||
import chorus_ops
|
||||
today_s = chorus.today()
|
||||
applied = gated = 0
|
||||
for p in valid:
|
||||
if p.get("_action") == "error":
|
||||
continue
|
||||
if p.get("_action") == "inbox":
|
||||
continue # routing an inbox line back to the inbox is a no-op, not a move
|
||||
bodyfile = chorus.temp_file((p.get("body") or "").encode()) if p.get("body") else None
|
||||
rc = chorus_ops.capture(
|
||||
p.get("kind"), p["title"], bodyfile,
|
||||
aliases=p.get("aliases") or [], sources=p.get("sources") or [],
|
||||
tags=p.get("tags") or [],
|
||||
date=p.get("date"), domain=p.get("domain", "business"))
|
||||
if rc == 76:
|
||||
gated += 1
|
||||
continue
|
||||
if rc != 0:
|
||||
continue
|
||||
applied += 1
|
||||
original = (p.get("line") or p["title"]).strip()
|
||||
chorus.cmd_append(f"{LOG_DIR}/{today_s}.md",
|
||||
f"- {original} → {p.get('_path', '?')}")
|
||||
print(f"triage: routed {applied}/{len(valid)} item(s); audit in {LOG_DIR}/{today_s}.md. "
|
||||
"Originals kept in the inbox (deletion is explicit-only)."
|
||||
+ (f" {gated} stopped at the duplicate gate — re-propose with the existing "
|
||||
f"entity's title or use capture --merge-into." if gated else ""))
|
||||
return 0
|
||||
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env python3
|
||||
"""migrate.py — bring an existing CHORUS vault up to the plugin's current schema.
|
||||
|
||||
Reads the marker's schema_version and applies each intervening migration in order.
|
||||
Migrations are idempotent and additive; every destructive step (DELETE/move) is
|
||||
gated behind --apply AND printed first. Default mode is a DRY-RUN plan.
|
||||
Cross-platform: pure Python via chorus.py.
|
||||
|
||||
Usage:
|
||||
migrate.py # print the migration plan (no changes)
|
||||
migrate.py --apply # perform the migration (moves/deletes included)
|
||||
|
||||
Env: CHORUS_BASE, CHORUS_KEY (via chorus.py).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(SCRIPT_DIR))
|
||||
|
||||
import chorus # noqa: E402
|
||||
|
||||
CURRENT_SCHEMA = 4
|
||||
|
||||
|
||||
def get_text(path: str) -> str | None:
|
||||
status, body = chorus.request("GET", chorus.vault_url(path))
|
||||
if status == 404:
|
||||
return None
|
||||
chorus.check(status, body, f"get {path}")
|
||||
return body.decode(errors="replace")
|
||||
|
||||
|
||||
def list_files(path: str) -> list[str]:
|
||||
if not path.endswith("/"):
|
||||
path += "/"
|
||||
status, body = chorus.request("GET", chorus.vault_url(path))
|
||||
if status == 404:
|
||||
return []
|
||||
chorus.check(status, body, f"ls {path}")
|
||||
try:
|
||||
payload = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
return []
|
||||
return [entry for entry in payload.get("files", []) if not entry.endswith("/")]
|
||||
|
||||
|
||||
def put_text(path: str, text: str) -> None:
|
||||
status, body = chorus.request("PUT", chorus.vault_url(path), data=text.encode(),
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
chorus.check(status, body, f"put {path}")
|
||||
|
||||
|
||||
def delete(path: str) -> None:
|
||||
status, body = chorus.request("DELETE", chorus.vault_url(path))
|
||||
if status != 404:
|
||||
chorus.check(status, body, f"delete {path}")
|
||||
|
||||
|
||||
def move(src: str, dst: str) -> None:
|
||||
text = get_text(src)
|
||||
if text is None:
|
||||
return
|
||||
put_text(dst, text)
|
||||
delete(src)
|
||||
|
||||
|
||||
def do_or_show(apply: bool, desc: str, func=None) -> None:
|
||||
if apply:
|
||||
print(f"migrate: APPLY {desc}")
|
||||
if func:
|
||||
func()
|
||||
else:
|
||||
print(f"migrate: PLAN {desc}")
|
||||
|
||||
|
||||
def find_marker() -> str:
|
||||
"""The vault's marker path: the CHORUS name, else a legacy pre-fork ECHO
|
||||
marker (the rename itself is a future migration step)."""
|
||||
for p in ("_agent/chorus-vault.md", "_agent/echo-vault.md"):
|
||||
if get_text(p) is not None:
|
||||
return p
|
||||
return "_agent/chorus-vault.md"
|
||||
|
||||
|
||||
def marker_schema() -> int:
|
||||
marker = get_text(find_marker())
|
||||
if marker is None:
|
||||
print("migrate: marker missing — vault not bootstrapped. Run bootstrap.py, not migrate.py.")
|
||||
raise SystemExit(3)
|
||||
for line in marker.splitlines():
|
||||
if line.startswith("schema_version:"):
|
||||
try:
|
||||
return int(line.split(":", 1)[1].strip())
|
||||
except ValueError:
|
||||
return 0
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Migrate CHORUS vault schema")
|
||||
parser.add_argument("--apply", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
start = marker_schema()
|
||||
print(f"migrate: vault schema_version={start}, plugin schema={CURRENT_SCHEMA} "
|
||||
f"{'(APPLY)' if args.apply else '(dry-run)'}")
|
||||
if start >= CURRENT_SCHEMA:
|
||||
print("migrate: up to date — nothing to do.")
|
||||
return 0
|
||||
|
||||
if start < 1:
|
||||
print("migrate: [0->1] retire in-vault control docs (CLAUDE/BOOTSTRAP/STRUCTURE/index.md)")
|
||||
for filename in ["CLAUDE.md", "BOOTSTRAP.md", "STRUCTURE.md", "index.md"]:
|
||||
if get_text(filename) is not None:
|
||||
do_or_show(args.apply, f"delete vault/{filename} (back it up outside the vault first)",
|
||||
lambda f=filename: delete(f))
|
||||
print("migrate: [0->1] reminder: scrub dangling control-doc [[links]] from Related sections.")
|
||||
|
||||
if start < 2:
|
||||
print("migrate: [1->2] fold reviews/ into journal/ and _agent/health/")
|
||||
for filename in list_files("reviews/weekly"):
|
||||
if filename.endswith(".md"):
|
||||
dst = f"journal/weekly/{filename.replace('-review.md', '.md')}"
|
||||
do_or_show(args.apply, f"move reviews/weekly/{filename} -> {dst}",
|
||||
lambda f=filename, d=dst: move(f"reviews/weekly/{f}", d))
|
||||
for filename in list_files("reviews/monthly"):
|
||||
if filename.endswith(".md"):
|
||||
dst = f"_agent/health/{filename}" if filename.endswith("vault-health.md") else f"journal/monthly/{filename}"
|
||||
do_or_show(args.apply, f"move reviews/monthly/{filename} -> {dst}",
|
||||
lambda f=filename, d=dst: move(f"reviews/monthly/{f}", d))
|
||||
for period in ["quarterly", "annual"]:
|
||||
for filename in list_files(f"reviews/{period}"):
|
||||
if filename.endswith(".md"):
|
||||
dst = f"journal/{period}/{filename}"
|
||||
do_or_show(args.apply, f"move reviews/{period}/{filename} -> {dst}",
|
||||
lambda p=period, f=filename, d=dst: move(f"reviews/{p}/{f}", d))
|
||||
print("migrate: [1->2] reminder: update inbound [[reviews/...]] wikilinks manually.")
|
||||
|
||||
if start < 3:
|
||||
print("migrate: [2->3] add the entity-index folder (cross-link / recall feature)")
|
||||
if get_text("_agent/index/README.md") is None:
|
||||
do_or_show(args.apply, "create _agent/index/README.md",
|
||||
lambda: put_text("_agent/index/README.md",
|
||||
"# index\n\nMachine-maintained entity index. See the chorus-memory plugin.\n"))
|
||||
print("migrate: [2->3] then run `sweep.py --apply` to build entities.json and symmetrize links.")
|
||||
|
||||
if start < 4:
|
||||
print("migrate: [3->4] add the recall (BM25) index — hybrid lexical+graph recall")
|
||||
print("migrate: [3->4] _agent/index/ already exists (schema 3); no moves needed.")
|
||||
print("migrate: [3->4] then run `sweep.py --apply` to build _agent/index/recall-index.json.")
|
||||
|
||||
marker_path = find_marker()
|
||||
do_or_show(args.apply, f"set {marker_path} schema_version -> {CURRENT_SCHEMA}",
|
||||
lambda: chorus.cmd_fm(marker_path, "schema_version", str(CURRENT_SCHEMA)))
|
||||
if args.apply:
|
||||
print(f"migrate: migration complete -> schema {CURRENT_SCHEMA}. Run sweep.py --apply, then vault_lint.py.")
|
||||
else:
|
||||
print("migrate: dry-run only. Re-run with --apply to perform the moves/deletes above.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"$comment": "CANONICAL machine-readable routing manifest for the CHORUS vault. This is the single source of truth for 'what paths may be written to'. The human-readable tables in SKILL.md, references/routing-map.md, and references/api-reference.md are DERIVED views of this file — when they disagree, this file wins. vault_lint.py consumes it to enforce the core rule (if a vault path matches no route here, and is not retired, nothing should be written to it); check_routing.py consumes it to verify the human routing docs stay in sync with this manifest. Patterns are Python regexes matched against vault-root-relative paths (no leading slash, no /vault/ prefix).",
|
||||
"schema_version": 2,
|
||||
"routes": [
|
||||
{ "id": "inbox-captures", "pattern": "^inbox/captures/inbox\\.md$", "method": "POST", "trigger": "Destination unknown at capture time", "distinct_because": "Only path whose contract is deferred routing" },
|
||||
{ "id": "inbox-imports", "pattern": "^inbox/imports/[^/]+\\.md$", "method": "PUT", "trigger": "Raw external material dropped wholesale", "distinct_because": "Bulk un-triaged material vs single-line captures" },
|
||||
{ "id": "inbox-processing-log", "pattern": "^inbox/processing-log/\\d{4}-\\d{2}-\\d{2}\\.md$", "method": "POST", "trigger": "An inbox item is routed to its real home", "distinct_because": "Audit trail of moves, not memory itself" },
|
||||
|
||||
{ "id": "journal-daily", "pattern": "^journal/daily/\\d{4}-\\d{2}-\\d{2}\\.md$", "method": "PATCH", "trigger": "First agent activity on a given day", "distinct_because": "Finest grain; PATCHed repeatedly within its period" },
|
||||
{ "id": "journal-weekly", "pattern": "^journal/weekly/\\d{4}-W\\d{2}\\.md$", "method": "PUT", "trigger": "First substantive session of a new ISO week (opt-in)", "distinct_because": "ISO-week grain" },
|
||||
{ "id": "journal-monthly", "pattern": "^journal/monthly/\\d{4}-\\d{2}\\.md$", "method": "PUT", "trigger": "First substantive session of a new month", "distinct_because": "Month grain" },
|
||||
{ "id": "journal-quarterly", "pattern": "^journal/quarterly/\\d{4}-Q[1-4]\\.md$", "method": "PUT", "trigger": "Manual / on request only", "distinct_because": "Strategic grain; never auto-fires" },
|
||||
{ "id": "journal-annual", "pattern": "^journal/annual/\\d{4}\\.md$", "method": "PUT", "trigger": "Manual / on request only", "distinct_because": "Coarsest grain; never auto-fires" },
|
||||
{ "id": "journal-templates", "pattern": "^journal/templates/.+\\.md$", "method": "PUT", "trigger": "Bootstrap seed only", "distinct_because": "Holds templates, not journal content" },
|
||||
|
||||
{ "id": "projects-active", "pattern": "^projects/active/[^/]+\\.md$", "method": "PUT", "trigger": "Work in motion now", "distinct_because": "Default state for anything being worked", "status": "active" },
|
||||
{ "id": "projects-incubating", "pattern": "^projects/incubating/[^/]+\\.md$", "method": "PUT", "trigger": "Idea captured, work not started", "distinct_because": "Pre-work", "status": "incubating" },
|
||||
{ "id": "projects-on-hold", "pattern": "^projects/on-hold/[^/]+\\.md$", "method": "PUT", "trigger": "Paused but still tracked", "distinct_because": "Resumable; not terminal", "status": "on-hold" },
|
||||
{ "id": "projects-archived", "pattern": "^projects/archived/[^/]+\\.md$", "method": "PUT", "trigger": "Done, abandoned, or rolled up", "distinct_because": "Terminal; kept for history", "status": "archived" },
|
||||
{ "id": "projects-template", "pattern": "^projects/project-template\\.md$", "method": "PUT", "trigger": "Bootstrap seed only", "distinct_because": "Template, not a project" },
|
||||
|
||||
{ "id": "areas", "pattern": "^areas/(business|personal|learning|systems)/[^/]+\\.md$", "method": "PUT", "trigger": "Ongoing domain with no finish line", "distinct_because": "No end state — disqualifies it from projects/" },
|
||||
|
||||
{ "id": "resources-people", "pattern": "^resources/people/[^/]+\\.md$", "method": "PUT", "trigger": "A fact about a specific person", "distinct_because": "Keyed to a person" },
|
||||
{ "id": "resources-companies", "pattern": "^resources/companies/[^/]+\\.md$", "method": "PUT", "trigger": "A fact about an organization", "distinct_because": "Keyed to an organization, not an individual" },
|
||||
{ "id": "resources-concepts", "pattern": "^resources/concepts/[^/]+\\.md$", "method": "PUT", "trigger": "A reusable concept/idea", "distinct_because": "An idea vs an external source" },
|
||||
{ "id": "resources-references", "pattern": "^resources/references/[^/]+\\.md$", "method": "PUT", "trigger": "An external source/link worth keeping", "distinct_because": "Points outward" },
|
||||
{ "id": "resources-meetings", "pattern": "^resources/meetings/\\d{4}-\\d{2}-\\d{2}-[^/]+\\.md$", "method": "PUT", "trigger": "Notes tied to a specific meeting", "distinct_because": "Event-anchored to a meeting" },
|
||||
|
||||
{ "id": "decisions-by-date", "pattern": "^decisions/by-date/\\d{4}-\\d{2}-\\d{2}-[^/]+\\.md$", "method": "PUT", "trigger": "A non-obvious decision worth recording", "distinct_because": "Chronological system of record for decisions" },
|
||||
{ "id": "decisions-template", "pattern": "^decisions/decision-template\\.md$", "method": "PUT", "trigger": "Bootstrap seed only", "distinct_because": "Template, not a decision" },
|
||||
|
||||
{ "id": "agent-marker", "pattern": "^_agent/chorus-vault\\.md$", "method": "PUT", "trigger": "Bootstrap / schema migration only", "distinct_because": "Plugin-owned probe; never hand-edited" },
|
||||
{ "id": "agent-marker-legacy", "pattern": "^_agent/echo-vault\\.md$", "method": "GET", "trigger": "Pre-fork ECHO vault adoption (read-only probe)", "distinct_because": "Legacy ECHO marker honored until the rename migration" },
|
||||
{ "id": "agent-context", "pattern": "^_agent/context/[^/]+\\.md$", "method": "PATCH", "trigger": "Active scope changes / task bundles", "distinct_because": "Single live scope pointer + bundles" },
|
||||
{ "id": "agent-semantic", "pattern": "^_agent/memory/semantic/[^/]+\\.md$", "method": "PUT", "trigger": "A durable fact/pattern (incl. operator-preferences.md)", "distinct_because": "Timeless fact" },
|
||||
{ "id": "agent-episodic", "pattern": "^_agent/memory/episodic/[^/]+\\.md$", "method": "PUT", "trigger": "A record of what happened, when", "distinct_because": "Anchored to an event in time" },
|
||||
{ "id": "agent-working", "pattern": "^_agent/memory/working/[^/]+\\.md$", "method": "PUT", "trigger": "Short-lived state for the current effort", "distinct_because": "Explicitly transient" },
|
||||
{ "id": "agent-sessions", "pattern": "^_agent/sessions/\\d{4}-\\d{2}-\\d{2}(-\\d{4})?-[^/]+\\.md$", "method": "PUT", "trigger": "A substantive session ends", "distinct_because": "Per-session record (new ones require HHMM)" },
|
||||
{ "id": "agent-health", "pattern": "^_agent/health/\\d{4}-\\d{2}-vault-health\\.md$", "method": "PUT", "trigger": "First substantive session of a month", "distinct_because": "Vault integrity, not work narrative" },
|
||||
{ "id": "agent-heartbeat", "pattern": "^_agent/heartbeat/[^/]+\\.md$", "method": "PUT", "trigger": "End of every session", "distinct_because": "O(1) orientation pointer; overwritten, never grows" },
|
||||
{ "id": "agent-templates", "pattern": "^_agent/templates/.+\\.md$", "method": "PUT", "trigger": "Bootstrap seed only", "distinct_because": "Holds templates, not memory" },
|
||||
{ "id": "agent-skills-active", "pattern": "^_agent/skills/active/[^/]+\\.md$", "method": "PUT", "trigger": "A skill/plugin catalogued as a capability", "distinct_because": "Catalogs a capability vs the build effort" },
|
||||
{ "id": "agent-skills-archived","pattern": "^_agent/skills/archived/[^/]+\\.md$", "method": "PUT", "trigger": "A catalogued skill is retired", "distinct_because": "Terminal state of the skill catalog" },
|
||||
{ "id": "agent-locks", "pattern": "^_agent/locks/[^/]+\\.lock$", "method": "PUT", "trigger": "Advisory multi-writer lock acquire/release", "distinct_because": "Concurrency coordination, not memory" },
|
||||
{ "id": "agent-index", "pattern": "^_agent/index/[^/]+\\.json$", "method": "PUT", "trigger": "Entity index rebuilt on write/sweep", "distinct_because": "Machine-maintained routing/recall registry, not a note" },
|
||||
|
||||
{ "id": "leaf-readme", "pattern": "^(.+/)?README\\.md$", "method": "PUT", "trigger": "Bootstrap leaf signpost / vault root README", "distinct_because": "Human signpost, not read for routing" }
|
||||
],
|
||||
"retired": [
|
||||
{ "pattern": "^reviews/", "retired_in_schema": 2, "replacement": "journal/{weekly,monthly,quarterly,annual}/ and _agent/health/" },
|
||||
{ "pattern": "^decisions/by-project/", "retired_in_schema": 1, "replacement": "[[wikilink]] under the project's ## Key Decisions" },
|
||||
{ "pattern": "^archive/", "retired_in_schema": 0, "replacement": "projects/archived/ and _agent/skills/archived/" },
|
||||
{ "pattern": "^(CLAUDE|BOOTSTRAP|STRUCTURE|index)\\.md$", "retired_in_schema": 1, "replacement": "All control logic lives in the plugin references/, not the vault" }
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,251 @@
|
||||
#!/usr/bin/env python3
|
||||
"""sweep.py — bring an existing CHORUS vault up to the current feature spec.
|
||||
|
||||
After upgrading the plugin (entity index, cross-linking, recall), run this once to
|
||||
backfill what older vaults don't have yet:
|
||||
|
||||
1. ensure the `_agent/index/` folder exists,
|
||||
2. **(re)build the entity index** (`_agent/index/entities.json`) from the notes
|
||||
already in the vault (people, companies, projects, concepts, references,
|
||||
meetings, areas, decisions, semantic/episodic memory, skills),
|
||||
3. **backfill incomplete entity frontmatter** — fill a missing/empty `status:` with
|
||||
the kind default and seed a missing/empty `tags:` with the note's kind, per the
|
||||
KIND_REQUIRED_FM/KIND_STATUS maps in chorus_index (what `/chorus-health` flags as
|
||||
`incomplete-frontmatter`),
|
||||
4. **symmetrize cross-links** — for every `## Related` link A -> B, ensure the
|
||||
reciprocal B -> A exists (it only adds the missing direction; it never invents
|
||||
new links from body text), and
|
||||
5. stamp the marker `schema_version` to the current schema (4).
|
||||
|
||||
Dry-run by default; pass --apply to write. READ-ONLY without --apply.
|
||||
Cross-platform: pure Python via chorus.py.
|
||||
|
||||
Usage: sweep.py [--apply]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(SCRIPT_DIR))
|
||||
|
||||
import chorus # noqa: E402
|
||||
import chorus_index as idx_mod # noqa: E402
|
||||
import chorus_links as links # noqa: E402
|
||||
import chorus_recall # noqa: E402
|
||||
|
||||
CURRENT_SCHEMA = 4
|
||||
TEMPLATE_RE = re.compile(r"(^|/)(templates/|.*-template\.md$)")
|
||||
SKIP_BASENAMES = {"README.md"}
|
||||
|
||||
|
||||
kind_for = idx_mod.kind_for_path
|
||||
|
||||
|
||||
def fm_block(text: str) -> str:
|
||||
if not text.startswith("---"):
|
||||
return ""
|
||||
end = text.find("\n---", 3)
|
||||
return text[:end] if end != -1 else text
|
||||
|
||||
|
||||
def fm_populated(text: str, field: str) -> bool:
|
||||
"""True when the frontmatter carries a real value for `field` (flow list `[x]`,
|
||||
block list items, or a non-empty scalar). Mirrors vault_lint.fm_field_populated."""
|
||||
fm = fm_block(text)
|
||||
m = re.search(rf"(?m)^{re.escape(field)}:[ \t]*(.*)$", fm)
|
||||
if not m:
|
||||
return False
|
||||
val = m.group(1).strip()
|
||||
if val == "[]":
|
||||
return False
|
||||
if val:
|
||||
return True
|
||||
rest = fm[m.end():].lstrip("\n")
|
||||
first = rest.splitlines()[0] if rest else ""
|
||||
return bool(re.match(r"^\s+-\s*\S", first))
|
||||
|
||||
|
||||
def get(path: str) -> str | None:
|
||||
status, body = chorus.request("GET", chorus.vault_url(path))
|
||||
if status == 404:
|
||||
return None
|
||||
chorus.check(status, body, f"get {path}")
|
||||
return body.decode(errors="replace")
|
||||
|
||||
|
||||
def list_dir(path: str):
|
||||
p = "" if path in ("", "/") else (path if path.endswith("/") else path + "/")
|
||||
body = get(p)
|
||||
if body is None:
|
||||
return [], []
|
||||
try:
|
||||
j = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
return [], []
|
||||
entries = list(j.get("files", [])) + list(j.get("folders", []))
|
||||
files = [e for e in entries if not e.endswith("/")]
|
||||
folders = [e[:-1] for e in entries if e.endswith("/")]
|
||||
return files, folders
|
||||
|
||||
|
||||
def walk(prefix: str = ""):
|
||||
files, folders = list_dir(prefix)
|
||||
for f in files:
|
||||
yield prefix + f
|
||||
for d in folders:
|
||||
yield from walk(f"{prefix}{d}/")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
ap = argparse.ArgumentParser(description="Bring an CHORUS vault up to the current feature spec")
|
||||
ap.add_argument("--apply", action="store_true")
|
||||
args = ap.parse_args(argv)
|
||||
apply = args.apply
|
||||
tag = "APPLY" if apply else "PLAN "
|
||||
|
||||
if get("_agent/chorus-vault.md") is None and get("_agent/echo-vault.md") is None:
|
||||
print("sweep: marker missing — vault not bootstrapped. Run bootstrap.py first.", file=sys.stderr)
|
||||
return 3
|
||||
|
||||
print(f"sweep: {'applying changes' if apply else 'dry-run (no writes)'}\n")
|
||||
|
||||
# ---- 1. index folder ------------------------------------------------------
|
||||
if get("_agent/index/README.md") is None:
|
||||
print(f"sweep: {tag} create _agent/index/README.md")
|
||||
if apply:
|
||||
chorus.request("PUT", chorus.vault_url("_agent/index/README.md"),
|
||||
data=b"# index\n\nMachine-maintained entity index. See the chorus-memory plugin.\n",
|
||||
headers={"Content-Type": "text/markdown"})
|
||||
|
||||
all_files = [p for p in walk()
|
||||
if p.endswith(".md") and p.rsplit("/", 1)[-1] not in SKIP_BASENAMES
|
||||
and not TEMPLATE_RE.search(p)]
|
||||
|
||||
# Fetch every note's content ONCE, concurrently. All three passes below (index,
|
||||
# recall, link symmetrize) read from this shared cache, so no file is fetched
|
||||
# twice and link targets aren't re-fetched per reference.
|
||||
texts = chorus.read_many(all_files)
|
||||
print(f"sweep: read {len(texts)} notes (concurrent, keep-alive)\n")
|
||||
|
||||
# ---- 2. (re)build entity index -------------------------------------------
|
||||
index = idx_mod.load()
|
||||
existing = dict(index.get("entities", {}))
|
||||
rebuilt = idx_mod.empty_index()
|
||||
added = updated = 0
|
||||
for path in all_files:
|
||||
kind = kind_for(path)
|
||||
if kind is None:
|
||||
continue
|
||||
base = path.rsplit("/", 1)[-1][:-3]
|
||||
slug = idx_mod.slugify(base)
|
||||
text = texts.get(path) or ""
|
||||
title = links.first_h1(text) or base
|
||||
prev = existing.get(slug)
|
||||
# Frontmatter aliases are authoritative and re-folded every rebuild; prior index
|
||||
# aliases (e.g. mentions learned by capture) are preserved; upsert adds title variants.
|
||||
prev_aliases = prev.get("aliases", []) if prev else []
|
||||
aliases = list(prev_aliases) + idx_mod.aliases_in_frontmatter(text)
|
||||
idx_mod.upsert(rebuilt, slug, path, kind, title, aliases)
|
||||
if not prev:
|
||||
added += 1
|
||||
elif prev.get("path") != path or prev.get("title") != title:
|
||||
updated += 1
|
||||
print(f"sweep: {tag} entity index — {len(rebuilt['entities'])} entities "
|
||||
f"({added} new, {updated} changed)")
|
||||
if apply:
|
||||
idx_mod.save(rebuilt)
|
||||
|
||||
# ---- 2b. (re)build the recall (BM25) index -------------------------------
|
||||
if apply:
|
||||
rix = chorus_recall.rebuild(prefetched=texts)
|
||||
print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25)")
|
||||
else:
|
||||
n_idx = sum(1 for p in all_files if chorus_recall._indexable(p))
|
||||
print(f"sweep: {tag} recall index — would rebuild BM25 over {n_idx} corpus notes "
|
||||
f"(entities + sessions/journal)")
|
||||
|
||||
# ---- 3. backfill incomplete entity frontmatter ----------------------------
|
||||
# What /chorus-health flags as incomplete-frontmatter, filled mechanically: a missing
|
||||
# or empty status gets the kind default; missing/empty tags get the kind as a
|
||||
# baseline tag. cmd_fm is create-or-replace, so absent keys are inserted surgically.
|
||||
fixes: list[tuple[str, str, str]] = []
|
||||
for path in all_files:
|
||||
kind = kind_for(path)
|
||||
if kind is None:
|
||||
continue
|
||||
text = texts.get(path) or ""
|
||||
if not text.startswith("---"):
|
||||
continue # no frontmatter block at all — lint flags it; not a mechanical fill
|
||||
for field in idx_mod.KIND_REQUIRED_FM.get(kind, ()):
|
||||
if not fm_populated(text, field):
|
||||
value = (idx_mod.KIND_STATUS.get(kind, "active") if field == "status"
|
||||
else json.dumps([kind]))
|
||||
fixes.append((path, field, value))
|
||||
print(f"sweep: {tag} frontmatter backfill — {len(fixes)} field(s) to fill")
|
||||
for path, field, value in fixes[:40]:
|
||||
print(f" {path}: {field} -> {value}")
|
||||
if len(fixes) > 40:
|
||||
print(f" ... and {len(fixes) - 40} more")
|
||||
if apply:
|
||||
for path, field, value in fixes:
|
||||
chorus.cmd_fm(path, field, value)
|
||||
|
||||
# ---- 4. symmetrize existing Related links --------------------------------
|
||||
fileset = set(all_files)
|
||||
basemap: dict[str, str] = {}
|
||||
for p in all_files:
|
||||
basemap.setdefault(idx_mod.slugify(p.rsplit("/", 1)[-1][:-3]), p)
|
||||
|
||||
def resolve_target(t: str) -> str | None:
|
||||
t = t.strip()
|
||||
if "/" in t:
|
||||
cand = t if t.endswith(".md") else t + ".md"
|
||||
return cand if cand in fileset else None
|
||||
return basemap.get(idx_mod.slugify(t))
|
||||
|
||||
missing = set() # (target_path, source_path) — reciprocal link to add on target
|
||||
for path in all_files:
|
||||
text = texts.get(path)
|
||||
if text is None:
|
||||
continue
|
||||
for tgt in links.related_targets(text):
|
||||
tp = resolve_target(tgt)
|
||||
if not tp or tp == path:
|
||||
continue
|
||||
back = {resolve_target(t) for t in links.related_targets(texts.get(tp) or "")}
|
||||
if path not in back:
|
||||
missing.add((tp, path))
|
||||
missing = sorted(missing)
|
||||
print(f"sweep: {tag} reciprocal links — {len(missing)} to add")
|
||||
for tp, sp in missing[:40]:
|
||||
print(f" {tp} += [[{links.link_token(sp)}]]")
|
||||
if len(missing) > 40:
|
||||
print(f" ... and {len(missing) - 40} more")
|
||||
if apply:
|
||||
for tp, sp in missing:
|
||||
links.add_one(tp, sp)
|
||||
|
||||
# ---- 5. stamp schema ------------------------------------------------------
|
||||
marker_path = "_agent/chorus-vault.md"
|
||||
marker = get(marker_path)
|
||||
if marker is None: # adopted pre-fork ECHO vault — stamp the marker it has
|
||||
marker_path = "_agent/echo-vault.md"
|
||||
marker = get(marker_path) or ""
|
||||
cur = next((int(m) for m in re.findall(r"(?m)^schema_version:\s*(\d+)", marker)), 0)
|
||||
if cur < CURRENT_SCHEMA:
|
||||
print(f"sweep: {tag} schema_version {cur} -> {CURRENT_SCHEMA}")
|
||||
if apply:
|
||||
chorus.cmd_fm(marker_path, "schema_version", str(CURRENT_SCHEMA))
|
||||
|
||||
print(f"\nsweep: {'done — run vault_lint.py to confirm invariants' if apply else 'dry-run complete. Re-run with --apply to write.'}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,233 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Offline regression tests for the CHORUS tooling. No network, no vault.
|
||||
|
||||
Run: python3 test_chorus_client.py (or: pytest test_chorus_client.py)
|
||||
|
||||
Covers chorus.py body/scalar normalization, the routing-view consistency checker's
|
||||
helpers, and — as the guard for improvement #4 — asserts the shipped docs are in
|
||||
sync with routing.json.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
import chorus # noqa: E402
|
||||
import check_routing # noqa: E402
|
||||
import chorus_index # noqa: E402
|
||||
import chorus_links # noqa: E402
|
||||
|
||||
|
||||
def test_heading_replace_body_is_newline_guarded() -> None:
|
||||
assert chorus.normalize_patch_body(b"- [x] done\n", "replace", "heading") == b"\n- [x] done\n"
|
||||
|
||||
|
||||
def test_heading_append_body_ends_with_newline() -> None:
|
||||
assert chorus.normalize_patch_body(b"- item", "append", "heading") == b"- item\n"
|
||||
|
||||
|
||||
def test_frontmatter_plain_string_becomes_json_scalar() -> None:
|
||||
assert json.loads(chorus.normalize_json_scalar("Fabrication Lead")) == "Fabrication Lead"
|
||||
|
||||
|
||||
def test_frontmatter_existing_json_is_preserved() -> None:
|
||||
assert chorus.normalize_json_scalar('"2026-06-20"') == b'"2026-06-20"'
|
||||
|
||||
|
||||
def test_read_many_dedups_and_maps_each_path() -> None:
|
||||
# Concurrency helper contract, no network: monkeypatch the per-file fetch.
|
||||
orig = chorus.get_text
|
||||
chorus.get_text = lambda p: f"T:{p}"
|
||||
try:
|
||||
assert chorus.read_many(["a", "b", "a"]) == {"a": "T:a", "b": "T:b"}
|
||||
finally:
|
||||
chorus.get_text = orig
|
||||
|
||||
|
||||
def test_read_many_empty_returns_empty_dict() -> None:
|
||||
assert chorus.read_many([]) == {}
|
||||
|
||||
|
||||
def test_read_many_tolerates_unreadable_file_as_none() -> None:
|
||||
orig = chorus.get_text
|
||||
chorus.get_text = lambda p: None if p == "bad" else f"T:{p}"
|
||||
try:
|
||||
assert chorus.read_many(["ok", "bad"]) == {"ok": "T:ok", "bad": None}
|
||||
finally:
|
||||
chorus.get_text = orig
|
||||
|
||||
|
||||
def test_plugin_description_under_500_chars() -> None:
|
||||
# Marketplace cap — the description has silently regressed past it before (a 525-char
|
||||
# relapse + an earlier "fix description length" commit). build.py fails the build on this;
|
||||
# this guards it in CI so it's caught before packaging.
|
||||
manifest = Path(__file__).resolve().parents[3] / ".claude-plugin" / "plugin.json"
|
||||
desc = json.loads(manifest.read_text(encoding="utf-8")).get("description", "")
|
||||
assert 0 < len(desc) < 500, f"plugin description is {len(desc)} chars (must be under 500)"
|
||||
|
||||
|
||||
def test_stem_extracts_literal_directory_prefix() -> None:
|
||||
assert check_routing.stem(r"^projects/active/[^/]+\.md$") == "projects/active/"
|
||||
assert check_routing.stem(r"^areas/(business|personal)/[^/]+\.md$") == "areas/"
|
||||
assert check_routing.stem(r"^journal/daily/\d{4}-\d{2}-\d{2}\.md$") == "journal/daily/"
|
||||
|
||||
|
||||
def test_concretize_substitutes_placeholders() -> None:
|
||||
assert check_routing.concretize("projects/<lifecycle>/<slug>.md") == "projects/active/sample.md"
|
||||
assert check_routing.concretize("_agent/sessions/YYYY-MM-DD-HHMM-<slug>.md") == "_agent/sessions/2026-01-02-0900-sample.md"
|
||||
assert check_routing.concretize("journal/weekly/YYYY-Www.md") == "journal/weekly/2026-W01.md"
|
||||
|
||||
|
||||
def test_looks_like_path_filters_non_paths() -> None:
|
||||
assert check_routing.looks_like_path("inbox/captures/inbox.md")
|
||||
assert check_routing.looks_like_path("_agent/locks/vault.lock")
|
||||
assert not check_routing.looks_like_path("application/vnd.olrapi.document-map+json")
|
||||
assert not check_routing.looks_like_path("Operator Preferences::Fact / Pattern")
|
||||
assert not check_routing.looks_like_path("status: active")
|
||||
|
||||
|
||||
def test_docs_are_in_sync_with_routing_json() -> None:
|
||||
# The guard for improvement #4: SKILL.md / routing-map.md / api-reference.md
|
||||
# must stay consistent with routing.json.
|
||||
assert check_routing.main() == 0
|
||||
|
||||
|
||||
def test_index_slugify_and_derive_path() -> None:
|
||||
assert chorus_index.slugify("Bob Smith!") == "bob-smith"
|
||||
assert chorus_index.derive_path("person", "bob-smith") == "resources/people/bob-smith.md"
|
||||
assert chorus_index.derive_path("project", "chorus") == "projects/active/chorus.md"
|
||||
assert chorus_index.derive_path("area", "ops", domain="business") == "areas/business/ops.md"
|
||||
assert chorus_index.derive_path("decision", "x", date="2026-01-02") == "decisions/by-date/2026-01-02-x.md"
|
||||
|
||||
|
||||
def test_index_resolve_matches_alias_and_title() -> None:
|
||||
index = {"entities": {"bob-smith": {"path": "resources/people/bob-smith.md",
|
||||
"kind": "person", "title": "Bob Smith", "aliases": ["bob"]}}}
|
||||
assert chorus_index.resolve(index, "Bob Smith")[0] == "bob-smith"
|
||||
assert chorus_index.resolve(index, "bob")[0] == "bob-smith"
|
||||
assert chorus_index.resolve(index, "nobody")[0] is None
|
||||
|
||||
|
||||
def test_kind_for_path() -> None:
|
||||
assert chorus_index.kind_for_path("resources/people/x.md") == "person"
|
||||
assert chorus_index.kind_for_path("projects/on-hold/x.md") == "project"
|
||||
assert chorus_index.kind_for_path("journal/daily/2026-01-01.md") is None
|
||||
|
||||
|
||||
def test_derive_aliases_produces_punctuation_variants() -> None:
|
||||
al = chorus_index.derive_aliases("CHORUS Memory")
|
||||
assert "chorus-memory" in al and "chorus memory" in al
|
||||
assert chorus_index.derive_aliases("") == []
|
||||
|
||||
|
||||
def test_upsert_folds_in_title_variants_and_drops_slug() -> None:
|
||||
index = {"entities": {}}
|
||||
e = chorus_index.upsert(index, "chorus-memory", "projects/active/chorus-memory.md",
|
||||
"project", "Chorus Memory")
|
||||
assert "chorus memory" in e["aliases"] # space variant auto-derived
|
||||
assert "chorus-memory" not in e["aliases"] # equals the slug -> not stored redundantly
|
||||
|
||||
|
||||
def test_fuzzy_candidates_finds_shortened_name() -> None:
|
||||
# The real bug: a project slugged "chorus" must surface for the mention "chorus memory".
|
||||
index = {"entities": {"chorus": {"path": "projects/active/chorus.md", "kind": "project",
|
||||
"title": "chorus", "aliases": []}}}
|
||||
cands = chorus_index.fuzzy_candidates(index, "chorus memory")
|
||||
assert cands and cands[0][0] == "chorus"
|
||||
# exact resolve still misses (it's exact-only) — fuzzy is the safety net:
|
||||
assert chorus_index.resolve(index, "chorus memory")[0] is None
|
||||
|
||||
|
||||
def test_fuzzy_candidates_ignores_common_and_short_tokens() -> None:
|
||||
index = {"entities": {"data": {"path": "x.md", "kind": "concept",
|
||||
"title": "data", "aliases": []}}}
|
||||
assert chorus_index.fuzzy_candidates(index, "data pipeline") == []
|
||||
|
||||
|
||||
def test_gate_candidates_blocks_same_kind_rare_token() -> None:
|
||||
# The blocking case the gate exists for: same kind, and the shared token is unique
|
||||
# to that one entity — "Robert Smith" vs the person bob-smith must gate.
|
||||
index = {"entities": {"bob-smith": {"path": "resources/people/bob-smith.md",
|
||||
"kind": "person", "title": "Bob Smith",
|
||||
"aliases": []}}}
|
||||
gated = chorus_index.gate_candidates(index, "Robert Smith", kind="person")
|
||||
assert gated and gated[0][0] == "bob-smith"
|
||||
|
||||
|
||||
def test_gate_candidates_skips_cross_kind() -> None:
|
||||
# A company/decision named after a person or project is intentional naming, not a
|
||||
# duplicate — cross-kind lookalikes warn but never block.
|
||||
index = {"entities": {"bob-smith": {"path": "resources/people/bob-smith.md",
|
||||
"kind": "person", "title": "Bob Smith",
|
||||
"aliases": []}}}
|
||||
assert chorus_index.gate_candidates(index, "Smith Consulting", kind="company") == []
|
||||
# ...but the advisory candidate list still surfaces it:
|
||||
assert chorus_index.fuzzy_candidates(index, "Smith Consulting")
|
||||
|
||||
|
||||
def test_gate_candidates_skips_single_vault_common_token() -> None:
|
||||
# The live 1.5.0 false positive: "chorus" appears in many entity names, so any title
|
||||
# containing it scored 1.0 against every single-token "chorus" entity. A lone shared
|
||||
# token only blocks when it is unique to that entity (df == 1).
|
||||
ents = {
|
||||
"chorus": {"path": "projects/active/chorus.md", "kind": "project",
|
||||
"title": "chorus", "aliases": []},
|
||||
"chorus-v-05": {"path": "projects/archived/chorus-v.05.md", "kind": "project",
|
||||
"title": "chorus-v.05", "aliases": []},
|
||||
}
|
||||
assert chorus_index.gate_candidates({"entities": ents}, "chorus handbook",
|
||||
kind="project") == []
|
||||
# multi-token overlap still blocks, even when the individual tokens are common:
|
||||
ents["chorus-memory-system"] = {"path": "projects/active/chorus-memory-system.md",
|
||||
"kind": "project", "title": "chorus memory system",
|
||||
"aliases": []}
|
||||
ents["chorus-memory-notes"] = {"path": "resources/concepts/chorus-memory-notes.md",
|
||||
"kind": "concept", "title": "chorus memory notes",
|
||||
"aliases": []}
|
||||
gated = chorus_index.gate_candidates({"entities": ents}, "chorus memory platform",
|
||||
kind="project")
|
||||
assert [s for s, _, _ in gated] == ["chorus-memory-system"]
|
||||
|
||||
|
||||
def test_aliases_in_frontmatter_flow_and_block() -> None:
|
||||
flow = '---\ntype: project\naliases: ["chorus memory", "chorus plugin"]\n---\n# x\n'
|
||||
assert chorus_index.aliases_in_frontmatter(flow) == ["chorus memory", "chorus plugin"]
|
||||
block = "---\ntype: project\naliases:\n - chorus memory\n - chorus plugin\n---\n# x\n"
|
||||
assert chorus_index.aliases_in_frontmatter(block) == ["chorus memory", "chorus plugin"]
|
||||
assert chorus_index.aliases_in_frontmatter("# no frontmatter\n") == []
|
||||
|
||||
|
||||
def test_links_add_related_is_idempotent_and_bidirectional_token() -> None:
|
||||
text = "---\ntype: person\n---\n\n# Bob\n\n## Related\n"
|
||||
new, changed = chorus_links.add_related(text, "resources/companies/acme.md")
|
||||
assert changed and "[[resources/companies/acme]]" in new
|
||||
again, changed2 = chorus_links.add_related(new, "resources/companies/acme.md")
|
||||
assert not changed2 and again == new
|
||||
|
||||
|
||||
def test_links_create_related_section_when_absent() -> None:
|
||||
text = "---\ntype: concept\n---\n\n# Alpha\n\nbody\n"
|
||||
new, changed = chorus_links.add_related(text, "resources/concepts/beta.md")
|
||||
assert changed and "## Related" in new and "[[resources/concepts/beta]]" in new
|
||||
|
||||
|
||||
def _run_all() -> int:
|
||||
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
|
||||
failed = 0
|
||||
for test in tests:
|
||||
try:
|
||||
test()
|
||||
print(f"ok {test.__name__}")
|
||||
except AssertionError as exc:
|
||||
failed += 1
|
||||
print(f"FAIL {test.__name__}: {exc}")
|
||||
print(f"\n{len(tests) - failed}/{len(tests)} passed")
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(_run_all())
|
||||
@@ -0,0 +1,138 @@
|
||||
#!/usr/bin/env python3
|
||||
"""test_v1_scaffold.py — H4 interface guard for the v1.0 scaffolds. [offline, no creds]
|
||||
|
||||
Asserts the scaffold contract holds so the skeleton can't silently rot while 1.0 is
|
||||
built: every planned module imports, the parts that ARE implemented work (BM25 ranking,
|
||||
queue/cache round-trip, slug helpers, secret resolution order), and the parts that are
|
||||
TODO raise NotImplementedError (so a stub can't masquerade as done).
|
||||
|
||||
Run: python test_v1_scaffold.py
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
failures: list[str] = []
|
||||
|
||||
|
||||
def check(name: str, cond: bool, detail: str = "") -> None:
|
||||
print(f"{'ok ' if cond else 'FAIL'} {name}" + (f" -- {detail}" if not cond else ""))
|
||||
if not cond:
|
||||
failures.append(name)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
# H1 — BM25 core is implemented; assert it actually ranks a relevant doc first.
|
||||
import chorus_recall as r
|
||||
ix = r.Bm25Index()
|
||||
ix.add("notes/mqtt.md", "the mqtt broker config and tls certificate rotation")
|
||||
ix.add("notes/payroll.md", "quarterly payroll run and employee deductions")
|
||||
top = ix.score("certificate rotation broker", limit=2)
|
||||
check("H1 BM25 ranks relevant doc first", bool(top) and top[0][0] == "notes/mqtt.md", str(top))
|
||||
check("H1 BM25 survives json round-trip",
|
||||
r.Bm25Index.from_json(ix.to_json()).score("payroll")[0][0] == "notes/payroll.md")
|
||||
ix.remove("notes/payroll.md")
|
||||
check("H1 BM25 remove() drops a doc's postings", ix.score("payroll") == [])
|
||||
check("H1 recall path is wired (not a stub)",
|
||||
all(callable(getattr(r, n, None))
|
||||
for n in ("recall", "rebuild", "update_note", "load_index", "save_index")))
|
||||
|
||||
# H2 — cache round-trips, enqueue dedups, flush is a no-op on an empty queue (offline-safe).
|
||||
import chorus_queue as q
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
os.environ["CHORUS_STATE_DIR"] = d
|
||||
q.cache_put("a/b.md", b"hello")
|
||||
check("H2 cache round-trip", q.cache_get("a/b.md") == b"hello")
|
||||
q.enqueue("PUT", "http://x/vault/a.md", b"body", {}, idem_key="k1")
|
||||
check("H2 enqueue persists a record", len(q.pending()) == 1)
|
||||
q.enqueue("PUT", "http://x/vault/a.md", b"body", {}, idem_key="k1")
|
||||
check("H2 enqueue dedups by idem_key", len(q.pending()) == 1)
|
||||
with tempfile.TemporaryDirectory() as d2:
|
||||
os.environ["CHORUS_STATE_DIR"] = d2
|
||||
check("H2 flush() returns 0 on an empty queue (no network)", q.flush() == 0)
|
||||
|
||||
# H3 — lock CM + atomic index update are stubs.
|
||||
import chorus_concurrency as c
|
||||
check("H3 auto_owner is stable", c.auto_owner() == c.auto_owner())
|
||||
check("H3 vault_lock returns a context manager (no network until entered)",
|
||||
hasattr(c.vault_lock(), "__enter__") and hasattr(c.vault_lock(), "__exit__"))
|
||||
check("H3 atomic_index_update is wired", callable(c.atomic_index_update))
|
||||
|
||||
# H5 — proposal validation is pure (offline): keeps valid (incl. inbox), drops the rest.
|
||||
import chorus_reflect as rf
|
||||
v, errs = rf.validate([
|
||||
{"title": "Ok", "kind": "person", "confidence": 0.9},
|
||||
{"title": "", "kind": "person"}, # missing title
|
||||
{"title": "Bad", "kind": "bogus"}, # unknown kind
|
||||
{"title": "Weak", "kind": "person", "confidence": 0.2}, # below confidence floor
|
||||
{"title": "Note", "inbox": True}, # inbox needs no kind
|
||||
])
|
||||
check("H5 validate keeps valid (incl. inbox), drops invalid", len(v) == 2 and len(errs) == 3, (len(v), errs))
|
||||
check("H5 classify/preview/apply wired",
|
||||
all(callable(getattr(rf, n, None)) for n in ("classify", "preview", "apply")))
|
||||
|
||||
# Config — owner/endpoint/key resolve from env -> machine-local config.json (no baked
|
||||
# defaults). Env wins per field; the file fills the rest; missing required -> raises.
|
||||
import chorus_config as cfg
|
||||
saved_env = {k: os.environ.get(k) for k in ("CHORUS_KEY", "CHORUS_BASE", "CHORUS_OWNER", "CHORUS_CONFIG")}
|
||||
try:
|
||||
for k in saved_env:
|
||||
os.environ.pop(k, None)
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
cfgfile = os.path.join(d, "config.json")
|
||||
os.environ["CHORUS_CONFIG"] = cfgfile
|
||||
# init writes the placeholder template only when absent.
|
||||
p, created = cfg.init_template()
|
||||
check("config init scaffolds the template", created and os.path.exists(cfgfile))
|
||||
p2, created2 = cfg.init_template()
|
||||
check("config init is idempotent (no clobber)", created2 is False)
|
||||
# write_config persists a filled-in config; load() reads it back.
|
||||
cfg.write_config("Owner Name", "https://obsidian.example.com/", "file-key-xyz")
|
||||
loaded = cfg.load()
|
||||
check("config file round-trips owner/endpoint/key",
|
||||
loaded == {"owner": "Owner Name",
|
||||
"endpoint": "https://obsidian.example.com", # trailing / stripped
|
||||
"key": "file-key-xyz"})
|
||||
# env overrides the file, per field.
|
||||
os.environ["CHORUS_KEY"] = "env-key-123"
|
||||
check("env CHORUS_KEY overrides config", cfg.resolve_key() == "env-key-123")
|
||||
os.environ.pop("CHORUS_KEY", None)
|
||||
# missing required field raises a helpful error.
|
||||
cfg.write_config("Owner Name", "", "")
|
||||
try:
|
||||
cfg.resolve_key()
|
||||
check("missing key raises", False)
|
||||
except RuntimeError:
|
||||
check("missing key raises", True)
|
||||
finally:
|
||||
for k, v in saved_env.items():
|
||||
if v is None:
|
||||
os.environ.pop(k, None)
|
||||
else:
|
||||
os.environ[k] = v
|
||||
|
||||
# M2 — slug collision + link-confidence helpers are implemented.
|
||||
import chorus_quality as qual
|
||||
check("M2 safe_slug disambiguates a collision",
|
||||
qual.safe_slug({"foo"}, "foo") != "foo")
|
||||
check("M2 short/common tokens are not confident links",
|
||||
qual.is_confident_link("API", "we built an api") is False)
|
||||
|
||||
# M3 — doctor module exposes run(); M4 — output envelope is implemented.
|
||||
import chorus_doctor as doc
|
||||
check("M3 doctor exposes run()", hasattr(doc, "run"))
|
||||
import chorus_output as out
|
||||
env = out.envelope("created", {"path": "p.md"})
|
||||
check("M4 envelope shape", env.get("ok") is True and env.get("action") == "created")
|
||||
|
||||
print(f"\n{len(failures)} failure(s)" if failures else "\nall scaffold interface checks passed")
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,360 @@
|
||||
#!/usr/bin/env python3
|
||||
"""vault_lint.py — mechanically assert CHORUS vault invariants. READ-ONLY.
|
||||
|
||||
Catches the recurring drift bugs prose rules can't enforce: folder<->status
|
||||
mismatch, duplicate slugs, wikilinks leaking into frontmatter, duplicate
|
||||
"## Agent Log" headings, stale active projects, aging inbox captures, impossible
|
||||
dates, missing frontmatter, broken source_notes, scope drift, and paths that no
|
||||
route in routing.json permits. Cross-platform: pure Python via chorus.py.
|
||||
|
||||
Exit status: 0 = clean, 1 = violations found, 2 = vault unreachable,
|
||||
3 = vault not bootstrapped (marker missing).
|
||||
|
||||
Config (env overrides):
|
||||
CHORUS_BASE, CHORUS_KEY (via chorus.py)
|
||||
CHORUS_TODAY (default machine date) — pass the conversation's currentDate so
|
||||
stale/aging math uses the same clock the agent writes with.
|
||||
STALE_DAYS (default 30) INBOX_DAYS (default 14) SCOPE_STALE_SESSIONS (default 3)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(SCRIPT_DIR))
|
||||
|
||||
import chorus # noqa: E402
|
||||
import chorus_index as idx_mod # noqa: E402
|
||||
import chorus_links as links # noqa: E402
|
||||
|
||||
TODAY = dt.date.fromisoformat(os.environ.get("CHORUS_TODAY") or dt.date.today().isoformat())
|
||||
STALE_DAYS = int(os.environ.get("STALE_DAYS", "30"))
|
||||
INBOX_DAYS = int(os.environ.get("INBOX_DAYS", "14"))
|
||||
SCOPE_STALE_SESSIONS = int(os.environ.get("SCOPE_STALE_SESSIONS", "3"))
|
||||
LIFECYCLES = ["active", "incubating", "on-hold", "archived"]
|
||||
SKIP = {"README.md", "project-template.md", "decision-template.md"}
|
||||
REQUIRED_FM = ("type", "created")
|
||||
|
||||
violations: list[tuple[str, str]] = []
|
||||
|
||||
|
||||
def flag(check: str, message: str) -> None:
|
||||
violations.append((check, message))
|
||||
|
||||
|
||||
def get(path: str) -> str | None:
|
||||
status, body = chorus.request("GET", chorus.vault_url(path))
|
||||
if status == 404:
|
||||
return None
|
||||
chorus.check(status, body, f"get {path}")
|
||||
return body.decode(errors="replace")
|
||||
|
||||
|
||||
def list_dir(path: str) -> tuple[list[str], list[str]]:
|
||||
if path and not path.endswith("/"):
|
||||
path += "/"
|
||||
body = get(path)
|
||||
if body is None:
|
||||
return [], []
|
||||
try:
|
||||
payload = json.loads(body)
|
||||
except json.JSONDecodeError:
|
||||
return [], []
|
||||
entries = list(payload.get("files", [])) + list(payload.get("folders", []))
|
||||
files = [entry for entry in entries if not entry.endswith("/")]
|
||||
folders = [entry[:-1] for entry in entries if entry.endswith("/")]
|
||||
return files, folders
|
||||
|
||||
|
||||
def walk(prefix: str = ""):
|
||||
files, folders = list_dir(prefix)
|
||||
for filename in files:
|
||||
yield prefix + filename
|
||||
for folder in folders:
|
||||
yield from walk(f"{prefix}{folder}/")
|
||||
|
||||
|
||||
def split_frontmatter(text: str) -> tuple[str, str]:
|
||||
lines = text.splitlines()
|
||||
if not lines or lines[0].strip() != "---":
|
||||
return "", text
|
||||
for index in range(1, len(lines)):
|
||||
if lines[index].strip() == "---":
|
||||
return "\n".join(lines[1:index]), "\n".join(lines[index + 1:])
|
||||
return "", text
|
||||
|
||||
|
||||
def parse_frontmatter(text: str) -> tuple[str, dict[str, object]]:
|
||||
raw, _ = split_frontmatter(text)
|
||||
fields: dict[str, object] = {}
|
||||
for line in raw.splitlines():
|
||||
match = re.match(r"^([A-Za-z_][\w-]*):\s*(.*)$", line)
|
||||
if not match:
|
||||
continue
|
||||
value: object = match.group(2).strip().strip('"').strip("'")
|
||||
if isinstance(value, str) and value.startswith("[") and value.endswith("]"):
|
||||
value = [part.strip().strip('"').strip("'") for part in value[1:-1].split(",") if part.strip()]
|
||||
fields[match.group(1)] = value
|
||||
return raw, fields
|
||||
|
||||
|
||||
def parse_date(value: object) -> dt.date | None:
|
||||
match = re.match(r"(\d{4}-\d{2}-\d{2})", str(value or ""))
|
||||
if not match:
|
||||
return None
|
||||
try:
|
||||
return dt.date.fromisoformat(match.group(1))
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def as_list(value: object) -> list[object]:
|
||||
if value in (None, ""):
|
||||
return []
|
||||
return value if isinstance(value, list) else [value]
|
||||
|
||||
|
||||
def fm_field_populated(raw: str, fields: dict[str, object], field: str) -> bool:
|
||||
"""True when `field` exists AND holds a real value. parse_frontmatter is line-based,
|
||||
so a block-style list (`tags:\n - x`) parses as "" — check the raw text for items
|
||||
before calling the field empty."""
|
||||
value = fields.get(field)
|
||||
if isinstance(value, list):
|
||||
return bool(value)
|
||||
if str(value or "").strip():
|
||||
return True
|
||||
lines = raw.splitlines()
|
||||
for index, line in enumerate(lines):
|
||||
if re.match(rf"^{re.escape(field)}:\s*$", line):
|
||||
return index + 1 < len(lines) and bool(re.match(r"^\s+-\s*\S", lines[index + 1]))
|
||||
return False
|
||||
|
||||
|
||||
def route_matchers():
|
||||
routing = json.loads((SCRIPT_DIR / "routing.json").read_text(encoding="utf-8"))
|
||||
routes = [(item["id"], re.compile(item["pattern"])) for item in routing.get("routes", [])]
|
||||
retired = [(re.compile(item["pattern"]), item.get("replacement", "")) for item in routing.get("retired", [])]
|
||||
return routes, retired
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
if get("_agent/chorus-vault.md") is None and get("_agent/echo-vault.md") is None:
|
||||
print("vault-lint: marker missing — vault not bootstrapped (run bootstrap.py).", file=sys.stderr)
|
||||
return 3
|
||||
except Exception as exc:
|
||||
print(f"vault-lint: vault unreachable ({exc}).", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
try:
|
||||
routes, retired = route_matchers()
|
||||
except Exception as exc:
|
||||
routes, retired = [], []
|
||||
flag("routing-manifest", f"could not load routing.json ({exc}) — path checks skipped")
|
||||
|
||||
all_files = list(walk())
|
||||
md_files = [p for p in all_files if p.endswith(".md")]
|
||||
md_set = set(md_files)
|
||||
# Fetch every markdown note ONCE, concurrently; all per-note checks below read from
|
||||
# this shared cache instead of issuing a fresh GET per file per section.
|
||||
texts = chorus.read_many(md_files)
|
||||
|
||||
# Path membership + retired-path detection
|
||||
for path in all_files:
|
||||
if routes and not any(rx.match(path) for _, rx in routes):
|
||||
replacement = next((repl for rx, repl in retired if rx.match(path)), None)
|
||||
if replacement is not None:
|
||||
flag("retired-path", f"{path}: retired location — should be {replacement}")
|
||||
else:
|
||||
flag("unknown-path", f"{path}: matches no route in routing.json")
|
||||
|
||||
# Per-note frontmatter checks
|
||||
template_re = re.compile(r"(^|/)(templates/|.*-template\.md$)")
|
||||
for path in all_files:
|
||||
base = path.rsplit("/", 1)[-1]
|
||||
if base in SKIP or template_re.search(path) or not path.endswith(".md"):
|
||||
continue
|
||||
text = texts.get(path) or ""
|
||||
raw, fm = parse_frontmatter(text)
|
||||
if "[[" in raw:
|
||||
flag("frontmatter-wikilink", f"{path}: '[[...]]' inside frontmatter")
|
||||
missing = [k for k in REQUIRED_FM if not str(fm.get(k, "")).strip()]
|
||||
if fm and missing:
|
||||
flag("missing-frontmatter", f"{path}: missing {', '.join(missing)}")
|
||||
# Kind-scoped completeness: entity notes must classify (status + tags). The
|
||||
# per-kind requirement map lives in chorus_index (KIND_REQUIRED_FM) so capture's
|
||||
# defaults, this check, and sweep's backfill all read the same source of truth.
|
||||
kind = idx_mod.kind_for_path(path)
|
||||
if fm and kind:
|
||||
for field in idx_mod.KIND_REQUIRED_FM.get(kind, ()):
|
||||
if not fm_field_populated(raw, fm, field):
|
||||
what = f"missing {field}" if field not in fm else f"empty {field}"
|
||||
flag("incomplete-frontmatter", f"{path}: {what}")
|
||||
created, updated = parse_date(fm.get("created")), parse_date(fm.get("updated"))
|
||||
if created and updated and updated < created:
|
||||
flag("date-order", f"{path}: updated {updated} is before created {created}")
|
||||
if updated and updated > TODAY:
|
||||
flag("future-date", f"{path}: updated {updated} is in the future (today {TODAY})")
|
||||
for source_note in as_list(fm.get("source_notes")):
|
||||
if "[[" in str(source_note):
|
||||
flag("source-notes-wikilink", f"{path}: source_notes contains a wikilink '{source_note}'")
|
||||
|
||||
# Projects: folder<->status, stale active, duplicate slugs
|
||||
slug_homes: dict[str, list[str]] = {}
|
||||
for lifecycle in LIFECYCLES:
|
||||
files, _ = list_dir(f"projects/{lifecycle}")
|
||||
for filename in files:
|
||||
if filename in SKIP or not filename.endswith(".md"):
|
||||
continue
|
||||
slug = filename[:-3]
|
||||
slug_homes.setdefault(slug, []).append(lifecycle)
|
||||
text = texts.get(f"projects/{lifecycle}/{filename}") or ""
|
||||
_, fm = parse_frontmatter(text)
|
||||
status = str(fm.get("status", "")).strip()
|
||||
if status and status != lifecycle:
|
||||
flag("folder/status", f"projects/{lifecycle}/{filename}: status='{status}' but folder='{lifecycle}'")
|
||||
if lifecycle == "active":
|
||||
updated = parse_date(fm.get("updated"))
|
||||
if updated and (TODAY - updated).days > STALE_DAYS:
|
||||
flag("stale-active", f"projects/active/{filename}: updated {updated} ({(TODAY - updated).days}d ago) — consider on-hold/")
|
||||
for slug, homes in slug_homes.items():
|
||||
if len(homes) > 1:
|
||||
flag("duplicate-slug", f"'{slug}' exists in {', '.join(homes)}")
|
||||
|
||||
# Daily notes: duplicate "## Agent Log" headings
|
||||
for path in all_files:
|
||||
if re.match(r"^journal/daily/.*\.md$", path):
|
||||
text = texts.get(path) or ""
|
||||
count = len(re.findall(r"(?m)^## Agent Log\s*$", text))
|
||||
if count > 1:
|
||||
flag("duplicate-agent-log", f"{path}: {count} '## Agent Log' headings")
|
||||
|
||||
# Inbox: captures aging past INBOX_DAYS
|
||||
inbox = get("inbox/captures/inbox.md") or ""
|
||||
for line in inbox.splitlines():
|
||||
match = re.match(r"^\s*-\s*(\d{4}-\d{2}-\d{2})\b", line)
|
||||
date = parse_date(match.group(1)) if match else None
|
||||
if date and (TODAY - date).days > INBOX_DAYS:
|
||||
flag("aging-inbox", f"inbox capture {date} ({(TODAY - date).days}d): {line.strip()[:80]}")
|
||||
|
||||
# Scope freshness (drift detector)
|
||||
context = get("_agent/context/current-context.md")
|
||||
if context is not None:
|
||||
_, fm = parse_frontmatter(context)
|
||||
scope_updated = parse_date(fm.get("scope_updated"))
|
||||
if scope_updated is None:
|
||||
flag("scope-no-timestamp",
|
||||
"_agent/context/current-context.md: no scope_updated frontmatter — scope drift cannot be detected; add it (bootstrap.py) and switch scope via `chorus.py scope set`")
|
||||
else:
|
||||
since = [
|
||||
path for path in all_files
|
||||
if (m := re.match(r"^_agent/sessions/(\d{4}-\d{2}-\d{2})", path))
|
||||
and (d := parse_date(m.group(1))) and d > scope_updated
|
||||
]
|
||||
if len(since) >= SCOPE_STALE_SESSIONS:
|
||||
flag("scope-stale",
|
||||
f"scope set {scope_updated}; {len(since)} session(s) logged since without a switch — confirm it still reflects current work (or run `chorus.py scope set`)")
|
||||
|
||||
# ---- Graph health: broken wikilinks, orphans, index drift ----------------
|
||||
# (md_files / md_set were built up top; reuse the shared `texts` cache.)
|
||||
basemap: dict[str, str] = {}
|
||||
for p in md_files:
|
||||
basemap.setdefault(idx_mod.slugify(p.rsplit("/", 1)[-1][:-3]), p)
|
||||
skip_link = re.compile(r"(^|/)(templates/|.*-template\.md$)")
|
||||
|
||||
def resolve_link(target: str):
|
||||
target = target.strip()
|
||||
if not target:
|
||||
return None
|
||||
if "/" in target:
|
||||
cand = target if target.endswith(".md") else target + ".md"
|
||||
return cand if cand in md_set else None
|
||||
return basemap.get(idx_mod.slugify(target))
|
||||
|
||||
inbound: set[str] = set()
|
||||
for path in md_files:
|
||||
if path.rsplit("/", 1)[-1] in SKIP or skip_link.search(path):
|
||||
continue
|
||||
text = texts.get(path) or ""
|
||||
_, body = split_frontmatter(text)
|
||||
for t in links.all_wikilinks(body):
|
||||
tp = resolve_link(t)
|
||||
if tp is None:
|
||||
flag("broken-wikilink", f"{path}: [[{t}]] resolves to no note")
|
||||
else:
|
||||
inbound.add(tp)
|
||||
|
||||
def is_indexable(path: str) -> bool:
|
||||
# READMEs and templates are signposts, not entity notes (sweep skips them too).
|
||||
base = path.rsplit("/", 1)[-1]
|
||||
return (idx_mod.kind_for_path(path) is not None
|
||||
and base not in SKIP and not template_re.search(path))
|
||||
|
||||
# orphan: an entity note with no inbound links and an empty ## Related
|
||||
for path in md_files:
|
||||
if not is_indexable(path) or path in inbound:
|
||||
continue
|
||||
text = texts.get(path) or ""
|
||||
if not links.related_targets(text):
|
||||
flag("orphan", f"{path}: no inbound links and empty ## Related ({idx_mod.kind_for_path(path)}) — link it for recall")
|
||||
|
||||
# index drift: entries pointing at gone files, or indexable notes not yet indexed
|
||||
try:
|
||||
index = idx_mod.load()
|
||||
idx_paths = {e.get("path") for e in index.get("entities", {}).values()}
|
||||
for slug, e in index.get("entities", {}).items():
|
||||
if e.get("path") not in md_set:
|
||||
flag("index-stale", f"index entry '{slug}' -> {e.get('path')} no longer exists (run sweep.py)")
|
||||
for path in md_files:
|
||||
if is_indexable(path) and path not in idx_paths:
|
||||
flag("index-missing", f"{path}: indexable note absent from entities.json (run sweep.py)")
|
||||
except Exception as exc:
|
||||
flag("index-error", f"could not check entity index ({exc})")
|
||||
|
||||
if not violations:
|
||||
print("vault-lint: clean — all invariants hold.")
|
||||
return 0
|
||||
|
||||
print(f"vault-lint: {len(violations)} violation(s) found\n")
|
||||
labels = {
|
||||
"folder/status": "Folder <-> status mismatch",
|
||||
"duplicate-slug": "Duplicate slug across lifecycle folders",
|
||||
"frontmatter-wikilink": "Wikilink in frontmatter (breaks reading view)",
|
||||
"duplicate-agent-log": "Duplicate '## Agent Log' heading",
|
||||
"stale-active": f"Stale active project (updated > {STALE_DAYS}d)",
|
||||
"aging-inbox": f"Inbox capture aging (> {INBOX_DAYS}d)",
|
||||
"unknown-path": "Path matches no route in routing.json",
|
||||
"retired-path": "Write to a retired/dead path",
|
||||
"missing-frontmatter": "Missing required frontmatter field",
|
||||
"incomplete-frontmatter": "Incomplete entity frontmatter (status/tags) — sweep.py --apply backfills",
|
||||
"date-order": "updated earlier than created",
|
||||
"future-date": "updated date is in the future",
|
||||
"source-notes-wikilink": "Wikilink in source_notes (must be plain paths)",
|
||||
"routing-manifest": "routing.json problem",
|
||||
"scope-no-timestamp": "current-context has no scope_updated (drift undetectable)",
|
||||
"scope-stale": f"Scope may have drifted (>= {SCOPE_STALE_SESSIONS} sessions since last switch)",
|
||||
"broken-wikilink": "Wikilink resolves to no note (dead link)",
|
||||
"orphan": "Orphan entity note (no inbound links, empty Related)",
|
||||
"index-stale": "Entity-index entry points at a missing file",
|
||||
"index-missing": "Indexable note absent from the entity index",
|
||||
"index-error": "Entity-index check failed",
|
||||
}
|
||||
grouped: dict[str, list[str]] = {}
|
||||
for check, message in violations:
|
||||
grouped.setdefault(check, []).append(message)
|
||||
for check, messages in grouped.items():
|
||||
print(f"## {labels.get(check, check)}")
|
||||
for message in messages:
|
||||
print(f" - {message}")
|
||||
print()
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user