Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| b4e923c2e7 | |||
| 882d21dd96 | |||
| 227e26c8db | |||
| e7792003a7 | |||
| 446cd9a0ff | |||
| 1deef7299e |
@@ -0,0 +1,11 @@
|
|||||||
|
# Only mcp-server/ + the plugin scripts reach the image; keep the context lean.
|
||||||
|
.git
|
||||||
|
*.plugin
|
||||||
|
dist/
|
||||||
|
CODEX/
|
||||||
|
docs/
|
||||||
|
eval/
|
||||||
|
echo-icon*
|
||||||
|
*.pdf
|
||||||
|
*.html
|
||||||
|
__pycache__/
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
name: Build and Push Docker Image
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
build:
|
||||||
|
# Runs on the forgerunner host: bundled Docker CLI + mounted /var/run/docker.sock.
|
||||||
|
runs-on: host
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
|
|
||||||
|
- name: Log in to Gitea Container Registry
|
||||||
|
uses: docker/login-action@v3
|
||||||
|
with:
|
||||||
|
registry: registry.alwisp.com
|
||||||
|
username: ${{ secrets.REGISTRY_USER }}
|
||||||
|
password: ${{ secrets.REGISTRY_TOKEN }}
|
||||||
|
|
||||||
|
- name: Build and Push
|
||||||
|
run: |
|
||||||
|
IMAGE="registry.alwisp.com/${{ gitea.repository }}"
|
||||||
|
GIT_SHA="$(git rev-parse --short HEAD)"
|
||||||
|
COMMIT_COUNT="$(git rev-list --count HEAD)"
|
||||||
|
docker build \
|
||||||
|
--label org.alwisp.git-sha="${{ gitea.sha }}" \
|
||||||
|
--label org.alwisp.version="v2.${COMMIT_COUNT}" \
|
||||||
|
--label org.alwisp.repo="${{ gitea.repository }}" \
|
||||||
|
-t "${IMAGE}:latest" .
|
||||||
|
docker push "${IMAGE}:latest"
|
||||||
|
|
||||||
|
# Dangling-only prune: removes untagged leftovers from previous builds. Never
|
||||||
|
# removes the tagged :latest or any image referenced by a running container.
|
||||||
|
- name: Prune dangling images on host
|
||||||
|
if: always()
|
||||||
|
run: docker image prune -f 2>/dev/null || true
|
||||||
|
|
||||||
|
- name: Trigger PORT redeploy
|
||||||
|
if: success()
|
||||||
|
run: |
|
||||||
|
# Repo is 'echo' but the container is 'echo-mcp' — target it explicitly.
|
||||||
|
curl -fsS -X POST https://port.alwisp.com/hooks/gitea \
|
||||||
|
-H "X-Deploy-Token: ${{ secrets.WEBHOOK_SECRET }}" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"container":"echo-mcp"}' || echo "PORT redeploy trigger failed (non-fatal)"
|
||||||
+172
@@ -1,5 +1,177 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## 2.3.0
|
||||||
|
|
||||||
|
**The index train** (IMPROVEMENT-PLANS #2/#4/#8) — one schema-bump release:
|
||||||
|
recall-index schema 3, entity-index schema 2. Old recall indexes are discarded
|
||||||
|
and rebuilt automatically (sub-second); no vault migration.
|
||||||
|
|
||||||
|
### Changed — the recall index is LOCAL-FIRST
|
||||||
|
|
||||||
|
The live BM25 index moves out of the vault into the machine state dir
|
||||||
|
(`recall-index-<endpoint-hash>.json` under `ECHO_STATE_DIR`). Capture's index
|
||||||
|
upkeep — previously GET-whole-index → PUT-whole-index against the vault under
|
||||||
|
the advisory lock, an O(vault) network round trip per write — is now a
|
||||||
|
zero-network atomic file write with no lock at all. The vault copy at
|
||||||
|
`_agent/index/recall-index.json` becomes a **snapshot**: written by
|
||||||
|
`sweep --apply` and best-effort at `session-end`, read only to seed a fresh
|
||||||
|
machine or to catch up after another client swept (checked at most every
|
||||||
|
`ECHO_RECALL_SYNC_HOURS`, default 24). The read path never PUTs the vault.
|
||||||
|
|
||||||
|
### Added — incremental sweep (`sweep --fast`) + content hashes
|
||||||
|
|
||||||
|
Entity-index entries and recall-index doc meta now carry a 16-char content
|
||||||
|
hash. `sweep --fast` walks the listing (cheap) and fetches only what is new,
|
||||||
|
gone, missing a hash, or in **today's rotating verification shard**
|
||||||
|
(`hash(path) % 7 == weekday` — the whole vault gets verified across a week of
|
||||||
|
fast sweeps while each run stays tiny; `--all-shards` forces everything).
|
||||||
|
Index-only and machine-owned, so no `--apply` gate; deletions are dropped from
|
||||||
|
both indexes. **`load` auto-runs it** when this machine's last sweep is older
|
||||||
|
than `ECHO_FAST_SWEEP_DAYS` (default 7) — edits made directly in Obsidian or by
|
||||||
|
other clients now reach the indexes without anyone remembering maintenance.
|
||||||
|
`doctor` reports index freshness.
|
||||||
|
|
||||||
|
### Added — stemming + alias query expansion
|
||||||
|
|
||||||
|
`echo_stem.py`: a conservative Porter-lite stemmer applied by `tokenize()` at
|
||||||
|
BOTH index and query time, so "deployed"/"deployment"/"deploys" meet at
|
||||||
|
"deploy" and "penalties" finds "penalty" (guards against over-stemming: "sing"
|
||||||
|
is not "s"; the -al/d~s irregulars are deliberately not conflated). At recall
|
||||||
|
time, a query that fuzzy-matches an entity (score ≥ 0.5) folds that entity's
|
||||||
|
title/alias vocabulary into the BM25 query at **half weight** — "turbokappa
|
||||||
|
tuning" finds the Kappa Engine note even though the alias appears in no body.
|
||||||
|
Expansion can only boost documents that actually contain the terms.
|
||||||
|
|
||||||
|
### Eval
|
||||||
|
|
||||||
|
Gold set grows 8 → **14 queries** (6 paraphrases the pre-2.3 lexical index
|
||||||
|
missed). v2.3.0: recall@5 / MRR **1.00 / 1.00**, 3/3 session-journal queries;
|
||||||
|
keyword baseline 0.75 / 0.79 and 0/3. Tests: +3 unit (stemmer families,
|
||||||
|
over-stemming guards, hash), +10 end-to-end (local-first no-snapshot-write,
|
||||||
|
local recallability, stemmed recall, alias expansion, fast-sweep pickup/
|
||||||
|
deletion/hash backfill). All suites — including the offline-queue, ops-api,
|
||||||
|
and MCP server suites — green; test harnesses now isolate `ECHO_STATE_DIR`.
|
||||||
|
|
||||||
|
## 2.2.0
|
||||||
|
|
||||||
|
### Added — echo-mcp: the containerized MCP server
|
||||||
|
|
||||||
|
ECHO's operations are now reachable as **typed MCP tools** from any surface with
|
||||||
|
an MCP connector (Claude Code, CoWork, claude.ai) — no Python-capable shell, no
|
||||||
|
`$ECHO` path resolution, no output re-parsing. New `mcp-server/` + `Dockerfile`
|
||||||
|
in this repo; deployed on ALPHA as the `echo-mcp` container behind
|
||||||
|
`https://echomcp.alwisp.com` (streamable HTTP, stateless JSON, bearer auth,
|
||||||
|
open `/health` for Docker/Kuma).
|
||||||
|
|
||||||
|
- **14 tools** wrapping the 2.1.1 `*_op` cores: `echo_load`, `echo_recall`
|
||||||
|
(score-packed `budget_chars`), `echo_resolve`, `echo_get_note`
|
||||||
|
(`section`/`max_chars`, traversal-guarded), `echo_get_scope`/`echo_set_scope`,
|
||||||
|
`echo_health` (`deep` runs the linter), `echo_capture` (gate as data —
|
||||||
|
`merge_into`/`force` are parameters), `echo_link` (paths OR resolvable names),
|
||||||
|
`echo_append_note`/`echo_patch_note` (invalid-target errors include the note's
|
||||||
|
actual headings), `echo_triage_inbox` (list/preview/apply in one tool),
|
||||||
|
`echo_reflect`, `echo_log_session` (the session-end bundle, heartbeat-last).
|
||||||
|
- **Tool profiles**: `ECHO_MCP_TOOLS=core` exposes only the six daily drivers.
|
||||||
|
- **Vault adjacency**: the container talks to the Obsidian REST API's HTTP
|
||||||
|
binding on the same box (`ECHO_BASE=http://10.2.0.35:27123`) — vault ops stop
|
||||||
|
depending on Cloudflare/NPM/DNS; the public chain only fronts the `echomcp`
|
||||||
|
ingress. Server-side offline queue at `/data`.
|
||||||
|
- **All MCP writes serialize** through the server process; the vault advisory
|
||||||
|
lock still coordinates with CLI clients.
|
||||||
|
- SKILL.md: prefer the `echo_*` tools when present; CLI recipes are the fallback.
|
||||||
|
- New suite `eval/test_mcp_server.py` (skips without the SDK): health/auth/
|
||||||
|
initialize/tools-list + the capture→gate→merge→recall→log_session flow over
|
||||||
|
real streamable HTTP.
|
||||||
|
|
||||||
|
The plugin itself is unchanged apart from the SKILL.md note — the container
|
||||||
|
vendors `skills/echo-memory/scripts/` at build time (one canonical tree).
|
||||||
|
|
||||||
|
## 2.1.1
|
||||||
|
|
||||||
|
### Changed — Phase 0 of the MCP build: every high-level op returns an envelope
|
||||||
|
|
||||||
|
Internal refactor (MCP-SERVER-SPEC §4), no behavior change intended: each
|
||||||
|
high-level operation now has a core `*_op` function that **returns an envelope
|
||||||
|
dict and never prints to stdout** — the transport-agnostic seam the upcoming MCP
|
||||||
|
server (2.2) wraps. The CLI verbs are thin wrappers that print exactly what they
|
||||||
|
printed before and map envelope outcomes to the same exit codes.
|
||||||
|
|
||||||
|
- `echo_ops`: `capture_op` / `resolve_op` / `link_op` — the duplicate gate and
|
||||||
|
offline queueing are now **data** (`action: duplicate-gate` with candidates;
|
||||||
|
`queued: true`), mapped to exit 76 / "queued" text only in the wrapper. The
|
||||||
|
`--json` stdout-swallowing hack is gone: helper chatter from the low-level
|
||||||
|
verbs routes to **stderr** wholesale inside the cores. `capture_op` accepts
|
||||||
|
`body_text=` directly (no temp file needed — the MCP path).
|
||||||
|
- `echo_recall.recall_op` — one structured result backs both the prose and
|
||||||
|
`--json` renderings (the duplicated prose path was deleted).
|
||||||
|
- `echo_triage.list_op` / `route_op`, `echo_reflect.apply_op`,
|
||||||
|
`echo_session.session_end_op` — same pattern; reflect/triage/session now call
|
||||||
|
`capture_op` internally and count gate outcomes from envelopes.
|
||||||
|
- `echo.scope_show_op` / `scope_set_op` / `load_op` (sections + brief digest as
|
||||||
|
data), `echo_doctor.run_op` (checks as a list).
|
||||||
|
|
||||||
|
New suite `eval/test_ops_api.py` asserts the contract for every core: envelope
|
||||||
|
shape AND stdout purity (a stray print would corrupt an MCP response stream).
|
||||||
|
All existing suites pass unchanged.
|
||||||
|
|
||||||
|
## 2.1.0
|
||||||
|
|
||||||
|
The "quick-wins train" from the 2026-07-28 review (`docs/IMPROVEMENT-PLANS.md` #1,
|
||||||
|
#3, #7). Three independently useful changes; no schema or layout changes.
|
||||||
|
|
||||||
|
### Added — `load --brief`: a token-budgeted cold-start digest
|
||||||
|
|
||||||
|
The SessionStart hook injected the full text of six files every session (measured
|
||||||
|
15.9KB and growing without bound — Observations and Scope History never shrink).
|
||||||
|
`load --brief` renders a digest instead: Fact / Pattern rules in full, the last ~10
|
||||||
|
Observations (with a `+N older` note), scope + `scope_updated` + sessions-since,
|
||||||
|
the heartbeat-pointed session log's key sections (Goal / Decisions Made / Open
|
||||||
|
Threads / Suggested Next Step), today's Agent Log lines, and the inbox as a
|
||||||
|
**count + oldest age** — everything the load-time reconcile needs. Over
|
||||||
|
`ECHO_LOAD_BUDGET` (default ~8000 chars), sections trim lowest-priority-first
|
||||||
|
(observations → agent log → session summary) with explicit `(truncated)` markers.
|
||||||
|
The hook now injects the brief form; `/echo-load` and the manual procedure keep the
|
||||||
|
full dump. All `load` reads (plus the sessions listing, which also feeds the
|
||||||
|
heartbeat-absent fallback) are fetched **in parallel** over the pooled connections.
|
||||||
|
|
||||||
|
### Fixed — capture is now offline-durable (the write path's inversion bug)
|
||||||
|
|
||||||
|
Low-level verbs queued on an outage, but the *default* write (`capture`) died: the
|
||||||
|
update path's POST/PATCH and `ensure_daily_log` bypassed the queue entirely, and an
|
||||||
|
unreachable index aborted the whole op. Now: (1) a fully-offline `capture` queues
|
||||||
|
the **whole operation as one semantic record** (title/kind/body/tags/…, plus the
|
||||||
|
capture-time date); `flush` replays it **through `capture`** so create-vs-update,
|
||||||
|
the duplicate gate, and aliasing re-run against the index as it is at replay time —
|
||||||
|
byte-level replay would freeze a decision made blind. (2) A duplicate-gate stop on
|
||||||
|
replay keeps the record, flags it `needs_attention` (surfaced by `flush` and
|
||||||
|
`load`), and does NOT block later records. (3) Mid-capture outages: the update
|
||||||
|
path and `ensure_daily_log` writes ride `safe_request` (accepted edge: a queued
|
||||||
|
agent-log PATCH 400-drops loudly if the daily note never existed — the trace line,
|
||||||
|
not the memory). Same-args offline captures dedupe by idem-key.
|
||||||
|
|
||||||
|
### Added — `session-end`: the one-call session close
|
||||||
|
|
||||||
|
`echo.py session-end <bundle.json> [--apply]` performs the whole end-of-session
|
||||||
|
sequence under ONE lock acquisition, in order: session log PUT → Agent Log line →
|
||||||
|
reflect proposals (via `capture`, gate-aware) → optional `scope set` → **heartbeat
|
||||||
|
LAST as the commit marker** (a part-way failure leaves the previous pointer intact,
|
||||||
|
never dangling). Dry-run by default previews the full plan including the reflect
|
||||||
|
classification. Filename HHMM from `$ECHO_NOW` (else local clock), validated
|
||||||
|
against the canonical session-log regex **before any write**. The Stop hook's nudge
|
||||||
|
now names the command, and recognizes a `session-end` invocation as
|
||||||
|
"already reflected".
|
||||||
|
|
||||||
|
### Fixed — helper-module errors exited 2 instead of their intended codes
|
||||||
|
|
||||||
|
When `echo.py` runs as `__main__`, helper modules' `import echo` creates a twin
|
||||||
|
module whose `EchoError` is a different class object, so `main()`'s handler missed
|
||||||
|
it and the raw traceback leaked. The handler now catches `RuntimeError` (both
|
||||||
|
classes subclass it) and honors `.code`.
|
||||||
|
|
||||||
|
Tests: +19 end-to-end checks (brief-load digest/budget, offline-capture queue /
|
||||||
|
replay-through-capture / gate-on-replay / idem-key dedup, session-end dry-run /
|
||||||
|
apply / ordering / validation-abort). All suites green.
|
||||||
|
|
||||||
## 2.0.0
|
## 2.0.0
|
||||||
|
|
||||||
**Packaging & structure major — nothing about memory behavior changes.** The major
|
**Packaging & structure major — nothing about memory behavior changes.** The major
|
||||||
|
|||||||
+26
@@ -0,0 +1,26 @@
|
|||||||
|
# echo-mcp — containerized MCP server over the ECHO vault (docs/MCP-SERVER-SPEC.md).
|
||||||
|
# LEGACY Dockerfile format on purpose: the git.alwisp.com CI runner has no BuildKit
|
||||||
|
# (no `# syntax=` line, no RUN --mount).
|
||||||
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
COPY mcp-server/requirements.txt /app/requirements.txt
|
||||||
|
RUN pip install --no-cache-dir -r /app/requirements.txt
|
||||||
|
|
||||||
|
# The canonical plugin scripts ARE the server's ops layer — vendored at build time,
|
||||||
|
# never a second source tree.
|
||||||
|
COPY echo-memory.plugin.src/skills/echo-memory/scripts /app/scripts
|
||||||
|
COPY mcp-server/app.py /app/app.py
|
||||||
|
|
||||||
|
ENV ECHO_STATE_DIR=/data \
|
||||||
|
ECHO_MCP_PORT=8765 \
|
||||||
|
PYTHONUNBUFFERED=1
|
||||||
|
VOLUME /data
|
||||||
|
EXPOSE 8765
|
||||||
|
|
||||||
|
# Probe 127.0.0.1, NOT localhost (::1-vs-IPv4 lesson from cpas/memer/breedr).
|
||||||
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
|
||||||
|
CMD python3 -c "import urllib.request;urllib.request.urlopen('http://127.0.0.1:8765/health', timeout=4)" || exit 1
|
||||||
|
|
||||||
|
CMD ["python3", "/app/app.py"]
|
||||||
@@ -1,4 +1,4 @@
|
|||||||
# echo-memory — v2.0.0
|
# echo-memory — v2.3.0
|
||||||
|
|
||||||
Persistent memory for Claude / CoWork sessions via the **ECHO** Obsidian vault, driven over the [Obsidian Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api). The skill makes direct REST calls through a bundled validated client (`scripts/echo.py`). The whole toolchain is **pure Python** (stdlib only), so it runs identically on Windows, macOS, and Linux — no bash, no platform-specific `date`.
|
Persistent memory for Claude / CoWork sessions via the **ECHO** Obsidian vault, driven over the [Obsidian Local REST API](https://github.com/coddingtonbear/obsidian-local-rest-api). The skill makes direct REST calls through a bundled validated client (`scripts/echo.py`). The whole toolchain is **pure Python** (stdlib only), so it runs identically on Windows, macOS, and Linux — no bash, no platform-specific `date`.
|
||||||
|
|
||||||
@@ -403,14 +403,14 @@ If the API returns a connection error, timeout, or `502` (usually Obsidian / the
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Eval metrics (v1.5.1, 2026-07-03)
|
## Eval metrics (v2.3.0, 2026-07-29)
|
||||||
|
|
||||||
From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate.
|
From the credential-free harness (`eval/run_eval.py` against the deterministic mock — no live vault; full detail in `eval/results/latest.json`). Baseline = pre-1.5 behavior: keyword substring ranking over entity notes only, no gate. The gold set grew to **14 queries in 2.3** — six are paraphrases (morphological variants like "penalties in the SLA" vs a note saying "penalty clause") that a purely lexical index misses; the stemmed index answers all of them.
|
||||||
|
|
||||||
| Metric | v1.5.1 | pre-1.5 baseline |
|
| Metric | v2.3.0 | pre-1.5 baseline |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Retrieval recall@5 / MRR (8-query gold set) | **1.00 / 1.00** | 0.75 / 0.75 |
|
| Retrieval recall@5 / MRR (14-query gold set incl. 6 paraphrases) | **1.00 / 1.00** | 0.75 / 0.79 |
|
||||||
| Queries answerable only from sessions/journal | **2/2** | 0/2 (not in corpus) |
|
| Queries answerable only from sessions/journal | **3/3** | 0/3 (not in corpus) |
|
||||||
| Freshness: live note outranks stale archived twin | **yes** | no |
|
| Freshness: live note outranks stale archived twin | **yes** | no |
|
||||||
| Duplicate notes created (3 renamed-entity captures) | **0** (gate blocks) | 3 |
|
| Duplicate notes created (3 renamed-entity captures) | **0** (gate blocks) | 3 |
|
||||||
| Legitimate captures wrongly blocked | **0** | 0 |
|
| Legitimate captures wrongly blocked | **0** | 0 |
|
||||||
@@ -421,6 +421,10 @@ From the credential-free harness (`eval/run_eval.py` against the deterministic m
|
|||||||
|
|
||||||
| Version | Highlights |
|
| Version | Highlights |
|
||||||
|---------|-----------|
|
|---------|-----------|
|
||||||
|
| **2.3.0** | **The index train (recall-index schema 3, entity-index schema 2).** (1) **Local-first recall index** — the live BM25 index moves to the machine state dir (keyed by endpoint); capture's index upkeep becomes a zero-network atomic file write instead of GET/PUT-whole-index under the vault lock; the vault copy is now a snapshot (written by sweep + session-end) that seeds fresh machines. (2) **Incremental sweep** — entity entries carry a content hash; `sweep --fast` fetches only new/gone/changed notes plus a rotating weekday shard (whole vault verified across a week); auto-runs at load when >`ECHO_FAST_SWEEP_DAYS` (7) — Obsidian-side edits now reach the indexes without manual maintenance. (3) **Stemming + alias expansion** — `echo_stem` (Porter-lite, conservative) applied at index+query time, and a query matching an entity's alias folds its title/alias vocabulary in at half weight. Eval gold set grows to 14 queries (6 paraphrases): recall@5 / MRR **1.00 / 1.00**. |
|
||||||
|
| **2.2.0** | **echo-mcp — the containerized MCP server.** ECHO as 14 typed MCP tools from any connector-capable surface (Claude Code, CoWork, claude.ai): `mcp-server/` + Dockerfile in-repo, deployed on ALPHA as `echo-mcp` behind `https://echomcp.alwisp.com` (streamable HTTP, stateless JSON, bearer auth, open `/health`). Talks to the Obsidian REST API directly on the LAN (`10.2.0.35:27123`) — one client round trip per tool call, all vault chatter host-local. Duplicate gate & offline queueing surface as data; `ECHO_MCP_TOOLS=core` trims the surface to six tools; writes serialize server-side. New `eval/test_mcp_server.py` e2e suite. Full spec: `docs/MCP-SERVER-SPEC.md`. |
|
||||||
|
| **2.1.1** | **MCP Phase 0 — return-not-print.** Every high-level op gains a core `*_op` function returning an envelope dict with a stdout-purity guarantee (helper chatter → stderr); CLI verbs become thin wrappers with identical output and exit codes. Duplicate gate and offline queueing become data (`action: duplicate-gate` / `queued: true`). New `eval/test_ops_api.py` contract suite. This is the seam the 2.2 containerized MCP server wraps. |
|
||||||
|
| **2.1.0** | **Quick-wins train: cheaper sessions, durable capture, one-call session end.** (1) **`load --brief`** — a token-budgeted cold-start digest (Fact/Pattern in full, last ~10 Observations, scope+freshness, last session's key sections, Agent-Log lines, inbox *count*; `ECHO_LOAD_BUDGET` default ~8000 chars) now injected by the SessionStart hook, replacing the full six-file dump that grew unboundedly; all `load` reads are fetched in parallel. (2) **Offline capture durability** — `capture` on an unreachable vault queues the *whole operation* as one semantic record; `flush` replays it **through capture** so routing/gate/aliasing re-run against the current index; a gate stop on replay is kept + flagged, never landed blind; `ensure_daily_log`/update-path writes ride the queue too. (3) **`session-end`** — one call (one lock) writes session log → Agent-Log line → reflect proposals → optional scope switch → **heartbeat last as the commit marker**; dry-run by default; `ECHO_NOW` pins the HHMM. Also fixes the `__main__` twin-module trap so helper-module `EchoError`s exit with their intended codes. +19 end-to-end checks. |
|
||||||
| **2.0.0** | **Packaging & structure major — memory behavior unchanged.** Skills-only: the legacy `commands/` directory is deleted (breaking for pre-skills clients; the 1.6.0-verified skills are the sole entry points). Artifact policy: the repo tracks only the `echo-memory.plugin` pointer; the 15 historical versioned zips leave the tree and versioned builds ship as **Gitea releases** (one per `v<version>` tag) from `v2.0.0` on; `.gitignore` blocks `*.plugin` except the pointer. README gains the "packaging for other agent runtimes" port note (build-target rule, never a second source tree). |
|
| **2.0.0** | **Packaging & structure major — memory behavior unchanged.** Skills-only: the legacy `commands/` directory is deleted (breaking for pre-skills clients; the 1.6.0-verified skills are the sole entry points). Artifact policy: the repo tracks only the `echo-memory.plugin` pointer; the 15 historical versioned zips leave the tree and versioned builds ship as **Gitea releases** (one per `v<version>` tag) from `v2.0.0` on; `.gitignore` blocks `*.plugin` except the pointer. README gains the "packaging for other agent runtimes" port note (build-target rule, never a second source tree). |
|
||||||
| **1.6.0** | **Skills-format migration + lint blind-spot fix.** The eight slash commands gain `skills/<name>/SKILL.md` twins (same names, byte-identical bodies; a same-name skill takes precedence, so `commands/` coexists until its 2.0 deletion): per-skill `allowed-tools` end the permission prompts, `disable-model-invocation: true` makes `/echo-sweep` + `/echo-triage` operator-only. `vault_lint` checks retired patterns **before** routes, so a seed README inside a retired tree (`archive/…`) is flagged instead of being absolved by the leaf-README route (+ regression test). `plugin.json` gains `homepage`/`repository`. |
|
| **1.6.0** | **Skills-format migration + lint blind-spot fix.** The eight slash commands gain `skills/<name>/SKILL.md` twins (same names, byte-identical bodies; a same-name skill takes precedence, so `commands/` coexists until its 2.0 deletion): per-skill `allowed-tools` end the permission prompts, `disable-model-invocation: true` makes `/echo-sweep` + `/echo-triage` operator-only. `vault_lint` checks retired patterns **before** routes, so a seed README inside a retired tree (`archive/…`) is flagged instead of being absolved by the leaf-README route (+ regression test). `plugin.json` gains `homepage`/`repository`. |
|
||||||
| **1.5.1** | **Duplicate-gate precision.** Live use caught the gate blocking a decision note because it shared the single vault-common token "echo" with an archived project (min-normalized scoring gives any single-token entity a 1.0 match). New `gate_candidates()` blocks only on **same-kind** candidates (cross-kind name collisions — a decision titled after its project — warn instead), and a **lone shared token blocks only when unique to that entity** (vault df == 1); multi-token overlaps still block. Advisory warnings (`fuzzy_candidates`) unchanged; exit-76/`--merge-into`/`--force` unchanged. +5 tests; verified against the live vault both ways (false positive creates cleanly, true lookalike still gates). |
|
| **1.5.1** | **Duplicate-gate precision.** Live use caught the gate blocking a decision note because it shared the single vault-common token "echo" with an archived project (min-normalized scoring gives any single-token entity a 1.0 match). New `gate_candidates()` blocks only on **same-kind** candidates (cross-kind name collisions — a decision titled after its project — warn instead), and a **lone shared token blocks only when unique to that entity** (vault df == 1); multi-token overlaps still block. Advisory warnings (`fuzzy_candidates`) unchanged; exit-76/`--merge-into`/`--force` unchanged. +5 tests; verified against the live vault both ways (false positive creates cleanly, true lookalike still gates). |
|
||||||
|
|||||||
+3
-2
@@ -35,8 +35,9 @@ coexisting (see `TODO-1.6.md`). 2.0 finishes it:
|
|||||||
|
|
||||||
- [x] **Delete the legacy `commands/` directory** — **done 2.0.0** (1.6.0's install
|
- [x] **Delete the legacy `commands/` directory** — **done 2.0.0** (1.6.0's install
|
||||||
verification on both surfaces cleared the gate; skills had precedence anyway).
|
verification on both surfaces cleared the gate; skills had precedence anyway).
|
||||||
- [ ] Re-verify desktop + CoWork installs of the skills-only artifact (operator step,
|
- [x] Re-verify desktop + CoWork installs of the skills-only artifact — **verified
|
||||||
post-2.0.0-install).
|
2026-07-28** on the baked 2.0.0 build: installs cleanly, all eight skills
|
||||||
|
register and work with no legacy `commands/` present.
|
||||||
- [x] Update SKILL.md / README / command docs that reference `commands/` — **done 2.0.0**.
|
- [x] Update SKILL.md / README / command docs that reference `commands/` — **done 2.0.0**.
|
||||||
|
|
||||||
## 3. Codex packaging (from MAINTENANCE › Canonical source tree)
|
## 3. Codex packaging (from MAINTENANCE › Canonical source tree)
|
||||||
|
|||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# echo-mcp — PORT deploy manifest (ALPHA / br0). See docs/MCP-SERVER-SPEC.md §8.
|
||||||
|
# Container name deliberately differs from the repo (echo): the repo ships the
|
||||||
|
# plugin AND this server; only the server is a container.
|
||||||
|
name: echo-mcp
|
||||||
|
image: registry.alwisp.com/jason/echo:latest
|
||||||
|
network: br0
|
||||||
|
ip: auto
|
||||||
|
ports: [] # br0 static IP — app serves :8765 directly
|
||||||
|
volumes:
|
||||||
|
- /mnt/user/appdata/echo-mcp:/data # outbox + read cache (+ backups/history, v1.1)
|
||||||
|
env:
|
||||||
|
# Vault adjacency: the Obsidian Local REST API's HTTP binding on the same box —
|
||||||
|
# never the public echoapi.alwisp.com hairpin (spec §8; cleartext stays on the LAN).
|
||||||
|
ECHO_BASE: http://10.2.0.35:27123
|
||||||
|
ECHO_OWNER: Jason Stedwell
|
||||||
|
ECHO_STATE_DIR: /data
|
||||||
|
ECHO_MCP_PORT: "8765"
|
||||||
|
ECHO_MCP_TOOLS: full
|
||||||
|
# Secrets from the PORT secret store — values never appear in this file or chat.
|
||||||
|
ECHO_KEY: SECRET:echo-vault-key
|
||||||
|
ECHO_MCP_TOKEN: SECRET:echo-mcp-token
|
||||||
|
health_check_path: /health
|
||||||
|
health_check_port: 8765
|
||||||
|
webui: https://echomcp.alwisp.com
|
||||||
|
proxy:
|
||||||
|
forward_port: 8765
|
||||||
|
forward_scheme: http
|
||||||
|
websockets: true
|
||||||
@@ -345,8 +345,9 @@ manifest-verification pass).
|
|||||||
| Release | Items | Rationale |
|
| Release | Items | Rationale |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **1.6.0** | TODO-1.6: skills-format migration (+ per-skill `allowed-tools`, `disable-model-invocation` on write-heavy entries, `$ECHO` block dedupe) · lint blind-spot fix (retired-dir/leaf-README) · plugin.json `homepage` | Restructures the command/SKILL files later work edits; allowed-tools removes prompt friction for every verb added below. Vault link repointing (TODO-1.6 §3) is vault data, not plugin code — any session. |
|
| **1.6.0** | TODO-1.6: skills-format migration (+ per-skill `allowed-tools`, `disable-model-invocation` on write-heavy entries, `$ECHO` block dedupe) · lint blind-spot fix (retired-dir/leaf-README) · plugin.json `homepage` | Restructures the command/SKILL files later work edits; allowed-tools removes prompt friction for every verb added below. Vault link repointing (TODO-1.6 §3) is vault data, not plugin code — any session. |
|
||||||
| **1.6.x quick wins** | #1 brief load · #3 offline capture · #7 session-end (+ return-not-print refactor, MCP spec Phase 0, if it fits) | Small, correctness/efficiency, no schema changes. #7 unblocks the MCP tool. Users get these without a reinstall — don't gate them behind 2.0. |
|
| **1.6.0 — SHIPPED 2026-07-28** | Skills migration + lint fix + homepage | Verified on desktop + CoWork. |
|
||||||
| **2.0** | ROADMAP-2.0 unchanged: delete legacy `commands/` · artifact policy (Gitea releases, gitignore, tags) · Codex note · README layout | Packaging only, no feature riders — a bad feature must never force reverting a packaging break. |
|
| **2.0.0 — SHIPPED 2026-07-28** | ROADMAP-2.0: commands/ deleted · release-based artifacts · gitignore · Codex note | Packaging-only major; skills-only install re-verified. |
|
||||||
| **2.1** | MCP server (containerized, `echomcp.alwisp.com`) — `docs/MCP-SERVER-SPEC.md` | Needs #7 + Phase 0. Lands after 2.0 so its docs target one command format and the Gitea release channel exists. Deployed via CI + PORT, not the plugin manifest. |
|
| **2.1.0 quick wins — SHIPPED 2026-07-28** | #1 brief load · #3 offline capture · #7 session-end | All three landed with +19 e2e checks; see CHANGELOG 2.1.0. The MCP spec's Phase 0 (return-not-print) did NOT ride along — it's the first phase of the MCP build session. |
|
||||||
| **2.2 index train** | #2 local-first recall (CLI-path half) · #4 incremental sweep · #8 stemming+expansion | One schema-bump train; all touch the same index/sweep loop. The container already keeps its index warm in memory (spec §7), so #2 here covers the CLI fallback path; #4's fast sweep also becomes the container's background job. |
|
| **2.2** | MCP server (containerized, `echomcp.alwisp.com`) — `docs/MCP-SERVER-SPEC.md` | Needs Phase 0 (return-not-print refactor, first step of the build session). #7 session-end shipped, so `echo_log_session` wraps a real verb. Deployed via CI + PORT, not the plugin manifest. |
|
||||||
| **2.3 memory train** | #5 lifecycle/decay · #6 supersession · #9 resolve-miss learning | Capability tier; #6 after #5 (status vocabulary). |
|
| **2.3 index train** | #2 local-first recall (CLI-path half) · #4 incremental sweep · #8 stemming+expansion | One schema-bump train; all touch the same index/sweep loop. The container already keeps its index warm in memory (spec §7), so #2 here covers the CLI fallback path; #4's fast sweep also becomes the container's background job. |
|
||||||
|
| **2.4 memory train** | #5 lifecycle/decay · #6 supersession · #9 resolve-miss learning | Capability tier; #6 after #5 (status vocabulary). |
|
||||||
|
|||||||
+12
-7
@@ -1,9 +1,9 @@
|
|||||||
# echo-mcp — MCP Server Build Spec (containerized)
|
# echo-mcp — MCP Server Build Spec (containerized)
|
||||||
|
|
||||||
> Status: **spec for a future build session** (written 2026-07-28 against v1.5.1;
|
> Status: **BUILT — shipped as 2.2.0, 2026-07-28** (`mcp-server/app.py`, 14 tools —
|
||||||
> revised same day: **remote container architecture**, operator decision — heavy
|
> the count below saying 13 undercounted the append/patch pair; e2e suite
|
||||||
> lifting belongs in a deployed container, not on any one machine).
|
> `eval/test_mcp_server.py`). This document remains the design record; §7.2 is the
|
||||||
> Companion plan for the other nine review items: `docs/IMPROVEMENT-PLANS.md`.
|
> live v1.1 backlog. Companion plan: `docs/IMPROVEMENT-PLANS.md`.
|
||||||
>
|
>
|
||||||
> **Prerequisites before starting this build:**
|
> **Prerequisites before starting this build:**
|
||||||
> 1. The `session-end` verb (IMPROVEMENT-PLANS #7) — the MCP tool wraps it.
|
> 1. The `session-end` verb (IMPROVEMENT-PLANS #7) — the MCP tool wraps it.
|
||||||
@@ -75,7 +75,7 @@ monitor, PORT deploy/rollback. Server down ⇒ exactly today's behavior.
|
|||||||
| Tool prefix | `echo_` | Namespace safety next to other servers. |
|
| Tool prefix | `echo_` | Namespace safety next to other servers. |
|
||||||
| Result shape | `structuredContent` (the `echo_output.envelope` dict) + a 1–3 line human text block | Envelope shape already exists. |
|
| Result shape | `structuredContent` (the `echo_output.envelope` dict) + a 1–3 line human text block | Envelope shape already exists. |
|
||||||
| Statefulness | Stateless protocol; warm in-process caches | See §7. |
|
| Statefulness | Stateless protocol; warm in-process caches | See §7. |
|
||||||
| Version target | **2.1** (after the 1.6.x train and the 2.0 packaging major) | Needs #7 + Phase 0; docs then target the single post-2.0 command format. |
|
| Version target | **2.2** — ALL prerequisites shipped 2026-07-28 (`session-end` in 2.1.0; **Phase 0 in 2.1.1**, contract-tested by `eval/test_ops_api.py`) | The build session starts directly at §5 (the server app). |
|
||||||
| Local stdio variant | **Dropped for v1** (documented, not built) | The CLI/skill fallback covers the no-server case; two transports = two test matrices for little gain. Phase 0 keeps the door open — the `*_op` core is transport-agnostic. |
|
| Local stdio variant | **Dropped for v1** (documented, not built) | The CLI/skill fallback covers the no-server case; two transports = two test matrices for little gain. Phase 0 keeps the door open — the `*_op` core is transport-agnostic. |
|
||||||
| Result budgets | Every read tool takes an explicit size budget (`budget_chars` / `max_chars`) and truncates with a marker + "call X for more" | The single biggest token lever: one right-sized answer instead of follow-up fetches. |
|
| Result budgets | Every read tool takes an explicit size budget (`budget_chars` / `max_chars`) and truncates with a marker + "call X for more" | The single biggest token lever: one right-sized answer instead of follow-up fetches. |
|
||||||
| Tool profiles | `ECHO_MCP_TOOLS=core\|full` env: `core` exposes only load/recall/capture/triage_inbox/log_session/health | Tool schemas cost context on every surface that lists them; the claude.ai connector doesn't need the escape hatches. Desktop registers `full`. |
|
| Tool profiles | `ECHO_MCP_TOOLS=core\|full` env: `core` exposes only load/recall/capture/triage_inbox/log_session/health | Tool schemas cost context on every surface that lists them; the claude.ai connector doesn't need the escape hatches. Desktop registers `full`. |
|
||||||
@@ -337,8 +337,13 @@ that division of labor is permanent, not a v1 scope cut).
|
|||||||
`https://echomcp.alwisp.com/mcp` with the `Authorization: Bearer` header
|
`https://echomcp.alwisp.com/mcp` with the `Authorization: Bearer` header
|
||||||
(project or user scope). No plugin-manifest coupling; the plugin does **not**
|
(project or user scope). No plugin-manifest coupling; the plugin does **not**
|
||||||
register the server.
|
register the server.
|
||||||
- **claude.ai**: custom connector with the same URL + token — ECHO memory from
|
- **claude.ai**: **blocked in v1 (found at 2.2.0 deploy)** — claude.ai custom
|
||||||
the browser/phone, a surface the plugin could never reach.
|
connectors run the MCP OAuth flow (Dynamic Client Registration against the
|
||||||
|
server) and offer no custom-header option, so a bearer-only server cannot
|
||||||
|
connect. Fix = a minimal single-user OAuth layer on echo-mcp (SDK auth
|
||||||
|
interface: DCR + one-time operator approval + token issuance) — **v2.2.x
|
||||||
|
backlog, alongside the nightly vault backup**. Token-in-URL workarounds are
|
||||||
|
rejected (credentials leak into logs).
|
||||||
- **SKILL.md** gains: *"When the `echo_*` MCP tools are available, prefer them
|
- **SKILL.md** gains: *"When the `echo_*` MCP tools are available, prefer them
|
||||||
for every operation they cover; the `$ECHO` CLI recipes are the fallback for
|
for every operation they cover; the `$ECHO` CLI recipes are the fallback for
|
||||||
hosts without the connector or when the server is unreachable."* Procedures
|
hosts without the connector or when the server is unreachable."* Procedures
|
||||||
|
|||||||
Binary file not shown.
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "echo-memory",
|
"name": "echo-memory",
|
||||||
"version": "2.0.0",
|
"version": "2.3.0",
|
||||||
"homepage": "https://git.alwisp.com/jason/echo",
|
"homepage": "https://git.alwisp.com/jason/echo",
|
||||||
"repository": "https://git.alwisp.com/jason/echo",
|
"repository": "https://git.alwisp.com/jason/echo",
|
||||||
"description": "Persistent memory via the ECHO Obsidian vault over the Obsidian Local REST API. Cross-platform Python client: one-call capture/resolve/recall/link/triage over an entity index, hybrid BM25 + graph recall spanning entities + sessions/journal (recency/status-aware), a pre-write duplicate gate, complete-frontmatter capture, session hooks that self-fire load/reflect, offline write-ahead queue, lock-guarded concurrency, linter-enforced routing, and /echo-* commands.",
|
"description": "Persistent memory via the ECHO Obsidian vault over the Obsidian Local REST API. Cross-platform Python client: one-call capture/resolve/recall/link/triage over an entity index, hybrid BM25 + graph recall spanning entities + sessions/journal (recency/status-aware), a pre-write duplicate gate, complete-frontmatter capture, session hooks that self-fire load/reflect, offline write-ahead queue, lock-guarded concurrency, linter-enforced routing, and /echo-* commands.",
|
||||||
|
|||||||
@@ -52,6 +52,19 @@ Executable logic ships under `scripts/` — **pure Python**, so the whole toolch
|
|||||||
- `scripts/check_routing.py` — verifies the routing docs stay in sync with `routing.json` (dev/CI; offline)
|
- `scripts/check_routing.py` — verifies the routing docs stay in sync with `routing.json` (dev/CI; offline)
|
||||||
- `scripts/bootstrap.py` / `scripts/migrate.py` — deterministic vault setup/repair and schema migration
|
- `scripts/bootstrap.py` / `scripts/migrate.py` — deterministic vault setup/repair and schema migration
|
||||||
|
|
||||||
|
## MCP tools first (when the echo-mcp connector is available)
|
||||||
|
|
||||||
|
When this session has the **`echo_*` MCP tools** (the deployed echo-mcp server —
|
||||||
|
`echo_load`, `echo_recall`, `echo_resolve`, `echo_get_note`, `echo_get_scope`/`echo_set_scope`,
|
||||||
|
`echo_health`, `echo_capture`, `echo_link`, `echo_append_note`/`echo_patch_note`,
|
||||||
|
`echo_triage_inbox`, `echo_reflect`, `echo_log_session`), **prefer them over the CLI
|
||||||
|
for every operation they cover** — typed calls, structured results, no path
|
||||||
|
resolution or quoting. The procedures in this skill (reconcile at load, search-first,
|
||||||
|
scope discipline, third person, preview-before-apply) are unchanged and apply to both
|
||||||
|
surfaces. A `duplicate-gate` tool result is the same contract as capture exit 76:
|
||||||
|
`merge_into` or confirmed `force`, never a blind retry. The CLI recipes below are the
|
||||||
|
fallback for hosts without the connector or when the server is unreachable.
|
||||||
|
|
||||||
## Bundled Tooling (prefer over raw curl)
|
## Bundled Tooling (prefer over raw curl)
|
||||||
|
|
||||||
All paths below are under `${CLAUDE_PLUGIN_ROOT}/skills/echo-memory/`. **Invoke with `python3`** (on Windows where that isn't on PATH, use `python` or `py -3`).
|
All paths below are under `${CLAUDE_PLUGIN_ROOT}/skills/echo-memory/`. **Invoke with `python3`** (on Windows where that isn't on PATH, use `python` or `py -3`).
|
||||||
@@ -140,7 +153,7 @@ Load at the start of any substantive conversation — anything beyond a single q
|
|||||||
|
|
||||||
### Loading procedure
|
### Loading procedure
|
||||||
|
|
||||||
**Run `python3 "$ECHO" load`** — one call that performs the six orientation reads below and prints each labeled section (404s on today's note / inbox show as absent, not errors; an absent marker is flagged). This replaces issuing six separate GETs by hand. The table documents what `load` reads and why:
|
**Run `python3 "$ECHO" load`** — one call that performs the six orientation reads below (fetched in parallel) and prints each labeled section (404s on today's note / inbox show as absent, not errors; an absent marker is flagged). This replaces issuing six separate GETs by hand. **`load --brief`** renders a token-budgeted digest instead — Fact/Pattern rules in full, the last ~10 Observations, scope + freshness, the last session's key sections, today's Agent Log lines, and the inbox as a *count* (`ECHO_LOAD_BUDGET`, default ~8000 chars, trims lowest-priority sections first). The SessionStart hook injects the brief form; `/echo-load` and this procedure use the full form. The table documents what `load` reads and why:
|
||||||
|
|
||||||
| # | GET | Notes |
|
| # | GET | Notes |
|
||||||
|---|-----|-------|
|
|---|-----|-------|
|
||||||
@@ -345,6 +358,8 @@ Run it after `migrate.py` (which handles structural/schema changes) — or any t
|
|||||||
|
|
||||||
The pass is cheap and pays for itself by catching drift before it requires a reorg. Write the findings as a digest; act on them only with the operator's go-ahead.
|
The pass is cheap and pays for itself by catching drift before it requires a reorg. Write the findings as a digest; act on them only with the operator's go-ahead.
|
||||||
|
|
||||||
|
**Incremental maintenance (2.3).** `sweep.py --fast` is the cheap, index-only mode: it fetches just what's new, gone, or changed (content hashes) plus a rotating daily verification shard, so edits made directly in Obsidian or by other clients reach the entity + recall indexes without a full pass. It runs **automatically at load** when this machine's last sweep is older than `ECHO_FAST_SWEEP_DAYS` (default 7) — you rarely need to invoke it. The recall index itself is **local-first**: capture maintains a machine-local copy with zero vault round-trips; the vault's `recall-index.json` is a snapshot (sweep/session-end) used to seed fresh machines.
|
||||||
|
|
||||||
**Performance.** Full-vault scripts (`sweep.py`, `vault_lint.py`, recall rebuild) are connection-pooled (keep-alive) and read the whole vault concurrently via `echo.read_many`, so they finish in about a second on a few-hundred-note vault instead of timing out on hundreds of serial TLS handshakes. Tune with `ECHO_WORKERS` (default 8, bulk-read concurrency) and `ECHO_TIMEOUT` (default 30s, per-request). For a vault large enough that even the concurrent pass nears the tool timeout, run the script with the harness's background-execution option rather than blocking the turn.
|
**Performance.** Full-vault scripts (`sweep.py`, `vault_lint.py`, recall rebuild) are connection-pooled (keep-alive) and read the whole vault concurrently via `echo.read_many`, so they finish in about a second on a few-hundred-note vault instead of timing out on hundreds of serial TLS handshakes. Tune with `ECHO_WORKERS` (default 8, bulk-read concurrency) and `ECHO_TIMEOUT` (default 30s, per-request). For a vault large enough that even the concurrent pass nears the tool timeout, run the script with the harness's background-execution option rather than blocking the turn.
|
||||||
|
|
||||||
## Daily Note — Agent Log
|
## Daily Note — Agent Log
|
||||||
@@ -405,25 +420,40 @@ At the end of substantive conversations (ones that produced decisions, artifacts
|
|||||||
|
|
||||||
**Filename format is canonical: `_agent/sessions/YYYY-MM-DD-HHMM-<slug>.md`.** Always include the four-digit local-time HHMM component — it makes filenames lexically sort in true chronological order, which is what Step 3 of loading relies on. Older session logs without the HHMM part exist; leave them alone, but every new one must use the full form.
|
**Filename format is canonical: `_agent/sessions/YYYY-MM-DD-HHMM-<slug>.md`.** Always include the four-digit local-time HHMM component — it makes filenames lexically sort in true chronological order, which is what Step 3 of loading relies on. Older session logs without the HHMM part exist; leave them alone, but every new one must use the full form.
|
||||||
|
|
||||||
Write the body (see `references/session-log-template.md`) to a file with the Write tool, then:
|
**Preferred — one call (`session-end`).** Write a bundle JSON with the Write tool and run it; it performs the whole session-end sequence under one lock, in order, with the **heartbeat written last as the commit marker** (a failure part-way leaves the previous pointer intact):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "slug": "<kebab-slug>",
|
||||||
|
"log_body": "<full session-log markdown, frontmatter included — see references/session-log-template.md>",
|
||||||
|
"agent_log_line": "- <currentDate>: <one-line summary>",
|
||||||
|
"scope": "<new scope text — omit to leave unchanged>",
|
||||||
|
"reflect": [ { "title": "…", "kind": "…", "body": "…", "confidence": 0.9 } ] }
|
||||||
|
```
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 "$ECHO" session-end bundle.json # dry-run: previews the full plan
|
||||||
|
ECHO_NOW=<HHMM> python3 "$ECHO" session-end bundle.json --apply
|
||||||
|
```
|
||||||
|
|
||||||
|
`--apply` writes: session log → Agent Log line → reflect proposals (via `capture`, gate-aware) → optional scope switch → heartbeat. Set `ECHO_NOW` to the conversation's local HHMM so the filename sorts truthfully. Reflect proposals still follow the reflect contract — preview with the dry-run and apply only with the operator's go-ahead. Offline, every step queues durably.
|
||||||
|
|
||||||
|
**Manual fallback** (only if `session-end` is unavailable) — the same sequence by hand:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 "$ECHO" put _agent/sessions/<currentDate>-HHMM-<slug>.md <bodyfile>
|
python3 "$ECHO" put _agent/sessions/<currentDate>-HHMM-<slug>.md <bodyfile>
|
||||||
```
|
```
|
||||||
|
|
||||||
Then add a one-line entry to today's daily note via the **Daily Note — Agent Log** procedure above.
|
Then add a one-line entry to today's daily note via the **Daily Note — Agent Log** procedure above, and finally update the heartbeat pointer so the next session can orient in one read (a pointer nobody writes is a pointer nobody can read) — a single line, `<session-log-path> @ <ISO-timestamp>`:
|
||||||
|
|
||||||
Finally, update the heartbeat pointer so the next session can orient in one read (this closes the loop with loading-procedure Step 4 — a pointer nobody writes is a pointer nobody can read). It is a single line, `<session-log-path> @ <ISO-timestamp>`; write it to a file and PUT it:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 "$ECHO" put _agent/heartbeat/last-session.md <bodyfile>
|
python3 "$ECHO" put _agent/heartbeat/last-session.md <bodyfile>
|
||||||
```
|
```
|
||||||
|
|
||||||
`last-session.md` is overwritten (PUT) each session end — never appended, so it can't grow or duplicate.
|
`last-session.md` is overwritten (PUT) each session end — never appended, so it can't grow or duplicate. Keep the heartbeat write **last** in the manual sequence too.
|
||||||
|
|
||||||
## Vault Unreachable
|
## Vault Unreachable
|
||||||
|
|
||||||
If the API returns a connection error, timeout, or `502`, tell the operator once that the memory vault is unreachable (a `502` usually means Obsidian/the REST plugin is not running on the backend), then proceed without memory. Do not retry repeatedly.
|
If the API returns a connection error, timeout, or `502`, tell the operator once that the memory vault is unreachable (a `502` usually means Obsidian/the REST plugin is not running on the backend), then proceed — reads degrade to the last-known-good cache at load, and **writes queue durably**: the low-level verbs queue the request, and `capture` queues the whole operation as one semantic record that replays *through capture* (routing + duplicate gate re-run against the index at replay time) on the next reachable session. A queued capture that stops at the duplicate gate on replay is kept and flagged (`flush` lists it) rather than landed blind. Do not retry repeatedly.
|
||||||
|
|
||||||
## Style Rules
|
## Style Rules
|
||||||
|
|
||||||
|
|||||||
@@ -549,13 +549,12 @@ def extract_heading(markdown: str, heading: str) -> str:
|
|||||||
return "\n".join(out).strip()
|
return "\n".join(out).strip()
|
||||||
|
|
||||||
|
|
||||||
def cmd_scope(subcommand: str, text: str | None = None, as_json: bool = False) -> int:
|
def scope_show_op() -> dict:
|
||||||
|
"""Core scope-show (Phase 0): active scope + freshness + sessions-since, as data."""
|
||||||
path = "_agent/context/current-context.md"
|
path = "_agent/context/current-context.md"
|
||||||
status, body = request("GET", vault_url(path))
|
status, body = request("GET", vault_url(path))
|
||||||
check(status, body, f"scope {subcommand}")
|
check(status, body, "scope show")
|
||||||
current = body.decode(errors="replace")
|
current = body.decode(errors="replace")
|
||||||
|
|
||||||
if subcommand == "show":
|
|
||||||
scope_text = extract_heading(current, "Scope")
|
scope_text = extract_heading(current, "Scope")
|
||||||
scope_updated = ""
|
scope_updated = ""
|
||||||
for line in current.splitlines():
|
for line in current.splitlines():
|
||||||
@@ -572,23 +571,22 @@ def cmd_scope(subcommand: str, text: str | None = None, as_json: bool = False) -
|
|||||||
sessions_since = sum(1 for f in files if f.endswith(".md") and f[:10] > scope_updated)
|
sessions_since = sum(1 for f in files if f.endswith(".md") and f[:10] > scope_updated)
|
||||||
except json.JSONDecodeError:
|
except json.JSONDecodeError:
|
||||||
pass
|
pass
|
||||||
if as_json:
|
return {"ok": True, "action": "scope-show", "scope": scope_text,
|
||||||
print(json.dumps({"ok": True, "action": "scope-show", "scope": scope_text,
|
"scope_updated": scope_updated or None, "sessions_since": sessions_since}
|
||||||
"scope_updated": scope_updated or None,
|
|
||||||
"sessions_since": sessions_since}, ensure_ascii=False))
|
|
||||||
return 0
|
|
||||||
print("-- Active scope --")
|
|
||||||
print(scope_text)
|
|
||||||
print(f"scope_updated: {scope_updated or '<missing — drift cannot be detected; run scope set or repair>'}")
|
|
||||||
if sessions_since is not None:
|
|
||||||
print(f"sessions logged since: {sessions_since}")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
if subcommand != "set":
|
|
||||||
raise EchoError("scope: use 'show' or 'set \"<text>\"'", 2)
|
def scope_set_op(text: str) -> dict:
|
||||||
|
"""Core scope-set (Phase 0): atomic switch (history + replace + stamp). Chatter
|
||||||
|
from the underlying PATCHes routes to stderr; returns the switch envelope."""
|
||||||
|
import contextlib
|
||||||
if not text:
|
if not text:
|
||||||
raise EchoError("scope set needs the new scope text", 2)
|
raise EchoError("scope set needs the new scope text", 2)
|
||||||
|
path = "_agent/context/current-context.md"
|
||||||
|
status, body = request("GET", vault_url(path))
|
||||||
|
check(status, body, "scope set")
|
||||||
|
current = body.decode(errors="replace")
|
||||||
prior = extract_heading(current, "Scope").replace("\n", " ").strip()[:140] or "(prior scope)"
|
prior = extract_heading(current, "Scope").replace("\n", " ").strip()[:140] or "(prior scope)"
|
||||||
|
with contextlib.redirect_stdout(sys.stderr):
|
||||||
cmd_patch(path, "prepend", "heading", "Current Context::Scope History",
|
cmd_patch(path, "prepend", "heading", "Current Context::Scope History",
|
||||||
temp_file(f"- {today()}: {prior}\n".encode()))
|
temp_file(f"- {today()}: {prior}\n".encode()))
|
||||||
cmd_patch(path, "replace", "heading", "Current Context::Scope",
|
cmd_patch(path, "replace", "heading", "Current Context::Scope",
|
||||||
@@ -601,13 +599,261 @@ def cmd_scope(subcommand: str, text: str | None = None, as_json: bool = False) -
|
|||||||
"scope set: body switched, but scope_updated frontmatter is missing "
|
"scope set: body switched, but scope_updated frontmatter is missing "
|
||||||
f"(run bootstrap.py to add it) [{exc}]"
|
f"(run bootstrap.py to add it) [{exc}]"
|
||||||
)
|
)
|
||||||
print(f"ok: scope switched (prior archived to Scope History; scope_updated={today()})")
|
return {"ok": True, "action": "scope-set", "scope": text, "prior": prior,
|
||||||
|
"scope_updated": today()}
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_scope(subcommand: str, text: str | None = None, as_json: bool = False) -> int:
|
||||||
|
if subcommand == "show":
|
||||||
|
env = scope_show_op()
|
||||||
|
if as_json:
|
||||||
|
print(json.dumps(env, ensure_ascii=False))
|
||||||
|
return 0
|
||||||
|
print("-- Active scope --")
|
||||||
|
print(env["scope"])
|
||||||
|
print(f"scope_updated: {env['scope_updated'] or '<missing — drift cannot be detected; run scope set or repair>'}")
|
||||||
|
if env["sessions_since"] is not None:
|
||||||
|
print(f"sessions logged since: {env['sessions_since']}")
|
||||||
|
return 0
|
||||||
|
if subcommand != "set":
|
||||||
|
raise EchoError("scope: use 'show' or 'set \"<text>\"'", 2)
|
||||||
|
env = scope_set_op(text or "")
|
||||||
|
print(f"ok: scope switched (prior archived to Scope History; scope_updated={env['scope_updated']})")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def cmd_load() -> int:
|
def _fetch_statuses(paths: list[str]) -> dict[str, tuple[int, bytes]]:
|
||||||
"""Cold-start orientation: the canonical 6 reads in one call. 404s on today's
|
"""Concurrently GET vault paths keeping (status, body) per path — unlike read_many,
|
||||||
daily note and the inbox are normal (printed as absent, not errors)."""
|
the 404-vs-offline distinction survives, which cmd_load's cache logic needs."""
|
||||||
|
if not paths:
|
||||||
|
return {}
|
||||||
|
workers = min(MAX_WORKERS, len(paths))
|
||||||
|
with ThreadPoolExecutor(max_workers=workers) as pool:
|
||||||
|
return dict(zip(paths, pool.map(lambda p: request("GET", vault_url(p)), paths)))
|
||||||
|
|
||||||
|
|
||||||
|
def _bullet_lines(text: str) -> list[str]:
|
||||||
|
return [ln for ln in text.splitlines() if ln.lstrip().startswith("- ")]
|
||||||
|
|
||||||
|
|
||||||
|
def _fm_line(text: str, field: str) -> str:
|
||||||
|
for ln in text.splitlines():
|
||||||
|
if ln.startswith(f"{field}:"):
|
||||||
|
return ln.split(":", 1)[1].strip().strip('"').strip("'")
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def _render_brief(texts: dict[str, str | None], listing_files: list[str],
|
||||||
|
offline: bool) -> str:
|
||||||
|
"""The token-budgeted cold-start digest (2.1.0). Selection over compression: Fact /
|
||||||
|
Pattern rules in full, recent Observations only, scope + freshness (not the whole
|
||||||
|
history), the last session's key sections, today's Agent Log lines, inbox COUNT.
|
||||||
|
Sections are trimmed lowest-priority-first when over ECHO_LOAD_BUDGET."""
|
||||||
|
budget = int(os.environ.get("ECHO_LOAD_BUDGET", "8000"))
|
||||||
|
S: list[tuple[str, str]] = [] # (section-name, text) in display order
|
||||||
|
|
||||||
|
marker = texts.get("marker")
|
||||||
|
if marker is None:
|
||||||
|
S.append(("marker", "marker: _agent/echo-vault.md ABSENT — vault not bootstrapped "
|
||||||
|
"(run bootstrap.py before relying on memory)"))
|
||||||
|
else:
|
||||||
|
ver = _fm_line(marker, "schema_version") or "?"
|
||||||
|
S.append(("marker", f"marker: _agent/echo-vault.md — bootstrapped, schema_version {ver}"))
|
||||||
|
|
||||||
|
ctx = texts.get("context")
|
||||||
|
if ctx:
|
||||||
|
scope = extract_heading(ctx, "Scope") or "(empty)"
|
||||||
|
upd = _fm_line(ctx, "scope_updated") or "unknown"
|
||||||
|
since = sum(1 for f in listing_files if f.endswith(".md") and f[:10] > upd) \
|
||||||
|
if upd != "unknown" else None
|
||||||
|
head = f"scope (updated {upd}"
|
||||||
|
if since is not None:
|
||||||
|
head += f", {since} session(s) logged since"
|
||||||
|
S.append(("scope", head + "):\n" + scope))
|
||||||
|
else:
|
||||||
|
S.append(("scope", "scope: current-context.md absent"))
|
||||||
|
|
||||||
|
prefs = texts.get("preferences")
|
||||||
|
if prefs:
|
||||||
|
fact = extract_heading(prefs, "Fact / Pattern")
|
||||||
|
if fact:
|
||||||
|
S.append(("facts", "preferences — Fact / Pattern:\n" + fact))
|
||||||
|
obs = _bullet_lines(extract_heading(prefs, "Observations"))
|
||||||
|
if obs:
|
||||||
|
shown = obs[-10:]
|
||||||
|
head = f"preferences — Observations (last {len(shown)} of {len(obs)}):"
|
||||||
|
S.append(("observations", head + "\n" + "\n".join(shown)))
|
||||||
|
|
||||||
|
hb = texts.get("heartbeat")
|
||||||
|
if hb and hb.strip():
|
||||||
|
pointer = hb.strip().splitlines()[0]
|
||||||
|
sess_path = pointer.split(" @ ", 1)[0].strip()
|
||||||
|
part = [f"last session: {pointer}"]
|
||||||
|
log_text = texts.get("_pointed_session")
|
||||||
|
if log_text:
|
||||||
|
for h in ("Goal", "Decisions Made", "Open Threads", "Suggested Next Step"):
|
||||||
|
sec = extract_heading(log_text, h)
|
||||||
|
if sec and sec.strip("() \n"):
|
||||||
|
part.append(f" {h}: " + " / ".join(ln.strip() for ln in sec.splitlines()
|
||||||
|
if ln.strip())[:400])
|
||||||
|
elif sess_path:
|
||||||
|
part.append(" (session log not readable — see the path above)")
|
||||||
|
S.append(("session", "\n".join(part)))
|
||||||
|
elif listing_files:
|
||||||
|
recent = sorted((f for f in listing_files if f.endswith(".md")), reverse=True)[:3]
|
||||||
|
S.append(("session", "last session: heartbeat absent — recent logs:\n"
|
||||||
|
+ "\n".join(f" _agent/sessions/{f}" for f in recent)))
|
||||||
|
|
||||||
|
today_note = texts.get("today")
|
||||||
|
if today_note:
|
||||||
|
log_lines = _bullet_lines(extract_heading(today_note, "Agent Log"))
|
||||||
|
S.append(("agent-log", "today's Agent Log:\n" + ("\n".join(log_lines) or " (empty)")))
|
||||||
|
else:
|
||||||
|
S.append(("agent-log", "today's Agent Log: (no daily note yet)"))
|
||||||
|
|
||||||
|
inbox = texts.get("inbox")
|
||||||
|
if inbox:
|
||||||
|
items = re.findall(r"(?m)^\s*-\s*(\d{4}-\d{2}-\d{2})", inbox)
|
||||||
|
if items:
|
||||||
|
try:
|
||||||
|
oldest = (dt.date.fromisoformat(today()) - dt.date.fromisoformat(min(items))).days
|
||||||
|
S.append(("inbox", f"inbox: {len(items)} capture(s), oldest {oldest}d — "
|
||||||
|
"offer triage if any are older than ~7 days"))
|
||||||
|
except ValueError:
|
||||||
|
S.append(("inbox", f"inbox: {len(items)} capture(s)"))
|
||||||
|
else:
|
||||||
|
S.append(("inbox", "inbox: empty"))
|
||||||
|
else:
|
||||||
|
S.append(("inbox", "inbox: empty"))
|
||||||
|
|
||||||
|
def total() -> int:
|
||||||
|
return sum(len(t) + 2 for _, t in S)
|
||||||
|
|
||||||
|
# Over budget -> trim lowest-priority sections first, with explicit markers.
|
||||||
|
for name, keep in (("observations", 4), ("agent-log", 4), ("session", 9)):
|
||||||
|
if total() <= budget:
|
||||||
|
break
|
||||||
|
for i, (n, t) in enumerate(S):
|
||||||
|
if n == name:
|
||||||
|
lines = t.splitlines()
|
||||||
|
if len(lines) > keep:
|
||||||
|
S[i] = (n, "\n".join(lines[:keep]) + "\n (truncated — /echo-load for full)")
|
||||||
|
out = f"ECHO load (brief) — {today()}\n\n" + "\n\n".join(t for _, t in S)
|
||||||
|
if len(out) > budget:
|
||||||
|
out = out[:budget] + "\n(truncated — /echo-load for full)"
|
||||||
|
if offline:
|
||||||
|
out += ("\n\nNOTE: vault unreachable — context above is last-known-good cache; "
|
||||||
|
"writes this session will be queued and synced when the vault returns.")
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
LOAD_TARGETS = [
|
||||||
|
("marker", "_agent/echo-vault.md"),
|
||||||
|
("preferences", "_agent/memory/semantic/operator-preferences.md"),
|
||||||
|
("context", "_agent/context/current-context.md"),
|
||||||
|
("heartbeat", "_agent/heartbeat/last-session.md"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _load_gather(targets):
|
||||||
|
"""Fetch the orientation reads (+ the sessions listing) in parallel and resolve
|
||||||
|
each to text via status/cache. Shared by cmd_load and load_op. Returns
|
||||||
|
(results, texts, listing_files, offline, marker_missing, heartbeat_absent)."""
|
||||||
|
import echo_queue
|
||||||
|
listing_path = "_agent/sessions/"
|
||||||
|
results = _fetch_statuses([p for _, p in targets] + [listing_path])
|
||||||
|
marker_missing = heartbeat_absent = offline = False
|
||||||
|
texts: dict[str, str | None] = {}
|
||||||
|
for label, path in targets:
|
||||||
|
status, body = results[path]
|
||||||
|
if status == 200:
|
||||||
|
echo_queue.cache_put(path, body) # refresh last-known-good for offline fallback
|
||||||
|
texts[label] = body.decode(errors="replace")
|
||||||
|
elif status == 0: # vault unreachable -> degrade to last-known-good cache
|
||||||
|
offline = True
|
||||||
|
cached = echo_queue.cache_get(path)
|
||||||
|
texts[label] = cached.decode(errors="replace") if cached is not None else None
|
||||||
|
else:
|
||||||
|
texts[label] = None
|
||||||
|
if status == 404:
|
||||||
|
if label == "marker":
|
||||||
|
marker_missing = True
|
||||||
|
if label == "heartbeat":
|
||||||
|
heartbeat_absent = True
|
||||||
|
lst_status, lst_body = results[listing_path]
|
||||||
|
listing_files: list[str] = []
|
||||||
|
if lst_status == 200:
|
||||||
|
try:
|
||||||
|
listing_files = list(json.loads(lst_body).get("files", []))
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
pass
|
||||||
|
return results, texts, listing_files, offline, marker_missing, heartbeat_absent
|
||||||
|
|
||||||
|
|
||||||
|
def _auto_fast_sweep(offline: bool) -> str | None:
|
||||||
|
"""Opportunistic index maintenance at load (2.3): when this machine's last fast
|
||||||
|
sweep is older than ECHO_FAST_SWEEP_DAYS (default 7), run `sweep --fast` so the
|
||||||
|
local recall + entity indexes true up against edits made by Obsidian or other
|
||||||
|
clients. Best-effort; skipped offline; never raises."""
|
||||||
|
if offline:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
import sweep as sweep_mod
|
||||||
|
days = float(os.environ.get("ECHO_FAST_SWEEP_DAYS", "7"))
|
||||||
|
age = sweep_mod.fast_sweep_age_days()
|
||||||
|
if age is not None and age < days:
|
||||||
|
return None
|
||||||
|
return sweep_mod.fast()
|
||||||
|
except Exception: # noqa: BLE001 — maintenance must never block a load
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch_pointed_session(texts: dict) -> None:
|
||||||
|
"""Follow the heartbeat pointer and stash the pointed session log for the digest."""
|
||||||
|
hb = texts.get("heartbeat")
|
||||||
|
if hb and hb.strip():
|
||||||
|
sess_path = hb.strip().splitlines()[0].split(" @ ", 1)[0].strip()
|
||||||
|
if sess_path:
|
||||||
|
st2, b2 = request("GET", vault_url(sess_path))
|
||||||
|
if st2 == 200:
|
||||||
|
texts["_pointed_session"] = b2.decode(errors="replace")
|
||||||
|
|
||||||
|
|
||||||
|
def load_op(brief: bool = True) -> dict:
|
||||||
|
"""Core load (Phase 0): the orientation reads as data — sections keyed by label,
|
||||||
|
plus offline/bootstrap flags, queue counts, recent sessions, and (with brief) the
|
||||||
|
rendered digest. Raises EchoError(78) when the machine is not configured."""
|
||||||
|
if not echo_config.is_configured(_CFG):
|
||||||
|
raise EchoError("NOT CONFIGURED — no usable ECHO key file on this machine "
|
||||||
|
f"(expected at {echo_config.config_path()})", 78)
|
||||||
|
import echo_queue
|
||||||
|
synced = flagged = 0
|
||||||
|
try:
|
||||||
|
synced = echo_queue.flush()
|
||||||
|
flagged = len(echo_queue.needs_attention())
|
||||||
|
except Exception: # noqa: BLE001 — queue upkeep must never block a load
|
||||||
|
pass
|
||||||
|
targets = LOAD_TARGETS + [("today", f"journal/daily/{today()}.md"),
|
||||||
|
("inbox", "inbox/captures/inbox.md")]
|
||||||
|
_, texts, listing_files, offline, marker_missing, _ = _load_gather(targets)
|
||||||
|
swept = _auto_fast_sweep(offline)
|
||||||
|
data = {"ok": True, "action": "load", "offline": offline,
|
||||||
|
"marker_missing": marker_missing, "synced": synced,
|
||||||
|
"needs_attention": flagged, "fast_sweep": swept,
|
||||||
|
"sections": {k: v for k, v in texts.items() if not k.startswith("_")},
|
||||||
|
"recent_sessions": sorted((f for f in listing_files if f.endswith(".md")),
|
||||||
|
reverse=True)[:5]}
|
||||||
|
if brief:
|
||||||
|
_fetch_pointed_session(texts)
|
||||||
|
data["brief"] = _render_brief(texts, listing_files, offline)
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def cmd_load(brief: bool = False) -> int:
|
||||||
|
"""Cold-start orientation: the canonical 6 reads in one call (fetched in parallel).
|
||||||
|
404s on today's daily note and the inbox are normal (printed as absent, not errors).
|
||||||
|
`brief` renders the token-budgeted digest the SessionStart hook injects; the default
|
||||||
|
full mode (and /echo-load) prints the raw sections unchanged."""
|
||||||
if not echo_config.is_configured(_CFG):
|
if not echo_config.is_configured(_CFG):
|
||||||
print("echo: NOT CONFIGURED — this machine has no usable ECHO key file yet.")
|
print("echo: NOT CONFIGURED — this machine has no usable ECHO key file yet.")
|
||||||
print(f"Expected at: {echo_config.config_path()}")
|
print(f"Expected at: {echo_config.config_path()}")
|
||||||
@@ -617,49 +863,51 @@ def cmd_load() -> int:
|
|||||||
print(" python3 echo.py config set --owner \"…\" --endpoint \"https://…\" --key \"…\"")
|
print(" python3 echo.py config set --owner \"…\" --endpoint \"https://…\" --key \"…\"")
|
||||||
print("Then re-run load. (Memory is unavailable until configured.)")
|
print("Then re-run load. (Memory is unavailable until configured.)")
|
||||||
return 78 # distinct: configuration required
|
return 78 # distinct: configuration required
|
||||||
targets = [
|
targets = LOAD_TARGETS + [("today", f"journal/daily/{today()}.md"),
|
||||||
("marker", "_agent/echo-vault.md"),
|
("inbox", "inbox/captures/inbox.md")]
|
||||||
("preferences", "_agent/memory/semantic/operator-preferences.md"),
|
|
||||||
("context", "_agent/context/current-context.md"),
|
|
||||||
("heartbeat", "_agent/heartbeat/last-session.md"),
|
|
||||||
("today", f"journal/daily/{today()}.md"),
|
|
||||||
("inbox", "inbox/captures/inbox.md"),
|
|
||||||
]
|
|
||||||
import echo_queue
|
import echo_queue
|
||||||
# H2: sync any writes queued during a prior outage, best-effort and quiet on empty.
|
# H2: sync any writes queued during a prior outage, best-effort and quiet on empty.
|
||||||
try:
|
try:
|
||||||
synced = echo_queue.flush()
|
synced = echo_queue.flush()
|
||||||
if synced:
|
if synced:
|
||||||
print(f"(synced {synced} write(s) queued during a prior offline session)\n")
|
print(f"(synced {synced} write(s) queued during a prior offline session)\n")
|
||||||
|
flagged = echo_queue.needs_attention()
|
||||||
|
if flagged:
|
||||||
|
print(f"NOTE: {len(flagged)} queued write(s) need attention (e.g. a capture "
|
||||||
|
"stopped at the duplicate gate on replay) — resolve with capture "
|
||||||
|
"--merge-into/--force; `echo.py flush` re-lists them.\n")
|
||||||
except Exception as exc: # noqa: BLE001 — never let queue upkeep block a load
|
except Exception as exc: # noqa: BLE001 — never let queue upkeep block a load
|
||||||
print(f"(queue flush skipped: {exc})\n", file=sys.stderr)
|
print(f"(queue flush skipped: {exc})\n", file=sys.stderr)
|
||||||
|
|
||||||
marker_missing = False
|
results, texts, listing_files, offline, marker_missing, heartbeat_absent = \
|
||||||
heartbeat_absent = False
|
_load_gather(targets)
|
||||||
offline = False
|
|
||||||
|
swept = _auto_fast_sweep(offline)
|
||||||
|
if swept:
|
||||||
|
print(f"({swept})\n")
|
||||||
|
|
||||||
|
if brief:
|
||||||
|
_fetch_pointed_session(texts)
|
||||||
|
print(_render_brief(texts, listing_files, offline))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
# ---- full mode: raw sections, output format unchanged ----
|
||||||
for label, path in targets:
|
for label, path in targets:
|
||||||
status, body = request("GET", vault_url(path))
|
status, body = results[path]
|
||||||
if status == 200:
|
if status == 200:
|
||||||
echo_queue.cache_put(path, body) # refresh last-known-good for offline fallback
|
|
||||||
print(f"===== {label}: {path} (HTTP 200) =====")
|
print(f"===== {label}: {path} (HTTP 200) =====")
|
||||||
text = body.decode(errors="replace")
|
text = texts[label] or ""
|
||||||
sys.stdout.write(text)
|
sys.stdout.write(text)
|
||||||
if not text.endswith("\n"):
|
if not text.endswith("\n"):
|
||||||
sys.stdout.write("\n")
|
sys.stdout.write("\n")
|
||||||
elif status == 404:
|
elif status == 404:
|
||||||
print(f"===== {label}: {path} (HTTP 404) =====")
|
print(f"===== {label}: {path} (HTTP 404) =====")
|
||||||
print("(absent — fine)")
|
print("(absent — fine)")
|
||||||
if label == "marker":
|
elif status == 0:
|
||||||
marker_missing = True
|
if texts[label] is not None:
|
||||||
if label == "heartbeat":
|
|
||||||
heartbeat_absent = True
|
|
||||||
elif status == 0: # vault unreachable -> degrade to last-known-good cache
|
|
||||||
offline = True
|
|
||||||
cached = echo_queue.cache_get(path)
|
|
||||||
if cached is not None:
|
|
||||||
print(f"===== {label}: {path} (OFFLINE — serving stale cache) =====")
|
print(f"===== {label}: {path} (OFFLINE — serving stale cache) =====")
|
||||||
sys.stdout.write(cached.decode(errors="replace"))
|
sys.stdout.write(texts[label])
|
||||||
if not cached.endswith(b"\n"):
|
if not texts[label].endswith("\n"):
|
||||||
sys.stdout.write("\n")
|
sys.stdout.write("\n")
|
||||||
else:
|
else:
|
||||||
print(f"===== {label}: {path} (OFFLINE — no cache) =====")
|
print(f"===== {label}: {path} (OFFLINE — no cache) =====")
|
||||||
@@ -669,15 +917,9 @@ def cmd_load() -> int:
|
|||||||
print(f"(error HTTP {status}: {body.decode(errors='replace')[:200]})")
|
print(f"(error HTTP {status}: {body.decode(errors='replace')[:200]})")
|
||||||
print()
|
print()
|
||||||
# M3: heartbeat pointer missing/stale -> fall back to the recent sessions listing
|
# M3: heartbeat pointer missing/stale -> fall back to the recent sessions listing
|
||||||
# (matches the documented loading procedure) so orientation works without the pointer.
|
# (already fetched in the batch) so orientation works without the pointer.
|
||||||
if heartbeat_absent and not offline:
|
if heartbeat_absent and not offline and listing_files:
|
||||||
st, body = request("GET", vault_url("_agent/sessions/"))
|
files = sorted((f for f in listing_files if f.endswith(".md")), reverse=True)
|
||||||
if st == 200:
|
|
||||||
try:
|
|
||||||
files = sorted((f for f in json.loads(body).get("files", []) if f.endswith(".md")),
|
|
||||||
reverse=True)
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
files = []
|
|
||||||
if files:
|
if files:
|
||||||
print("===== recent sessions (heartbeat absent — fallback) =====")
|
print("===== recent sessions (heartbeat absent — fallback) =====")
|
||||||
for f in files[:5]:
|
for f in files[:5]:
|
||||||
@@ -732,7 +974,8 @@ def cmd_config(args) -> int:
|
|||||||
def build_parser() -> argparse.ArgumentParser:
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
parser = argparse.ArgumentParser(description="Validated cross-platform ECHO vault client")
|
parser = argparse.ArgumentParser(description="Validated cross-platform ECHO vault client")
|
||||||
sub = parser.add_subparsers(dest="cmd", required=True)
|
sub = parser.add_subparsers(dest="cmd", required=True)
|
||||||
sub.add_parser("load")
|
p = sub.add_parser("load")
|
||||||
|
p.add_argument("--brief", action="store_true") # token-budgeted digest (hook default)
|
||||||
sub.add_parser("flush") # H2: replay writes queued during a prior outage
|
sub.add_parser("flush") # H2: replay writes queued during a prior outage
|
||||||
sub.add_parser("doctor") # M3: one-call readiness check
|
sub.add_parser("doctor") # M3: one-call readiness check
|
||||||
sub.add_parser("get").add_argument("path")
|
sub.add_parser("get").add_argument("path")
|
||||||
@@ -761,6 +1004,9 @@ def build_parser() -> argparse.ArgumentParser:
|
|||||||
p = sub.add_parser("reflect") # H5: apply a JSON proposal set (dry-run unless --apply)
|
p = sub.add_parser("reflect") # H5: apply a JSON proposal set (dry-run unless --apply)
|
||||||
p.add_argument("file", nargs="?")
|
p.add_argument("file", nargs="?")
|
||||||
p.add_argument("--apply", action="store_true")
|
p.add_argument("--apply", action="store_true")
|
||||||
|
p = sub.add_parser("session-end") # 2.1.0: one-call session end (log+agent-log+reflect+scope+heartbeat)
|
||||||
|
p.add_argument("file", nargs="?") # bundle JSON; - or stdin
|
||||||
|
p.add_argument("--apply", action="store_true")
|
||||||
p = sub.add_parser("triage") # one-tap inbox triage (reflect pipeline + audit log)
|
p = sub.add_parser("triage") # one-tap inbox triage (reflect pipeline + audit log)
|
||||||
p.add_argument("file", nargs="?") # proposals JSON; omit to list
|
p.add_argument("file", nargs="?") # proposals JSON; omit to list
|
||||||
p.add_argument("--list", action="store_true", dest="list_inbox")
|
p.add_argument("--list", action="store_true", dest="list_inbox")
|
||||||
@@ -796,21 +1042,27 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
args = build_parser().parse_args(argv)
|
args = build_parser().parse_args(argv)
|
||||||
try:
|
try:
|
||||||
if args.cmd == "load":
|
if args.cmd == "load":
|
||||||
return cmd_load()
|
return cmd_load(brief=args.brief)
|
||||||
if args.cmd == "doctor":
|
if args.cmd == "doctor":
|
||||||
import echo_doctor
|
import echo_doctor
|
||||||
return echo_doctor.run()
|
return echo_doctor.run()
|
||||||
if args.cmd == "flush":
|
if args.cmd == "flush":
|
||||||
import echo_queue
|
import echo_queue
|
||||||
n = echo_queue.flush()
|
n = echo_queue.flush()
|
||||||
remaining = len(echo_queue.pending())
|
remaining = echo_queue.pending()
|
||||||
|
flagged = [r for r in remaining if r.get("needs_attention")]
|
||||||
if n:
|
if n:
|
||||||
print(f"ok: flushed {n} queued write(s)"
|
print(f"ok: flushed {n} queued write(s)"
|
||||||
+ (f"; {remaining} still queued (vault unreachable)" if remaining else ""))
|
+ (f"; {len(remaining)} still queued" if remaining else ""))
|
||||||
elif remaining:
|
elif remaining:
|
||||||
print(f"ok: {remaining} write(s) still queued (vault unreachable)")
|
print(f"ok: {len(remaining)} write(s) still queued")
|
||||||
else:
|
else:
|
||||||
print("ok: queue empty")
|
print("ok: queue empty")
|
||||||
|
for r in flagged:
|
||||||
|
what = (f"capture '{(r.get('args') or {}).get('title')}'" if r.get("op") == "capture"
|
||||||
|
else f"{r.get('method')} {r.get('url')}")
|
||||||
|
print(f" needs attention ({r['needs_attention']}): {what} — resolve "
|
||||||
|
"(e.g. capture --merge-into <slug> / --force), it will not auto-land")
|
||||||
return 0
|
return 0
|
||||||
if args.cmd == "config":
|
if args.cmd == "config":
|
||||||
return cmd_config(args)
|
return cmd_config(args)
|
||||||
@@ -852,6 +1104,14 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
if not isinstance(proposals, list):
|
if not isinstance(proposals, list):
|
||||||
raise EchoError("reflect: proposals must be a JSON array of objects", 2)
|
raise EchoError("reflect: proposals must be a JSON array of objects", 2)
|
||||||
return echo_reflect.apply(proposals, confirm=args.apply)
|
return echo_reflect.apply(proposals, confirm=args.apply)
|
||||||
|
if args.cmd == "session-end":
|
||||||
|
import echo_session
|
||||||
|
raw = read_body(args.file).decode("utf-8", errors="replace")
|
||||||
|
try:
|
||||||
|
bundle = json.loads(raw)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise EchoError(f"session-end: bundle must be a JSON object ({exc})", 2)
|
||||||
|
return echo_session.session_end(bundle, apply=args.apply)
|
||||||
if args.cmd == "triage":
|
if args.cmd == "triage":
|
||||||
import echo_triage
|
import echo_triage
|
||||||
if args.list_inbox or not args.file:
|
if args.list_inbox or not args.file:
|
||||||
@@ -880,9 +1140,12 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
date=args.date, domain=args.domain, inbox=args.inbox, no_log=args.no_log,
|
date=args.date, domain=args.domain, inbox=args.inbox, no_log=args.no_log,
|
||||||
as_json=args.json, dry_run=args.dry_run,
|
as_json=args.json, dry_run=args.dry_run,
|
||||||
force=args.force, merge_into=args.merge_into)
|
force=args.force, merge_into=args.merge_into)
|
||||||
except EchoError as exc:
|
except RuntimeError as exc:
|
||||||
|
# Catches EchoError from THIS module and from helper modules' own `import echo`
|
||||||
|
# twin (when echo.py runs as __main__ the two class objects differ, but both
|
||||||
|
# subclass RuntimeError and carry .code).
|
||||||
print(f"echo.py: {exc}", file=sys.stderr)
|
print(f"echo.py: {exc}", file=sys.stderr)
|
||||||
return exc.code
|
return getattr(exc, "code", 1)
|
||||||
return 2
|
return 2
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -22,17 +22,18 @@ import echo # noqa: E402
|
|||||||
MIN_PY = (3, 9)
|
MIN_PY = (3, 9)
|
||||||
|
|
||||||
|
|
||||||
def run() -> int:
|
def run_op() -> dict:
|
||||||
reds = 0
|
"""Core doctor (Phase 0): the readiness checks as data — {checks: [{ok, label,
|
||||||
|
detail}], endpoint, fatal, ok}. `fatal` names an early-exit condition (not
|
||||||
|
configured / unreachable) after which later checks were skipped."""
|
||||||
|
checks: list[dict] = []
|
||||||
|
|
||||||
def line(ok: bool, label: str, detail: str = "") -> None:
|
def line(ok: bool, label: str, detail: str = "") -> None:
|
||||||
nonlocal reds
|
checks.append({"ok": ok, "label": label, "detail": detail})
|
||||||
reds += 0 if ok else 1
|
|
||||||
print(f" [{'OK ' if ok else 'RED'}] {label}" + (f" — {detail}" if detail else ""))
|
|
||||||
|
|
||||||
import echo_config
|
import echo_config
|
||||||
cfg = echo_config.load()
|
cfg = echo_config.load()
|
||||||
print(f"echo doctor — endpoint {cfg['endpoint'] or '(not configured)'}")
|
fatal = None
|
||||||
|
|
||||||
# 1. interpreter
|
# 1. interpreter
|
||||||
line(sys.version_info >= MIN_PY, f"python >= {MIN_PY[0]}.{MIN_PY[1]}",
|
line(sys.version_info >= MIN_PY, f"python >= {MIN_PY[0]}.{MIN_PY[1]}",
|
||||||
@@ -53,18 +54,16 @@ def run() -> int:
|
|||||||
line(bool(cfg["owner"]), "vault owner set",
|
line(bool(cfg["owner"]), "vault owner set",
|
||||||
cfg["owner"] if cfg["owner"] else "optional, but recommended for third-person writes")
|
cfg["owner"] if cfg["owner"] else "optional, but recommended for third-person writes")
|
||||||
if not echo_config.is_configured(cfg):
|
if not echo_config.is_configured(cfg):
|
||||||
print("\ndoctor: not configured — ask the operator for their key file and install it "
|
fatal = "not-configured"
|
||||||
"(`echo.py config import <file>`, or `config set --owner … --endpoint … --key …`), "
|
else:
|
||||||
"then re-run.")
|
|
||||||
return 1
|
|
||||||
|
|
||||||
# 3. reachability + auth + marker, in one GET of the bootstrap marker
|
# 3. reachability + auth + marker, in one GET of the bootstrap marker
|
||||||
status, body = echo.request("GET", echo.vault_url("_agent/echo-vault.md"))
|
status, body = echo.request("GET", echo.vault_url("_agent/echo-vault.md"))
|
||||||
if status == 0:
|
if status == 0:
|
||||||
line(False, "vault reachable", body.decode(errors="replace")[:120])
|
line(False, "vault reachable", body.decode(errors="replace")[:120])
|
||||||
print("\ndoctor: endpoint unreachable — is Obsidian + the Local REST API running?")
|
fatal = "unreachable"
|
||||||
return 1
|
else:
|
||||||
line(status not in (401, 403), "auth accepted", f"HTTP {status}" if status in (401, 403) else "")
|
line(status not in (401, 403), "auth accepted",
|
||||||
|
f"HTTP {status}" if status in (401, 403) else "")
|
||||||
if status == 404:
|
if status == 404:
|
||||||
line(False, "vault bootstrapped", "marker absent — run bootstrap.py")
|
line(False, "vault bootstrapped", "marker absent — run bootstrap.py")
|
||||||
elif status == 200:
|
elif status == 200:
|
||||||
@@ -75,11 +74,42 @@ def run() -> int:
|
|||||||
else:
|
else:
|
||||||
line(status < 400, "marker fetch", f"HTTP {status}")
|
line(status < 400, "marker fetch", f"HTTP {status}")
|
||||||
|
|
||||||
# 4. invariants pointer.
|
# Index freshness (2.3) — informational, never red: how stale this machine's
|
||||||
print(" [i] invariants — run `/echo-health` (vault_lint.py) for the full check")
|
# local indexes are (the fast sweep trues them up at load past 7 days).
|
||||||
|
if fatal is None:
|
||||||
|
try:
|
||||||
|
import sweep as sweep_mod
|
||||||
|
age = sweep_mod.fast_sweep_age_days()
|
||||||
|
line(True, "index freshness",
|
||||||
|
f"last local sweep {age:.1f}d ago" if age is not None
|
||||||
|
else "no local sweep yet — runs automatically at the next load")
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
pass
|
||||||
|
|
||||||
print(f"\ndoctor: {'all green' if reds == 0 else f'{reds} issue(s) — see RED above'}")
|
reds = sum(1 for c in checks if not c["ok"])
|
||||||
return 1 if reds else 0
|
return {"ok": reds == 0 and fatal is None, "action": "doctor",
|
||||||
|
"endpoint": cfg["endpoint"], "fatal": fatal, "reds": reds, "checks": checks}
|
||||||
|
|
||||||
|
|
||||||
|
def run() -> int:
|
||||||
|
env = run_op()
|
||||||
|
print(f"echo doctor — endpoint {env['endpoint'] or '(not configured)'}")
|
||||||
|
for c in env["checks"]:
|
||||||
|
print(f" [{'OK ' if c['ok'] else 'RED'}] {c['label']}"
|
||||||
|
+ (f" — {c['detail']}" if c["detail"] else ""))
|
||||||
|
if env["fatal"] == "not-configured":
|
||||||
|
print("\ndoctor: not configured — ask the operator for their key file and install it "
|
||||||
|
"(`echo.py config import <file>`, or `config set --owner … --endpoint … --key …`), "
|
||||||
|
"then re-run.")
|
||||||
|
return 1
|
||||||
|
if env["fatal"] == "unreachable":
|
||||||
|
print("\ndoctor: endpoint unreachable — is Obsidian + the Local REST API running?")
|
||||||
|
return 1
|
||||||
|
# invariants pointer.
|
||||||
|
print(" [i] invariants — run `/echo-health` (vault_lint.py) for the full check")
|
||||||
|
summary = "all green" if env["reds"] == 0 else f"{env['reds']} issue(s) — see RED above"
|
||||||
|
print(f"\ndoctor: {summary}")
|
||||||
|
return 1 if env["reds"] else 0
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
@@ -42,7 +42,7 @@ def main() -> int:
|
|||||||
try:
|
try:
|
||||||
import echo
|
import echo
|
||||||
with contextlib.redirect_stdout(buf):
|
with contextlib.redirect_stdout(buf):
|
||||||
rc = echo.cmd_load()
|
rc = echo.cmd_load(brief=True) # token-budgeted digest; /echo-load stays full
|
||||||
text = buf.getvalue()
|
text = buf.getvalue()
|
||||||
if rc == 78:
|
if rc == 78:
|
||||||
context = ("[echo-memory] ECHO is NOT CONFIGURED on this machine. Before "
|
context = ("[echo-memory] ECHO is NOT CONFIGURED on this machine. Before "
|
||||||
|
|||||||
@@ -30,9 +30,11 @@ MIN_TURNS = int(os.environ.get("ECHO_REFLECT_MIN_TURNS", "5"))
|
|||||||
REASON = (
|
REASON = (
|
||||||
"[echo-memory] Session-end reflection has not run yet. If this session produced "
|
"[echo-memory] Session-end reflection has not run yet. If this session produced "
|
||||||
"durable facts, decisions, commitments, or artifacts worth remembering: run the "
|
"durable facts, decisions, commitments, or artifacts worth remembering: run the "
|
||||||
"/echo-reflect flow (extract -> preview -> apply only with the operator's confirm) "
|
"/echo-reflect flow (extract -> preview -> apply only with the operator's confirm), "
|
||||||
"and write the session log + heartbeat per the echo-memory skill. If nothing "
|
"then finish with ONE call — `echo.py session-end <bundle.json> --apply` — which "
|
||||||
"durable emerged, simply finish — do not invent memories. "
|
"writes the session log, Agent Log line, reflect proposals, optional scope switch, "
|
||||||
|
"and the heartbeat (last, as the commit marker) together. If nothing durable "
|
||||||
|
"emerged, simply finish — do not invent memories. "
|
||||||
"(This reminder fires once per session.)"
|
"(This reminder fires once per session.)"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -64,6 +66,7 @@ def already_reflected(raw: str) -> bool:
|
|||||||
"""Transcript evidence that reflection or session logging already happened."""
|
"""Transcript evidence that reflection or session logging already happened."""
|
||||||
return ("/echo-reflect" in raw
|
return ("/echo-reflect" in raw
|
||||||
or re.search(r'echo\.py"?\s+reflect\b', raw) is not None
|
or re.search(r'echo\.py"?\s+reflect\b', raw) is not None
|
||||||
|
or re.search(r'echo\.py"?\s+session-end\b', raw) is not None
|
||||||
or "put _agent/sessions/" in raw
|
or "put _agent/sessions/" in raw
|
||||||
or "put _agent/heartbeat/last-session.md" in raw)
|
or "put _agent/heartbeat/last-session.md" in raw)
|
||||||
|
|
||||||
|
|||||||
@@ -168,7 +168,10 @@ def kind_for_path(path: str) -> str | None:
|
|||||||
|
|
||||||
|
|
||||||
def empty_index() -> dict:
|
def empty_index() -> dict:
|
||||||
return {"schema": 1, "updated": echo.today(), "entities": {}}
|
# Schema 2 (2.3): entries may carry `h`, a short content hash of the note body —
|
||||||
|
# the incremental sweep's change detector. Tolerated-absent (schema-1 entries
|
||||||
|
# simply get fetched and hashed on their first fast sweep).
|
||||||
|
return {"schema": 2, "updated": echo.today(), "entities": {}}
|
||||||
|
|
||||||
|
|
||||||
def load() -> dict:
|
def load() -> dict:
|
||||||
@@ -287,13 +290,20 @@ def gate_candidates(index: dict, mention: str, kind: str | None = None,
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def content_hash(text: str) -> str:
|
||||||
|
import hashlib
|
||||||
|
return hashlib.sha1(str(text).encode("utf-8", "replace")).hexdigest()[:16]
|
||||||
|
|
||||||
|
|
||||||
def upsert(index: dict, slug: str, path: str, kind: str, title: str | None = None,
|
def upsert(index: dict, slug: str, path: str, kind: str, title: str | None = None,
|
||||||
aliases=None) -> dict:
|
aliases=None, h: str | None = None) -> dict:
|
||||||
ents = index.setdefault("entities", {})
|
ents = index.setdefault("entities", {})
|
||||||
e = ents.get(slug, {})
|
e = ents.get(slug, {})
|
||||||
e["path"] = path
|
e["path"] = path
|
||||||
e["kind"] = kind
|
e["kind"] = kind
|
||||||
e["title"] = title or e.get("title") or slug
|
e["title"] = title or e.get("title") or slug
|
||||||
|
if h:
|
||||||
|
e["h"] = h
|
||||||
# Keep any prior + explicit aliases, and ALWAYS fold in the safe title variants so a
|
# Keep any prior + explicit aliases, and ALWAYS fold in the safe title variants so a
|
||||||
# hyphen/space/case form of the name resolves even if no one passed --aliases. Drop any
|
# hyphen/space/case form of the name resolves even if no one passed --aliases. Drop any
|
||||||
# alias that just equals the slug (resolve already checks the slug).
|
# alias that just equals the slug (resolve already checks the slug).
|
||||||
|
|||||||
@@ -37,12 +37,13 @@ LOG_HEADING = {
|
|||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- resolve -----
|
# ---------------------------------------------------------------- resolve -----
|
||||||
def resolve(mention: str) -> int:
|
def resolve_op(mention: str) -> dict:
|
||||||
|
"""Core resolve: mention -> match/candidates dict. No printing (Phase 0 — the same
|
||||||
|
return value backs the CLI wrapper and the MCP `echo_resolve` tool)."""
|
||||||
index = idx_mod.load()
|
index = idx_mod.load()
|
||||||
slug, e = idx_mod.resolve(index, mention)
|
slug, e = idx_mod.resolve(index, mention)
|
||||||
if e:
|
if e:
|
||||||
print(json.dumps({"match": True, "slug": slug, **e}, ensure_ascii=False, indent=2))
|
return {"match": True, "slug": slug, **e}
|
||||||
return 0
|
|
||||||
# No exact match — surface fuzzy candidates so a shortened/expanded name (e.g. "echo
|
# No exact match — surface fuzzy candidates so a shortened/expanded name (e.g. "echo
|
||||||
# memory" for the project "echo") reveals the existing note instead of looking absent.
|
# memory" for the project "echo") reveals the existing note instead of looking absent.
|
||||||
cands = idx_mod.fuzzy_candidates(index, mention)
|
cands = idx_mod.fuzzy_candidates(index, mention)
|
||||||
@@ -54,21 +55,30 @@ def resolve(mention: str) -> int:
|
|||||||
"these before creating a new note; reuse the right path or `echo.py link`.")
|
"these before creating a new note; reuse the right path or `echo.py link`.")
|
||||||
else:
|
else:
|
||||||
out["note"] = "no index entry — derive a path with the right --kind and create it"
|
out["note"] = "no index entry — derive a path with the right --kind and create it"
|
||||||
print(json.dumps(out, ensure_ascii=False, indent=2))
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def resolve(mention: str) -> int:
|
||||||
|
print(json.dumps(resolve_op(mention), ensure_ascii=False, indent=2))
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------- link -----
|
# ------------------------------------------------------------------- link -----
|
||||||
def link(a_path: str, b_path: str, as_json: bool = False) -> int:
|
def link_op(a_path: str, b_path: str) -> dict:
|
||||||
a_changed, b_changed = links.link_bidirectional(a_path, b_path)
|
"""Core link: reciprocal `## Related` links, returns the change envelope."""
|
||||||
if as_json:
|
|
||||||
import echo_output
|
import echo_output
|
||||||
env = echo_output.envelope("link", {"a": a_path, "b": b_path,
|
a_changed, b_changed = links.link_bidirectional(a_path, b_path)
|
||||||
|
return echo_output.envelope("link", {"a": a_path, "b": b_path,
|
||||||
"a_changed": a_changed, "b_changed": b_changed})
|
"a_changed": a_changed, "b_changed": b_changed})
|
||||||
|
|
||||||
|
|
||||||
|
def link(a_path: str, b_path: str, as_json: bool = False) -> int:
|
||||||
|
env = link_op(a_path, b_path)
|
||||||
|
if as_json:
|
||||||
print(json.dumps(env, ensure_ascii=False))
|
print(json.dumps(env, ensure_ascii=False))
|
||||||
return 0
|
return 0
|
||||||
print(f"ok: linked {a_path} <-> {b_path} "
|
print(f"ok: linked {env['a']} <-> {env['b']} "
|
||||||
f"(added: {'A' if a_changed else '-'}{'B' if b_changed else '-'})")
|
f"(added: {'A' if env['a_changed'] else '-'}{'B' if env['b_changed'] else '-'})")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
@@ -83,8 +93,13 @@ def recall(query, limit: int = 8, as_json: bool = False) -> int:
|
|||||||
# ----------------------------------------------------------- agent log -------
|
# ----------------------------------------------------------- agent log -------
|
||||||
def ensure_daily_log(line: str) -> None:
|
def ensure_daily_log(line: str) -> None:
|
||||||
"""Resilient: ensure today's daily note + its `## Agent Log` heading exist, then
|
"""Resilient: ensure today's daily note + its `## Agent Log` heading exist, then
|
||||||
idempotently append the line. Best-effort — never raises into the caller."""
|
idempotently append the line. Best-effort — never raises into the caller. Writes go
|
||||||
|
through the offline queue (2.1.0), so a mid-session outage queues the line instead
|
||||||
|
of silently dropping it. Offline edge accepted: if the daily note still doesn't
|
||||||
|
exist at replay time, the queued PATCH 400-drops with a loud warning — the agent-log
|
||||||
|
line is a convenience trace, not the memory itself."""
|
||||||
try:
|
try:
|
||||||
|
import echo_queue
|
||||||
date = echo.today()
|
date = echo.today()
|
||||||
path = f"journal/daily/{date}.md"
|
path = f"journal/daily/{date}.md"
|
||||||
status, body = echo.request("GET", echo.vault_url(path))
|
status, body = echo.request("GET", echo.vault_url(path))
|
||||||
@@ -92,18 +107,19 @@ def ensure_daily_log(line: str) -> None:
|
|||||||
ts, tb = echo.request("GET", echo.vault_url("journal/templates/daily-note-template.md"))
|
ts, tb = echo.request("GET", echo.vault_url("journal/templates/daily-note-template.md"))
|
||||||
tmpl = tb.decode(errors="replace") if ts == 200 else f"# {date}\n\n## Agent Log\n\n## Related\n"
|
tmpl = tb.decode(errors="replace") if ts == 200 else f"# {date}\n\n## Agent Log\n\n## Related\n"
|
||||||
tmpl = tmpl.replace("{{date:YYYY-MM-DD}}", date).replace("{{DATE}}", date)
|
tmpl = tmpl.replace("{{date:YYYY-MM-DD}}", date).replace("{{DATE}}", date)
|
||||||
echo.request("PUT", echo.vault_url(path), data=tmpl.encode(),
|
echo_queue.safe_request("PUT", echo.vault_url(path), data=tmpl.encode(),
|
||||||
headers={"Content-Type": "text/markdown"})
|
headers={"Content-Type": "text/markdown"})
|
||||||
text = tmpl
|
text = tmpl
|
||||||
else:
|
else:
|
||||||
text = body.decode(errors="replace")
|
text = body.decode(errors="replace") if status == 200 else ""
|
||||||
if not re.search(r"(?m)^## Agent Log\s*$", text):
|
if status == 200 and not re.search(r"(?m)^## Agent Log\s*$", text):
|
||||||
echo.request("POST", echo.vault_url(path), data=b"\n\n## Agent Log\n",
|
echo_queue.safe_request("POST", echo.vault_url(path), data=b"\n\n## Agent Log\n",
|
||||||
headers={"Content-Type": "text/markdown"})
|
headers={"Content-Type": "text/markdown"})
|
||||||
status, body = echo.request("GET", echo.vault_url(path))
|
st2, body2 = echo.request("GET", echo.vault_url(path))
|
||||||
if status == 200 and line in body.decode(errors="replace"):
|
if st2 == 200 and line in body2.decode(errors="replace"):
|
||||||
return
|
return
|
||||||
echo.request("PATCH", echo.vault_url(path),
|
echo_queue.safe_request(
|
||||||
|
"PATCH", echo.vault_url(path),
|
||||||
data=echo.normalize_patch_body((line + "\n").encode(), "append", "heading"),
|
data=echo.normalize_patch_body((line + "\n").encode(), "append", "heading"),
|
||||||
headers={"Operation": "append", "Target-Type": "heading",
|
headers={"Operation": "append", "Target-Type": "heading",
|
||||||
"Target": f"{date}::Agent Log", "Content-Type": "text/markdown"})
|
"Target": f"{date}::Agent Log", "Content-Type": "text/markdown"})
|
||||||
@@ -148,81 +164,106 @@ def _dated_block(today_s: str, body_text: str) -> tuple[str, str]:
|
|||||||
|
|
||||||
|
|
||||||
def _append_to_existing(path: str, kind: str, today_s: str, body_text: str) -> None:
|
def _append_to_existing(path: str, kind: str, today_s: str, body_text: str) -> None:
|
||||||
|
# Writes route through the offline queue (2.1.0) so a mid-capture outage queues the
|
||||||
|
# update instead of silently losing it. (A fully-offline capture never reaches here —
|
||||||
|
# the short-circuit in capture() queues the whole op as one semantic record.)
|
||||||
|
import echo_queue
|
||||||
text = links.get_text(path) or ""
|
text = links.get_text(path) or ""
|
||||||
h1 = links.first_h1(text) or path.rsplit("/", 1)[-1][:-3]
|
h1 = links.first_h1(text) or path.rsplit("/", 1)[-1][:-3]
|
||||||
heading = LOG_HEADING.get(kind, "Notes")
|
heading = LOG_HEADING.get(kind, "Notes")
|
||||||
if not re.search(rf"(?m)^##\s+{re.escape(heading)}\s*$", text):
|
if not re.search(rf"(?m)^##\s+{re.escape(heading)}\s*$", text):
|
||||||
echo.request("POST", echo.vault_url(path), data=f"\n\n## {heading}\n".encode(),
|
echo_queue.safe_request("POST", echo.vault_url(path), data=f"\n\n## {heading}\n".encode(),
|
||||||
headers={"Content-Type": "text/markdown"})
|
headers={"Content-Type": "text/markdown"})
|
||||||
bullet, block = _dated_block(today_s, body_text)
|
bullet, block = _dated_block(today_s, body_text)
|
||||||
status, body = echo.request("GET", echo.vault_url(path))
|
status, body = echo.request("GET", echo.vault_url(path))
|
||||||
if status == 200 and bullet in body.decode(errors="replace"):
|
if status == 200 and bullet in body.decode(errors="replace"):
|
||||||
return
|
return
|
||||||
echo.request("PATCH", echo.vault_url(path),
|
echo_queue.safe_request(
|
||||||
|
"PATCH", echo.vault_url(path),
|
||||||
data=echo.normalize_patch_body((block + "\n").encode(), "append", "heading"),
|
data=echo.normalize_patch_body((block + "\n").encode(), "append", "heading"),
|
||||||
headers={"Operation": "append", "Target-Type": "heading",
|
headers={"Operation": "append", "Target-Type": "heading",
|
||||||
"Target": f"{h1}::{heading}", "Content-Type": "text/markdown"})
|
"Target": f"{h1}::{heading}", "Content-Type": "text/markdown"})
|
||||||
echo.cmd_fm(path, "updated", json.dumps(today_s))
|
echo.cmd_fm(path, "updated", json.dumps(today_s))
|
||||||
|
|
||||||
|
|
||||||
def capture(kind: str | None, title: str, file_arg: str | None, status_v: str = "",
|
def capture_op(kind: str | None, title: str, file_arg: str | None = None, status_v: str = "",
|
||||||
aliases=None, sources=None, tags=None, date: str | None = None,
|
aliases=None, sources=None, tags=None, date: str | None = None,
|
||||||
domain: str = "business", inbox: bool = False, no_log: bool = False,
|
domain: str = "business", inbox: bool = False, no_log: bool = False,
|
||||||
as_json: bool = False, dry_run: bool = False, force: bool = False,
|
dry_run: bool = False, force: bool = False,
|
||||||
merge_into: str | None = None) -> int:
|
merge_into: str | None = None, body_text: str | None = None) -> dict:
|
||||||
|
"""Core capture (Phase 0): route + frontmatter + index + auto-link + agent-log,
|
||||||
|
returning an envelope dict — never printing to stdout (helper chatter from the
|
||||||
|
low-level verbs is redirected to stderr). Special outcomes are DATA, not exit
|
||||||
|
codes: `duplicate-gate` (ok=false, candidates), `queued:capture` (queued=true),
|
||||||
|
`dry-run:*`. Raises echo.EchoError on hard failure. `body_text` may be passed
|
||||||
|
directly (MCP path); otherwise it is read from file_arg/stdin."""
|
||||||
import contextlib
|
import contextlib
|
||||||
import io
|
|
||||||
import echo_output
|
import echo_output
|
||||||
import echo_quality
|
import echo_quality
|
||||||
|
|
||||||
real_stdout = sys.stdout
|
def chatter():
|
||||||
|
# Low-level verbs (cmd_put/cmd_append/cmd_fm) print progress lines; keep
|
||||||
def quiet():
|
# stdout clean for the envelope by routing them to stderr wholesale.
|
||||||
# M4: in --json mode, swallow the helper "ok:" chatter so stdout is clean JSON.
|
return contextlib.redirect_stdout(sys.stderr)
|
||||||
return contextlib.redirect_stdout(io.StringIO()) if as_json else contextlib.nullcontext()
|
|
||||||
|
|
||||||
def done(action: str, path: str, links: int = 0, ok: bool = True, dry: bool = False,
|
|
||||||
near=None) -> int:
|
|
||||||
if as_json:
|
|
||||||
act = f"dry-run:{action}" if dry else action
|
|
||||||
data = {"kind": kind, "path": path, "title": title, "links_added": links}
|
|
||||||
if near:
|
|
||||||
data["near_duplicates"] = near
|
|
||||||
env = echo_output.envelope(act, data, ok=ok)
|
|
||||||
print(json.dumps(env, ensure_ascii=False), file=real_stdout)
|
|
||||||
elif dry:
|
|
||||||
print(f"would {action} {kind or '-'} -> {path}")
|
|
||||||
# non-dry human output is the helper "ok:" lines + the summary printed by the caller.
|
|
||||||
return 0 if ok else 1
|
|
||||||
|
|
||||||
|
if body_text is None:
|
||||||
body_text = echo.read_body(file_arg).decode("utf-8", errors="replace")
|
body_text = echo.read_body(file_arg).decode("utf-8", errors="replace")
|
||||||
today_s = echo.today()
|
today_s = echo.today()
|
||||||
aliases = [a.strip() for a in (aliases or []) if a.strip()]
|
aliases = [a.strip() for a in (aliases or []) if a.strip()]
|
||||||
sources = [s.strip() for s in (sources or []) if s.strip()]
|
sources = [s.strip() for s in (sources or []) if s.strip()]
|
||||||
tags = [t.strip() for t in (tags or []) if t.strip()]
|
tags = [t.strip() for t in (tags or []) if t.strip()]
|
||||||
|
|
||||||
|
def env_for(action: str, path: str, links: int = 0, ok: bool = True, dry: bool = False,
|
||||||
|
near=None, **extra) -> dict:
|
||||||
|
act = f"dry-run:{action}" if dry else action
|
||||||
|
data = {"kind": kind, "path": path, "title": title, "links_added": links}
|
||||||
|
if near:
|
||||||
|
data["near_duplicates"] = near
|
||||||
|
data.update(extra)
|
||||||
|
return echo_output.envelope(act, data, ok=ok)
|
||||||
|
|
||||||
# Unknown home -> defer to the inbox (a single idempotent capture line).
|
# Unknown home -> defer to the inbox (a single idempotent capture line).
|
||||||
if inbox or not kind:
|
if inbox or not kind:
|
||||||
if dry_run:
|
if dry_run:
|
||||||
return done("inbox", "inbox/captures/inbox.md", dry=True)
|
return env_for("inbox", "inbox/captures/inbox.md", dry=True)
|
||||||
line = f"- {today_s}: {title}"
|
line = f"- {today_s}: {title}"
|
||||||
if body_text.strip():
|
if body_text.strip():
|
||||||
line += f" — {body_text.strip().splitlines()[0]}"
|
line += f" — {body_text.strip().splitlines()[0]}"
|
||||||
with quiet():
|
with chatter():
|
||||||
rc = echo.cmd_append("inbox/captures/inbox.md", line)
|
rc = echo.cmd_append("inbox/captures/inbox.md", line)
|
||||||
return done("inbox", "inbox/captures/inbox.md", ok=rc == 0)
|
return env_for("inbox", "inbox/captures/inbox.md", ok=rc == 0)
|
||||||
|
|
||||||
slug = idx_mod.slugify(title)
|
slug = idx_mod.slugify(title)
|
||||||
|
# Offline short-circuit (2.1.0). The routed path needs the entity index and several
|
||||||
|
# round-trips; when the vault is unreachable, queue the WHOLE capture as one semantic
|
||||||
|
# record — flush replays it back through this function against the index as it is at
|
||||||
|
# replay time, so routing, the duplicate gate, and aliasing re-run with fresh state.
|
||||||
|
# (Byte-level request replay would freeze a create-vs-update decision made blind.)
|
||||||
|
try:
|
||||||
index = idx_mod.load()
|
index = idx_mod.load()
|
||||||
|
except echo.EchoError as exc:
|
||||||
|
if "unreachable" not in str(exc):
|
||||||
|
raise
|
||||||
|
if dry_run:
|
||||||
|
return echo_output.envelope("dry-run:queued:capture",
|
||||||
|
{"kind": kind, "title": title, "queued": True})
|
||||||
|
import echo_queue
|
||||||
|
echo_queue.enqueue_capture({
|
||||||
|
"kind": kind, "title": title, "body_text": body_text, "status_v": status_v,
|
||||||
|
"aliases": aliases, "sources": sources, "tags": tags, "date": date,
|
||||||
|
"domain": domain, "inbox": False, "no_log": no_log,
|
||||||
|
"force": force, "merge_into": merge_into, "today": today_s,
|
||||||
|
})
|
||||||
|
return echo_output.envelope("queued:capture",
|
||||||
|
{"kind": kind, "title": title, "queued": True})
|
||||||
match_slug, existing = idx_mod.resolve(index, title)
|
match_slug, existing = idx_mod.resolve(index, title)
|
||||||
# --merge-into: the operator has already identified the canonical entity (e.g. after
|
# --merge-into: the operator has already identified the canonical entity (e.g. after
|
||||||
# a duplicate-gate stop) — route this capture as an UPDATE to it, whatever the title.
|
# a duplicate-gate stop) — route this capture as an UPDATE to it, whatever the title.
|
||||||
if merge_into and not existing:
|
if merge_into and not existing:
|
||||||
match_slug, existing = idx_mod.resolve(index, merge_into)
|
match_slug, existing = idx_mod.resolve(index, merge_into)
|
||||||
if not existing:
|
if not existing:
|
||||||
print(f"echo_ops: --merge-into '{merge_into}' matches no entity in the index "
|
raise echo.EchoError(f"--merge-into '{merge_into}' matches no entity in the "
|
||||||
f"(try `resolve` first)", file=sys.stderr)
|
"index (try `resolve` first)", 2)
|
||||||
return 2
|
|
||||||
existing_reachable = bool(existing and echo.request("GET", echo.vault_url(existing["path"]))[0] == 200)
|
existing_reachable = bool(existing and echo.request("GET", echo.vault_url(existing["path"]))[0] == 200)
|
||||||
|
|
||||||
# Pre-write duplicate gate: a strong fuzzy candidate means this title is very likely
|
# Pre-write duplicate gate: a strong fuzzy candidate means this title is very likely
|
||||||
@@ -237,40 +278,26 @@ def capture(kind: str | None, title: str, file_arg: str | None, status_v: str =
|
|||||||
for s, c, sc in idx_mod.gate_candidates(index, title, kind=kind,
|
for s, c, sc in idx_mod.gate_candidates(index, title, kind=kind,
|
||||||
threshold=DUP_GATE)]
|
threshold=DUP_GATE)]
|
||||||
if gate_hits:
|
if gate_hits:
|
||||||
if as_json:
|
return echo_output.envelope("duplicate-gate", {
|
||||||
env = echo_output.envelope("duplicate-gate", {
|
|
||||||
"kind": kind, "title": title, "candidates": gate_hits,
|
"kind": kind, "title": title, "candidates": gate_hits,
|
||||||
"note": "likely duplicate — re-run with --merge-into <slug> to update the "
|
"note": "likely duplicate — re-run with --merge-into <slug> to update the "
|
||||||
"existing entity, or --force to create anyway"}, ok=False)
|
"existing entity, or --force to create anyway"}, ok=False)
|
||||||
print(json.dumps(env, ensure_ascii=False), file=real_stdout)
|
|
||||||
else:
|
|
||||||
print(f"STOP: '{title}' likely already exists — not creating a duplicate.",
|
|
||||||
file=real_stdout)
|
|
||||||
for c in gate_hits:
|
|
||||||
print(f" candidate: {c['slug']} -> {c['path']} (score {c['score']})",
|
|
||||||
file=real_stdout)
|
|
||||||
print(" re-run with --merge-into <slug> to update the existing entity, "
|
|
||||||
"or --force to create anyway.", file=real_stdout)
|
|
||||||
return 76 # distinct exit: duplicate gate (cf. 75 lock, 78 config)
|
|
||||||
|
|
||||||
if dry_run:
|
if dry_run:
|
||||||
if existing_reachable:
|
if existing_reachable:
|
||||||
return done("update", existing["path"], dry=True)
|
return env_for("update", existing["path"], dry=True)
|
||||||
s2 = echo_quality.safe_slug(set(index.get("entities", {}).keys()), slug)
|
s2 = echo_quality.safe_slug(set(index.get("entities", {}).keys()), slug)
|
||||||
near = [c.get("path") for _, c, _ in idx_mod.fuzzy_candidates(index, title)][:3]
|
near = [c.get("path") for _, c, _ in idx_mod.fuzzy_candidates(index, title)][:3]
|
||||||
plan = done("create", idx_mod.derive_path(kind, s2, date=date, domain=domain),
|
would_gate = (not force
|
||||||
dry=True, near=near)
|
and bool(idx_mod.gate_candidates(index, title, kind=kind,
|
||||||
if (not as_json and not force
|
threshold=DUP_GATE)))
|
||||||
and idx_mod.gate_candidates(index, title, kind=kind, threshold=DUP_GATE)):
|
return env_for("create", idx_mod.derive_path(kind, s2, date=date, domain=domain),
|
||||||
# (in --json mode the near_duplicates field carries this; keep stdout clean)
|
dry=True, near=near, would_gate=would_gate)
|
||||||
print("note: a real run would STOP at the duplicate gate — use --merge-into "
|
|
||||||
"or --force.", file=real_stdout)
|
|
||||||
return plan
|
|
||||||
|
|
||||||
near_dupes: list[str] = []
|
near_dupes: list[str] = []
|
||||||
index_title = title # title to record in the index entry
|
index_title = title # title to record in the index entry
|
||||||
index_aliases = list(aliases)
|
index_aliases = list(aliases)
|
||||||
with quiet():
|
with chatter():
|
||||||
if existing_reachable:
|
if existing_reachable:
|
||||||
# Reuse the matched entity's CANONICAL slug — never re-slug from the mention, or
|
# Reuse the matched entity's CANONICAL slug — never re-slug from the mention, or
|
||||||
# two index slugs end up pointing at one note. Keep its title; learn the mention
|
# two index slugs end up pointing at one note. Keep its title; learn the mention
|
||||||
@@ -306,15 +333,11 @@ def capture(kind: str | None, title: str, file_arg: str | None, status_v: str =
|
|||||||
echo.cmd_put(path, echo.temp_file(note.encode()))
|
echo.cmd_put(path, echo.temp_file(note.encode()))
|
||||||
action = "created"
|
action = "created"
|
||||||
|
|
||||||
# H3: the entity-index write is the race the review flagged — a bare load->save lets
|
# auto-link FIRST (it mutates the note's ## Related section), so the content
|
||||||
# two concurrent captures clobber each other's entry. Route it through a lock-guarded,
|
# hash stored below reflects the note's final state. Uses the pre-update index
|
||||||
# fresh-re-read transaction. Returns the fresh index (used by auto-link below).
|
# — it only needs to find OTHER entities named in the body. (M2: is_confident_link
|
||||||
import echo_concurrency
|
# rejects short/common tokens so a 3-char alias or a word like "API" can't link
|
||||||
index = echo_concurrency.atomic_index_update(
|
# everywhere.)
|
||||||
lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases))
|
|
||||||
|
|
||||||
# auto-link: any other known entity CONFIDENTLY named in the body (M2: is_confident_link
|
|
||||||
# rejects short/common tokens so a 3-char alias or a word like "API" can't link everywhere).
|
|
||||||
linked = 0
|
linked = 0
|
||||||
for s2, e2 in index.get("entities", {}).items():
|
for s2, e2 in index.get("entities", {}).items():
|
||||||
if e2.get("path") in (None, path):
|
if e2.get("path") in (None, path):
|
||||||
@@ -324,24 +347,85 @@ def capture(kind: str | None, title: str, file_arg: str | None, status_v: str =
|
|||||||
ca, cb = links.link_bidirectional(path, e2["path"])
|
ca, cb = links.link_bidirectional(path, e2["path"])
|
||||||
linked += int(ca or cb)
|
linked += int(ca or cb)
|
||||||
|
|
||||||
# Keep the BM25 recall index current for this note (best-effort; lock-guarded inside
|
# The note's final content: hashed into the entity index (the incremental
|
||||||
# update_note, so it can't clobber a concurrent writer either).
|
# sweep's change detector, 2.3) and fed to the local recall index.
|
||||||
|
final_text = links.get_text(path) or ""
|
||||||
|
note_h = idx_mod.content_hash(final_text) if final_text else None
|
||||||
|
|
||||||
|
# H3: the entity-index write is the race the review flagged — a bare load->save lets
|
||||||
|
# two concurrent captures clobber each other's entry. Route it through a lock-guarded,
|
||||||
|
# fresh-re-read transaction.
|
||||||
|
import echo_concurrency
|
||||||
|
index = echo_concurrency.atomic_index_update(
|
||||||
|
lambda idx: idx_mod.upsert(idx, slug, path, kind, index_title, index_aliases,
|
||||||
|
h=note_h))
|
||||||
|
|
||||||
|
# Keep the local BM25 recall index current for this note (best-effort; a
|
||||||
|
# zero-network atomic file write since 2.3).
|
||||||
try:
|
try:
|
||||||
import echo_recall
|
import echo_recall
|
||||||
echo_recall.update_note(path, links.get_text(path) or "")
|
echo_recall.update_note(path, final_text)
|
||||||
except Exception as exc: # never let recall-index upkeep fail a capture
|
except Exception as exc: # never let recall-index upkeep fail a capture
|
||||||
print(f"echo_ops: recall-index update skipped ({exc})", file=sys.stderr)
|
print(f"echo_ops: recall-index update skipped ({exc})", file=sys.stderr)
|
||||||
|
|
||||||
if not no_log:
|
if not no_log:
|
||||||
ensure_daily_log(f"- {today_s}: {action} {kind} [[{links.link_token(path)}]]")
|
ensure_daily_log(f"- {today_s}: {action} {kind} [[{links.link_token(path)}]]")
|
||||||
|
|
||||||
|
return env_for(action, path, links=linked, near=near_dupes)
|
||||||
|
|
||||||
|
|
||||||
|
def capture(kind: str | None, title: str, file_arg: str | None, status_v: str = "",
|
||||||
|
aliases=None, sources=None, tags=None, date: str | None = None,
|
||||||
|
domain: str = "business", inbox: bool = False, no_log: bool = False,
|
||||||
|
as_json: bool = False, dry_run: bool = False, force: bool = False,
|
||||||
|
merge_into: str | None = None) -> int:
|
||||||
|
"""CLI wrapper around capture_op: prints exactly what pre-Phase-0 capture printed
|
||||||
|
(human prose or the --json envelope) and maps envelope outcomes to exit codes
|
||||||
|
(76 duplicate-gate, 2 usage-class errors, 0 otherwise)."""
|
||||||
|
try:
|
||||||
|
env = capture_op(kind, title, file_arg, status_v=status_v, aliases=aliases,
|
||||||
|
sources=sources, tags=tags, date=date, domain=domain,
|
||||||
|
inbox=inbox, no_log=no_log, dry_run=dry_run, force=force,
|
||||||
|
merge_into=merge_into)
|
||||||
|
except echo.EchoError as exc:
|
||||||
|
print(f"echo_ops: {exc}", file=sys.stderr)
|
||||||
|
return getattr(exc, "code", 1)
|
||||||
|
action = env.get("action", "")
|
||||||
|
|
||||||
if as_json:
|
if as_json:
|
||||||
done(action, path, links=linked, near=near_dupes)
|
print(json.dumps(env, ensure_ascii=False))
|
||||||
else:
|
if action == "duplicate-gate":
|
||||||
print(f"ok: {action} {kind} -> {path}" + (f"; auto-linked {linked}" if linked else ""))
|
return 76
|
||||||
if near_dupes:
|
return 0 if env.get("ok") else 1
|
||||||
kind_word = "entity" if len(near_dupes) == 1 else "entities"
|
|
||||||
print(f"WARNING: similar existing {kind_word} ({', '.join(near_dupes)}) — if this is "
|
if action == "duplicate-gate":
|
||||||
f"the same thing, merge or `echo.py link` instead of keeping a duplicate.",
|
print(f"STOP: '{title}' likely already exists — not creating a duplicate.")
|
||||||
file=real_stdout)
|
for c in env.get("candidates", []):
|
||||||
|
print(f" candidate: {c['slug']} -> {c['path']} (score {c['score']})")
|
||||||
|
print(" re-run with --merge-into <slug> to update the existing entity, "
|
||||||
|
"or --force to create anyway.")
|
||||||
|
return 76 # distinct exit: duplicate gate (cf. 75 lock, 78 config)
|
||||||
|
if action == "queued:capture":
|
||||||
|
print(f"queued (offline): capture {kind} '{title}' — will replay through "
|
||||||
|
"capture on the next reachable session")
|
||||||
return 0
|
return 0
|
||||||
|
if action == "dry-run:queued:capture":
|
||||||
|
print("offline: vault unreachable — a real run would queue this capture "
|
||||||
|
"for replay on the next reachable session.")
|
||||||
|
return 0
|
||||||
|
if action.startswith("dry-run:"):
|
||||||
|
print(f"would {action.split(':', 1)[1]} {kind or '-'} -> {env.get('path')}")
|
||||||
|
if env.get("would_gate"):
|
||||||
|
print("note: a real run would STOP at the duplicate gate — use --merge-into "
|
||||||
|
"or --force.")
|
||||||
|
return 0
|
||||||
|
if action == "inbox":
|
||||||
|
return 0 if env.get("ok") else 1
|
||||||
|
print(f"ok: {action} {kind} -> {env.get('path')}"
|
||||||
|
+ (f"; auto-linked {env['links_added']}" if env.get("links_added") else ""))
|
||||||
|
near = env.get("near_duplicates")
|
||||||
|
if near:
|
||||||
|
kind_word = "entity" if len(near) == 1 else "entities"
|
||||||
|
print(f"WARNING: similar existing {kind_word} ({', '.join(near)}) — if this is "
|
||||||
|
f"the same thing, merge or `echo.py link` instead of keeping a duplicate.")
|
||||||
|
return 0 if env.get("ok") else 1
|
||||||
|
|||||||
@@ -81,6 +81,23 @@ def enqueue(method: str, url: str, data: bytes | None, headers: dict, idem_key:
|
|||||||
fh.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
fh.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||||
|
|
||||||
|
|
||||||
|
def enqueue_capture(args: dict) -> None:
|
||||||
|
"""Queue a whole capture as ONE semantic record (2.1.0). Byte-level replay is wrong
|
||||||
|
for the routed capture path: the create-vs-update decision, duplicate gate, and
|
||||||
|
aliasing must re-run against the index AS IT IS AT REPLAY TIME, so flush re-invokes
|
||||||
|
echo_ops.capture with the recorded arguments instead of replaying stale requests.
|
||||||
|
`args` carries body_text inline (never a temp-file path) plus `today` (the capture-
|
||||||
|
time ECHO_TODAY) so replayed frontmatter/log dates reflect when it was said."""
|
||||||
|
key = "capture:" + hashlib.sha1(
|
||||||
|
json.dumps(args, sort_keys=True, ensure_ascii=False).encode("utf-8")).hexdigest()
|
||||||
|
if any(rec.get("idem_key") == key for rec in pending()):
|
||||||
|
return
|
||||||
|
_ensure_dirs()
|
||||||
|
rec = {"op": "capture", "args": args, "idem_key": key}
|
||||||
|
with outbox_path().open("a", encoding="utf-8") as fh:
|
||||||
|
fh.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||||
|
|
||||||
|
|
||||||
def pending() -> list[dict]:
|
def pending() -> list[dict]:
|
||||||
p = outbox_path()
|
p = outbox_path()
|
||||||
if not p.exists():
|
if not p.exists():
|
||||||
@@ -88,6 +105,12 @@ def pending() -> list[dict]:
|
|||||||
return [json.loads(ln) for ln in p.read_text(encoding="utf-8").splitlines() if ln.strip()]
|
return [json.loads(ln) for ln in p.read_text(encoding="utf-8").splitlines() if ln.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def needs_attention() -> list[dict]:
|
||||||
|
"""Queued records that replay could not land automatically (e.g. a duplicate-gate
|
||||||
|
stop) — surfaced by `flush` and `load` so the operator resolves them deliberately."""
|
||||||
|
return [rec for rec in pending() if rec.get("needs_attention")]
|
||||||
|
|
||||||
|
|
||||||
def _rewrite(records: list[dict]) -> None:
|
def _rewrite(records: list[dict]) -> None:
|
||||||
p = outbox_path()
|
p = outbox_path()
|
||||||
if not records:
|
if not records:
|
||||||
@@ -108,9 +131,57 @@ def _rebase(url: str) -> str:
|
|||||||
return echo.BASE + parts.path + (("?" + parts.query) if parts.query else "")
|
return echo.BASE + parts.path + (("?" + parts.query) if parts.query else "")
|
||||||
|
|
||||||
|
|
||||||
|
def _replay_capture(rec: dict) -> str:
|
||||||
|
"""Replay a queued semantic capture through echo_ops.capture against the CURRENT
|
||||||
|
index. Returns 'ok', 'retry' (still offline), or 'gated' (duplicate gate / needs
|
||||||
|
the operator — the record is kept and flagged, later records still replay)."""
|
||||||
|
# Cheap reachability probe FIRST: if the vault is still down, echo_ops.capture's
|
||||||
|
# own offline short-circuit would try to re-enqueue this very record — probe and
|
||||||
|
# bail as 'retry' instead of recursing into the queue.
|
||||||
|
st, _ = echo.request("GET", echo.vault_url("_agent/echo-vault.md"))
|
||||||
|
if st == 0:
|
||||||
|
return "retry"
|
||||||
|
import echo_ops
|
||||||
|
a = dict(rec.get("args") or {})
|
||||||
|
body_text = a.pop("body_text", "") or ""
|
||||||
|
day = a.pop("today", None)
|
||||||
|
bodyfile = echo.temp_file(body_text.encode("utf-8")) if body_text else None
|
||||||
|
prev = os.environ.get("ECHO_TODAY")
|
||||||
|
try:
|
||||||
|
if day:
|
||||||
|
os.environ["ECHO_TODAY"] = day # replayed dates reflect when it was said
|
||||||
|
rc = echo_ops.capture(
|
||||||
|
a.get("kind"), a.get("title") or "(untitled)", bodyfile,
|
||||||
|
status_v=a.get("status_v", ""), aliases=a.get("aliases") or [],
|
||||||
|
sources=a.get("sources") or [], tags=a.get("tags") or [],
|
||||||
|
date=a.get("date"), domain=a.get("domain", "business"),
|
||||||
|
inbox=bool(a.get("inbox")), no_log=bool(a.get("no_log")),
|
||||||
|
force=bool(a.get("force")), merge_into=a.get("merge_into"))
|
||||||
|
except Exception as exc: # noqa: BLE001 — a broken record must not wedge the queue
|
||||||
|
print(f"echo_queue: queued capture '{a.get('title')}' failed on replay ({exc}) "
|
||||||
|
"— kept and flagged", file=sys.stderr)
|
||||||
|
return "gated"
|
||||||
|
finally:
|
||||||
|
if day:
|
||||||
|
if prev is None:
|
||||||
|
os.environ.pop("ECHO_TODAY", None)
|
||||||
|
else:
|
||||||
|
os.environ["ECHO_TODAY"] = prev
|
||||||
|
if rc == 0:
|
||||||
|
return "ok"
|
||||||
|
if rc == 76:
|
||||||
|
print(f"echo_queue: queued capture '{a.get('title')}' stopped at the duplicate "
|
||||||
|
"gate on replay — resolve with capture --merge-into <slug> or --force, "
|
||||||
|
"then remove it from the outbox", file=sys.stderr)
|
||||||
|
return "gated"
|
||||||
|
|
||||||
|
|
||||||
def _replay(rec: dict) -> str:
|
def _replay(rec: dict) -> str:
|
||||||
"""Re-issue one queued write idempotently. Returns 'ok' (landed/already-present),
|
"""Re-issue one queued write idempotently. Returns 'ok' (landed/already-present),
|
||||||
'drop' (permanent 4xx — can never succeed), or 'retry' (still offline / 5xx)."""
|
'drop' (permanent 4xx — can never succeed), 'gated' (kept + flagged for the
|
||||||
|
operator), or 'retry' (still offline / 5xx)."""
|
||||||
|
if rec.get("op") == "capture":
|
||||||
|
return _replay_capture(rec)
|
||||||
method = rec["method"]
|
method = rec["method"]
|
||||||
url = _rebase(rec["url"])
|
url = _rebase(rec["url"])
|
||||||
data = base64.b64decode(rec["body_b64"]) if rec.get("body_b64") else None
|
data = base64.b64decode(rec["body_b64"]) if rec.get("body_b64") else None
|
||||||
@@ -137,21 +208,30 @@ def _replay(rec: dict) -> str:
|
|||||||
|
|
||||||
def flush() -> int:
|
def flush() -> int:
|
||||||
"""Replay queued writes in order; stop at the first still-unreachable one (keeping it
|
"""Replay queued writes in order; stop at the first still-unreachable one (keeping it
|
||||||
and the rest). Returns the number actually replayed. Safe to call when empty."""
|
and everything after). A 'gated' record (duplicate gate / replay error) is kept and
|
||||||
|
flagged but does NOT block later records — they are independent writes. Returns the
|
||||||
|
number actually replayed. Safe to call when empty."""
|
||||||
items = pending()
|
items = pending()
|
||||||
if not items:
|
if not items:
|
||||||
return 0
|
return 0
|
||||||
replayed = 0
|
replayed = 0
|
||||||
remaining: list[dict] = []
|
remaining: list[dict] = []
|
||||||
for i, rec in enumerate(items):
|
stopped = False
|
||||||
|
for rec in items:
|
||||||
|
if stopped:
|
||||||
|
remaining.append(rec)
|
||||||
|
continue
|
||||||
result = _replay(rec)
|
result = _replay(rec)
|
||||||
if result == "ok":
|
if result == "ok":
|
||||||
replayed += 1
|
replayed += 1
|
||||||
elif result == "drop":
|
elif result == "drop":
|
||||||
continue # warned in _replay; remove from queue
|
continue # warned in _replay; remove from queue
|
||||||
|
elif result == "gated":
|
||||||
|
rec["needs_attention"] = rec.get("needs_attention") or "replay-gated"
|
||||||
|
remaining.append(rec)
|
||||||
else: # retry -> still offline; keep this and everything after, in order
|
else: # retry -> still offline; keep this and everything after, in order
|
||||||
remaining = items[i:]
|
remaining.append(rec)
|
||||||
break
|
stopped = True
|
||||||
_rewrite(remaining)
|
_rewrite(remaining)
|
||||||
return replayed
|
return replayed
|
||||||
|
|
||||||
|
|||||||
@@ -52,9 +52,17 @@ sys.path.insert(0, str(Path(__file__).resolve().parent))
|
|||||||
import echo # noqa: E402
|
import echo # noqa: E402
|
||||||
import echo_index as idx_mod # noqa: E402
|
import echo_index as idx_mod # noqa: E402
|
||||||
import echo_links as links # noqa: E402
|
import echo_links as links # noqa: E402
|
||||||
|
import echo_stem # noqa: E402
|
||||||
|
|
||||||
RECALL_INDEX_PATH = "_agent/index/recall-index.json"
|
RECALL_INDEX_PATH = "_agent/index/recall-index.json"
|
||||||
INDEX_SCHEMA = 2 # bumped: schema 2 adds per-doc meta (weight/updated/status)
|
# Schema 3 (2.3, the "index train"): postings are STEMMED (echo_stem), per-doc meta
|
||||||
|
# gains a content hash `h` (incremental sweep), and the index carries a `built`
|
||||||
|
# stamp + endpoint hash. The LIVE index is LOCAL-FIRST — it lives in the state dir
|
||||||
|
# and is updated with zero vault round-trips; the vault copy at RECALL_INDEX_PATH
|
||||||
|
# is a snapshot written by sweep/session-end, used to seed fresh machines. Older
|
||||||
|
# schemas are discarded and rebuilt (sub-second since the 1.1 network work).
|
||||||
|
INDEX_SCHEMA = 3
|
||||||
|
SYNC_HOURS = float(os.environ.get("ECHO_RECALL_SYNC_HOURS", "24"))
|
||||||
|
|
||||||
# Tuning knobs. Defaults are standard BM25 (k1/b) + a 0.6 hop decay over <=2 hops.
|
# Tuning knobs. Defaults are standard BM25 (k1/b) + a 0.6 hop decay over <=2 hops.
|
||||||
K1 = 1.5
|
K1 = 1.5
|
||||||
@@ -131,8 +139,11 @@ _SKIP_BASENAMES = {"README.md"}
|
|||||||
|
|
||||||
|
|
||||||
def tokenize(text: str) -> list[str]:
|
def tokenize(text: str) -> list[str]:
|
||||||
"""Lowercase word tokens, stopworded. Frontmatter should be stripped by the caller."""
|
"""Lowercase word tokens, stopworded, STEMMED (2.3) — applied identically at
|
||||||
return [t for t in _WORD.findall(text.lower()) if t not in _STOP and len(t) > 1]
|
index and query time, so 'deployed'/'deployment' meet at 'deploy'.
|
||||||
|
Frontmatter should be stripped by the caller."""
|
||||||
|
return [echo_stem.stem(t) for t in _WORD.findall(text.lower())
|
||||||
|
if t not in _STOP and len(t) > 1]
|
||||||
|
|
||||||
|
|
||||||
def strip_frontmatter(text: str) -> str:
|
def strip_frontmatter(text: str) -> str:
|
||||||
@@ -152,9 +163,10 @@ class Bm25Index:
|
|||||||
self.df: Counter[str] = Counter() # term -> #docs containing it
|
self.df: Counter[str] = Counter() # term -> #docs containing it
|
||||||
self.postings: dict[str, dict[str, int]] = {} # term -> {doc_path: tf}
|
self.postings: dict[str, dict[str, int]] = {} # term -> {doc_path: tf}
|
||||||
self.length: dict[str, int] = {} # doc_path -> token count
|
self.length: dict[str, int] = {} # doc_path -> token count
|
||||||
self.meta: dict[str, dict] = {} # doc_path -> {w, u, s} priors
|
self.meta: dict[str, dict] = {} # doc_path -> {w, u, s, h} priors
|
||||||
self.n_docs = 0
|
self.n_docs = 0
|
||||||
self.avg_len = 0.0
|
self.avg_len = 0.0
|
||||||
|
self.built = "" # ISO stamp of last full (re)build
|
||||||
|
|
||||||
def _recompute(self) -> None:
|
def _recompute(self) -> None:
|
||||||
self.n_docs = len(self.length)
|
self.n_docs = len(self.length)
|
||||||
@@ -165,12 +177,19 @@ class Bm25Index:
|
|||||||
self.remove(path)
|
self.remove(path)
|
||||||
toks = tokenize(strip_frontmatter(raw_text))
|
toks = tokenize(strip_frontmatter(raw_text))
|
||||||
self.length[path] = len(toks)
|
self.length[path] = len(toks)
|
||||||
self.meta[path] = doc_meta(path, raw_text)
|
meta = doc_meta(path, raw_text)
|
||||||
|
# Content hash: the incremental sweep's change detector (2.3).
|
||||||
|
import hashlib
|
||||||
|
meta["h"] = hashlib.sha1(raw_text.encode("utf-8", "replace")).hexdigest()[:16]
|
||||||
|
self.meta[path] = meta
|
||||||
for term, tf in Counter(toks).items():
|
for term, tf in Counter(toks).items():
|
||||||
self.postings.setdefault(term, {})[path] = tf
|
self.postings.setdefault(term, {})[path] = tf
|
||||||
self.df[term] += 1
|
self.df[term] += 1
|
||||||
self._recompute()
|
self._recompute()
|
||||||
|
|
||||||
|
def content_hash(self, path: str) -> str | None:
|
||||||
|
return (self.meta.get(path) or {}).get("h")
|
||||||
|
|
||||||
def remove(self, path: str) -> None:
|
def remove(self, path: str) -> None:
|
||||||
if path not in self.length:
|
if path not in self.length:
|
||||||
return
|
return
|
||||||
@@ -192,11 +211,17 @@ class Bm25Index:
|
|||||||
* freshness(m.get("u"), today_s)
|
* freshness(m.get("u"), today_s)
|
||||||
* STATUS_FACTOR.get(m.get("s", ""), 1.0))
|
* STATUS_FACTOR.get(m.get("s", ""), 1.0))
|
||||||
|
|
||||||
def score(self, query: str, limit: int = 10,
|
def score(self, query: str, limit: int = 10, today_s: str | None = None,
|
||||||
today_s: str | None = None) -> list[tuple[str, float]]:
|
extra_terms: dict[str, float] | None = None) -> list[tuple[str, float]]:
|
||||||
|
"""BM25 x priors. `extra_terms` (term -> weight) are alias-expansion terms
|
||||||
|
(2.3): they contribute at reduced weight and can only boost documents that
|
||||||
|
actually contain them — expansion never invents corpus-free hits."""
|
||||||
today_s = today_s or echo.today()
|
today_s = today_s or echo.today()
|
||||||
|
weighted: dict[str, float] = {t: 1.0 for t in set(tokenize(query))}
|
||||||
|
for t, w in (extra_terms or {}).items():
|
||||||
|
weighted.setdefault(t, w)
|
||||||
scores: Counter[str] = Counter()
|
scores: Counter[str] = Counter()
|
||||||
for term in set(tokenize(query)):
|
for term, weight in weighted.items():
|
||||||
posting = self.postings.get(term)
|
posting = self.postings.get(term)
|
||||||
if not posting:
|
if not posting:
|
||||||
continue
|
continue
|
||||||
@@ -204,26 +229,29 @@ class Bm25Index:
|
|||||||
for path, tf in posting.items():
|
for path, tf in posting.items():
|
||||||
dl = self.length.get(path, 0)
|
dl = self.length.get(path, 0)
|
||||||
denom = tf + K1 * (1 - B + B * (dl / self.avg_len if self.avg_len else 1))
|
denom = tf + K1 * (1 - B + B * (dl / self.avg_len if self.avg_len else 1))
|
||||||
scores[path] += idf * (tf * (K1 + 1) / denom if denom else 0)
|
scores[path] += weight * idf * (tf * (K1 + 1) / denom if denom else 0)
|
||||||
fused = {p: s * self._doc_factor(p, today_s) for p, s in scores.items()}
|
fused = {p: s * self._doc_factor(p, today_s) for p, s in scores.items()}
|
||||||
ranked = sorted(fused.items(), key=lambda kv: -kv[1])[:limit]
|
ranked = sorted(fused.items(), key=lambda kv: -kv[1])[:limit]
|
||||||
return [(p, s) for p, s in ranked if s > MIN_SCORE]
|
return [(p, s) for p, s in ranked if s > MIN_SCORE]
|
||||||
|
|
||||||
# ---- serialization (compact; rebuildable, so the format can change freely) ----
|
# ---- serialization (compact; rebuildable, so the format can change freely) ----
|
||||||
def to_json(self) -> dict:
|
def to_json(self) -> dict:
|
||||||
return {"schema": INDEX_SCHEMA, "n_docs": self.n_docs, "avg_len": self.avg_len,
|
return {"schema": INDEX_SCHEMA, "built": self.built, "n_docs": self.n_docs,
|
||||||
"length": self.length, "postings": self.postings, "meta": self.meta}
|
"avg_len": self.avg_len, "length": self.length,
|
||||||
|
"postings": self.postings, "meta": self.meta}
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_json(cls, d: dict) -> "Bm25Index":
|
def from_json(cls, d: dict) -> "Bm25Index":
|
||||||
if d.get("schema") != INDEX_SCHEMA:
|
if d.get("schema") != INDEX_SCHEMA:
|
||||||
return cls() # older/newer format -> empty; recall's cold path rebuilds
|
return cls() # older/newer format -> empty; recall's cold path rebuilds
|
||||||
|
# (a pre-stemming index MUST NOT be reused: its postings are unstemmed)
|
||||||
ix = cls()
|
ix = cls()
|
||||||
ix.length = d.get("length", {})
|
ix.length = d.get("length", {})
|
||||||
ix.postings = d.get("postings", {})
|
ix.postings = d.get("postings", {})
|
||||||
ix.meta = d.get("meta", {})
|
ix.meta = d.get("meta", {})
|
||||||
ix.n_docs = d.get("n_docs", len(ix.length))
|
ix.n_docs = d.get("n_docs", len(ix.length))
|
||||||
ix.avg_len = d.get("avg_len", 0.0)
|
ix.avg_len = d.get("avg_len", 0.0)
|
||||||
|
ix.built = d.get("built", "")
|
||||||
ix.df = Counter({term: len(posting) for term, posting in ix.postings.items()})
|
ix.df = Counter({term: len(posting) for term, posting in ix.postings.items()})
|
||||||
return ix
|
return ix
|
||||||
|
|
||||||
@@ -271,27 +299,84 @@ def _indexable(path: str) -> bool:
|
|||||||
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------- persistence
|
# ------------------------------------------------------------------- persistence
|
||||||
def load_index() -> Bm25Index:
|
# LOCAL-FIRST (2.3). The live index is a machine-local file — updating it on capture
|
||||||
status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH))
|
# is a zero-network atomic file write instead of GET-whole-index -> PUT-whole-index
|
||||||
if status == 404:
|
# under the vault lock (the old per-write O(vault) round trip). The vault copy is a
|
||||||
return Bm25Index()
|
# SNAPSHOT: written by sweep (and best-effort at session end), read only to seed a
|
||||||
echo.check(status, body, "recall-index load")
|
# fresh machine or to catch up after another client swept (checked at most every
|
||||||
|
# ECHO_RECALL_SYNC_HOURS). Staleness across clients is tolerable — recall degrades
|
||||||
|
# gracefully and the fast sweep trues it up.
|
||||||
|
|
||||||
|
def _local_path() -> Path:
|
||||||
|
import hashlib
|
||||||
|
import echo_queue
|
||||||
|
h = hashlib.sha1((echo.BASE or "").encode()).hexdigest()[:12]
|
||||||
|
return echo_queue.state_dir() / f"recall-index-{h}.json"
|
||||||
|
|
||||||
|
|
||||||
|
def _load_local() -> Bm25Index | None:
|
||||||
|
p = _local_path()
|
||||||
|
if not p.exists():
|
||||||
|
return None
|
||||||
try:
|
try:
|
||||||
|
return Bm25Index.from_json(json.loads(p.read_text(encoding="utf-8")))
|
||||||
|
except (json.JSONDecodeError, KeyError, TypeError, OSError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def save_local(ix: Bm25Index) -> None:
|
||||||
|
p = _local_path()
|
||||||
|
p.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
tmp = p.with_suffix(".json.tmp")
|
||||||
|
tmp.write_text(json.dumps(ix.to_json(), ensure_ascii=False), encoding="utf-8")
|
||||||
|
os.replace(tmp, p) # atomic: a crash mid-write can't corrupt the live index
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch_snapshot() -> Bm25Index | None:
|
||||||
|
try:
|
||||||
|
status, body = echo.request("GET", echo.vault_url(RECALL_INDEX_PATH))
|
||||||
|
if status != 200:
|
||||||
|
return None
|
||||||
return Bm25Index.from_json(json.loads(body))
|
return Bm25Index.from_json(json.loads(body))
|
||||||
except (json.JSONDecodeError, KeyError, TypeError):
|
except Exception: # noqa: BLE001 — the snapshot is an optimization, never required
|
||||||
return Bm25Index()
|
return None
|
||||||
|
|
||||||
|
|
||||||
def save_index(ix: Bm25Index) -> None:
|
def save_snapshot(ix: Bm25Index) -> None:
|
||||||
|
"""PUT the vault snapshot — sweep + session-end only, never the per-capture path."""
|
||||||
body = json.dumps(ix.to_json(), ensure_ascii=False).encode("utf-8")
|
body = json.dumps(ix.to_json(), ensure_ascii=False).encode("utf-8")
|
||||||
status, b = echo.request("PUT", echo.vault_url(RECALL_INDEX_PATH), data=body,
|
status, b = echo.request("PUT", echo.vault_url(RECALL_INDEX_PATH), data=body,
|
||||||
headers={"Content-Type": "application/json"})
|
headers={"Content-Type": "application/json"})
|
||||||
echo.check(status, b, "recall-index save")
|
echo.check(status, b, "recall-index snapshot save")
|
||||||
|
|
||||||
|
|
||||||
def rebuild(prefetched: dict | None = None) -> Bm25Index:
|
def load_index() -> Bm25Index:
|
||||||
"""Full rebuild from every indexable entity note. Called by sweep.py and as the
|
"""The live index: local file first; the vault snapshot only seeds a fresh
|
||||||
cold-cache path inside recall(). Saves and returns the index.
|
machine or replaces a local copy that a newer sweep elsewhere has superseded
|
||||||
|
(checked when the local build stamp is older than SYNC_HOURS)."""
|
||||||
|
local = _load_local()
|
||||||
|
if local and local.n_docs:
|
||||||
|
try:
|
||||||
|
age_ok = local.built and (
|
||||||
|
_dt.datetime.now(_dt.timezone.utc)
|
||||||
|
- _dt.datetime.strptime(local.built, "%Y-%m-%dT%H:%M:%SZ").replace(
|
||||||
|
tzinfo=_dt.timezone.utc)
|
||||||
|
).total_seconds() < SYNC_HOURS * 3600
|
||||||
|
except ValueError:
|
||||||
|
age_ok = False
|
||||||
|
if age_ok:
|
||||||
|
return local
|
||||||
|
snap = _fetch_snapshot()
|
||||||
|
if snap and snap.n_docs and (not local or (snap.built or "") > (local.built or "")):
|
||||||
|
save_local(snap)
|
||||||
|
return snap
|
||||||
|
return local or Bm25Index()
|
||||||
|
|
||||||
|
|
||||||
|
def rebuild(prefetched: dict | None = None, snapshot: bool = False) -> Bm25Index:
|
||||||
|
"""Full rebuild from every indexable note. Saves the LOCAL index always; writes
|
||||||
|
the vault snapshot only when `snapshot=True` (sweep) — the read path never
|
||||||
|
surprises the vault with a PUT.
|
||||||
|
|
||||||
`prefetched` (a {path: text} map, e.g. from echo.read_many) lets sweep.py reuse the
|
`prefetched` (a {path: text} map, e.g. from echo.read_many) lets sweep.py reuse the
|
||||||
content it already pulled instead of walking and re-fetching the whole vault again."""
|
content it already pulled instead of walking and re-fetching the whole vault again."""
|
||||||
@@ -299,27 +384,30 @@ def rebuild(prefetched: dict | None = None) -> Bm25Index:
|
|||||||
if prefetched is not None:
|
if prefetched is not None:
|
||||||
items = ((p, t) for p, t in prefetched.items() if _indexable(p))
|
items = ((p, t) for p, t in prefetched.items() if _indexable(p))
|
||||||
else:
|
else:
|
||||||
items = ((p, _get(p)) for p in _walk() if _indexable(p))
|
items = echo.read_many([p for p in _walk() if _indexable(p)]).items()
|
||||||
for path, text in items:
|
for path, text in items:
|
||||||
if text is not None:
|
if text is not None:
|
||||||
ix.add(path, text) # add() strips frontmatter + extracts doc meta itself
|
ix.add(path, text) # add() strips frontmatter + extracts doc meta itself
|
||||||
save_index(ix)
|
ix.built = echo.now_iso()
|
||||||
|
save_local(ix)
|
||||||
|
if snapshot:
|
||||||
|
save_snapshot(ix)
|
||||||
return ix
|
return ix
|
||||||
|
|
||||||
|
|
||||||
def update_note(path: str, text: str) -> None:
|
def update_note(path: str, text: str) -> None:
|
||||||
"""Best-effort incremental upkeep for a single note (mirrors how entities.json is
|
"""Best-effort incremental upkeep for a single note — a LOCAL atomic file write,
|
||||||
maintained on capture). Never raises into the caller. The load->save is wrapped in the
|
no vault round-trip, no advisory lock (2.3). Maintains an existing local index
|
||||||
advisory lock with a fresh re-read (H3), so concurrent captures can't clobber the
|
only: with no local index yet, the next recall's cold path (or a sweep) builds
|
||||||
recall index either."""
|
one, and a one-note index must never shadow the snapshot. Never raises."""
|
||||||
try:
|
try:
|
||||||
if not _indexable(path):
|
if not _indexable(path):
|
||||||
return # only entity notes are part of the recall corpus
|
return # only corpus notes are part of the recall index
|
||||||
import echo_concurrency
|
ix = _load_local()
|
||||||
with echo_concurrency.vault_lock():
|
if ix is None or not ix.n_docs:
|
||||||
ix = load_index() # fresh read inside the lock
|
return
|
||||||
ix.add(path, text) # raw text: add() strips + extracts meta
|
ix.add(path, text) # raw text: add() strips + extracts meta
|
||||||
save_index(ix)
|
save_local(ix)
|
||||||
except Exception as exc: # noqa: BLE001 — upkeep must never fail a capture
|
except Exception as exc: # noqa: BLE001 — upkeep must never fail a capture
|
||||||
print(f"echo_recall: index update skipped ({exc})", file=sys.stderr)
|
print(f"echo_recall: index update skipped ({exc})", file=sys.stderr)
|
||||||
|
|
||||||
@@ -396,44 +484,36 @@ def _doc_summary(path: str, text: str) -> dict:
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
def _brief(path: str, score: float | None = None, via: str | None = None) -> None:
|
|
||||||
text = links.get_text(path)
|
|
||||||
if text is None:
|
|
||||||
return
|
|
||||||
info = _doc_summary(path, text)
|
|
||||||
head = f"\n### {path}"
|
|
||||||
if score is not None:
|
|
||||||
head += f" (score {score:.2f})"
|
|
||||||
if via:
|
|
||||||
head += f" (via {via})"
|
|
||||||
print(head)
|
|
||||||
stamps = []
|
|
||||||
if info.get("type"):
|
|
||||||
stamps.append(f"type: {info['type']}")
|
|
||||||
if info.get("updated"):
|
|
||||||
stamps.append(f"updated: {info['updated']}")
|
|
||||||
if info.get("status"):
|
|
||||||
stamps.append(f"status: {info['status']}")
|
|
||||||
if stamps:
|
|
||||||
print("_" + " · ".join(stamps) + "_") # staleness is visible, not guessed
|
|
||||||
if info.get("excerpt"):
|
|
||||||
print(info["excerpt"])
|
|
||||||
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------- entrypoint
|
# ------------------------------------------------------------------- entrypoint
|
||||||
def recall(query, limit: int = 8, as_json: bool = False) -> int:
|
def recall_op(query, limit: int = 8) -> dict:
|
||||||
|
"""Core recall (Phase 0): hybrid BM25 + graph, returning the structured envelope
|
||||||
|
({query, primary, linked}) that backs both the CLI renderings and the MCP tool."""
|
||||||
q = " ".join(query) if isinstance(query, list) else query
|
q = " ".join(query) if isinstance(query, list) else query
|
||||||
today_s = echo.today()
|
today_s = echo.today()
|
||||||
index = idx_mod.load()
|
index = idx_mod.load()
|
||||||
nmap = idx_mod.name_map(index)
|
nmap = idx_mod.name_map(index)
|
||||||
|
|
||||||
|
# --- alias query expansion (2.3): a query phrased by an entity's alias or a
|
||||||
|
# partial name folds that entity's title/alias vocabulary into the BM25 query at
|
||||||
|
# half weight — "the ECHO plugin" also scores `echo-memory` terms. Expansion can
|
||||||
|
# only boost documents that actually contain the terms.
|
||||||
|
extra_terms: dict[str, float] = {}
|
||||||
|
q_toks = set(tokenize(q))
|
||||||
|
for _slug, e, sc in idx_mod.fuzzy_candidates(index, q)[:2]:
|
||||||
|
if sc < 0.5:
|
||||||
|
continue
|
||||||
|
for name in (e.get("title", ""), _slug, *e.get("aliases", [])):
|
||||||
|
for tok in tokenize(str(name).replace("-", " ")):
|
||||||
|
if tok not in q_toks:
|
||||||
|
extra_terms[tok] = 0.5
|
||||||
|
|
||||||
# --- lexical layer (BM25); rebuild the cache on a cold miss --------------
|
# --- lexical layer (BM25); rebuild the cache on a cold miss --------------
|
||||||
lexical: list[tuple[str, float]] = []
|
lexical: list[tuple[str, float]] = []
|
||||||
try:
|
try:
|
||||||
ix = load_index()
|
ix = load_index()
|
||||||
if ix.n_docs == 0:
|
if ix.n_docs == 0:
|
||||||
ix = rebuild()
|
ix = rebuild()
|
||||||
lexical = ix.score(q, limit=limit, today_s=today_s)
|
lexical = ix.score(q, limit=limit, today_s=today_s, extra_terms=extra_terms)
|
||||||
except echo.EchoError as exc:
|
except echo.EchoError as exc:
|
||||||
print(f"recall: BM25 index unavailable ({exc}); falling back to server search",
|
print(f"recall: BM25 index unavailable ({exc}); falling back to server search",
|
||||||
file=sys.stderr)
|
file=sys.stderr)
|
||||||
@@ -464,7 +544,6 @@ def recall(query, limit: int = 8, as_json: bool = False) -> int:
|
|||||||
# --- graph layer ----------------------------------------------------------
|
# --- graph layer ----------------------------------------------------------
|
||||||
neighbours = expand_graph(hits, nmap, base, max_hops=MAX_HOPS)
|
neighbours = expand_graph(hits, nmap, base, max_hops=MAX_HOPS)
|
||||||
|
|
||||||
if as_json:
|
|
||||||
import echo_output
|
import echo_output
|
||||||
texts = echo.read_many(hits + [p for p, _ in neighbours[: 2 * limit]])
|
texts = echo.read_many(hits + [p for p, _ in neighbours[: 2 * limit]])
|
||||||
primary = []
|
primary = []
|
||||||
@@ -479,16 +558,35 @@ def recall(query, limit: int = 8, as_json: bool = False) -> int:
|
|||||||
continue
|
continue
|
||||||
linked.append({**_doc_summary(p, texts[p]),
|
linked.append({**_doc_summary(p, texts[p]),
|
||||||
"score": round(sc, 3), "via": via})
|
"score": round(sc, 3), "via": via})
|
||||||
env = echo_output.envelope("recall", {"query": q, "primary": primary,
|
return echo_output.envelope("recall", {"query": q, "primary": primary,
|
||||||
"linked": linked})
|
"linked": linked})
|
||||||
|
|
||||||
|
|
||||||
|
def _print_hit(info: dict) -> None:
|
||||||
|
"""Prose rendering of one recall hit — same fields the envelope carries."""
|
||||||
|
head = f"\n### {info['path']}"
|
||||||
|
if info.get("score") is not None:
|
||||||
|
head += f" (score {info['score']:.2f})"
|
||||||
|
if info.get("via"):
|
||||||
|
head += f" (via {info['via']})"
|
||||||
|
print(head)
|
||||||
|
stamps = [f"{k}: {info[k]}" for k in ("type", "updated", "status") if info.get(k)]
|
||||||
|
if stamps:
|
||||||
|
print("_" + " · ".join(stamps) + "_") # staleness is visible, not guessed
|
||||||
|
if info.get("excerpt"):
|
||||||
|
print(info["excerpt"])
|
||||||
|
|
||||||
|
|
||||||
|
def recall(query, limit: int = 8, as_json: bool = False) -> int:
|
||||||
|
env = recall_op(query, limit=limit)
|
||||||
|
if as_json:
|
||||||
print(json.dumps(env, ensure_ascii=False))
|
print(json.dumps(env, ensure_ascii=False))
|
||||||
return 0
|
return 0
|
||||||
|
print(f"== recall: {env['query']} ==")
|
||||||
print(f"== recall: {q} ==")
|
print(f"\n# Primary hits ({len(env['primary'])})")
|
||||||
print(f"\n# Primary hits ({len(hits)})")
|
for info in env["primary"]:
|
||||||
for p in hits:
|
_print_hit(info)
|
||||||
_brief(p, score=base.get(p))
|
print(f"\n# Linked context ({len(env['linked'])})")
|
||||||
print(f"\n# Linked context ({len(neighbours)})")
|
for info in env["linked"]:
|
||||||
for p, (sc, via) in neighbours[: 2 * limit]:
|
_print_hit(info)
|
||||||
_brief(p, score=sc, via=via)
|
|
||||||
return 0
|
return 0
|
||||||
|
|||||||
@@ -99,43 +99,69 @@ def preview(proposals: list[dict]) -> str:
|
|||||||
return "\n".join(rows)
|
return "\n".join(rows)
|
||||||
|
|
||||||
|
|
||||||
def apply(proposals: list[dict], confirm: bool = False) -> int:
|
def apply_op(proposals: list[dict], confirm: bool = False) -> dict:
|
||||||
"""Validate -> classify -> preview; with confirm=True, apply each via echo_ops.capture
|
"""Core reflect (Phase 0): validate -> classify -> (apply via capture_op). Returns
|
||||||
(which routes/indexes/links/logs, each self-lock-guarded). Without confirm it is a
|
an envelope — rows carry the classified preview; with confirm, `results` carries
|
||||||
dry-run that writes nothing — the preview IS the confirmation step."""
|
each proposal's capture outcome. Never prints to stdout."""
|
||||||
|
import echo_output
|
||||||
valid, errors = validate(proposals)
|
valid, errors = validate(proposals)
|
||||||
for e in errors:
|
rows: list[dict] = []
|
||||||
print(f"skip: {e}", file=sys.stderr)
|
counts: dict[str, int] = {}
|
||||||
if not valid:
|
if valid:
|
||||||
print("reflect: no valid proposals to apply.")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
classify(valid)
|
classify(valid)
|
||||||
counts = {a: sum(1 for p in valid if p.get("_action") == a) for a in ("create", "update", "inbox", "error")}
|
counts = {a: sum(1 for p in valid if p.get("_action") == a)
|
||||||
print(f"reflect: {len(valid)} proposal(s) — "
|
for a in ("create", "update", "inbox", "error")}
|
||||||
f"{counts['create']} new, {counts['update']} update, {counts['inbox']} inbox"
|
rows = [{"action": p.get("_action"), "kind": p.get("kind"),
|
||||||
+ (f", {counts['error']} error" if counts["error"] else ""))
|
"title": p["title"], "path": p.get("_path")} for p in valid]
|
||||||
print(preview(valid))
|
data = {"proposals": len(proposals or []), "valid": len(valid), "errors": errors,
|
||||||
|
"counts": counts, "rows": rows, "dry_run": not confirm,
|
||||||
if not confirm:
|
"applied": 0, "gated": 0, "results": []}
|
||||||
print("\nreflect: dry-run — re-run with --apply to write these to memory.")
|
if not valid or not confirm:
|
||||||
return 0
|
return echo_output.envelope("reflect", data)
|
||||||
|
|
||||||
import echo_ops # lazy: the apply path pulls in the capture/index/link stack
|
import echo_ops # lazy: the apply path pulls in the capture/index/link stack
|
||||||
applied = gated = 0
|
applied = gated = 0
|
||||||
|
results = []
|
||||||
for p in valid:
|
for p in valid:
|
||||||
if p.get("_action") == "error":
|
if p.get("_action") == "error":
|
||||||
continue
|
continue
|
||||||
bodyfile = echo.temp_file((p.get("body") or "").encode()) if p.get("body") else None
|
env = echo_ops.capture_op(
|
||||||
rc = echo_ops.capture(
|
p.get("kind"), p["title"], body_text=p.get("body") or "",
|
||||||
p.get("kind"), p["title"], bodyfile,
|
|
||||||
aliases=p.get("aliases") or [], sources=p.get("sources") or [],
|
aliases=p.get("aliases") or [], sources=p.get("sources") or [],
|
||||||
tags=p.get("tags") or [],
|
tags=p.get("tags") or [],
|
||||||
date=p.get("date"), domain=p.get("domain", "business"),
|
date=p.get("date"), domain=p.get("domain", "business"),
|
||||||
inbox=bool(p.get("inbox")) or not p.get("kind"))
|
inbox=bool(p.get("inbox")) or not p.get("kind"))
|
||||||
applied += 1 if rc == 0 else 0
|
act = env.get("action", "")
|
||||||
gated += 1 if rc == 76 else 0
|
if act == "duplicate-gate":
|
||||||
print(f"reflect: applied {applied}/{len(valid)} proposal(s)."
|
gated += 1
|
||||||
|
elif env.get("ok"):
|
||||||
|
applied += 1
|
||||||
|
results.append({"title": p["title"], "action": act,
|
||||||
|
"path": env.get("path"), "ok": bool(env.get("ok"))})
|
||||||
|
data.update(applied=applied, gated=gated, results=results)
|
||||||
|
return echo_output.envelope("reflect", data)
|
||||||
|
|
||||||
|
|
||||||
|
def apply(proposals: list[dict], confirm: bool = False) -> int:
|
||||||
|
"""CLI wrapper: same preview/summary text as pre-Phase-0. Without confirm it is a
|
||||||
|
dry-run that writes nothing — the preview IS the confirmation step."""
|
||||||
|
env = apply_op(proposals, confirm=confirm)
|
||||||
|
for e in env["errors"]:
|
||||||
|
print(f"skip: {e}", file=sys.stderr)
|
||||||
|
if not env["valid"]:
|
||||||
|
print("reflect: no valid proposals to apply.")
|
||||||
|
return 0
|
||||||
|
counts = env["counts"]
|
||||||
|
print(f"reflect: {env['valid']} proposal(s) — "
|
||||||
|
f"{counts['create']} new, {counts['update']} update, {counts['inbox']} inbox"
|
||||||
|
+ (f", {counts['error']} error" if counts["error"] else ""))
|
||||||
|
print("\n".join(f" {r['action'] or '?':7} | {('-' if r['action'] == 'inbox' else r['kind'] or '?'):9} "
|
||||||
|
f"| {r['title']} -> {r['path'] or '?'}" for r in env["rows"]))
|
||||||
|
if env["dry_run"]:
|
||||||
|
print("\nreflect: dry-run — re-run with --apply to write these to memory.")
|
||||||
|
return 0
|
||||||
|
gated = env["gated"]
|
||||||
|
print(f"reflect: applied {env['applied']}/{env['valid']} proposal(s)."
|
||||||
+ (f" {gated} stopped at the duplicate gate — re-propose with the existing "
|
+ (f" {gated} stopped at the duplicate gate — re-propose with the existing "
|
||||||
f"entity's title (or capture --merge-into <slug>)." if gated else ""))
|
f"entity's title (or capture --merge-into <slug>)." if gated else ""))
|
||||||
return 0
|
return 0
|
||||||
|
|||||||
@@ -0,0 +1,177 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""echo_session.py — the session-end bundle: one call ends a session correctly. [2.1.0]
|
||||||
|
|
||||||
|
Ending a substantive session used to take four-plus separate invocations (session-log
|
||||||
|
PUT, Agent Log append, heartbeat PUT, optional scope set, optional reflect apply) —
|
||||||
|
a cost per session, and a partial failure left broken orientation state (log written
|
||||||
|
but heartbeat stale). This verb does all of it in one call, under ONE advisory-lock
|
||||||
|
acquisition, in a fixed order with the heartbeat written LAST as the commit marker:
|
||||||
|
a failure part-way leaves the previous pointer intact, never a dangling one.
|
||||||
|
|
||||||
|
Bundle shape (JSON object):
|
||||||
|
{
|
||||||
|
"slug": "echo-mcp-spec", # required, kebab-case
|
||||||
|
"log_body": "<full session-log markdown, frontmatter included>", # required
|
||||||
|
"agent_log_line": "- 2026-07-28: ...", # optional; derived when omitted
|
||||||
|
"scope": "new scope text", # optional -> scope set
|
||||||
|
"reflect": [ { ...PROPOSAL_SCHEMA... } ] # optional -> routed via capture
|
||||||
|
}
|
||||||
|
|
||||||
|
Dry-run by default (previews the whole plan, reflect included); --apply writes.
|
||||||
|
Times: the filename HHMM comes from $ECHO_NOW (HHMM) else the local clock; the date
|
||||||
|
from ECHO_TODAY via echo.today(). Offline: every step rides the offline queue, so a
|
||||||
|
session end during an outage queues durably instead of failing.
|
||||||
|
|
||||||
|
CLI: echo.py session-end <bundle.json> [--apply]
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import datetime as dt
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
import echo # noqa: E402
|
||||||
|
|
||||||
|
SESSION_RE = re.compile(r"^\d{4}-\d{2}-\d{2}-\d{4}-[a-z0-9-]+\.md$")
|
||||||
|
SLUG_RE = re.compile(r"^[a-z0-9][a-z0-9-]*$")
|
||||||
|
|
||||||
|
|
||||||
|
def _hhmm() -> str:
|
||||||
|
v = os.environ.get("ECHO_NOW", "").strip()
|
||||||
|
if v:
|
||||||
|
if not re.match(r"^\d{4}$", v):
|
||||||
|
raise echo.EchoError(f"session-end: $ECHO_NOW must be HHMM, got '{v}'", 2)
|
||||||
|
return v
|
||||||
|
return dt.datetime.now().strftime("%H%M")
|
||||||
|
|
||||||
|
|
||||||
|
def validate(bundle: dict) -> tuple[str, str]:
|
||||||
|
"""Return (session_path, agent_log_line) or raise EchoError(2). Validation runs
|
||||||
|
BEFORE any write — a bad bundle writes nothing at all."""
|
||||||
|
if not isinstance(bundle, dict):
|
||||||
|
raise echo.EchoError("session-end: bundle must be a JSON object", 2)
|
||||||
|
slug = str(bundle.get("slug", "")).strip()
|
||||||
|
if not SLUG_RE.match(slug):
|
||||||
|
raise echo.EchoError(f"session-end: slug must be kebab-case, got '{slug}'", 2)
|
||||||
|
if not str(bundle.get("log_body", "")).strip():
|
||||||
|
raise echo.EchoError("session-end: log_body is required (the full session-log markdown)", 2)
|
||||||
|
reflect = bundle.get("reflect")
|
||||||
|
if reflect is not None and not isinstance(reflect, list):
|
||||||
|
raise echo.EchoError("session-end: reflect must be a JSON array of proposals", 2)
|
||||||
|
fname = f"{echo.today()}-{_hhmm()}-{slug}.md"
|
||||||
|
if not SESSION_RE.match(fname):
|
||||||
|
raise echo.EchoError(f"session-end: derived filename '{fname}' violates the "
|
||||||
|
"canonical YYYY-MM-DD-HHMM-<slug>.md form", 2)
|
||||||
|
path = f"_agent/sessions/{fname}"
|
||||||
|
line = str(bundle.get("agent_log_line") or "").strip() \
|
||||||
|
or f"- {echo.today()}: session logged [[{path[:-3]}]]"
|
||||||
|
return path, line
|
||||||
|
|
||||||
|
|
||||||
|
def session_end_op(bundle: dict, apply: bool = False) -> dict:
|
||||||
|
"""Core session-end (Phase 0): validate -> plan -> (apply). Returns an envelope
|
||||||
|
(plan rows + per-step results); never prints to stdout — helper chatter from the
|
||||||
|
underlying verbs routes to stderr. Raises echo.EchoError(2) on a bad bundle,
|
||||||
|
BEFORE any write."""
|
||||||
|
import contextlib
|
||||||
|
import echo_output
|
||||||
|
import echo_reflect
|
||||||
|
path, line = validate(bundle)
|
||||||
|
scope = str(bundle.get("scope") or "").strip()
|
||||||
|
proposals = bundle.get("reflect") or []
|
||||||
|
|
||||||
|
valid, errors = echo_reflect.validate(proposals)
|
||||||
|
rows: list[dict] = []
|
||||||
|
if valid:
|
||||||
|
echo_reflect.classify(valid)
|
||||||
|
rows = [{"action": p.get("_action"), "kind": p.get("kind"),
|
||||||
|
"title": p["title"], "path": p.get("_path")} for p in valid]
|
||||||
|
data = {"path": path, "agent_log_line": line, "scope": scope or None,
|
||||||
|
"reflect_rows": rows, "reflect_errors": errors,
|
||||||
|
"dry_run": not apply, "steps": {}}
|
||||||
|
if not apply:
|
||||||
|
return echo_output.envelope("session-end", data)
|
||||||
|
|
||||||
|
import echo_concurrency
|
||||||
|
import echo_ops
|
||||||
|
steps: dict[str, str] = {}
|
||||||
|
with echo_concurrency.vault_lock(), contextlib.redirect_stdout(sys.stderr):
|
||||||
|
# 1. The session log itself. A hard failure here aborts the whole bundle —
|
||||||
|
# nothing after it (including the heartbeat) runs, so orientation state
|
||||||
|
# can never point at a log that was never written.
|
||||||
|
echo.cmd_put(path, echo.temp_file(str(bundle["log_body"]).encode("utf-8")))
|
||||||
|
steps["session_log"] = "ok"
|
||||||
|
# 2. Agent-log line (best-effort by contract; queues itself when offline).
|
||||||
|
echo_ops.ensure_daily_log(line)
|
||||||
|
steps["agent_log"] = "ok"
|
||||||
|
# 3. Reflect proposals through the normal capture path (gate-aware).
|
||||||
|
if valid:
|
||||||
|
gated = 0
|
||||||
|
for p in valid:
|
||||||
|
if p.get("_action") == "error":
|
||||||
|
continue
|
||||||
|
env = echo_ops.capture_op(
|
||||||
|
p.get("kind"), p["title"], body_text=p.get("body") or "",
|
||||||
|
aliases=p.get("aliases") or [], sources=p.get("sources") or [],
|
||||||
|
tags=p.get("tags") or [], date=p.get("date"),
|
||||||
|
domain=p.get("domain", "business"),
|
||||||
|
inbox=bool(p.get("inbox")) or not p.get("kind"))
|
||||||
|
gated += 1 if env.get("action") == "duplicate-gate" else 0
|
||||||
|
steps["reflect"] = f"{len(valid) - gated}/{len(valid)} applied" \
|
||||||
|
+ (f", {gated} gated (re-propose with --merge-into)" if gated else "")
|
||||||
|
else:
|
||||||
|
steps["reflect"] = "skipped (none)"
|
||||||
|
# 4. Scope switch (optional).
|
||||||
|
if scope:
|
||||||
|
echo.cmd_scope("set", scope)
|
||||||
|
steps["scope"] = "ok"
|
||||||
|
else:
|
||||||
|
steps["scope"] = "unchanged"
|
||||||
|
# 5. Heartbeat LAST — the commit marker for the whole bundle.
|
||||||
|
echo.cmd_put("_agent/heartbeat/last-session.md",
|
||||||
|
echo.temp_file(f"{path} @ {echo.now_iso()}\n".encode("utf-8")))
|
||||||
|
steps["heartbeat"] = "ok"
|
||||||
|
# Index maintenance (2.3), NOT part of the bundle contract: push this
|
||||||
|
# machine's local recall index to the vault snapshot so other machines
|
||||||
|
# can seed/catch up. Best-effort — a failure never taints the bundle.
|
||||||
|
try:
|
||||||
|
import echo_recall
|
||||||
|
rix = echo_recall._load_local()
|
||||||
|
if rix and rix.n_docs:
|
||||||
|
rix.built = rix.built or echo.now_iso()
|
||||||
|
echo_recall.save_snapshot(rix)
|
||||||
|
steps["recall_snapshot"] = "ok"
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
steps["recall_snapshot"] = "skipped"
|
||||||
|
|
||||||
|
data["steps"] = steps
|
||||||
|
return echo_output.envelope("session-end", data)
|
||||||
|
|
||||||
|
|
||||||
|
def session_end(bundle: dict, apply: bool = False) -> int:
|
||||||
|
"""CLI wrapper: same plan/summary text as before; envelope logic in session_end_op."""
|
||||||
|
env = session_end_op(bundle, apply=apply)
|
||||||
|
for e in env["reflect_errors"]:
|
||||||
|
print(f"skip: {e}", file=sys.stderr)
|
||||||
|
print(f"session-end plan ({'APPLY' if apply else 'dry-run'}):")
|
||||||
|
print(f" 1. session log -> {env['path']}")
|
||||||
|
print(f" 2. agent-log line: {env['agent_log_line']}")
|
||||||
|
rows = env["reflect_rows"]
|
||||||
|
if rows:
|
||||||
|
print(f" 3. reflect: {len(rows)} proposal(s)")
|
||||||
|
print("\n".join(f" {r['action'] or '?':7} | {('-' if r['action'] == 'inbox' else r['kind'] or '?'):9} "
|
||||||
|
f"| {r['title']} -> {r['path'] or '?'}" for r in rows))
|
||||||
|
else:
|
||||||
|
print(" 3. reflect: (none)")
|
||||||
|
print(f" 4. scope set: {env['scope']!r}" if env["scope"] else " 4. scope: (unchanged)")
|
||||||
|
print(f" 5. heartbeat -> {env['path']} @ <now> (written LAST — the commit marker)")
|
||||||
|
if env["dry_run"]:
|
||||||
|
print("\nsession-end: dry-run — re-run with --apply to write.")
|
||||||
|
return 0
|
||||||
|
print("\nsession-end: done — "
|
||||||
|
+ "; ".join(f"{k}: {v}" for k, v in env["steps"].items()))
|
||||||
|
return 0
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""echo_stem.py — a deliberately light, dependency-free English stemmer. [2.3]
|
||||||
|
|
||||||
|
BM25 is purely lexical: "deploy", "deployed", and "deployment" are three unrelated
|
||||||
|
terms to it, so a paraphrased recall query misses notes stored with a different
|
||||||
|
inflection. This Porter-LITE stemmer conflates the common inflection families while
|
||||||
|
staying conservative — when in doubt it leaves the token alone, because an
|
||||||
|
over-stemmed index silently merges unrelated words (far worse than a missed match).
|
||||||
|
|
||||||
|
Applied by echo_recall.tokenize() at BOTH index and query time (recall-index
|
||||||
|
schema 3; older indexes are discarded and rebuilt). Pure function, unit-tested in
|
||||||
|
test_echo_client.py.
|
||||||
|
|
||||||
|
Design notes:
|
||||||
|
* plural / -ed / -ing families first (with the classic restore rules: doubled
|
||||||
|
consonant undoubling, at/bl/iz +e, cvc +e), then one derivational suffix pass,
|
||||||
|
then y->i and final-e normalization so "navigate"/"navigation"/"navigating"
|
||||||
|
all land on "navigat".
|
||||||
|
* every rule guards a minimum remaining stem length — "sing" is not "s".
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import re
|
||||||
|
|
||||||
|
_VOWEL = re.compile(r"[aeiouy]")
|
||||||
|
_DOUBLE = re.compile(r"([^aeiouylsz])\1$")
|
||||||
|
_CVC = re.compile(r"[^aeiou][aeiouy][^aeiouwxy]$")
|
||||||
|
|
||||||
|
# One derivational pass, most-specific first. Replacements keep families together:
|
||||||
|
# navigational/navigation -> navigat · saturation/saturated -> saturat ·
|
||||||
|
# deployment/deployed -> deploy · usefulness/useful -> use... (guarded by length).
|
||||||
|
_SUFFIXES = (
|
||||||
|
("ization", "ize"), ("ational", "ate"), ("fulness", "ful"), ("ousness", "ous"),
|
||||||
|
("iveness", "ive"), ("tional", "t"), ("biliti", "ble"), ("ements", ""),
|
||||||
|
("ment", ""), ("ness", ""), ("tion", "t"), ("sion", "s"), ("ance", ""),
|
||||||
|
("ence", ""), ("able", ""), ("ible", ""), ("ally", "al"), ("ously", "ous"),
|
||||||
|
("ively", "ive"), ("ly", ""),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def stem(t: str) -> str:
|
||||||
|
"""Stem one lowercase token. Never raises; returns the token unchanged when no
|
||||||
|
rule applies safely."""
|
||||||
|
if len(t) <= 3 or not t.isascii():
|
||||||
|
return t
|
||||||
|
# --- plurals -------------------------------------------------------------
|
||||||
|
if t.endswith("sses"):
|
||||||
|
t = t[:-2]
|
||||||
|
elif t.endswith("ies") and len(t) > 4:
|
||||||
|
t = t[:-3] + "i"
|
||||||
|
elif t.endswith("s") and not t.endswith(("ss", "us", "is")):
|
||||||
|
t = t[:-1]
|
||||||
|
# --- -ed / -ing (with restore rules) --------------------------------------
|
||||||
|
for suf in ("ingly", "edly", "ing", "ed"):
|
||||||
|
if t.endswith(suf):
|
||||||
|
base = t[: -len(suf)]
|
||||||
|
if len(base) >= 3 and _VOWEL.search(base):
|
||||||
|
t = base
|
||||||
|
if t.endswith(("at", "bl", "iz")):
|
||||||
|
t += "e"
|
||||||
|
elif _DOUBLE.search(t):
|
||||||
|
t = t[:-1]
|
||||||
|
elif len(t) == 3 and _CVC.search(t):
|
||||||
|
t += "e"
|
||||||
|
break
|
||||||
|
# --- one derivational suffix pass -----------------------------------------
|
||||||
|
for suf, rep in _SUFFIXES:
|
||||||
|
if t.endswith(suf):
|
||||||
|
base = t[: -len(suf)]
|
||||||
|
if len(base) >= 3 and len(base + rep) >= 3:
|
||||||
|
t = base + rep
|
||||||
|
break
|
||||||
|
# --- normalize: y->i (so penalty/penalties agree), then drop a final e -----
|
||||||
|
if len(t) > 3 and t.endswith("y") and t[-2] not in "aeiou":
|
||||||
|
t = t[:-1] + "i"
|
||||||
|
if len(t) > 4 and t.endswith("e"):
|
||||||
|
t = t[:-1]
|
||||||
|
return t
|
||||||
@@ -55,19 +55,25 @@ def parse_inbox(text: str) -> list[dict]:
|
|||||||
return items
|
return items
|
||||||
|
|
||||||
|
|
||||||
def list_inbox(as_json: bool = False) -> int:
|
def list_op() -> dict:
|
||||||
|
"""Core inbox listing (Phase 0): structured captures envelope, no printing."""
|
||||||
|
import echo_output
|
||||||
status, body = echo.request("GET", echo.vault_url(INBOX_PATH))
|
status, body = echo.request("GET", echo.vault_url(INBOX_PATH))
|
||||||
if status == 404:
|
if status == 404:
|
||||||
items = []
|
items = []
|
||||||
else:
|
else:
|
||||||
echo.check(status, body, f"triage list {INBOX_PATH}")
|
echo.check(status, body, f"triage list {INBOX_PATH}")
|
||||||
items = parse_inbox(body.decode(errors="replace"))
|
items = parse_inbox(body.decode(errors="replace"))
|
||||||
if as_json:
|
return echo_output.envelope("triage-list", {"path": INBOX_PATH, "items": items,
|
||||||
import echo_output
|
|
||||||
env = echo_output.envelope("triage-list", {"path": INBOX_PATH, "items": items,
|
|
||||||
"count": len(items)})
|
"count": len(items)})
|
||||||
|
|
||||||
|
|
||||||
|
def list_inbox(as_json: bool = False) -> int:
|
||||||
|
env = list_op()
|
||||||
|
if as_json:
|
||||||
print(json.dumps(env, ensure_ascii=False))
|
print(json.dumps(env, ensure_ascii=False))
|
||||||
return 0
|
return 0
|
||||||
|
items = env["items"]
|
||||||
if not items:
|
if not items:
|
||||||
print("triage: inbox is empty — nothing to route.")
|
print("triage: inbox is empty — nothing to route.")
|
||||||
return 0
|
return 0
|
||||||
@@ -80,53 +86,76 @@ def list_inbox(as_json: bool = False) -> int:
|
|||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def apply(proposals: list[dict], confirm: bool = False) -> int:
|
def route_op(proposals: list[dict], confirm: bool = False) -> dict:
|
||||||
"""Validate -> classify -> preview; with confirm, route each via capture AND write
|
"""Core triage routing (Phase 0): reflect pipeline + processing-log audit lines.
|
||||||
the processing-log audit line. Mirrors echo_reflect.apply's contract exactly."""
|
Returns an envelope (rows = classified preview; results = per-item outcomes);
|
||||||
|
never prints to stdout (audit-append chatter routes to stderr)."""
|
||||||
|
import contextlib
|
||||||
|
import echo_output
|
||||||
import echo_reflect
|
import echo_reflect
|
||||||
valid, errors = echo_reflect.validate(proposals)
|
valid, errors = echo_reflect.validate(proposals)
|
||||||
for e in errors:
|
rows: list[dict] = []
|
||||||
print(f"skip: {e}", file=sys.stderr)
|
counts: dict[str, int] = {}
|
||||||
if not valid:
|
if valid:
|
||||||
print("triage: no valid proposals to route.")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
echo_reflect.classify(valid)
|
echo_reflect.classify(valid)
|
||||||
counts = {a: sum(1 for p in valid if p.get("_action") == a)
|
counts = {a: sum(1 for p in valid if p.get("_action") == a)
|
||||||
for a in ("create", "update", "inbox", "error")}
|
for a in ("create", "update", "inbox", "error")}
|
||||||
print(f"triage: {len(valid)} proposal(s) — "
|
rows = [{"action": p.get("_action"), "kind": p.get("kind"),
|
||||||
f"{counts['create']} new, {counts['update']} update, {counts['inbox']} stay-in-inbox"
|
"title": p["title"], "path": p.get("_path")} for p in valid]
|
||||||
+ (f", {counts['error']} error" if counts["error"] else ""))
|
data = {"proposals": len(proposals or []), "valid": len(valid), "errors": errors,
|
||||||
print(echo_reflect.preview(valid))
|
"counts": counts, "rows": rows, "dry_run": not confirm,
|
||||||
|
"routed": 0, "gated": 0, "log": f"{LOG_DIR}/{echo.today()}.md", "results": []}
|
||||||
if not confirm:
|
if not valid or not confirm:
|
||||||
print("\ntriage: dry-run — re-run with --apply to route these and log the moves.")
|
return echo_output.envelope("triage", data)
|
||||||
return 0
|
|
||||||
|
|
||||||
import echo_ops
|
import echo_ops
|
||||||
today_s = echo.today()
|
today_s = echo.today()
|
||||||
applied = gated = 0
|
routed = gated = 0
|
||||||
|
results = []
|
||||||
for p in valid:
|
for p in valid:
|
||||||
if p.get("_action") == "error":
|
if p.get("_action") in ("error", "inbox"):
|
||||||
continue
|
|
||||||
if p.get("_action") == "inbox":
|
|
||||||
continue # routing an inbox line back to the inbox is a no-op, not a move
|
continue # routing an inbox line back to the inbox is a no-op, not a move
|
||||||
bodyfile = echo.temp_file((p.get("body") or "").encode()) if p.get("body") else None
|
env = echo_ops.capture_op(
|
||||||
rc = echo_ops.capture(
|
p.get("kind"), p["title"], body_text=p.get("body") or "",
|
||||||
p.get("kind"), p["title"], bodyfile,
|
|
||||||
aliases=p.get("aliases") or [], sources=p.get("sources") or [],
|
aliases=p.get("aliases") or [], sources=p.get("sources") or [],
|
||||||
tags=p.get("tags") or [],
|
tags=p.get("tags") or [],
|
||||||
date=p.get("date"), domain=p.get("domain", "business"))
|
date=p.get("date"), domain=p.get("domain", "business"))
|
||||||
if rc == 76:
|
act = env.get("action", "")
|
||||||
|
results.append({"title": p["title"], "action": act,
|
||||||
|
"path": env.get("path"), "ok": bool(env.get("ok"))})
|
||||||
|
if act == "duplicate-gate":
|
||||||
gated += 1
|
gated += 1
|
||||||
continue
|
continue
|
||||||
if rc != 0:
|
if not env.get("ok"):
|
||||||
continue
|
continue
|
||||||
applied += 1
|
routed += 1
|
||||||
original = (p.get("line") or p["title"]).strip()
|
original = (p.get("line") or p["title"]).strip()
|
||||||
|
with contextlib.redirect_stdout(sys.stderr):
|
||||||
echo.cmd_append(f"{LOG_DIR}/{today_s}.md",
|
echo.cmd_append(f"{LOG_DIR}/{today_s}.md",
|
||||||
f"- {original} → {p.get('_path', '?')}")
|
f"- {original} → {p.get('_path', '?')}")
|
||||||
print(f"triage: routed {applied}/{len(valid)} item(s); audit in {LOG_DIR}/{today_s}.md. "
|
data.update(routed=routed, gated=gated, results=results)
|
||||||
|
return echo_output.envelope("triage", data)
|
||||||
|
|
||||||
|
|
||||||
|
def apply(proposals: list[dict], confirm: bool = False) -> int:
|
||||||
|
"""CLI wrapper: same preview/summary text as pre-Phase-0. Mirrors reflect's contract."""
|
||||||
|
env = route_op(proposals, confirm=confirm)
|
||||||
|
for e in env["errors"]:
|
||||||
|
print(f"skip: {e}", file=sys.stderr)
|
||||||
|
if not env["valid"]:
|
||||||
|
print("triage: no valid proposals to route.")
|
||||||
|
return 0
|
||||||
|
counts = env["counts"]
|
||||||
|
print(f"triage: {env['valid']} proposal(s) — "
|
||||||
|
f"{counts['create']} new, {counts['update']} update, {counts['inbox']} stay-in-inbox"
|
||||||
|
+ (f", {counts['error']} error" if counts["error"] else ""))
|
||||||
|
print("\n".join(f" {r['action'] or '?':7} | {('-' if r['action'] == 'inbox' else r['kind'] or '?'):9} "
|
||||||
|
f"| {r['title']} -> {r['path'] or '?'}" for r in env["rows"]))
|
||||||
|
if env["dry_run"]:
|
||||||
|
print("\ntriage: dry-run — re-run with --apply to route these and log the moves.")
|
||||||
|
return 0
|
||||||
|
gated = env["gated"]
|
||||||
|
print(f"triage: routed {env['routed']}/{env['valid']} item(s); audit in {env['log']}. "
|
||||||
"Originals kept in the inbox (deletion is explicit-only)."
|
"Originals kept in the inbox (deletion is explicit-only)."
|
||||||
+ (f" {gated} stopped at the duplicate gate — re-propose with the existing "
|
+ (f" {gated} stopped at the duplicate gate — re-propose with the existing "
|
||||||
f"entity's title or use capture --merge-into." if gated else ""))
|
f"entity's title or use capture --merge-into." if gated else ""))
|
||||||
|
|||||||
@@ -18,9 +18,19 @@ backfill what older vaults don't have yet:
|
|||||||
5. stamp the marker `schema_version` to the current schema (4).
|
5. stamp the marker `schema_version` to the current schema (4).
|
||||||
|
|
||||||
Dry-run by default; pass --apply to write. READ-ONLY without --apply.
|
Dry-run by default; pass --apply to write. READ-ONLY without --apply.
|
||||||
|
|
||||||
|
**--fast (2.3)** is the incremental mode: it walks the listing (cheap), then fetches
|
||||||
|
only what's new, gone, missing a content hash, or in today's rotating verification
|
||||||
|
shard (hash(path) % 7 == weekday — the whole vault gets verified over a week of
|
||||||
|
fast sweeps while each run stays tiny; --all-shards forces everything). It updates
|
||||||
|
the LOCAL recall index and the entity index (both machine-owned), so it needs no
|
||||||
|
--apply gate and writes no notes. `load` auto-runs it when the last fast sweep is
|
||||||
|
older than ECHO_FAST_SWEEP_DAYS (default 7). This is what keeps the indexes honest
|
||||||
|
against edits made directly in Obsidian or by other clients.
|
||||||
|
|
||||||
Cross-platform: pure Python via echo.py.
|
Cross-platform: pure Python via echo.py.
|
||||||
|
|
||||||
Usage: sweep.py [--apply]
|
Usage: sweep.py [--apply] | sweep.py --fast [--all-shards]
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -102,10 +112,128 @@ def walk(prefix: str = ""):
|
|||||||
yield from walk(f"{prefix}{d}/")
|
yield from walk(f"{prefix}{d}/")
|
||||||
|
|
||||||
|
|
||||||
|
def _stamp_path() -> Path:
|
||||||
|
import echo_queue
|
||||||
|
return echo_queue.state_dir() / "fast-sweep.stamp"
|
||||||
|
|
||||||
|
|
||||||
|
def _stamp_fast_sweep() -> None:
|
||||||
|
try:
|
||||||
|
p = _stamp_path()
|
||||||
|
p.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
p.write_text(echo.now_iso() + "\n", encoding="utf-8")
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def fast_sweep_age_days() -> float | None:
|
||||||
|
"""Days since the last fast/full sweep on THIS machine, or None if never."""
|
||||||
|
import datetime as dt
|
||||||
|
try:
|
||||||
|
stamp = _stamp_path().read_text(encoding="utf-8").strip()
|
||||||
|
then = dt.datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=dt.timezone.utc)
|
||||||
|
return (dt.datetime.now(dt.timezone.utc) - then).total_seconds() / 86400
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def fast(all_shards: bool = False) -> str:
|
||||||
|
"""Incremental index maintenance (2.3): true up the LOCAL recall index and the
|
||||||
|
entity index against the vault, fetching only new / gone / hash-missing notes
|
||||||
|
plus today's rotating verification shard. Index-only — writes no notes; the
|
||||||
|
indexes are machine-owned, so no --apply gate. Returns a one-line summary."""
|
||||||
|
import datetime as dt
|
||||||
|
import echo_recall
|
||||||
|
|
||||||
|
if get("_agent/echo-vault.md") is None:
|
||||||
|
return "fast-sweep skipped: vault not bootstrapped"
|
||||||
|
|
||||||
|
listing = [p for p in walk()
|
||||||
|
if p.endswith(".md") and p.rsplit("/", 1)[-1] not in SKIP_BASENAMES
|
||||||
|
and not TEMPLATE_RE.search(p)]
|
||||||
|
lset = set(listing)
|
||||||
|
|
||||||
|
rix = echo_recall._load_local()
|
||||||
|
if rix is None or not rix.n_docs:
|
||||||
|
# Nothing to maintain incrementally — build the local index outright
|
||||||
|
# (sub-second on a few-hundred-note vault; no vault snapshot write).
|
||||||
|
rix = echo_recall.rebuild()
|
||||||
|
_stamp_fast_sweep()
|
||||||
|
return f"fast-sweep: no local index — built one ({rix.n_docs} docs)"
|
||||||
|
|
||||||
|
index = idx_mod.load()
|
||||||
|
ents = index.get("entities", {})
|
||||||
|
removed = 0
|
||||||
|
for slug in [s for s, e in list(ents.items()) if e.get("path") not in lset]:
|
||||||
|
del ents[slug]
|
||||||
|
removed += 1
|
||||||
|
for p in [p for p in list(rix.length) if p not in lset]:
|
||||||
|
rix.remove(p)
|
||||||
|
removed += 1
|
||||||
|
|
||||||
|
try:
|
||||||
|
weekday = dt.date.fromisoformat(echo.today()).weekday()
|
||||||
|
except ValueError:
|
||||||
|
weekday = dt.date.today().weekday()
|
||||||
|
|
||||||
|
def in_shard(p: str) -> bool:
|
||||||
|
return int(idx_mod.content_hash(p), 16) % 7 == weekday
|
||||||
|
|
||||||
|
cands: set[str] = set()
|
||||||
|
for p in listing:
|
||||||
|
if not echo_recall._indexable(p):
|
||||||
|
continue
|
||||||
|
if p not in rix.length: # new note (any client, any tool)
|
||||||
|
cands.add(p)
|
||||||
|
continue
|
||||||
|
kind = kind_for(p)
|
||||||
|
if kind:
|
||||||
|
slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3])
|
||||||
|
if not (ents.get(slug) or {}).get("h"): # pre-2.3 entry: hash it once
|
||||||
|
cands.add(p)
|
||||||
|
continue
|
||||||
|
if all_shards or in_shard(p): # rotating verification shard
|
||||||
|
cands.add(p)
|
||||||
|
|
||||||
|
texts = echo.read_many(sorted(cands))
|
||||||
|
changed = 0
|
||||||
|
ents_dirty = removed > 0
|
||||||
|
for p, t in texts.items():
|
||||||
|
if t is None:
|
||||||
|
continue
|
||||||
|
h = idx_mod.content_hash(t)
|
||||||
|
if rix.content_hash(p) != h:
|
||||||
|
rix.add(p, t)
|
||||||
|
changed += 1
|
||||||
|
kind = kind_for(p)
|
||||||
|
if kind:
|
||||||
|
slug = idx_mod.slugify(p.rsplit("/", 1)[-1][:-3])
|
||||||
|
prev = ents.get(slug) or {}
|
||||||
|
if prev.get("h") != h or prev.get("path") != p:
|
||||||
|
title = links.first_h1(t) or p.rsplit("/", 1)[-1][:-3]
|
||||||
|
aliases = list(prev.get("aliases", [])) + idx_mod.aliases_in_frontmatter(t)
|
||||||
|
idx_mod.upsert(index, slug, p, kind, title, aliases, h=h)
|
||||||
|
ents_dirty = True
|
||||||
|
|
||||||
|
echo_recall.save_local(rix)
|
||||||
|
if ents_dirty:
|
||||||
|
idx_mod.save(index)
|
||||||
|
_stamp_fast_sweep()
|
||||||
|
return (f"fast-sweep: {len(cands)} note(s) checked, {changed} reindexed, "
|
||||||
|
f"{removed} removed" + (" (all shards)" if all_shards else ""))
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str] | None = None) -> int:
|
def main(argv: list[str] | None = None) -> int:
|
||||||
ap = argparse.ArgumentParser(description="Bring an ECHO vault up to the current feature spec")
|
ap = argparse.ArgumentParser(description="Bring an ECHO vault up to the current feature spec")
|
||||||
ap.add_argument("--apply", action="store_true")
|
ap.add_argument("--apply", action="store_true")
|
||||||
|
ap.add_argument("--fast", action="store_true",
|
||||||
|
help="incremental index maintenance (local recall + entity index only)")
|
||||||
|
ap.add_argument("--all-shards", action="store_true",
|
||||||
|
help="with --fast: verify every indexed note, not just today's shard")
|
||||||
args = ap.parse_args(argv)
|
args = ap.parse_args(argv)
|
||||||
|
if args.fast:
|
||||||
|
print(fast(all_shards=args.all_shards))
|
||||||
|
return 0
|
||||||
apply = args.apply
|
apply = args.apply
|
||||||
tag = "APPLY" if apply else "PLAN "
|
tag = "APPLY" if apply else "PLAN "
|
||||||
|
|
||||||
@@ -151,7 +279,8 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
# aliases (e.g. mentions learned by capture) are preserved; upsert adds title variants.
|
# aliases (e.g. mentions learned by capture) are preserved; upsert adds title variants.
|
||||||
prev_aliases = prev.get("aliases", []) if prev else []
|
prev_aliases = prev.get("aliases", []) if prev else []
|
||||||
aliases = list(prev_aliases) + idx_mod.aliases_in_frontmatter(text)
|
aliases = list(prev_aliases) + idx_mod.aliases_in_frontmatter(text)
|
||||||
idx_mod.upsert(rebuilt, slug, path, kind, title, aliases)
|
idx_mod.upsert(rebuilt, slug, path, kind, title, aliases,
|
||||||
|
h=idx_mod.content_hash(text))
|
||||||
if not prev:
|
if not prev:
|
||||||
added += 1
|
added += 1
|
||||||
elif prev.get("path") != path or prev.get("title") != title:
|
elif prev.get("path") != path or prev.get("title") != title:
|
||||||
@@ -163,8 +292,9 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
|
|
||||||
# ---- 2b. (re)build the recall (BM25) index -------------------------------
|
# ---- 2b. (re)build the recall (BM25) index -------------------------------
|
||||||
if apply:
|
if apply:
|
||||||
rix = echo_recall.rebuild(prefetched=texts)
|
rix = echo_recall.rebuild(prefetched=texts, snapshot=True)
|
||||||
print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25)")
|
_stamp_fast_sweep()
|
||||||
|
print(f"sweep: {tag} recall index — {rix.n_docs} docs (BM25; local + vault snapshot)")
|
||||||
else:
|
else:
|
||||||
n_idx = sum(1 for p in all_files if echo_recall._indexable(p))
|
n_idx = sum(1 for p in all_files if echo_recall._indexable(p))
|
||||||
print(f"sweep: {tag} recall index — would rebuild BM25 over {n_idx} corpus notes "
|
print(f"sweep: {tag} recall index — would rebuild BM25 over {n_idx} corpus notes "
|
||||||
|
|||||||
@@ -215,6 +215,42 @@ def test_links_create_related_section_when_absent() -> None:
|
|||||||
assert changed and "## Related" in new and "[[resources/concepts/beta]]" in new
|
assert changed and "## Related" in new and "[[resources/concepts/beta]]" in new
|
||||||
|
|
||||||
|
|
||||||
|
def test_stemmer_conflates_inflection_families() -> None:
|
||||||
|
import echo_stem
|
||||||
|
families = [
|
||||||
|
["deploy", "deploys", "deployed", "deploying", "deployment"],
|
||||||
|
["navigate", "navigation", "navigating", "navigational"],
|
||||||
|
["renew", "renews", "renewing"],
|
||||||
|
["prefer", "prefers", "preferring", "preferred"],
|
||||||
|
["penalty", "penalties"],
|
||||||
|
["saturate", "saturation", "saturated"],
|
||||||
|
["frequency", "frequencies"],
|
||||||
|
["expand", "expanded", "expanding"],
|
||||||
|
["meet", "meeting", "meetings"],
|
||||||
|
["service", "services"],
|
||||||
|
]
|
||||||
|
for fam in families:
|
||||||
|
stems = {echo_stem.stem(w) for w in fam}
|
||||||
|
assert len(stems) == 1, f"{fam} -> {stems}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_stemmer_leaves_short_and_risky_tokens_alone() -> None:
|
||||||
|
import echo_stem
|
||||||
|
# over-stemming merges unrelated words — worse than a missed match
|
||||||
|
for w in ("sing", "thing", "was", "is", "bus", "this", "echo", "vault", "mpm"):
|
||||||
|
assert echo_stem.stem(w) == w, f"{w} mangled to {echo_stem.stem(w)}"
|
||||||
|
# documented non-conflations (the -al / d~s irregulars full Porter also skips)
|
||||||
|
assert echo_stem.stem("renewal") != echo_stem.stem("renew")
|
||||||
|
assert echo_stem.stem("expansion") != echo_stem.stem("expand")
|
||||||
|
|
||||||
|
|
||||||
|
def test_content_hash_stable_and_short() -> None:
|
||||||
|
import echo_index as ix
|
||||||
|
a = ix.content_hash("hello world")
|
||||||
|
assert a == ix.content_hash("hello world") and len(a) == 16
|
||||||
|
assert a != ix.content_hash("hello world!")
|
||||||
|
|
||||||
|
|
||||||
def _run_all() -> int:
|
def _run_all() -> int:
|
||||||
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
|
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
|
||||||
failed = 0
|
failed = 0
|
||||||
|
|||||||
@@ -30,4 +30,9 @@ python3 "$ECHO" reflect proposals.json --apply # write — routes each via ca
|
|||||||
`reflect` dedups every proposal against the entity index (so it shows create-vs-update),
|
`reflect` dedups every proposal against the entity index (so it shows create-vs-update),
|
||||||
skips below-confidence items, and routes `inbox` proposals to the capture inbox. Each
|
skips below-confidence items, and routes `inbox` proposals to the capture inbox. Each
|
||||||
applied item goes through the normal `capture` path — canonical frontmatter, auto-linking,
|
applied item goes through the normal `capture` path — canonical frontmatter, auto-linking,
|
||||||
and the recall index — and the writes are lock-guarded. $ARGUMENTS
|
and the recall index — and the writes are lock-guarded.
|
||||||
|
|
||||||
|
**Finishing the session?** Prefer the one-call bundle — `python3 "$ECHO" session-end
|
||||||
|
bundle.json --apply` — which applies the reflect proposals AND writes the session log,
|
||||||
|
Agent Log line, optional scope switch, and the heartbeat (last, as the commit marker)
|
||||||
|
together. See the echo-memory skill's **Session Logging** section for the bundle shape. $ARGUMENTS
|
||||||
|
|||||||
@@ -3,17 +3,17 @@
|
|||||||
"date": "2026-07-03",
|
"date": "2026-07-03",
|
||||||
"retrieval": {
|
"retrieval": {
|
||||||
"current (hybrid+priors, full corpus)": {
|
"current (hybrid+priors, full corpus)": {
|
||||||
"precision_at_5": 0.25,
|
"precision_at_5": 0.243,
|
||||||
"recall_at_5": 1.0,
|
"recall_at_5": 1.0,
|
||||||
"mrr": 1.0,
|
"mrr": 1.0,
|
||||||
"session_journal_queries_answered": "2/2",
|
"session_journal_queries_answered": "3/3",
|
||||||
"freshness_top1_correct": true
|
"freshness_top1_correct": true
|
||||||
},
|
},
|
||||||
"baseline (keyword, entities only)": {
|
"baseline (keyword, entities only)": {
|
||||||
"precision_at_5": 0.2,
|
"precision_at_5": 0.186,
|
||||||
"recall_at_5": 0.75,
|
"recall_at_5": 0.75,
|
||||||
"mrr": 0.75,
|
"mrr": 0.786,
|
||||||
"session_journal_queries_answered": "0/2",
|
"session_journal_queries_answered": "0/3",
|
||||||
"freshness_top1_correct": false
|
"freshness_top1_correct": false
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -135,6 +135,16 @@ QUERIES = [
|
|||||||
("contract renewal Q3", {"resources/companies/vantage-systems.md"}, False),
|
("contract renewal Q3", {"resources/companies/vantage-systems.md"}, False),
|
||||||
("renegotiation kickoff", {"journal/daily/2026-06-20.md"}, True),
|
("renegotiation kickoff", {"journal/daily/2026-06-20.md"}, True),
|
||||||
("graph neighbourhood expansion", {"resources/concepts/graph-expansion.md"}, False),
|
("graph neighbourhood expansion", {"resources/concepts/graph-expansion.md"}, False),
|
||||||
|
# ---- paraphrase set (2.3): morphological variants + rephrasings the pre-2.3
|
||||||
|
# purely-lexical index missed — the stemming gain is measured, not assumed.
|
||||||
|
("renewing enterprise contracts", {"resources/companies/vantage-systems.md"}, False),
|
||||||
|
("navigational cleanup", {"projects/active/website-redesign.md",
|
||||||
|
"projects/archived/old-website.md"}, False),
|
||||||
|
("preferring uv for tooling", {"_agent/memory/semantic/tooling-preferences.md"}, False),
|
||||||
|
("penalties in the SLA",
|
||||||
|
{"_agent/sessions/2026-06-20-1000-vantage-contract-review.md"}, True),
|
||||||
|
("saturated term frequencies", {"resources/concepts/bm25-ranking.md"}, False),
|
||||||
|
("expanded linked neighbourhoods", {"resources/concepts/graph-expansion.md"}, False),
|
||||||
]
|
]
|
||||||
FRESHNESS_QUERY = "marketing site navigation"
|
FRESHNESS_QUERY = "marketing site navigation"
|
||||||
FRESHNESS_TOP1 = "projects/active/website-redesign.md"
|
FRESHNESS_TOP1 = "projects/active/website-redesign.md"
|
||||||
|
|||||||
+118
-3
@@ -27,7 +27,9 @@ def check(name, cond, detail=""):
|
|||||||
|
|
||||||
class Harness:
|
class Harness:
|
||||||
def __init__(self, base):
|
def __init__(self, base):
|
||||||
|
import tempfile
|
||||||
self.base = base
|
self.base = base
|
||||||
|
self.state_dir = tempfile.mkdtemp()
|
||||||
|
|
||||||
def http(self, method, url, body=None, headers=None):
|
def http(self, method, url, body=None, headers=None):
|
||||||
data = body.encode() if isinstance(body, str) else body
|
data = body.encode() if isinstance(body, str) else body
|
||||||
@@ -52,8 +54,10 @@ class Harness:
|
|||||||
return None if body == "<<MISSING>>" else body
|
return None if body == "<<MISSING>>" else body
|
||||||
|
|
||||||
def echo(self, *args):
|
def echo(self, *args):
|
||||||
|
# ECHO_STATE_DIR isolation matters since 2.3: the recall index is LOCAL-first,
|
||||||
|
# and without this the suite would write into the real ~/.echo-memory.
|
||||||
env = dict(os.environ, ECHO_BASE=self.base, ECHO_KEY=KEY, ECHO_VERIFY="1",
|
env = dict(os.environ, ECHO_BASE=self.base, ECHO_KEY=KEY, ECHO_VERIFY="1",
|
||||||
ECHO_TODAY="2026-06-21")
|
ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=self.state_dir)
|
||||||
return subprocess.run([sys.executable, str(args[0]), *args[1:]],
|
return subprocess.run([sys.executable, str(args[0]), *args[1:]],
|
||||||
capture_output=True, text=True, env=env)
|
capture_output=True, text=True, env=env)
|
||||||
|
|
||||||
@@ -230,8 +234,9 @@ def main():
|
|||||||
check("v1.5 recall finds a session log by body term",
|
check("v1.5 recall finds a session log by body term",
|
||||||
"_agent/sessions/2026-06-15-1200-mpm-review.md" in r.stdout, r.stdout)
|
"_agent/sessions/2026-06-15-1200-mpm-review.md" in r.stdout, r.stdout)
|
||||||
rix2 = h.ground("_agent/index/recall-index.json") or ""
|
rix2 = h.ground("_agent/index/recall-index.json") or ""
|
||||||
check("v1.5 recall index carries doc meta (schema 2)",
|
check("v2.3 recall snapshot is schema 3 with build stamp + content hashes",
|
||||||
'"schema": 2' in rix2 or '"schema":2' in rix2, rix2[:200])
|
('"schema": 3' in rix2 or '"schema":3' in rix2)
|
||||||
|
and '"built"' in rix2 and '"h"' in rix2, rix2[:200])
|
||||||
|
|
||||||
# 14. recency-aware ranking: same term, fresher note wins.
|
# 14. recency-aware ranking: same term, fresher note wins.
|
||||||
h.seed("resources/concepts/old-idea.md",
|
h.seed("resources/concepts/old-idea.md",
|
||||||
@@ -345,6 +350,116 @@ def main():
|
|||||||
check("v1.5 session-start hook stays quiet on resume",
|
check("v1.5 session-start hook stays quiet on resume",
|
||||||
r.returncode == 0 and not r.stdout.strip(), r.stdout)
|
r.returncode == 0 and not r.stdout.strip(), r.stdout)
|
||||||
|
|
||||||
|
# 18. (2.1.0) brief load: digest form, budget respected, key facts present.
|
||||||
|
h.seed("_agent/memory/semantic/operator-preferences.md",
|
||||||
|
"---\ntype: semantic-memory\ncreated: 2026-06-01\n---\n# Operator Preferences\n\n"
|
||||||
|
"## Fact / Pattern\n- prefers concise output\n\n## Observations\n"
|
||||||
|
+ "\n".join(f"- 2026-06-{i:02d}: observation {i}" for i in range(1, 15)) + "\n")
|
||||||
|
h.seed("inbox/captures/inbox.md", "- 2026-06-01: old capture\n- 2026-06-20: new capture\n")
|
||||||
|
r = h.echo(ECHO, "load", "--brief")
|
||||||
|
check("v2.1 brief load renders the digest header",
|
||||||
|
"ECHO load (brief)" in r.stdout, r.stdout[:200])
|
||||||
|
check("v2.1 brief load keeps Fact / Pattern",
|
||||||
|
"prefers concise output" in r.stdout, r.stdout[:800])
|
||||||
|
check("v2.1 brief load trims observations to the last 10",
|
||||||
|
"last 10 of 14" in r.stdout and "observation 14" in r.stdout
|
||||||
|
and "observation 2" not in r.stdout.replace("observation 2026", ""), r.stdout[-1200:])
|
||||||
|
check("v2.1 brief load reports inbox as a count",
|
||||||
|
"inbox: 2 capture(s), oldest 20d" in r.stdout, r.stdout[-400:])
|
||||||
|
check("v2.1 brief load includes the marker line",
|
||||||
|
"echo-vault.md" in r.stdout and "marker" in r.stdout, r.stdout[:300])
|
||||||
|
|
||||||
|
# 19. (2.1.0) session-end: dry-run writes nothing; apply lands all steps with
|
||||||
|
# the heartbeat LAST; a bad bundle aborts before any write.
|
||||||
|
se_env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1",
|
||||||
|
ECHO_TODAY="2026-06-21", ECHO_NOW="2359")
|
||||||
|
bundle = {"slug": "quickwins-ship",
|
||||||
|
"log_body": "---\ntype: session-log\nstatus: complete\ncreated: 2026-06-21\n---\n"
|
||||||
|
"# Session Log\n\n## Goal\nShip the quick wins.\n",
|
||||||
|
"agent_log_line": "- 2026-06-21: shipped the quick wins",
|
||||||
|
"reflect": [{"title": "Quark Preference", "kind": "semantic",
|
||||||
|
"body": "The operator prefers quark.", "confidence": 0.9}]}
|
||||||
|
bfile = HERE / "_session_bundle.json"
|
||||||
|
bfile.write_text(json.dumps(bundle), encoding="utf-8")
|
||||||
|
sess_path = "_agent/sessions/2026-06-21-2359-quickwins-ship.md"
|
||||||
|
r = subprocess.run([sys.executable, str(ECHO), "session-end", str(bfile)],
|
||||||
|
capture_output=True, text=True, env=se_env)
|
||||||
|
check("v2.1 session-end dry-run previews the plan",
|
||||||
|
"dry-run" in r.stdout and sess_path in r.stdout, r.stdout)
|
||||||
|
check("v2.1 session-end dry-run writes nothing", h.ground(sess_path) is None)
|
||||||
|
r = subprocess.run([sys.executable, str(ECHO), "session-end", str(bfile), "--apply"],
|
||||||
|
capture_output=True, text=True, env=se_env)
|
||||||
|
check("v2.1 session-end apply lands the session log",
|
||||||
|
"Ship the quick wins" in (h.ground(sess_path) or ""), r.stdout + r.stderr)
|
||||||
|
check("v2.1 session-end apply routes the reflect proposal",
|
||||||
|
h.ground("_agent/memory/semantic/quark-preference.md") is not None)
|
||||||
|
check("v2.1 session-end apply writes the heartbeat last (commit marker)",
|
||||||
|
sess_path in (h.ground("_agent/heartbeat/last-session.md") or ""))
|
||||||
|
daily = h.ground("journal/daily/2026-06-21.md") or ""
|
||||||
|
check("v2.1 session-end apply appends the agent-log line",
|
||||||
|
"shipped the quick wins" in daily, daily[-300:])
|
||||||
|
bad = dict(bundle, slug="Bad_Slug!")
|
||||||
|
bfile.write_text(json.dumps(bad), encoding="utf-8")
|
||||||
|
r = subprocess.run([sys.executable, str(ECHO), "session-end", str(bfile), "--apply"],
|
||||||
|
capture_output=True, text=True, env=se_env)
|
||||||
|
bfile.unlink()
|
||||||
|
check("v2.1 session-end rejects a bad slug before any write",
|
||||||
|
r.returncode == 2 and sess_path in (h.ground("_agent/heartbeat/last-session.md") or ""),
|
||||||
|
r.stderr)
|
||||||
|
|
||||||
|
# ---------------- v2.3 index train --------------------------------------
|
||||||
|
# 20. local-first recall index: capture updates the LOCAL index with ZERO
|
||||||
|
# vault snapshot writes — the vault copy only changes on sweep/session-end.
|
||||||
|
snap_before = h.ground("_agent/index/recall-index.json") or ""
|
||||||
|
r = h.echo(ECHO, "capture", "Quixotic Widget", "--kind", "concept", "-")
|
||||||
|
r = subprocess.run([sys.executable, str(ECHO), "capture", "Quixotic Widget2", "--kind",
|
||||||
|
"concept", "-"], input="A gadget for quixotry deployment.\n",
|
||||||
|
capture_output=True, text=True,
|
||||||
|
env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY,
|
||||||
|
ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=h.state_dir))
|
||||||
|
snap_after = h.ground("_agent/index/recall-index.json") or ""
|
||||||
|
check("v2.3 capture does NOT write the vault snapshot", snap_before == snap_after)
|
||||||
|
local_files = [p for p in Path(h.state_dir).glob("recall-index-*.json")]
|
||||||
|
check("v2.3 capture maintains the LOCAL index", len(local_files) == 1,
|
||||||
|
str(local_files))
|
||||||
|
r = h.echo(ECHO, "recall", "quixotry")
|
||||||
|
check("v2.3 locally-indexed capture is recallable",
|
||||||
|
"resources/concepts/quixotic-widget2.md" in r.stdout, r.stdout[:400])
|
||||||
|
|
||||||
|
# 21. stemming: a morphologically different query still hits (deployment ~ deploying).
|
||||||
|
r = h.echo(ECHO, "recall", "deploying quixotry")
|
||||||
|
check("v2.3 stemmed recall bridges inflections",
|
||||||
|
"resources/concepts/quixotic-widget2.md" in r.stdout, r.stdout[:400])
|
||||||
|
|
||||||
|
# 22. alias query expansion: an alias-phrased query surfaces the entity even
|
||||||
|
# though the alias never appears in any note body.
|
||||||
|
r = subprocess.run([sys.executable, str(ECHO), "capture", "Kappa Engine", "--kind",
|
||||||
|
"concept", "-", "--aliases", "turbokappa"],
|
||||||
|
input="Compression tuning notes for the kappa engine core.\n",
|
||||||
|
capture_output=True, text=True,
|
||||||
|
env=dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY,
|
||||||
|
ECHO_TODAY="2026-06-21", ECHO_STATE_DIR=h.state_dir))
|
||||||
|
r = h.echo(ECHO, "recall", "turbokappa compression")
|
||||||
|
check("v2.3 alias expansion surfaces the aliased entity",
|
||||||
|
"resources/concepts/kappa-engine.md" in r.stdout, r.stdout[:400])
|
||||||
|
|
||||||
|
# 23. incremental fast sweep: an out-of-band edit (seeded directly, as if by
|
||||||
|
# Obsidian) and a deletion are trued up without a full rebuild.
|
||||||
|
h.seed("resources/concepts/quixotic-widget.md",
|
||||||
|
"---\ntype: concept\nstatus: active\ncreated: 2026-06-21\nupdated: 2026-06-21\n"
|
||||||
|
"tags: [concept]\n---\n# Quixotic Widget\n\nNow mentions zanzibar explicitly.\n")
|
||||||
|
h.http("DELETE", f"{base}/vault/resources/concepts/quixotic-widget2.md")
|
||||||
|
r = h.echo(SWEEP, "--fast", "--all-shards")
|
||||||
|
check("v2.3 fast sweep reports its work",
|
||||||
|
"fast-sweep:" in r.stdout and "reindexed" in r.stdout, r.stdout + r.stderr)
|
||||||
|
r = h.echo(ECHO, "recall", "zanzibar")
|
||||||
|
check("v2.3 fast sweep picked up the out-of-band edit",
|
||||||
|
"resources/concepts/quixotic-widget.md" in r.stdout, r.stdout[:400])
|
||||||
|
idx5 = h.ground("_agent/index/entities.json") or ""
|
||||||
|
check("v2.3 fast sweep drops deleted notes from the entity index",
|
||||||
|
"quixotic-widget2" not in idx5)
|
||||||
|
check("v2.3 entity index carries content hashes", '"h"' in idx5, idx5[:300])
|
||||||
|
|
||||||
print(f"\n{len(failures)} failure(s)" if failures else "\nall feature tests passed")
|
print(f"\n{len(failures)} failure(s)" if failures else "\nall feature tests passed")
|
||||||
return 1 if failures else 0
|
return 1 if failures else 0
|
||||||
finally:
|
finally:
|
||||||
|
|||||||
@@ -0,0 +1,171 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""test_mcp_server.py — echo-mcp (2.2) end-to-end against the mock vault.
|
||||||
|
|
||||||
|
Spawns mcp-server/app.py + mock_olrapi and drives the real streamable-HTTP MCP
|
||||||
|
protocol: health (open), auth (401), initialize, tools/list, and the memory-day
|
||||||
|
tools/call flow (capture -> duplicate gate as DATA -> merge_into -> recall ->
|
||||||
|
get_note -> log_session with heartbeat commit).
|
||||||
|
|
||||||
|
Requires the `mcp` package in the interpreter that runs the SERVER. Set
|
||||||
|
ECHO_MCP_PYTHON to a venv python that has it; otherwise the current interpreter
|
||||||
|
is tried and the suite SKIPS (exit 0) when the SDK is absent — the other suites
|
||||||
|
don't depend on it.
|
||||||
|
|
||||||
|
Run: python test_mcp_server.py [--port 8862] [--mcp-port 8767]
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
HERE = Path(__file__).resolve().parent
|
||||||
|
APP = HERE.parent / "mcp-server" / "app.py"
|
||||||
|
KEY = "test-key-not-a-real-secret"
|
||||||
|
TOKEN = "local-test-token"
|
||||||
|
|
||||||
|
failures = []
|
||||||
|
|
||||||
|
|
||||||
|
def check(name, cond, detail=""):
|
||||||
|
print(f"{'ok ' if cond else 'FAIL'} {name}" + (f" -- {detail}" if not cond else ""))
|
||||||
|
if not cond:
|
||||||
|
failures.append(name)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--port", type=int, default=8862)
|
||||||
|
ap.add_argument("--mcp-port", type=int, default=8767)
|
||||||
|
a = ap.parse_args()
|
||||||
|
mock_base = f"http://127.0.0.1:{a.port}"
|
||||||
|
mcp_url = f"http://127.0.0.1:{a.mcp_port}/mcp"
|
||||||
|
|
||||||
|
server_py = os.environ.get("ECHO_MCP_PYTHON") or sys.executable
|
||||||
|
probe = subprocess.run([server_py, "-c", "import mcp"], capture_output=True)
|
||||||
|
if probe.returncode != 0:
|
||||||
|
print("SKIP: `mcp` SDK not installed for the server interpreter "
|
||||||
|
"(set ECHO_MCP_PYTHON to a venv python that has it)")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
def http(method, url, body=None, headers=None, timeout=30):
|
||||||
|
data = body.encode() if isinstance(body, str) else body
|
||||||
|
req = urllib.request.Request(url, data=data, method=method, headers=headers or {})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||||
|
return r.status, r.read().decode("utf-8", "replace")
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
return e.code, e.read().decode("utf-8", "replace")
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
return 0, str(e)
|
||||||
|
|
||||||
|
_id = [0]
|
||||||
|
|
||||||
|
def rpc(method, params):
|
||||||
|
_id[0] += 1
|
||||||
|
st, body = http("POST", mcp_url, json.dumps(
|
||||||
|
{"jsonrpc": "2.0", "id": _id[0], "method": method, "params": params}),
|
||||||
|
headers={"Content-Type": "application/json",
|
||||||
|
"Accept": "application/json, text/event-stream",
|
||||||
|
"Authorization": f"Bearer {TOKEN}"})
|
||||||
|
return st, (json.loads(body) if body.strip().startswith("{") else {})
|
||||||
|
|
||||||
|
def call(name, args):
|
||||||
|
st, d = rpc("tools/call", {"name": name, "arguments": args})
|
||||||
|
res = d.get("result") or {}
|
||||||
|
payload = {}
|
||||||
|
if res.get("content"):
|
||||||
|
try:
|
||||||
|
payload = json.loads(res["content"][0]["text"])
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
payload = {}
|
||||||
|
return res.get("isError", False), payload
|
||||||
|
|
||||||
|
mock = subprocess.Popen([sys.executable, str(HERE / "mock_olrapi.py"), "--port", str(a.port)],
|
||||||
|
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
||||||
|
srv = None
|
||||||
|
try:
|
||||||
|
for _ in range(50):
|
||||||
|
try:
|
||||||
|
urllib.request.urlopen(f"{mock_base}/__debug__reset", data=b"", timeout=1)
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
time.sleep(0.1)
|
||||||
|
http("PUT", f"{mock_base}/vault/_agent/echo-vault.md",
|
||||||
|
"---\nschema_version: 4\n---\n# marker\n",
|
||||||
|
headers={"Authorization": f"Bearer {KEY}"})
|
||||||
|
|
||||||
|
env = dict(os.environ, ECHO_BASE=mock_base, ECHO_KEY=KEY, ECHO_MCP_TOKEN=TOKEN,
|
||||||
|
ECHO_MCP_PORT=str(a.mcp_port), ECHO_STATE_DIR=tempfile.mkdtemp(),
|
||||||
|
ECHO_TODAY="2026-07-28")
|
||||||
|
srv = subprocess.Popen([server_py, str(APP)], env=env,
|
||||||
|
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
||||||
|
for _ in range(60):
|
||||||
|
st, _b = http("GET", f"http://127.0.0.1:{a.mcp_port}/health", timeout=2)
|
||||||
|
if st == 200:
|
||||||
|
break
|
||||||
|
time.sleep(0.5)
|
||||||
|
|
||||||
|
st, body = http("GET", f"http://127.0.0.1:{a.mcp_port}/health")
|
||||||
|
try:
|
||||||
|
hb = json.loads(body)
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
hb = {}
|
||||||
|
check("health is open and green", st == 200 and hb.get("ok") is True
|
||||||
|
and hb.get("vault_reachable") is True, body[:200])
|
||||||
|
st, _b = http("POST", mcp_url, "{}", headers={"Content-Type": "application/json"})
|
||||||
|
check("MCP endpoint requires the bearer token (401)", st == 401, str(st))
|
||||||
|
|
||||||
|
st, d = rpc("initialize", {"protocolVersion": "2025-06-18", "capabilities": {},
|
||||||
|
"clientInfo": {"name": "eval", "version": "0"}})
|
||||||
|
check("initialize answers with serverInfo",
|
||||||
|
d.get("result", {}).get("serverInfo", {}).get("name") == "echo-mcp", json.dumps(d)[:200])
|
||||||
|
st, d = rpc("tools/list", {})
|
||||||
|
tools = [t["name"] for t in d.get("result", {}).get("tools", [])]
|
||||||
|
check("tools/list exposes the full profile (14 tools)", len(tools) == 14, str(tools))
|
||||||
|
|
||||||
|
e, b = call("echo_capture", {"title": "Vera Lumen", "kind": "person",
|
||||||
|
"body": "CTO at Fluxcorp, met at the summit."})
|
||||||
|
check("capture create over MCP", not e and b.get("action") == "created"
|
||||||
|
and b.get("path") == "resources/people/vera-lumen.md", json.dumps(b))
|
||||||
|
e, b = call("echo_capture", {"title": "Vera Lumen Jr", "kind": "person"})
|
||||||
|
check("duplicate gate is data, not isError", not e
|
||||||
|
and b.get("action") == "duplicate-gate" and b.get("candidates"), json.dumps(b))
|
||||||
|
e, b = call("echo_capture", {"title": "Vera Lumen Jr", "kind": "person",
|
||||||
|
"merge_into": "vera-lumen", "body": "Follow-up."})
|
||||||
|
check("merge_into resolves the gate as an update", not e and b.get("action") == "updated",
|
||||||
|
json.dumps(b))
|
||||||
|
e, b = call("echo_recall", {"query": "fluxcorp", "budget_chars": 900})
|
||||||
|
check("recall over MCP finds by body term", not e and any(
|
||||||
|
h["path"] == "resources/people/vera-lumen.md" for h in b.get("primary", [])),
|
||||||
|
json.dumps(b)[:300])
|
||||||
|
e, b = call("echo_get_note", {"path": "resources/people/vera-lumen.md"})
|
||||||
|
check("get_note returns frontmatter + content", not e
|
||||||
|
and b.get("frontmatter", {}).get("type") == "person", json.dumps(b)[:200])
|
||||||
|
e, b = call("echo_get_note", {"path": "../etc/passwd"})
|
||||||
|
check("get_note rejects path traversal", not b.get("ok"))
|
||||||
|
e, b = call("echo_log_session", {"slug": "mcp-e2e", "hhmm": "2345", "apply": True,
|
||||||
|
"log_body": "---\ntype: session-log\n---\n# S\n\n## Goal\ne2e\n"})
|
||||||
|
check("log_session commits with the heartbeat", not e
|
||||||
|
and b.get("steps", {}).get("heartbeat") == "ok", json.dumps(b))
|
||||||
|
e, b = call("echo_capture", {"title": "Bad Kind", "kind": "wizard"})
|
||||||
|
check("unknown kind rejected with the valid list", not b.get("ok")
|
||||||
|
and "wizard" in b.get("error", ""), json.dumps(b))
|
||||||
|
|
||||||
|
print(f"\n{len(failures)} failure(s)" if failures else "\nall mcp-server tests passed")
|
||||||
|
return 1 if failures else 0
|
||||||
|
finally:
|
||||||
|
if srv:
|
||||||
|
srv.terminate()
|
||||||
|
mock.terminate()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -115,6 +115,40 @@ def main():
|
|||||||
check("offline load flags OFFLINE", "OFFLINE" in r.stdout, r.stdout)
|
check("offline load flags OFFLINE", "OFFLINE" in r.stdout, r.stdout)
|
||||||
check("offline load serves cached marker", "EVALMARK-marker" in r.stdout, r.stdout)
|
check("offline load serves cached marker", "EVALMARK-marker" in r.stdout, r.stdout)
|
||||||
|
|
||||||
|
# --- Phase 5 (2.1.0): vault DOWN -> capture queues a SEMANTIC record ------
|
||||||
|
r = echo(DEAD, "capture", "Offline Person", "--kind", "person")
|
||||||
|
check("offline capture exits 0 (queued)", r.returncode == 0, r.stdout + r.stderr)
|
||||||
|
check("offline capture reports queued", "queued (offline): capture" in r.stdout, r.stdout)
|
||||||
|
outbox = Path(state) / "outbox.ndjson"
|
||||||
|
check("capture queued as an op record",
|
||||||
|
outbox.exists() and '"op": "capture"' in outbox.read_text(encoding="utf-8"))
|
||||||
|
# same capture again while offline -> deduped by idem_key, not double-queued
|
||||||
|
echo(DEAD, "capture", "Offline Person", "--kind", "person")
|
||||||
|
recs = [ln for ln in outbox.read_text(encoding="utf-8").splitlines() if '"op": "capture"' in ln]
|
||||||
|
check("offline capture is idempotent in the queue", len(recs) == 1, str(len(recs)))
|
||||||
|
|
||||||
|
# --- Phase 6 (2.1.0): vault UP -> flush replays capture THROUGH capture ---
|
||||||
|
srv = start_mock()
|
||||||
|
r = echo(mock_base, "flush")
|
||||||
|
check("flush replays the queued capture", "flushed" in r.stdout, r.stdout + r.stderr)
|
||||||
|
note = ground("resources/people/offline-person.md")
|
||||||
|
check("replayed capture created the routed note",
|
||||||
|
note is not None and "type: person" in note, str(note)[:200])
|
||||||
|
check("replayed capture used the capture-time date",
|
||||||
|
note is not None and "created: 2026-06-22" in note, str(note)[:200])
|
||||||
|
|
||||||
|
# --- Phase 7 (2.1.0): duplicate gate ON REPLAY keeps + flags the record ---
|
||||||
|
echo(mock_base, "capture", "Zebulon Quargle", "--kind", "person") # the existing entity
|
||||||
|
r = echo(DEAD, "capture", "Zebulon Quargle Junior", "--kind", "person")
|
||||||
|
check("lookalike capture queues offline", "queued (offline)" in r.stdout, r.stdout)
|
||||||
|
r = echo(mock_base, "flush")
|
||||||
|
check("gated replay is flagged, not landed",
|
||||||
|
"needs attention" in r.stdout and "replay-gated" in r.stdout, r.stdout + r.stderr)
|
||||||
|
check("gated replay did NOT create the duplicate",
|
||||||
|
ground("resources/people/zebulon-quargle-junior.md") is None)
|
||||||
|
check("gated record survives in the outbox",
|
||||||
|
'"needs_attention"' in (outbox.read_text(encoding="utf-8") if outbox.exists() else ""))
|
||||||
|
|
||||||
print(f"\n{len(failures)} failure(s)" if failures else "\nall offline-queue tests passed")
|
print(f"\n{len(failures)} failure(s)" if failures else "\nall offline-queue tests passed")
|
||||||
return 1 if failures else 0
|
return 1 if failures else 0
|
||||||
finally:
|
finally:
|
||||||
|
|||||||
@@ -0,0 +1,182 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""test_ops_api.py — Phase 0 (return-not-print) contract: every high-level op has a
|
||||||
|
core `*_op` function that RETURNS an envelope dict and prints nothing to stdout.
|
||||||
|
|
||||||
|
This is the seam the MCP server wraps (docs/MCP-SERVER-SPEC.md §4): the CLI wrappers
|
||||||
|
are tested by the other suites; here we call the cores in-process against the mock and
|
||||||
|
assert (a) the envelope shapes and (b) stdout purity — a stray print() would corrupt
|
||||||
|
an MCP stdio/HTTP response stream.
|
||||||
|
|
||||||
|
Run: python test_ops_api.py [--port 8850]
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import contextlib
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
import urllib.request
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
HERE = Path(__file__).resolve().parent
|
||||||
|
SCRIPTS = HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts"
|
||||||
|
KEY = "test-key-not-a-real-secret"
|
||||||
|
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--port", type=int, default=8850)
|
||||||
|
a = ap.parse_args()
|
||||||
|
BASE = f"http://127.0.0.1:{a.port}"
|
||||||
|
|
||||||
|
# env must be set BEFORE importing echo (it resolves config at import time)
|
||||||
|
os.environ.update(ECHO_BASE=BASE, ECHO_KEY=KEY, ECHO_TODAY="2026-06-21",
|
||||||
|
ECHO_NOW="2300", ECHO_STATE_DIR=tempfile.mkdtemp(), ECHO_VERIFY="0")
|
||||||
|
sys.path.insert(0, str(SCRIPTS))
|
||||||
|
|
||||||
|
failures = []
|
||||||
|
|
||||||
|
|
||||||
|
def check(name, cond, detail=""):
|
||||||
|
print(f"{'ok ' if cond else 'FAIL'} {name}" + (f" -- {detail}" if not cond else ""))
|
||||||
|
if not cond:
|
||||||
|
failures.append(name)
|
||||||
|
|
||||||
|
|
||||||
|
def http(method, url, body=None):
|
||||||
|
data = body.encode() if isinstance(body, str) else body
|
||||||
|
req = urllib.request.Request(url, data=data, method=method,
|
||||||
|
headers={"Authorization": f"Bearer {KEY}"})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=10) as r:
|
||||||
|
return r.status, r.read().decode("utf-8", "replace")
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
return getattr(e, "code", 0), ""
|
||||||
|
|
||||||
|
|
||||||
|
def pure(fn, *args, **kw):
|
||||||
|
"""Call fn capturing stdout; return (result, captured_stdout)."""
|
||||||
|
buf = io.StringIO()
|
||||||
|
with contextlib.redirect_stdout(buf):
|
||||||
|
out = fn(*args, **kw)
|
||||||
|
return out, buf.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
srv = subprocess.Popen([sys.executable, str(HERE / "mock_olrapi.py"), "--port", str(a.port)],
|
||||||
|
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
||||||
|
try:
|
||||||
|
for _ in range(50):
|
||||||
|
try:
|
||||||
|
urllib.request.urlopen(f"{BASE}/__debug__reset", data=b"", timeout=1)
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
time.sleep(0.1)
|
||||||
|
http("PUT", f"{BASE}/vault/_agent/echo-vault.md", "---\nschema_version: 4\n---\n# marker\n")
|
||||||
|
http("PUT", f"{BASE}/vault/_agent/context/current-context.md",
|
||||||
|
"---\ntype: context-bundle\nscope_updated: \"2026-06-20\"\ncreated: 2026-06-01\n---\n"
|
||||||
|
"# Current Context\n\n## Scope\ntesting phase 0\n\n## Scope History\n")
|
||||||
|
|
||||||
|
import echo
|
||||||
|
import echo_doctor
|
||||||
|
import echo_ops
|
||||||
|
import echo_recall
|
||||||
|
import echo_reflect
|
||||||
|
import echo_session
|
||||||
|
import echo_triage
|
||||||
|
|
||||||
|
# capture_op: create / duplicate-gate / dry-run — envelopes, stdout pure
|
||||||
|
env, out = pure(echo_ops.capture_op, "person", "Zara Quix", body_text="Met at the expo.")
|
||||||
|
check("capture_op create envelope", env.get("ok") is True and env.get("action") == "created"
|
||||||
|
and env.get("path") == "resources/people/zara-quix.md", json.dumps(env))
|
||||||
|
check("capture_op prints nothing to stdout", out == "", out[:200])
|
||||||
|
env, out = pure(echo_ops.capture_op, "person", "Zara Quix Junior", body_text="")
|
||||||
|
check("capture_op duplicate-gate is data, not an exit code",
|
||||||
|
env.get("ok") is False and env.get("action") == "duplicate-gate"
|
||||||
|
and env.get("candidates"), json.dumps(env))
|
||||||
|
check("capture_op gate prints nothing", out == "", out[:200])
|
||||||
|
env, out = pure(echo_ops.capture_op, "company", "Plan Co", body_text="", dry_run=True)
|
||||||
|
check("capture_op dry-run envelope", env.get("action") == "dry-run:create"
|
||||||
|
and env.get("path") == "resources/companies/plan-co.md", json.dumps(env))
|
||||||
|
|
||||||
|
# resolve_op / link_op
|
||||||
|
env, out = pure(echo_ops.resolve_op, "zara quix")
|
||||||
|
check("resolve_op returns the match dict", env.get("match") is True
|
||||||
|
and env.get("path") == "resources/people/zara-quix.md", json.dumps(env))
|
||||||
|
pure(echo_ops.capture_op, "concept", "Gizmo", body_text="")
|
||||||
|
env, out = pure(echo_ops.link_op, "resources/people/zara-quix.md", "resources/concepts/gizmo.md")
|
||||||
|
check("link_op envelope", env.get("ok") is True and env.get("a_changed") in (True, False),
|
||||||
|
json.dumps(env))
|
||||||
|
check("link_op prints nothing", out == "", out[:200])
|
||||||
|
|
||||||
|
# recall_op
|
||||||
|
env, out = pure(echo_recall.recall_op, "expo")
|
||||||
|
check("recall_op envelope shape", env.get("action") == "recall"
|
||||||
|
and isinstance(env.get("primary"), list) and isinstance(env.get("linked"), list),
|
||||||
|
json.dumps(env)[:200])
|
||||||
|
check("recall_op prints nothing", out == "", out[:200])
|
||||||
|
|
||||||
|
# triage list_op / route_op (dry-run)
|
||||||
|
http("PUT", f"{BASE}/vault/inbox/captures/inbox.md", "- 2026-06-10: try the quorlab tool\n")
|
||||||
|
env, out = pure(echo_triage.list_op)
|
||||||
|
check("triage list_op envelope", env.get("count") == 1
|
||||||
|
and env["items"][0]["age_days"] == 11, json.dumps(env))
|
||||||
|
props = [{"title": "Quorlab Tool", "kind": "reference", "confidence": 0.9,
|
||||||
|
"line": "- 2026-06-10: try the quorlab tool"}]
|
||||||
|
env, out = pure(echo_triage.route_op, props, False)
|
||||||
|
check("triage route_op dry-run rows", env.get("dry_run") is True
|
||||||
|
and env["rows"][0]["action"] == "create", json.dumps(env))
|
||||||
|
check("triage route_op prints nothing", out == "", out[:200])
|
||||||
|
|
||||||
|
# reflect apply_op (applied path, using capture_op internally)
|
||||||
|
env, out = pure(echo_reflect.apply_op,
|
||||||
|
[{"title": "Blorp Pattern", "kind": "semantic",
|
||||||
|
"body": "The operator prefers blorp.", "confidence": 0.9}], True)
|
||||||
|
check("reflect apply_op applies via capture_op", env.get("applied") == 1
|
||||||
|
and env["results"][0]["action"] == "created", json.dumps(env))
|
||||||
|
check("reflect apply_op prints nothing", out == "", out[:200])
|
||||||
|
|
||||||
|
# scope ops
|
||||||
|
env, out = pure(echo.scope_show_op)
|
||||||
|
check("scope_show_op returns scope + freshness", env.get("scope") == "testing phase 0"
|
||||||
|
and env.get("scope_updated") == "2026-06-20", json.dumps(env))
|
||||||
|
env, out = pure(echo.scope_set_op, "phase zero refactor")
|
||||||
|
check("scope_set_op envelope", env.get("action") == "scope-set"
|
||||||
|
and env.get("scope_updated") == "2026-06-21", json.dumps(env))
|
||||||
|
check("scope_set_op prints nothing", out == "", out[:200])
|
||||||
|
|
||||||
|
# load_op (brief)
|
||||||
|
env, out = pure(echo.load_op, True)
|
||||||
|
check("load_op returns sections + brief", "marker" in env.get("sections", {})
|
||||||
|
and "ECHO load (brief)" in env.get("brief", ""), json.dumps(env)[:200])
|
||||||
|
check("load_op prints nothing", out == "", out[:200])
|
||||||
|
|
||||||
|
# doctor run_op
|
||||||
|
env, out = pure(echo_doctor.run_op)
|
||||||
|
check("doctor run_op checks list", isinstance(env.get("checks"), list)
|
||||||
|
and env.get("fatal") is None and env.get("ok") is True, json.dumps(env))
|
||||||
|
check("doctor run_op prints nothing", out == "", out[:200])
|
||||||
|
|
||||||
|
# session_end_op: dry-run envelope; bad bundle raises EchoError(2) pre-write
|
||||||
|
bundle = {"slug": "phase-zero", "log_body": "---\ntype: session-log\n---\n# S\n\n## Goal\nx\n"}
|
||||||
|
env, out = pure(echo_session.session_end_op, bundle, False)
|
||||||
|
check("session_end_op dry-run envelope", env.get("dry_run") is True
|
||||||
|
and env.get("path") == "_agent/sessions/2026-06-21-2300-phase-zero.md", json.dumps(env))
|
||||||
|
check("session_end_op prints nothing", out == "", out[:200])
|
||||||
|
try:
|
||||||
|
echo_session.session_end_op({"slug": "Bad Slug!", "log_body": "x"}, True)
|
||||||
|
check("session_end_op rejects a bad slug", False, "no exception raised")
|
||||||
|
except RuntimeError as exc:
|
||||||
|
check("session_end_op rejects a bad slug", getattr(exc, "code", 0) == 2, str(exc))
|
||||||
|
|
||||||
|
print(f"\n{len(failures)} failure(s)" if failures else "\nall ops-api tests passed")
|
||||||
|
return 1 if failures else 0
|
||||||
|
finally:
|
||||||
|
srv.terminate()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -59,7 +59,9 @@ def main():
|
|||||||
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
|
||||||
|
|
||||||
def echo(*args):
|
def echo(*args):
|
||||||
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="0")
|
import tempfile
|
||||||
|
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="0",
|
||||||
|
ECHO_STATE_DIR=globals().setdefault("_STATE_DIR", tempfile.mkdtemp()))
|
||||||
return subprocess.run([sys.executable, str(ECHO), *args], capture_output=True, text=True, env=env)
|
return subprocess.run([sys.executable, str(ECHO), *args], capture_output=True, text=True, env=env)
|
||||||
|
|
||||||
def ground(path):
|
def ground(path):
|
||||||
|
|||||||
@@ -51,7 +51,9 @@ def main():
|
|||||||
return getattr(e, "code", 0), ""
|
return getattr(e, "code", 0), ""
|
||||||
|
|
||||||
def echo(*args, stdin=None):
|
def echo(*args, stdin=None):
|
||||||
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1", ECHO_TODAY="2026-06-22")
|
import tempfile
|
||||||
|
env = dict(os.environ, ECHO_BASE=base, ECHO_KEY=KEY, ECHO_VERIFY="1",
|
||||||
|
ECHO_TODAY="2026-06-22", ECHO_STATE_DIR=tempfile.mkdtemp())
|
||||||
return subprocess.run([sys.executable, str(ECHO), *args], input=stdin,
|
return subprocess.run([sys.executable, str(ECHO), *args], input=stdin,
|
||||||
capture_output=True, text=True, env=env)
|
capture_output=True, text=True, env=env)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,417 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""echo-mcp — the containerized MCP server over the ECHO ops layer. [2.2]
|
||||||
|
|
||||||
|
Streamable-HTTP (stateless JSON) MCP server exposing the plugin's `*_op` cores
|
||||||
|
(Phase 0, shipped 2.1.1) as typed tools. Runs next to the Obsidian REST API on
|
||||||
|
the same box, so every vault round-trip behind a tool call is LAN-local.
|
||||||
|
|
||||||
|
Spec: docs/MCP-SERVER-SPEC.md. Highlights implemented here:
|
||||||
|
* bearer-token auth on everything except /health (constant-time compare);
|
||||||
|
* tool results = the op envelopes, returned as structured JSON;
|
||||||
|
* duplicate gate / offline queueing are DATA (never protocol errors), so the
|
||||||
|
model deliberately chooses merge_into/force instead of blind-retrying;
|
||||||
|
* tool profiles: ECHO_MCP_TOOLS=core exposes only the six daily drivers
|
||||||
|
(schema tokens cost context on every surface that lists them);
|
||||||
|
* result budgets: echo_recall(budget_chars), echo_get_note(section/max_chars);
|
||||||
|
* per-write serialization (one process mediates all MCP writes, so a plain
|
||||||
|
lock removes the advisory-lock race window for server-mediated writes);
|
||||||
|
* startup validation: fail fast (crash-loop visibly) on missing env.
|
||||||
|
|
||||||
|
Deliberately NOT here in v1 (spec §7.2 backlog): entity-index TTL cache — the
|
||||||
|
atomic_index_update correctness fix depends on a FRESH re-read under the lock,
|
||||||
|
so caching idx_mod.load would reintroduce the clobber race it closed. LAN
|
||||||
|
adjacency + the warm keep-alive pool make the read ~ms anyway.
|
||||||
|
|
||||||
|
Env: ECHO_BASE, ECHO_KEY, ECHO_MCP_TOKEN (required) · ECHO_OWNER ·
|
||||||
|
ECHO_MCP_PORT=8765 · ECHO_MCP_TOOLS=full|core · ECHO_STATE_DIR=/data
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import contextlib
|
||||||
|
import hmac
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# --- locate the plugin scripts (image layout first, repo layout for local dev) ---
|
||||||
|
_HERE = Path(__file__).resolve().parent
|
||||||
|
for _cand in (_HERE / "scripts",
|
||||||
|
_HERE.parent / "echo-memory.plugin.src" / "skills" / "echo-memory" / "scripts"):
|
||||||
|
if _cand.is_dir():
|
||||||
|
SCRIPTS = _cand
|
||||||
|
break
|
||||||
|
else: # pragma: no cover
|
||||||
|
sys.exit("echo-mcp: cannot locate the ECHO scripts directory")
|
||||||
|
sys.path.insert(0, str(SCRIPTS))
|
||||||
|
|
||||||
|
# --- startup validation: a misdeployed container must crash-loop visibly ---------
|
||||||
|
_missing = [k for k in ("ECHO_BASE", "ECHO_KEY", "ECHO_MCP_TOKEN") if not os.environ.get(k)]
|
||||||
|
if _missing:
|
||||||
|
sys.exit(f"echo-mcp: missing required env: {', '.join(_missing)} "
|
||||||
|
"— check the PORT template / secret refs")
|
||||||
|
|
||||||
|
# Modules aliased *_mod: several tool functions below share a module's name
|
||||||
|
# (echo_recall, echo_reflect, ...) and would shadow it at module scope otherwise.
|
||||||
|
import echo # noqa: E402
|
||||||
|
import echo_doctor as doctor_mod # noqa: E402
|
||||||
|
import echo_index as idx_mod # noqa: E402
|
||||||
|
import echo_ops as ops_mod # noqa: E402
|
||||||
|
import echo_queue as queue_mod # noqa: E402
|
||||||
|
import echo_recall as recall_mod # noqa: E402
|
||||||
|
import echo_reflect as reflect_mod # noqa: E402
|
||||||
|
import echo_session as session_mod # noqa: E402
|
||||||
|
import echo_triage as triage_mod # noqa: E402
|
||||||
|
|
||||||
|
from mcp.server.fastmcp import FastMCP # noqa: E402
|
||||||
|
from starlette.requests import Request # noqa: E402
|
||||||
|
from starlette.responses import JSONResponse, Response # noqa: E402
|
||||||
|
from starlette.middleware.base import BaseHTTPMiddleware # noqa: E402
|
||||||
|
|
||||||
|
PORT = int(os.environ.get("ECHO_MCP_PORT", "8765"))
|
||||||
|
TOKEN = os.environ["ECHO_MCP_TOKEN"]
|
||||||
|
PROFILE = os.environ.get("ECHO_MCP_TOOLS", "full").strip().lower()
|
||||||
|
KINDS = sorted(idx_mod.KIND_FOLDER)
|
||||||
|
|
||||||
|
mcp = FastMCP(
|
||||||
|
"echo-mcp",
|
||||||
|
instructions=(
|
||||||
|
"Persistent memory over the operator's ECHO Obsidian vault. Use echo_load at "
|
||||||
|
"the start of substantive sessions, echo_recall/echo_resolve to read, "
|
||||||
|
"echo_capture as the default write, and echo_log_session to end a session. "
|
||||||
|
"A duplicate-gate result is a decision point, not an error — merge_into or "
|
||||||
|
"(confirmed-distinct) force. Write about the operator in third person."),
|
||||||
|
host="0.0.0.0",
|
||||||
|
port=PORT,
|
||||||
|
stateless_http=True,
|
||||||
|
json_response=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
WRITE_LOCK = threading.Lock() # all MCP-path writes serialize through this process
|
||||||
|
|
||||||
|
|
||||||
|
def _guard(write: bool, fn, *args, **kw) -> dict:
|
||||||
|
"""Run an op core with stdout redirected to stderr (belt-and-braces stream
|
||||||
|
purity) and EchoError mapped to an actionable error envelope, not a protocol
|
||||||
|
error. Write ops serialize on WRITE_LOCK."""
|
||||||
|
lock = WRITE_LOCK if write else contextlib.nullcontext()
|
||||||
|
try:
|
||||||
|
with lock, contextlib.redirect_stdout(sys.stderr):
|
||||||
|
return fn(*args, **kw)
|
||||||
|
except RuntimeError as exc: # EchoError (either module instance) subclasses it
|
||||||
|
code = getattr(exc, "code", 1)
|
||||||
|
err = {"ok": False, "error": str(exc), "code": code}
|
||||||
|
if code == 78:
|
||||||
|
err["hint"] = ("server deployment is missing vault credentials — operator: "
|
||||||
|
"check the PORT template env (SECRET: refs)")
|
||||||
|
elif code == 44:
|
||||||
|
err["code_name"] = "not-found"
|
||||||
|
elif "unreachable" in str(exc):
|
||||||
|
err["hint"] = ("vault unreachable (Obsidian/REST plugin likely down) — "
|
||||||
|
"proceed without memory; writes queue durably")
|
||||||
|
return err
|
||||||
|
|
||||||
|
|
||||||
|
_SAFE_PATH = re.compile(r"^[^/][^\0]*$")
|
||||||
|
|
||||||
|
|
||||||
|
def _check_path(path: str) -> str | None:
|
||||||
|
if not path or path.startswith("/") or ".." in path.split("/") or not _SAFE_PATH.match(path):
|
||||||
|
return "invalid path: must be vault-relative, no leading '/' and no '..'"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _frontmatter(text: str) -> dict:
|
||||||
|
out: dict = {}
|
||||||
|
if text.startswith("---"):
|
||||||
|
end = text.find("\n---", 3)
|
||||||
|
for ln in text[3:end if end != -1 else len(text)].splitlines():
|
||||||
|
m = re.match(r"^([A-Za-z_][\w-]*):\s*(.*)$", ln)
|
||||||
|
if m:
|
||||||
|
out[m.group(1)] = m.group(2).strip().strip('"').strip("'")
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ tools -------
|
||||||
|
CORE_TOOLS = {"echo_load", "echo_recall", "echo_capture", "echo_triage_inbox",
|
||||||
|
"echo_log_session", "echo_health"}
|
||||||
|
|
||||||
|
|
||||||
|
def tool(fn):
|
||||||
|
"""Register `fn` as an MCP tool unless the core profile excludes it."""
|
||||||
|
if PROFILE == "core" and fn.__name__ not in CORE_TOOLS:
|
||||||
|
return fn
|
||||||
|
return mcp.tool()(fn)
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_load(brief: bool = True) -> dict:
|
||||||
|
"""Cold-start memory orientation — call at the start of a substantive session.
|
||||||
|
brief=true (default) returns a token-budgeted digest plus structured sections;
|
||||||
|
brief=false returns the six raw sections. Also flushes writes queued offline."""
|
||||||
|
return _guard(True, echo.load_op, brief)
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_recall(query: str, limit: int = 6, budget_chars: int = 4000,
|
||||||
|
include_linked: bool = True) -> dict:
|
||||||
|
"""Search memory: ranked matches for a topic/person/project PLUS their linked
|
||||||
|
neighbourhood. Excerpts are packed into budget_chars by score — call
|
||||||
|
echo_get_note for any hit's full content. Freshness/status-aware ranking."""
|
||||||
|
limit = max(1, min(int(limit), 20))
|
||||||
|
budget = max(500, min(int(budget_chars), 20000))
|
||||||
|
env = _guard(False, recall_mod.recall_op, query, limit)
|
||||||
|
if not env.get("ok"):
|
||||||
|
return env
|
||||||
|
if not include_linked:
|
||||||
|
env["linked"] = []
|
||||||
|
used = 0
|
||||||
|
for hit in (env.get("primary") or []) + (env.get("linked") or []):
|
||||||
|
ex = hit.get("excerpt") or ""
|
||||||
|
room = max(0, budget - used)
|
||||||
|
if len(ex) > room:
|
||||||
|
hit["excerpt"] = ex[:room] + ("… (truncated — echo_get_note for full)" if room else "")
|
||||||
|
hit["truncated"] = True
|
||||||
|
used += len(hit.get("excerpt") or "")
|
||||||
|
return env
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_resolve(mention: str) -> dict:
|
||||||
|
"""Resolve a name/mention to its canonical vault note (alias-aware), or get
|
||||||
|
did-you-mean candidates. Call before creating any note by hand; echo_capture
|
||||||
|
does this automatically."""
|
||||||
|
return _guard(False, ops_mod.resolve_op, mention)
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_get_note(path: str, section: str = "", max_chars: int = 8000) -> dict:
|
||||||
|
"""Read one vault note (vault-relative path). section='Status' returns only that
|
||||||
|
## section. Content over max_chars is truncated with a marker."""
|
||||||
|
bad = _check_path(path)
|
||||||
|
if bad:
|
||||||
|
return {"ok": False, "error": bad}
|
||||||
|
|
||||||
|
def _read() -> dict:
|
||||||
|
status, body = echo.request("GET", echo.vault_url(path))
|
||||||
|
if status == 404:
|
||||||
|
return {"ok": False, "code_name": "not-found", "path": path,
|
||||||
|
"error": f"{path}: not found"}
|
||||||
|
echo.check(status, body, f"get {path}")
|
||||||
|
text = body.decode(errors="replace")
|
||||||
|
fm = _frontmatter(text)
|
||||||
|
if section:
|
||||||
|
text = echo.extract_heading(text, section)
|
||||||
|
if not text:
|
||||||
|
return {"ok": False, "path": path,
|
||||||
|
"error": f"no '## {section}' section in {path}"}
|
||||||
|
truncated = len(text) > max_chars
|
||||||
|
return {"ok": True, "path": path, "frontmatter": fm,
|
||||||
|
"content": text[:max_chars] + ("\n… (truncated)" if truncated else ""),
|
||||||
|
"truncated": truncated}
|
||||||
|
return _guard(False, _read)
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_get_scope() -> dict:
|
||||||
|
"""The operator's active scope + freshness. If sessions_since >= 3, treat the
|
||||||
|
recorded scope as suspect and confirm with the operator before working."""
|
||||||
|
return _guard(False, echo.scope_show_op)
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_set_scope(scope: str) -> dict:
|
||||||
|
"""Switch the active scope atomically (archives the prior scope to Scope History
|
||||||
|
and stamps freshness). Use when the session's work diverges from the recorded
|
||||||
|
scope."""
|
||||||
|
return _guard(True, echo.scope_set_op, scope)
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_health(deep: bool = False) -> dict:
|
||||||
|
"""ECHO readiness: config, vault reachability, auth, bootstrap/schema, and the
|
||||||
|
offline-queue depth. deep=true also runs the full vault-invariant linter."""
|
||||||
|
env = _guard(False, doctor_mod.run_op)
|
||||||
|
try:
|
||||||
|
env["outbox_depth"] = len(queue_mod.pending())
|
||||||
|
env["needs_attention"] = len(queue_mod.needs_attention())
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
pass
|
||||||
|
if deep and env.get("ok"):
|
||||||
|
r = subprocess.run([sys.executable, str(SCRIPTS / "vault_lint.py")],
|
||||||
|
capture_output=True, text=True,
|
||||||
|
env=dict(os.environ), timeout=120)
|
||||||
|
env["lint_exit"] = r.returncode
|
||||||
|
env["lint_report"] = (r.stdout or r.stderr)[-6000:]
|
||||||
|
return env
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_capture(title: str, kind: str = "", body: str = "", tags: list[str] = [],
|
||||||
|
aliases: list[str] = [], sources: list[str] = [], status: str = "",
|
||||||
|
date: str = "", domain: str = "business", merge_into: str = "",
|
||||||
|
force: bool = False, dry_run: bool = False) -> dict:
|
||||||
|
"""The default memory write: routes by kind (person, company, concept, reference,
|
||||||
|
meeting, project, area, semantic, episodic, working, skill, decision), stamps
|
||||||
|
complete frontmatter, indexes, auto-links, and writes the Agent-Log line — one
|
||||||
|
call. Omit kind for an inbox capture. If the result is action='duplicate-gate',
|
||||||
|
do NOT retry blindly: call again with merge_into=<candidate slug> to update the
|
||||||
|
existing entity, or force=true only after confirming they are genuinely distinct.
|
||||||
|
Offline, the whole capture queues durably (queued=true)."""
|
||||||
|
if kind and kind not in KINDS:
|
||||||
|
return {"ok": False, "error": f"unknown kind '{kind}' — one of: {', '.join(KINDS)}"}
|
||||||
|
return _guard(True, ops_mod.capture_op, kind or None, title,
|
||||||
|
body_text=body or "", status_v=status, aliases=aliases,
|
||||||
|
sources=sources, tags=tags, date=date or None, domain=domain,
|
||||||
|
inbox=not kind, dry_run=dry_run, force=force,
|
||||||
|
merge_into=merge_into or None)
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_link(a: str, b: str) -> dict:
|
||||||
|
"""Add reciprocal '## Related' links between two notes. Accepts vault paths or
|
||||||
|
resolvable entity names (aliases work)."""
|
||||||
|
def _to_path(x: str) -> str | None:
|
||||||
|
if "/" in x:
|
||||||
|
return x if x.endswith(".md") else x + ".md"
|
||||||
|
nmap = idx_mod.name_map(idx_mod.load())
|
||||||
|
return nmap.get(idx_mod.slugify(x))
|
||||||
|
def _do() -> dict:
|
||||||
|
pa, pb = _to_path(a), _to_path(b)
|
||||||
|
if not pa or not pb:
|
||||||
|
missing = a if not pa else b
|
||||||
|
return {"ok": False, "error": f"'{missing}' resolves to no note — "
|
||||||
|
"pass a vault path or a known entity name"}
|
||||||
|
return ops_mod.link_op(pa, pb)
|
||||||
|
return _guard(True, _do)
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_append_note(path: str, line: str) -> dict:
|
||||||
|
"""Append one line to a note, idempotently (skipped if the exact line already
|
||||||
|
exists). For inbox lines, Agent-Log entries, Observations bullets."""
|
||||||
|
bad = _check_path(path)
|
||||||
|
if bad:
|
||||||
|
return {"ok": False, "error": bad}
|
||||||
|
def _do() -> dict:
|
||||||
|
rc = echo.cmd_append(path, line)
|
||||||
|
return {"ok": rc == 0, "action": "append", "path": path}
|
||||||
|
return _guard(True, _do)
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_patch_note(path: str, operation: str, target_type: str, target: str,
|
||||||
|
content: str) -> dict:
|
||||||
|
"""Targeted edit: append/prepend/replace under a heading, frontmatter field, or
|
||||||
|
block. HEADING TARGETS must be the full '::'-delimited path from the top-level
|
||||||
|
heading (e.g. 'Operator Preferences::Fact / Pattern') — on an invalid target the
|
||||||
|
error includes the note's actual headings. replace overwrites the section."""
|
||||||
|
bad = _check_path(path)
|
||||||
|
if bad:
|
||||||
|
return {"ok": False, "error": bad}
|
||||||
|
def _do() -> dict:
|
||||||
|
rc = echo.cmd_patch(path, operation, target_type, target,
|
||||||
|
echo.temp_file(content.encode("utf-8")))
|
||||||
|
return {"ok": rc == 0, "action": f"patch:{operation}", "path": path,
|
||||||
|
"target": target}
|
||||||
|
env = _guard(True, _do)
|
||||||
|
if not env.get("ok") and "HTTP 400" in str(env.get("error", "")) and target_type == "heading":
|
||||||
|
st, body = echo.request("GET", echo.vault_url(path),
|
||||||
|
headers={"Accept": "application/vnd.olrapi.document-map+json"})
|
||||||
|
if st == 200:
|
||||||
|
try:
|
||||||
|
headings = [h.get("heading") or h for h in
|
||||||
|
json.loads(body).get("headings", [])][:40]
|
||||||
|
env["available_headings"] = headings
|
||||||
|
env["hint"] = "use one of available_headings verbatim as the Target"
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
pass
|
||||||
|
return env
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_triage_inbox(proposals: list[dict] = [], apply: bool = False) -> dict:
|
||||||
|
"""Inbox triage. No proposals => structured listing of captures (line, date, text,
|
||||||
|
age_days). With proposals (reflect schema + optional 'line'): apply=false previews
|
||||||
|
the routing; apply=true routes via capture and writes the processing-log audit.
|
||||||
|
List first, propose, preview, then apply only after the operator confirms."""
|
||||||
|
if not proposals:
|
||||||
|
return _guard(False, triage_mod.list_op)
|
||||||
|
return _guard(True, triage_mod.route_op, proposals, apply)
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_reflect(proposals: list[dict], apply: bool = False) -> dict:
|
||||||
|
"""Session-reflection proposals: validate -> classify against the entity index ->
|
||||||
|
preview (apply=false) -> apply (routes each via capture). Never apply without the
|
||||||
|
operator's go-ahead; never invent memories to have something to save."""
|
||||||
|
return _guard(True, reflect_mod.apply_op, proposals, apply)
|
||||||
|
|
||||||
|
|
||||||
|
@tool
|
||||||
|
def echo_log_session(slug: str, log_body: str, agent_log_line: str = "",
|
||||||
|
scope: str = "", reflect: list[dict] = [],
|
||||||
|
apply: bool = False, hhmm: str = "") -> dict:
|
||||||
|
"""End a substantive session in ONE call: session log -> Agent-Log line -> reflect
|
||||||
|
proposals -> optional scope switch -> heartbeat LAST (the commit marker).
|
||||||
|
apply=false previews the plan. Pass hhmm (local time, e.g. '1430') so the log
|
||||||
|
filename sorts truthfully."""
|
||||||
|
bundle = {"slug": slug, "log_body": log_body}
|
||||||
|
if agent_log_line:
|
||||||
|
bundle["agent_log_line"] = agent_log_line
|
||||||
|
if scope:
|
||||||
|
bundle["scope"] = scope
|
||||||
|
if reflect:
|
||||||
|
bundle["reflect"] = reflect
|
||||||
|
def _do() -> dict:
|
||||||
|
prev = os.environ.get("ECHO_NOW")
|
||||||
|
try:
|
||||||
|
if hhmm:
|
||||||
|
os.environ["ECHO_NOW"] = hhmm
|
||||||
|
return session_mod.session_end_op(bundle, apply=apply)
|
||||||
|
finally:
|
||||||
|
if hhmm:
|
||||||
|
if prev is None:
|
||||||
|
os.environ.pop("ECHO_NOW", None)
|
||||||
|
else:
|
||||||
|
os.environ["ECHO_NOW"] = prev
|
||||||
|
return _guard(True, _do)
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------- health + auth ----
|
||||||
|
@mcp.custom_route("/health", methods=["GET"])
|
||||||
|
async def health(_request: Request) -> JSONResponse:
|
||||||
|
"""Unauthenticated liveness for Docker HEALTHCHECK + Kuma. No vault data."""
|
||||||
|
st, _ = echo.request("GET", echo.vault_url("_agent/echo-vault.md"))
|
||||||
|
try:
|
||||||
|
outbox = len(queue_mod.pending())
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
outbox = -1
|
||||||
|
return JSONResponse({"ok": True, "vault_reachable": st != 0,
|
||||||
|
"outbox_depth": outbox, "profile": PROFILE})
|
||||||
|
|
||||||
|
|
||||||
|
class BearerAuth(BaseHTTPMiddleware):
|
||||||
|
async def dispatch(self, request, call_next):
|
||||||
|
if request.url.path == "/health":
|
||||||
|
return await call_next(request)
|
||||||
|
auth = request.headers.get("authorization", "")
|
||||||
|
if not hmac.compare_digest(auth, f"Bearer {TOKEN}"):
|
||||||
|
return Response(status_code=401)
|
||||||
|
return await call_next(request)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
import uvicorn
|
||||||
|
app = mcp.streamable_http_app()
|
||||||
|
app.add_middleware(BearerAuth)
|
||||||
|
print(f"echo-mcp: serving on :{PORT} (profile={PROFILE}, vault={echo.BASE})",
|
||||||
|
file=sys.stderr)
|
||||||
|
uvicorn.run(app, host="0.0.0.0", port=PORT, log_level="info")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
# echo-mcp server deps. The PLUGIN stays pure-stdlib; the dependency budget for the
|
||||||
|
# container is deliberately tiny: the official MCP SDK (brings starlette/uvicorn/
|
||||||
|
# pydantic) and nothing else.
|
||||||
|
mcp>=1.9,<2
|
||||||
Reference in New Issue
Block a user