The MCP `mempalace_status` tool was returning the server's absolute
`_config.palace_path` to any connected client on both the main
(ChromaDB-backed) path and the sqlite fallback path that runs when
HNSW divergence is detected (#1222). On a single-user local deployment
this is self-disclosure, but in nested-agent or multi-server MCP
topologies the client is a separate trust domain and the absolute
path has no documented client-side use.
Clients that legitimately need the palace path continue to have three
documented channels: the `MEMPALACE_PALACE_PATH` env var (primary) or
its legacy `MEMPAL_PALACE_PATH` alias, the `~/.mempalace/config.json`
file, and the `--palace` CLI flag on most subcommands.
Also corrects stale docs that claimed `mempalace_reconnect` returned a
`palace_path` field; the code returns `{success, message, drawers,
vector_disabled[, vector_disabled_reason]}` on success, plus a no-palace
shape and an exception shape.
- mempalace/mcp_server.py: drop palace_path from tool_status() and
_tool_status_via_sqlite() result dicts
- website/reference/mcp-tools.md: update documented return shapes for
mempalace_status (fix) and mempalace_reconnect (stale-docs correction)
Authored-by: Aaron Salsitz (ICCI LLC, @icciaaron). Claude Code was used
as an authoring and review-orchestration tool, with human-in-the-loop
oversight at every step: Aaron wrote the prompts, reviewed each draft,
called for three independent review passes (drafting / post-rebase
technical / CISA-aligned disclosure-leak), and verified the final patch
behavior before commit.
- Resolve the EF inside the two reopen branches that actually call
`client.get_collection` / `client.create_collection`, so warm-cache
reads stay zero-cost (no `MempalaceConfig()` / `_resolve_providers`
on every tool call).
- Reuse `ChromaBackend._resolve_embedding_function()` instead of
duplicating its try/except + log message + None-fallback.
- Reword the inline + CHANGELOG explanation to clarify that ChromaDB 1.x
persists the EF *identity* (its `name()`) but not the *instance/
configuration* — `mempalace.embedding` documents this and spoofs
`name()` to `"default"` precisely so the identity check passes; the
bug was the *provider list* (lazy ONNX selection) silently differing.
`mcp_server._get_collection` bypassed `ChromaBackend.get_collection`
and called `client.get_collection` / `client.create_collection` without
`embedding_function=`. ChromaDB 1.x does not persist the EF identity
with the collection, so the MCP server's reopen silently bound
chromadb's built-in `DefaultEmbeddingFunction` while the miner / Stop
hook ingest path bound `mempalace.embedding.get_embedding_function()`.
On bleeding-edge interpreters (python 3.14 + chromadb 1.5.x on Apple
Silicon, per #1299) the default EF's lazy ONNX provider selection could
SIGSEGV the host process on first `col.add()`, killing the MCP stdio
server and leaving every subsequent tool call returning
`Connection closed` until Claude Code was relaunched. Reads worked
because `col.get(ids=...)` and metadata fetches don't invoke the EF;
the auto-ingest path worked because mining routes through the backend
abstraction. Diary writes were the consistent failure surface.
Resolve the EF up front (matching `ChromaBackend._resolve_embedding_function`)
and pass it into both reopen branches. Falls back to the chromadb default
only if `mempalace.embedding.get_embedding_function` itself raises.
Regression test patches the chromadb client class to capture
`embedding_function=` on every `get_collection` / `create_collection`
call from `_get_collection(create=True)` and `_get_collection()`, and
fails if any call omits it.
Follow-up to #1262 / #1289 (which fixed the metadata-mismatch SIGSEGV
path); this addresses the EF-mismatch SIGSEGV path on the same surface.
#1262 split `get_or_create_collection` into `get_collection` + fallback
`create_collection` inside `ChromaBackend.get_collection`, fixing the
chromadb 1.5.x Rust-binding SIGSEGV that fires when stored collection
metadata differs from the call-site's `_HNSW_BLOAT_GUARD` payload.
The MCP server's `_get_collection(create=True)` carries the same metadata
payload at `mcp_server.py:287` and routes through chromadb's Python
client directly, bypassing the backend layer. Both `tool_add_drawer`
and `tool_diary_write` reach this site on every invocation, and the
Stop hook fires `mempalace_diary_write` at session end — which was
exactly the crash path #1089 named.
Apply the same try/except split here so legacy palaces whose stored
metadata predates the bloat-guard expansion no longer crash on the
MCP-server reopen path. Regression test patches
`get_or_create_collection` at the chromadb client class level (not the
instance — chromadb's mtime-change detection rebuilds the client between
calls, so an instance-level spy doesn't survive) and asserts the second
`_get_collection(create=True)` call never reaches it.
Sets `hnsw:batch_size` and `hnsw:sync_threshold` to 50_000 at every
collection-creation call site:
* `mempalace/backends/chroma.py` — `get_collection(create=True)` and the
legacy `create_collection()` path. Preserves existing `hnsw:space`,
`hnsw:num_threads=1` (race fix from #976), and `**ef_kwargs`
(embedding-function plumbing from a4868a3).
* `mempalace/mcp_server.py` — the direct `client.get_or_create_collection`
path used when a palace is first opened by the MCP server. Without this
third site, MCP-bootstrapped palaces would skip the guard and could
still trigger the original bloat.
Without these defaults, mining ~10K+ drawers triggers ~30 HNSW index
resizes and hundreds of persistDirty() calls. persistDirty uses relative
seek positioning in link_lists.bin; accumulated seek drift across resize
cycles causes the OS to extend the sparse file with zero-filled regions,
each cycle compounding the next. Result: link_lists.bin grows into
hundreds of GB sparse, after which `status`, `search`, and `repair` all
segfault and the palace is unrecoverable.
Empirical: rebuilt a palace from scratch on 39,792 drawers across 5
wings with this fix applied. Final palace 376 MB, link_lists.bin stays
at 0 bytes across both Chroma collection dirs, status and search both
return cleanly. Same workload without the fix bloated the palace to
565 GB sparse (30 GB on disk) and segfaulted at ~15K drawers.
Migration note: chromadb 1.5.x exposes a
`collection.modify(configuration={"hnsw": {...}})` retrofit path for
already-created collections (`UpdateHNSWConfiguration`), but this PR
doesn't pursue it — by the time link_lists.bin has bloated the index
is already corrupt and the only known recovery is a fresh mine.
Tests assert both keys land on the persisted collection metadata in
both `ChromaBackend` code paths, which also covers the #1161 "config
silently dropped" concern at CI time. A separate smoke test was used
to verify the metadata round-trips through `chromadb.PersistentClient`
reopen on chromadb 1.5.8.
Closes#344
Supersedes #346
Co-authored-by: robot-rocket-science <robot-rocket-science@users.noreply.github.com>
Five Copilot review issues + the Python 3.9 CI failure rolled into one
follow-up:
* Replace ``dict | None`` annotated assignment with a type-comment so
module load doesn't evaluate PEP 604 syntax on Python 3.9 (CI red).
* Drop ``mempalace repair rebuild`` — the CLI only ships ``mempalace
repair`` (rebuild) and ``mempalace repair-status``. Updated all
user-facing messages, docstrings, and test assertions.
* Replace ``_get_client()`` in ``tool_search`` with the safe
``_refresh_vector_disabled_flag`` probe so the fallback isn't
defeated by the very chromadb client load it's trying to avoid.
* Short-circuit ``tool_status`` to a pure-sqlite reader
(``_tool_status_via_sqlite``) when divergence is detected so wing /
room counts come back without ever opening the persistent client.
* Wrap the recency-window query in ``_bm25_only_via_sqlite`` with an
``id``-ordered fallback so legacy schemas missing ``created_at``
don't break BM25 search.
New test covers the sqlite-status short-circuit. 1409 tests pass.
When chromadb's HNSW segment freezes at a stale max_elements while
sqlite keeps accumulating embeddings, the next chromadb open segfaults
the MCP server on every tool call. Adds a pure-filesystem capacity probe
(zero chromadb interaction), a `mempalace repair-status` read-only
health check, and a BM25-only sqlite fallback so the palace stays
reachable even when vector search is unavailable.
* `hnsw_capacity_status` reads sqlite + index_metadata.pickle directly
via a tight-allowlist unpickler — no hnswlib import, no segment load.
* MCP server runs the probe at startup and after every reconnect; sets
`_vector_disabled` and routes search to the sqlite FTS5 + BM25 path.
* `tool_status` and `tool_reconnect` surface the fallback state.
* Threshold tuned for chromadb 1.5.x async-flush lag (2× sync_threshold).
Addresses remaining PR #976 review items after rebase on develop.
`get_collection(create=False)` previously returned existing collections without
re-applying `hnsw:num_threads=1`, so palaces created before the fix kept the
unsafe parallel-insert path. Add `_pin_hnsw_threads()` helper that calls
`collection.modify(configuration=UpdateCollectionConfiguration(
hnsw=UpdateHNSWConfiguration(num_threads=1)))` best-effort on every
`get_collection` call (including the MCP server's `_get_collection`).
In chromadb 1.5.x the runtime config does not persist to disk across
`PersistentClient` reopens, so the retrofit is re-applied each process start
rather than being a one-shot migration. Fresh palaces keep the metadata-based
pin as primary defense; legacy palaces now also get per-session protection
without requiring `mempalace nuke` + re-mine.
After the rebase on develop, `hook_precompact` delegates to `_mine_sync` and
no longer emits `decision: block`, so the attempt-cap constant was orphaned.
Grep confirms 0 usages in the repo — remove it.
- `_pin_hnsw_threads` retrofits legacy collection (num_threads None -> 1)
- `_pin_hnsw_threads` swallows all errors (never raises)
- `ChromaBackend.get_collection(create=False)` applies retrofit on legacy palace
- 62 tests pass (10 backends + 6 palace locks + 46 hooks_cli)
Addresses the six Copilot review comments on the initial commit.
1) #6 (critical) — mcp_server.py `_get_collection` bypassed ChromaBackend
The MCP server creates its palace collection directly via
`chromadb.PersistentClient.get_or_create_collection` in `_get_collection`,
not through `ChromaBackend.get_collection`. That path was missing the
`hnsw:num_threads=1` metadata, so the primary crash surface for #974
and #965 was untouched by the original patch. Fixed by passing
`hnsw:num_threads=1` at the mcp_server create site too. Documented
in a code comment that the setting is only honored at creation
time — existing palaces created before this fix still need a
`mempalace nuke` + re-mine to gain the protection.
2) #3 — mine_global_lock over-serialized mines across unrelated palaces
Replaced the single global lock file `mine_global.lock` with a
per-palace lock keyed by `sha256(os.path.abspath(palace_path))`
(`mine_palace_<hash>.lock`). Mines against the same palace still
collapse to a single runner (the correctness boundary), but mines
against *different* palaces are now free to run in parallel.
`mine_global_lock` is kept as a backward-compatible alias for
`mine_palace_lock` so any external callers that imported the
previous name keep working.
3) #1 — hook_precompact swallowed OSError but not subprocess.TimeoutExpired
`subprocess.run(..., timeout=60)` raises `TimeoutExpired` on slow
palaces. The previous `except OSError` clause didn't catch it, so
the hook could raise and fail to emit any JSON decision — leaving
the harness without a block/passthrough signal. Fixed by catching
`(OSError, subprocess.TimeoutExpired)` together and always falling
through to the block decision so the hook reliably emits a response.
4) #2 + #4 — tests
- tests/test_hooks_cli.py: added
`test_precompact_first_two_attempts_block`,
`test_precompact_passes_through_after_cap`, and
`test_precompact_counter_is_per_session` to lock in the #955
deadlock fix.
- tests/test_palace_locks.py (new): covers `mine_palace_lock`
single-acquire, reuse-after-release, cross-process serialization
on the same palace, non-interference across different palaces,
path normalization, and the `mine_global_lock` back-compat alias.
5) #5 — known limitation, documented but not auto-fixed
Copilot suggested detecting collections missing `hnsw:num_threads=1`
and calling `collection.modify(metadata=...)` to retrofit existing
palaces. Verified against chromadb 1.5.7: `modify(metadata=...)`
replaces metadata rather than merging, and re-passing
`hnsw:space="cosine"` then raises `ValueError: Changing the
distance function of a collection once it is created is not
supported currently.` The HNSW runtime configuration
(`configuration_json`) also does not expose `num_threads` in
chromadb 1.5.x, so the flag appears to be read only at creation
time. Rather than paper over the limitation with a best-effort
`modify` that silently drops `hnsw:space`, documented in the
mcp_server comment that pre-existing palaces need a
`mempalace nuke` + re-mine to gain the protection. Fresh palaces
are always protected.
Testing
- pytest tests/test_palace_locks.py tests/test_hooks_cli.py
tests/test_backends.py tests/test_cli.py → **98 passed, 0 failed**.
- Runtime validation with two concurrent `mempalace mine` calls:
- Different palaces → both complete in parallel ✓
- Same palace → one completes, the other exits with
"another `mine` is already running against <palace> — exiting
cleanly." ✓
#1097 fixed mempalace_search to treat empty-string wing/room as
no filter, matching how LLM agents default to filling every optional
parameter with ''. The same pattern wasn't applied to diary_read:
passing wing='' defaulted to wing_<agent_name>, siloing away entries
that hooks had written to project-derived wings per #659.
When wing is empty/omitted, filter only on agent + room=diary so
callers get a unified view of the agent's journal across every wing
it has written to. Explicit wing=<name> continues to scope reads
to that wing only.
Adds test covering empty-wing read after writing to both the default
and a non-default wing.
* fix: add wing param to diary_write/diary_read, derive from transcript path
Without a wing override, all diary entries from the stop hook land in
wing_session-hook regardless of which project the session is in, making
per-project diary search impossible.
- tool_diary_write(): add optional `wing` param; sanitize and use it when
provided, fall back to wing_{agent_name} when omitted
- tool_diary_read(): add optional `wing` param for filtering by target wing
- TOOLS dict: expose `wing` in input_schema for both diary tools
- hooks_cli: add _wing_from_transcript_path() helper that extracts the
project name from Claude Code paths like
~/.claude/projects/-home-jp-Projects-kiyo-xhci-fix/... → kiyo-xhci-fix
- hook_stop: derive project wing and append wing= hint to block reason so
Claude writes diary entries to the correct per-project wing
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: sanitize wing param, cross-platform paths, tighten test assertions
Addresses Copilot review feedback on #659.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: wing_ prefix + agent filter on diary_read
Addresses bensig's 2-issue review on this PR.
1. _wing_from_transcript_path() was returning bare project names
(e.g. "myproject") while all existing wings follow the wing_*
convention from AAAK_SPEC. Entries landed in wing="myproject"
while diary_read defaulted to wing="wing_<agent_name>" —
orphaning every diary entry written by the stop hook. Now
returns "wing_<project>" and falls back to "wing_sessions".
2. tool_diary_read() did not include agent_name in the ChromaDB
where filter when a custom wing was provided — any caller with
a shared wing could read entries written by other agents.
Add {"agent": agent_name} to the $and clause. Also flagged by
Qudo and left unresolved until now.
Tests updated to expect the wing_ prefix (6 tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The MCP server config used `python -m mempalace.mcp_server` which fails
when mempalace is installed via pipx or uv, since the system python
cannot find the module in the isolated venv. Adding a `mempalace-mcp`
console_scripts entry point ensures the MCP server works regardless of
installation method (pip, pipx, uv, conda).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Four more MCP handlers iterate a metadata list and call m.get(...)
unconditionally. When the cache contains a None entry (drawers with no
metadata, common on older mining paths), the try block catches the
AttributeError and marks the response "partial: true" with an
error message — visible as {"error": "'NoneType' object has no
attribute 'get'", "partial": true} returned from mempalace_status even
though the palace data is otherwise fetchable.
Same m = m or {} guard we applied to searcher.py (d3a2d22, a51c3c2)
and miner.status() (66f08a1). None-metadata drawers now roll up under
the existing "unknown" fallback bucket instead of poisoning the
response with a misleading partial flag.
Regression test: mock the metadata cache with a None in the middle,
assert tool_status returns clean counts and no error/partial fields.
Verified the test fails without the guard.
998 tests pass.
agent_name and entry are validated via sanitize_name/sanitize_content,
but topic is stored raw into ChromaDB metadata. Apply the same
sanitize_name guard to reject null bytes, path traversal, and
oversized payloads.
* fix: restrict file permissions on sensitive palace data
On Linux with default umask (022), several files and directories
containing personal data were created world-readable. This patch
applies chmod 0o700 to directories and 0o600 to files immediately
after creation, wrapped in try/except for Windows compatibility.
Files hardened:
- hooks_cli.py: hook_state/ directory and hook.log
- entity_registry.py: entity_registry.json (names, relationships)
- knowledge_graph.py: knowledge_graph.sqlite3 parent directory
- exporter.py: export output directory and wing subdirectories
- config.py: people_map.json (name mappings)
- mcp_server.py: WAL file creation uses atomic os.open (TOCTOU fix)
Refs: MemPalace/mempalace#809
* fix: avoid redundant chmod calls on hot paths
- hooks_cli.py: chmod STATE_DIR and hook.log only on first creation,
not on every _log() call (hooks fire on every Stop event)
- exporter.py: track created wing dirs to skip redundant makedirs +
chmod on the same directory across batches
- mcp_server.py: remove redundant _WAL_FILE.chmod after os.open
already set mode=0o600 atomically
Refs: MemPalace/mempalace#809
* fix(mcp): redirect stdout to stderr during import to protect JSON-RPC channel (#225)
Fixes#225.
Several transitive dependencies (chromadb, onnxruntime, posthog) print
banners and warnings to stdout — sometimes at the C level — during the
mcp_server import chain. Because the MCP protocol multiplexes JSON-RPC
over stdio, any non-JSON output on stdout corrupted the message stream
and broke Claude Desktop's parser with errors like:
MCP mempalace: Unexpected token '*', "**********"... is not valid JSON
MCP mempalace: Unexpected token 'E', "EP Error D"... is not valid JSON
MCP mempalace: Unexpected token 'F', "Falling ba"... is not valid JSON
Reproduced on Windows 11 with mempalace 3.0.0 / Python 3.10 / Claude
Desktop 1.1062.0.
Fix: at module load, redirect stdout to stderr at both the Python level
(sys.stdout = sys.stderr) and the file-descriptor level (os.dup2(2, 1))
to catch C-level prints, while preserving the real stdout for later
restore. main() calls _restore_stdout() right before entering the
protocol loop so JSON-RPC responses still go to the real stdout.
Adds tests/test_mcp_stdio_protection.py with three regression tests:
- module-level redirect is in place after import
- _restore_stdout() restores the original stdout (idempotent)
- 'python -m mempalace.mcp_server' with empty stdin emits no stdout
* style: reformat with ruff 0.4 (CI version) for #225
tool_status() called _get_collection() with the default create=False,
which throws when the ChromaDB collection does not exist yet (valid
palace, zero drawers). The exception was swallowed and status returned
"No palace found" even though init had completed successfully.
Switching to create=True bootstraps an empty collection on first
status call, matching what the write path already does.
Fix suggested by @hkevinchu in the issue.
sanitize_name rejects commas, colons, parentheses, and slashes — characters
that commonly appear in knowledge graph subject/object values. Adds
sanitize_kg_value for KG entity fields (subject, object, entity) while
keeping sanitize_name for predicates and wing/room names.
Prerequisite for RFC 001 (plugin spec, #743). Removes every direct
`import chromadb` outside the ChromaDB backend itself so the core
modules depend only on the backend abstraction layer.
Extends ChromaBackend with make_client, get_or_create_collection,
delete_collection, create_collection, and backend_version. Adds
update() to the BaseCollection contract. Non-backend callers
(mcp_server, dedup, repair, migrate, cli) now go through the
abstraction; tests patch ChromaBackend instead of chromadb.
With this landed, the RFC 001 spec can be enforced and PalaceStore
(#643) can ship as a plugin without touching core modules.
Merges the full hardened stack (up through #791 drawer-grep) and turns
fact_checker from "dead code hidden behind bare except" into an
actually-working offline contradiction detector with tests.
## Dead paths the PR body advertised but the code never executed
Both buried by a single outer ``except Exception: pass``:
* ``kg.query(subject)`` — ``KnowledgeGraph`` has no ``query()`` method;
it has ``query_entity()``. The attribute error was silently swallowed
and the entire KG branch always returned ``[]``. Now using
``kg.query_entity(subject, direction="outgoing")`` with proper
handling of the ``predicate``/``object``/``current``/``valid_to``
fields the real API returns.
* ``KnowledgeGraph(palace_path=palace_path)`` — the constructor's only
kwarg is ``db_path``. Passing ``palace_path`` raised TypeError,
silently swallowed. Now computing the db_path correctly from
``<palace>/knowledge_graph.sqlite3``, matching the convention the
MCP server already uses.
## Contradiction logic rewritten
The previous ``if kg_pred in claim and fact.object not in claim`` only
fired when text used the SAME predicate word as the KG fact — the exact
opposite of the stated use case ("Bob is Alice's brother" when KG says
husband" would NOT have fired). Replaced with a proper parse → lookup
→ compare pipeline:
* ``_extract_claims`` parses two surface forms ("X is Y's Z" and
"X's Z is Y") into ``(subject, predicate, object)`` triples.
* ``_check_kg_contradictions`` pulls the subject's outgoing facts
and flags two classes:
- ``relationship_mismatch`` when a current KG fact matches the
same ``(subject, object)`` pair but with a different predicate.
- ``stale_fact`` when the exact triple exists but is
``valid_to``-closed in the past.
* Stale-fact detection is now implemented (the PR body claimed it;
the old code silently didn't implement it).
## Performance fix — O(n²) → O(mentioned × n)
``_check_entity_confusion`` previously computed Levenshtein for every
pair of registered names on every ``check_text`` call. For 1,000
registered names that's ~500K edit-distance calls per hook invocation.
Now we first identify which registry names actually appear in the text
(single regex scan), then only compute edit distance between mentioned
and unmentioned names. Pinned by a test that asserts <200ms on a 500-
name registry with zero mentions.
Also: when *both* similar names are mentioned in the text, we no
longer flag them — the user clearly knows they're different people.
## Shared entity-registry loader
``mempalace/miner.py`` already had an mtime-cached loader for
``~/.mempalace/known_entities.json``. fact_checker had a duplicate
implementation that leaked file handles and ignored caching. Extended
miner's cache to expose both the flat set (``_load_known_entities``)
and the raw category dict (``_load_known_entities_raw``); fact_checker
now imports the latter. No more double disk reads, no more handle leak.
## Tests — 24 cases in tests/test_fact_checker.py
All three detection paths + both dead-code regressions:
* ``test_kg_init_uses_db_path_not_palace_path_kwarg`` — pins the
correct KG constructor signature so the ``palace_path=`` bug can't
come back.
* ``test_relationship_mismatch_detected`` — the headline example from
the PR body now actually fires.
* ``test_stale_fact_detected`` — valid_to-closed triple is flagged.
* ``test_current_fact_same_triple_is_not_flagged`` — no false positive
on a still-valid match.
* ``test_performance_bounded_by_mentioned_names`` — 500-name registry,
zero mentions, <200ms. Regression for the O(n²) blowup.
* ``test_no_false_positive_when_both_names_mentioned`` — Mila and
Milla in the same text is fine.
* Plus claim extraction, flatten_names shapes, CLI exit code, empty
text handling, missing-palace graceful fallback, registry-dict
shape support.
785/785 suite pass. ruff + format clean on CI-pinned 0.4.x.
Merges the hardened closet/entity/BM25/diary stack from #789 and fixes
five correctness/durability issues in the tunnels module plus the
directional/symmetric design question.
## Design: tunnels are now symmetric
Per review discussion: a tunnel represents "these two things relate",
not "A causes B". The canonical ID now hashes the *sorted* endpoint
pair, so ``create_tunnel(A, B)`` and ``create_tunnel(B, A)`` resolve to
the same record and the second call updates the label rather than
creating a duplicate. ``follow_tunnels`` can be called from either
endpoint and surfaces the other side consistently.
The returned dict still preserves ``source``/``target`` in the order
the caller supplied, so UIs that want to render the connection
directionally can do so.
## Correctness fixes
* **Atomic write** — ``_save_tunnels`` writes to ``tunnels.json.tmp``
and ``os.replace``s it into place. A crash mid-write can no longer
leave a truncated file that silently reads back as ``[]`` and wipes
every tunnel. Includes ``f.flush() + os.fsync`` before replace on
platforms that support it.
* **Concurrent-write lock** — ``create_tunnel`` and ``delete_tunnel``
wrap the load→mutate→save cycle in ``mine_lock(_TUNNEL_FILE)``.
Without this, two agents creating tunnels simultaneously would both
read the same snapshot and the later writer would drop the earlier
writer's tunnel.
* **Corrupt-file tolerance** — ``_load_tunnels`` now uses a context
manager, validates that the loaded JSON is a list, and returns ``[]``
for any read failure. Subsequent ``create_tunnel`` then overwrites
the corrupt file via atomic write — no manual recovery needed.
* **Input validation** — new ``_require_name`` helper rejects empty or
whitespace-only wing/room names with a clear ``ValueError``. Prevents
phantom tunnels with blank endpoints from ever reaching the JSON
store.
* **Timezone-aware timestamps** — ``created_at`` / ``updated_at`` now
use ``datetime.now(timezone.utc).isoformat()``, matching diary ingest
and other recent modules.
## Tests (12 in TestTunnels)
5 original + 7 regression cases:
* ``test_tunnel_is_symmetric`` — A↔B and B↔A dedupe to one record.
* ``test_follow_tunnels_works_from_either_endpoint`` — symmetric surface.
* ``test_empty_endpoint_fields_rejected`` — validation guard.
* ``test_corrupt_tunnel_file_does_not_lose_new_writes`` — truncated
JSON treated as empty; next create persists cleanly.
* ``test_atomic_write_leaves_no_stray_tmp_file`` — no leftover ``.tmp``.
* ``test_concurrent_creates_preserve_all_tunnels`` — 5 threads each
create a distinct tunnel; all 5 persisted (regression for the
read-modify-write race).
* ``test_created_at_is_timezone_aware`` — ISO8601 has tz suffix.
Merge resolutions: tests/test_closets.py combined develop's hardened
closet/entity/BM25/diary tests with this PR's TestTunnels class.
755/755 tests pass. ruff + format clean under CI-pinned 0.4.x.
Adds active tunnel creation alongside passive tunnel discovery.
Passive tunnels (existing): rooms with the same name across wings.
Explicit tunnels (new): agent-created links between specific
locations. "This API design in project_api relates to the database
schema in project_database."
New functions in palace_graph.py:
- create_tunnel() — link two wing/room pairs with a label
- list_tunnels() — list all explicit tunnels, filter by wing
- delete_tunnel() — remove a tunnel by ID
- follow_tunnels() — from a room, find all connected rooms in
other wings with drawer content previews
New MCP tools:
- mempalace_create_tunnel
- mempalace_list_tunnels
- mempalace_delete_tunnel
- mempalace_follow_tunnels
Tunnels stored in ~/.mempalace/tunnels.json (persists across
palace rebuilds). Deduplicated by endpoint pair.
689/689 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When external tools write to the palace database (CLI mining, scripts), the MCP server's cached ChromaDB collection becomes stale — its HNSW index doesn't know about new vectors. Develop already invalidates on inode changes (catches rebuilds) but not on mtime changes (misses in-place writes).
This PR:
- Adds st_mtime tracking alongside st_ino in _get_client; invalidates the cached client on either change.
- Adds the mempalace_reconnect MCP tool for explicit cache flush.
Original author: @jphein (#663). Original approval: @Ari4ka.
Skips test_missing_db_invalidates_cache on Windows (ChromaDB holds chroma.sqlite3 open).
* fix: skip arg whitelist for handlers accepting **kwargs (#572)
The schema-based argument filter (from #647) strips all kwargs not
declared in input_schema. This breaks handlers that accept **kwargs
for pass-through to ChromaDB or other backends.
Add inspect.Parameter.VAR_KEYWORD check before filtering — handlers
with **kwargs receive all arguments unfiltered.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: guard inspect.signature failure, default to filtering
Wrap inspect.signature() in try/except — on failure, default to
filtering (safe fallback). Addresses Copilot feedback on fragility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: add stage-1 backend abstraction seam
Introduce the first upstreamable storage seam for MemPalace without
bringing in the PostgreSQL spike or any benchmark artifacts.
This change adds a small backend package with:
- BaseCollection as the minimal collection contract
- ChromaBackend/ChromaCollection as the default implementation
It then routes the main runtime collection consumers through that seam:
- palace.py
- searcher.py
- layers.py
- palace_graph.py
- mcp_server.py
- miner.status()
Behavioral constraints kept for stage 1:
- ChromaDB remains the only backend and the default path
- no config/env backend selection yet
- no PostgreSQL code
- no benchmark or research files
- existing tests stay unchanged
Important compatibility details:
- read paths now call the seam with create=False so they still surface
the existing 'no palace found' behavior instead of silently creating
empty collections
- write paths keep create=True semantics through palace.get_collection()
- layers/searcher retain a chromadb module attribute so the existing
mock-based tests can keep patching PersistentClient unchanged
- ChromaBackend only creates palace directories on create=True, which
preserves mocked read-path tests that use fake read-only paths
Verification:
- python3 -m py_compile mempalace/backends/__init__.py mempalace/backends/base.py mempalace/backends/chroma.py mempalace/palace.py mempalace/searcher.py mempalace/layers.py mempalace/palace_graph.py mempalace/mcp_server.py mempalace/miner.py
- pytest -q # 529 passed, 106 deselected
* refactor: clean up stage-1 seam compatibility shims
Tighten the stage-1 backend abstraction branch after review.
This follow-up does three small things:
- keep the chromadb compatibility hook in searcher.py and layers.py,
but express it through the backends.chroma module so it no longer
reads like an accidental unused import
- fix the palace_graph.py helper alias to avoid the local name collision
flagged by ruff (imported helper vs local _get_collection wrapper)
- preserve the existing mock-based test patch points unchanged while
keeping the new backend seam intact
Why this matters:
- the direct form looked like a
dead import in review, even though it was intentionally preserving the
existing test seam ( and
)
- palace_graph.py had a real lint issue ( redefinition) that was
small but worth fixing before a public PR
Verification:
- /opt/homebrew/bin/ruff check mempalace/backends/__init__.py mempalace/backends/base.py mempalace/backends/chroma.py mempalace/palace.py mempalace/searcher.py mempalace/layers.py mempalace/palace_graph.py mempalace/mcp_server.py mempalace/miner.py
- pytest -q tests/test_layers.py tests/test_searcher.py
- pytest -q # 529 passed, 106 deselected
* docs: explain backend shim imports in search paths
Add short code comments in searcher.py and layers.py explaining why the
module-level `chromadb` alias remains after the stage-1 backend seam
refactor.
The alias is intentional: it preserves the existing mock patch points used
by the current test suite (`mempalace.searcher.chromadb.PersistentClient`
and `mempalace.layers.chromadb.PersistentClient`) while the runtime logic
now flows through the backend abstraction.
This keeps the public PR easier to review because the apparent "unused
import" now has an explicit reason next to it.
Verification:
- /opt/homebrew/bin/ruff check mempalace/searcher.py mempalace/layers.py
- pytest -q tests/test_layers.py tests/test_searcher.py
* refactor: reuse a default backend instance in palace helper
Tighten the stage-1 backend seam by promoting the default Chroma backend
adapter to a module-level singleton in `mempalace/palace.py`.
This keeps the stage-1 scope unchanged — Chroma is still the only backend
wired in this branch — but avoids constructing a fresh `ChromaBackend()`
object on every `get_collection()` call. The backend is stateless today,
so this is a readability/cleanup change rather than a behavioral one.
Why this helps:
- makes `palace.get_collection()` read like a real default factory instead
of an inline constructor call
- keeps the stage-1 branch a little cleaner before opening the public PR
- does not widen the backend surface or change any config/runtime behavior
Verification:
- python3 -m py_compile mempalace/palace.py
- pytest -q tests/test_miner.py tests/test_layers.py tests/test_searcher.py
- pytest -q # 529 passed, 106 deselected
* fix: harden read-only seam behavior and update seam tests
Preserve the stage-1 backend abstraction while closing the real read-path
regression surfaced in PR review.
What changed:
- make ChromaBackend.get_collection(create=False) fail fast when the palace
directory does not exist instead of letting PersistentClient create it as a
side effect
- update miner.status() to call get_collection(..., create=False) so status
keeps the historical 'No palace found' behavior
- remove the temporary chromadb shim aliases from layers.py and searcher.py
now that the tests patch the seam directly
- add focused tests for the new backends package, including ChromaCollection
delegation and ChromaBackend create=True/create=False behavior
- retarget layer/searcher tests to patch the backend seam instead of patching
chromadb.PersistentClient inside production modules
- add a regression test that status() does not create an empty palace when the
target path is missing
Verification:
- ruff check .
- uv run pytest -q
- uv run pytest -q tests/test_backends.py tests/test_cli.py tests/test_mcp_server.py tests/test_layers.py tests/test_searcher.py tests/test_miner.py
Notes:
- the separate benchmark/slow/stress layer was started as a soak but not used
as the merge gate for this PR branch
* refactor: drop duplicate mcp collection cache declaration
Remove a redundant `_collection_cache = None` assignment in
`mempalace/mcp_server.py` left over after the stage-1 backend seam refactor.
This does not change behavior; it only trims review noise in the MCP server
module after the read-path hardening pass.
Verification:
- ruff check mempalace/mcp_server.py
- uv run pytest -q tests/test_mcp_server.py
---------
Co-authored-by: Sergey Kuznetsov <sergey@iterudit.com>
- Replace 'while offset < count/total' with 'while True' + break on short batch
- Fixes tool_list_rooms iterating over unfiltered col.count() when wing filter active
- Fixes all 4 paginated functions (tool_status, tool_list_wings, tool_list_rooms,
tool_get_taxonomy) missing early-exit when batch smaller than batch_size
- Remove unused 'total' variable in tool_list_wings, tool_list_rooms, tool_get_taxonomy
(replaced col.count() with accessibility check only)
Per bensig review comments on PR #371
Three critical bugfixes:
1. MCP server hangs on null arguments (#394) — `params.get("arguments", {})`
returns None when JSON has `"arguments": null`. Changed to `or {}`.
2. cmd_repair infinite recursion (#395) — trailing slash on palace_path
caused backup_path to be inside the source dir. Strip trailing sep.
3. OOM on large transcript files (#396) — split_mega_files.py and
normalize.py load entire files into memory. Added 500MB safety limit
with clear skip/error messages.
Closes#394, #395, #396.
Review fixes (from Sage's review):
- Restore mtime check in file_already_mined (check_mtime=True for miner)
- Restore limit=10000 on MCP metadata fetches to prevent OOM on large palaces
- Apply _SAFE_NAME_RE regex in sanitize_name (was dead code)
- Drop raw_aaak metadata duplication in diary_write
- chmod 0o700 on WAL dir, 0o600 on WAL file
- Add check_same_thread=False on KnowledgeGraph SQLite connection
- Remove __del__ (unreliable) and dead PRAGMA foreign_keys=ON
Remove datetime.now() from drawer_id hash so same content + wing + room
always produces the same ID. This enables the idempotency check that
returns "already_exists" on duplicate writes.
The 30s TTL metadata cache returned stale data between test runs and
after write operations. Reverted to direct col.get() reads which match
the original behavior and pass all tests.
- _client → _client_cache to match conftest.py reset fixture
- _get_collection now uses _get_client() return value instead of stale ref
- Restore .pytest_cache and other dirs missing from palace.py SKIP_DIRS
icciAaron