On Windows, projects containing git-submodule junctions or dev-drive
reparse points cause iterdir() to list the entry successfully but
Path.is_dir() to raise OSError when it calls stat() internally.
Reproducer: any Windows project with a submodule checked out as a
junction (e.g. skills/pr-perfect) crashes mempalace init with:
OSError: [WinError 448] The path cannot be traversed because it
contains an untrusted mount point
Fix: wrap every is_dir() call in detect_rooms_from_folders with
try/except OSError so the scanner skips inaccessible entries and
continues rather than aborting.
Covers both the top-level pass and the one-level-deep nested pass.
Two new tests mock the OSError on specific paths and verify the
function returns correct rooms from the remaining accessible entries.
- Replace 'while offset < count/total' with 'while True' + break on short batch
- Fixes tool_list_rooms iterating over unfiltered col.count() when wing filter active
- Fixes all 4 paginated functions (tool_status, tool_list_wings, tool_list_rooms,
tool_get_taxonomy) missing early-exit when batch smaller than batch_size
- Remove unused 'total' variable in tool_list_wings, tool_list_rooms, tool_get_taxonomy
(replaced col.count() with accessibility check only)
Per bensig review comments on PR #371
Delete existing drawers for a file before re-inserting fresh chunks.
Converts re-mines from upsert (hnswlib updatePoint path, thread-unsafe
on macOS ARM + chromadb 0.6.3) into delete+insert (safe addPoint path).
Credit: @StefanKremen (#523)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reads documents and metadata directly from ChromaDB's SQLite (bypassing
the API that fails on version-mismatched databases), then reimports into
a fresh palace using the currently installed ChromaDB.
Fixes the 3.0.0 → 3.1.0 upgrade path where chromadb was downgraded from
1.5.x to 0.6.x, breaking the on-disk storage format.
- Detects chromadb version from SQLite schema (0.6.x vs 1.x)
- Extracts all drawers with full metadata via raw SQL
- Builds fresh palace in temp dir, swaps atomically
- Backs up original palace before any changes
- Supports --dry-run to preview without modifying
Fixes#457
Addresses community feedback:
- Add --wing flag to scope dedup to a single wing (catches cross-wing
duplicates when same source mined into multiple wings)
- Document that threshold is cosine distance (not similarity) with
guidance on values: 0.15 for near-identical, 0.3-0.4 for paraphrased
- Confirmed shutil import is present in repair.py (line 32)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: convo_miner.py used collection.add() instead of upsert(),
so repeated mine runs pushed duplicate entries into the HNSW graph.
At scale (50K+ drawers) this causes link_lists.bin to grow to terabytes
and eventually segfault.
Changes:
- convo_miner.py: add() → upsert() (the one-line root cause fix)
- repair.py: new module — scan for corrupt IDs, prune them, or rebuild
the HNSW index from scratch. Backs up only chroma.sqlite3 (not the
bloated HNSW files). Recreates collection with hnsw:space=cosine.
- dedup.py: new module — detect and remove near-duplicate drawers from
the same source file using cosine similarity. No API calls.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reads documents and metadata directly from ChromaDB's SQLite (bypassing
the API that fails on version-mismatched databases), then reimports into
a fresh palace using the currently installed ChromaDB.
Fixes the 3.0.0 → 3.1.0 upgrade path where chromadb was downgraded from
1.5.x to 0.6.x, breaking the on-disk storage format.
- Detects chromadb version from SQLite schema (0.6.x vs 1.x)
- Extracts all drawers with full metadata via raw SQL
- Builds fresh palace in temp dir, swaps atomically
- Backs up original palace before any changes
- Supports --dry-run to preview without modifying
Fixes#457
Three critical bugfixes:
1. MCP server hangs on null arguments (#394) — `params.get("arguments", {})`
returns None when JSON has `"arguments": null`. Changed to `or {}`.
2. cmd_repair infinite recursion (#395) — trailing slash on palace_path
caused backup_path to be inside the source dir. Strip trailing sep.
3. OOM on large transcript files (#396) — split_mega_files.py and
normalize.py load entire files into memory. Added 500MB safety limit
with clear skip/error messages.
Closes#394, #395, #396.
Review fixes (from Sage's review):
- Restore mtime check in file_already_mined (check_mtime=True for miner)
- Restore limit=10000 on MCP metadata fetches to prevent OOM on large palaces
- Apply _SAFE_NAME_RE regex in sanitize_name (was dead code)
- Drop raw_aaak metadata duplication in diary_write
- chmod 0o700 on WAL dir, 0o600 on WAL file
- Add check_same_thread=False on KnowledgeGraph SQLite connection
- Remove __del__ (unreliable) and dead PRAGMA foreign_keys=ON
Remove datetime.now() from drawer_id hash so same content + wing + room
always produces the same ID. This enables the idempotency check that
returns "already_exists" on duplicate writes.
The 30s TTL metadata cache returned stale data between test runs and
after write operations. Reverted to direct col.get() reads which match
the original behavior and pass all tests.
- _client → _client_cache to match conftest.py reset fixture
- _get_collection now uses _get_client() return value instead of stale ref
- Restore .pytest_cache and other dirs missing from palace.py SKIP_DIRS
Addresses Issue #333: AI agents prepending system prompts to search queries
causes embedding retrieval to collapse (89.8% → 1.0% R@10).
Mitigation approach (減災):
- New query_sanitizer.py with 4-stage pipeline:
Step 1: passthrough for short queries (≤200 chars)
Step 2: question extraction (finds ? sentences) → ~85-89% recovery
Step 3: tail sentence extraction → ~80-89% recovery
Step 4: tail truncation fallback → ~70-80% recovery
Worst case without sanitizer: 1.0% (catastrophic)
Worst case with sanitizer: ~70-80% (survivable)
- mcp_server.py: tool_search applies sanitizer before ChromaDB query
- MCP schema: query description warns agents not to include prompts
- New 'context' parameter separates background info from search intent
- Sanitizer metadata included in response when triggered
22 new tests covering all pipeline stages and real-world scenarios.
Made-with: Cursor
The _count_human_messages() function previously only handled Claude Code
transcript format: {"message": {"role": "user", "content": "..."}}
Codex CLI transcripts use a different schema:
{"type": "event_msg", "payload": {"type": "user_message", "message": "..."}}
This meant the stop-hook auto-save threshold never triggered for Codex
sessions because the count always returned 0.
Added detection for the Codex format so both Claude Code and Codex CLI
transcripts are counted correctly.
The initialize handler hardcoded protocolVersion "2024-11-05", which
causes newer MCP clients (e.g. Claude Code) to reject the connection
when they negotiate "2025-11-25" or later.
Echo the client's requested version if it is in the supported set,
otherwise fall back to the latest supported version. This keeps
backwards compatibility with older clients while allowing newer ones
to connect.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI was installing latest ruff (0.15.x) which has different formatting
rules than our local 0.4.x. Pin to ruff>=0.4.0,<0.5 for consistency.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add tests for config, convo_miner, spellcheck, knowledge_graph
- Fix Windows PermissionError in test cleanup (chromadb file locks)
- Add UTF-8 encoding to split_mega_files, entity_registry, hooks_cli
- Fix mcp_server parse_known_args logging for unknown args
- Set coverage threshold to 85 in pyproject.toml and CI
- Reset all version files to 3.0.11
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
grtninja