The save hook and precompact hook were telling the agent to write
diary entries, add drawers, and add KG triples IN THE CHAT WINDOW.
Every line written stays in conversation history and retransmits on
every subsequent turn — ~$1/session in wasted tokens.
Fix: hooks now say "saved in background, no action needed" and use
decision: allow instead of block. The agent continues working without
interruption. All filing happens via the background pipeline.
Also updated hooks README with:
- Known limitation: hooks require session restart after install
- Updated cost section: zero tokens, background-only
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Non-trivial merge in convo_miner.py: this branch's _file_convo_chunks
(purge stale + upsert with normalize_version) and develop's
_file_chunks_locked (mine_lock + double-checked file_already_mined)
both touched the same critical section. Combined into a single
_file_chunks_locked helper that does lock → double-check → purge →
upsert, preserving both the multi-agent safety guarantee from #784
and the schema-rebuild contract from this PR.
Also folds develop's mine_lock import into both miner.py and
convo_miner.py alongside NORMALIZE_VERSION.
707/707 tests pass, ruff + format clean under CI-pinned 0.4.x.
Without this, the strip_noise improvement only helps new mines. Every
user who had already mined Claude Code JSONL sessions would keep their
noise-polluted drawers forever, because convo_miner's file_already_mined
skip short-circuits before re-processing.
Adds a versioned schema gate so upgrades propagate silently:
- palace.NORMALIZE_VERSION=2 — bumped when the normalization pipeline
changes shape (this PR's strip_noise is the v1→v2 bump).
- file_already_mined now returns False if the stored normalize_version
is missing or less than current, triggering a rebuild on next mine.
- Both miners stamp drawers with the current normalize_version.
- convo_miner now purges stale drawers before inserting fresh chunks
(mirrors miner.py's existing delete+insert), extracted into
_file_convo_chunks helper to keep mine_convos under ruff's C901 limit.
User experience: upgrade mempalace, run `mempalace mine` as usual, old
noisy drawers get silently replaced with clean ones. No erase needed,
no "you need to rebuild" changelog footgun.
Tests:
- test_file_already_mined_returns_false_for_stale_normalize_version —
pins the version gate contract for missing/v1/current.
- test_add_drawer_stamps_normalize_version — fresh project-miner drawers
carry the field.
- test_mine_convos_rebuilds_stale_drawers_after_schema_bump — end-to-end
proof that a pre-v2 palace gets silently cleaned on next mine, with
orphan drawers purged and NOT skipped.
Existing test_file_already_mined_check_mtime updated to include the
new field; all other tests unaffected.
The initial strip_noise() regressed on three fronts when audited against
adversarial user content — each verified with executable repros against
the cherry-picked code:
1. `<tag>.*?</tag>` with re.DOTALL span-ate across messages: one
stray unclosed <system-reminder> anywhere in a session merged with
the next closing tag, silently deleting everything between them
(including full assistant replies).
2. `.*\(ctrl\+o to expand\).*\n?` nuked entire lines of user prose
whenever a user happened to document the TUI shortcut.
3. `Ran \d+ (?:stop|pre|post)\s*hook.*` with IGNORECASE ate the
second sentence from "our CI has a stop hook ... Ran 2 stop hooks
last week" — legitimate user commentary.
These are unambiguous violations of the project's "Verbatim always"
design principle.
Fixes:
- All tag patterns are now line-anchored (`(?m)^(?:> )?<tag>`) and their
body forbids crossing a blank line (`(?:(?!\n\s*\n)[\s\S])*?`), so a
dangling open tag cannot eat neighboring messages.
- `_NOISE_LINE_PREFIXES` are line-anchored and case-sensitive — user
prose mentioning "CURRENT TIME:" mid-sentence is preserved.
- Hook-run chrome requires `(?m)^`, explicit hook names (Stop,
PreCompact, PreToolUse, etc.), and no IGNORECASE.
- "… +N lines" is line-anchored.
- "(ctrl+o to expand)" only matches Claude Code's actual collapsed-
output chrome shape `[N tokens] (ctrl+o to expand)`; a bare
parenthetical in user prose stays intact.
Scope:
- `strip_noise()` is no longer called on every normalization path.
Only `_try_claude_code_jsonl` invokes it, per-extracted-message — so
Claude.ai exports, ChatGPT exports, Slack JSON, Codex JSONL, and
plain text with `>` markers pass through fully verbatim. Per-message
application also makes span-eating structurally impossible.
Tests:
- 15 new tests in test_normalize.py pin the boundary: 6 guard user
content that must survive (each of the adversarial repros), 9 assert
real system chrome is still stripped. All pass; full suite 702 pass
(2 failures are the unrelated pre-existing version.py bug, cleared
by #820).
Known limitation (not fixed here): convo_miner.py does not delete
drawers on re-mine, so transcripts mined before this PR keep noise-
filled drawers until the user manually erases + re-mines. Proper fix
needs a schema-version field on drawer metadata + re-mine trigger —
out of scope for this PR.
Adding the per-file lock + double-checked file_already_mined() in the
previous commit pushed mine_convos cyclomatic complexity from 25 to 26,
just over ruff's max-complexity threshold. Hoist the locked critical
section into _file_chunks_locked() so the outer loop stays within
budget. No behavior change.
normalize.py now strips before filing:
- <system-reminder>, <command-message>, <command-name> tags
- <task-notification>, <user-prompt-submit-hook>, <hook_output> tags
- Hook status messages (CURRENT TIME, Checking verified facts, etc.)
- Claude Code UI chrome (ctrl+o to expand, progress bars, etc.)
- Collapsed runs of blank lines
This noise was going straight into drawers, wasting storage space
and polluting search results. strip_noise() runs on all normalized
output regardless of input format (JSONL, JSON, plain text).
689/689 tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: when multiple agents mine simultaneously, both pass
file_already_mined() check, both delete+insert the same file's
drawers, creating duplicates or losing data.
Fix: mine_lock() in palace.py — cross-platform file lock (fcntl on
Unix, msvcrt on Windows). Both miner.py and convo_miner.py now lock
per-file during the delete+insert cycle and re-check after acquiring
the lock.
Tested:
- Lock acquires and releases correctly
- Second agent blocks until first releases (0.25s wait)
- 33/33 existing tests pass
- Cross-platform: fcntl (macOS/Linux), msvcrt (Windows)
Based on v3.2.0 tag.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: parse Claude.ai privacy export with messages key and sender field (#677)
The privacy-export branch in _try_claude_ai_json only checked for the
"chat_messages" key, missing exports that use "messages" instead. It
also only read the "role" field while real privacy exports use "sender".
Both gaps caused the file to fall through to plain-text, producing a
single giant drawer.
Changes:
- Accept "messages" alongside "chat_messages" in the conversation-object
guard and inner extraction.
- Accept "sender" alongside "role" as the author field.
- Fall back to a top-level "text" key when content blocks are empty.
- Produce one transcript per conversation instead of concatenating all
conversations into a single blob.
- Extract shared logic into _collect_claude_messages helper.
- Add 6 regression tests covering each variant.
* style: apply ruff format to normalize.py
* fix: guard against null text field in Claude.ai export parsing
item.get("text", "").strip() crashes when "text" is explicitly null
in the JSON (legal and observed in some exports). Use
(item.get("text") or "").strip() and add a regression test.
---------
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
When external tools write to the palace database (CLI mining, scripts), the MCP server's cached ChromaDB collection becomes stale — its HNSW index doesn't know about new vectors. Develop already invalidates on inode changes (catches rebuilds) but not on mtime changes (misses in-place writes).
This PR:
- Adds st_mtime tracking alongside st_ino in _get_client; invalidates the cached client on either change.
- Adds the mempalace_reconnect MCP tool for explicit cache flush.
Original author: @jphein (#663). Original approval: @Ari4ka.
Skips test_missing_db_invalidates_cache on Windows (ChromaDB holds chroma.sqlite3 open).
'(r)roject' had a duplicate 'r', making it read as '(r)roject'
instead of the intended '(r)project'.
Small UX fix — no behavior change.
Co-authored-by: Arnold Wender <arnold.wender@gmail.com>
Full changelog from git history and merged PRs:
- v3.0.0 (2026-04-06): initial public release
- v3.1.0 (2026-04-09): 80+ commits, security hardening, Windows compat, tests 20→92
- Unreleased/v3.2.0: 50+ commits, i18n, backend seam, migrate command, more security
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- query_sanitizer: require matching quote pair in _strip_wrapping_quotes
- query_sanitizer: re-check MIN_QUERY_LENGTH after trim in tail_sentence path
- migrate: neutral confirmation message accurate for both migrate and repair
- cli: os.path.normpath instead of rstrip to handle '/' root edge case
* docs: add CLAUDE.md + mission/principles to AGENTS.md
Add project mission, design principles, and contribution guidelines
to CLAUDE.md (new file) and prepend them to AGENTS.md.
Six non-negotiable principles: verbatim always, incremental only,
entity-first, local-first zero API, performance budgets, privacy
by architecture.
* docs: update CLAUDE.md with Milla's edits, add MISSION.md, symlink AGENTS.md
CLAUDE.md:
- Add Zettelkasten + AAAK explanation to mission (Milla's edit)
- Add 7th principle: background everything
- Update project structure to match current develop (all modules)
- Fix hooks listing to match actual public repo
- Add backends/ to structure and key files
MISSION.md:
- Milla's personal narrative on why MemPalace exists
- Origin story, AAAK inside joke, v4 goals
AGENTS.md:
- Now symlinks to CLAUDE.md (single source of truth)
* docs: trim MISSION.md — remove v4 notes and workflow (moved to private)
* fix: register 0-chunk files to prevent re-processing on every mine (#654)
mine_convos() has three early-exit paths (OSError, content too short,
zero chunks) that skip writing anything to ChromaDB. Since
file_already_mined() checks for the presence of a document with a
matching source_file, these files are re-read and re-processed on
every subsequent run.
Add _register_file() that upserts a lightweight sentinel document
(room="_registry", ingest_mode="registry") so file_already_mined()
returns True on future runs.
Note: Bug 2 from the issue (drawers_added counter always 0) was
already resolved upstream via the switch from collection.add() to
collection.upsert().
* fix: resolve macOS path symlink in test + remove unused variable
The _chunk_by_exchange() function was silently truncating AI responses
to 8 lines via ai_lines[:8]. Any content beyond line 8 was discarded,
violating the project's verbatim storage principle.
Now the full AI response is preserved. When a combined exchange exceeds
CHUNK_SIZE (800 chars, aligned with miner.py), it is split across
consecutive drawers instead of being truncated.
* fix: return "general" room from process_file error paths (#586)
process_file() returned (0, None) for already-mined, unreadable, and
too-short files. In --dry-run mode the caller always enters the
room_counts branch, so None ended up as a dict key and crashed the
summary printer with "unsupported format string passed to
NoneType.__format__".
Returning "general" instead of None makes the function contract
explicit: it always yields (int, str). This matches the consensus
fix discussed in the issue thread.
* style: apply ruff format to test_miner.py
* fix: skip arg whitelist for handlers accepting **kwargs (#572)
The schema-based argument filter (from #647) strips all kwargs not
declared in input_schema. This breaks handlers that accept **kwargs
for pass-through to ChromaDB or other backends.
Add inspect.Parameter.VAR_KEYWORD check before filtering — handlers
with **kwargs receive all arguments unfiltered.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: guard inspect.signature failure, default to filtering
Wrap inspect.signature() in try/except — on failure, default to
filtering (safe fallback). Addresses Copilot feedback on fragility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: allow Unicode in sanitize_name() — Latvian, CJK, Cyrillic names (#637)
_SAFE_NAME_RE was ASCII-only ([a-zA-Z0-9]), rejecting valid Unicode
names like "Jānis" or "太郎". Changed to \w which matches Unicode
word characters (letters, digits, underscore) in Python 3.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: tighten Unicode regex, add sanitize_name tests
Use [^\W_] for first/last char to allow Unicode letters/digits but
reject leading/trailing underscores (Copilot feedback). Add 7 tests
covering Latvian, CJK, Cyrillic, path traversal, and edge cases.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
AI agents calling `mempalace init` hang on the interactive confirmation
prompt. The --yes flag skips it, enabling automated setup.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unused `json` and `current_lang` imports from
mempalace/i18n/test_i18n.py (F401)
- Reformat Dialect.__init__ signature in mempalace/dialect.py
(ruff format collapses multi-line signature, adds blank line
after lazy import)
Both auto-fixes from `ruff check --fix` / `ruff format`. No behavioral
changes.
Add language dictionaries: English, French, Korean, Japanese, Spanish,
German, Simplified Chinese, Traditional Chinese.
Each language is a single JSON file with:
- Localized terms (palace, wing, closet, drawer, etc.)
- CLI output strings with {var} interpolation
- AAAK compression instructions in that language
- Regex patterns for offline topic/quote/action extraction
Usage: Dialect(lang="ko") or set "language": "ko" in config.
Contributors can add new languages by copying en.json and translating.
Dialect class now accepts lang param and loads AAAK instruction +
regex patterns from the i18n dictionary automatically.
Tests: mempalace/i18n/test_i18n.py — all 8 languages pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The repo moved to the MemPalace org but several docs still point at the
old milla-jovovich URLs. Also, CONTRIBUTING.md tells people to PR
against main while the actual workflow (per ROADMAP.md) targets develop.
Files touched:
- CONTRIBUTING.md: clone URL, issues URL, PR target branch
- examples/gemini_cli_setup.md: clone URL
- integrations/openclaw/SKILL.md: homepage and license URLs
The `_load_api_key()` function in longmemeval_bench.py and locomo_bench.py
searched for API keys in a fixed path (`~/.config/lu/keys.json`) using
personal key names (`anthropic_milla`, `anthropic_claude_code_main`).
This leaks internal infrastructure details into the public codebase and
trains contributors to store credentials in a non-standard location
rather than using the standard ANTHROPIC_API_KEY env var.
Simplified to: CLI flag > env var > empty string. Updated help text
and HYBRID_MODE.md docs to match.
Co-authored-by: Tadao <tadao@travisfixes.com>
Path("~/foo") does not expand tilde on its own, causing
`mempalace split ~/some/dir` to silently find no files.
Fix by calling .expanduser().resolve() in both places the
path is constructed: cmd_split in cli.py (defensive, at the
CLI boundary) and main() in split_mega_files.py (the root cause).
Co-authored-by: Brooke Whatnall <brookewhatnall@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Igor Lins e Silva