entity_detector.py was refactored in #911 to load candidate patterns
from i18n locale JSON files, supporting non-Latin scripts (Cyrillic,
accented Latin, etc.). But three other code paths still hardcoded the
ASCII-only regex [A-Z][a-z]{2,}, silently missing non-Latin entity
names in metadata tagging, closet indexing, and registry lookups.
Replace the hardcoded regex with a shared _candidate_entity_words()
helper that reuses the same i18n candidate_patterns as entity_detector.
- repair.py: define backup_path before the conditional block so it is
always in scope when the except handler references it
- migrate.py: restore old palace from .old if both os.rename and
shutil.move fail during the swap step
- repair.py: wrap upsert loop in try/except; restore from backup on
failure instead of leaving a partially rebuilt collection
- migrate.py: replace non-atomic rmtree+move with rename-aside swap
so a crash between the two calls does not destroy both copies
- cli.py: use offset += len(batch["ids"]) with empty-batch guard
instead of fixed offset += batch_size to prevent skipping drawers
Python's \b is a \w/non-\w transition. Devanagari vowel signs (matras)
like ा ी ु are Unicode category Mc (Mark, Spacing Combining) — not \w.
This means \b splits mid-word on every matra: names like अनीता (Anita)
truncate to अनीत, and person-verb patterns like \bराज\s+ने\s+कहा\b
never match because \b fails after the final matra of कहा.
Same issue affects Arabic, Hebrew, Thai, Tamil, and every other script
whose words contain combining marks.
Fix: locales with combining-mark scripts declare a boundary_chars field
in their entity section (e.g. "\\w\\u0900-\\u097F" for Hindi). The i18n
loader replaces every \b in that locale's patterns with a script-aware
lookaround that treats the declared characters as "inside-word", and
pre-wraps candidate/multi_word patterns with the same boundary.
Default behavior (no boundary_chars) keeps standard \b — en, pt-br, ru,
it are unchanged.
Changes:
- mempalace/i18n/__init__.py: add _script_boundary, _expand_b,
_wrap_candidate, _collect_entity_section; candidate_patterns are now
returned fully-wrapped (boundary + capture group applied)
- mempalace/entity_detector.py: extract_candidates compiles pre-wrapped
candidate patterns directly instead of re-wrapping with \b
- tests/test_entity_detector.py: 5 new tests for Devanagari boundaries
(name extraction with/without boundary_chars, person-verb firing,
English regression)
BCP 47 language tags are case-insensitive (RFC 5646 §2.1.1) but the
locale files mix conventions (pt-br.json vs zh-CN.json). On
case-sensitive filesystems, '--lang PT-BR' or '--lang zh-cn' silently
missed the file, _load_entity_section returned {}, and entity
detection ran in English with no warning.
The cache key in get_entity_patterns was built from raw input, so
('PT-BR',) and ('pt-br',) produced two distinct entries, both wrong.
Add _canonical_lang(lang) that resolves any casing to the on-disk
filename stem via lowercase comparison, and route load_lang,
_load_entity_section, and the cache key through it.
Closes#927
Introduces a version label to the command-line interface, displaying the current MemPalace version in the help text. Adds a `--version` flag to allow users to easily check the version and exit.
CLI strings, AAAK instruction, regex patterns, and entity section
with person-verb, pronoun, dialogue, and candidate patterns for
Latin+diacritics names (Joao, Ines, Angela).
Follows the i18n entity framework from #911.
Adds 34 prepositions and conjunctions to reduce false positives
in entity detection when these words appear sentence-initial.
Co-Authored-By: almirus <almirus@users.noreply.github.com>
Add ru.json with full Russian translations for CLI strings, palace
terminology, AAAK compression instruction, and regex patterns for
topic/action extraction with Cyrillic character classes.
No code changes needed -- the i18n module auto-discovers language
files via *.json glob in the i18n directory.
Move all entity-detection lexical patterns (person verbs, pronouns,
dialogue markers, project verbs, stopwords, candidate character class)
out of hardcoded module-level constants and into the entity section of
each locale's JSON in mempalace/i18n/. Adds a languages parameter to
every public function so callers union patterns across the desired
locales. The default stays ("en",), so all existing callers and tests
behave unchanged.
Also adds:
- get_entity_patterns(langs) helper in mempalace/i18n/ that merges
patterns across requested languages, dedupes lists, unions stopwords,
and falls back to English for unknown locales
- MempalaceConfig.entity_languages property + setter, with env var
override (MEMPALACE_ENTITY_LANGUAGES, comma-separated)
- mempalace init --lang en,pt-br flag (persists to config.json)
- Per-language candidate_pattern so non-Latin scripts (Cyrillic,
Devanagari, CJK) can register their own character classes instead of
being silently dropped by the ASCII-only [A-Z][a-z]+ default
- _build_patterns LRU cache keyed by (name, languages) so multi-language
callers don't poison each other's cache slots
Why now: the open language PRs (#760 ru, #773 hi, #778 id, #907 it) only
add CLI strings via mempalace/i18n/. PR #156 (pt-br) is the first that
needed entity_detector changes and inlined a _PTBR variant of every
constant. That doesn't scale past 2-3 languages — every text gets
checked against every language's patterns regardless of relevance, and
candidate extraction still drops accented and non-Latin names.
This PR sets the standard so future locale contributors only edit one
JSON file (no Python changes), and entity detection scales linearly
with how many languages a user actually enabled, not how many ship.
* fix: restrict file permissions on sensitive palace data
On Linux with default umask (022), several files and directories
containing personal data were created world-readable. This patch
applies chmod 0o700 to directories and 0o600 to files immediately
after creation, wrapped in try/except for Windows compatibility.
Files hardened:
- hooks_cli.py: hook_state/ directory and hook.log
- entity_registry.py: entity_registry.json (names, relationships)
- knowledge_graph.py: knowledge_graph.sqlite3 parent directory
- exporter.py: export output directory and wing subdirectories
- config.py: people_map.json (name mappings)
- mcp_server.py: WAL file creation uses atomic os.open (TOCTOU fix)
Refs: MemPalace/mempalace#809
* fix: avoid redundant chmod calls on hot paths
- hooks_cli.py: chmod STATE_DIR and hook.log only on first creation,
not on every _log() call (hooks fire on every Stop event)
- exporter.py: track created wing dirs to skip redundant makedirs +
chmod on the same directory across batches
- mcp_server.py: remove redundant _WAL_FILE.chmod after os.open
already set mode=0o600 atomically
Refs: MemPalace/mempalace#809
* fix: add provenance header and speaker IDs to Slack transcript imports
Slack exports are multi-party chats where no speaker is inherently
the "user" or "assistant". The parser previously assigned these roles
purely by position, allowing a crafted export to place attacker text
in the "user" role — making it appear as the memory owner's words
in all future retrieval (data poisoning via stored memory).
Changes:
- Add provenance header marking Slack transcripts as multi-party
with positional (unverified) role assignment
- Prefix each message with the original speaker ID ([U1], [U2], etc.)
so downstream consumers can distinguish authors
- Keep user/assistant role alternation for exchange-pair chunking
compatibility with convo_miner.py
Tests:
- Provenance header presence and content
- Speaker ID preservation in output
- Attacker-first-message attribution verification
Refs: MemPalace/mempalace#809
* fix: move Slack provenance to footer, sanitize speaker IDs, extract constant
- Move provenance notice from header to footer to prevent it becoming
a standalone ChromaDB drawer via paragraph chunking on exports
with fewer than 3 exchange pairs (violates verbatim-always principle)
- Sanitize speaker user_id/username: strip brackets, newlines, and
control characters to prevent chunk-boundary injection via crafted
Slack exports
- Extract header string to _SLACK_PROVENANCE_FOOTER module constant,
consistent with _TOOL_RESULT_* constants pattern; tests import it
instead of duplicating the literal
Refs: MemPalace/mempalace#809
* feat: include created_at timestamp in search results (closes#465)
Surface the existing filed_at metadata as created_at in search result
objects returned by search_memories(). Enables temporal reasoning over
search hits without additional queries.
* Feat: add fallback for missing filed_at metadata
* fix(hooks): stop precompact hook from blocking compaction
The precompact hook unconditionally returned {"decision": "block"},
which in Claude Code means "cancel compaction" with no retry mechanism.
This made /compact permanently broken for all plugin users.
Changed hook_precompact() to mine the transcript synchronously (so data
lands before compaction) and return {"decision": "allow"}. This matches
the standalone bash hook in hooks/ which already uses allow.
Also extracted _get_mine_dir() and _mine_sync() helpers so precompact
can mine from the transcript directory, not just MEMPAL_DIR.
Stop hook behavior is unchanged -- left for #673 which implements the
full silent save path.
Closes#856, closes#858.
* fix: use empty JSON instead of invalid \"allow\" decision value
Claude Code only recognizes \"block\" as a top-level decision value.
\"allow\" is a permissionDecision value for PreToolUse hooks, not a
valid top-level decision. The correct way to not block is to return
empty JSON. Caught by #872.
* fix(mcp): redirect stdout to stderr during import to protect JSON-RPC channel (#225)
Fixes#225.
Several transitive dependencies (chromadb, onnxruntime, posthog) print
banners and warnings to stdout — sometimes at the C level — during the
mcp_server import chain. Because the MCP protocol multiplexes JSON-RPC
over stdio, any non-JSON output on stdout corrupted the message stream
and broke Claude Desktop's parser with errors like:
MCP mempalace: Unexpected token '*', "**********"... is not valid JSON
MCP mempalace: Unexpected token 'E', "EP Error D"... is not valid JSON
MCP mempalace: Unexpected token 'F', "Falling ba"... is not valid JSON
Reproduced on Windows 11 with mempalace 3.0.0 / Python 3.10 / Claude
Desktop 1.1062.0.
Fix: at module load, redirect stdout to stderr at both the Python level
(sys.stdout = sys.stderr) and the file-descriptor level (os.dup2(2, 1))
to catch C-level prints, while preserving the real stdout for later
restore. main() calls _restore_stdout() right before entering the
protocol loop so JSON-RPC responses still go to the real stdout.
Adds tests/test_mcp_stdio_protection.py with three regression tests:
- module-level redirect is in place after import
- _restore_stdout() restores the original stdout (idempotent)
- 'python -m mempalace.mcp_server' with empty stdin emits no stdout
* style: reformat with ruff 0.4 (CI version) for #225
Partially addresses #185.
`mempalace init <dir>` writes `mempalace.yaml` and `entities.json` into
the project root. When <dir> is a git repository, those files have no
default protection and risk being committed by accident — the loudest
concern in the original report.
This PR adds `_ensure_mempalace_files_gitignored()` which runs at the
end of cmd_init: if <dir>/.git exists, append the two filenames to
.gitignore (creating it if necessary) under a clearly-marked block.
The helper is conservative:
- only runs when <dir>/.git is present (no-op for non-git projects)
- skips entries already present (no duplicates)
- preserves existing .gitignore content
- handles files without trailing newlines
This does NOT relocate the files to ~/.mempalace/wings/<wing>/ as the
issue's 'Expected' section proposes — that's a behavioral change with
miner/config implications and warrants a separate design discussion.
The gitignore safeguard removes the immediate risk without breaking any
existing flow.
Tests: 5 cases in tests/test_init_gitignore_protection.py covering
no-op, fresh creation, partial append, idempotency, and missing-newline
edge case.
Fixes#195.
When ChromaDB returns no documents (empty palace, or wing/room filter
that excludes everything), it returns the shape:
{"documents": [], "metadatas": [], "distances": []}
Indexing `results["documents"][0]` blindly raises IndexError instead of
the expected 'no results' response. Affected: searcher.search(),
searcher.search_memories() (drawer + closet branches plus the
total_before_filter aggregate), and Layer3.search() / Layer3.search_raw().
Adds a tiny private helper `searcher._first_or_empty(results, key)` that
safely extracts the inner list, returning [] for any of: missing key,
empty outer list, [None], or [[]]. layers.py imports the same helper to
avoid duplicating the guard.
Tests: tests/test_empty_chromadb_results.py covers all observed shapes
plus a documentation-style test that pins the original IndexError so
future readers understand why the helper exists.
tool_status() called _get_collection() with the default create=False,
which throws when the ChromaDB collection does not exist yet (valid
palace, zero drawers). The exception was swallowed and status returned
"No palace found" even though init had completed successfully.
Switching to create=True bootstraps an empty collection on first
status call, matching what the write path already does.
Fix suggested by @hkevinchu in the issue.
* fix: make entity_registry.research() local-only by default
research() previously called _wikipedia_lookup() unconditionally,
sending entity names to en.wikipedia.org on every uncached lookup.
This violates the project's local-first and privacy-by-architecture
principles documented in CLAUDE.md.
Changes:
- research() now returns "unknown" for uncached words by default
- New allow_network=True parameter required for Wikipedia lookups
- Wikipedia 404 now returns "unknown" instead of asserting "person"
with 0.70 confidence, preventing entity registry poisoning
- Added privacy warning docstring to _wikipedia_lookup()
- Added tests for local-only default, opt-in network, 404 handling,
and cache-not-persisted-on-local-only behaviour
Refs: MemPalace/mempalace#809
* fix: improve research() cache read path and deduplicate test mocks
- Use .get() instead of .setdefault() for cache reads in research()
so the local-only path never mutates _data unnecessarily
- Move .setdefault() to the network-write path only
- Use result.setdefault() for word/confirmed keys to ensure
consistent return shape across all _wikipedia_lookup error paths
- Extract duplicated mock_result dict into _MOCK_SAOIRSE_PERSON
constant shared by 3 test functions
Fixes#210.
The CLI requires a positional <dir> argument. Previous docs emphasized
that init 'sets up ~/.mempalace/' which misled users into expecting
no arguments. Now the docs show <dir> is required, offer '.' as the
usage for the current directory, and reword the description so the
project-directory scan is listed first.
Three issues flagged by bensig on the i18n PR before merge:
1. ko.json: status_drawers used {drawers} instead of {count}, causing
the Korean UI to show the raw template string instead of the actual
drawer count. All other 7 languages use {count}.
2. Test file was shipped inside the package at mempalace/i18n/test_i18n.py
with a sys.path.insert hack. Moved to tests/test_i18n.py per the
project convention in AGENTS.md.
3. Dialect.from_config() passed lang=config.get("lang") which defaults
to None, causing __init__ to inherit whatever language was loaded
earlier via module-level state. Now defaults to "en" explicitly so
from_config is deterministic regardless of prior load_lang() calls.
Added two regression tests for the ko.json fix and the state leak.
The regression-guard tests added in #835 were pinned to the old
README shape (tool table + file-reference table). When #897 slimmed
the README and moved that content to the website, three tests
started failing:
TestReadmeToolsExistInCode.test_every_readme_tool_exists_in_tools_dict
TestNoUnlistedTools.test_no_undocumented_tools
TestReadmeDialectNotLossless.test_readme_dialect_line_not_lossless
Changes in this commit:
1. Update the 3 tests to track the new canonical docs surfaces
- Tool list -> website/reference/mcp-tools.md
(tests parse `### \`mempalace_xxx\`` headings instead of
markdown table rows).
- dialect.py lossless disclaimer -> website/reference/modules.md
(any line mentioning dialect.py must not also say "lossless").
2. Fix the website to make "no undocumented tools" true
Add the 10 tools that existed in TOOLS but were missing from
website/reference/mcp-tools.md (create_tunnel, delete_tunnel,
follow_tunnels, list_tunnels, get_drawer, list_drawers,
update_drawer, hook_settings, memories_filed_away, reconnect).
Page header now correctly says "all 29 MCP tools".
3. Align pre-commit ruff pin to match CI (0.4.x)
.pre-commit-config.yaml was pinning ruff v0.9.0, while
.github/workflows/ci.yml installs ruff>=0.4.0,<0.5. The two
formatters produce incompatible output (e.g. v0.9.0 reformats
`assert (x), msg` -> `assert x, (msg)` in a way v0.4.x rejects),
which would cause the pre-commit hook to modify files that CI
then flags as unformatted. Pinning the hook to v0.4.10 keeps
the dev loop and CI in lock-step.
Full suite: 887 passed, 0 failed.
Remaining in-repo surfaces carrying the same retracted or broken
claims as the public pages fixed in the previous two commits.
CONTRIBUTING.md
- "Palace structure matters ... 34% retrieval improvement" → reframed
as scoping (same rewording applied to the website equivalents).
benchmarks/BENCHMARKS.md
- Add a prominent "Important caveat" block at the top of the
"Comparison vs Published Systems" table explaining that R@5
(retrieval recall) and QA accuracy are different metrics, with
citations to Mastra, Mem0, and Supermemory's own published
methodology pages. Annotate the specific competitor rows whose
numbers are QA accuracy, not retrieval recall.
- Annotate the `hybrid v4 + rerank 100%` row to note that the 99.4
→ 100 step was tuned on 3 specific wrong answers (already disclosed
further down in the doc under "Benchmark Integrity"); the honest
hybrid figure is held-out 98.4%.
- Fix the broken clone URL — `aya-thekeeper/mempal` no longer points
at anything; now `MemPalace/mempalace`.
benchmarks/README.md + benchmarks/HYBRID_MODE.md
- Same clone-URL fix applied.
CHANGELOG.md
- Add a ### Documentation entry under [Unreleased] v3.3.0 that names
#875 and summarises the scope of the rewrite.
Part of #875. Bring the VitePress site into line with the new README
and the reproducibility scorecard: drop category-error comparisons,
drop retracted claims, retain only metrics and caveats that survive
audit.
website/index.md
- New tagline matches README (local-first, verbatim, pluggable backend,
96.6% R@5 raw, zero API calls).
- Replace the "MemPalace hybrid 100% / Supermemory ~99% / Mastra
94.87% / Mem0 ~85%" comparison table with a single honest table
showing MemPalace's own retrieval-recall numbers (raw 96.6%,
hybrid v4 held-out 98.4%). Add an explicit sentence explaining why
we no longer publish a cross-system table on the landing page
(retrieval recall vs QA accuracy are different metrics).
- Soften the "ChromaDB-powered vector search" feature blurb to be
backend-agnostic, since the retrieval layer is pluggable.
website/reference/benchmarks.md
- Full rewrite of the retrieval-recall tables. No more "100%"
headline; honest held-out 98.4% R@5 replaces it. Added the
model-agnostic rerank result (99.2% R@5 / 100% R@10 with
minimax-m2.7 via Ollama) to show the pipeline is not Haiku-specific.
- Drop the LoCoMo "Hybrid v5 + Sonnet rerank (top-50) 100%" row.
With per-conversation session counts of 19-32 and top_k=50, the
retrieval stage returns every session by construction — the number
measures an LLM's reading comprehension, not retrieval.
- Drop the cross-system comparison tables. Link out to each project's
own research page (Mastra, Mem0, Supermemory) for their published
numbers and metric definitions.
- Rewrite reproduction commands to use the correct repository and
demonstrate the new --llm-backend ollama flag.
website/concepts/the-palace.md
- Remove the "+34%" row / paragraph. Wing/room filtering is standard
metadata filtering in the vector store, not a novel retrieval
mechanism — the April-7 note already retracted that framing; this
finishes the retraction on the website where it had remained.
website/guide/searching.md
- Same treatment for "34% retrieval improvement". Reframe as
operational scoping, not a novel boost.
website/reference/contributing.md
- Update the "palace structure matters" bullet to reflect the same
framing: scoping-not-magic.
website/concepts/knowledge-graph.md
- Replace the MemPalace-vs-Zep feature matrix with a short "related
work" note that links to Zep's own documentation for authoritative
details on their deployment model. Avoids claims we cannot verify
at source.
Addresses #875. The previous README was 755 lines mixing six purposes
(scam alert, hero, two mea-culpa notes, install guide, architecture
explainer, API reference, file map). Rework it as a pure entry point:
what MemPalace is, how to install, honest benchmark numbers, links to
the website for concept/architecture documentation.
Key content changes:
- Drop the "highest-scoring AI memory system ever benchmarked" framing.
- New tagline: "Local-first AI memory. Verbatim storage, pluggable
backend, 96.6% R@5 raw on LongMemEval — zero API calls." Avoids
naming a specific vector-store implementation since the backend is
pluggable (see mempalace/backends/base.py).
- Remove the cross-system comparison table. Retrieval recall (R@5)
and end-to-end QA accuracy are different metrics and are not
comparable; placing MemPalace's R@5 next to competitor QA accuracy
under a single column header was a category error.
- The "100%" LongMemEval headline is no longer the lead. The honest
held-out figure is 98.4% R@5 on 450 unseen questions. The rerank
pipeline reaches >=99% with any capable LLM (reproduced with
Claude Haiku, Sonnet, and minimax-m2.7 via Ollama) — pipeline-level,
not model-specific.
- Benchmark reproduction commands now reference the correct repo
(MemPalace/mempalace, not the defunct aya-thekeeper/mempal branch).
New file: docs/HISTORY.md as the canonical home for post-launch
corrections, public notices, and retractions. Contains verbatim:
- 2026-04-14 note on this rewrite (links to #875)
- 2026-04-11 impostor-domain notice (moved from README header)
- 2026-04-07 "A Note from Milla & Ben" (moved from README body)
README keeps a one-line scam-alert callout that links to
docs/HISTORY.md for the full timeline.
Addresses #875: every internal BENCHMARKS.md claim reproduced
on Linux x86_64 (v3.3.0 tag, deterministic ChromaDB embeddings,
seed=42 for the LongMemEval dev/held-out split).
Scorecard — all reproduce exactly:
LongMemEval
raw R@5 96.6% (500/500) ✅
hybrid_v4 held-out 450 R@5 98.4% (442/450) ✅
hybrid_v4 + minimax rerank R@5 99.2% (496/500) *
hybrid_v4 + minimax rerank R@10 100.0% (500/500) *
LoCoMo (session, top-10)
raw 60.3% (1986q) ✅
hybrid v5 88.9% (1986q) ✅
ConvoMem all-categories (250 items) 92.9% ✅
MemBench all-categories (8500) 80.3% ✅
* The minimax-m2.7:cloud rerank run replicates the "100%" claim
with a different LLM family (no Anthropic dependency). R@10 is
a perfect reproduction; R@5 misses 4 questions that the
published Haiku run caught — consistent with BENCHMARKS.md's own
disclosure that hybrid_v4 includes three question-specific fixes
developed by inspecting misses, i.e. teaching to the test.
The committed 50/450 split is the deterministic (seed=42) split
BENCHMARKS.md references but wasn't previously in the repo.
Full result JSONLs include every question, every retrieved id,
and every score — auditable end-to-end.
The rerank pipeline was hardcoded to Anthropic's /v1/messages.
Add a backend flag so the same code path can be exercised with
any OpenAI-compatible endpoint — local Ollama, Ollama Cloud,
or any gateway that speaks /v1/chat/completions.
Enables independent verification of the "100% with Haiku rerank"
claim by running the full benchmark with a different LLM family
(e.g. minimax-m2.7:cloud) and zero Anthropic dependency.
Both longmemeval_bench.py and locomo_bench.py:
- llm_rerank*() gain backend= / base_url= kwargs
- CLI: --llm-backend {anthropic,ollama}, --llm-base-url
- API key required only when backend=anthropic (diary/palace modes still require it)
- Parse last integer in response (reasoning models emit multi-int output)
- Fallback to message.reasoning when content is empty
- Raise max_tokens to 1024 for reasoning models
mvalentsev