jason/mempalace
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
688 Commits 2 Branches 0 Tags
e9222b4c7b98bc25942008942a96c5c9e2784795
Commit Graph

7 Commits

Author SHA1 Message Date
bensig 49e9e04a12 fix: guard Layer3.search_raw against None doc/meta from ChromaDB (#1011) 
Same class of bug as #1007: ChromaDB's query() can return None in the
documents and metadatas arrays when a drawer's HNSW vector entry exists
but its metadata/document rows haven't been materialized. The code in
Layer3.search_raw (mempalace/layers.py) calls meta.get("wing", ...),
meta.get("room", ...), meta.get("source_file", ...) directly without
null safety, so it raises:

  AttributeError: 'NoneType' object has no attribute 'get'

Two-line defensive coercion matching the pattern in #1009 /
PR #999 for searcher.py: meta = meta or {}, doc = doc or "".
The hit still appears with its real distance; source/wing/room
fall back to their fallback values where the metadata row is missing.

Frequently hit on chromadb 1.5.x (root cause #1006). Even after the
chromadb floor lands (#1010), partial-state results remain possible
during interrupted mines and schema upgrade boundaries, so the guard
is worth having on its own.

Fixes #1011.
 2026-04-18 13:30:57 -07:00
Arnold Wender 6a73eb2e20 fix(searcher): guard against empty ChromaDB query results (#195) (#865) 
Fixes #195.

When ChromaDB returns no documents (empty palace, or wing/room filter
that excludes everything), it returns the shape:

    {"documents": [], "metadatas": [], "distances": []}

Indexing `results["documents"][0]` blindly raises IndexError instead of
the expected 'no results' response. Affected: searcher.search(),
searcher.search_memories() (drawer + closet branches plus the
total_before_filter aggregate), and Layer3.search() / Layer3.search_raw().

Adds a tiny private helper `searcher._first_or_empty(results, key)` that
safely extracts the inner list, returning [] for any of: missing key,
empty outer list, [None], or [[]]. layers.py imports the same helper to
avoid duplicating the guard.

Tests: tests/test_empty_chromadb_results.py covers all observed shapes
plus a documentation-style test that pins the original IndexError so
future readers understand why the helper exists.
 2026-04-15 00:26:38 -07:00
Ben Sigman 4621f85d7c style: ruff format all Python files (#675) 2026-04-11 22:59:34 -07:00
Ben Sigman 20c8f8e57b feat: new MCP tools — get/list/update drawer, hook settings, export (resolves #635) (#667) 
* feat: MCP reliability — inode detection, WAL rotation, metadata cache, search limits

Infrastructure hardening for the MCP server:
- Detect palace DB replacement via inode tracking (repair command support)
- WAL rotation to prevent unbounded WAL growth
- _fetch_all_metadata() + _get_cached_metadata() with 60s TTL for taxonomy/status
- _MAX_RESULTS cap (100) with limit clamping [1, _MAX_RESULTS]
- max_distance parameter for similarity threshold in search
- Handle all notifications/* methods, null arguments, method=None
- Remove duplicate _client_cache = None declarations
- searcher.py max_distance parameter passthrough

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: new MCP tools (get/list/update drawer, hook settings, memories filed), export, normalize

New MCP tools:
- mempalace_get_drawer: fetch single drawer by ID with full content
- mempalace_list_drawers: paginated listing with wing/room filter
- mempalace_update_drawer: update content/wing/room on existing drawers
- mempalace_hook_settings: get/set hook behavior (silent_save, desktop_toast)
- mempalace_memories_filed_away: check latest checkpoint status

Also includes:
- exporter.py: export palace as browsable markdown files
- normalize.py: tool_use/tool_result capture for richer transcript mining
- layers.py: updated for new tool integration
- config.py: hook settings properties (hook_silent_save, hook_desktop_toast)

Depends on PR 3 (reliability) for _MAX_RESULTS, _metadata_cache, WAL logging.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: normalize.py handles string messages and Read offset type mismatch

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: params null guard, L2→cosine docs, empty tool_use_map key guard

- Handle explicit null in MCP params (request.get("params") or {})
- Fix search tool description: L2 → cosine distance (collection uses hnsw:space=cosine)
- Guard against empty string key in tool_use_map from malformed JSONL entries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: rename ambiguous var 'l' to 'line' (E741 lint)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address code review findings (5 issues)

1. min_similarity backwards-compat: convert similarity to distance scale
   (1.0 - similarity) instead of passing raw value as max_distance
2. Restore structured error reporting (error + partial fields) in
   tool_status, tool_list_wings, tool_list_rooms, tool_get_taxonomy
   — reverts silent except:pass that dropped #647 security hardening
3. inode cache: remove falsy-zero short-circuit so missing DB file
   triggers reconnect instead of reusing stale client
4. _fetch_all_metadata: check for empty batch before extending/advancing
   offset to prevent infinite loop on concurrent deletion
5. KG initialization: only override path when --palace is explicit;
   default runs use KnowledgeGraph's built-in default path

Co-authored-by: jphein <jphein@users.noreply.github.com>

---------

Co-authored-by: jp <jp@jphein.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: jphein <jphein@users.noreply.github.com>
 2026-04-11 21:25:04 -07:00
Sergey Kuznetsov ae5196bc8d Мempalace backend seam (#413) 
* refactor: add stage-1 backend abstraction seam

Introduce the first upstreamable storage seam for MemPalace without
bringing in the PostgreSQL spike or any benchmark artifacts.

This change adds a small backend package with:
- BaseCollection as the minimal collection contract
- ChromaBackend/ChromaCollection as the default implementation

It then routes the main runtime collection consumers through that seam:
- palace.py
- searcher.py
- layers.py
- palace_graph.py
- mcp_server.py
- miner.status()

Behavioral constraints kept for stage 1:
- ChromaDB remains the only backend and the default path
- no config/env backend selection yet
- no PostgreSQL code
- no benchmark or research files
- existing tests stay unchanged

Important compatibility details:
- read paths now call the seam with create=False so they still surface
  the existing 'no palace found' behavior instead of silently creating
  empty collections
- write paths keep create=True semantics through palace.get_collection()
- layers/searcher retain a chromadb module attribute so the existing
  mock-based tests can keep patching PersistentClient unchanged
- ChromaBackend only creates palace directories on create=True, which
  preserves mocked read-path tests that use fake read-only paths

Verification:
- python3 -m py_compile mempalace/backends/__init__.py mempalace/backends/base.py mempalace/backends/chroma.py mempalace/palace.py mempalace/searcher.py mempalace/layers.py mempalace/palace_graph.py mempalace/mcp_server.py mempalace/miner.py
- pytest -q  # 529 passed, 106 deselected

* refactor: clean up stage-1 seam compatibility shims

Tighten the stage-1 backend abstraction branch after review.

This follow-up does three small things:
- keep the chromadb compatibility hook in searcher.py and layers.py,
  but express it through the backends.chroma module so it no longer
  reads like an accidental unused import
- fix the palace_graph.py helper alias to avoid the local name collision
  flagged by ruff (imported helper vs local _get_collection wrapper)
- preserve the existing mock-based test patch points unchanged while
  keeping the new backend seam intact

Why this matters:
- the direct  form looked like a
  dead import in review, even though it was intentionally preserving the
  existing test seam ( and
  )
- palace_graph.py had a real lint issue ( redefinition) that was
  small but worth fixing before a public PR

Verification:
- /opt/homebrew/bin/ruff check mempalace/backends/__init__.py mempalace/backends/base.py mempalace/backends/chroma.py mempalace/palace.py mempalace/searcher.py mempalace/layers.py mempalace/palace_graph.py mempalace/mcp_server.py mempalace/miner.py
- pytest -q tests/test_layers.py tests/test_searcher.py
- pytest -q  # 529 passed, 106 deselected

* docs: explain backend shim imports in search paths

Add short code comments in searcher.py and layers.py explaining why the
module-level `chromadb` alias remains after the stage-1 backend seam
refactor.

The alias is intentional: it preserves the existing mock patch points used
by the current test suite (`mempalace.searcher.chromadb.PersistentClient`
and `mempalace.layers.chromadb.PersistentClient`) while the runtime logic
now flows through the backend abstraction.

This keeps the public PR easier to review because the apparent "unused
import" now has an explicit reason next to it.

Verification:
- /opt/homebrew/bin/ruff check mempalace/searcher.py mempalace/layers.py
- pytest -q tests/test_layers.py tests/test_searcher.py

* refactor: reuse a default backend instance in palace helper

Tighten the stage-1 backend seam by promoting the default Chroma backend
adapter to a module-level singleton in `mempalace/palace.py`.

This keeps the stage-1 scope unchanged — Chroma is still the only backend
wired in this branch — but avoids constructing a fresh `ChromaBackend()`
object on every `get_collection()` call. The backend is stateless today,
so this is a readability/cleanup change rather than a behavioral one.

Why this helps:
- makes `palace.get_collection()` read like a real default factory instead
  of an inline constructor call
- keeps the stage-1 branch a little cleaner before opening the public PR
- does not widen the backend surface or change any config/runtime behavior

Verification:
- python3 -m py_compile mempalace/palace.py
- pytest -q tests/test_miner.py tests/test_layers.py tests/test_searcher.py
- pytest -q  # 529 passed, 106 deselected

* fix: harden read-only seam behavior and update seam tests

Preserve the stage-1 backend abstraction while closing the real read-path
regression surfaced in PR review.

What changed:
- make ChromaBackend.get_collection(create=False) fail fast when the palace
  directory does not exist instead of letting PersistentClient create it as a
  side effect
- update miner.status() to call get_collection(..., create=False) so status
  keeps the historical 'No palace found' behavior
- remove the temporary chromadb shim aliases from layers.py and searcher.py
  now that the tests patch the seam directly
- add focused tests for the new backends package, including ChromaCollection
  delegation and ChromaBackend create=True/create=False behavior
- retarget layer/searcher tests to patch the backend seam instead of patching
  chromadb.PersistentClient inside production modules
- add a regression test that status() does not create an empty palace when the
  target path is missing

Verification:
- ruff check .
- uv run pytest -q
- uv run pytest -q tests/test_backends.py tests/test_cli.py tests/test_mcp_server.py tests/test_layers.py tests/test_searcher.py tests/test_miner.py

Notes:
- the separate benchmark/slow/stress layer was started as a soak but not used
  as the merge gate for this PR branch

* refactor: drop duplicate mcp collection cache declaration

Remove a redundant `_collection_cache = None` assignment in
`mempalace/mcp_server.py` left over after the stage-1 backend seam refactor.

This does not change behavior; it only trims review noise in the MCP server
module after the read-path hardening pass.

Verification:
- ruff check mempalace/mcp_server.py
- uv run pytest -q tests/test_mcp_server.py

---------

Co-authored-by: Sergey Kuznetsov <sergey@iterudit.com>
 2026-04-11 16:16:49 -07:00
Maurice Wen 0e77981dec fix: batch ChromaDB reads to avoid SQLite variable limit 
col.get() without limit generates SELECT ... WHERE id IN (...) with all
document IDs, which exceeds SQLite's ~999 variable limit when a palace
has more than ~1000 drawers.

This breaks both `mempalace compress` and `mempalace wake-up` on large
palaces. Reproduced on a 13880-file codebase (242K+ drawers).

Fix: paginate reads in batches of 500 using ChromaDB's offset/limit
parameters in both Layer1.generate() and cmd_compress().
 2026-04-07 21:40:12 +08:00
Milla Jovovich 068dbd9a7b MemPalace: palace architecture, AAAK compression, knowledge graph 
The memory system:
- Palace structure: Wings (people/projects) → Rooms (topics) → Closets (AAAK compressed) → Drawers (verbatim transcripts)
- Halls connect related rooms within a wing
- Tunnels cross-reference rooms across wings
- AAAK: 30x lossless compression dialect for AI agents
- Knowledge graph: temporal entity-relationship triples (SQLite)
- Palace graph: room-based navigation with tunnel detection
- MCP server: 19 tools — search, graph traversal, agent diary, AAAK auto-teach
- Onboarding: guided setup generates wing config + AAAK entity registry
- Contradiction detection: catches wrong pronouns, names, ages
- Auto-save hooks for Claude Code

96.6% Recall@5 on LongMemEval — highest zero-API score published.
100% with optional Haiku rerank (500/500).
Local. Free. No API key required.
 2026-04-04 18:16:04 -07:00