jason/mempalace
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Мempalace backend seam (#413)

Browse Source 
* refactor: add stage-1 backend abstraction seam

Introduce the first upstreamable storage seam for MemPalace without
bringing in the PostgreSQL spike or any benchmark artifacts.

This change adds a small backend package with:
- BaseCollection as the minimal collection contract
- ChromaBackend/ChromaCollection as the default implementation

It then routes the main runtime collection consumers through that seam:
- palace.py
- searcher.py
- layers.py
- palace_graph.py
- mcp_server.py
- miner.status()

Behavioral constraints kept for stage 1:
- ChromaDB remains the only backend and the default path
- no config/env backend selection yet
- no PostgreSQL code
- no benchmark or research files
- existing tests stay unchanged

Important compatibility details:
- read paths now call the seam with create=False so they still surface
  the existing 'no palace found' behavior instead of silently creating
  empty collections
- write paths keep create=True semantics through palace.get_collection()
- layers/searcher retain a chromadb module attribute so the existing
  mock-based tests can keep patching PersistentClient unchanged
- ChromaBackend only creates palace directories on create=True, which
  preserves mocked read-path tests that use fake read-only paths

Verification:
- python3 -m py_compile mempalace/backends/__init__.py mempalace/backends/base.py mempalace/backends/chroma.py mempalace/palace.py mempalace/searcher.py mempalace/layers.py mempalace/palace_graph.py mempalace/mcp_server.py mempalace/miner.py
- pytest -q  # 529 passed, 106 deselected

* refactor: clean up stage-1 seam compatibility shims

Tighten the stage-1 backend abstraction branch after review.

This follow-up does three small things:
- keep the chromadb compatibility hook in searcher.py and layers.py,
  but express it through the backends.chroma module so it no longer
  reads like an accidental unused import
- fix the palace_graph.py helper alias to avoid the local name collision
  flagged by ruff (imported helper vs local _get_collection wrapper)
- preserve the existing mock-based test patch points unchanged while
  keeping the new backend seam intact

Why this matters:
- the direct  form looked like a
  dead import in review, even though it was intentionally preserving the
  existing test seam ( and
  )
- palace_graph.py had a real lint issue ( redefinition) that was
  small but worth fixing before a public PR

Verification:
- /opt/homebrew/bin/ruff check mempalace/backends/__init__.py mempalace/backends/base.py mempalace/backends/chroma.py mempalace/palace.py mempalace/searcher.py mempalace/layers.py mempalace/palace_graph.py mempalace/mcp_server.py mempalace/miner.py
- pytest -q tests/test_layers.py tests/test_searcher.py
- pytest -q  # 529 passed, 106 deselected

* docs: explain backend shim imports in search paths

Add short code comments in searcher.py and layers.py explaining why the
module-level `chromadb` alias remains after the stage-1 backend seam
refactor.

The alias is intentional: it preserves the existing mock patch points used
by the current test suite (`mempalace.searcher.chromadb.PersistentClient`
and `mempalace.layers.chromadb.PersistentClient`) while the runtime logic
now flows through the backend abstraction.

This keeps the public PR easier to review because the apparent "unused
import" now has an explicit reason next to it.

Verification:
- /opt/homebrew/bin/ruff check mempalace/searcher.py mempalace/layers.py
- pytest -q tests/test_layers.py tests/test_searcher.py

* refactor: reuse a default backend instance in palace helper

Tighten the stage-1 backend seam by promoting the default Chroma backend
adapter to a module-level singleton in `mempalace/palace.py`.

This keeps the stage-1 scope unchanged — Chroma is still the only backend
wired in this branch — but avoids constructing a fresh `ChromaBackend()`
object on every `get_collection()` call. The backend is stateless today,
so this is a readability/cleanup change rather than a behavioral one.

Why this helps:
- makes `palace.get_collection()` read like a real default factory instead
  of an inline constructor call
- keeps the stage-1 branch a little cleaner before opening the public PR
- does not widen the backend surface or change any config/runtime behavior

Verification:
- python3 -m py_compile mempalace/palace.py
- pytest -q tests/test_miner.py tests/test_layers.py tests/test_searcher.py
- pytest -q  # 529 passed, 106 deselected

* fix: harden read-only seam behavior and update seam tests

Preserve the stage-1 backend abstraction while closing the real read-path
regression surfaced in PR review.

What changed:
- make ChromaBackend.get_collection(create=False) fail fast when the palace
  directory does not exist instead of letting PersistentClient create it as a
  side effect
- update miner.status() to call get_collection(..., create=False) so status
  keeps the historical 'No palace found' behavior
- remove the temporary chromadb shim aliases from layers.py and searcher.py
  now that the tests patch the seam directly
- add focused tests for the new backends package, including ChromaCollection
  delegation and ChromaBackend create=True/create=False behavior
- retarget layer/searcher tests to patch the backend seam instead of patching
  chromadb.PersistentClient inside production modules
- add a regression test that status() does not create an empty palace when the
  target path is missing

Verification:
- ruff check .
- uv run pytest -q
- uv run pytest -q tests/test_backends.py tests/test_cli.py tests/test_mcp_server.py tests/test_layers.py tests/test_searcher.py tests/test_miner.py

Notes:
- the separate benchmark/slow/stress layer was started as a soak but not used
  as the merge gate for this PR branch

* refactor: drop duplicate mcp collection cache declaration

Remove a redundant `_collection_cache = None` assignment in
`mempalace/mcp_server.py` left over after the stage-1 backend seam refactor.

This does not change behavior; it only trims review noise in the MCP server
module after the read-path hardening pass.

Verification:
- ruff check mempalace/mcp_server.py
- uv run pytest -q tests/test_mcp_server.py

---------

Co-authored-by: Sergey Kuznetsov <sergey@iterudit.com>
This commit is contained in:
Sergey Kuznetsov
2026-04-11 19:16:49 -04:00
committed by GitHub
parent 154e8a78ec
commit ae5196bc8d
13 changed files with 271 additions and 161 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_layers.py
+33 -95
View File
@@ -71,16 +71,14 @@ def test_layer0_default_path():
def _mock_chromadb_for_layer(docs, metas, monkeypatch=None):
"""Return a mock PersistentClient whose collection.get returns docs/metas."""
"""Return a mock collection whose get() returns docs/metas."""
mock_col = MagicMock()
# First batch returns data, second batch returns empty (end of pagination)
mock_col.get.side_effect = [
{"documents": docs, "metadatas": metas},
{"documents": [], "metadatas": []},
]
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
return mock_client
return mock_col
def test_layer1_no_palace():
@@ -101,11 +99,11 @@ def test_layer1_generates_essential_story():
{"room": "decisions", "source_file": "meeting.txt", "importance": 5},
{"room": "architecture", "source_file": "design.txt", "importance": 4},
]
mock_client = _mock_chromadb_for_layer(docs, metas)
mock_col = _mock_chromadb_for_layer(docs, metas)
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer1(palace_path="/fake")
@@ -118,12 +116,9 @@ def test_layer1_generates_essential_story():
def test_layer1_empty_palace():
mock_col = MagicMock()
mock_col.get.return_value = {"documents": [], "metadatas": []}
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer1(palace_path="/fake")
@@ -135,11 +130,11 @@ def test_layer1_empty_palace():
def test_layer1_with_wing_filter():
docs = ["Memory about project X"]
metas = [{"room": "general", "source_file": "x.txt", "importance": 3}]
mock_client = _mock_chromadb_for_layer(docs, metas)
mock_col = _mock_chromadb_for_layer(docs, metas)
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer1(palace_path="/fake", wing="project_x")
@@ -147,18 +142,18 @@ def test_layer1_with_wing_filter():
assert "ESSENTIAL STORY" in result
# Verify wing filter was passed
call_kwargs = mock_client.get_collection.return_value.get.call_args_list[0][1]
call_kwargs = mock_col.get.call_args_list[0][1]
assert call_kwargs.get("where") == {"wing": "project_x"}
def test_layer1_truncates_long_snippets():
docs = ["A" * 300]
metas = [{"room": "general", "source_file": "long.txt"}]
mock_client = _mock_chromadb_for_layer(docs, metas)
mock_col = _mock_chromadb_for_layer(docs, metas)
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer1(palace_path="/fake")
@@ -171,11 +166,11 @@ def test_layer1_respects_max_chars():
"""L1 stops adding entries once MAX_CHARS is reached."""
docs = [f"Memory number {i} with substantial content padding here" for i in range(30)]
metas = [{"room": "general", "source_file": f"f{i}.txt", "importance": 5} for i in range(30)]
mock_client = _mock_chromadb_for_layer(docs, metas)
mock_col = _mock_chromadb_for_layer(docs, metas)
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer1(palace_path="/fake")
@@ -193,11 +188,11 @@ def test_layer1_importance_from_various_keys():
{"room": "r", "weight": 1},
{"room": "r"}, # no weight key, defaults to 3
]
mock_client = _mock_chromadb_for_layer(docs, metas)
mock_col = _mock_chromadb_for_layer(docs, metas)
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer1(palace_path="/fake")
@@ -213,12 +208,9 @@ def test_layer1_batch_exception_breaks():
{"documents": ["doc1"], "metadatas": [{"room": "r"}]},
RuntimeError("batch error"),
]
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer1(palace_path="/fake")
@@ -244,12 +236,9 @@ def test_layer2_retrieve_with_wing():
"documents": ["Some memory about the project"],
"metadatas": [{"room": "backend", "source_file": "notes.txt"}],
}
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer2(palace_path="/fake")
@@ -265,12 +254,9 @@ def test_layer2_retrieve_with_room():
"documents": ["Backend architecture notes"],
"metadatas": [{"room": "architecture", "source_file": "arch.txt"}],
}
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer2(palace_path="/fake")
@@ -285,12 +271,9 @@ def test_layer2_retrieve_wing_and_room():
"documents": ["Filtered result"],
"metadatas": [{"room": "backend", "source_file": "x.txt"}],
}
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer2(palace_path="/fake")
@@ -304,12 +287,9 @@ def test_layer2_retrieve_wing_and_room():
def test_layer2_retrieve_empty():
mock_col = MagicMock()
mock_col.get.return_value = {"documents": [], "metadatas": []}
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer2(palace_path="/fake")
@@ -321,12 +301,9 @@ def test_layer2_retrieve_empty():
def test_layer2_retrieve_no_filter():
mock_col = MagicMock()
mock_col.get.return_value = {"documents": [], "metadatas": []}
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer2(palace_path="/fake")
@@ -340,12 +317,9 @@ def test_layer2_retrieve_no_filter():
def test_layer2_retrieve_error():
mock_col = MagicMock()
mock_col.get.side_effect = RuntimeError("db error")
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer2(palace_path="/fake")
@@ -360,12 +334,9 @@ def test_layer2_truncates_long_snippets():
"documents": ["B" * 400],
"metadatas": [{"room": "r", "source_file": "s.txt"}],
}
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer2(palace_path="/fake")
@@ -408,12 +379,9 @@ def test_layer3_search_with_results():
[{"wing": "project", "room": "backend", "source_file": "notes.txt"}],
[0.2],
)
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer3(palace_path="/fake")
@@ -427,12 +395,9 @@ def test_layer3_search_with_results():
def test_layer3_search_no_results():
mock_col = MagicMock()
mock_col.query.return_value = _mock_query_results([], [], [])
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer3(palace_path="/fake")
@@ -448,12 +413,9 @@ def test_layer3_search_with_wing_filter():
[{"wing": "proj", "room": "r"}],
[0.1],
)
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer3(palace_path="/fake")
@@ -470,12 +432,9 @@ def test_layer3_search_with_room_filter():
[{"wing": "w", "room": "backend"}],
[0.1],
)
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer3(palace_path="/fake")
@@ -492,12 +451,9 @@ def test_layer3_search_with_wing_and_room():
[{"wing": "proj", "room": "backend"}],
[0.1],
)
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer3(palace_path="/fake")
@@ -510,12 +466,9 @@ def test_layer3_search_with_wing_and_room():
def test_layer3_search_error():
mock_col = MagicMock()
mock_col.query.side_effect = RuntimeError("search failed")
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer3(palace_path="/fake")
@@ -531,12 +484,9 @@ def test_layer3_search_truncates_long_docs():
[{"wing": "w", "room": "r", "source_file": "s.txt"}],
[0.1],
)
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer3(palace_path="/fake")
@@ -552,12 +502,9 @@ def test_layer3_search_raw_returns_dicts():
[{"wing": "proj", "room": "backend", "source_file": "f.txt"}],
[0.3],
)
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer3(palace_path="/fake")
@@ -577,12 +524,9 @@ def test_layer3_search_raw_with_filters():
[{"wing": "w", "room": "r"}],
[0.1],
)
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer3(palace_path="/fake")
@@ -595,12 +539,9 @@ def test_layer3_search_raw_with_filters():
def test_layer3_search_raw_error():
mock_col = MagicMock()
mock_col.query.side_effect = RuntimeError("fail")
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
layer = Layer3(palace_path="/fake")
@@ -701,12 +642,9 @@ def test_memory_stack_status_with_palace(tmp_path):
mock_col = MagicMock()
mock_col.count.return_value = 42
mock_client = MagicMock()
mock_client.get_collection.return_value = mock_col
with (
patch("mempalace.layers.MempalaceConfig") as mock_cfg,
patch("mempalace.layers.chromadb.PersistentClient", return_value=mock_client),
patch("mempalace.layers._get_collection", return_value=mock_col),
):
mock_cfg.return_value.palace_path = "/fake"
stack = MemoryStack(