Same class of bug as #1007: ChromaDB's query() can return None in the
documents and metadatas arrays when a drawer's HNSW vector entry exists
but its metadata/document rows haven't been materialized. The code in
Layer3.search_raw (mempalace/layers.py) calls meta.get("wing", ...),
meta.get("room", ...), meta.get("source_file", ...) directly without
null safety, so it raises:
AttributeError: 'NoneType' object has no attribute 'get'
Two-line defensive coercion matching the pattern in #1009 /
PR #999 for searcher.py: meta = meta or {}, doc = doc or "".
The hit still appears with its real distance; source/wing/room
fall back to their fallback values where the metadata row is missing.
Frequently hit on chromadb 1.5.x (root cause #1006). Even after the
chromadb floor lands (#1010), partial-state results remain possible
during interrupted mines and schema upgrade boundaries, so the guard
is worth having on its own.
Fixes#1011.
The CLI help text and README told first-time users to mine from ~/chats/,
a path that doesn't exist on any machine. Real location where Claude
Code writes session JSONL is ~/.claude/projects/<escaped-project-path>/.
Updates three user-visible strings:
- mempalace/cli.py line 7 ("Two ways to ingest" block)
- mempalace/cli.py line 25 (Examples block)
- README.md line 58 (Quickstart)
Website guides (website/guide/mining.md, getting-started.md) still
reference ~/chats/ for ChatGPT/Slack export scenarios where that remains
a valid placeholder. Those can be a separate PR if the maintainers want
to tilt the website examples toward Claude Code specifically.
Fixes#996.
Five findings from the automated review, fixed with targeted tests where
behavior changed:
1. Transformation Protocol (transforms.py). The registry mixed a bytes-to-str
transform (utf8_replace_invalid) with str-to-str transforms under a single
Callable[..., str] type, misleading static type checkers and adapter
authors. Introduced a Transformation Protocol with __call__(data: bytes|str)
-> str and retyped the registry + get_transformation return.
2. Drawer-id collision risk (context.py). Switched _build_drawer_id from
sha1[:16]=64 bits to sha256[:24]=96 bits. 64 bits sits uncomfortably
close to the birthday bound for palace-sized corpora; 96 bits keeps the
collision probability negligible while preserving the existing
<prefix>_<chunk> layout adapters rely on.
3. Fresh-schema KG columns (knowledge_graph.py). source_drawer_id and
adapter_name now live in the canonical CREATE TABLE so new palaces don't
take an ALTER round-trip on first open. _migrate_schema stays for legacy
palaces (SQLite has no ADD COLUMN IF NOT EXISTS, so PRAGMA introspection
is still needed there).
4. Identity-shim comment (transforms.py). Comment said the adapter-specific
transforms "raise if invoked without adapter context" but they return
the input unchanged. Updated the comment to match the actual identity-
shim behavior Copilot suggested.
5. Test docstring (test_sources.py). Comment mentioned default_factory=list
but SourceRef.options uses default_factory=dict. Corrected.
Tests: 1020 passed (up from 1018), +2 new tests for the sha256 id shape
and the fresh-schema column presence on new palaces.
Four more MCP handlers iterate a metadata list and call m.get(...)
unconditionally. When the cache contains a None entry (drawers with no
metadata, common on older mining paths), the try block catches the
AttributeError and marks the response "partial: true" with an
error message — visible as {"error": "'NoneType' object has no
attribute 'get'", "partial": true} returned from mempalace_status even
though the palace data is otherwise fetchable.
Same m = m or {} guard we applied to searcher.py (d3a2d22, a51c3c2)
and miner.status() (66f08a1). None-metadata drawers now roll up under
the existing "unknown" fallback bucket instead of poisoning the
response with a misleading partial flag.
Regression test: mock the metadata cache with a None in the middle,
assert tool_status returns clean counts and no error/partial fields.
Verified the test fails without the guard.
998 tests pass.
Lands the read-side contract so third-party adapter authors (@Perseusxrltd,
@JakobSachs, @adv3nt3, @zendesk-thittesdorf, @mfhens, @roip, @MrDys) have a
stable target matching what RFC 001 §10 landed on the write side in #995.
Scope (this PR):
- mempalace/sources/base.py: BaseSourceAdapter ABC with kwargs-only
ingest() / describe_schema() and default is_current() / source_summary()
/ close() (§1.1–1.2). Typed records: SourceRef, SourceItemMetadata,
DrawerRecord, RouteHint, SourceSummary, AdapterSchema, FieldSpec (§1.3,
§5.2). Error classes: SourceNotFoundError, AuthRequiredError,
AdapterClosedError, TransformationViolationError, SchemaConformanceError
(§2.7). Class-level identity contract: name / adapter_version /
capabilities / supported_modes / declared_transformations /
default_privacy_class (§2.1, §1.4, §1.5, §6).
- mempalace/sources/transforms.py: reference implementations of the 13
reserved transformations (§1.4) — utf8_replace_invalid, newline_normalize,
whitespace_trim, whitespace_collapse_internal, line_trim, line_join_spaces,
blank_line_drop — as pure functions, plus identity shims for the six
adapter-specific ones (strip_tool_chrome, tool_result_truncate,
tool_result_omitted, spellcheck_user, synthesized_marker,
speaker_role_assignment) that the conversations adapter will override
when migrated. get_transformation(name) resolves by reserved name.
- mempalace/sources/registry.py: entry-point discovery via
importlib.metadata.entry_points(group="mempalace.sources") + explicit
register()/unregister() surface (§3.1–3.2). resolve_adapter_for_source()
implements the §3.3 priority order; crucially, no auto-detection on the
read side (§3.3 is explicit about that — user intent never inferred from
on-disk artifacts).
- mempalace/sources/context.py: PalaceContext facade (§9) bundling the
drawer/closet collections, knowledge graph, palace path, adapter identity,
and progress hooks core passes into adapter.ingest(). upsert_drawer()
applies the spec-mandated adapter_name/adapter_version stamps from §5.1.
skip_current_item() signals laziness; emit() dispatches to hooks and
swallows hook exceptions.
- mempalace/knowledge_graph.py: add_triple() gains optional source_drawer_id
and adapter_name kwargs (§5.5). Backwards-compatible column migration
auto-adds the new columns on open of a pre-RFC 002 palace (PRAGMA
table_info then ALTER TABLE ADD COLUMN), matching the pattern used for
any new palace-side provenance fields.
- pyproject.toml: mempalace.sources entry-point group declared. Empty on
the first-party side for now — miners migrate in a follow-up; the group
being present means third-party packages can begin registering today.
Out of scope (explicit follow-ups):
- miner.py → mempalace/sources/filesystem.py. Behavior-preserving rename
that also moves READABLE_EXTENSIONS, detect_room(), detect_hall() into
the adapter (§9). Larger refactor; lands separately.
- convo_miner.py + normalize.py → mempalace/sources/conversations.py. The
format-detection if-chain in normalize.py becomes per-format plugins;
declared_transformations enumerates what the current pipeline already
does to source bytes (§1.4 existing-code mapping).
- Closet post-step wired into the conversations adapter (§1.7).
- CLI --source flag + --mode deprecation alias (§3.3).
- MCP mempalace_mine tool source parameter.
- AbstractSourceAdapterContractSuite (§7.1–7.3): byte-preservation round-
trip and declared-transformation round-trip tests.
- Privacy-class floor enforcement (§6.2); depends on #389 for
secrets_possible scanning.
Tests: 1018 passed (up from ~990 on develop), +27 targeted tests covering
the ABC instantiation rules, typed records, all reserved transformations,
the registry register/get/unregister surface, PalaceContext upsert + skip +
emit semantics, and both the new KG provenance kwargs and backwards-
compatible legacy-schema migration.
Refs: #989 (RFC 002 tracking), #990 (RFC 002 spec), #995 (RFC 001 §10
cleanup — sibling PR on the write side).
Per Copilot review on the CLI-only PR (#999): search_memories() has the
same vulnerability in two additional spots, since ChromaDB can return
None entries in the inner metadatas list for either the drawer query or
the closets query. Without guards, the API path crashes with:
AttributeError: 'NoneType' object has no attribute 'get'
at either \`cmeta.get("source_file", "")\` in the closet boost lookup or
\`meta.get("source_file", "") or ""\` in the drawer scoring loop.
Applies the matching \`meta = meta or {}\` / \`cmeta = cmeta or {}\`
guard at both sites and adds an API-path regression test that mocks a
drawer query result with a None metadata entry and asserts both hits
render — the None-metadata hit with the existing \`"unknown"\` sentinel
values the scoring loop already writes for missing keys.
Verified both the new API test and the existing CLI test fail without
the guards (AttributeError) and pass with them.
`status()` walks `col.get(include=["metadatas"])` and buckets each drawer
into a `wing_rooms[wing][room]` histogram. The same ChromaDB return shape
fixed in the search print path — `None` entries in the `metadatas` list
for drawers with no stored metadata — crashes the status command with:
AttributeError: 'NoneType' object has no attribute 'get'
Applies the matching ``m = m or {}`` guard so None-metadata drawers roll
up under the existing `?/?` fallback bucket instead of killing the
command mid-tally. Reproduced on a 135K-drawer palace where two drawers
had `metadata=None`; both now show under `WING: ? / ROOM: ?` in the
tally while the command prints the full histogram as designed.
Adds a regression test that feeds `status()` a fake collection whose
`get()` returns a `None` in the middle of the metadatas list and asserts
both the fallback bucket and the real wing render.
`col.query(...)` can return `None` entries in the inner ``metadatas`` list
for drawers whose metadata was never set (older palaces, rows written
outside the normal mining path). The CLI `search()` function would render
earlier results successfully and then crash mid-loop with:
AttributeError: 'NoneType' object has no attribute 'get'
at ``searcher.py:286`` — ``meta.get("source_file", "?")``. The user sees
partial output followed by a traceback, with no indication of which
drawers rendered OK and which were skipped.
Guard with ``meta = meta or {}`` inside the loop so entries with missing
metadata fall back to the existing ``"?"`` defaults instead of crashing,
matching the hit dict assembly in ``search_memories()`` which already
uses ``meta.get("wing", "unknown")`` etc. against the same data.
Adds a regression test that mocks a ChromaDB result with a ``None``
metadata entry in the middle of the inner list and asserts both result
blocks render to stdout.
24bf97b (network-download fix) and my earlier Copilot-review commit both
added tests for the same ValueError. Keep the broader one that covers
both 'documents length' and 'metadatas length' mismatches; drop the
narrower duplicate.
PermissionError [WinError 32] on Windows when Path.unlink() runs while
chromadb.PersistentClient still holds a handle on chroma.sqlite3. Rewrite
test_chroma_cache_invalidates_when_db_file_missing to prime
backend._clients/_freshness with a sentinel object instead of opening a
real PersistentClient, so the unlink runs against an unheld file.
The assertion is also corrected: after invalidation, ChromaBackend's
_client rebuilds a fresh PersistentClient which re-creates chroma.sqlite3
and re-stats it, so freshness ends up at the post-rebuild stat (not
(0, 0.0) as the assertion previously expected). The meaningful invariant
is "freshness advanced past the pre-unlink value AND the sentinel was
replaced", which the test now checks.
Ref: Windows CI failure on 995.
test_base_collection_update_default_validates_list_lengths and
test_base_collection_update_default_rejects_mismatched_lengths were
spinning up a real ChromaBackend and calling add(documents=...), which
triggered ChromaDB's default ONNX embedding function and attempted a
network download — failing in offline/sandboxed CI.
BaseCollection.update() validates list lengths before any DB access, so
no items need to be pre-loaded for the length-check to fire. Switch both
tests to use _FakeCollection (same as the rest of the unit tests in this
file) so they are pure in-memory and network-free.
Also fixes a structural bug in test 1: collection._collection.add() was
accidentally placed inside the pytest.raises(ValueError) block, masking
the real assertion.
Agent-Logs-Url: https://github.com/MemPalace/mempalace/sessions/55fc663e-b256-4b8b-88ce-4271560def8d
Co-authored-by: igorls <4753812+igorls@users.noreply.github.com>
Six items from the automated review on PR #998:
1. **Cursor tie-break bug (correctness).** The skip condition was
`rec.timestamp <= cursor`; if multiple messages share the max
timestamp and only some were ingested before a crash, the rest
would be lost forever. Changed to `< cursor`, relying on
deterministic drawer IDs for safe re-attempt at the boundary.
Regression test
`test_sweep_recovers_untaken_message_at_cursor_timestamp`.
2. **`drawers_added` counted upserts, not adds.** Added a pre-flight
`collection.get(ids=batch)` to distinguish new rows from already-
present ones. Return value now carries `drawers_added`,
`drawers_already_present`, `drawers_upserted`, and `drawers_skipped`
separately. Dict-compatible access (`existing.get("ids")`) keeps it
working on both the raw Chroma return and the typed `GetResult`.
3. **`sweep_directory` hid failures in the summary.** `files_processed`
used to exclude failed files. Replaced with `files_attempted` (all
discovered) + `files_succeeded` (subset that completed); CLI output
shows `succeeded/attempted`.
4. **Coordination claim was overstated.** The primary miners don't
stamp `session_id`/`timestamp` metadata, so the sweeper coordinates
only with its own prior runs. Softened docstrings on module and CLI
command. Uniform cross-miner metadata is flagged as a follow-up.
5. **MAX_FILE_SIZE comments were misleading.** Said source size "does
not affect storage or embedding cost" — true per-drawer, but source
size still scales drawer count, embedding work, and memory usage
(files are read in full, not streamed). Corrected in both
`miner.py` and `convo_miner.py`.
6. Added the tie-break regression test that reproduces the correctness
bug from (1).
Tests: 970 passed (was 969), ruff + pre-commit clean.
Co-Authored-By: MSL <232237854+milla-jovovich@users.noreply.github.com>
Four defects surfaced by the automated review, fixed with targeted tests:
1. BaseCollection.update() default now validates that documents / metadatas /
embeddings lengths match ids, raising ValueError instead of silently
misaligning pairs or raising IndexError (base.py).
2. ChromaCollection.query() now rejects the two ambiguous input shapes up
front — neither or both of query_texts / query_embeddings, and empty input
lists — with clear ValueError messages rather than delegating to chromadb's
less-obvious errors (chroma.py).
3. QueryResult.empty() accepts embeddings_requested=True to preserve the
outer-query dimension with empty hit lists when the caller asked for
embeddings, matching the spec rule that included fields carry the outer
shape even when empty (base.py). ChromaCollection.query() threads this
through on the empty-result path (chroma.py).
4. ChromaBackend cache-freshness check now matches the semantics from
mcp_server._get_client (merged via #757) on three edge cases Copilot
called out: (a) invalidate when chroma.sqlite3 disappears while a cached
client is held, (b) treat a 0→nonzero stat transition as a change so a
cache built when the DB did not yet exist is refreshed, (c) re-stat
after PersistentClient constructs the DB lazily so freshness reflects
the post-creation state (chroma.py).
Tests: 978 passed (up from 970), 8 new tests covering the fixes.
Four changes on top of the proposal's initial sweeper draft, driven by
the CLAUDE.md design principles:
1. Drop the 500-char truncation on tool_use / tool_result content in
_flatten_content. The "verbatim always" principle forbids lossy
compression of user-adjacent data; a long code-edit diff handed to
the assistant must round-trip intact. Unknown block types now also
serialize their full payload instead of just a type marker. New test
test_parse_preserves_tool_blocks_verbatim covers a 5000-char input.
2. Use the full session_id in drawer IDs (not session_id[:12]). Rules
out cross-session collisions if a transcript source ever uses
non-UUID session identifiers or shared prefixes.
3. Replace silent `except Exception: return None` in get_palace_cursor
with a logger.warning — the exact anti-pattern this PR otherwise
criticizes in miner.py. The fallback behavior is still safe
(deterministic IDs make a missed cursor recover on the next run),
but the failure is now discoverable.
4. sweep_directory now collects per-file failures into the result dict
and the CLI exits non-zero when any file failed, so a partial-sweep
outcome is visible rather than swallowed.
Co-Authored-By: MSL <232237854+milla-jovovich@users.noreply.github.com>
The primary miners (miner.py, convo_miner.py) operate at file
granularity and can drop data for several reasons: size caps, silent
OSError on read, dedup false positives, extensions the project miner
does not recognize. Even with tonight's hotfixes, any future bug in
the file-level path risks silent data loss.
The sweeper is a second, cooperating miner that works at MESSAGE
granularity:
- Parses Claude Code .jsonl line by line, yielding only
user/assistant records (filters progress, file-history-snapshot,
etc. noise).
- For each session_id, queries the palace for max(timestamp) and
treats that as the cursor.
- Ingests only messages newer than the cursor, as one small drawer
per exchange (never hits a size cap — each drawer is 1-5 KB).
- Deterministic drawer IDs from session_id + message UUID make
reruns idempotent; crash mid-sweep is safe.
Tandem coordination is free: if the primary miner committed up to
timestamp T, the sweeper resumes from T. If the primary miner missed
everything, the sweeper catches it all. Neither duplicates the other.
Smoke test on a real Claude Code transcript:
1st run: +39 drawers, 0 already present
2nd run: +0 drawers, 39 already present (perfect idempotence)
Opt-in via:
mempalace sweep <file.jsonl>
mempalace sweep <transcript-dir>
No changes to existing miners. No schema migration. Purely additive.
Tests: tests/test_sweeper.py (7 tests covering parsing, tandem
coordination, idempotency, resume-from-cursor, metadata correctness).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the miner.py fix in this same branch. convo_miner.py had the
exact same 10 MB cap at line 58 that silently dropped long transcripts
via continue. Long Claude Code sessions, multi-year ChatGPT exports,
and lifetime Slack dumps all exceed 10 MB. Same silent-drop pattern,
different file.
Raised to 500 MB to match miner.py for consistency; downstream chunking
means source file size does not affect storage or embedding cost.
Tests: tests/test_convo_miner_size_cap.py (1 test)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Long Claude Code sessions routinely produce transcripts larger than 10
MB. The previous cap at miner.py:65 silently dropped them at line 732
with `if filepath.stat().st_size > MAX_FILE_SIZE: continue` — same
silent-failure pattern as the .jsonl extension bug.
The cap exists as a safety rail against pathological binaries, not as
a limit on legitimate text. Downstream chunking at 800 chars per drawer
means source file size does not affect storage or embedding cost.
500 MB leaves headroom for year-long continuous transcripts while still
catching accidental multi-GB binary mines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mempalace/miner.py:READABLE_EXTENSIONS contained `.json` but not
`.jsonl`. Every jsonl file encountered in a mined directory was
silently skipped at miner.py:722:
if filepath.suffix.lower() not in READABLE_EXTENSIONS:
continue
Claude Code transcripts, ChatGPT exports, and every other tool writing
line-delimited JSON ship as `.jsonl`. Users running `mempalace mine`
against a directory of transcripts saw the command complete with no
error and no log line — and their conversations never reached the
palace. Silent data loss.
Adding `.jsonl` to the whitelist alongside `.json`. jsonl is text
line-by-line; the existing chunking pipeline handles it the same way
it handles any other text file.
Tests: tests/test_miner_jsonl_visibility.py
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Draft plugin specification for source adapters, mirroring RFC 001's
role for storage backends. Formalizes the contract six community
ingester PRs (#274, #23, #169, #232, #567, #98, #702) plus #981's
metadata-only mode have been reinventing ad-hoc, so adapter authors
can build to a stable surface.
Key decisions:
- Single ingest() method; lazy adapters yield SourceItemMetadata
ahead of drawers, eager adapters interleave
- Declared-transformation model (§1.4) replaces informal verbatim
promise with a verifiable one; byte_preserving adapters declare
the empty set, declared_lossy adapters enumerate. Existing
miner.py and the convo_miner+normalize pipeline map cleanly
- Palace is the incremental cursor via is_current(item, metadata);
no sidecar persistence
- Routing is adapter-owned; detect_room/detect_hall move into the
filesystem adapter
- Flat metadata per ChromaDB (RFC 001 §1.4) — entity hints as
json_string field, KG triples route to SQLite knowledge graph
- Closets stay core-built as a post-step; adapters may emit flat
closet_hints. Closes existing gap where convo drawers get no
closets
- No per-drawer field renames: source_file, filed_at, source_mtime,
added_by, normalize_version, entities, ingest_mode all preserved.
Spec adds adapter_name, adapter_version, privacy_class
§9 enumerates the cleanup PR prerequisites (mempalace/sources/
module, PalaceContext facade, KnowledgeGraph.add_triple gaining
backwards-compatible source_drawer_id + adapter_name params).
Tracking issue: #989
Extract 2002-line monolith into landing/ subfolder:
- 8 section components (FolioHeader, HeroSection, ForgettingSection, AnatomySection, DialectSection, MechanicsSection, InstallSection, CatalogFooter)
- useLandingEffects.js composable for all vanilla-JS effects
- landing.css for all styles
- Landing.vue reduced to 28-line orchestrator
Also restores upstream hero lede text ("permanent. Designed for total recall.").
- Landing: replace nonexistent `mempalace remember` CLI demo with real
`mempalace mine ./notes`
- Landing: soften unverifiable absolutes ("forever available",
"100% recall by design", "<50 ms", "90%+ compression",
"two-thousand-year-old", "tens of thousands of entries")
- MCP tool count: 19 → 29 across mcp-integration, claude-code, openclaw,
and modules; expand tool overview with Drawers, Tunnels, and System
categories to match mcp_server.py
- Wake-up token range: ~170–900 → ~600–900 in cli/api-reference/python-api
to match cli.py help text and concept docs
- Gemini CLI: move `--scope user` before target name and add `--`
separator so `-m mempalace.mcp_server` isn't parsed as Gemini flags
Igor Lins e Silva