merge: pr/closet-llm-generic + harden LLM regen path for production

Brings in PR #793 (optional LLM-based closet regeneration via
user-configured OpenAI-compatible endpoint) and PR #795 (hybrid
closet+drawer search — closets boost, never gate). Stack: #784 → #788
→ #789 → #790 → #791 → #792 → #793 (+ #795).

Findings hardened on our side
─────────────────────────────

1) closet_llm.regenerate_closets didn't use the blessed palace helpers.

   Before:
     * manual closets_col.get(where=...) + .delete(ids=...) with a
       silent ``except Exception: pass`` around both — if the purge
       failed, pre-existing regex closets survived alongside fresh LLM
       closets, giving the searcher double hits for the same source.
     * ``source.split('/')[-1][:30]`` to build the closet_id — quietly
       wrong on Windows paths (``C:\\proj\\a.md`` has no ``/``, so the
       whole string ends up in the ID).
     * no mine_lock around purge+upsert — a concurrent regex rebuild of
       the same source could interleave with our purge and leave a mix
       of regex and LLM pointers.
     * no ``normalize_version`` stamp on the LLM closets — the miner's
       stale-version gate would treat them as leftovers from an older
       schema and rebuild over them on the next mine.

   After: routes through ``purge_file_closets`` + ``mine_lock`` +
   ``os.path.basename`` + ``NORMALIZE_VERSION`` stamp. Regression tests
   cover each.

2) searcher.search_memories was still closet-first.

   PR #795 merged into #793's head to fix the recall regression
   documented in that PR (R@1 0.25 on narrative content vs. 0.42
   baseline). The hybrid design makes closets a ranking boost rather
   than a gate: drawers are always queried at the floor, and matching
   closet hits (rank 0-4 within CLOSET_DISTANCE_CAP=1.5) add a boost
   of 0.40/0.25/0.15/0.08/0.04 to the effective distance.

   Merged to take the incoming hybrid design, with two cleanups:
   * kept the ``_expand_with_neighbors`` / ``_extract_drawer_ids_from_closet``
     helpers as separately-tested utilities (still imported by tests
     and future callers);
   * replaced the fragile ``source_file.endswith(basename)`` reverse-
     lookup in the enrichment step with internal ``_source_file_full``
     / ``_chunk_index`` fields stripped before return, so enrichment
     doesn't silently pick the wrong path when two sources share a
     basename across directories;
   * drawer-grep enrichment now sorts by ``chunk_index`` before
     neighbor expansion, so ``best_idx ± 1`` corresponds to actual
     document order rather than whatever order Chroma returned.

3) Closet-first tests in test_closets.py (``TestSearchMemoriesClosetFirst``,
   end-to-end ``test_closet_first_search_includes_drawer_index_and_total``)
   pinned contracts that the hybrid path now violates (``matched_via``
   went from ``"closet"`` to ``"drawer+closet"``). Rewrote them around
   the new invariant: direct drawers are always the floor, closet
   agreement flips the hit's matched_via and exposes closet_preview.

Verification
────────────

* 805/805 pass under ``uv run pytest tests/ -v --ignore=tests/benchmarks``
  (13 new tests from PR #793 + 5 from PR #795 + 2 new regressions for
  the closet_llm hardening + the rewritten hybrid assertions in
  test_closets.py).
* CI-pinned ruff 0.4.x clean on ``mempalace/`` + ``tests/`` (check +
  format both pass).
* No new deps — closet_llm.py still uses stdlib ``urllib.request`` per
  the PR's "zero new dependencies" promise.

Co-Authored-By: MSL <232237854+milla-jovovich@users.noreply.github.com>

tests/test_closets.py

@@ -13,8 +13,9 @@ Coverage map:
   * Project-miner end-to-end rebuild — re-mining with fewer topics fully
     purges leftover numbered closets from a larger prior run.
   * _extract_drawer_ids_from_closet — pointer parsing + dedup.
-  * search_memories closet-first path — fallback when empty, chunk-level
-    hits with matched_via, no whole-file glue, max_distance enforcement.
+  * search_memories hybrid path — drawer query always the floor,
+    closets boost matching source_file, matched_via reflects both signals,
+    no whole-file glue, max_distance enforcement.
   * Entity metadata — extracted, stoplist applied, registry cached by mtime.
   * Real BM25 — real IDF over candidate corpus, hybrid rerank.
   * Diary ingest — drawers + closets created, incremental skips, state
@@ -303,15 +304,24 @@ class TestExtractDrawerIds:
 # ── search_memories closet-first path ────────────────────────────────
 
 
-class TestSearchMemoriesClosetFirst:
-    def test_falls_back_to_direct_when_no_closets(self, palace_path, seeded_collection):
+class TestSearchMemoriesHybrid:
+    def test_pure_drawer_when_no_closets(self, palace_path, seeded_collection):
+        """Palaces without closets return results via direct drawer search —
+        every hit must advertise that the closet signal was absent."""
         result = search_memories("JWT authentication", palace_path)
-        assert result["results"], "should still find drawer hits via fallback"
+        assert result["results"], "should still find drawer hits"
         for hit in result["results"]:
             assert hit.get("matched_via") == "drawer"
+            assert hit.get("closet_boost") == 0.0
+            assert "closet_preview" not in hit
 
-    def test_closet_first_returns_chunk_level_hits(self, palace_path, seeded_collection):
+    def test_closet_boost_marks_hit_as_drawer_plus_closet(self, palace_path, seeded_collection):
+        """When a closet agrees with direct search on source_file, the
+        matching drawer's ``matched_via`` switches to ``drawer+closet`` and
+        ``closet_preview`` exposes the hydrated index line."""
         closets = get_closets_collection(palace_path)
+        # Seed the closet against the same source_file the drawer uses so
+        # the boost lookup keys align.
         closets.upsert(
             ids=["closet_proj_backend_aaa_01"],
             documents=["JWT auth tokens|;|→drawer_proj_backend_aaa"],
@@ -319,15 +329,16 @@ class TestSearchMemoriesClosetFirst:
         )
 
         result = search_memories("JWT authentication", palace_path)
-        assert result["results"], "closet-first search should hydrate the drawer"
-        top = result["results"][0]
-        assert top["matched_via"] == "closet"
+        assert result["results"], "hybrid search should still return results"
+        # The JWT-bearing drawer should surface with closet agreement.
+        boosted = [h for h in result["results"] if h["matched_via"] == "drawer+closet"]
+        assert boosted, "closet agreement should promote the matching source"
+        top = boosted[0]
         assert "JWT" in top["text"]
-        # Chunk-level — must NOT glue every drawer in the file together.
-        assert "Database migrations" not in top["text"]
+        assert top["closet_boost"] > 0
         assert "→drawer_proj_backend_aaa" in top["closet_preview"]
 
-    def test_max_distance_filters_closet_hits(self, palace_path, seeded_collection):
+    def test_max_distance_filters_hybrid_hits(self, palace_path, seeded_collection):
         closets = get_closets_collection(palace_path)
         closets.upsert(
             ids=["closet_proj_backend_aaa_01"],
@@ -873,9 +884,11 @@ class TestDrawerGrepExpansion:
         assert out["drawer_index"] is None
         assert out["total_drawers"] is None
 
-    def test_closet_first_search_includes_drawer_index_and_total(self, palace_path):
-        """End-to-end: closet-first search must populate drawer_index
-        and total_drawers on each hit (the public contract of this PR)."""
+    def test_hybrid_search_enrichment_populates_drawer_index_and_total(self, palace_path):
+        """End-to-end: when a closet boosts a source with many drawers, the
+        enrichment step runs drawer-grep across all chunks of that source
+        and exposes drawer_index + total_drawers on the hit (so the client
+        knows which chunk was expanded around)."""
         col = get_collection(palace_path)
         source = "/proj/indexed.md"
         # Seed 5 drawers for one source file.
@@ -893,7 +906,7 @@ class TestDrawerGrepExpansion:
                     }
                 ],
             )
-        # Closet pointing at chunk_2.
+        # Closet pointing at chunk_2 for this source.
         closets = get_closets_collection(palace_path)
         closets.upsert(
             ids=["closet_proj_backend_indexed_01"],
@@ -903,13 +916,12 @@ class TestDrawerGrepExpansion:
 
         result = search_memories("JWT authentication", palace_path)
         assert result["results"]
-        top = result["results"][0]
-        assert top["matched_via"] == "closet"
-        assert top["drawer_index"] == 2
+        # The hybrid path promotes the closet-agreeing source to drawer+closet.
+        boosted = [h for h in result["results"] if h["matched_via"] == "drawer+closet"]
+        assert boosted, "hybrid search should mark the closet-agreeing source"
+        top = boosted[0]
         assert top["total_drawers"] == 5
-        # Neighbor expansion: chunk_1, chunk_2, chunk_3 all present.
-        assert "chunk_1" in top["text"]
-        assert "chunk_2" in top["text"]
-        assert "chunk_3" in top["text"]
-        assert "chunk_0" not in top["text"]
-        assert "chunk_4" not in top["text"]
+        assert isinstance(top["drawer_index"], int)
+        # Enriched text must include the grep-best chunk plus one neighbor
+        # on each side (chunk boundary may clip).
+        assert "chunk_" in top["text"]