Commit Graph

263 Commits

Author SHA1 Message Date
bensig 58b8d5b198 fix: release ChromaDB handles before rmtree on Windows 2026-04-09 09:31:55 -07:00
bensig 1c48f4d2c3 fix: use os.utime in mtime test for Windows compatibility 2026-04-09 09:23:08 -07:00
Ben Sigman 252e440df5 Merge pull request #324 from virgil-at-biocompute/fix/mcp-protocol-version-negotiation
fix: negotiate MCP protocol version instead of hardcoding
2026-04-09 09:17:38 -07:00
Ben Sigman e293e290d5 Merge branch 'main' into fix/mcp-protocol-version-negotiation 2026-04-09 09:15:06 -07:00
Ben Sigman 39855df3fb Merge pull request #387 from milla-jovovich/ben/security-hardening
security: harden inputs, fix shell injection, optimize DB access
2026-04-09 09:13:09 -07:00
bensig 2448ac0026 test: add coverage for file_already_mined mtime check
Covers the check_mtime=True path in palace.py to meet 85% coverage threshold.
2026-04-09 08:56:28 -07:00
bensig c2308a1e36 fix: address code review — restore mtime check, bound metadata reads, harden security
Review fixes (from Sage's review):
- Restore mtime check in file_already_mined (check_mtime=True for miner)
- Restore limit=10000 on MCP metadata fetches to prevent OOM on large palaces
- Apply _SAFE_NAME_RE regex in sanitize_name (was dead code)
- Drop raw_aaak metadata duplication in diary_write
- chmod 0o700 on WAL dir, 0o600 on WAL file
- Add check_same_thread=False on KnowledgeGraph SQLite connection
- Remove __del__ (unreliable) and dead PRAGMA foreign_keys=ON
2026-04-09 08:52:24 -07:00
bensig 0717caea5c fix: make drawer_id deterministic for idempotent writes
Remove datetime.now() from drawer_id hash so same content + wing + room
always produces the same ID. This enables the idempotency check that
returns "already_exists" on duplicate writes.
2026-04-09 08:26:47 -07:00
bensig 32297fdae8 fix: remove metadata cache that broke test isolation
The 30s TTL metadata cache returned stale data between test runs and
after write operations. Reverted to direct col.get() reads which match
the original behavior and pass all tests.
2026-04-09 08:22:17 -07:00
bensig 455871a0ef fix: align cache variable names with test fixtures, restore full SKIP_DIRS
- _client → _client_cache to match conftest.py reset fixture
- _get_collection now uses _get_client() return value instead of stale ref
- Restore .pytest_cache and other dirs missing from palace.py SKIP_DIRS
2026-04-09 08:13:32 -07:00
Ben Sigman 725fa2b6f1 Merge branch 'main' into fix/query-sanitizer-prompt-contamination 2026-04-09 08:11:39 -07:00
Ben Sigman 70f2160bd6 Merge branch 'main' into fix/mcp-protocol-version-negotiation 2026-04-09 08:09:57 -07:00
Ben Sigman 0126f750d7 Merge branch 'main' into fix/issue-339-338-silent-exceptions-pagination 2026-04-09 08:09:38 -07:00
bensig 1d19dfc9d5 security: harden inputs, fix shell injection, optimize DB access
- Fix command injection in hook script (pass paths via sys.argv)
- Add sanitize_name/sanitize_content validators in config.py
- Add 10MB file size guard + symlink skip in miners
- Fix SQLite connection leak in knowledge_graph.py (reuse connection)
- Use `with conn:` for proper transaction handling
- Consolidate shared palace operations into palace.py
- Add write-ahead log for audit trail on writes/deletes
- Add metadata cache with 30s TTL for status/taxonomy calls
- Upgrade md5 → sha256 for drawer/triple IDs
- Harden file permissions (0o700/0o600)
- Pin chromadb>=0.5.0,<0.7

Based on PR #252 by @anthonyonazure with lint fixes applied.

Co-Authored-By: anthonyonazure <anthonyonazure@users.noreply.github.com>
2026-04-09 08:06:30 -07:00
Ben Sigman 963c04cf45 Merge pull request #281 from tmuskal/main
Increase test coverage from 30% to 85% and fix Windows encoding bugs
2026-04-09 07:53:32 -07:00
matrix9neonebuchadnezzar2199-sketch 7509a72502 fix: mitigate system prompt contamination in search queries (#333)
Addresses Issue #333: AI agents prepending system prompts to search queries
causes embedding retrieval to collapse (89.8% → 1.0% R@10).

Mitigation approach (減災):
- New query_sanitizer.py with 4-stage pipeline:
  Step 1: passthrough for short queries (≤200 chars)
  Step 2: question extraction (finds ? sentences) → ~85-89% recovery
  Step 3: tail sentence extraction → ~80-89% recovery
  Step 4: tail truncation fallback → ~70-80% recovery
  Worst case without sanitizer: 1.0% (catastrophic)
  Worst case with sanitizer: ~70-80% (survivable)

- mcp_server.py: tool_search applies sanitizer before ChromaDB query
- MCP schema: query description warns agents not to include prompts
- New 'context' parameter separates background info from search intent
- Sanitizer metadata included in response when triggered

22 new tests covering all pipeline stages and real-world scenarios.

Made-with: Cursor
2026-04-09 23:28:59 +09:00
Luna Mira e5440e31af fix: count Codex user_message turns in _count_human_messages (#347)
The _count_human_messages() function previously only handled Claude Code
transcript format: {"message": {"role": "user", "content": "..."}}

Codex CLI transcripts use a different schema:
{"type": "event_msg", "payload": {"type": "user_message", "message": "..."}}

This meant the stop-hook auto-save threshold never triggered for Codex
sessions because the count always returned 0.

Added detection for the Codex format so both Claude Code and Codex CLI
transcripts are counted correctly.
2026-04-09 13:33:45 +01:00
Openclaw d20c8ab992 fix: paginate large collection reads and surface errors in MCP tools (#339, #338) 2026-04-09 13:33:45 +01:00
Tal Muskal da64016a94 fix: format test_layers_bench.py with ruff to pass CI lint
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-09 08:24:51 +03:00
Tal Muskal 667d895fb9 Merge branch 'main' into main 2026-04-09 08:17:39 +03:00
virgil-at-biocompute 950d52baf2 fix: negotiate MCP protocol version instead of hardcoding
The initialize handler hardcoded protocolVersion "2024-11-05", which
causes newer MCP clients (e.g. Claude Code) to reject the connection
when they negotiate "2025-11-25" or later.

Echo the client's requested version if it is in the supported set,
otherwise fall back to the latest supported version. This keeps
backwards compatibility with older clients while allowing newer ones
to connect.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 22:57:32 -04:00
github-actions[bot] 26835e30ef chore: bump version to 3.0.14 2026-04-08 23:54:30 +00:00
Ben Sigman a5d9baf589 Merge pull request #165 from adv3nt3/fix/miner-room-stats
fix: use actual detected room in mine summary stats
2026-04-08 16:54:22 -07:00
Ben Sigman d26606b2f9 Merge branch 'main' into main 2026-04-08 14:07:33 -07:00
github-actions[bot] b370e86f96 chore: bump version to 3.0.13 2026-04-08 20:56:15 +00:00
Ben Sigman 9b705d651f Merge pull request #223 from igorls/bench/scale-test-suite
bench: add scale benchmark suite (106 tests)
2026-04-08 13:56:06 -07:00
Igor Lins e Silva c4e52954fe Merge upstream/main into bench/scale-test-suite to resolve conflicts
Merged both the PR's benchmark suite additions (psutil dep, pytest
markers, --ignore=tests/benchmarks) and upstream's coverage changes
(pytest-cov, --cov-fail-under=30, coverage config) so both coexist.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
2026-04-08 16:28:06 -03:00
Tal Muskal 28de031f25 fix: remove stale palace_path reference in test helper
_patch_mcp_server had palace_path removed from its signature but the
assertion body still referenced it, causing NameError at runtime and
F821 from ruff.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 22:07:46 +03:00
Tal Muskal dbf456b73b Merge branch 'main' into main 2026-04-08 22:02:50 +03:00
github-actions[bot] cef5994ea6 chore: bump version to 3.0.12 2026-04-08 18:58:39 +00:00
Ben Sigman 2c4abb9b59 Merge pull request #140 from igorls/fix/data-integrity
fix: use upsert and deterministic IDs to prevent data stagnation
2026-04-08 11:58:11 -07:00
Tal Muskal 4ce0d8491e fix: pin ruff <0.5 in CI to match local formatting, reset version to 3.0.11
CI was installing latest ruff (0.15.x) which has different formatting
rules than our local 0.4.x. Pin to ruff>=0.4.0,<0.5 for consistency.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 21:42:02 +03:00
github-actions[bot] ea32c2e659 chore: bump version to 3.0.12 2026-04-08 18:38:38 +00:00
Tal Muskal abd52534bb test: bring coverage to 85%, set threshold to 85, reset version to 3.0.11
- Add tests for config, convo_miner, spellcheck, knowledge_graph
- Fix Windows PermissionError in test cleanup (chromadb file locks)
- Add UTF-8 encoding to split_mega_files, entity_registry, hooks_cli
- Fix mcp_server parse_known_args logging for unknown args
- Set coverage threshold to 85 in pyproject.toml and CI
- Reset all version files to 3.0.11

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 21:38:12 +03:00
github-actions[bot] f47ca8a122 chore: bump version to 3.0.15 2026-04-08 18:23:32 +00:00
Tal Muskal 43c5a47172 ci: add separate jobs for Windows and macOS testing 2026-04-08 21:23:20 +03:00
Igor Lins e Silva edf8f36099 fix: use parse_known_args to allow importing mcp_server during pytest collection 2026-04-08 15:18:40 -03:00
Igor Lins e Silva a0bcd0c836 fix: ruff format test_hooks_cli.py and test_knowledge_graph.py 2026-04-08 15:12:12 -03:00
Igor Lins e Silva af42a850f6 fix: split semicolon statements onto two lines for ruff E702 2026-04-08 15:11:55 -03:00
Igor Lins e Silva bf88daa649 fix: address review — re-mine modified files, idempotent add_drawer, cleanup ChromaDB handles 2026-04-08 15:11:55 -03:00
Igor Lins e Silva a4149ab248 fix: use upsert and deterministic IDs to prevent data stagnation
MCP tool_add_drawer:
- Make drawer_id content-based: hash full content instead of
  content[:100] + timestamp. Same content → same ID, eliminating
  TOCTOU race conditions
- Switch from col.add() to col.upsert() so re-filing with updated
  content updates the existing drawer

miner.add_drawer:
- Switch from collection.add() to collection.upsert() so re-mining
  a modified file updates instead of silently failing
- Remove the try/except catching 'already exists' — upsert handles
  this naturally

Findings: #11 (HIGH — add ignores updates), #6 (MEDIUM — TOCTOU),
          #13 (MEDIUM — non-deterministic IDs)

Includes test infrastructure from PR #131.
92 tests pass.
2026-04-08 15:11:55 -03:00
github-actions[bot] 5cee307e7c chore: bump version to 3.0.14 2026-04-08 18:09:18 +00:00
Tal Muskal 9ca70264f3 style: format test files with ruff
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 21:08:49 +03:00
github-actions[bot] 65ac8ad977 chore: bump version to 3.0.13 2026-04-08 18:07:33 +00:00
Tal Muskal e24d8ca733 test: expand coverage to 70%, fix mcp_server CI crash (threshold 60%)
Add/expand tests for normalize (39%→97%), searcher (39%→100%),
layers (28%→97%), split_mega_files (34%→72%).

Fix mcp_server.py parse_args→parse_known_args to prevent SystemExit
when imported during pytest (CI was crashing on all test jobs).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 21:07:03 +03:00
github-actions[bot] ab66c0e945 chore: bump version to 3.0.12 2026-04-08 17:55:26 +00:00
Tal Muskal 03e9b57108 test: add comprehensive test coverage (35% → 58%, threshold 50%)
Add 180+ new tests across 10 test files covering previously untested modules:
- instructions_cli (0% → 100%), hooks_cli (73% → 96%), spellcheck (28% → 84%)
- palace_graph (9% → 91%), general_extractor (0% → 92%), entity_detector (0% → 69%)
- entity_registry (0% → 70%), room_detector_local (0% → 55%), layers (0% → 28%)
- onboarding (0% → 36%)

Also fixes Windows encoding bug in onboarding.py (write_text without encoding="utf-8").

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 20:54:56 +03:00
github-actions[bot] fcc9ce84f2 chore: bump version to 3.0.11 2026-04-08 17:46:56 +00:00
Ben Sigman c3ea596855 Merge pull request #264 from showaykerker/fix/palace-flag-ignored
fix: honour --palace flag in mcp_server
2026-04-08 10:46:46 -07:00
Ben Sigman ede3ad10c2 Merge pull request #236 from marerem/fix/silence-chromadb-telemetry-warnings
fix: silence ChromaDB telemetry warnings and CoreML segfault on Apple Silicon
2026-04-08 10:46:43 -07:00