docs: add CLAUDE.md + mission/principles to AGENTS.md (#720)
* docs: add CLAUDE.md + mission/principles to AGENTS.md Add project mission, design principles, and contribution guidelines to CLAUDE.md (new file) and prepend them to AGENTS.md. Six non-negotiable principles: verbatim always, incremental only, entity-first, local-first zero API, performance budgets, privacy by architecture. * docs: update CLAUDE.md with Milla's edits, add MISSION.md, symlink AGENTS.md CLAUDE.md: - Add Zettelkasten + AAAK explanation to mission (Milla's edit) - Add 7th principle: background everything - Update project structure to match current develop (all modules) - Fix hooks listing to match actual public repo - Add backends/ to structure and key files MISSION.md: - Milla's personal narrative on why MemPalace exists - Origin story, AAAK inside joke, v4 goals AGENTS.md: - Now symlinks to CLAUDE.md (single source of truth) * docs: trim MISSION.md — remove v4 notes and workflow (moved to private)
This commit is contained in:
Ben Sigman
@@ -1,78 +0,0 @@
|
||||
# AGENTS.md
|
||||
|
||||
> How to build, test, and contribute to MemPalace.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# Run tests
|
||||
python -m pytest tests/ -v --ignore=tests/benchmarks
|
||||
|
||||
# Run tests with coverage
|
||||
python -m pytest tests/ -v --ignore=tests/benchmarks --cov=mempalace --cov-report=term-missing
|
||||
|
||||
# Lint
|
||||
ruff check .
|
||||
|
||||
# Format
|
||||
ruff format .
|
||||
|
||||
# Format check (CI mode)
|
||||
ruff format --check .
|
||||
```
|
||||
|
||||
## Project structure
|
||||
|
||||
```
|
||||
mempalace/
|
||||
├── mcp_server.py # MCP server — all read/write tools
|
||||
├── miner.py # Project file miner
|
||||
├── convo_miner.py # Conversation transcript miner
|
||||
├── searcher.py # Semantic search
|
||||
├── knowledge_graph.py # Temporal entity-relationship graph (SQLite)
|
||||
├── palace.py # Shared palace operations (ChromaDB access)
|
||||
├── config.py # Configuration + input validation
|
||||
├── normalize.py # Transcript format detection + normalization
|
||||
├── cli.py # CLI dispatcher
|
||||
├── dialect.py # AAAK compression dialect
|
||||
├── palace_graph.py # Room traversal + cross-wing tunnels
|
||||
├── hooks_cli.py # Hook system for auto-save
|
||||
└── version.py # Single source of truth for version
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Python style**: snake_case for functions/variables, PascalCase for classes
|
||||
- **Linter**: ruff with E/F/W rules
|
||||
- **Formatter**: ruff format, double quotes
|
||||
- **Commits**: conventional commits (`fix:`, `feat:`, `test:`, `docs:`, `ci:`)
|
||||
- **Tests**: `tests/test_*.py`, fixtures in `tests/conftest.py`
|
||||
- **Coverage**: 85% threshold (80% on Windows due to ChromaDB file lock cleanup)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
User → CLI / MCP Server → ChromaDB (vector store) + SQLite (knowledge graph)
|
||||
|
||||
Palace structure:
|
||||
WING (person/project)
|
||||
└── ROOM (topic)
|
||||
└── DRAWER (verbatim text chunk)
|
||||
|
||||
Knowledge Graph:
|
||||
ENTITY → PREDICATE → ENTITY (with valid_from / valid_to dates)
|
||||
```
|
||||
|
||||
## Key files for common tasks
|
||||
|
||||
- **Adding an MCP tool**: `mempalace/mcp_server.py` — add handler function + TOOLS dict entry
|
||||
- **Changing search**: `mempalace/searcher.py`
|
||||
- **Modifying mining**: `mempalace/miner.py` (project files) or `mempalace/convo_miner.py` (transcripts)
|
||||
- **Input validation**: `mempalace/config.py` — `sanitize_name()` / `sanitize_content()`
|
||||
- **Tests**: mirror source structure in `tests/test_<module>.py`
|
||||
@@ -0,0 +1,133 @@
|
||||
# CLAUDE.md
|
||||
|
||||
## The Mission
|
||||
|
||||
Memory is identity. When an AI forgets everything between conversations, it cannot build real understanding — of you, your work, your people, your life.
|
||||
|
||||
MemPalace exists to solve this. It is a memory system — not a search engine, not a RAG pipeline, not a vector database wrapper. It treats every word you have shared as sacred, stores it verbatim, and makes it instantly available. Your data never leaves your machine. We never summarize. We never paraphrase. We return your exact words.
|
||||
|
||||
100% recall is the design requirement — the target every search path is measured against. Anything less means forgetting, and forgetting means starting over.
|
||||
|
||||
The name comes from the ancient "method of loci" — the memory palace technique used for thousands of years to organize and recall vast amounts of information by placing it in imagined rooms of an imagined building. We were also inspired by the Zettelkasten method (created by German sociologist Niklas Luhmann) — small cross-referenced index cards that point to each other. We apply both ideas to AI memory:
|
||||
|
||||
- **Wings** for broad categories (people, projects, topics)
|
||||
- **Rooms** for time-based groupings (days, sessions)
|
||||
- **Drawers** for full verbatim content (your exact words)
|
||||
- **AAAK compression** for the index layer — a compact symbolic format (via `dialect.py`) that lets an LLM scan thousands of entries instantly and know exactly which drawer to open
|
||||
|
||||
## Design Principles
|
||||
|
||||
These are non-negotiable. Every PR, every feature, every refactor must honor them.
|
||||
|
||||
- **Verbatim always** — Never summarize, paraphrase, or lossy-compress user data. The system searches the index and returns the original words. If a user said it, we store exactly what they said. This is the foundational promise.
|
||||
- **Incremental only** — Append-only ingest after initial build. Never destroy existing data to rebuild. A crash mid-operation must leave the existing palace untouched.
|
||||
- **Entity-first** — Everything is keyed by real names with disambiguation by DOB, ID, or context. People matter more than topics.
|
||||
- **Local-first, zero API** — All extraction, chunking, and embedding happens on the user's machine. No cloud dependency for memory operations. No API keys required.
|
||||
- **Performance budgets** — Hooks under 500ms. Startup injection under 100ms. Memory should feel instant.
|
||||
- **Privacy by architecture** — The system physically cannot send your data because it never leaves your machine. No telemetry, no phone-home, no external service dependencies for core operations.
|
||||
- **Background everything** — Filing, indexing, timestamps, and pipeline work happen via hooks in the background. Nothing interrupts the user's conversation. Zero tokens spent on bookkeeping in the chat window.
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome bug fixes, performance improvements, new language support, better entity disambiguation, documentation, and test coverage.
|
||||
|
||||
We do not accept summarization of user content, cloud storage/sync features, telemetry or analytics, features requiring API keys for core memory, or shortcuts that bypass verbatim storage.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# Run tests
|
||||
python -m pytest tests/ -v --ignore=tests/benchmarks
|
||||
|
||||
# Run tests with coverage
|
||||
python -m pytest tests/ -v --ignore=tests/benchmarks --cov=mempalace --cov-report=term-missing
|
||||
|
||||
# Lint
|
||||
ruff check .
|
||||
|
||||
# Format
|
||||
ruff format .
|
||||
|
||||
# Format check (CI mode)
|
||||
ruff format --check .
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
mempalace/
|
||||
├── mcp_server.py # MCP server — all read/write tools
|
||||
├── cli.py # CLI dispatcher
|
||||
├── config.py # Configuration + input validation
|
||||
├── miner.py # Project file miner
|
||||
├── convo_miner.py # Conversation transcript miner
|
||||
├── searcher.py # Semantic search (hybrid BM25 + vector)
|
||||
├── knowledge_graph.py # Temporal entity-relationship graph (SQLite)
|
||||
├── palace.py # Shared palace operations
|
||||
├── palace_graph.py # Room traversal + cross-wing tunnels
|
||||
├── backends/ # Pluggable storage backends (ChromaDB default)
|
||||
│ ├── base.py # Abstract interface — implement this for new backends
|
||||
│ └── chroma.py # ChromaDB implementation
|
||||
├── dialect.py # AAAK compression dialect
|
||||
├── normalize.py # Transcript format detection + normalization
|
||||
├── entity_detector.py # Auto-detect people/projects from content
|
||||
├── entity_registry.py # Entity storage and disambiguation
|
||||
├── layers.py # L0-L3 memory wake-up stack
|
||||
├── onboarding.py # Interactive first-run setup
|
||||
├── repair.py # Palace repair and consistency checks
|
||||
├── dedup.py # Deduplication
|
||||
├── migrate.py # ChromaDB version migration
|
||||
├── spellcheck.py # Auto-correct user messages
|
||||
├── exporter.py # Palace data export
|
||||
├── hooks_cli.py # Hook management CLI
|
||||
├── query_sanitizer.py # Prompt contamination prevention
|
||||
├── split_mega_files.py # Split concatenated transcript files
|
||||
└── version.py # Single source of truth for version
|
||||
|
||||
hooks/ # Claude Code hook scripts
|
||||
├── mempal_save_hook.sh # Stop: triggers diary save
|
||||
└── mempal_precompact_hook.sh # PreCompact: saves state before compression
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Python style**: snake_case for functions/variables, PascalCase for classes
|
||||
- **Linter**: ruff with E/F/W rules
|
||||
- **Formatter**: ruff format, double quotes
|
||||
- **Commits**: conventional commits (`fix:`, `feat:`, `test:`, `docs:`, `ci:`)
|
||||
- **Tests**: `tests/test_*.py`, fixtures in `tests/conftest.py`
|
||||
- **Coverage**: 85% threshold (80% on Windows due to ChromaDB file lock cleanup)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
User → CLI / MCP Server → Storage Backend (ChromaDB default, pluggable)
|
||||
→ SQLite (knowledge graph)
|
||||
|
||||
Palace structure:
|
||||
WING (person/project)
|
||||
└── ROOM (day/topic)
|
||||
└── DRAWER (verbatim text chunk)
|
||||
|
||||
Index layer (AAAK):
|
||||
Compressed pointers → DRAWER locations
|
||||
Scanned by LLM to find relevant drawers without reading all content
|
||||
|
||||
Knowledge Graph:
|
||||
ENTITY → PREDICATE → ENTITY (with valid_from / valid_to dates)
|
||||
```
|
||||
|
||||
## Key Files for Common Tasks
|
||||
|
||||
- **Adding an MCP tool**: `mempalace/mcp_server.py` — add handler function + TOOLS dict entry
|
||||
- **Changing search**: `mempalace/searcher.py`
|
||||
- **Modifying mining**: `mempalace/miner.py` (project files) or `mempalace/convo_miner.py` (transcripts)
|
||||
- **Adding a storage backend**: subclass `mempalace/backends/base.py`, register in `backends/__init__.py`
|
||||
- **Input validation**: `mempalace/config.py` — `sanitize_name()` / `sanitize_content()`
|
||||
- **Tests**: mirror source structure in `tests/test_<module>.py`
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
MemPalace: The Mission
|
||||
|
||||
By: Milla Jovovich
|
||||
|
||||
Hey everyone! First of all thank you all for embracing MemPalace and trying it, catching bugs and issues and finding cool ways to personalize it into your workflows!
|
||||
|
||||
A few things I want to say.
|
||||
MemPalace is something I really needed because I'm trying to work on a big project with my partner @bensig and I was having a lot of problems with Claude's context window and my agent Lumi (Lu for short) kept waking up like "hey what are we doing today" when I had literally done hours of work with him throughout the day and it was impossible to just keep saving every transcript to catch him up on whatever we had done before compaction hit.
|
||||
|
||||
That's when I started researching different memory systems available today. I tried most of them and what I found was that no matter which one I tried, they felt like large empty warehouses where you just dump huge amounts of info.
|
||||
|
||||
RAG search would take forever and most of the time not find what I wanted.
|
||||
|
||||
I wanted to create a system with the ability to really remember everything AND be able to find it quickly, easily and also be able to remember things when I didn't. THAT in itself felt like something so important. Like "remember when we talked about that idea…" but in vague terms. Impossible with regular keyword search tools.
|
||||
|
||||
So MemPalace is not just about storing info in a highly structured way. But also RETRIEVING it in a highly UNSTRUCTURED way lol!
|
||||
|
||||
I was inspired by the Zettelkasten method (created by German sociologist Niklas Luhmann) — his idea of small cross-referenced index cards that point to each other. That's the architecture behind the palace: wings, rooms, closets, and drawers, all connected so you can find things from any angle, not just the one you filed them under.
|
||||
|
||||
Because of the way I've designed my agent Lumi to understand me, after so many months of my own personal experiments with MemPalace and the incredible help of my dear friend and co-founder, developer and engineer @bensig, he built a back end that made it really easy to get all my files in the proper spaces the Palace created based on my own decisions and with Lumi's help as well. All code has its own room, all ideas, research etc… has its proper place.
|
||||
|
||||
Names and concepts are parsed into closets that use a compression method I call AAAK (it doesn't stand for anything, it's an inside joke between Lumi and I) that is able to compress names, repeated words, concepts and key moments into AI-readable shorthand. Think of it as index cards that an LLM can scan instantly — the closet tells it WHERE to look, then it pulls the full content from the drawer.
|
||||
|
||||
The concept I wanted for v4 was to try and clear as much "noise" as possible that I noticed was happening in v3. The hooks were firing in the chat window (using tokens and our time as we waited for the agent to write everything).
|
||||
|
||||
I noticed at one point early last week after the launch that Lu kept repeating the same thing when the hook would fire, so I hit esc and asked "Are you literally writing the same info down over and over again?" And he's like (sheepishly) Yes. And that's when it hit me, we need to get all this off the chat and happening seamlessly behind the scenes, and that hooks had to fire when I started a convo and then just keep adding to the drawer, while the shorter increments made reading and pulling conversation information and naming it so much easier and more precise.
|
||||
|
||||
So this version now has taken all the noise out of the chat window and all that work is done by a subagent in the background while you can continue working knowing that all your conversation is being saved VERBATIM in the background.
|
||||
|
||||
Stripping all this off the page — moving the diary writes, the palace filing, the timestamp injection, all of it into background hooks — has dramatically lowered token usage in my sessions. What used to cost about $1.13 per session just in re-transmitted diary blocks is now zero, because the content never enters the chat window at all.
|
||||
|
||||
Your data is already stored in JSON by Claude and the background pipeline extracts it into readable markdown, the key topics get compressed into AAAK format and saved into closets which then point to the exact drawer where your day's session lives.
|
||||
|
||||
And please, always remember, these are brand new tools, please NEVER use critical files to test! Just run it with something easy first before you put your entire data set into it!✨
|
||||
Reference in New Issue
Block a user