1
0
forked from jason/echo

ver 1.5.1 — duplicate-gate precision (common tokens, cross-kind)

Live 1.5.0 use caught the gate blocking a decision note over the single
vault-common token "echo" (min-normalized overlap gives any one-token
entity a 1.0 score). New echo_index.gate_candidates() + token_df():

- gate blocks same-kind candidates only; cross-kind name collisions
  (a decision titled after its project) warn instead of blocking
- a lone shared token blocks only when unique to that entity (df == 1);
  multi-token overlaps still block

fuzzy_candidates (advisory warnings) and exit-76/--merge-into/--force
semantics unchanged. +3 offline unit tests, gate e2e cases restructured
+2 new; verified live both directions. All suites green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jason Stedwell
2026-07-03 13:03:07 -05:00
parent be3fd36ebf
commit 3b6c3053cb
10 changed files with 155 additions and 15 deletions
@@ -22,6 +22,7 @@ from __future__ import annotations
import json
import re
import sys
from collections import Counter
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
@@ -236,6 +237,56 @@ def fuzzy_candidates(index: dict, mention: str, limit: int = 5):
return [(slug, e, score) for score, _, slug, e in scored[:limit]]
def _entity_tokens(slug: str, e: dict) -> set[str]:
toks: set[str] = set()
for name in (slug, e.get("title", ""), *e.get("aliases", [])):
toks |= _sig_tokens(name)
return toks
def token_df(index: dict) -> Counter:
"""Vault-wide token document frequency: token -> how many entities carry it in their
slug/title/aliases. A token shared by several entities ("echo" in an echo-centric
vault) is a weak duplicate signal on its own; df lets the gate discount it."""
df: Counter = Counter()
for slug, e in index.get("entities", {}).items():
for t in _entity_tokens(slug, e):
df[t] += 1
return df
def gate_candidates(index: dict, mention: str, kind: str | None = None,
threshold: float = 0.6):
"""The subset of fuzzy_candidates strong enough to BLOCK a create (capture's
duplicate gate). Stricter than the advisory warning list, by two rules learned
from live use (1.5.1):
* same-kind only — a cross-kind name collision (a decision titled after its
project, a meeting named for a company) is usually intentional naming, not a
duplicate. Cross-kind lookalikes still surface as warnings, never as a block.
* no single-common-token blocks — score = overlap/min(|tokens|) means an entity
whose ONE distinctive token appears in the mention scores 1.0, so any title
containing a vault-common word ("echo") would be blocked by every such entity.
A lone shared token only blocks when it is unique to that entity (df == 1).
Returns [(slug, entry, score)] like fuzzy_candidates."""
mtoks = _sig_tokens(mention)
if not mtoks:
return []
df = token_df(index)
out = []
for slug, e, score in fuzzy_candidates(index, mention):
if score < threshold:
continue
if kind and e.get("kind") and e["kind"] != kind:
continue
overlap = mtoks & _entity_tokens(slug, e)
if len(overlap) == 1 and df[next(iter(overlap))] > 1:
continue
out.append((slug, e, score))
return out
def upsert(index: dict, slug: str, path: str, kind: str, title: str | None = None,
aliases=None) -> dict:
ents = index.setdefault("entities", {})