forked from jason/echo
ver 1.5.1 — duplicate-gate precision (common tokens, cross-kind)
Live 1.5.0 use caught the gate blocking a decision note over the single vault-common token "echo" (min-normalized overlap gives any one-token entity a 1.0 score). New echo_index.gate_candidates() + token_df(): - gate blocks same-kind candidates only; cross-kind name collisions (a decision titled after its project) warn instead of blocking - a lone shared token blocks only when unique to that entity (df == 1); multi-token overlaps still block fuzzy_candidates (advisory warnings) and exit-76/--merge-into/--force semantics unchanged. +3 offline unit tests, gate e2e cases restructured +2 new; verified live both directions. All suites green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -22,6 +22,7 @@ from __future__ import annotations
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
@@ -236,6 +237,56 @@ def fuzzy_candidates(index: dict, mention: str, limit: int = 5):
|
||||
return [(slug, e, score) for score, _, slug, e in scored[:limit]]
|
||||
|
||||
|
||||
def _entity_tokens(slug: str, e: dict) -> set[str]:
|
||||
toks: set[str] = set()
|
||||
for name in (slug, e.get("title", ""), *e.get("aliases", [])):
|
||||
toks |= _sig_tokens(name)
|
||||
return toks
|
||||
|
||||
|
||||
def token_df(index: dict) -> Counter:
|
||||
"""Vault-wide token document frequency: token -> how many entities carry it in their
|
||||
slug/title/aliases. A token shared by several entities ("echo" in an echo-centric
|
||||
vault) is a weak duplicate signal on its own; df lets the gate discount it."""
|
||||
df: Counter = Counter()
|
||||
for slug, e in index.get("entities", {}).items():
|
||||
for t in _entity_tokens(slug, e):
|
||||
df[t] += 1
|
||||
return df
|
||||
|
||||
|
||||
def gate_candidates(index: dict, mention: str, kind: str | None = None,
|
||||
threshold: float = 0.6):
|
||||
"""The subset of fuzzy_candidates strong enough to BLOCK a create (capture's
|
||||
duplicate gate). Stricter than the advisory warning list, by two rules learned
|
||||
from live use (1.5.1):
|
||||
|
||||
* same-kind only — a cross-kind name collision (a decision titled after its
|
||||
project, a meeting named for a company) is usually intentional naming, not a
|
||||
duplicate. Cross-kind lookalikes still surface as warnings, never as a block.
|
||||
* no single-common-token blocks — score = overlap/min(|tokens|) means an entity
|
||||
whose ONE distinctive token appears in the mention scores 1.0, so any title
|
||||
containing a vault-common word ("echo") would be blocked by every such entity.
|
||||
A lone shared token only blocks when it is unique to that entity (df == 1).
|
||||
|
||||
Returns [(slug, entry, score)] like fuzzy_candidates."""
|
||||
mtoks = _sig_tokens(mention)
|
||||
if not mtoks:
|
||||
return []
|
||||
df = token_df(index)
|
||||
out = []
|
||||
for slug, e, score in fuzzy_candidates(index, mention):
|
||||
if score < threshold:
|
||||
continue
|
||||
if kind and e.get("kind") and e["kind"] != kind:
|
||||
continue
|
||||
overlap = mtoks & _entity_tokens(slug, e)
|
||||
if len(overlap) == 1 and df[next(iter(overlap))] > 1:
|
||||
continue
|
||||
out.append((slug, e, score))
|
||||
return out
|
||||
|
||||
|
||||
def upsert(index: dict, slug: str, path: str, kind: str, title: str | None = None,
|
||||
aliases=None) -> dict:
|
||||
ents = index.setdefault("entities", {})
|
||||
|
||||
Reference in New Issue
Block a user