Feature: match CJK terms through their bigram fields in place (#14156)

QUERY-mode searches blended a separate bigram clause in at the top of the
query, built from the parsed AST's free-text tokens. Because it sat beside
the exact clause rather than inside the query, nothing around a CJK term
constrained its bigram match: an exclusion that was one OR branch's own
condition could never reach it, so "(東京 AND NOT secret) OR bill" still
returned the secret document.

Widen each CJK leaf where it sits instead, through emit()'s rewrite_leaf
hook, so every AND, NOT, REQUIRE, boost and field restriction around the
leaf applies to its bigram match too. Negated leaves are widened on
purpose, so "NOT X" excludes exactly what "X" matches.
This commit is contained in:
Trenton H authored and GitHub committed 2026-09-17 14:55:13 -07:00
1 parent 420bf503e8
commit 762e8cf4d1
16 files changed
+2019 -252

No files matched your search

@@ -51,7 +51,7 @@ class TestFuzzyClauseParseFailureDegradesGracefully:
THEN:
- It returns None instead of propagating, so a fuzzy word
string tantivy's own parser rejects only drops the fuzzy
clause: the exact/CJK clauses still stand rather than the
clause: the exact clause still stands rather than the
whole query failing. The ValueError guard is insurance (the
word string is plain tokens, so tantivy accepting it is
expected, not assumed)