The fuzzy clause's word string is cut to \w+ runs so no query grammar
reaches index.parse_query, but tantivy's boolean keywords are themselves
word runs. Under analyzed=True the field analyzer lowercased them into
ordinary terms before they got that far; now that the words are raw
query text, an uppercase keyword out of a quoted phrase arrives as
grammar: '"tax AND reports"' quietly made the clause a conjunction,
'"tax NOT reports"' gave it its own exclusion, and '"tax AND"' (or IN
anywhere) failed the parse and cost the query its fuzzy clause outright.
Lowercase exactly AND/OR/NOT/IN, which is what the analyzer used to do
and is the only spelling tantivy reads as grammar ("And" is a term).
Nothing else is touched: tantivy already lowercases query terms with the
field's analyzer, and doing it ourselves first is not the same operation
for every input (Python folds a final sigma differently, and turns 'İ'
into a sequence tantivy then splits in two), which would search for
terms the index does not contain.
Also pins two behaviours that were reasoned about but untested: the
fielded-CJK test now runs with the fuzzy clause on as well, where the
clause's documented unfielded contribution does bring the other document
back, and the negation tests pin the CJK over-admission for an exclusion
under an Or, which cannot be hoisted without dropping the other branch's
documents.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_try_parse_fuzzy_query collected free_text_tokens with the default
analyzed=True and handed the analyzer's output back to
index.parse_query, which analyzes it again. Analysis is not idempotent:
'universities' stems to 'univers', and re-stemming that yields 'univ', a
term the index does not contain. prefix=True hid the mistake as
over-broad matching rather than as no matches at all, which is why no
test caught it: searching 'universities' also returned documents whose
only relevant word was 'univalent' or 'unicycle'.
Collect the raw text instead. Raw text has not been tokenized, so the
whole-token \w+ filter that keeps tantivy query grammar out of the
re-parse would now reject ordinary input outright: 'COVID-19',
'hello@example.com' and the phrase "tax reports" each arrive as a single
token containing punctuation, and a query made only of such terms would
lose its fuzzy clause entirely. Cut each token into its word runs and
keep those, which recovers the terms and keeps the guarantee the filter
exists for: only word characters ever reach the parser.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stumpylogandClaude Opus 5