docs(search): pin the stem-substitution limit wildcards inherit

Stemming substitutes as well as truncates ("copy" and "copies" both index as
"copi" while "copyright" keeps its literal y), so a stemmed pattern reaches a
word's inflections but no longer reaches compounds that keep the surface
spelling. That trade is accepted: the same substitution is what makes company*
and library* work, and no rule over one normalized string separates them. So
usage.md stops claiming a trailing star just works, and a test pins copy* to the
base word rather than the compound. Also adds a parity test tying
stem_pattern_text to paperless_text_analyzer's own output, so a filter added to
the index analyzer alone cannot silently diverge, and corrects the docstring
claim that a run can analyze to several tokens - the raw tokenizer emits one
token whatever the input, so only the remove_long zero-token case can fire.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
stumpylog
2026-08-20 08:01:02 -07:00
co-authored by Claude Opus 5
parent e311c84139
commit e48182c358
3 changed files with 63 additions and 10 deletions
+6 -4
View File
@@ -927,10 +927,12 @@ title:Invoice*
```
Wildcards are matched against the _stemmed_ terms stored in the index, not
against the words as they appear in the document. A trailing `*` therefore
works as expected (`invoice*` finds "invoice", "invoices" and "invoiced"), but
a pattern whose text continues past where stemming cuts a word off cannot
match: `productname` is indexed as `productnam`, so `produ*name` finds nothing.
against the words as they appear in the document. A trailing `*` matches a word
and its inflections (`invoice*` finds "invoice", "invoices" and "invoiced"),
but not every longer word that starts with the same letters: `copy*` finds
"copy" and "copies", not "copyright". For the same reason, a pattern whose text
continues past where stemming cuts a word off cannot match at all:
`productname` is indexed as `productnam`, so `produ*name` finds nothing.
Matching natural date keywords: