feat(search)!: let a wildcard match the typed run or its stem

whoosh-compat's pattern_normalizer now accepts several alternative forms
per literal run, ORed and deduplicated by the emitter, so the
shorter-of-the-two heuristic that had to pick one form is gone. The typed
run and its stem are both offered: neither is a prefix of the other once
the stemmer substitutes rather than truncates ("copy" -> "copi"), so
"copy*" now reaches "copies" and "copyright" alike instead of trading one
for the other.

checksum keeps _fold_normalizer. It is the only KEYWORD field, indexed
with the raw tokenizer, and a stemmed prefix there ("ceded" -> "cede")
returns documents whose checksum does not start with what was typed.

Also picks up two date-grammar fixes from the same library release: a
reversed relative range now swaps its bounds like the absolute case
instead of day-bumping the upper one, and a date value the grammar can
only half-consume (a bare, unquoted "added:2005-03-04T15:30:00Z") is
rejected as an InvalidDateQuery rather than silently matching nothing.

docs/usage.md: the "copy* does not find copyright" caveat is no longer
true; the bare timestamp is now an error rather than a silent non-match;
and the range-bracket quoting rule was wrong in a user-visible way. Only
double-quoted bounds are rejected, single-quoted ones parse.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
stumpylog
2026-08-20 15:02:33 -07:00
co-authored by Claude Opus 5
parent d17232043b
commit 5ecd237a3d
7 changed files with 188 additions and 99 deletions
+13 -9
View File
@@ -933,7 +933,7 @@ original_filename:invoice.pdf
- `asn` matches a document's Archive Serial Number.
- `page_count` matches a document's page count.
- `num_notes` matches how many notes a document has.
- `checksum` matches the checksum of the original document file (not the archived/processed version). Unlike the text fields, this one is stored verbatim rather than tokenized, so only a complete, lowercase checksum matches. To search by the first few characters instead, use a wildcard: `checksum:9f86d081*`. Wildcard patterns on the text fields are stemmed to line up with the stemmed index, but `checksum` is indexed without stemming, so its patterns are not stemmed either and the prefix is matched exactly as typed.
- `checksum` matches the checksum of the original document file (not the archived/processed version). Unlike the text fields, this one is stored verbatim rather than tokenized, so only a complete, lowercase checksum matches. To search by the first few characters instead, use a wildcard: `checksum:9f86d081*`. Wildcard patterns on the text fields are also tried stemmed, to line up with the stemmed index, but `checksum` is indexed without stemming, so its patterns are matched exactly as typed and nothing else.
- `original_filename` matches the filename of the document as originally consumed.
`asn`, `page_count` and `num_notes` are numeric and also accept ranges, for example `asn:[50 to 150]`.
@@ -946,12 +946,16 @@ title:Invoice*
```
Wildcards are matched against the _stemmed_ terms stored in the index, not
against the words as they appear in the document. A trailing `*` matches a word
and its inflections (`invoice*` finds "invoice", "invoices" and "invoiced"),
but not every longer word that starts with the same letters: `copy*` finds
"copy" and "copies", not "copyright". For the same reason, a pattern whose text
continues past where stemming cuts a word off cannot match at all:
`productname` is indexed as `productnam`, so `produ*name` finds nothing.
against the words as they appear in the document. Each literal part of a
pattern is tried both as you typed it and in its stemmed form, so a trailing
`*` matches a word and its inflections (`invoice*` finds "invoice", "invoices"
and "invoiced") as well as longer words whose stored term still begins with
what you typed (`copy*` finds "copyright" alongside "copy" and "copies").
It is still not a plain prefix search over the original text: where stemming
shortens a word, a pattern that reaches past the point it was cut off matches
nothing. `productname` is stored as `productnam`, so `produ*name` finds
nothing, and "happiness" is stored as `happi`, so `happine*` does not find it.
Matching natural date keywords:
@@ -990,8 +994,8 @@ added:[2005-06-15T09:00:00Z to 2005-06-15T17:00:00Z]
- An ISO date such as `2005-03-04` covers that whole day, and `2005-01` covers that whole month.
- A month name such as `january` covers that whole month in the current year.
- `next <weekday>` and `last <weekday>` each cover that whole day and must be quoted. A bare weekday name such as `monday` is not accepted.
- A full timestamp such as `2005-01-01T00:00:00Z` matches that exact instant. Like the other expressions above, it has to be quoted when it stands on its own: `added:"2005-01-01T00:00:00Z"`.
- A range takes two of the above as its bounds, for example `created:[2005 to 2009]` or `added:[2005-01-01 to 2005-01-31]`. Bounds may carry a time of day, and inside the brackets they are written without quotes.
- A full timestamp such as `2005-01-01T00:00:00Z` matches that exact instant. Like the other expressions above, it has to be quoted when it stands on its own: `added:"2005-01-01T00:00:00Z"`. The unquoted spelling is rejected with an error rather than searched, because only part of it can be read as a date.
- A range takes two of the above as its bounds, for example `created:[2005 to 2009]` or `added:[2005-01-01 to 2005-01-31]`. Bounds may carry a time of day. A bound is normally written without quotes; if you do quote one, use single quotes (`added:['-1 week' to now]`), because a double-quoted bound is rejected with an error.
!!! warning
+20 -14
View File
@@ -12,7 +12,7 @@ from documents.search._tokenizer import paperless_text_analyzer
from documents.search._tokenizer import stem_pattern_text
if TYPE_CHECKING:
from collections.abc import Callable
from whoosh_compat import PatternNormalizer
_registry_cache: dict[str | None, FieldRegistry] = {}
@@ -27,29 +27,35 @@ def _fold_normalizer(text: str) -> str:
return ascii_fold(text.lower())
def _make_pattern_normalizer(language: str | None) -> Callable[[str], str]:
def _make_pattern_normalizer(language: str | None) -> PatternNormalizer:
"""Build the wildcard/regex literal-run normalizer for a search language."""
def _pattern_normalizer(text: str) -> str:
"""Normalize a literal run so it can match indexed terms.
def _pattern_normalizer(text: str) -> tuple[str, ...]:
"""Normalize a literal run into the forms a term may match.
TEXT index terms go through lowercase -> ascii_fold -> stem, so a
pattern that skips stemming can never match one: "invoice*" would look
for a term starting with "invoice" while the index holds "invoic". The
run is therefore stemmed here too. KEYWORD fields are indexed raw and
get _fold_normalizer instead, so their patterns stay literal.
run is therefore offered stemmed as well. KEYWORD fields are indexed
raw and get _fold_normalizer instead, so their patterns stay literal.
A stem can be longer than the fragment the user typed, though, and a
longer prefix matches nothing, so the stem is used only when it is no
longer than the typed run. Length is a proxy for "the stem stayed close
to what was typed", not a guarantee of wider recall: a stem that
substitutes rather than truncates ("copy" -> "copi") moves the pattern
sideways instead of widening it, so "copy*" gains "copies" and loses
"copyright". test_pattern_stemming.py pins that trade.
Both forms are returned, as alternatives, because neither is a prefix
of the other in general: English stemming substitutes as well as
truncates ("copy" -> "copi"), so the stem alone loses the compounds
the typed run reaches ("copyright") while the typed run alone loses
the inflections the stem reaches ("copies"). whoosh-compat ORs the
alternatives per literal run and deduplicates them, so a run the
stemmer leaves alone costs exactly the one branch it did before.
Inside a bracket class the emitter calls this once per character and
uses the answer only if it is a single one-character form; two forms
there leave the character as typed. A stemmer does not change a lone
character, so the two forms deduplicate to one and the class body is
folded as before.
"""
folded = ascii_fold(text.lower())
stemmed = stem_pattern_text(folded, language)
return folded if len(stemmed) > len(folded) else stemmed
return (folded, stemmed)
return _pattern_normalizer
@@ -4,8 +4,9 @@ Every query here appears verbatim, or as a direct paraphrase, in the
"Document searches" section of ``docs/usage.md``. Each case indexes real
documents and asserts on matched document IDs rather than on the parsed
query, because a query that parses cleanly is not necessarily a query that
means what the documentation says it means: ``added:now - 3 days`` parses
without a single diagnostic and then matches nothing.
means what the documentation says it means: ``added:now`` parses without a
single diagnostic and then matches nothing, because it resolves to an
instant rather than to a span.
The negative cases matter as much as the positive ones. They pin the
behaviours the docs explicitly warn about, so that if any of them ever
@@ -25,6 +26,7 @@ import time_machine
from documents.models import Document
from documents.models import Note
from documents.models import Tag
from documents.search._errors import InvalidDateQuery
if TYPE_CHECKING:
from collections.abc import Generator
@@ -239,6 +241,9 @@ class TestDocumentedDateForms:
# spelling is pinned as a non-match below.
('added:"2005-03-04T15:30:00Z"', "old"),
("added:[2005-03-04T09:00:00Z to 2005-03-04T17:00:00Z]", "old"),
# A quoted range bound works when the quotes are single ones; the
# double-quoted spelling is pinned as an error below.
("added:['2005-03-04' to 2005-03-05]", "old"),
],
)
def test_documented_date_form_matches_its_day_or_month(
@@ -264,15 +269,6 @@ class TestDocumentedDateForms:
# of the resulting range, not the way the value is delimited.
'added:"now"',
'added:"midnight"',
'added:"-3 days"',
'added:"-1 week"',
# The trap: this reports no diagnostics but parses as
# And(added:now, "3", "days") - added:now plus stray text.
"added:now - 3 days",
# The bare, unquoted spelling of a full timestamp. The quoted and
# range-bound spellings above do work and match this fixture's
# document; only this one silently degrades to a non-match.
"added:2005-03-04T15:30:00Z",
],
)
def test_forms_the_docs_warn_about_match_nothing(
@@ -282,3 +278,36 @@ class TestDocumentedDateForms:
query: str,
) -> None:
assert _matched_ids(backend, query) == set()
def test_bare_timestamp_is_rejected_rather_than_matching_nothing(
self,
backend: TantivyBackend,
dated: dict[str, int],
) -> None:
"""The bare, unquoted spelling of a full timestamp. The quoted and
range-bound spellings pinned above do work and match this fixture's
document; this one is a user-fixable error rather than an empty
result set, so the docs tell the user to quote it.
The reported value is the prefix the date grammar could consume, not
the whole of what the user typed: paperless's own message, not this
value, is what has to carry the "quote it" guidance.
"""
with pytest.raises(InvalidDateQuery) as exc_info:
_matched_ids(backend, "added:2005-03-04T15:30:00Z")
assert exc_info.value.field == "added"
assert exc_info.value.value == "2005-03-"
def test_double_quoted_range_bound_is_rejected(
self,
backend: TantivyBackend,
dated: dict[str, int],
) -> None:
"""Quoting a range bound is allowed, but only with single quotes: the
double-quoted spelling reaches the date grammar with its quotes still
attached and is not a recognizable date. The docs say so, so pin which
of the two quote characters is the one that fails.
"""
with pytest.raises(InvalidDateQuery) as exc_info:
_matched_ids(backend, 'added:["2005-03-04" to 2005-03-05]')
assert exc_info.value.value == '"2005-03-04"'
@@ -17,9 +17,8 @@ from documents.models import Document
from documents.search._registry import get_field_registry
if TYPE_CHECKING:
from collections.abc import Callable
from whoosh_compat import FieldRegistry
from whoosh_compat import PatternNormalizer
from documents.search._backend import TantivyBackend
@@ -29,7 +28,7 @@ CEDEF00D = "cedef00ddeadbeef0123456789abcdef01234567"
CEDEDEAD = "cededeadbeef567801234567" + "89abcdef01234567"
def _normalizer(registry: FieldRegistry, name: str) -> Callable[[str], str]:
def _normalizer(registry: FieldRegistry, name: str) -> PatternNormalizer:
ref = registry.make_ref(name)
assert ref is not None
resolved = registry.resolve(ref)
@@ -48,12 +47,17 @@ class TestKeywordPatternNormalizer:
],
)
def test_keyword_runs_are_folded_not_stemmed(self, run: str) -> None:
"""One form, the run as typed: a KEYWORD pattern must never be widened
to a stem, which would return checksums that do not start with what
the user typed."""
normalize = _normalizer(get_field_registry("en"), "checksum")
assert normalize(run) == run
def test_text_runs_are_still_stemmed(self) -> None:
def test_text_runs_still_offer_their_stem(self) -> None:
"""A TEXT field offers the stem alongside the typed run, so a term
matching either one is reachable."""
normalize = _normalizer(get_field_registry("en"), "title")
assert normalize("Running") == "run"
assert tuple(normalize("Running")) == ("running", "run")
class TestChecksumPrefixQueries:
@@ -20,6 +20,8 @@ from documents.search._tokenizer import paperless_text_analyzer
from documents.search._tokenizer import stem_pattern_text
if TYPE_CHECKING:
from whoosh_compat import PatternNormalizer
from documents.search._backend import TantivyBackend
pytestmark = [pytest.mark.search, pytest.mark.django_db]
@@ -82,9 +84,10 @@ class TestPrefixStemming:
indexed_doc: Document,
query: str,
) -> None:
"""A partial prefix keeps matching: "librar" stems to "librari", which
is longer than what was typed, so the typed run is kept. Using the
shorter of the two widens recall rather than failing closed."""
"""A partial prefix keeps matching. "librar" stems to "librari", which
is longer than what was typed and so matches no term on its own, but
the run is offered as typed alongside its stem and that form reaches
"librari" in the index."""
assert _matched_ids(backend, query) == {indexed_doc.id}
def test_pattern_past_the_stem_boundary_is_documented_not_fixed(
@@ -97,20 +100,17 @@ class TestPrefixStemming:
is deliberate, not accidental."""
assert _matched_ids(backend, "produ*name") == set()
def test_stem_substitution_loses_compounds_accepted_trade(
def test_stem_substitution_reaches_both_the_inflection_and_the_compound(
self,
backend: TantivyBackend,
indexed_doc: Document,
) -> None:
"""English stemming substitutes as well as truncates: "copy" and
"copies" both index as "copi", while "copyright" keeps its literal "y".
So "copy*" reaches the base word and its inflections but no longer
reaches the compound, which it did before patterns were stemmed. That
trade is deliberate: the same substitution is what makes "company*" and
"library*" work at all, and no rule over one normalized string tells the
two apart. Matching both would need the pattern to be emitted as a
disjunction of the folded and stemmed forms, which belongs in the
emitter, not here.
Neither form is a prefix of the other, so no single normalized string
reaches both. The run is therefore emitted as a disjunction of the
folded and stemmed forms, and "copy*" reaches the base word, its
inflections and the compound alike.
"""
compound = Document.objects.create(
title="Copyright notice",
@@ -120,7 +120,7 @@ class TestPrefixStemming:
)
backend.add_or_update(compound)
assert _matched_ids(backend, "copy*") == {indexed_doc.id}
assert _matched_ids(backend, "copy*") == {indexed_doc.id, compound.id}
assert _matched_ids(backend, "copyright*") == {compound.id}
@@ -143,34 +143,67 @@ class TestStemsMatchTheIndexAnalyzer:
assert stem_pattern_text(ascii_fold(word.lower()), language) == indexed
def _forms(normalize: PatternNormalizer, text: str) -> tuple[str, ...]:
"""The distinct forms a term may match, in order, the way the emitter reads
the normalizer's answer (see whoosh_compat.PatternNormalizer)."""
result = normalize(text)
if isinstance(result, str):
return (result,)
return tuple(dict.fromkeys(result))
class TestPatternNormalizer:
@pytest.mark.parametrize(
("text", "expected"),
[
("Invoice", "invoic"),
("companies", "compani"),
# y -> i: same length as typed, and the index only holds the stem
("library", "librari"),
("invoic", "invoic"),
("Universit", "universit"),
("Café", "cafe"),
("Invoice", ("invoice", "invoic")),
("companies", ("companies", "compani")),
# y -> i is a substitution, so both forms are needed: the index
# holds "librari" for "library" and "library" for "librarian".
("library", ("library", "librari")),
# A run the stemmer leaves alone collapses back to one form, so it
# costs exactly the one regex branch it did before.
("invoic", ("invoic",)),
("Universit", ("universit",)),
("Café", ("cafe",)),
],
)
def test_shorter_of_the_typed_run_and_its_stem(
def test_offers_the_typed_run_and_its_stem(
self,
text: str,
expected: str,
expected: tuple[str, ...],
) -> None:
assert _make_pattern_normalizer("en")(text) == expected
assert _forms(_make_pattern_normalizer("en"), text) == expected
def test_run_that_yields_no_token_falls_back_to_the_typed_run(self) -> None:
"""A run past the remove_long limit analyzes to zero tokens, so there is
no stem to substitute and the folded run is used as typed."""
no stem to offer and only the folded run remains."""
over_long = "invoices" * 20
assert _make_pattern_normalizer("en")(over_long) == over_long
assert _forms(_make_pattern_normalizer("en"), over_long) == (over_long,)
@pytest.mark.parametrize("language", [None, "klingon"])
def test_unstemmed_language_folds_only(self, language: str | None) -> None:
"""With no stemmer configured, or one this build has no stemmer for, the
index holds surface forms and the pattern must keep them too."""
assert _make_pattern_normalizer(language)("Invoices") == "invoices"
assert _forms(_make_pattern_normalizer(language), "Invoices") == ("invoices",)
@pytest.mark.parametrize("char", ["a", "Z", "é"])
def test_a_single_character_collapses_to_one_folded_form(self, char: str) -> None:
"""A bracket class body is normalized one character at a time and the
answer is used only when it is a single one-character form, so a
stemmer that changed a lone character would silently disable folding
inside classes."""
forms = _forms(_make_pattern_normalizer("en"), char)
assert len(forms) == 1
assert len(forms[0]) == 1
class TestBracketClassStillFolds:
def test_class_body_matches_case_insensitively(
self,
backend: TantivyBackend,
indexed_doc: Document,
) -> None:
"""The class body is folded per character, which the alternatives
contract preserves only because a lone character stems to itself."""
assert _matched_ids(backend, "title:[IP]nvoice*") == {indexed_doc.id}
+22 -5
View File
@@ -1,3 +1,5 @@
from collections.abc import Sequence
import pytest
from whoosh_compat import FieldKind
from whoosh_compat import FieldRegistry
@@ -20,6 +22,15 @@ def _resolve(registry: FieldRegistry, name: str) -> ResolvedField:
return resolved
def _distinct_forms(result: str | Sequence[str]) -> tuple[str, ...]:
"""The forms a term may match, in order, the way whoosh-compat's emitter
reads a pattern_normalizer's answer: a bare str is one form, a sequence is
several, deduplicated."""
if isinstance(result, str):
return (result,)
return tuple(dict.fromkeys(result))
class TestFieldRegistry:
def test_internal_id_fields_are_not_registered(
self,
@@ -103,16 +114,22 @@ class TestFieldRegistry:
self,
registry: FieldRegistry,
) -> None:
# Index terms are stemmed, so patterns are too, using the registry's
# own language: "Running" has to reach the indexed "run". Without a
# language the index holds surface forms, so it only case/accent-folds.
# Index terms are stemmed, so patterns offer their stem too, using the
# registry's own language: "Running" has to reach the indexed "run".
# Without a language the index holds surface forms, so there is no
# second form and the run is only case/accent-folded.
resolved = _resolve(registry, "title")
assert resolved.spec.pattern_normalizer is not None
assert resolved.spec.pattern_normalizer("Running") == "running"
assert _distinct_forms(resolved.spec.pattern_normalizer("Running")) == (
"running",
)
resolved_en = _resolve(get_field_registry("en"), "title")
assert resolved_en.spec.pattern_normalizer is not None
assert resolved_en.spec.pattern_normalizer("Running") == "run"
assert _distinct_forms(resolved_en.spec.pattern_normalizer("Running")) == (
"running",
"run",
)
def test_registry_is_cached_per_language(self) -> None:
a = get_field_registry("en")
@@ -1,26 +1,23 @@
"""Reversed date ranges: an internal inconsistency between relative and
absolute bounds, pinned exactly as measured rather than "fixed" here.
"""Reversed date ranges swap their bounds back into order.
Measured end to end (FROZEN_NOW = 2026-06-15T12:00:00Z):
added:[now-1h to now+1h] -> 2h window, correct order
added:[now+1h to now-1h] -> ~22h window, DAY-BUMPED, not swapped
added:[now+1h to now-1h] -> the same 2h window, SWAPPED
added:[2020-01-01 to 2019-01-01] -> 366 days, SWAPPED to the forward order
added:[2019-01-01 to 2020-01-01] -> 366 days (same result either way)
Absolute reversed ranges swap their bounds back into order, matching
whoosh's own behavior. Relative (``now±``) reversed ranges do not swap --
whoosh-compat's date grammar instead adds a day to the upper bound,
producing a much wider window than either the forward or a swapped
reading would give. This is a library-level inconsistency between the two
range kinds, not an application-level rewrite paperless performs (there is
no reversed-range handling in documents/search/_query.py), so it is not
"fixed" here: fixing it belongs in whoosh-compat's date grammar, not in a
pre-parse rewrite on this side (see also the CJK/pre-parse-rewrites
docstrings elsewhere in this suite for the same discipline). Pinned as a
known inconsistency and a library follow-up. When whoosh-compat's date
grammar is fixed to swap consistently, the relative case's test below
inverts -- that inversion is the intended, visible signal.
Both range kinds behave the same way, matching whoosh's own behavior.
Relative (``now±``) reversed ranges used to differ: whoosh-compat's date
grammar added a day to the upper bound instead of swapping, producing a
much wider window than either the forward or a swapped reading gives. That
was a library-level inconsistency between the two range kinds, not an
application-level rewrite paperless performs (there is still no
reversed-range handling in documents/search/_query.py), and it was fixed in
whoosh-compat's date grammar rather than in a pre-parse rewrite on this
side (see also the CJK/pre-parse-rewrites docstrings elsewhere in this
suite for the same discipline). Both cases are pinned below so a future
divergence between them is visible again.
"""
from __future__ import annotations
@@ -52,13 +49,13 @@ def _index(backend: TantivyBackend, **kwargs: object) -> Document:
return doc
class TestRelativeReversedRangeDayBumpsInsteadOfSwapping:
class TestRelativeReversedRangeSwaps:
@pytest.fixture
def docs(self, backend: TantivyBackend) -> dict[str, int]:
with time_machine.travel(FROZEN_NOW, tick=False):
return {
# Inside the correct (forward) 2h window [11:00, 13:00],
# outside the day-bumped window [13:00, next-day 11:00).
# Inside the 2h window [11:00, 13:00] that both the forward
# and the reversed spelling resolve to.
"in_forward_window": _index(
backend,
title="Forward window doc",
@@ -66,13 +63,14 @@ class TestRelativeReversedRangeDayBumpsInsteadOfSwapping:
checksum="reversed-relative-forward",
added=FROZEN_NOW,
).pk,
# Outside the correct 2h window, inside the day-bumped
# window the reversed query actually produces.
"in_daybumped_window": _index(
# Outside that window, but inside the wider window the
# reversed spelling produced when it day-bumped instead of
# swapping. It must not match either query now.
"outside_the_window": _index(
backend,
title="Day-bumped window doc",
title="Later same-week doc",
content="x",
checksum="reversed-relative-daybumped",
checksum="reversed-relative-outside",
added=datetime(2026, 6, 16, 8, 0, tzinfo=UTC),
).pk,
}
@@ -87,18 +85,16 @@ class TestRelativeReversedRangeDayBumpsInsteadOfSwapping:
docs["in_forward_window"],
}
def test_reversed_range_day_bumps_rather_than_swapping(
def test_reversed_range_swaps_to_the_same_window(
self,
backend: TantivyBackend,
docs: dict[str, int],
) -> None:
# If this swapped like the absolute case below, it would match
# "in_forward_window" (the same set as the forward query). It
# instead matches only the day-bumped document -- the documented
# library inconsistency.
# The same set as the forward query, and in particular not the
# document that only a day-bumped upper bound would have reached.
with time_machine.travel(FROZEN_NOW, tick=False):
assert _matched_ids(backend, "added:[now+1h to now-1h]") == {
docs["in_daybumped_window"],
docs["in_forward_window"],
}