mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-08-25 20:23:18 +00:00
feat(search)!: let a wildcard match the typed run or its stem
whoosh-compat's pattern_normalizer now accepts several alternative forms
per literal run, ORed and deduplicated by the emitter, so the
shorter-of-the-two heuristic that had to pick one form is gone. The typed
run and its stem are both offered: neither is a prefix of the other once
the stemmer substitutes rather than truncates ("copy" -> "copi"), so
"copy*" now reaches "copies" and "copyright" alike instead of trading one
for the other.
checksum keeps _fold_normalizer. It is the only KEYWORD field, indexed
with the raw tokenizer, and a stemmed prefix there ("ceded" -> "cede")
returns documents whose checksum does not start with what was typed.
Also picks up two date-grammar fixes from the same library release: a
reversed relative range now swaps its bounds like the absolute case
instead of day-bumping the upper one, and a date value the grammar can
only half-consume (a bare, unquoted "added:2005-03-04T15:30:00Z") is
rejected as an InvalidDateQuery rather than silently matching nothing.
docs/usage.md: the "copy* does not find copyright" caveat is no longer
true; the bare timestamp is now an error rather than a silent non-match;
and the range-bracket quoting rule was wrong in a user-visible way. Only
double-quoted bounds are rejected, single-quoted ones parse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
d17232043b
commit
5ecd237a3d
+13
-9
@@ -933,7 +933,7 @@ original_filename:invoice.pdf
|
||||
- `asn` matches a document's Archive Serial Number.
|
||||
- `page_count` matches a document's page count.
|
||||
- `num_notes` matches how many notes a document has.
|
||||
- `checksum` matches the checksum of the original document file (not the archived/processed version). Unlike the text fields, this one is stored verbatim rather than tokenized, so only a complete, lowercase checksum matches. To search by the first few characters instead, use a wildcard: `checksum:9f86d081*`. Wildcard patterns on the text fields are stemmed to line up with the stemmed index, but `checksum` is indexed without stemming, so its patterns are not stemmed either and the prefix is matched exactly as typed.
|
||||
- `checksum` matches the checksum of the original document file (not the archived/processed version). Unlike the text fields, this one is stored verbatim rather than tokenized, so only a complete, lowercase checksum matches. To search by the first few characters instead, use a wildcard: `checksum:9f86d081*`. Wildcard patterns on the text fields are also tried stemmed, to line up with the stemmed index, but `checksum` is indexed without stemming, so its patterns are matched exactly as typed and nothing else.
|
||||
- `original_filename` matches the filename of the document as originally consumed.
|
||||
|
||||
`asn`, `page_count` and `num_notes` are numeric and also accept ranges, for example `asn:[50 to 150]`.
|
||||
@@ -946,12 +946,16 @@ title:Invoice*
|
||||
```
|
||||
|
||||
Wildcards are matched against the _stemmed_ terms stored in the index, not
|
||||
against the words as they appear in the document. A trailing `*` matches a word
|
||||
and its inflections (`invoice*` finds "invoice", "invoices" and "invoiced"),
|
||||
but not every longer word that starts with the same letters: `copy*` finds
|
||||
"copy" and "copies", not "copyright". For the same reason, a pattern whose text
|
||||
continues past where stemming cuts a word off cannot match at all:
|
||||
`productname` is indexed as `productnam`, so `produ*name` finds nothing.
|
||||
against the words as they appear in the document. Each literal part of a
|
||||
pattern is tried both as you typed it and in its stemmed form, so a trailing
|
||||
`*` matches a word and its inflections (`invoice*` finds "invoice", "invoices"
|
||||
and "invoiced") as well as longer words whose stored term still begins with
|
||||
what you typed (`copy*` finds "copyright" alongside "copy" and "copies").
|
||||
|
||||
It is still not a plain prefix search over the original text: where stemming
|
||||
shortens a word, a pattern that reaches past the point it was cut off matches
|
||||
nothing. `productname` is stored as `productnam`, so `produ*name` finds
|
||||
nothing, and "happiness" is stored as `happi`, so `happine*` does not find it.
|
||||
|
||||
Matching natural date keywords:
|
||||
|
||||
@@ -990,8 +994,8 @@ added:[2005-06-15T09:00:00Z to 2005-06-15T17:00:00Z]
|
||||
- An ISO date such as `2005-03-04` covers that whole day, and `2005-01` covers that whole month.
|
||||
- A month name such as `january` covers that whole month in the current year.
|
||||
- `next <weekday>` and `last <weekday>` each cover that whole day and must be quoted. A bare weekday name such as `monday` is not accepted.
|
||||
- A full timestamp such as `2005-01-01T00:00:00Z` matches that exact instant. Like the other expressions above, it has to be quoted when it stands on its own: `added:"2005-01-01T00:00:00Z"`.
|
||||
- A range takes two of the above as its bounds, for example `created:[2005 to 2009]` or `added:[2005-01-01 to 2005-01-31]`. Bounds may carry a time of day, and inside the brackets they are written without quotes.
|
||||
- A full timestamp such as `2005-01-01T00:00:00Z` matches that exact instant. Like the other expressions above, it has to be quoted when it stands on its own: `added:"2005-01-01T00:00:00Z"`. The unquoted spelling is rejected with an error rather than searched, because only part of it can be read as a date.
|
||||
- A range takes two of the above as its bounds, for example `created:[2005 to 2009]` or `added:[2005-01-01 to 2005-01-31]`. Bounds may carry a time of day. A bound is normally written without quotes; if you do quote one, use single quotes (`added:['-1 week' to now]`), because a double-quoted bound is rejected with an error.
|
||||
|
||||
!!! warning
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ from documents.search._tokenizer import paperless_text_analyzer
|
||||
from documents.search._tokenizer import stem_pattern_text
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
from whoosh_compat import PatternNormalizer
|
||||
|
||||
_registry_cache: dict[str | None, FieldRegistry] = {}
|
||||
|
||||
@@ -27,29 +27,35 @@ def _fold_normalizer(text: str) -> str:
|
||||
return ascii_fold(text.lower())
|
||||
|
||||
|
||||
def _make_pattern_normalizer(language: str | None) -> Callable[[str], str]:
|
||||
def _make_pattern_normalizer(language: str | None) -> PatternNormalizer:
|
||||
"""Build the wildcard/regex literal-run normalizer for a search language."""
|
||||
|
||||
def _pattern_normalizer(text: str) -> str:
|
||||
"""Normalize a literal run so it can match indexed terms.
|
||||
def _pattern_normalizer(text: str) -> tuple[str, ...]:
|
||||
"""Normalize a literal run into the forms a term may match.
|
||||
|
||||
TEXT index terms go through lowercase -> ascii_fold -> stem, so a
|
||||
pattern that skips stemming can never match one: "invoice*" would look
|
||||
for a term starting with "invoice" while the index holds "invoic". The
|
||||
run is therefore stemmed here too. KEYWORD fields are indexed raw and
|
||||
get _fold_normalizer instead, so their patterns stay literal.
|
||||
run is therefore offered stemmed as well. KEYWORD fields are indexed
|
||||
raw and get _fold_normalizer instead, so their patterns stay literal.
|
||||
|
||||
A stem can be longer than the fragment the user typed, though, and a
|
||||
longer prefix matches nothing, so the stem is used only when it is no
|
||||
longer than the typed run. Length is a proxy for "the stem stayed close
|
||||
to what was typed", not a guarantee of wider recall: a stem that
|
||||
substitutes rather than truncates ("copy" -> "copi") moves the pattern
|
||||
sideways instead of widening it, so "copy*" gains "copies" and loses
|
||||
"copyright". test_pattern_stemming.py pins that trade.
|
||||
Both forms are returned, as alternatives, because neither is a prefix
|
||||
of the other in general: English stemming substitutes as well as
|
||||
truncates ("copy" -> "copi"), so the stem alone loses the compounds
|
||||
the typed run reaches ("copyright") while the typed run alone loses
|
||||
the inflections the stem reaches ("copies"). whoosh-compat ORs the
|
||||
alternatives per literal run and deduplicates them, so a run the
|
||||
stemmer leaves alone costs exactly the one branch it did before.
|
||||
|
||||
Inside a bracket class the emitter calls this once per character and
|
||||
uses the answer only if it is a single one-character form; two forms
|
||||
there leave the character as typed. A stemmer does not change a lone
|
||||
character, so the two forms deduplicate to one and the class body is
|
||||
folded as before.
|
||||
"""
|
||||
folded = ascii_fold(text.lower())
|
||||
stemmed = stem_pattern_text(folded, language)
|
||||
return folded if len(stemmed) > len(folded) else stemmed
|
||||
return (folded, stemmed)
|
||||
|
||||
return _pattern_normalizer
|
||||
|
||||
|
||||
@@ -4,8 +4,9 @@ Every query here appears verbatim, or as a direct paraphrase, in the
|
||||
"Document searches" section of ``docs/usage.md``. Each case indexes real
|
||||
documents and asserts on matched document IDs rather than on the parsed
|
||||
query, because a query that parses cleanly is not necessarily a query that
|
||||
means what the documentation says it means: ``added:now - 3 days`` parses
|
||||
without a single diagnostic and then matches nothing.
|
||||
means what the documentation says it means: ``added:now`` parses without a
|
||||
single diagnostic and then matches nothing, because it resolves to an
|
||||
instant rather than to a span.
|
||||
|
||||
The negative cases matter as much as the positive ones. They pin the
|
||||
behaviours the docs explicitly warn about, so that if any of them ever
|
||||
@@ -25,6 +26,7 @@ import time_machine
|
||||
from documents.models import Document
|
||||
from documents.models import Note
|
||||
from documents.models import Tag
|
||||
from documents.search._errors import InvalidDateQuery
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Generator
|
||||
@@ -239,6 +241,9 @@ class TestDocumentedDateForms:
|
||||
# spelling is pinned as a non-match below.
|
||||
('added:"2005-03-04T15:30:00Z"', "old"),
|
||||
("added:[2005-03-04T09:00:00Z to 2005-03-04T17:00:00Z]", "old"),
|
||||
# A quoted range bound works when the quotes are single ones; the
|
||||
# double-quoted spelling is pinned as an error below.
|
||||
("added:['2005-03-04' to 2005-03-05]", "old"),
|
||||
],
|
||||
)
|
||||
def test_documented_date_form_matches_its_day_or_month(
|
||||
@@ -264,15 +269,6 @@ class TestDocumentedDateForms:
|
||||
# of the resulting range, not the way the value is delimited.
|
||||
'added:"now"',
|
||||
'added:"midnight"',
|
||||
'added:"-3 days"',
|
||||
'added:"-1 week"',
|
||||
# The trap: this reports no diagnostics but parses as
|
||||
# And(added:now, "3", "days") - added:now plus stray text.
|
||||
"added:now - 3 days",
|
||||
# The bare, unquoted spelling of a full timestamp. The quoted and
|
||||
# range-bound spellings above do work and match this fixture's
|
||||
# document; only this one silently degrades to a non-match.
|
||||
"added:2005-03-04T15:30:00Z",
|
||||
],
|
||||
)
|
||||
def test_forms_the_docs_warn_about_match_nothing(
|
||||
@@ -282,3 +278,36 @@ class TestDocumentedDateForms:
|
||||
query: str,
|
||||
) -> None:
|
||||
assert _matched_ids(backend, query) == set()
|
||||
|
||||
def test_bare_timestamp_is_rejected_rather_than_matching_nothing(
|
||||
self,
|
||||
backend: TantivyBackend,
|
||||
dated: dict[str, int],
|
||||
) -> None:
|
||||
"""The bare, unquoted spelling of a full timestamp. The quoted and
|
||||
range-bound spellings pinned above do work and match this fixture's
|
||||
document; this one is a user-fixable error rather than an empty
|
||||
result set, so the docs tell the user to quote it.
|
||||
|
||||
The reported value is the prefix the date grammar could consume, not
|
||||
the whole of what the user typed: paperless's own message, not this
|
||||
value, is what has to carry the "quote it" guidance.
|
||||
"""
|
||||
with pytest.raises(InvalidDateQuery) as exc_info:
|
||||
_matched_ids(backend, "added:2005-03-04T15:30:00Z")
|
||||
assert exc_info.value.field == "added"
|
||||
assert exc_info.value.value == "2005-03-"
|
||||
|
||||
def test_double_quoted_range_bound_is_rejected(
|
||||
self,
|
||||
backend: TantivyBackend,
|
||||
dated: dict[str, int],
|
||||
) -> None:
|
||||
"""Quoting a range bound is allowed, but only with single quotes: the
|
||||
double-quoted spelling reaches the date grammar with its quotes still
|
||||
attached and is not a recognizable date. The docs say so, so pin which
|
||||
of the two quote characters is the one that fails.
|
||||
"""
|
||||
with pytest.raises(InvalidDateQuery) as exc_info:
|
||||
_matched_ids(backend, 'added:["2005-03-04" to 2005-03-05]')
|
||||
assert exc_info.value.value == '"2005-03-04"'
|
||||
|
||||
@@ -17,9 +17,8 @@ from documents.models import Document
|
||||
from documents.search._registry import get_field_registry
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from whoosh_compat import FieldRegistry
|
||||
from whoosh_compat import PatternNormalizer
|
||||
|
||||
from documents.search._backend import TantivyBackend
|
||||
|
||||
@@ -29,7 +28,7 @@ CEDEF00D = "cedef00ddeadbeef0123456789abcdef01234567"
|
||||
CEDEDEAD = "cededeadbeef567801234567" + "89abcdef01234567"
|
||||
|
||||
|
||||
def _normalizer(registry: FieldRegistry, name: str) -> Callable[[str], str]:
|
||||
def _normalizer(registry: FieldRegistry, name: str) -> PatternNormalizer:
|
||||
ref = registry.make_ref(name)
|
||||
assert ref is not None
|
||||
resolved = registry.resolve(ref)
|
||||
@@ -48,12 +47,17 @@ class TestKeywordPatternNormalizer:
|
||||
],
|
||||
)
|
||||
def test_keyword_runs_are_folded_not_stemmed(self, run: str) -> None:
|
||||
"""One form, the run as typed: a KEYWORD pattern must never be widened
|
||||
to a stem, which would return checksums that do not start with what
|
||||
the user typed."""
|
||||
normalize = _normalizer(get_field_registry("en"), "checksum")
|
||||
assert normalize(run) == run
|
||||
|
||||
def test_text_runs_are_still_stemmed(self) -> None:
|
||||
def test_text_runs_still_offer_their_stem(self) -> None:
|
||||
"""A TEXT field offers the stem alongside the typed run, so a term
|
||||
matching either one is reachable."""
|
||||
normalize = _normalizer(get_field_registry("en"), "title")
|
||||
assert normalize("Running") == "run"
|
||||
assert tuple(normalize("Running")) == ("running", "run")
|
||||
|
||||
|
||||
class TestChecksumPrefixQueries:
|
||||
|
||||
@@ -20,6 +20,8 @@ from documents.search._tokenizer import paperless_text_analyzer
|
||||
from documents.search._tokenizer import stem_pattern_text
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from whoosh_compat import PatternNormalizer
|
||||
|
||||
from documents.search._backend import TantivyBackend
|
||||
|
||||
pytestmark = [pytest.mark.search, pytest.mark.django_db]
|
||||
@@ -82,9 +84,10 @@ class TestPrefixStemming:
|
||||
indexed_doc: Document,
|
||||
query: str,
|
||||
) -> None:
|
||||
"""A partial prefix keeps matching: "librar" stems to "librari", which
|
||||
is longer than what was typed, so the typed run is kept. Using the
|
||||
shorter of the two widens recall rather than failing closed."""
|
||||
"""A partial prefix keeps matching. "librar" stems to "librari", which
|
||||
is longer than what was typed and so matches no term on its own, but
|
||||
the run is offered as typed alongside its stem and that form reaches
|
||||
"librari" in the index."""
|
||||
assert _matched_ids(backend, query) == {indexed_doc.id}
|
||||
|
||||
def test_pattern_past_the_stem_boundary_is_documented_not_fixed(
|
||||
@@ -97,20 +100,17 @@ class TestPrefixStemming:
|
||||
is deliberate, not accidental."""
|
||||
assert _matched_ids(backend, "produ*name") == set()
|
||||
|
||||
def test_stem_substitution_loses_compounds_accepted_trade(
|
||||
def test_stem_substitution_reaches_both_the_inflection_and_the_compound(
|
||||
self,
|
||||
backend: TantivyBackend,
|
||||
indexed_doc: Document,
|
||||
) -> None:
|
||||
"""English stemming substitutes as well as truncates: "copy" and
|
||||
"copies" both index as "copi", while "copyright" keeps its literal "y".
|
||||
So "copy*" reaches the base word and its inflections but no longer
|
||||
reaches the compound, which it did before patterns were stemmed. That
|
||||
trade is deliberate: the same substitution is what makes "company*" and
|
||||
"library*" work at all, and no rule over one normalized string tells the
|
||||
two apart. Matching both would need the pattern to be emitted as a
|
||||
disjunction of the folded and stemmed forms, which belongs in the
|
||||
emitter, not here.
|
||||
Neither form is a prefix of the other, so no single normalized string
|
||||
reaches both. The run is therefore emitted as a disjunction of the
|
||||
folded and stemmed forms, and "copy*" reaches the base word, its
|
||||
inflections and the compound alike.
|
||||
"""
|
||||
compound = Document.objects.create(
|
||||
title="Copyright notice",
|
||||
@@ -120,7 +120,7 @@ class TestPrefixStemming:
|
||||
)
|
||||
backend.add_or_update(compound)
|
||||
|
||||
assert _matched_ids(backend, "copy*") == {indexed_doc.id}
|
||||
assert _matched_ids(backend, "copy*") == {indexed_doc.id, compound.id}
|
||||
assert _matched_ids(backend, "copyright*") == {compound.id}
|
||||
|
||||
|
||||
@@ -143,34 +143,67 @@ class TestStemsMatchTheIndexAnalyzer:
|
||||
assert stem_pattern_text(ascii_fold(word.lower()), language) == indexed
|
||||
|
||||
|
||||
def _forms(normalize: PatternNormalizer, text: str) -> tuple[str, ...]:
|
||||
"""The distinct forms a term may match, in order, the way the emitter reads
|
||||
the normalizer's answer (see whoosh_compat.PatternNormalizer)."""
|
||||
result = normalize(text)
|
||||
if isinstance(result, str):
|
||||
return (result,)
|
||||
return tuple(dict.fromkeys(result))
|
||||
|
||||
|
||||
class TestPatternNormalizer:
|
||||
@pytest.mark.parametrize(
|
||||
("text", "expected"),
|
||||
[
|
||||
("Invoice", "invoic"),
|
||||
("companies", "compani"),
|
||||
# y -> i: same length as typed, and the index only holds the stem
|
||||
("library", "librari"),
|
||||
("invoic", "invoic"),
|
||||
("Universit", "universit"),
|
||||
("Café", "cafe"),
|
||||
("Invoice", ("invoice", "invoic")),
|
||||
("companies", ("companies", "compani")),
|
||||
# y -> i is a substitution, so both forms are needed: the index
|
||||
# holds "librari" for "library" and "library" for "librarian".
|
||||
("library", ("library", "librari")),
|
||||
# A run the stemmer leaves alone collapses back to one form, so it
|
||||
# costs exactly the one regex branch it did before.
|
||||
("invoic", ("invoic",)),
|
||||
("Universit", ("universit",)),
|
||||
("Café", ("cafe",)),
|
||||
],
|
||||
)
|
||||
def test_shorter_of_the_typed_run_and_its_stem(
|
||||
def test_offers_the_typed_run_and_its_stem(
|
||||
self,
|
||||
text: str,
|
||||
expected: str,
|
||||
expected: tuple[str, ...],
|
||||
) -> None:
|
||||
assert _make_pattern_normalizer("en")(text) == expected
|
||||
assert _forms(_make_pattern_normalizer("en"), text) == expected
|
||||
|
||||
def test_run_that_yields_no_token_falls_back_to_the_typed_run(self) -> None:
|
||||
"""A run past the remove_long limit analyzes to zero tokens, so there is
|
||||
no stem to substitute and the folded run is used as typed."""
|
||||
no stem to offer and only the folded run remains."""
|
||||
over_long = "invoices" * 20
|
||||
assert _make_pattern_normalizer("en")(over_long) == over_long
|
||||
assert _forms(_make_pattern_normalizer("en"), over_long) == (over_long,)
|
||||
|
||||
@pytest.mark.parametrize("language", [None, "klingon"])
|
||||
def test_unstemmed_language_folds_only(self, language: str | None) -> None:
|
||||
"""With no stemmer configured, or one this build has no stemmer for, the
|
||||
index holds surface forms and the pattern must keep them too."""
|
||||
assert _make_pattern_normalizer(language)("Invoices") == "invoices"
|
||||
assert _forms(_make_pattern_normalizer(language), "Invoices") == ("invoices",)
|
||||
|
||||
@pytest.mark.parametrize("char", ["a", "Z", "é"])
|
||||
def test_a_single_character_collapses_to_one_folded_form(self, char: str) -> None:
|
||||
"""A bracket class body is normalized one character at a time and the
|
||||
answer is used only when it is a single one-character form, so a
|
||||
stemmer that changed a lone character would silently disable folding
|
||||
inside classes."""
|
||||
forms = _forms(_make_pattern_normalizer("en"), char)
|
||||
assert len(forms) == 1
|
||||
assert len(forms[0]) == 1
|
||||
|
||||
|
||||
class TestBracketClassStillFolds:
|
||||
def test_class_body_matches_case_insensitively(
|
||||
self,
|
||||
backend: TantivyBackend,
|
||||
indexed_doc: Document,
|
||||
) -> None:
|
||||
"""The class body is folded per character, which the alternatives
|
||||
contract preserves only because a lone character stems to itself."""
|
||||
assert _matched_ids(backend, "title:[IP]nvoice*") == {indexed_doc.id}
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
from collections.abc import Sequence
|
||||
|
||||
import pytest
|
||||
from whoosh_compat import FieldKind
|
||||
from whoosh_compat import FieldRegistry
|
||||
@@ -20,6 +22,15 @@ def _resolve(registry: FieldRegistry, name: str) -> ResolvedField:
|
||||
return resolved
|
||||
|
||||
|
||||
def _distinct_forms(result: str | Sequence[str]) -> tuple[str, ...]:
|
||||
"""The forms a term may match, in order, the way whoosh-compat's emitter
|
||||
reads a pattern_normalizer's answer: a bare str is one form, a sequence is
|
||||
several, deduplicated."""
|
||||
if isinstance(result, str):
|
||||
return (result,)
|
||||
return tuple(dict.fromkeys(result))
|
||||
|
||||
|
||||
class TestFieldRegistry:
|
||||
def test_internal_id_fields_are_not_registered(
|
||||
self,
|
||||
@@ -103,16 +114,22 @@ class TestFieldRegistry:
|
||||
self,
|
||||
registry: FieldRegistry,
|
||||
) -> None:
|
||||
# Index terms are stemmed, so patterns are too, using the registry's
|
||||
# own language: "Running" has to reach the indexed "run". Without a
|
||||
# language the index holds surface forms, so it only case/accent-folds.
|
||||
# Index terms are stemmed, so patterns offer their stem too, using the
|
||||
# registry's own language: "Running" has to reach the indexed "run".
|
||||
# Without a language the index holds surface forms, so there is no
|
||||
# second form and the run is only case/accent-folded.
|
||||
resolved = _resolve(registry, "title")
|
||||
assert resolved.spec.pattern_normalizer is not None
|
||||
assert resolved.spec.pattern_normalizer("Running") == "running"
|
||||
assert _distinct_forms(resolved.spec.pattern_normalizer("Running")) == (
|
||||
"running",
|
||||
)
|
||||
|
||||
resolved_en = _resolve(get_field_registry("en"), "title")
|
||||
assert resolved_en.spec.pattern_normalizer is not None
|
||||
assert resolved_en.spec.pattern_normalizer("Running") == "run"
|
||||
assert _distinct_forms(resolved_en.spec.pattern_normalizer("Running")) == (
|
||||
"running",
|
||||
"run",
|
||||
)
|
||||
|
||||
def test_registry_is_cached_per_language(self) -> None:
|
||||
a = get_field_registry("en")
|
||||
|
||||
@@ -1,26 +1,23 @@
|
||||
"""Reversed date ranges: an internal inconsistency between relative and
|
||||
absolute bounds, pinned exactly as measured rather than "fixed" here.
|
||||
"""Reversed date ranges swap their bounds back into order.
|
||||
|
||||
Measured end to end (FROZEN_NOW = 2026-06-15T12:00:00Z):
|
||||
|
||||
added:[now-1h to now+1h] -> 2h window, correct order
|
||||
added:[now+1h to now-1h] -> ~22h window, DAY-BUMPED, not swapped
|
||||
added:[now+1h to now-1h] -> the same 2h window, SWAPPED
|
||||
added:[2020-01-01 to 2019-01-01] -> 366 days, SWAPPED to the forward order
|
||||
added:[2019-01-01 to 2020-01-01] -> 366 days (same result either way)
|
||||
|
||||
Absolute reversed ranges swap their bounds back into order, matching
|
||||
whoosh's own behavior. Relative (``now±``) reversed ranges do not swap --
|
||||
whoosh-compat's date grammar instead adds a day to the upper bound,
|
||||
producing a much wider window than either the forward or a swapped
|
||||
reading would give. This is a library-level inconsistency between the two
|
||||
range kinds, not an application-level rewrite paperless performs (there is
|
||||
no reversed-range handling in documents/search/_query.py), so it is not
|
||||
"fixed" here: fixing it belongs in whoosh-compat's date grammar, not in a
|
||||
pre-parse rewrite on this side (see also the CJK/pre-parse-rewrites
|
||||
docstrings elsewhere in this suite for the same discipline). Pinned as a
|
||||
known inconsistency and a library follow-up. When whoosh-compat's date
|
||||
grammar is fixed to swap consistently, the relative case's test below
|
||||
inverts -- that inversion is the intended, visible signal.
|
||||
Both range kinds behave the same way, matching whoosh's own behavior.
|
||||
Relative (``now±``) reversed ranges used to differ: whoosh-compat's date
|
||||
grammar added a day to the upper bound instead of swapping, producing a
|
||||
much wider window than either the forward or a swapped reading gives. That
|
||||
was a library-level inconsistency between the two range kinds, not an
|
||||
application-level rewrite paperless performs (there is still no
|
||||
reversed-range handling in documents/search/_query.py), and it was fixed in
|
||||
whoosh-compat's date grammar rather than in a pre-parse rewrite on this
|
||||
side (see also the CJK/pre-parse-rewrites docstrings elsewhere in this
|
||||
suite for the same discipline). Both cases are pinned below so a future
|
||||
divergence between them is visible again.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -52,13 +49,13 @@ def _index(backend: TantivyBackend, **kwargs: object) -> Document:
|
||||
return doc
|
||||
|
||||
|
||||
class TestRelativeReversedRangeDayBumpsInsteadOfSwapping:
|
||||
class TestRelativeReversedRangeSwaps:
|
||||
@pytest.fixture
|
||||
def docs(self, backend: TantivyBackend) -> dict[str, int]:
|
||||
with time_machine.travel(FROZEN_NOW, tick=False):
|
||||
return {
|
||||
# Inside the correct (forward) 2h window [11:00, 13:00],
|
||||
# outside the day-bumped window [13:00, next-day 11:00).
|
||||
# Inside the 2h window [11:00, 13:00] that both the forward
|
||||
# and the reversed spelling resolve to.
|
||||
"in_forward_window": _index(
|
||||
backend,
|
||||
title="Forward window doc",
|
||||
@@ -66,13 +63,14 @@ class TestRelativeReversedRangeDayBumpsInsteadOfSwapping:
|
||||
checksum="reversed-relative-forward",
|
||||
added=FROZEN_NOW,
|
||||
).pk,
|
||||
# Outside the correct 2h window, inside the day-bumped
|
||||
# window the reversed query actually produces.
|
||||
"in_daybumped_window": _index(
|
||||
# Outside that window, but inside the wider window the
|
||||
# reversed spelling produced when it day-bumped instead of
|
||||
# swapping. It must not match either query now.
|
||||
"outside_the_window": _index(
|
||||
backend,
|
||||
title="Day-bumped window doc",
|
||||
title="Later same-week doc",
|
||||
content="x",
|
||||
checksum="reversed-relative-daybumped",
|
||||
checksum="reversed-relative-outside",
|
||||
added=datetime(2026, 6, 16, 8, 0, tzinfo=UTC),
|
||||
).pk,
|
||||
}
|
||||
@@ -87,18 +85,16 @@ class TestRelativeReversedRangeDayBumpsInsteadOfSwapping:
|
||||
docs["in_forward_window"],
|
||||
}
|
||||
|
||||
def test_reversed_range_day_bumps_rather_than_swapping(
|
||||
def test_reversed_range_swaps_to_the_same_window(
|
||||
self,
|
||||
backend: TantivyBackend,
|
||||
docs: dict[str, int],
|
||||
) -> None:
|
||||
# If this swapped like the absolute case below, it would match
|
||||
# "in_forward_window" (the same set as the forward query). It
|
||||
# instead matches only the day-bumped document -- the documented
|
||||
# library inconsistency.
|
||||
# The same set as the forward query, and in particular not the
|
||||
# document that only a day-bumped upper bound would have reached.
|
||||
with time_machine.travel(FROZEN_NOW, tick=False):
|
||||
assert _matched_ids(backend, "added:[now+1h to now-1h]") == {
|
||||
docs["in_daybumped_window"],
|
||||
docs["in_forward_window"],
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user