mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-08-27 05:03:20 +00:00
docs(search): state the wildcard rule the alternatives actually implement
The wildcard caveat still described the pre-alternatives world: "where stemming shortens a word, a pattern that reaches past the point it was cut off matches nothing". Measured against a real index that is false for most of the set it ranges over. university*, companie*, happiness* and universities* all match, because the stem of the typed run is offered as a second branch. Only a fragment landing strictly between the stem and the surface form fails, which is what both of the examples happened to be, so the sentence read as convincing. A user who believed it would type a shorter fragment, which genuinely finds nothing, instead of the full word, which works. The rule stated now is the measured one: a trailing * matches a stored term when either the run as typed or its stem is a prefix of that term. Both branches are load-bearing, verified one word per document. copy* reaches "copies" only via the stem and "copyright" only via the surface form, so a rule naming just the stem would predict the wrong answer for half of it. Also corrects two more claims that were each defensible alone and wrong together, both measured before rewriting: - checksum patterns are lowercased even though checksum terms are stored verbatim, so checksum:9F86D081* does match; "matched exactly as typed and nothing else" told the reader otherwise. Pinned as a positive case beside the uppercase *term* that really does fail. - relative offsets and long-form dates must be quoted only as a value standing alone; added:[-1 week to now] works unquoted and returns the same documents as the quoted spelling. test_pattern_stemming.py's partial-prefix docstring was fabricated: it said "librar" stems to the longer "librari" and so matches nothing on its own. Measured, the stemmer leaves "librar" alone, "univers" stems to the shorter "univ", and librari* does match this file's own fixture. Rewritten against measured values, and renamed, since a future editor reasoning from it would have believed these params exercise the two-alternative path. They do not, so a regression breaking that path would have left them green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
8f371e39ae
commit
a65efd3e4c
+13
-6
@@ -933,7 +933,7 @@ original_filename:invoice.pdf
|
||||
- `asn` matches a document's Archive Serial Number.
|
||||
- `page_count` matches a document's page count.
|
||||
- `num_notes` matches how many notes a document has.
|
||||
- `checksum` matches the checksum of the original document file (not the archived/processed version). Unlike the text fields, this one is stored verbatim rather than tokenized, so only a complete, lowercase checksum matches. To search by the first few characters instead, use a wildcard: `checksum:9f86d081*`. Wildcard patterns on the text fields are also tried stemmed, to line up with the stemmed index, but `checksum` is indexed without stemming, so its patterns are matched exactly as typed and nothing else.
|
||||
- `checksum` matches the checksum of the original document file (not the archived/processed version). Unlike the text fields, this one is stored verbatim rather than tokenized, so only a complete, lowercase checksum matches. To search by the first few characters instead, use a wildcard: `checksum:9f86d081*`. Wildcard patterns on the text fields are also tried stemmed, to line up with the stemmed index, but `checksum` is indexed without stemming, so its patterns are not stemmed either: a wildcard prefix is matched literally, apart from being lowercased first. `checksum:9F86D081*` therefore does find the document, even though the plain uppercase term does not.
|
||||
- `original_filename` matches the filename of the document as originally consumed.
|
||||
|
||||
`asn`, `page_count` and `num_notes` are numeric and also accept ranges, for example `asn:[50 to 150]`.
|
||||
@@ -952,10 +952,15 @@ pattern is tried both as you typed it and in its stemmed form, so a trailing
|
||||
and "invoiced") as well as longer words whose stored term still begins with
|
||||
what you typed (`copy*` finds "copyright" alongside "copy" and "copies").
|
||||
|
||||
It is still not a plain prefix search over the original text: where stemming
|
||||
shortens a word, a pattern that reaches past the point it was cut off matches
|
||||
nothing. `productname` is stored as `productnam`, so `produ*name` finds
|
||||
nothing, and "happiness" is stored as `happi`, so `happine*` does not find it.
|
||||
It is still not a plain prefix search over the original text. A trailing `*`
|
||||
matches a stored term when either the run you typed or its stemmed form is a
|
||||
prefix of that term, so a fragment that stops part-way between the two matches
|
||||
neither: `universities*` finds "university" and "universities", which are both
|
||||
stored as `univers`, while the shorter `universit*` finds nothing at all. For
|
||||
the same reason `happine*` does not find "happiness", which is stored as
|
||||
`happi`. And a pattern that requires letters after the wildcard which stemming
|
||||
has removed cannot match either: `productname` is stored as `productnam`, so
|
||||
`produ*name` finds nothing.
|
||||
|
||||
Matching natural date keywords:
|
||||
|
||||
@@ -963,7 +968,9 @@ The multi-word date keywords listed below work quoted or unquoted after a
|
||||
date field (`added:"previous month"` and `added:previous month` are
|
||||
equivalent); elsewhere in a query the same words are treated as ordinary
|
||||
search text. Other date expressions the parser accepts (relative offsets
|
||||
like `-1 week`, or specific dates like `12 december 2019`) must be quoted.
|
||||
like `-1 week`, or specific dates like `12 december 2019`) must be quoted when
|
||||
they stand alone as a value; inside a range's brackets they work unquoted, as
|
||||
in `added:[-1 week to now]`.
|
||||
|
||||
```
|
||||
added:today
|
||||
|
||||
@@ -182,6 +182,11 @@ class TestArchiveMetadataFields:
|
||||
"original_filename:invoice.pdf",
|
||||
f"checksum:{DOC_CHECKSUM}",
|
||||
"checksum:9f86d081*",
|
||||
# A checksum term is stored verbatim, but a checksum *pattern* is
|
||||
# lowercased before it is matched, which the docs now say outright
|
||||
# next to the "only a complete, lowercase checksum matches" rule
|
||||
# that the uppercase term in the negative list below pins.
|
||||
"checksum:9F86D081*",
|
||||
],
|
||||
)
|
||||
def test_documented_metadata_query_matches(
|
||||
|
||||
@@ -78,18 +78,42 @@ class TestPrefixStemming:
|
||||
assert _matched_ids(backend, query) == {indexed_doc.id}
|
||||
|
||||
@pytest.mark.parametrize("query", ["univers*", "librar*"])
|
||||
def test_partial_prefix_is_not_lengthened_by_its_stem(
|
||||
def test_partial_prefix_reaches_the_stemmed_term(
|
||||
self,
|
||||
backend: TantivyBackend,
|
||||
indexed_doc: Document,
|
||||
query: str,
|
||||
) -> None:
|
||||
"""A partial prefix keeps matching. "librar" stems to "librari", which
|
||||
is longer than what was typed and so matches no term on its own, but
|
||||
the run is offered as typed alongside its stem and that form reaches
|
||||
"librari" in the index."""
|
||||
"""A prefix shorter than a whole word still matches, and neither of
|
||||
these needs the two-alternative path to do it.
|
||||
|
||||
Measured under "en": the stemmer leaves "librar" alone, so it has one
|
||||
form, and that form is a prefix of the "librari" the index holds for
|
||||
"library". "univers" stems to the *shorter* "univ", and the run as
|
||||
typed and its stem are both prefixes of the "univers" the index holds
|
||||
for "university". The case where the two forms genuinely diverge, and
|
||||
only one of them matches, is
|
||||
test_stem_substitution_reaches_both_the_inflection_and_the_compound.
|
||||
"""
|
||||
assert _matched_ids(backend, query) == {indexed_doc.id}
|
||||
|
||||
def test_full_word_reaches_the_stem_but_a_fragment_of_it_does_not(
|
||||
self,
|
||||
backend: TantivyBackend,
|
||||
indexed_doc: Document,
|
||||
) -> None:
|
||||
"""The alternatives widen recall without turning a wildcard into a
|
||||
prefix search over the original text.
|
||||
|
||||
"university" is stored as "univers". The stem of "universities" is
|
||||
that same "univers", so the longer word matches; "universit" is a
|
||||
prefix of neither its own stem nor the stored term, so the *shorter*
|
||||
fragment matches nothing. usage.md names this pair, so a reader told
|
||||
that `universit*` fails is also told which spelling works.
|
||||
"""
|
||||
assert _matched_ids(backend, "universities*") == {indexed_doc.id}
|
||||
assert _matched_ids(backend, "universit*") == set()
|
||||
|
||||
def test_pattern_past_the_stem_boundary_is_documented_not_fixed(
|
||||
self,
|
||||
backend: TantivyBackend,
|
||||
|
||||
Reference in New Issue
Block a user