docs(search): state the wildcard rule the alternatives actually implement

The wildcard caveat still described the pre-alternatives world: "where
stemming shortens a word, a pattern that reaches past the point it was cut
off matches nothing". Measured against a real index that is false for most
of the set it ranges over. university*, companie*, happiness* and
universities* all match, because the stem of the typed run is offered as a
second branch. Only a fragment landing strictly between the stem and the
surface form fails, which is what both of the examples happened to be, so
the sentence read as convincing. A user who believed it would type a
shorter fragment, which genuinely finds nothing, instead of the full word,
which works.

The rule stated now is the measured one: a trailing * matches a stored term
when either the run as typed or its stem is a prefix of that term. Both
branches are load-bearing, verified one word per document. copy* reaches
"copies" only via the stem and "copyright" only via the surface form, so a
rule naming just the stem would predict the wrong answer for half of it.

Also corrects two more claims that were each defensible alone and wrong
together, both measured before rewriting:

- checksum patterns are lowercased even though checksum terms are stored
  verbatim, so checksum:9F86D081* does match; "matched exactly as typed and
  nothing else" told the reader otherwise. Pinned as a positive case beside
  the uppercase *term* that really does fail.
- relative offsets and long-form dates must be quoted only as a value
  standing alone; added:[-1 week to now] works unquoted and returns the same
  documents as the quoted spelling.

test_pattern_stemming.py's partial-prefix docstring was fabricated: it said
"librar" stems to the longer "librari" and so matches nothing on its own.
Measured, the stemmer leaves "librar" alone, "univers" stems to the shorter
"univ", and librari* does match this file's own fixture. Rewritten against
measured values, and renamed, since a future editor reasoning from it would
have believed these params exercise the two-alternative path. They do not,
so a regression breaking that path would have left them green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
stumpylog
2026-08-20 16:37:38 -07:00
co-authored by Claude Opus 5
parent 8f371e39ae
commit a65efd3e4c
3 changed files with 47 additions and 11 deletions
+13 -6
View File
@@ -933,7 +933,7 @@ original_filename:invoice.pdf
- `asn` matches a document's Archive Serial Number.
- `page_count` matches a document's page count.
- `num_notes` matches how many notes a document has.
- `checksum` matches the checksum of the original document file (not the archived/processed version). Unlike the text fields, this one is stored verbatim rather than tokenized, so only a complete, lowercase checksum matches. To search by the first few characters instead, use a wildcard: `checksum:9f86d081*`. Wildcard patterns on the text fields are also tried stemmed, to line up with the stemmed index, but `checksum` is indexed without stemming, so its patterns are matched exactly as typed and nothing else.
- `checksum` matches the checksum of the original document file (not the archived/processed version). Unlike the text fields, this one is stored verbatim rather than tokenized, so only a complete, lowercase checksum matches. To search by the first few characters instead, use a wildcard: `checksum:9f86d081*`. Wildcard patterns on the text fields are also tried stemmed, to line up with the stemmed index, but `checksum` is indexed without stemming, so its patterns are not stemmed either: a wildcard prefix is matched literally, apart from being lowercased first. `checksum:9F86D081*` therefore does find the document, even though the plain uppercase term does not.
- `original_filename` matches the filename of the document as originally consumed.
`asn`, `page_count` and `num_notes` are numeric and also accept ranges, for example `asn:[50 to 150]`.
@@ -952,10 +952,15 @@ pattern is tried both as you typed it and in its stemmed form, so a trailing
and "invoiced") as well as longer words whose stored term still begins with
what you typed (`copy*` finds "copyright" alongside "copy" and "copies").
It is still not a plain prefix search over the original text: where stemming
shortens a word, a pattern that reaches past the point it was cut off matches
nothing. `productname` is stored as `productnam`, so `produ*name` finds
nothing, and "happiness" is stored as `happi`, so `happine*` does not find it.
It is still not a plain prefix search over the original text. A trailing `*`
matches a stored term when either the run you typed or its stemmed form is a
prefix of that term, so a fragment that stops part-way between the two matches
neither: `universities*` finds "university" and "universities", which are both
stored as `univers`, while the shorter `universit*` finds nothing at all. For
the same reason `happine*` does not find "happiness", which is stored as
`happi`. And a pattern that requires letters after the wildcard which stemming
has removed cannot match either: `productname` is stored as `productnam`, so
`produ*name` finds nothing.
Matching natural date keywords:
@@ -963,7 +968,9 @@ The multi-word date keywords listed below work quoted or unquoted after a
date field (`added:"previous month"` and `added:previous month` are
equivalent); elsewhere in a query the same words are treated as ordinary
search text. Other date expressions the parser accepts (relative offsets
like `-1 week`, or specific dates like `12 december 2019`) must be quoted.
like `-1 week`, or specific dates like `12 december 2019`) must be quoted when
they stand alone as a value; inside a range's brackets they work unquoted, as
in `added:[-1 week to now]`.
```
added:today
@@ -182,6 +182,11 @@ class TestArchiveMetadataFields:
"original_filename:invoice.pdf",
f"checksum:{DOC_CHECKSUM}",
"checksum:9f86d081*",
# A checksum term is stored verbatim, but a checksum *pattern* is
# lowercased before it is matched, which the docs now say outright
# next to the "only a complete, lowercase checksum matches" rule
# that the uppercase term in the negative list below pins.
"checksum:9F86D081*",
],
)
def test_documented_metadata_query_matches(
@@ -78,18 +78,42 @@ class TestPrefixStemming:
assert _matched_ids(backend, query) == {indexed_doc.id}
@pytest.mark.parametrize("query", ["univers*", "librar*"])
def test_partial_prefix_is_not_lengthened_by_its_stem(
def test_partial_prefix_reaches_the_stemmed_term(
self,
backend: TantivyBackend,
indexed_doc: Document,
query: str,
) -> None:
"""A partial prefix keeps matching. "librar" stems to "librari", which
is longer than what was typed and so matches no term on its own, but
the run is offered as typed alongside its stem and that form reaches
"librari" in the index."""
"""A prefix shorter than a whole word still matches, and neither of
these needs the two-alternative path to do it.
Measured under "en": the stemmer leaves "librar" alone, so it has one
form, and that form is a prefix of the "librari" the index holds for
"library". "univers" stems to the *shorter* "univ", and the run as
typed and its stem are both prefixes of the "univers" the index holds
for "university". The case where the two forms genuinely diverge, and
only one of them matches, is
test_stem_substitution_reaches_both_the_inflection_and_the_compound.
"""
assert _matched_ids(backend, query) == {indexed_doc.id}
def test_full_word_reaches_the_stem_but_a_fragment_of_it_does_not(
self,
backend: TantivyBackend,
indexed_doc: Document,
) -> None:
"""The alternatives widen recall without turning a wildcard into a
prefix search over the original text.
"university" is stored as "univers". The stem of "universities" is
that same "univers", so the longer word matches; "universit" is a
prefix of neither its own stem nor the stored term, so the *shorter*
fragment matches nothing. usage.md names this pair, so a reader told
that `universit*` fails is also told which spelling works.
"""
assert _matched_ids(backend, "universities*") == {indexed_doc.id}
assert _matched_ids(backend, "universit*") == set()
def test_pattern_past_the_stem_boundary_is_documented_not_fixed(
self,
backend: TantivyBackend,