User/DocumentType/StoragePath were imported inside individual test bodies
despite the module already importing documents.models at top level -
nothing here needed deferred import. Also add an _index() helper
(Document.objects.create + backend.add_or_update in one call) for the many
sites where nothing needs to happen between creating a document and
indexing it; the two-step ceremony was outweighing the fixture data at
every call site. Left as two explicit steps wherever a Note or
CustomFieldInstance genuinely has to be attached before indexing.
Deletes TestCommaValueLists, TestMultitokenInNestedOr, TestRfc3339TZDateRange,
TestCreatedTimezoneInvariance, and TestReversedDateRange: none of them
exercise any paperless-specific pre/post-processing code. Comma-list AND
semantics, multitoken resolution, RFC3339 T/Z UTC math, date-only timezone
invariance, and reversed-range disambiguation are all entirely
whoosh-compat's own grammar/semantics, already covered by its own test
suite. The comma_values flag paperless does own is still covered cheaply in
test_fields.py; the date_only flag is still covered in test_registry.py.
Also trims verbose docstrings/comments across _query.py and the surviving
acceptance tests: cuts references to whoosh-compat's internal
DIVERGENCES.md entry numbers and paperless v2/Whoosh-era implementation
history down to the user-facing behavior that actually matters, without
losing the substance.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RVj8NFy821G3YhNf68PF6X
test_date_grammar_parity.py was added and deleted entirely within this
feature branch's own history; it never existed in dev. Referencing its
deletion in a docstring only makes sense while reading this branch's
intermediate commits, not once this merges - unlike PR #13010 or
whoosh-compat's DIVERGENCES.md, which are permanent, externally
verifiable references.
Four targeted additions, no production code:
The type-alias test asserted only that a query object was built, and a
naive result-level replacement turned out equally vacuous for a subtle
reason: document_type is itself a default search field, so a broken
alias resolution demoting "type:invoice" to unfielded text STILL
matches the typed document through the field value under test. Both
alias tests (type/document_type, path/storage_path) now use
discriminating decoys carrying the query word in content, so demotion
matches the decoy and fails the exact-set assertion; the old
parse-shape test is deleted.
A new schema test pins that every PublicField.fast flag equals the
built tantivy schema's per-field fast option, in both drift directions:
whoosh-compat trusts the declared flag when resolving field:* existence
checks, and build_schema() only honors it for U64 and DATE kinds, so a
future fast=True TEXT/KEYWORD/JSON entry would otherwise make those
searches silently match nothing at query time.
Two result-level date pins restore behaviors whose assertions were lost
in the test migration: a created date matches regardless of the active
timezone (the America/New_York leg is the discriminating one: a
tz-applying implementation shifts the window past the naive-midnight
indexed value), and a reversed created:[2025 TO 2020] range still
matches its span through the joint-disambiguation swap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
The TestUnregisteredIdFieldFoldsToLiteralText docstring pointed readers
at docs/usage.md's advanced-search note about the dropped *_id
prefixes, which a later commit removed. State the rationale directly
instead: the *_id fields were always internal index columns (v2
consumed them for permission filtering and its own criteria), and their
queryability as search syntax was an accident of whoosh resolving any
schema field name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
The v2 whoosh schema had plural notes/custom_fields TEXT fields (notes
indexed the joined note texts, custom_fields indexed joined
"name : value" strings), so "notes:foo" and "custom_fields:foo" were
valid fielded searches in released paperless and through the deleted
translation layer. On the whoosh-compat registry those names are JSON
fields addressable only via subpaths, and the bare spelling silently
demoted to an unfielded text search of the words themselves, matching
unrelated documents that merely contain "notes" or "custom".
parse_user_query now rewrites the bare prefixes live to the same
targets migration 0017 chose for the singular whoosh-era spellings:
notes: becomes notes.note: and custom_fields: becomes
custom_fields.value:, with 0017's lookbehind guard so subpath spellings
and words merely ending in the prefix are untouched. Prefix
substitution only; values ride through unchanged, and every value shape
lands in a documented outcome downstream (ranges, wildcards and exists
on JSON subpaths are typed errors, not crashes). The inherited
trade-off stands: custom_fields.value: drops the name-matching half of
v2's combined indexing, with custom_fields.name: available for it.
Acceptance tests pin the rewrite with decoy documents whose content
contains the literal prefix words, which the old demotion matched and
the fielded search must not, plus untouched-subpath controls.
docs/usage.md documents the bare prefixes as subpath shorthand.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
"added:previous month" returned HTTP 400 after the whoosh-compat
migration. The unquoted spelling was never parser-native anywhere: v2
rewrote it to explicit bracket ranges app-side before whoosh saw the
string, and the deleted translation layer consumed it itself, so users
and saved views have relied on it continuously while whoosh-compat
deliberately scopes it out of its parser (its DIVERGENCES.md entry 19)
and understands the phrases natively only as quoted values.
parse_user_query now quotes the closed six-phrase vocabulary (previous
week/month/quarter/year, this month/year) when it directly follows a
date field's colon, before parsing. Only quoting happens app-side; every
date computation stays in whoosh-compat's grammar, unlike v2's rewrite,
which computed the ranges itself. Date field names derive from
PUBLIC_FIELDS, the field name matches case-sensitively (the parser's own
field tagging is case-sensitive), the phrase case-insensitively (the
grammar accepts any case in the quoted form), and already-quoted
spellings, TEXT fields, unfielded words and bracketed ranges are
untouched.
The previously xfailed end-to-end regression test now passes as a plain
test, and a new acceptance class pins unquoted == quoted == mixed-case
result sets on a boundary fixture, no-error parsing for the whole
vocabulary across all three date fields, and that "title:previous month"
stays an ordinary text search. docs/usage.md now states the two
spellings are equivalent after a date field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
The fuzzy blend clause handed the raw query string to tantivy's own
parser, which rejects whoosh-only grammar (date keywords, whoosh ranges,
aliases needing resolution), so any mixed query silently lost its fuzzy
clause: a typo'd word beside "added:today" stopped matching the moment
the date keyword appeared, while the same typo without it still matched.
Before the whoosh-compat migration the parser received the translated
string, so fuzzy survived mixed queries.
The clause is now built from whoosh_compat.free_text_tokens over the
already-parsed AST: the query's free-text words, analyzed, deduplicated,
with negated terms excluded so a NOT'd word cannot resurface through the
fuzzy clause. The joined word string is always plain tokens, so tantivy
always parses it; a defensive word-character filter guards any future
field whose analyzer passes punctuation through, and the ValueError skip
remains as insurance. One chosen trade-off is documented in the
docstring: a term fielded on a default search field contributes its text
unfielded, widening fuzzy recall on the 0.1-boosted secondary clause.
Two result-level acceptance tests pin the behavior: the mixed
typo-plus-date-keyword query matches its document again, and a NOT'd
word does not fuzzy-resurface (shaped so the assertion genuinely fails
under a naive all-words implementation: the excluded word's document is
the only candidate hit, so score normalization cannot mask it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
A prior commit deleted test_query.py's parametrized "doesn't raise" coverage
for this shape (created:[...T...Z TO ...] and comma-combined ranges), which
was also the only place PR #13010's T/Z backward-compat guarantee was
exercised. Nothing in paperless's suite proved the full parse_user_query() ->
tantivy Query -> matched-document pipeline still honors it after the
whoosh-compat grammar fix (commit f936143 in the whoosh-compat repo). Add
result-level acceptance cases: an in/out-of-range T/Z bracket range, PR
#13010's original comma-combined two-field shape, and an exact-boundary case
proving a Z-suffixed bound is absolute UTC, not shifted by the local search
timezone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RVj8NFy821G3YhNf68PF6X
test_unregistered_id_field_folds_to_literal_text_not_error only checked
that parse_user_query() didn't raise for a query like tag_id:5. Add a
result-level acceptance test (matching test_acceptance.py's
_matched_ids pattern, indexed against real documents) that asserts the
matched-document-ID set is genuinely empty, not just that the parse
step succeeds.
Replaces test_query.py's intermediate-AST/query-string checks with a
result-level acceptance corpus that indexes real documents and asserts
matched-ID sets through parse_user_query(), covering the #13568
bracket-wildcard regression, comma value lists, field boosts, JSON subpaths,
and Multitoken-in-OR nesting. Removes TestCreatedDateField, TestDateTimeFields,
TestWhooshQueryRewriting, TestYearRangeRewriting, TestNonDateFieldsNotRewritten,
TestPassthrough, TestNormalizeQuery, and TestParseUserQuery's
test_advanced_search_queries_do_not_raise from test_query.py, since they test
translate_query/_dates.py internals or a diagnostics-free-parse guarantee
whoosh-compat's own suite already covers.
Trenton Holmes
