InvalidDateQuery/InvalidNumberQuery/MultipleSearchQueryErrors/
SearchQueryError are all already imported at module top; three test
bodies re-imported them locally for no reason.
Three classes in test_query.py each defined an identical query_index
fixture. None of these tests write documents to the index, so consolidate
into one module-level, module-scoped fixture (mirroring conftest.py's
index fixture rationale) instead of three copies to keep in sync.
Deliberately NOT merged with conftest.py's own index fixture: that one
registers tokenizers with "english" (stemming on), while these tests rely
on "" (stemming off) - a real behavioral difference, not incidental.
Deletes TestCommaValueLists, TestMultitokenInNestedOr, TestRfc3339TZDateRange,
TestCreatedTimezoneInvariance, and TestReversedDateRange: none of them
exercise any paperless-specific pre/post-processing code. Comma-list AND
semantics, multitoken resolution, RFC3339 T/Z UTC math, date-only timezone
invariance, and reversed-range disambiguation are all entirely
whoosh-compat's own grammar/semantics, already covered by its own test
suite. The comma_values flag paperless does own is still covered cheaply in
test_fields.py; the date_only flag is still covered in test_registry.py.
Also trims verbose docstrings/comments across _query.py and the surviving
acceptance tests: cuts references to whoosh-compat's internal
DIVERGENCES.md entry numbers and paperless v2/Whoosh-era implementation
history down to the user-facing behavior that actually matters, without
losing the substance.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RVj8NFy821G3YhNf68PF6X
Four targeted additions, no production code:
The type-alias test asserted only that a query object was built, and a
naive result-level replacement turned out equally vacuous for a subtle
reason: document_type is itself a default search field, so a broken
alias resolution demoting "type:invoice" to unfielded text STILL
matches the typed document through the field value under test. Both
alias tests (type/document_type, path/storage_path) now use
discriminating decoys carrying the query word in content, so demotion
matches the decoy and fails the exact-set assertion; the old
parse-shape test is deleted.
A new schema test pins that every PublicField.fast flag equals the
built tantivy schema's per-field fast option, in both drift directions:
whoosh-compat trusts the declared flag when resolving field:* existence
checks, and build_schema() only honors it for U64 and DATE kinds, so a
future fast=True TEXT/KEYWORD/JSON entry would otherwise make those
searches silently match nothing at query time.
Two result-level date pins restore behaviors whose assertions were lost
in the test migration: a created date matches regardless of the active
timezone (the America/New_York leg is the discriminating one: a
tz-applying implementation shifts the window past the naive-midnight
indexed value), and a reversed created:[2025 TO 2020] range still
matches its span through the joint-disambiguation swap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
Four pieces of the same surface:
whoosh-compat's emit() documents a two-part host contract: both a parse
diagnostic and the QueryEmitError/UnsupportedQueryError pair are
user-input errors. Only the latter half was caught; QueryEmitError now
maps to SearchQueryError too. Messages pass through a cleanup that
strips the library's DIVERGENCES.md references and replaces the
fast=True host-configuration advice with user language, so no
library-internal vocabulary reaches a searching user.
The bulk selection paths (bulk edit, the legacy bulk endpoint, bulk
download) reached the backend with no SearchQueryError handler, so a
bad date or number in a selection filter raised straight to a DRF 500.
They now share the search list endpoint's exact mapping (a new
search_query_error_messages helper flattens MultipleSearchQueryErrors
in one place), returning the same 400 body for the same bad query.
QueryParserError means a whoosh-compat parser bug, not user-fixable
input, per its own contract; the list endpoint's blanket handler was
converting it to a generic 400. It now re-raises and surfaces as a 500
that monitoring can see.
All behavior is pinned test-first: bulk edit and bulk download API
tests assert 400s naming the bad value (previously unhandled
exceptions), a unit test pins the QueryEmitError mapping, three
parametrized checks assert no internal vocabulary leaks for the
unsupported query shapes, and a mocked parser-bug test asserts the 500.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
- Update stale test comments in test_query.py that described string
rewriting / raw-query fallback behavior that no longer exists post
whoosh-compat migration; rename
test_date_rewriting_applied_before_tantivy_parse to
test_date_keyword_resolves_without_raising to match.
- views.py: move the local MultipleSearchQueryErrors import up into the
existing local-import block near the top of list(), consistent with
the other documents.search imports there, instead of importing it
again inside the except SearchQueryError clause.
- test_api_search.py: assert response.status_code explicitly before
indexing into response.data["results"] in
test_search_added_previous_month_excludes_next_period_start, and tie
the xfail marker to AssertionError instead of the incidental KeyError
that indexing a 400 response's missing "results" key produced.
test_unregistered_id_field_folds_to_literal_text_not_error only checked
that parse_user_query() didn't raise for a query like tag_id:5. Add a
result-level acceptance test (matching test_acceptance.py's
_matched_ids pattern, indexed against real documents) that asserts the
matched-document-ID set is genuinely empty, not just that the parse
step succeeds.
The fuzzy blend clause in parse_user_query() fed the raw, whoosh-syntax
query string directly to tantivy's own query parser. Since the
whoosh-compat migration, raw_query still contains whoosh grammar (date
keywords, whoosh-style ranges, bracket-class wildcards) that tantivy's
parser rejects with ValueError, which escaped parse_user_query and
turned into a generic HTTP 400 for the entire query whenever
ADVANCED_FUZZY_SEARCH_THRESHOLD was configured.
Deriving a clean plain-text-only extraction for the fuzzy clause was
ruled out: wc.parse() already expands unfielded terms into per-default-
field copies in the AST, so there's no "still unfielded" marker left to
walk without duplicating whoosh-compat's own expansion logic. Instead,
scope a narrow try/except ValueError around exactly the
index.parse_query() call and skip the fuzzy clause (logged at debug)
when it can't parse, leaving the exact/CJK clauses unaffected.
Replaces test_query.py's intermediate-AST/query-string checks with a
result-level acceptance corpus that indexes real documents and asserts
matched-ID sets through parse_user_query(), covering the #13568
bracket-wildcard regression, comma value lists, field boosts, JSON subpaths,
and Multitoken-in-OR nesting. Removes TestCreatedDateField, TestDateTimeFields,
TestWhooshQueryRewriting, TestYearRangeRewriting, TestNonDateFieldsNotRewritten,
TestPassthrough, TestNormalizeQuery, and TestParseUserQuery's
test_advanced_search_queries_do_not_raise from test_query.py, since they test
translate_query/_dates.py internals or a diagnostics-free-parse guarantee
whoosh-compat's own suite already covers.
whoosh-compat's date grammar now accepts "T" as a date/time separator
and a trailing "Z" UTC designator (paperless-ngx PR #13010
back-compat), so these advanced-search queries no longer raise.
Rewires parse_user_query() to parse via wc.parse()/tantivy_emit() against
the shared FieldRegistry instead of the string-based translate_query()
pipeline, so diagnostics map to typed SearchQueryError subclasses
(InvalidDateQuery/InvalidNumberQuery/MultipleSearchQueryErrors) and every
bad field is reported, not just the first.
Marks three pre-existing tests xfail (2 in test_query.py, 1 in
test_api_search.py) for confirmed whoosh-compat grammar gaps found while
verifying this rewrite: unquoted multi-word date keywords (e.g.
`added:previous month`) and RFC3339 T/Z datetime range bounds no longer
parse.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move SearchQueryError and InvalidDateQuery from _translate.py to _query.py and
add two new exception classes: InvalidNumberQuery and MultipleSearchQueryErrors.
Update _translate.py to re-export the exceptions for backward compatibility
until the translation module is removed. Update __init__.py to export all
four exception classes from _query.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix: exclude next-period start from relative date-range filters
Tantivy's [lo TO hi] range is inclusive on both ends, but computed upper
bounds (keyword ranges, YYYY/YYYYMM/YYYYMMDD tokens) represent the start of
the next period. Use half-open [lo TO hi} for those so e.g. "previous month"
no longer matches the 1st of the current month.
* Adds a regression test down to the second check for the hi range
Trenton Holmes
shamoonandGitHub
