test_date_grammar_parity.py was added and deleted entirely within this
feature branch's own history; it never existed in dev. Referencing its
deletion in a docstring only makes sense while reading this branch's
intermediate commits, not once this merges - unlike PR #13010 or
whoosh-compat's DIVERGENCES.md, which are permanent, externally
verifiable references.
The design spec, implementation plan, and dev-skill for this migration
are no longer needed now that the migration is complete and merged into
this branch.
Four targeted additions, no production code:
The type-alias test asserted only that a query object was built, and a
naive result-level replacement turned out equally vacuous for a subtle
reason: document_type is itself a default search field, so a broken
alias resolution demoting "type:invoice" to unfielded text STILL
matches the typed document through the field value under test. Both
alias tests (type/document_type, path/storage_path) now use
discriminating decoys carrying the query word in content, so demotion
matches the decoy and fails the exact-set assertion; the old
parse-shape test is deleted.
A new schema test pins that every PublicField.fast flag equals the
built tantivy schema's per-field fast option, in both drift directions:
whoosh-compat trusts the declared flag when resolving field:* existence
checks, and build_schema() only honors it for U64 and DATE kinds, so a
future fast=True TEXT/KEYWORD/JSON entry would otherwise make those
searches silently match nothing at query time.
Two result-level date pins restore behaviors whose assertions were lost
in the test migration: a created date matches regardless of the active
timezone (the America/New_York leg is the discriminating one: a
tz-applying implementation shifts the window past the naive-midnight
indexed value), and a reversed created:[2025 TO 2020] range still
matches its span through the joint-disambiguation swap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
Four pieces of the same surface:
whoosh-compat's emit() documents a two-part host contract: both a parse
diagnostic and the QueryEmitError/UnsupportedQueryError pair are
user-input errors. Only the latter half was caught; QueryEmitError now
maps to SearchQueryError too. Messages pass through a cleanup that
strips the library's DIVERGENCES.md references and replaces the
fast=True host-configuration advice with user language, so no
library-internal vocabulary reaches a searching user.
The bulk selection paths (bulk edit, the legacy bulk endpoint, bulk
download) reached the backend with no SearchQueryError handler, so a
bad date or number in a selection filter raised straight to a DRF 500.
They now share the search list endpoint's exact mapping (a new
search_query_error_messages helper flattens MultipleSearchQueryErrors
in one place), returning the same 400 body for the same bad query.
QueryParserError means a whoosh-compat parser bug, not user-fixable
input, per its own contract; the list endpoint's blanket handler was
converting it to a generic 400. It now re-raises and surfaces as a 500
that monitoring can see.
All behavior is pinned test-first: bulk edit and bulk download API
tests assert 400s naming the bad value (previously unhandled
exceptions), a unit test pins the QueryEmitError mapping, three
parametrized checks assert no internal vocabulary leaks for the
unsupported query shapes, and a mocked parser-bug test asserts the 500.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
The TestUnregisteredIdFieldFoldsToLiteralText docstring pointed readers
at docs/usage.md's advanced-search note about the dropped *_id
prefixes, which a later commit removed. State the rationale directly
instead: the *_id fields were always internal index columns (v2
consumed them for permission filtering and its own criteria), and their
queryability as search syntax was an accident of whoosh resolving any
schema field name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
The v2 whoosh schema had plural notes/custom_fields TEXT fields (notes
indexed the joined note texts, custom_fields indexed joined
"name : value" strings), so "notes:foo" and "custom_fields:foo" were
valid fielded searches in released paperless and through the deleted
translation layer. On the whoosh-compat registry those names are JSON
fields addressable only via subpaths, and the bare spelling silently
demoted to an unfielded text search of the words themselves, matching
unrelated documents that merely contain "notes" or "custom".
parse_user_query now rewrites the bare prefixes live to the same
targets migration 0017 chose for the singular whoosh-era spellings:
notes: becomes notes.note: and custom_fields: becomes
custom_fields.value:, with 0017's lookbehind guard so subpath spellings
and words merely ending in the prefix are untouched. Prefix
substitution only; values ride through unchanged, and every value shape
lands in a documented outcome downstream (ranges, wildcards and exists
on JSON subpaths are typed errors, not crashes). The inherited
trade-off stands: custom_fields.value: drops the name-matching half of
v2's combined indexing, with custom_fields.name: available for it.
Acceptance tests pin the rewrite with decoy documents whose content
contains the literal prefix words, which the old demotion matched and
the fielded search must not, plus untouched-subpath controls.
docs/usage.md documents the bare prefixes as subpath shorthand.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
"added:previous month" returned HTTP 400 after the whoosh-compat
migration. The unquoted spelling was never parser-native anywhere: v2
rewrote it to explicit bracket ranges app-side before whoosh saw the
string, and the deleted translation layer consumed it itself, so users
and saved views have relied on it continuously while whoosh-compat
deliberately scopes it out of its parser (its DIVERGENCES.md entry 19)
and understands the phrases natively only as quoted values.
parse_user_query now quotes the closed six-phrase vocabulary (previous
week/month/quarter/year, this month/year) when it directly follows a
date field's colon, before parsing. Only quoting happens app-side; every
date computation stays in whoosh-compat's grammar, unlike v2's rewrite,
which computed the ranges itself. Date field names derive from
PUBLIC_FIELDS, the field name matches case-sensitively (the parser's own
field tagging is case-sensitive), the phrase case-insensitively (the
grammar accepts any case in the quoted form), and already-quoted
spellings, TEXT fields, unfielded words and bracketed ranges are
untouched.
The previously xfailed end-to-end regression test now passes as a plain
test, and a new acceptance class pins unquoted == quoted == mixed-case
result sets on a boundary fixture, no-error parsing for the whole
vocabulary across all three date fields, and that "title:previous month"
stays an ordinary text search. docs/usage.md now states the two
spellings are equivalent after a date field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
The fuzzy blend clause handed the raw query string to tantivy's own
parser, which rejects whoosh-only grammar (date keywords, whoosh ranges,
aliases needing resolution), so any mixed query silently lost its fuzzy
clause: a typo'd word beside "added:today" stopped matching the moment
the date keyword appeared, while the same typo without it still matched.
Before the whoosh-compat migration the parser received the translated
string, so fuzzy survived mixed queries.
The clause is now built from whoosh_compat.free_text_tokens over the
already-parsed AST: the query's free-text words, analyzed, deduplicated,
with negated terms excluded so a NOT'd word cannot resurface through the
fuzzy clause. The joined word string is always plain tokens, so tantivy
always parses it; a defensive word-character filter guards any future
field whose analyzer passes punctuation through, and the ValueError skip
remains as insurance. One chosen trade-off is documented in the
docstring: a term fielded on a default search field contributes its text
unfielded, widening fuzzy recall on the 0.1-boosted secondary clause.
Two result-level acceptance tests pin the behavior: the mixed
typo-plus-date-keyword query matches its document again, and a NOT'd
word does not fuzzy-resurface (shaped so the assertion genuinely fails
under a naive all-words implementation: the excluded word's document is
the only candidate hit, so score normalization cannot mask it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
These prefixes were never documented public API (undocumented internal
fields the old KNOWN_FIELDS happened to accept), so their removal isn't a
user-facing regression worth calling out in usage.md. The behavior is still
covered by test_acceptance.py's TestUnregisteredIdFieldFoldsToLiteralText.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RVj8NFy821G3YhNf68PF6X
A prior commit deleted test_query.py's parametrized "doesn't raise" coverage
for this shape (created:[...T...Z TO ...] and comma-combined ranges), which
was also the only place PR #13010's T/Z backward-compat guarantee was
exercised. Nothing in paperless's suite proved the full parse_user_query() ->
tantivy Query -> matched-document pipeline still honors it after the
whoosh-compat grammar fix (commit f936143 in the whoosh-compat repo). Add
result-level acceptance cases: an in/out-of-range T/Z bracket range, PR
#13010's original comma-combined two-field shape, and an exact-boundary case
proving a Z-suffixed bound is absolute UTC, not shifted by the local search
timezone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RVj8NFy821G3YhNf68PF6X
- Update stale test comments in test_query.py that described string
rewriting / raw-query fallback behavior that no longer exists post
whoosh-compat migration; rename
test_date_rewriting_applied_before_tantivy_parse to
test_date_keyword_resolves_without_raising to match.
- views.py: move the local MultipleSearchQueryErrors import up into the
existing local-import block near the top of list(), consistent with
the other documents.search imports there, instead of importing it
again inside the except SearchQueryError clause.
- test_api_search.py: assert response.status_code explicitly before
indexing into response.data["results"] in
test_search_added_previous_month_excludes_next_period_start, and tie
the xfail marker to AssertionError instead of the incidental KeyError
that indexing a 400 response's missing "results" key produced.
Add a sentence to the "Supported date keywords" advanced-search section
noting that multi-word date keywords must be quoted (e.g.
added:"previous month") -- whoosh-compat requires quoting where the
unquoted form used to work. Also document that the old undocumented
*_id field aliases (tag_id, owner_id, viewer_id, correspondent_id,
document_type_id, storage_path_id, type_id, path_id) are no longer
recognized: a query using one now silently folds to a literal-text
search instead of matching the intended structured field.
test_unregistered_id_field_folds_to_literal_text_not_error only checked
that parse_user_query() didn't raise for a query like tag_id:5. Add a
result-level acceptance test (matching test_acceptance.py's
_matched_ids pattern, indexed against real documents) that asserts the
matched-document-ID set is genuinely empty, not just that the parse
step succeeds.
The fuzzy blend clause in parse_user_query() fed the raw, whoosh-syntax
query string directly to tantivy's own query parser. Since the
whoosh-compat migration, raw_query still contains whoosh grammar (date
keywords, whoosh-style ranges, bracket-class wildcards) that tantivy's
parser rejects with ValueError, which escaped parse_user_query and
turned into a generic HTTP 400 for the entire query whenever
ADVANCED_FUZZY_SEARCH_THRESHOLD was configured.
Deriving a clean plain-text-only extraction for the fuzzy clause was
ruled out: wc.parse() already expands unfielded terms into per-default-
field copies in the AST, so there's no "still unfielded" marker left to
walk without duplicating whoosh-compat's own expansion logic. Instead,
scope a narrow try/except ValueError around exactly the
index.parse_query() call and skip the fuzzy clause (logged at debug)
when it can't parse, leaving the exact/CJK clauses unaffected.
Replaces test_query.py's intermediate-AST/query-string checks with a
result-level acceptance corpus that indexes real documents and asserts
matched-ID sets through parse_user_query(), covering the #13568
bracket-wildcard regression, comma value lists, field boosts, JSON subpaths,
and Multitoken-in-OR nesting. Removes TestCreatedDateField, TestDateTimeFields,
TestWhooshQueryRewriting, TestYearRangeRewriting, TestNonDateFieldsNotRewritten,
TestPassthrough, TestNormalizeQuery, and TestParseUserQuery's
test_advanced_search_queries_do_not_raise from test_query.py, since they test
translate_query/_dates.py internals or a diagnostics-free-parse guarantee
whoosh-compat's own suite already covers.
When parse_user_query() raises MultipleSearchQueryErrors due to multiple
field parsing failures (e.g. both an invalid date and an invalid number
in a single query), the exception handler now surfaces all error messages
in the 400 response, allowing users to fix them all in one round-trip
instead of discovering them one at a time.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
whoosh-compat's date grammar now accepts "T" as a date/time separator
and a trailing "Z" UTC designator (paperless-ngx PR #13010
back-compat), so these advanced-search queries no longer raise.
Rewires parse_user_query() to parse via wc.parse()/tantivy_emit() against
the shared FieldRegistry instead of the string-based translate_query()
pipeline, so diagnostics map to typed SearchQueryError subclasses
(InvalidDateQuery/InvalidNumberQuery/MultipleSearchQueryErrors) and every
bad field is reported, not just the first.
Marks three pre-existing tests xfail (2 in test_query.py, 1 in
test_api_search.py) for confirmed whoosh-compat grammar gaps found while
verifying this rewrite: unquoted multi-word date keywords (e.g.
`added:previous month`) and RFC3339 T/Z datetime range bounds no longer
parse.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move SearchQueryError and InvalidDateQuery from _translate.py to _query.py and
add two new exception classes: InvalidNumberQuery and MultipleSearchQueryErrors.
Update _translate.py to re-export the exceptions for backward compatibility
until the translation module is removed. Update __init__.py to export all
four exception classes from _query.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Create the shared field-definition table consumed by the schema builder
(_schema.py) and the whoosh-compat field registry (_registry.py). This
eliminates drift between what the index exposes and what queries can address.
- Create PublicField frozen dataclass with field metadata
- Define PUBLIC_FIELDS tuple with 16 searchable fields
- Add comprehensive test suite covering field properties
The whoosh-compat pyproject.toml dependency addition is added in a
follow-up commit, with the correct [tantivy] extra, source comment, and a
matching uv.lock update.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The library changed after these were written and more changes are already
decided upstream. Records what is wrong today, what to write toward, the
one question still open, and the fast-JSON-field trap, rather than
silently leaving code that would fail on contact.
Field references became a typed value rather than a dotted string, so
diagnostics carry one too and the registry exposes a single resolver.
Also records that the JSON fields must stay non-fast while existence
checks against a fast JSON field return inverted results.
Diagnostics now carry field and raw_value, so the transition guidance
points at those instead of parsing human-readable message text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
16 bite-sized, TDD tasks across the design spec's 4-PR stack, each with
a suggested subagent type/model for delegated execution. Test/fixture
code in the acceptance-corpus and API-expansion tasks was verified
against the real codebase (documents/tests/search/conftest.py's
existing backend/index fixtures, test_backend.py's pytestmark
convention, CustomFieldInstance's typed value_text field) rather than
guessed, and the date-grammar parity test's AST-shape assumption was
confirmed by actually running whoosh_compat.parse() against a real
DATE FieldRegistry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Agent review (source-verified against both repos) confirmed the spec's
claims accurate throughout, with one real gap: the JSON-subpath
tantivy-py carve-out (index.parse_query fallback for notes.*/
custom_fields.* until tantivy-py#716 ships) interacts with paperless's
pinned tantivy~=0.26.0 and wasn't mentioned. Also added two footnotes:
FieldRegistry forces date_only=True on any DATE spec regardless of the
PublicField default, and the date-grammar parity audit implicitly
grants new keyword vocabulary as a side effect.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Design for replacing _translate.py/_dates.py with whoosh-compat: shared
field-definition table driving both the Tantivy schema and the query
FieldRegistry, diagnostics->exception mapping (aggregating all errors,
not just the first), a 4-PR stack with no rollout flag, and a
result-level acceptance corpus + date-grammar parity audit as the
safety net instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Encodes the settled integration decisions for replacing the
hand-maintained search translation layer with whoosh-compat:
user-typed query surface policy, analyzer seam, diagnostics-before-emit
contract, mandatory date parity audit, test churn, and rollout plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Trenton Holmes
shamoon
GitHub Actions