test_empty_query_returns_empty_query and test_all_operators_returns_empty_query
asserted isinstance(result, tantivy.Query), which parse_simple_text_highlight_query
cannot fail to satisfy: it either returns a Query or raises. Replacing its
`return tantivy.Query.empty_query()` with `all_query()` left both green, so
the contract they were named for -- highlight nothing, rather than highlight
every document -- was unpinned.
They now count hits against an index holding one document. A third test
asserts a real token matches that document, so an empty corpus cannot make
the other two pass for a match-everything query.
Under the empty_query -> all_query mutation both new tests fail; before this
change the mutation killed nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TestFieldAliases in test_documented_syntax.py pinned type:/path: against a
single indexed document with no decoy, so neither test could tell alias
resolution from the demotion that happens without it. Stripping `aliases`
from every FieldSpec and clearing the registry cache left both green:
- type:invoice demotes to unfielded text, and document_type is a default
search field, so the token still matched the typed document.
- path:archive demotes and matched via the title: the fixture title was
"Pathed", and with SEARCH_LANGUAGE=en that stems to "path".
test_acceptance.py::TestFieldAliases covers the same syntax at the same
result level with content decoys chosen to make demotion visible, and dies
on that mutation. Deleted rather than given decoys of their own, so the
property has one home instead of two.
Under the alias-strip mutation the suite now fails 4 tests: both
test_acceptance.py::TestFieldAliases cases and both
test_registry.py resolution cases.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GlobalSearchView calls the backend directly rather than through the shared
helper the cap lives in, so "every query string is length-checked" was a
claim with an exception rather than an invariant.
It hardcodes SearchMode.TEXT, which is linear rather than quadratic, so
this path was never the CPU-exhaustion vector and this is not a fix for
one. It is capped so the invariant holds without a footnote: the view
already bounds the query from below, and a later change letting it select
a search mode would otherwise reopen the hole with nothing to catch it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
whoosh-compat's fieldname tagger is O(n^2) in plain word characters,
reachable only through SearchMode.QUERY's whoosh grammar. Measured
against the real field registry: ~1s at 10k chars, ~3.7s at 20k, ~14.4s
at 40k. The POST selection-filter path (bulk edit, bulk download) has
no server-imposed length bound the way the GET path incidentally does
via header limits, making an unbounded query a single-request CPU
exhaustion vector.
Cap both entry points at their shared choke point,
_get_tantivy_query_and_mode, with a new QueryTooLongError that reuses
the existing SearchQueryError -> 400 routing both callers already
have. 4096 chars bounds the worst case to roughly 0.16s by quadratic
extrapolation, far beyond any plausible hand-typed query. TEXT and
TITLE modes route through simple_search_tokens instead and measure
linear even at 20k chars, so the same cap is hygiene for them rather
than a fix. Hardcoded rather than a PAPERLESS_* setting: this is a
security boundary, and a raisable ceiling could reintroduce the exact
hazard it exists to close.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The lock carried ~490 redundant `sys_platform == 'darwin' or sys_platform
== 'linux'` markers and had quietly pinned sqlparse to 0.5.5 and
pymdown-extensions to 11.0, both older than dev and neither related to
search. Re-resolving with uv 0.12.x (CI's pinned series) drops the markers
and restores both to dev's versions, taking the lock's diff against dev
from 1428 lines to 51. The only version line that now differs from dev is
whoosh-compat's own.
Three versions move here, not two: whoosh-compat also goes 0.1.0.dev0 ->
0.1.0, picked up from the sibling repo's own bump through the local path
dependency rather than from anything this re-lock decided.
A plain `uv lock` is a no-op here, since the lock is already
self-consistent; the markers are only recomputed when the resolution
actually re-runs.
Also drops a TODO's pointer to a spec file deleted in e50421542, and
corrects a comment claiming custom fields have a companion text field for
full-text search. They do not: no such field is ever written, and their
values are reachable only through the JSON field. Records that notes_text
is absent from _DEFAULT_SEARCH_FIELDS, so it is highlight-only by design.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
parse_simple_text_highlight_query re-parsed simple-search tokens through
Tantivy's query-string parser without quoting, so any token carrying
Tantivy grammar (a bare quote, a colon, brackets, a slash) raised an
unguarded ValueError once the search itself had already matched a
document. With DocumentViewSet.list's blanket exception handler narrowed
earlier on this branch, that ValueError now reaches the client as a bare
500, not the 400 it used to be -- confirmed against the real endpoint
before this change.
Quote and escape each token as its own phrase before parsing so ordinary
punctuation in a plain-text search no longer trips the grammar parser,
and keeps producing real highlight snippets instead of none. Still guard
the call with a narrow ValueError catch (matching the sibling notes_text
guard's shape, not its broader Exception catch) as defense in depth for
inputs quoting alone cannot save, falling back to the query that already
matched.
The prune replaced a generic "no PUBLIC_FIELDS name ends in _id" invariant
with a fixed list of the seven names that were dropped. That list catches
the seven; nothing catches the eighth.
Dropping write-only *_id fields from the query surface is the whole point
of the schema change earlier in this branch, so the generic form is what
guards the class against recurrence. Both tests coexist: the list pins
that specific names stay unregistered, this pins that no new one leaks in.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- test_fields.py: reduced to test_json_fields_have_subpaths - the rest
asserted properties of PUBLIC_FIELDS' 16-line literal tuple, already
covered behaviourally by test_registry.py's resolve()-based tests.
- test_registry.py::TestJsonSubpathCoupling: deleted - its own docstring
admitted it hardcodes both sides of the comparison it claims to guard.
test_acceptance.py::TestJsonSubpaths already proves the coupling
against a real index, and the new test_json_subpath_completeness.py
proves it exhaustively for every declared subpath.
- test_query.py: dropped the asn/checksum isinstance-only parse checks,
now duplicated by result-level matches in test_documented_syntax.py
and test_api_search.py.
- conftest.py: dropped the module-scoped `index` fixture, dead since
test_translate.py was deleted.
419 passed (documents/tests/search/ + test_api_search.py +
test_api_search_errors.py), down from 432 before this prune.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds dedicated result-level tests, each indexing real documents and
asserting on matched-ID sets rather than parse shape:
- Deferred `-term` negation (G1): bare `-taxes` requires the term, and
fielded `-title:alpha` drops the negation entirely, both matching v2.
- Reversed date ranges: absolute bounds swap (whoosh parity); relative
`now±` bounds day-bump instead, an inconsistency pinned as a
whoosh-compat follow-up rather than "fixed" locally.
- The six whoosh unit abbreviations (yrs/mos/wks/hrs/mins/secs).
- Unterminated `[` date range brackets 400 at the API level.
- `tag:foo,bar` comma value lists are conjunctive, with a decoy proving
`correspondent:foo,bar` is not treated as a list.
- Date keyword phrases (`today`) honour the active (non-UTC) timezone,
not just relative ranges.
- `_DEFAULT_SEARCH_FIELDS` stays a subset of PUBLIC_FIELDS.
- Every declared JSON subpath is actually written at index time.
Also replaces test_schema.py::TestFastFlagAgreement's tantivy-py
__reduce__() pickling probe with one built on the paperless-owned,
tantivy-independent field_descriptors().
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit's warning said a timestamp carrying a time of day was
not understood at all, and told users to fall back to whole-day range
bounds. Both halves were wrong. Only the bare unquoted spelling fails:
added:2005-01-01T00:00:00Z -> no match
added:"2005-01-01T00:00:00Z" -> matches
added:[2005-01-01T00:00:00Z to 2006-01-01T00:00:00Z] -> matches
That is the ordinary quoting rule the surrounding docs already state, the
same one "-1 week" and "next monday" obey, so present the timestamp as a
working form rather than as a limitation and drop the false workaround:
range bounds carrying a time of day work fine.
A bound must be bare inside range brackets, where quoting it is rejected
outright, so document both halves of the rule rather than just "quote it".
Keep the zero-width warning distinct from the quoting rule now sitting above
it, since a reader who just learned quoting rescues "next monday" would
otherwise assume it rescues "-3 days". It does not: re-verified against
documents added at exactly those instants, the quoted offsets still match
only that one instant.
Pin the working spellings, which is the assertion that was missing: nothing
covered the quoted or range-bound forms, so a regression of a working
feature went undetected. Also pin that quoting does not rescue the
zero-width group.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
usage.md linked to tantivy's QueryParser documentation, promising a grammar
paperless neither implements nor intends to. Replace the link with a
description of the surface that was verified end-to-end against a real index,
and correct the claims that did not survive that verification.
- checksum: the field is stored verbatim, so only a complete lowercase
checksum matches. The old a1b2c3d4 example matched nothing.
- A leading - is not negation. Separators are stripped at index time, so
"invoice -secret" requires "secret", the opposite of the intent. Document
NOT as the way to exclude a term.
- Document the aliases type: and path:, num_notes:, numeric ranges, quoted
phrases, tag:'s comma list (which requires all listed tags, not any), and
the date forms that resolve to a real span: tomorrow, ISO dates, month
names, "next monday"/"last monday".
- Warn about now/noon/midnight and offsets like "-3 days": they parse, but
resolve to a single instant rather than a span, so they match nothing.
Likewise a T/Z timestamp, whose time portion is split off as loose text.
Add test_documented_syntax.py, which asserts on matched document IDs rather
than on parsed queries, so the docs cannot drift from the code again. Its
negative cases pin the behaviours the warnings describe.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both comments still described the pre-fingerprint layout: _fields.py said
the internal-only fields "stay hardcoded in build_schema()", and the
fast-flag test said build_schema() honors the flag only in its U64 and
DATE branches. Both now live in field_descriptors(), and _fields.py's
header is the one thing a future editor reads before touching the field
table, so a stale pointer there is the expensive kind.
Comments only; no executable line is touched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
build_schema() was half table-driven and half hardcoded, so editing it for
parser reasons could change the on-disk field list without anyone bumping
SCHEMA_VERSION. tantivy compares schemas by ordered field list, so such an
edit leaves reads working while every write raises.
Complete the table: build_schema() now iterates an explicit list of field
descriptors covering id, the PUBLIC_FIELDS expansion, the sort shadow,
bigram, simple_* and autocomplete fields and the permission columns. The
same list is hashed into a schema_fingerprint() that is stamped into
.index_settings.json and compared by needs_rebuild() as a fourth check
alongside the existing schema version and language checks.
The fingerprint is computed from paperless' own descriptors rather than
tantivy's schema representation, so a tantivy-py option-key rename or
addition cannot silently force a global reindex.
The emitted schema is byte-identical to the previous one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fuzzy clause's word string is cut to \w+ runs so no query grammar
reaches index.parse_query, but tantivy's boolean keywords are themselves
word runs. Under analyzed=True the field analyzer lowercased them into
ordinary terms before they got that far; now that the words are raw
query text, an uppercase keyword out of a quoted phrase arrives as
grammar: '"tax AND reports"' quietly made the clause a conjunction,
'"tax NOT reports"' gave it its own exclusion, and '"tax AND"' (or IN
anywhere) failed the parse and cost the query its fuzzy clause outright.
Lowercase exactly AND/OR/NOT/IN, which is what the analyzer used to do
and is the only spelling tantivy reads as grammar ("And" is a term).
Nothing else is touched: tantivy already lowercases query terms with the
field's analyzer, and doing it ourselves first is not the same operation
for every input (Python folds a final sigma differently, and turns 'İ'
into a sequence tantivy then splits in two), which would search for
terms the index does not contain.
Also pins two behaviours that were reasoned about but untested: the
fielded-CJK test now runs with the fuzzy clause on as well, where the
clause's documented unfielded contribution does bring the other document
back, and the negation tests pin the CJK over-admission for an exclusion
under an Or, which cannot be hoisted without dropping the other branch's
documents.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_try_parse_fuzzy_query collected free_text_tokens with the default
analyzed=True and handed the analyzer's output back to
index.parse_query, which analyzes it again. Analysis is not idempotent:
'universities' stems to 'univers', and re-stemming that yields 'univ', a
term the index does not contain. prefix=True hid the mistake as
over-broad matching rather than as no matches at all, which is why no
test caught it: searching 'universities' also returned documents whose
only relevant word was 'univalent' or 'unicycle'.
Collect the raw text instead. Raw text has not been tokenized, so the
whole-token \w+ filter that keeps tantivy query grammar out of the
re-parse would now reject ordinary input outright: 'COVID-19',
'hello@example.com' and the phrase "tax reports" each arrive as a single
token containing punctuation, and a query made only of such terms would
lose its fuzzy clause entirely. Cut each token into its word runs and
keep those, which recovers the terms and keeps the guarantee the filter
exists for: only word characters ever reach the parser.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
parse_user_query ORs three top-level Should clauses: the exact query, an
optional fuzzy blend and an optional CJK bigram clause. The latter two
are built from positive terms only and cannot express an exclusion, so
each one re-admitted precisely the documents the exact clause had
excluded: 'invoice NOT secret' returned the secret document as soon as
ADVANCED_FUZZY_SEARCH_THRESHOLD was set, and '東京 NOT secret' returned
it unconditionally, since nothing gates the CJK clause.
Building the CJK clause from the AST does not fix this: there the
excluded term is not the CJK one, so the clause legitimately contains
東京 and still matches the document.
Hoist the exclusions instead. _ConjunctiveNegations walks the parsed
tree for the subtrees that constrain every matching document, and each
is emitted as an ordinary positive query attached with MustNot above the
Must-ed blend. Or is not descended into: in 'invoice OR NOT secret' the
negation is one branch's condition, and hoisting it would drop documents
the other branch matches.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_build_cjk_query scanned the raw query string for CJK runs, so a CJK term
the user negated ('invoice NOT 漢字') or restricted to one field
('title:漢字', 'notes:漢字') came straight back as a top-level Should
clause over every bigram field. The fuzzy clause already collects its
words from the parsed tree for exactly this reason; the CJK clause a few
lines below did not.
Collect the CJK runs from whoosh_compat's free_text_tokens over
result.ast instead, one default field at a time so the tokens keep their
field attribution: a bare term (already copied onto every default field
by the parser) still searches every bigram field, while title:東京
reaches bigram_title alone, and a term on a non-default field
contributes nothing. Fields sharing identical CJK text share one parse.
The raw-string builder stays for the simple TEXT/TITLE modes, whose
input is plain text with no query grammar to respect, as does
extract_cjk_text, which the indexing side calls per bigram field.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
whoosh-compat's grammar already accepts the closed multi-word date
keyword vocabulary (previous month, this year, etc.) unquoted after a
date field, making _quote_date_keyword_phrases redundant. Like its
sibling rewrite removed in an earlier commit, it was not quote-aware
and could insert quotes mid-phrase inside an unrelated quoted string
(e.g. title:"see added:previous month notes"), corrupting the parse.
Deleting it removes that hazard entirely.
Docs are adjusted to scope the quoted-or-unquoted equivalence to the
documented keyword list; other date expressions the grammar accepts
(relative offsets, absolute dates) still require quoting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The regex rewrite that turned bare notes:/custom_fields: prefixes into
their subpath spelling was blind to quoting: content:"payment notes:
none" was silently rewritten mid-phrase into a notes-field search and
matched zero documents. whoosh-compat's FieldSpec now supports a
default subpath per JSON field (SubpathSpec(default=True)), which
resolves during parsing where quoting is already understood, so the
pre-parse string rewrite is no longer needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The docstring said a shorter prefix "only widens recall". That holds for a
stem that truncates, not for one that substitutes: English y -> i moves the
pattern sideways, so "copy*" gains "copies" and loses "copyright". Stating
it as a general invariant is what hid that class in the first place.
Length stays the rule; only its justification is corrected. No behavior
change -- no executable line is touched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stemming substitutes as well as truncates ("copy" and "copies" both index as
"copi" while "copyright" keeps its literal y), so a stemmed pattern reaches a
word's inflections but no longer reaches compounds that keep the surface
spelling. That trade is accepted: the same substitution is what makes company*
and library* work, and no rule over one normalized string separates them. So
usage.md stops claiming a trailing star just works, and a test pins copy* to the
base word rather than the compound. Also adds a parity test tying
stem_pattern_text to paperless_text_analyzer's own output, so a filter added to
the index analyzer alone cannot silently diverge, and corrects the docstring
claim that a run can analyze to several tokens - the raw tokenizer emits one
token whatever the input, so only the remove_long zero-token case can fire.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Index terms are stemmed but query patterns were not, so invoice* matched nothing
while invoic* worked. v2's index was unstemmed (whoosh TEXT() defaults to
StandardAnalyzer), so this regressed against both baselines, not just dev. Uses
the typed run's stem unless the stem is longer than the run, since a stem can be
longer than a partial prefix and a shorter prefix only widens recall. Patterns
spanning the stem boundary (produ*name) still cannot match a stemmed index, so
usage.md loses that example rather than advertising a broken one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This reverts ea883f416, which suppressed repeat MISCONFIGURED logs to
once per field per process.
A misconfigured field is a static condition an operator can fix in one
change, so the repetition is the prompt to fix it rather than noise to
suppress, and it stops on its own once the schema is corrected. Keeping
the suppression meant carrying machinery whose key boundedness and
check-then-add race both had to be reasoned about, to solve a problem
that ends when someone fixes the config.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DocumentViewSet.list's trailing except Exception clause was catching the
re-raised QueryParserError/INTERNAL-cause QueryError that _map_emit_error
and whoosh-compat's own parse() deliberately let escape, and turning them
into a generic 400 -- exactly the outcome that routing exists to prevent.
Remove the catch-all (and the now-redundant QueryParserError re-raise it
made pointless) so a whoosh-compat library defect surfaces as a
monitorable 500 instead of blaming the user for it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
EXISTS_REQUIRES_FAST is MISCONFIGURED and reachable from ordinary query text
(notes.user:*), so the operator alert added with the Cause routing fired on
every such request. An alert that repeats on every user query is one operators
learn to filter out, which defeats routing MISCONFIGURED to an operator at all.
The condition is a static configuration fact: it stays true until an operator
changes the schema and reindexes, so the first log carries the same information
as the ten-thousandth. Deduped on (kind, field) in a per-process set; a restart
re-logs, re-surfacing the condition after a config change. The 400 is not
deduped: every request still gets its response and its message.
The key is bounded by the registry, not by query text. emit() only reports
MISCONFIGURED for a field it resolved, and FieldRegistry.resolve returns None
for any name or JSON subpath the registry does not declare.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The except QueryError arm converted every kind to a 400 on the strength of a
comment asserting the INTERNAL kinds could not occur. SCHEMA_FIELD_MISSING
fires on registry/schema drift, which deriving both from PUBLIC_FIELDS newly
makes possible, so a defect in our own wiring was reported to the user as a bad
query and never reached monitoring.
Diagnostics now route on Cause: INVALID_INPUT/UNSUPPORTED are a 400,
MISCONFIGURED is logged at error level naming the field and then a 400 (the
registry and the schema disagree, which only an operator can fix, but a request
is still waiting and the query cannot run either way), and INTERNAL is
re-raised rather than converted.
Messages, parse-time as well as emit-time, are built from the Diagnostic's
structured fields; d.message is documented as unstable developer output and
PATTERN_TOO_COMPLEX embedded raw backend error text in the 400 body.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
.superpowers/ holds per-plan ledgers, task briefs and review diffs. Untracked
scratch in the tree is how unrelated files get swept into commits, and in a
sibling repo the same directory was silently pulled into an sdist by the build
backend's default include.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
whoosh-compat replaced both exception classes with a single QueryError
carrying a structured Diagnostic (kind/cause/field_kind); the old
message-text regex stripping is now dead weight since the library no
longer embeds host-facing wording (DIVERGENCES refs, fast=True advice)
in Diagnostic.message. Branch on diagnostic.kind instead.
Also trims a test that was re-asserting whoosh-compat's own message
contract (now covered by its own test_kind_matrix.py) down to the one
rewrite paperless still owns: EXISTS_REQUIRES_FAST's user-facing message.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_quote_date_keyword_phrases and _rewrite_bare_json_field_prefixes both
regex-match anywhere in raw_query, with no awareness of whether the match
falls inside an already-quoted phrase on an unrelated field. Unlikely in
practice and not fixed (quote-aware scanning is real work for an edge
case), but now called out explicitly like this file's other accepted
trade-offs, instead of being the one undocumented one.
_build_cjk_query silently swallowed a parse failure with no log line,
while _try_parse_fuzzy_query logs at debug for the same "skip this
optional clause" situation. Add the matching debug log.
Deliberately NOT narrowing except Exception to except ValueError here to
match the fuzzy path: the fuzzy blend's word string is pre-filtered to
\\w+-only tokens before it ever reaches index.parse_query, so ValueError
is the only realistic failure mode there. cjk_text has no equivalent
filter, so narrowing this catch without verifying tantivy's actual
exception behavior for CJK input would risk letting something other than
ValueError propagate uncaught - the same class of mistake as the fuzzy
blend regression this migration already fixed once, in the other
direction.
_query.py mixed three unrelated responsibilities: the SearchQueryError
family (paperless's public error-surface API, re-exported by __init__.py),
the actual query rewrite/parse/emit/blend pipeline, and
build_permission_filter, which has nothing to do with query parsing and
is consumed only by _backend.py.
- New _errors.py: SearchQueryError, InvalidDateQuery, InvalidNumberQuery,
MultipleSearchQueryErrors, search_query_error_messages. _query.py now
imports these instead of defining them.
- build_permission_filter moves to _backend.py, next to its one caller
(TantivyBackend._build_permission_filter).
- __init__.py re-exports the error classes from _errors.py instead of
_query.py; the package's public API (documents.search import ...) is
unchanged for every caller going through it (views.py etc.).
_query.py now reads top-to-bottom as rewrite -> parse -> emit -> blend,
matching what parse_user_query's own docstring already claimed the file
was.
DEFAULT_SEARCH_FIELDS/SIMPLE_SEARCH_FIELDS/TITLE_SEARCH_FIELDS looked
public but are only ever used inside _query.py itself, sitting next to
underscore-prefixed constants at the same scope (_CJK_ALL_FIELDS etc.).
Rename to match, and add Final like their neighbors already have.
Both test_fields.py and test_registry.py repeated the same generator-next
lookup by field name. Add a module-level {name: field} dict in each and
use it instead.
Also collapse test_fields.py's five single-attribute tests
(document_type/storage_path aliases, tag's comma_values, notes/
custom_fields subpaths) into one parametrized test_field_attributes -
they were really one table-consistency check split into five copies of
the same three-line shape.
test_search_by_asn/page_count/original_filename/checksum were all
create-doc -> index -> GET -> assert 200 and doc.id in results, repeated
verbatim four times. Extract _assert_query_finds() so each test states
only its distinguishing field and query.
InvalidDateQuery/InvalidNumberQuery/MultipleSearchQueryErrors/
SearchQueryError are all already imported at module top; three test
bodies re-imported them locally for no reason.
User/DocumentType/StoragePath were imported inside individual test bodies
despite the module already importing documents.models at top level -
nothing here needed deferred import. Also add an _index() helper
(Document.objects.create + backend.add_or_update in one call) for the many
sites where nothing needs to happen between creating a document and
indexing it; the two-step ceremony was outweighing the fixture data at
every call site. Left as two explicit steps wherever a Note or
CustomFieldInstance genuinely has to be attached before indexing.
get_backend was imported alone, three statements ran, then
SearchQueryError/search_query_error_messages were imported separately -
one function's imports split across two blocks with code between them.
Merge into the single existing local-import block.
PublicField duplicated seven fields whoosh-compat's own FieldSpec already
has (name/kind/aliases/comma_values/date_only/fast/subpaths), and
_registry.py hand-copied all of them across on every registry build.
FieldSpec is a frozen dataclass with analyzer/pattern_normalizer already
optional (default None), so PUBLIC_FIELDS can just BE the FieldSpec tuple -
_schema.py only ever read name/kind/fast off it and needs no changes.
_registry.py now attaches the per-language analyzer/pattern_normalizer via
dataclasses.replace() instead of reconstructing every field from scratch.
FieldSpec.__post_init__ normalizes subpaths into a MappingProxyType, so
test_fields.py's exact-tuple-equality subpath assertions become set
comparisons; a genuinely empty subpaths is now `not field.subpaths` rather
than `== ()`.
Verified test_api_trash.py::test_api_trash's "Schema error: An index exists
but the schema does not match" failure is a pre-existing, unrelated local
environment issue (a stale, untracked data/index/ directory in this
checkout) - reproduces identically with this commit's changes stashed out.
It called simple_search_tokens() and nothing else, with a comment
duplicating that function's own docstring. Call sites now call
simple_search_tokens() directly.
Four call sites in _query.py each hand-rolled "no clauses -> empty, one
clause -> return it bare, many -> wrap in boolean_query" - one of them also
handling the empty case, one written as a ternary, one returning a captured
variable instead of clauses[0][1] (same value, different spelling). Extract
_any_of() so the collapsing logic and its rationale (skip a wasted
single-clause boolean_query wrap) live in one place.
_schema_field_names and TestFastFlagAgreement each independently reached
into Schema.__reduce__()[1][0] - the one fragile, version-coupled
expression this test suite depends on. Rename to _schema_fields, return
{name: field-state} instead of just names, and have both call sites use
it, so a tantivy-py upgrade that changes this shape breaks in one place.
Three classes in test_query.py each defined an identical query_index
fixture. None of these tests write documents to the index, so consolidate
into one module-level, module-scoped fixture (mirroring conftest.py's
index fixture rationale) instead of three copies to keep in sync.
Deliberately NOT merged with conftest.py's own index fixture: that one
registers tokenizers with "english" (stemming on), while these tests rely
on "" (stemming off) - a real behavioral difference, not incidental.
Every test repeated the same 5-line make_ref/resolve/assert-not-None
sequence before its one real assertion. Extract a registry fixture and a
typed _resolve() helper so each test states one fact in one line.
_build_cjk_query re-derived the same CJK-run extraction extract_cjk_text
already implements, despite a docstring claiming they mirror each other.
Call it directly so the mirroring is structural, not a copy to keep in sync.
Deletes TestCommaValueLists, TestMultitokenInNestedOr, TestRfc3339TZDateRange,
TestCreatedTimezoneInvariance, and TestReversedDateRange: none of them
exercise any paperless-specific pre/post-processing code. Comma-list AND
semantics, multitoken resolution, RFC3339 T/Z UTC math, date-only timezone
invariance, and reversed-range disambiguation are all entirely
whoosh-compat's own grammar/semantics, already covered by its own test
suite. The comma_values flag paperless does own is still covered cheaply in
test_fields.py; the date_only flag is still covered in test_registry.py.
Also trims verbose docstrings/comments across _query.py and the surviving
acceptance tests: cuts references to whoosh-compat's internal
DIVERGENCES.md entry numbers and paperless v2/Whoosh-era implementation
history down to the user-facing behavior that actually matters, without
losing the substance.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RVj8NFy821G3YhNf68PF6X
test_date_grammar_parity.py was added and deleted entirely within this
feature branch's own history; it never existed in dev. Referencing its
deletion in a docstring only makes sense while reading this branch's
intermediate commits, not once this merges - unlike PR #13010 or
whoosh-compat's DIVERGENCES.md, which are permanent, externally
verifiable references.
The design spec, implementation plan, and dev-skill for this migration
are no longer needed now that the migration is complete and merged into
this branch.
Four targeted additions, no production code:
The type-alias test asserted only that a query object was built, and a
naive result-level replacement turned out equally vacuous for a subtle
reason: document_type is itself a default search field, so a broken
alias resolution demoting "type:invoice" to unfielded text STILL
matches the typed document through the field value under test. Both
alias tests (type/document_type, path/storage_path) now use
discriminating decoys carrying the query word in content, so demotion
matches the decoy and fails the exact-set assertion; the old
parse-shape test is deleted.
A new schema test pins that every PublicField.fast flag equals the
built tantivy schema's per-field fast option, in both drift directions:
whoosh-compat trusts the declared flag when resolving field:* existence
checks, and build_schema() only honors it for U64 and DATE kinds, so a
future fast=True TEXT/KEYWORD/JSON entry would otherwise make those
searches silently match nothing at query time.
Two result-level date pins restore behaviors whose assertions were lost
in the test migration: a created date matches regardless of the active
timezone (the America/New_York leg is the discriminating one: a
tz-applying implementation shifts the window past the naive-midnight
indexed value), and a reversed created:[2025 TO 2020] range still
matches its span through the joint-disambiguation swap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
Four pieces of the same surface:
whoosh-compat's emit() documents a two-part host contract: both a parse
diagnostic and the QueryEmitError/UnsupportedQueryError pair are
user-input errors. Only the latter half was caught; QueryEmitError now
maps to SearchQueryError too. Messages pass through a cleanup that
strips the library's DIVERGENCES.md references and replaces the
fast=True host-configuration advice with user language, so no
library-internal vocabulary reaches a searching user.
The bulk selection paths (bulk edit, the legacy bulk endpoint, bulk
download) reached the backend with no SearchQueryError handler, so a
bad date or number in a selection filter raised straight to a DRF 500.
They now share the search list endpoint's exact mapping (a new
search_query_error_messages helper flattens MultipleSearchQueryErrors
in one place), returning the same 400 body for the same bad query.
QueryParserError means a whoosh-compat parser bug, not user-fixable
input, per its own contract; the list endpoint's blanket handler was
converting it to a generic 400. It now re-raises and surfaces as a 500
that monitoring can see.
All behavior is pinned test-first: bulk edit and bulk download API
tests assert 400s naming the bad value (previously unhandled
exceptions), a unit test pins the QueryEmitError mapping, three
parametrized checks assert no internal vocabulary leaks for the
unsupported query shapes, and a mocked parser-bug test asserts the 500.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
stumpylogandClaude Opus 5