The warning said relative offsets such as "-1 week" "in practice mean no
documents at all", unqualified, one bullet after the range-bound rule
offered added:['-1 week' to now] as the example of quoting a bound. Both
sentences were true in their own scope, but read together they talk a
user out of a query that works: the bare value is a zero-width instant,
the same offset as a bound is a real seven-day window (verified:
lo=2026-06-08T12:00, hi=2026-06-15T12:00 from a frozen 2026-06-15T12:00).
Also widens the last sentence to say now-3days and "3 days ago" are
rejected wherever they appear, having checked they are rejected as range
bounds too, single- or double-quoted, not only as bare values.
Restores the 'added:"-1 week"' case to the standalone-forms list, which
the docs name by that exact spelling, and pins the bound reading beside
it so neither half of the warning is an unpinned claim.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
whoosh-compat's pattern_normalizer now accepts several alternative forms
per literal run, ORed and deduplicated by the emitter, so the
shorter-of-the-two heuristic that had to pick one form is gone. The typed
run and its stem are both offered: neither is a prefix of the other once
the stemmer substitutes rather than truncates ("copy" -> "copi"), so
"copy*" now reaches "copies" and "copyright" alike instead of trading one
for the other.
checksum keeps _fold_normalizer. It is the only KEYWORD field, indexed
with the raw tokenizer, and a stemmed prefix there ("ceded" -> "cede")
returns documents whose checksum does not start with what was typed.
Also picks up two date-grammar fixes from the same library release: a
reversed relative range now swaps its bounds like the absolute case
instead of day-bumping the upper one, and a date value the grammar can
only half-consume (a bare, unquoted "added:2005-03-04T15:30:00Z") is
rejected as an InvalidDateQuery rather than silently matching nothing.
docs/usage.md: the "copy* does not find copyright" caveat is no longer
true; the bare timestamp is now an error rather than a silent non-match;
and the range-bracket quoting rule was wrong in a user-visible way. Only
double-quoted bounds are rejected, single-quoted ones parse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_map_emit_error's docstring pointed at the QueryParserError arm of views.py's
generic handler, which was deleted; QueryParserError no longer appears in
production code at all. _REGEX_TIMEOUT justified itself as ReDoS protection,
but the two pre-parse rewrites it was written for are gone and its one
remaining use is a character class that cannot backtrack. The timeout stays,
its rationale is corrected. administration.md listed two triggers for
--if-needed; the schema fingerprint is now a third.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
get_field_registry() branched the analyzer by field kind but gave every field
the same stemming pattern normalizer. checksum is indexed with the raw
tokenizer, so its terms are neither folded nor stemmed, yet its wildcard
patterns were: "checksum:ceded*" normalized to "cede*" and matched a document
whose checksum starts with "cedef00d". About 2.8% of random hex prefixes were
rewritten this way. Always over-matching rather than missing, but for a field
whose whole purpose is exact identification, returning a different checksum is
a wrong answer.
KEYWORD fields now get a fold-and-lower normalizer, which is what every field
used before pattern stemming was added; TEXT fields keep the stemming one so
"invoice*" still reaches the indexed "invoic".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit's warning said a timestamp carrying a time of day was
not understood at all, and told users to fall back to whole-day range
bounds. Both halves were wrong. Only the bare unquoted spelling fails:
added:2005-01-01T00:00:00Z -> no match
added:"2005-01-01T00:00:00Z" -> matches
added:[2005-01-01T00:00:00Z to 2006-01-01T00:00:00Z] -> matches
That is the ordinary quoting rule the surrounding docs already state, the
same one "-1 week" and "next monday" obey, so present the timestamp as a
working form rather than as a limitation and drop the false workaround:
range bounds carrying a time of day work fine.
A bound must be bare inside range brackets, where quoting it is rejected
outright, so document both halves of the rule rather than just "quote it".
Keep the zero-width warning distinct from the quoting rule now sitting above
it, since a reader who just learned quoting rescues "next monday" would
otherwise assume it rescues "-3 days". It does not: re-verified against
documents added at exactly those instants, the quoted offsets still match
only that one instant.
Pin the working spellings, which is the assertion that was missing: nothing
covered the quoted or range-bound forms, so a regression of a working
feature went undetected. Also pin that quoting does not rescue the
zero-width group.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
usage.md linked to tantivy's QueryParser documentation, promising a grammar
paperless neither implements nor intends to. Replace the link with a
description of the surface that was verified end-to-end against a real index,
and correct the claims that did not survive that verification.
- checksum: the field is stored verbatim, so only a complete lowercase
checksum matches. The old a1b2c3d4 example matched nothing.
- A leading - is not negation. Separators are stripped at index time, so
"invoice -secret" requires "secret", the opposite of the intent. Document
NOT as the way to exclude a term.
- Document the aliases type: and path:, num_notes:, numeric ranges, quoted
phrases, tag:'s comma list (which requires all listed tags, not any), and
the date forms that resolve to a real span: tomorrow, ISO dates, month
names, "next monday"/"last monday".
- Warn about now/noon/midnight and offsets like "-3 days": they parse, but
resolve to a single instant rather than a span, so they match nothing.
Likewise a T/Z timestamp, whose time portion is split off as loose text.
Add test_documented_syntax.py, which asserts on matched document IDs rather
than on parsed queries, so the docs cannot drift from the code again. Its
negative cases pin the behaviours the warnings describe.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
whoosh-compat's grammar already accepts the closed multi-word date
keyword vocabulary (previous month, this year, etc.) unquoted after a
date field, making _quote_date_keyword_phrases redundant. Like its
sibling rewrite removed in an earlier commit, it was not quote-aware
and could insert quotes mid-phrase inside an unrelated quoted string
(e.g. title:"see added:previous month notes"), corrupting the parse.
Deleting it removes that hazard entirely.
Docs are adjusted to scope the quoted-or-unquoted equivalence to the
documented keyword list; other date expressions the grammar accepts
(relative offsets, absolute dates) still require quoting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stemming substitutes as well as truncates ("copy" and "copies" both index as
"copi" while "copyright" keeps its literal y), so a stemmed pattern reaches a
word's inflections but no longer reaches compounds that keep the surface
spelling. That trade is accepted: the same substitution is what makes company*
and library* work, and no rule over one normalized string separates them. So
usage.md stops claiming a trailing star just works, and a test pins copy* to the
base word rather than the compound. Also adds a parity test tying
stem_pattern_text to paperless_text_analyzer's own output, so a filter added to
the index analyzer alone cannot silently diverge, and corrects the docstring
claim that a run can analyze to several tokens - the raw tokenizer emits one
token whatever the input, so only the remove_long zero-token case can fire.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Index terms are stemmed but query patterns were not, so invoice* matched nothing
while invoic* worked. v2's index was unstemmed (whoosh TEXT() defaults to
StandardAnalyzer), so this regressed against both baselines, not just dev. Uses
the typed run's stem unless the stem is longer than the run, since a stem can be
longer than a partial prefix and a shorter prefix only widens recall. Patterns
spanning the stem boundary (produ*name) still cannot match a stemmed index, so
usage.md loses that example rather than advertising a broken one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The design spec, implementation plan, and dev-skill for this migration
are no longer needed now that the migration is complete and merged into
this branch.
The v2 whoosh schema had plural notes/custom_fields TEXT fields (notes
indexed the joined note texts, custom_fields indexed joined
"name : value" strings), so "notes:foo" and "custom_fields:foo" were
valid fielded searches in released paperless and through the deleted
translation layer. On the whoosh-compat registry those names are JSON
fields addressable only via subpaths, and the bare spelling silently
demoted to an unfielded text search of the words themselves, matching
unrelated documents that merely contain "notes" or "custom".
parse_user_query now rewrites the bare prefixes live to the same
targets migration 0017 chose for the singular whoosh-era spellings:
notes: becomes notes.note: and custom_fields: becomes
custom_fields.value:, with 0017's lookbehind guard so subpath spellings
and words merely ending in the prefix are untouched. Prefix
substitution only; values ride through unchanged, and every value shape
lands in a documented outcome downstream (ranges, wildcards and exists
on JSON subpaths are typed errors, not crashes). The inherited
trade-off stands: custom_fields.value: drops the name-matching half of
v2's combined indexing, with custom_fields.name: available for it.
Acceptance tests pin the rewrite with decoy documents whose content
contains the literal prefix words, which the old demotion matched and
the fielded search must not, plus untouched-subpath controls.
docs/usage.md documents the bare prefixes as subpath shorthand.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
"added:previous month" returned HTTP 400 after the whoosh-compat
migration. The unquoted spelling was never parser-native anywhere: v2
rewrote it to explicit bracket ranges app-side before whoosh saw the
string, and the deleted translation layer consumed it itself, so users
and saved views have relied on it continuously while whoosh-compat
deliberately scopes it out of its parser (its DIVERGENCES.md entry 19)
and understands the phrases natively only as quoted values.
parse_user_query now quotes the closed six-phrase vocabulary (previous
week/month/quarter/year, this month/year) when it directly follows a
date field's colon, before parsing. Only quoting happens app-side; every
date computation stays in whoosh-compat's grammar, unlike v2's rewrite,
which computed the ranges itself. Date field names derive from
PUBLIC_FIELDS, the field name matches case-sensitively (the parser's own
field tagging is case-sensitive), the phrase case-insensitively (the
grammar accepts any case in the quoted form), and already-quoted
spellings, TEXT fields, unfielded words and bracketed ranges are
untouched.
The previously xfailed end-to-end regression test now passes as a plain
test, and a new acceptance class pins unquoted == quoted == mixed-case
result sets on a boundary fixture, no-error parsing for the whole
vocabulary across all three date fields, and that "title:previous month"
stays an ordinary text search. docs/usage.md now states the two
spellings are equivalent after a date field.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
These prefixes were never documented public API (undocumented internal
fields the old KNOWN_FIELDS happened to accept), so their removal isn't a
user-facing regression worth calling out in usage.md. The behavior is still
covered by test_acceptance.py's TestUnregisteredIdFieldFoldsToLiteralText.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RVj8NFy821G3YhNf68PF6X
Add a sentence to the "Supported date keywords" advanced-search section
noting that multi-word date keywords must be quoted (e.g.
added:"previous month") -- whoosh-compat requires quoting where the
unquoted form used to work. Also document that the old undocumented
*_id field aliases (tag_id, owner_id, viewer_id, correspondent_id,
document_type_id, storage_path_id, type_id, path_id) are no longer
recognized: a query using one now silently folds to a literal-text
search instead of matching the intended structured field.
The library changed after these were written and more changes are already
decided upstream. Records what is wrong today, what to write toward, the
one question still open, and the fast-JSON-field trap, rather than
silently leaving code that would fail on contact.
16 bite-sized, TDD tasks across the design spec's 4-PR stack, each with
a suggested subagent type/model for delegated execution. Test/fixture
code in the acceptance-corpus and API-expansion tasks was verified
against the real codebase (documents/tests/search/conftest.py's
existing backend/index fixtures, test_backend.py's pytestmark
convention, CustomFieldInstance's typed value_text field) rather than
guessed, and the date-grammar parity test's AST-shape assumption was
confirmed by actually running whoosh_compat.parse() against a real
DATE FieldRegistry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Agent review (source-verified against both repos) confirmed the spec's
claims accurate throughout, with one real gap: the JSON-subpath
tantivy-py carve-out (index.parse_query fallback for notes.*/
custom_fields.* until tantivy-py#716 ships) interacts with paperless's
pinned tantivy~=0.26.0 and wasn't mentioned. Also added two footnotes:
FieldRegistry forces date_only=True on any DATE spec regardless of the
PublicField default, and the date-grammar parity audit implicitly
grants new keyword vocabulary as a side effect.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Design for replacing _translate.py/_dates.py with whoosh-compat: shared
field-definition table driving both the Tantivy schema and the query
FieldRegistry, diagnostics->exception mapping (aggregating all errors,
not just the first), a 4-PR stack with no rollout flag, and a
result-level acceptance corpus + date-grammar parity audit as the
safety net instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Feature: Allow configuring the compression type and compression levels during export
Building on the zip export improvements, this now allows users to further configure the
zip to fit their needs. A simple stored zip for speed, or a high compression zstd for
the smallest archive. Full validation of the method and levels at the command line
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
Added a new --url argument to specify the base URL of the Paperless instance, allowing matched documents to be displayed as clickable links. Updated the logic to fetch document titles based on the presence of the base URL.
* Have the remote parser respect the provided produce_archive_file setting, as already determined via the consumer checks
* Updates the documentation to be correct about the respecting now
* merge conflict fixing
Addresses discussion #13373. Documents that password removal creates a
new document version via re-consumption of the decrypted file rather
than editing in place, and explains why the Consumption Started trigger
produces an initial un-OCR'd version followed by a properly processed
one.
stumpylogandClaude Opus 5
JaydenandGitHub
SandroandGitHub
dependabot[bot]andGitHub
Albert MikaelyanandGitHub