These prefixes were never documented public API (undocumented internal
fields the old KNOWN_FIELDS happened to accept), so their removal isn't a
user-facing regression worth calling out in usage.md. The behavior is still
covered by test_acceptance.py's TestUnregisteredIdFieldFoldsToLiteralText.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RVj8NFy821G3YhNf68PF6X
A prior commit deleted test_query.py's parametrized "doesn't raise" coverage
for this shape (created:[...T...Z TO ...] and comma-combined ranges), which
was also the only place PR #13010's T/Z backward-compat guarantee was
exercised. Nothing in paperless's suite proved the full parse_user_query() ->
tantivy Query -> matched-document pipeline still honors it after the
whoosh-compat grammar fix (commit f936143 in the whoosh-compat repo). Add
result-level acceptance cases: an in/out-of-range T/Z bracket range, PR
#13010's original comma-combined two-field shape, and an exact-boundary case
proving a Z-suffixed bound is absolute UTC, not shifted by the local search
timezone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RVj8NFy821G3YhNf68PF6X
- Update stale test comments in test_query.py that described string
rewriting / raw-query fallback behavior that no longer exists post
whoosh-compat migration; rename
test_date_rewriting_applied_before_tantivy_parse to
test_date_keyword_resolves_without_raising to match.
- views.py: move the local MultipleSearchQueryErrors import up into the
existing local-import block near the top of list(), consistent with
the other documents.search imports there, instead of importing it
again inside the except SearchQueryError clause.
- test_api_search.py: assert response.status_code explicitly before
indexing into response.data["results"] in
test_search_added_previous_month_excludes_next_period_start, and tie
the xfail marker to AssertionError instead of the incidental KeyError
that indexing a 400 response's missing "results" key produced.
Add a sentence to the "Supported date keywords" advanced-search section
noting that multi-word date keywords must be quoted (e.g.
added:"previous month") -- whoosh-compat requires quoting where the
unquoted form used to work. Also document that the old undocumented
*_id field aliases (tag_id, owner_id, viewer_id, correspondent_id,
document_type_id, storage_path_id, type_id, path_id) are no longer
recognized: a query using one now silently folds to a literal-text
search instead of matching the intended structured field.
test_unregistered_id_field_folds_to_literal_text_not_error only checked
that parse_user_query() didn't raise for a query like tag_id:5. Add a
result-level acceptance test (matching test_acceptance.py's
_matched_ids pattern, indexed against real documents) that asserts the
matched-document-ID set is genuinely empty, not just that the parse
step succeeds.
The fuzzy blend clause in parse_user_query() fed the raw, whoosh-syntax
query string directly to tantivy's own query parser. Since the
whoosh-compat migration, raw_query still contains whoosh grammar (date
keywords, whoosh-style ranges, bracket-class wildcards) that tantivy's
parser rejects with ValueError, which escaped parse_user_query and
turned into a generic HTTP 400 for the entire query whenever
ADVANCED_FUZZY_SEARCH_THRESHOLD was configured.
Deriving a clean plain-text-only extraction for the fuzzy clause was
ruled out: wc.parse() already expands unfielded terms into per-default-
field copies in the AST, so there's no "still unfielded" marker left to
walk without duplicating whoosh-compat's own expansion logic. Instead,
scope a narrow try/except ValueError around exactly the
index.parse_query() call and skip the fuzzy clause (logged at debug)
when it can't parse, leaving the exact/CJK clauses unaffected.
Replaces test_query.py's intermediate-AST/query-string checks with a
result-level acceptance corpus that indexes real documents and asserts
matched-ID sets through parse_user_query(), covering the #13568
bracket-wildcard regression, comma value lists, field boosts, JSON subpaths,
and Multitoken-in-OR nesting. Removes TestCreatedDateField, TestDateTimeFields,
TestWhooshQueryRewriting, TestYearRangeRewriting, TestNonDateFieldsNotRewritten,
TestPassthrough, TestNormalizeQuery, and TestParseUserQuery's
test_advanced_search_queries_do_not_raise from test_query.py, since they test
translate_query/_dates.py internals or a diagnostics-free-parse guarantee
whoosh-compat's own suite already covers.
When parse_user_query() raises MultipleSearchQueryErrors due to multiple
field parsing failures (e.g. both an invalid date and an invalid number
in a single query), the exception handler now surfaces all error messages
in the 400 response, allowing users to fix them all in one round-trip
instead of discovering them one at a time.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
whoosh-compat's date grammar now accepts "T" as a date/time separator
and a trailing "Z" UTC designator (paperless-ngx PR #13010
back-compat), so these advanced-search queries no longer raise.
Rewires parse_user_query() to parse via wc.parse()/tantivy_emit() against
the shared FieldRegistry instead of the string-based translate_query()
pipeline, so diagnostics map to typed SearchQueryError subclasses
(InvalidDateQuery/InvalidNumberQuery/MultipleSearchQueryErrors) and every
bad field is reported, not just the first.
Marks three pre-existing tests xfail (2 in test_query.py, 1 in
test_api_search.py) for confirmed whoosh-compat grammar gaps found while
verifying this rewrite: unquoted multi-word date keywords (e.g.
`added:previous month`) and RFC3339 T/Z datetime range bounds no longer
parse.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move SearchQueryError and InvalidDateQuery from _translate.py to _query.py and
add two new exception classes: InvalidNumberQuery and MultipleSearchQueryErrors.
Update _translate.py to re-export the exceptions for backward compatibility
until the translation module is removed. Update __init__.py to export all
four exception classes from _query.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Create the shared field-definition table consumed by the schema builder
(_schema.py) and the whoosh-compat field registry (_registry.py). This
eliminates drift between what the index exposes and what queries can address.
- Create PublicField frozen dataclass with field metadata
- Define PUBLIC_FIELDS tuple with 16 searchable fields
- Add comprehensive test suite covering field properties
The whoosh-compat pyproject.toml dependency addition is added in a
follow-up commit, with the correct [tantivy] extra, source comment, and a
matching uv.lock update.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The library changed after these were written and more changes are already
decided upstream. Records what is wrong today, what to write toward, the
one question still open, and the fast-JSON-field trap, rather than
silently leaving code that would fail on contact.
Field references became a typed value rather than a dotted string, so
diagnostics carry one too and the registry exposes a single resolver.
Also records that the JSON fields must stay non-fast while existence
checks against a fast JSON field return inverted results.
Diagnostics now carry field and raw_value, so the transition guidance
points at those instead of parsing human-readable message text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
16 bite-sized, TDD tasks across the design spec's 4-PR stack, each with
a suggested subagent type/model for delegated execution. Test/fixture
code in the acceptance-corpus and API-expansion tasks was verified
against the real codebase (documents/tests/search/conftest.py's
existing backend/index fixtures, test_backend.py's pytestmark
convention, CustomFieldInstance's typed value_text field) rather than
guessed, and the date-grammar parity test's AST-shape assumption was
confirmed by actually running whoosh_compat.parse() against a real
DATE FieldRegistry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Agent review (source-verified against both repos) confirmed the spec's
claims accurate throughout, with one real gap: the JSON-subpath
tantivy-py carve-out (index.parse_query fallback for notes.*/
custom_fields.* until tantivy-py#716 ships) interacts with paperless's
pinned tantivy~=0.26.0 and wasn't mentioned. Also added two footnotes:
FieldRegistry forces date_only=True on any DATE spec regardless of the
PublicField default, and the date-grammar parity audit implicitly
grants new keyword vocabulary as a side effect.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Design for replacing _translate.py/_dates.py with whoosh-compat: shared
field-definition table driving both the Tantivy schema and the query
FieldRegistry, diagnostics->exception mapping (aggregating all errors,
not just the first), a 4-PR stack with no rollout flag, and a
result-level acceptance corpus + date-grammar parity audit as the
safety net instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Encodes the settled integration decisions for replacing the
hand-maintained search translation layer with whoosh-compat:
user-typed query surface policy, analyzer seam, diagnostics-before-emit
contract, mandatory date parity audit, test churn, and rollout plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AI Suggestions previously invented near-duplicate metadata because the classification
prompt had no knowledge of the installation's own taxonomy. This surfaces
a small, ranked, permission-filtered set of existing tags/document
types/correspondents/storage paths - drawn from the document's RAG
neighbors plus its own already-assigned metadata - so the model prefers
reusing what already exists.
The LLM response schema now returns existing_ids (IDs of reused
candidates) separately from new_names (genuinely new suggestions).
Only new_names goes through localization and fuzzy name-matching;
existing_ids is resolved deterministically and never touched by the
localization pass, so exact matches can no longer be silently
corrupted by translation.
Trenton HolmesandClaude Sonnet 5
shamoon
GitHub Actions