fix(search): restore unquoted multi-word date keywords via pre-parse quoting

"added:previous month" returned HTTP 400 after the whoosh-compat
migration. The unquoted spelling was never parser-native anywhere: v2
rewrote it to explicit bracket ranges app-side before whoosh saw the
string, and the deleted translation layer consumed it itself, so users
and saved views have relied on it continuously while whoosh-compat
deliberately scopes it out of its parser (its DIVERGENCES.md entry 19)
and understands the phrases natively only as quoted values.

parse_user_query now quotes the closed six-phrase vocabulary (previous
week/month/quarter/year, this month/year) when it directly follows a
date field's colon, before parsing. Only quoting happens app-side; every
date computation stays in whoosh-compat's grammar, unlike v2's rewrite,
which computed the ranges itself. Date field names derive from
PUBLIC_FIELDS, the field name matches case-sensitively (the parser's own
field tagging is case-sensitive), the phrase case-insensitively (the
grammar accepts any case in the quoted form), and already-quoted
spellings, TEXT fields, unfielded words and bracketed ranges are
untouched.

The previously xfailed end-to-end regression test now passes as a plain
test, and a new acceptance class pins unquoted == quoted == mixed-case
result sets on a boundary fixture, no-error parsing for the whole
vocabulary across all three date fields, and that "title:previous month"
stays an ordinary text search. docs/usage.md now states the two
spellings are equivalent after a date field.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WMsn6DgzbvSqh1pwy66VVF
This commit is contained in:
Trenton Holmes
2026-08-18 11:05:04 -07:00
co-authored by Claude Fable 5
parent a717684a60
commit 7bab9622c8
4 changed files with 154 additions and 16 deletions
+59 -1
View File
@@ -13,6 +13,7 @@ from whoosh_compat.errors import Diagnostic
from whoosh_compat.errors import DiagnosticKind
from whoosh_compat.errors import UnsupportedQueryError
from documents.search._fields import PUBLIC_FIELDS
from documents.search._registry import get_field_registry
from documents.search._tokenizer import simple_search_tokens
@@ -70,6 +71,59 @@ _REGEX_TIMEOUT: Final[float] = 1.0
# Uses Unicode properties to cover all blocks including Extension B+ planes.
_CJK_RE: Final = regex.compile(r"[\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}]+")
# The closed multi-word date-keyword vocabulary, unchanged since paperless
# v2's rewrite_natural_date_keywords. whoosh-compat's date grammar
# understands every one of these natively, but only as a QUOTED value
# (its DIVERGENCES.md entry 19: unquoted multi-word values split at
# whitespace, faithfully to whoosh); the unquoted spelling has been
# honored continuously since the whoosh era by an app-level assist, so
# _quote_date_keyword_phrases below keeps honoring it by inserting the
# quotes and nothing else. Single-word keywords (today, yesterday) parse
# unquoted already and need no entry.
_DATE_KEYWORD_PHRASES: Final = (
"previous week",
"previous month",
"previous quarter",
"previous year",
"this month",
"this year",
)
# Field names are case-sensitive (matching the parser's own field
# tagging); the keyword phrase is case-insensitive (matching the date
# grammar's leniency for the quoted form). Date fields derived from
# PUBLIC_FIELDS, never hand-listed.
_DATE_KEYWORD_PHRASE_RE: Final = regex.compile(
r"\b("
+ "|".join(
regex.escape(f.name)
for f in PUBLIC_FIELDS
if f.kind in (wc.FieldKind.DATE, wc.FieldKind.DATETIME)
)
+ r"):((?i:"
+ "|".join(_DATE_KEYWORD_PHRASES)
+ r"))\b",
)
def _quote_date_keyword_phrases(raw_query: str) -> str:
"""Quote unquoted multi-word date keyword phrases on date fields.
``added:previous month`` becomes ``added:"previous month"``; the
already-quoted spellings don't match the pattern (the colon must be
followed directly by the phrase), and the same words after a TEXT
field or standing alone are ordinary text and untouched. Only quoting
happens here: every date computation stays in whoosh-compat's
grammar, which parses exactly this phrase vocabulary as quoted
values. This is deliberately NOT a revival of the deleted
translation layer, which computed the ranges app-side.
"""
return _DATE_KEYWORD_PHRASE_RE.sub(
r'\1:"\2"',
raw_query,
timeout=_REGEX_TIMEOUT,
)
def _has_cjk(text: str) -> bool:
"""Return True if text contains any CJK characters."""
@@ -285,7 +339,10 @@ def parse_user_query(
"""
Parse user query through whoosh-compat, then blend in fuzzy/CJK clauses.
1. wc.parse() against the shared FieldRegistry (whoosh grammar -> AST).
1. Unquoted multi-word date keyword phrases on date fields are quoted
(_quote_date_keyword_phrases) so the historically honored
"added:previous month" spelling keeps working; then wc.parse()
against the shared FieldRegistry (whoosh grammar -> AST).
2. Any diagnostics (bad dates/numbers) map to SearchQueryError subclasses
and raise — the view returns HTTP 400 with every offending field
listed, not just the first.
@@ -303,6 +360,7 @@ def parse_user_query(
never went through the pre-whoosh-compat translation layer either.
"""
registry = get_field_registry(settings.SEARCH_LANGUAGE)
raw_query = _quote_date_keyword_phrases(raw_query)
result = wc.parse(
raw_query,
registry=registry,