mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-08-18 08:43:22 +00:00
More done work
This commit is contained in:
Trenton Holmes
committed by
stumpylog
stumpylog
parent
60e4715a00
commit
b47cc92b29
@@ -0,0 +1,462 @@
|
||||
# Unicode NFC Normalization for Filesystem Paths Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Ensure all filesystem paths stored in the database and written to disk use NFC Unicode normalization, preventing "file not found" failures caused by byte-level mismatches between visually identical filenames (e.g., NFD `ü` = `u + combining diaeresis` vs NFC `ü` = single codepoint U+00FC).
|
||||
|
||||
**Architecture:** The fix has two layers. The primary fix normalizes the output of `clean_filepath()` in `FilePathTemplate.render()` — this is the single choke point through which all template-rendered filenames pass. Defense-in-depth changes normalize input strings before `pathvalidate.sanitize_filename()` in the context builder functions. A separate fix normalizes mail attachment filenames at the entry point. Existing documents with NFD paths will be transparently migrated to NFC on their next save (the file move logic already handles the case where old and new paths differ).
|
||||
|
||||
**Tech Stack:** Python `unicodedata.normalize('NFC', ...)`, `pathvalidate`, Django, Jinja2, pytest
|
||||
|
||||
---
|
||||
|
||||
## Background: The Bug
|
||||
|
||||
`pathvalidate.sanitize_filename()` removes illegal filesystem characters but does **not** normalize Unicode. NFC `ü` (UTF-8: `c3 bc`) and NFD `ü` (UTF-8: `75 cc 88`) are visually identical but produce different byte sequences. On Linux filesystems with no normalization (default ZFS, ext4), these are treated as distinct filenames. If an LLM or OCR engine produces NFD text for a document title, the generated filesystem path contains NFD bytes. If the same title is later regenerated in NFC form (LLM output is non-deterministic), the path lookup fails: `old_source_path.is_file()` returns `False` even though a file with the same visual name exists on disk.
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Change |
|
||||
| ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `src/documents/templating/filepath.py` | Add NFC normalization in `clean_filepath()` (primary fix) + input normalization in `get_basic_metadata_context()`, `get_tags_context()`, `get_custom_fields_context()` (defense-in-depth) |
|
||||
| `src/paperless_mail/mail.py` | Normalize attachment filenames before `pathvalidate.sanitize_filename()` |
|
||||
| `src/documents/tests/test_file_handling.py` | Tests for NFC normalization in `generate_filename()` |
|
||||
| `src/paperless_mail/tests/test_mail.py` | Tests for NFC normalization in mail attachment handling |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Normalize `clean_filepath()` output (primary fix)
|
||||
|
||||
This is the single choke point. ALL template-rendered paths pass through `clean_filepath()` before being stored in `document.filename`. Fixing this alone prevents the bug for every path generated via the filename format system — including `{{ title }}` (sanitized context), `{{ document.title }}` (raw context), `{{ correspondent }}`, and every other template variable.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/documents/templating/filepath.py:36-48`
|
||||
- Test: `src/documents/tests/test_file_handling.py`
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
|
||||
Add these tests to `src/documents/tests/test_file_handling.py`, inside `class TestFileHandling`:
|
||||
|
||||
```python
|
||||
import unicodedata
|
||||
|
||||
@override_settings(FILENAME_FORMAT="{{ title }}")
|
||||
def test_generate_filename_nfc_normalizes_nfd_title(self) -> None:
|
||||
"""NFD title (u + combining diaeresis) must produce NFC path bytes."""
|
||||
nfd_title = unicodedata.normalize("NFD", "Gemüse")
|
||||
nfc_title = unicodedata.normalize("NFC", "Gemüse")
|
||||
assert nfd_title != nfc_title # confirm inputs differ at byte level
|
||||
|
||||
doc = Document.objects.create(title=nfd_title, mime_type="application/pdf")
|
||||
result = generate_filename(doc)
|
||||
|
||||
assert str(result) == f"{nfc_title}.pdf"
|
||||
assert str(result).encode() == f"{nfc_title}.pdf".encode()
|
||||
|
||||
@override_settings(FILENAME_FORMAT="{{ correspondent }}/{{ title }}")
|
||||
def test_generate_filename_nfc_normalizes_nfd_correspondent(self) -> None:
|
||||
"""NFD correspondent name must produce NFC path component."""
|
||||
nfd_name = unicodedata.normalize("NFD", "Müller")
|
||||
nfc_name = unicodedata.normalize("NFC", "Müller")
|
||||
|
||||
correspondent = Correspondent.objects.create(name=nfd_name)
|
||||
doc = Document.objects.create(
|
||||
title="invoice",
|
||||
correspondent=correspondent,
|
||||
mime_type="application/pdf",
|
||||
)
|
||||
result = generate_filename(doc)
|
||||
|
||||
assert str(result) == f"{nfc_name}/invoice.pdf"
|
||||
assert str(result).encode() == f"{nfc_name}/invoice.pdf".encode()
|
||||
|
||||
@override_settings(FILENAME_FORMAT="{{ document.title }}")
|
||||
def test_generate_filename_nfc_normalizes_raw_document_title_in_template(self) -> None:
|
||||
"""NFD title accessed via document.title (unsanitized context) must also be NFC."""
|
||||
nfd_title = unicodedata.normalize("NFD", "Café")
|
||||
nfc_title = unicodedata.normalize("NFC", "Café")
|
||||
|
||||
doc = Document.objects.create(title=nfd_title, mime_type="application/pdf")
|
||||
result = generate_filename(doc)
|
||||
|
||||
assert str(result) == f"{nfc_title}.pdf"
|
||||
assert str(result).encode() == f"{nfc_title}.pdf".encode()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
```bash
|
||||
uv run pytest --override-ini="addopts=" src/documents/tests/test_file_handling.py::TestFileHandling::test_generate_filename_nfc_normalizes_nfd_title src/documents/tests/test_file_handling.py::TestFileHandling::test_generate_filename_nfc_normalizes_nfd_correspondent src/documents/tests/test_file_handling.py::TestFileHandling::test_generate_filename_nfc_normalizes_raw_document_title_in_template -v
|
||||
```
|
||||
|
||||
Expected: all three FAIL (NFD title produces NFD path, assertion fails).
|
||||
|
||||
- [ ] **Step 3: Add NFC normalization to `clean_filepath()`**
|
||||
|
||||
In `src/documents/templating/filepath.py`, add `import unicodedata` at the top of the file and modify `clean_filepath()`:
|
||||
|
||||
```python
|
||||
import unicodedata # add to top-of-file imports
|
||||
|
||||
class FilePathTemplate(Template):
|
||||
def render(self, *args, **kwargs) -> str:
|
||||
def clean_filepath(value: str) -> str:
|
||||
"""
|
||||
Clean up a filepath by:
|
||||
1. Normalizing to NFC Unicode form to prevent byte-level mismatches
|
||||
between visually identical filenames on case-sensitive filesystems
|
||||
2. Removing newlines and carriage returns
|
||||
3. Removing extra spaces before and after forward slashes
|
||||
4. Preserving spaces in other parts of the path
|
||||
"""
|
||||
value = unicodedata.normalize("NFC", value)
|
||||
value = value.replace("\n", "").replace("\r", "")
|
||||
value = re.sub(r"\s*/\s*", "/", value)
|
||||
|
||||
# We remove trailing and leading separators, as these are always relative paths, not absolute, even if the user
|
||||
# tries
|
||||
return value.strip().strip(os.sep)
|
||||
|
||||
original_render = super().render(*args, **kwargs)
|
||||
|
||||
return clean_filepath(original_render)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
```bash
|
||||
uv run pytest --override-ini="addopts=" src/documents/tests/test_file_handling.py::TestFileHandling::test_generate_filename_nfc_normalizes_nfd_title src/documents/tests/test_file_handling.py::TestFileHandling::test_generate_filename_nfc_normalizes_nfd_correspondent src/documents/tests/test_file_handling.py::TestFileHandling::test_generate_filename_nfc_normalizes_raw_document_title_in_template -v
|
||||
```
|
||||
|
||||
Expected: all three PASS.
|
||||
|
||||
- [ ] **Step 5: Run the full file-handling test suite to check for regressions**
|
||||
|
||||
```bash
|
||||
uv run pytest --override-ini="addopts=" src/documents/tests/test_file_handling.py -v
|
||||
```
|
||||
|
||||
Expected: all existing tests continue to pass (ASCII titles are unaffected by NFC normalization).
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add src/documents/templating/filepath.py src/documents/tests/test_file_handling.py
|
||||
git commit -m "Fix: normalize filesystem paths to NFC Unicode to prevent byte-level mismatches"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Defense-in-depth normalization in context builders
|
||||
|
||||
`clean_filepath()` (Task 1) fixes the rendered path. These changes normalize the input strings that go into `pathvalidate.sanitize_filename()` within the context builders — belt-and-suspenders so the sanitized shorthand variables (`{{ title }}`, `{{ correspondent }}`, `{{ tag_list }}`, `{{ custom_fields }}`) are also NFC before sanitization. This matters because the sanitized strings could theoretically be compared directly against DB-stored values in other contexts.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/documents/templating/filepath.py:171-319`
|
||||
- Test: `src/documents/tests/test_file_handling.py`
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
|
||||
Add these tests to `TestFileHandling` in `src/documents/tests/test_file_handling.py`:
|
||||
|
||||
```python
|
||||
@override_settings(FILENAME_FORMAT="{{ tag_list }}/{{ title }}")
|
||||
def test_generate_filename_nfc_normalizes_nfd_tag_list(self) -> None:
|
||||
"""NFD tag names must produce NFC path component in tag_list."""
|
||||
nfd_name = unicodedata.normalize("NFD", "Büro")
|
||||
nfc_name = unicodedata.normalize("NFC", "Büro")
|
||||
|
||||
doc = Document.objects.create(title="doc", mime_type="application/pdf")
|
||||
doc.tags.create(name=nfd_name)
|
||||
result = generate_filename(doc)
|
||||
|
||||
assert str(result) == f"{nfc_name}/doc.pdf"
|
||||
assert str(result).encode() == f"{nfc_name}/doc.pdf".encode()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
```bash
|
||||
uv run pytest --override-ini="addopts=" src/documents/tests/test_file_handling.py::TestFileHandling::test_generate_filename_nfc_normalizes_nfd_tag_list -v
|
||||
```
|
||||
|
||||
Expected: FAIL. (The tag_list is already caught by `clean_filepath()` from Task 1, but we want a test that directly validates input normalization through the sanitize call.)
|
||||
|
||||
Note: this test may already pass after Task 1 due to `clean_filepath()`. If so, keep the test as a regression guard and move straight to the implementation.
|
||||
|
||||
- [ ] **Step 3: Normalize inputs in `get_basic_metadata_context()`**
|
||||
|
||||
In `src/documents/templating/filepath.py`, update `get_basic_metadata_context()`. The `unicodedata` import was added in Task 1.
|
||||
|
||||
```python
|
||||
def get_basic_metadata_context(
|
||||
document: Document,
|
||||
*,
|
||||
no_value_default: str = NO_VALUE_PLACEHOLDER,
|
||||
) -> dict[str, str]:
|
||||
"""
|
||||
Given a Document, constructs some basic information about it. If certain values are not set,
|
||||
they will be replaced with the no_value_default.
|
||||
|
||||
Regardless of set or not, the values will be sanitized
|
||||
"""
|
||||
return {
|
||||
"title": pathvalidate.sanitize_filename(
|
||||
unicodedata.normalize("NFC", document.title),
|
||||
replacement_text="-",
|
||||
),
|
||||
"correspondent": pathvalidate.sanitize_filename(
|
||||
unicodedata.normalize("NFC", document.correspondent.name),
|
||||
replacement_text="-",
|
||||
)
|
||||
if document.correspondent
|
||||
else no_value_default,
|
||||
"document_type": pathvalidate.sanitize_filename(
|
||||
unicodedata.normalize("NFC", document.document_type.name),
|
||||
replacement_text="-",
|
||||
)
|
||||
if document.document_type
|
||||
else no_value_default,
|
||||
"asn": str(document.archive_serial_number)
|
||||
if document.archive_serial_number
|
||||
else no_value_default,
|
||||
"owner_username": document.owner.username
|
||||
if document.owner
|
||||
else no_value_default,
|
||||
"original_name": PurePath(document.original_filename).with_suffix("").name
|
||||
if document.original_filename
|
||||
else no_value_default,
|
||||
"doc_pk": f"{document.pk:07}",
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Normalize inputs in `get_tags_context()`**
|
||||
|
||||
Update `get_tags_context()` in the same file:
|
||||
|
||||
```python
|
||||
def get_tags_context(tags: Iterable[Tag]) -> dict[str, str | list[str]]:
|
||||
"""
|
||||
Given an Iterable of tags, constructs some context from them for usage
|
||||
"""
|
||||
return {
|
||||
"tag_list": pathvalidate.sanitize_filename(
|
||||
",".join(
|
||||
sorted(unicodedata.normalize("NFC", tag.name) for tag in tags),
|
||||
),
|
||||
replacement_text="-",
|
||||
),
|
||||
# Assumed to be ordered, but a template could loop through to find what they want
|
||||
"tag_name_list": [unicodedata.normalize("NFC", x.name) for x in tags],
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Normalize string-type inputs in `get_custom_fields_context()`**
|
||||
|
||||
Update `get_custom_fields_context()` in the same file. Only string-type fields (MONETARY, STRING, URL, LONG_TEXT, SELECT) go through `sanitize_filename()`; the others (dates, numbers, booleans) cannot contain non-ASCII unicode. Also normalize the field name itself.
|
||||
|
||||
```python
|
||||
def get_custom_fields_context(
|
||||
custom_fields: Iterable[CustomFieldInstance],
|
||||
) -> dict[str, dict[str, dict[str, str]]]:
|
||||
"""
|
||||
Given an Iterable of CustomFieldInstance, builds a dictionary mapping the field name
|
||||
to its type and value
|
||||
"""
|
||||
field_data = {"custom_fields": {}}
|
||||
for field_instance in custom_fields:
|
||||
type_ = pathvalidate.sanitize_filename(
|
||||
field_instance.field.data_type,
|
||||
replacement_text="-",
|
||||
)
|
||||
if field_instance.value is None:
|
||||
value = None
|
||||
# String types need to be sanitized
|
||||
elif field_instance.field.data_type in {
|
||||
CustomField.FieldDataType.MONETARY,
|
||||
CustomField.FieldDataType.STRING,
|
||||
CustomField.FieldDataType.URL,
|
||||
CustomField.FieldDataType.LONG_TEXT,
|
||||
}:
|
||||
value = pathvalidate.sanitize_filename(
|
||||
unicodedata.normalize("NFC", field_instance.value),
|
||||
replacement_text="-",
|
||||
)
|
||||
elif (
|
||||
field_instance.field.data_type == CustomField.FieldDataType.SELECT
|
||||
and field_instance.field.extra_data["select_options"] is not None
|
||||
):
|
||||
options = field_instance.field.extra_data["select_options"]
|
||||
value = pathvalidate.sanitize_filename(
|
||||
unicodedata.normalize(
|
||||
"NFC",
|
||||
next(
|
||||
option["label"]
|
||||
for option in options
|
||||
if option["id"] == field_instance.value
|
||||
),
|
||||
),
|
||||
replacement_text="-",
|
||||
)
|
||||
else:
|
||||
value = field_instance.value
|
||||
field_data["custom_fields"][
|
||||
pathvalidate.sanitize_filename(
|
||||
unicodedata.normalize("NFC", field_instance.field.name),
|
||||
replacement_text="-",
|
||||
)
|
||||
] = {
|
||||
"type": type_,
|
||||
"value": value,
|
||||
}
|
||||
return field_data
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the new test and full test suite**
|
||||
|
||||
```bash
|
||||
uv run pytest --override-ini="addopts=" src/documents/tests/test_file_handling.py -v
|
||||
```
|
||||
|
||||
Expected: all tests pass, including the new tag test.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add src/documents/templating/filepath.py src/documents/tests/test_file_handling.py
|
||||
git commit -m "Fix: normalize context builder inputs to NFC before sanitize_filename (defense-in-depth)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Normalize mail attachment filenames
|
||||
|
||||
Email attachment filenames come from MIME headers and can be in any Unicode normalization depending on the sending client. These flow into `document.original_filename` and then into `{{ original_name }}` template context. They also become the temp file name created on disk.
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `src/paperless_mail/mail.py`
|
||||
- Test: `src/paperless_mail/tests/test_mail.py`
|
||||
|
||||
- [ ] **Step 1: Find the exact lines in mail.py**
|
||||
|
||||
```bash
|
||||
grep -n "sanitize_filename" src/paperless_mail/mail.py
|
||||
```
|
||||
|
||||
Expected output (line numbers may vary):
|
||||
|
||||
```
|
||||
NNN: attachment_name = pathvalidate.sanitize_filename(att.filename)
|
||||
NNN: filename=pathvalidate.sanitize_filename(att.filename),
|
||||
NNN: filename=pathvalidate.sanitize_filename(f"{message.subject}.eml"),
|
||||
```
|
||||
|
||||
Note the line numbers for the next step.
|
||||
|
||||
- [ ] **Step 2: Write a failing test**
|
||||
|
||||
Find an existing test in `src/paperless_mail/tests/test_mail.py` that exercises attachment filename handling (search for `sanitize_filename` or `att.filename` in that file to find a good base test to copy). Add a new test that uses an NFD attachment filename.
|
||||
|
||||
The following test goes into the appropriate `TestCase` class in `src/paperless_mail/tests/test_mail.py`. Look at the file first to confirm the right class and mock patterns — the test below follows the existing pattern for mocking `MailMessage` and `Attachment` objects:
|
||||
|
||||
```python
|
||||
def test_attachment_filename_nfd_normalized_to_nfc(self) -> None:
|
||||
"""Mail attachment filenames with NFD encoding must be normalized to NFC."""
|
||||
import unicodedata
|
||||
nfd_name = unicodedata.normalize("NFD", "Rechnung März.pdf")
|
||||
nfc_name = unicodedata.normalize("NFC", "Rechnung März.pdf")
|
||||
assert nfd_name != nfc_name # confirm inputs differ at byte level
|
||||
|
||||
# Use whatever mock/factory pattern exists in this test file for creating
|
||||
# a fake attachment with a specific filename, then run the mail handler,
|
||||
# and assert that document.original_filename == nfc_name (not nfd_name).
|
||||
# Adapt the mock setup to match the test file's existing patterns exactly.
|
||||
```
|
||||
|
||||
To find the right mock pattern: `grep -n "att.filename\|Attachment\|MailMessage\|MagicMock" src/paperless_mail/tests/test_mail.py | head -20`
|
||||
|
||||
- [ ] **Step 3: Run the test to verify it fails**
|
||||
|
||||
```bash
|
||||
uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py -k "test_attachment_filename_nfd" -v
|
||||
```
|
||||
|
||||
Expected: FAIL.
|
||||
|
||||
- [ ] **Step 4: Add `import unicodedata` to mail.py**
|
||||
|
||||
At the top of `src/paperless_mail/mail.py`, add:
|
||||
|
||||
```python
|
||||
import unicodedata
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Normalize attachment filenames in mail.py**
|
||||
|
||||
At each of the three `pathvalidate.sanitize_filename` call sites found in Step 1, wrap the input string with `unicodedata.normalize("NFC", ...)`:
|
||||
|
||||
For the attachment temp file creation:
|
||||
|
||||
```python
|
||||
attachment_name = pathvalidate.sanitize_filename(
|
||||
unicodedata.normalize("NFC", att.filename)
|
||||
)
|
||||
```
|
||||
|
||||
For the metadata override filename:
|
||||
|
||||
```python
|
||||
filename=pathvalidate.sanitize_filename(
|
||||
unicodedata.normalize("NFC", att.filename)
|
||||
),
|
||||
```
|
||||
|
||||
For the EML subject filename:
|
||||
|
||||
```python
|
||||
filename=pathvalidate.sanitize_filename(
|
||||
unicodedata.normalize("NFC", f"{message.subject}.eml")
|
||||
),
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the mail test suite**
|
||||
|
||||
```bash
|
||||
uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py -v
|
||||
```
|
||||
|
||||
Expected: all tests pass, including the new NFD normalization test.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add src/paperless_mail/mail.py src/paperless_mail/tests/test_mail.py
|
||||
git commit -m "Fix: normalize mail attachment filenames to NFC Unicode"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Checklist
|
||||
|
||||
### Spec coverage
|
||||
|
||||
| Requirement | Covered by |
|
||||
| --------------------------------------------------------- | ----------------------------------------------------- |
|
||||
| `clean_filepath()` normalizes all template-rendered paths | Task 1 Step 3 |
|
||||
| `{{ title }}` (sanitized context) produces NFC output | Task 1 test + Task 2 Step 3 |
|
||||
| `{{ document.title }}` (raw context) produces NFC output | Task 1 test |
|
||||
| `{{ correspondent }}` produces NFC output | Task 1 test + Task 2 Step 3 |
|
||||
| `{{ tag_list }}` and `tag_name_list` produce NFC output | Task 2 Steps 1+4 |
|
||||
| Custom field string values produce NFC output | Task 2 Step 5 |
|
||||
| Mail attachment filenames normalized at entry point | Task 3 |
|
||||
| Existing NFD files auto-migrate to NFC on next save | Handled by existing move logic; no code change needed |
|
||||
|
||||
### Notes for implementer
|
||||
|
||||
- The `FILENAME_FORMAT` setting accepts old-style `{title}` format strings, which `convert_format_str_to_template_format()` converts to Jinja2 `{{ title }}` before rendering. Tests using `@override_settings(FILENAME_FORMAT="{{ title }}")` use Jinja2 syntax directly.
|
||||
- Run tests with `--override-ini="addopts="` to disable coverage and parallelism for faster iteration.
|
||||
- The `unicodedata` module is part of the Python standard library — no new dependency.
|
||||
- NFC is the right normalization form for filenames: it is the default on macOS (HFS+/APFS) and the form most databases and text processing tools produce. NFD is what macOS HFS+ _internally_ normalizes to when writing (but presents as NFC), and what some OCR/LLM outputs occasionally produce.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,407 @@
|
||||
# Design: Whoosh→Tantivy Advanced-Query Translation Layer
|
||||
|
||||
**Date:** 2026-06-14
|
||||
**Status:** Phase 1 implemented on branch `fix/search-query-translation` (string-pipeline translation layer in `_translate.py`/`_dates.py`, wired into `parse_user_query`). Phase 2 (Query objects) remains gated on the tantivy-py release noted in §8/§9. Plan: `docs/superpowers/plans/2026-06-14-search-query-translation.md`.
|
||||
**Branch context:** `beta`. Search code: `src/documents/search/`.
|
||||
**Related:** `SEARCH_TANTIVY_WHOOSH_COMPAT.md` (repo root) — full empirical gap matrix and reproduction harnesses. Open branch `fix/scope-comma-expansion` (commit `d8fa97232`) — partial comma fix this design subsumes.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem
|
||||
|
||||
Paperless migrated full-text search from Whoosh (v2) to Tantivy (v3, commit `aed9abe48`, #12471). A
|
||||
compatibility layer in `_query.py` rewrites old Whoosh query syntax into Tantivy syntax via a stack of
|
||||
ordered regex substitutions before calling `tantivy.Index.parse_query`.
|
||||
|
||||
That regex stack is piecemeal and has hit its complexity ceiling:
|
||||
|
||||
- **No structural awareness.** It runs regex on a flat string, so it cannot distinguish a comma inside
|
||||
`[...]` from a top-level clause separator, or know whether a `:` is a field prefix or text. This causes
|
||||
real bugs (e.g. `title:x,created:[2020 TO 2021]` rewrites to malformed `title:x AND title:created:[...]`).
|
||||
- **Order-dependence.** Six rewriters with implicit ordering contracts (14-digit before 8-digit, year-range
|
||||
before 8-digit, etc.). Each new date form means reasoning about all interactions again.
|
||||
|
||||
The result is a class of v2-valid queries that now return **HTTP 400**. There is no fallback: any syntax
|
||||
Tantivy rejects raises out of `parse_query`, propagates through `_backend.py` (no try/except), and is caught
|
||||
by the generic handler in `views.py:2471-2475` → `HttpResponseBadRequest`, with the real error only in logs.
|
||||
|
||||
### Confirmed regressions (empirically reproduced; full table in `SEARCH_TANTIVY_WHOOSH_COMPAT.md` §5)
|
||||
|
||||
| Class | Example | Today | Whoosh v2 |
|
||||
| ------------------------ | -------------------------------------------------------------- | ---------------------- | --------------------------- |
|
||||
| Bare date on date field | `created:2020`, `created:202003` | 400 | full-year / full-month span |
|
||||
| Bracketed absolute range | `created:[20200101 TO 20201231]`, `[2020-01-01 TO 2020-12-31]` | 400 | floor/ceil range |
|
||||
| Open-ended range | `created:[2020 to]`, `created:[to 2020]` | 400 | `>=` / `<=` range |
|
||||
| Comma between clauses | `title:x,created:[...]` | 400 (malformed) | AND, both sides |
|
||||
| Comma value-list scope | `tag:foo,type:bar` | wrong (`tag:type:bar`) | `tag:foo AND type:bar` |
|
||||
| Invalid date | `created:202023` | 400 | NullQuery (no-match) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Goals / Non-goals
|
||||
|
||||
**Goals**
|
||||
|
||||
- Eliminate the date- and comma-class 400s by translating those forms to valid Tantivy syntax.
|
||||
- Replace the order-dependent regex stack with a structural, context-aware pass.
|
||||
- Match empirically-verified Whoosh v2 semantics (see §3).
|
||||
- Additive tests: existing suite stays green during transition.
|
||||
- **Field-name aliasing for the four renamed Whoosh→Tantivy fields** (added to scope 2026-06-14):
|
||||
`type`→`document_type`, `type_id`→`document_type_id`, `path`→`storage_path`, `path_id`→`storage_path_id`.
|
||||
These are the only fields the Tantivy migration renamed; v2 queries using the old names currently 400.
|
||||
Both old and new spellings work after aliasing (new names pass through verbatim). The alias targets are the
|
||||
text "name" fields (`document_type` is populated from `document_type.name`), so `type:invoice` →
|
||||
`document_type:invoice` is correct. Fields with no Tantivy equivalent (`owner`, the `has_*` booleans,
|
||||
`is_shared`, `custom_field_count`, `custom_fields_id`) are NOT aliased and remain out of scope.
|
||||
|
||||
**Non-goals (explicitly out of scope)**
|
||||
|
||||
- Full Whoosh query-language parity.
|
||||
- Other Whoosh divergences: unknown-field-degrades-to-text (`http://x/a,b` → 400 on the `http:` unknown
|
||||
field), tolerant unbalanced parens, case-insensitive `AND/OR/NOT`. These pass through to Tantivy unchanged
|
||||
and are recorded as separate, known gaps (§10).
|
||||
- `>`/`<`/`>=`/`<=` comparison operators — never supported in paperless-Whoosh (no `GtLtPlugin`); adding them
|
||||
would be a new feature, not a compat fix.
|
||||
|
||||
---
|
||||
|
||||
## 3. Empirical ground truth (verified, not inferred)
|
||||
|
||||
Both engines were run directly; do not regress these without re-checking.
|
||||
|
||||
**Whoosh v2** (paperless's exact `MultifieldParser([...]) + DateParserPlugin(basedate=...)` setup):
|
||||
|
||||
- `created:2020` → `DateRange(2020-01-01 .. 2020-12-31 23:59:59)`; `created:202003` → March 2020.
|
||||
- `created:202023` (month 23) → `<_NullQuery>` — **invalid dates match nothing, never error.**
|
||||
- `created:[202001 TO 202006]` → floor/ceil partial-date bounds; `[2020 to]` / `[to 2020]` → open bounds.
|
||||
- `created:-1week` → an exact-microsecond `Term` — parsed but matches ~nothing (useless in v2).
|
||||
- Comma = AND between clauses, both preserved: `created:[r],added:[r]`, `correspondent:acme,created:[...]`,
|
||||
`invoice,created:2020`.
|
||||
- Comma value-list **only** for `KEYWORD(commas=True)` fields (`tag`, `tag_id`, `viewer_id`):
|
||||
`tag:a,b` → `tag:a AND tag:b`. Text-field commas (`correspondent:foo,bar`, `title:10,20`) are split by the
|
||||
field **analyzer** at parse time, not the comma plugin.
|
||||
- `title:x,created:[...]` → only the DateRange (Whoosh drops `title:x`) — a v2 free-mode **bug**; the correct
|
||||
target keeps both sides.
|
||||
|
||||
**Tantivy 0.26.0** (`tantivy v0.26.0, index_format v7`):
|
||||
|
||||
- Date fields require RFC3339 (`...Z`) literals; rejects bare `2020`, `20200101`, `2020-01-01`, lowercase
|
||||
open ranges.
|
||||
- Text-field commas parse fine verbatim (`correspondent:foo,bar`, `title:10,20`, `content:a,b,c`).
|
||||
- Boolean/paren/phrase structure parses correctly, so a translated date token can sit anywhere:
|
||||
`created:[...Z TO ...Z] OR foo` and `(created:[...] OR foo)` both parse.
|
||||
- String date sentinels `0001-01-01T00:00:00Z` and `9999-12-31T23:59:59Z` both parse on a date field.
|
||||
|
||||
---
|
||||
|
||||
## 4. Architecture (Approach 1: flat tokenizing scanner + single date translator)
|
||||
|
||||
The scanner specializes only the date/comma tokens and treats everything else (operators, parens, phrases,
|
||||
words, wildcards) as opaque passthrough. Tantivy keeps doing boolean/grouping/phrase parsing. A `field:value`
|
||||
span is locally recognizable regardless of surrounding boolean context, so the scanner needs no understanding
|
||||
of `AND/OR/NOT`.
|
||||
|
||||
### 4.1 Module layout
|
||||
|
||||
New module `src/documents/search/_translate.py` — single source of truth:
|
||||
|
||||
```
|
||||
translate_query(raw: str, tz) -> str # top-level: scan → transform → recombine
|
||||
scan(raw) -> list[Token] # depth-aware char-walk tokenizer
|
||||
_resolve_commas(tokens) -> list[Token] # comma → AND / value-list / literal
|
||||
translate_date_value(field, raw, tz) -> str # shape-dispatch date translator
|
||||
```
|
||||
|
||||
Date-boundary math (`_date_only_range`, `_datetime_range`, floor/ceil helpers) **moves** from `_query.py`
|
||||
into `_translate.py` (or a small shared `_dates.py`) so there is one home. The existing math is reused
|
||||
verbatim — not rewritten.
|
||||
|
||||
### 4.2 Data flow
|
||||
|
||||
```
|
||||
parse_user_query(raw, tz)
|
||||
→ translate_query(raw, tz) # NEW pipeline
|
||||
→ index.parse_query(translated, DEFAULT_SEARCH_FIELDS, field_boosts=_FIELD_BOOSTS)
|
||||
```
|
||||
|
||||
### 4.3 Transition (delegate + planned removal)
|
||||
|
||||
- `rewrite_natural_date_keywords` and `normalize_query` become thin delegators to `translate_query` (or its
|
||||
sub-steps) so their existing assertions still pass.
|
||||
- The plan **explicitly schedules deleting both functions and their string-output tests** once
|
||||
`test_translate.py` covers them. Single source of truth, no lingering dead code.
|
||||
|
||||
### 4.4 Safety net
|
||||
|
||||
`parse_user_query` wraps `translate_query` in try/except. On any unexpected scanner error it falls back to the
|
||||
**raw** query string (today's behavior) and logs a warning. The new layer can never regress below current
|
||||
behavior; worst case equals the status quo.
|
||||
|
||||
---
|
||||
|
||||
## 5. Scanner token model
|
||||
|
||||
`scan()` is a single left-to-right char walk tracking **quote state** and **`[]`/`{}` bracket depth**. Token
|
||||
kinds:
|
||||
|
||||
- **`FieldValue(field, value)`** — `field:value`, value a single bare token (no brackets). Recognized when,
|
||||
outside quotes/brackets, it sees `\w+:` followed by a non-bracket value. Value runs until whitespace, a
|
||||
resolved clause-comma, `)`, or end (may itself be quoted: `correspondent:"A B"`).
|
||||
- **`FieldValueList(field, [v1, v2, …])`** — value-list, **only** for `field ∈ {tag, tag_id, viewer_id}`. A
|
||||
`FieldValue` whose value is immediately followed by `,term` runs with **no spaces and no colon** in the
|
||||
continuation terms. The no-colon rule fixes `tag:foo,type:bar` (the `type:bar` is not swallowed).
|
||||
- **`FieldRange(field, open, lo, hi, close)`** — `field:[lo TO hi]` / `{…}`. Split on case-insensitive
|
||||
`TO`; `lo`/`hi` may be empty (open). Consumed to the matching close bracket.
|
||||
- **`Comma`** — emitted only when a depth-0 comma resolves to a clause separator (see §7).
|
||||
- **`Passthrough(raw)`** — everything else, byte-for-byte: operators (`AND OR NOT + -`), parens, bare words,
|
||||
wildcards, phrases/quoted spans, whitespace.
|
||||
|
||||
**Key properties**
|
||||
|
||||
- `field:value` is recognized at any paren depth but **never inside `[]`/`{}` or quotes** — so
|
||||
`(created:2020 OR foo)` still finds the date token, and commas inside `[2020 TO 2021]` or `"a,b"` are never
|
||||
clause separators.
|
||||
- Only date fields (`created`, `modified`, `added`) trigger date translation. Every other `field:value` /
|
||||
`field:range` (`tag:`, `asn:`, unknown fields) and every `Passthrough` is re-emitted verbatim — preserving
|
||||
queries Tantivy already handles.
|
||||
- Multi-valued set is exactly `{tag, tag_id, viewer_id}`. `custom_fields` is now a JSON structure in the index
|
||||
(Whoosh smashed it into a comma-keyword field; the JSON path handles it better) and is **not** comma-split.
|
||||
|
||||
---
|
||||
|
||||
## 6. `translate_date_value` — shape dispatch
|
||||
|
||||
One entry point per token type, both emitting `field:[<ISO-Z> TO <ISO-Z>]`. `created` uses date-only
|
||||
(UTC-midnight) boundaries; `added`/`modified` use local-tz-midnight→UTC. All boundary math reuses the
|
||||
existing tested helpers.
|
||||
|
||||
### Scalar value (`FieldValue` on a date field)
|
||||
|
||||
| Shape | Example | Result | Status |
|
||||
| ----------------------- | ---------------------------------- | ------------------------------------------------------------- | ----------- |
|
||||
| Keyword (opt. quoted) | `created:today`, `"previous week"` | existing keyword ranges | works today |
|
||||
| 4-digit `YYYY` | `created:2020` | full-year span, emitted as `[2020-01-01T…Z TO 2021-01-01T…Z]` | NEW |
|
||||
| 6-digit `YYYYMM` | `created:202003` | month span | NEW |
|
||||
| 8-digit `YYYYMMDD` | `created:20200101` | day span | works today |
|
||||
| 14-digit | `…120000` | exact-second point `[t TO t]` | works today |
|
||||
| ISO dashed | `created:2020-01`, `2020-01-01` | strip separators → digit-precision span | NEW |
|
||||
| Bare relative `-N unit` | `created:-1week` | `[t TO t]` instant (effectively no-match, matches v2) | NEW (P3) |
|
||||
| Invalid / unparsable | `created:202023` | **no-match clause, never 400** | NEW |
|
||||
|
||||
### Range (`FieldRange`)
|
||||
|
||||
Parse each bound with the same shape parser, then `floor(lo)` / `ceil(hi)`:
|
||||
|
||||
- Partial / ISO / 8-digit / 14-digit bounds: `[202001 TO 202006]`, `[2020-01-01 TO 2020-12-31]` — NEW.
|
||||
- `now` bound: `[20200101 TO now]` — NEW.
|
||||
- Open bound (empty side): `[2020 to]`, `[to 2020]` → sentinel far-past floor / far-future ceil (§8) — NEW.
|
||||
- Relative bound: generalize existing `[-N unit to now]` so `-N unit` works on either side.
|
||||
- Reversed (`lo>hi`): swap (existing year-range `min/max` + Whoosh `disambiguated` behavior).
|
||||
- Bare year range `[2005 to 2009]`: unchanged (works today).
|
||||
|
||||
**Boundary convention:** keep the existing "ceil = start of next period, inclusive bracket" (e.g.
|
||||
`[2005-01-01 .. 2010-01-01]`) that current tests encode. Do not switch to Whoosh's `23:59:59.999999`; document
|
||||
the one-instant boundary difference.
|
||||
|
||||
---
|
||||
|
||||
## 7. Comma resolution
|
||||
|
||||
A depth-0 comma is resolved three ways (this single rule set subsumes both `fix/scope-comma-expansion` and
|
||||
the unstaged `]`/`"` fix, and fixes Gap E):
|
||||
|
||||
1. **Value-list** — preceding token is a `FieldValue`/`FieldValueList` on `{tag, tag_id, viewer_id}` and the
|
||||
following continuation is a bare, colon-free term → repeat the field: `tag:a,b,c` → `tag:a AND tag:b AND tag:c`.
|
||||
2. **Clause separator → `AND`** — fires only at a structured boundary:
|
||||
- (a) the comma is preceded by a closing `]` or `"` (`created:[r],added:[r]`, `correspondent:"A B",created:[r]`), or
|
||||
- (b) the comma is followed by a **known schema** `field:` (`title:foo,created:[r]`, `correspondent:foo,created:[r]`).
|
||||
Requiring a _known_ field for (b) prevents `http://x,…`-style misfires.
|
||||
3. **Literal** — anything else (a comma followed by a bare term on a non-multivalue field) stays in place:
|
||||
`correspondent:foo,bar`, `title:10,20`, URLs. Tantivy's analyzer tokenizes these on punctuation, matching
|
||||
Whoosh's analyzer behavior.
|
||||
|
||||
---
|
||||
|
||||
## 8. Open-range handling & the two phases
|
||||
|
||||
**Phase 1 (this work) — string output, no tantivy change.**
|
||||
Open bounds use verified string sentinels: lower-open → `0001-01-01T00:00:00Z`, upper-open → `9999-12-31T23:59:59Z`
|
||||
(both confirmed to parse on a date field in 0.26.0). No-match (invalid date) uses a degenerate date range
|
||||
(exact representation flagged for verification in §11).
|
||||
|
||||
**Phase 2 (stretch) — build `tantivy.Query` objects for date clauses.**
|
||||
`Query.range_query(..., lower_bound=None/upper_bound=None)` gives true open bounds and `empty_query()` gives a
|
||||
real no-match, eliminating all string hacks. **Gated only on a released `tantivy-py` > 0.26.0 that includes
|
||||
#655 + #666 — the code already exists on `tantivy-py` `master`, it just postdates the `0.26.0` wheel we pin
|
||||
(`pyproject.toml`: `tantivy~=0.26.0`); see §9.** Splicing a Query object into an otherwise-string boolean query
|
||||
is non-trivial, so Phase 2 is a separate, later effort; Phase 1 ships independently.
|
||||
|
||||
Phase 2 also folds in the deferred Phase-1 cleanup (maintainer decision, 2026-06-15):
|
||||
|
||||
- Replace the `NO_MATCH` degenerate-range sentinel with `Query.empty_query()` (this also retires the cosmetic
|
||||
issue that `NO_MATCH` always names the `created` field regardless of the queried field).
|
||||
- Replace `OPEN_LO`/`OPEN_HI` string sentinels with `range_query(..., None)` open bounds.
|
||||
- Retire the now-dead `_rewrite_*` helpers and the `rewrite_natural_date_keywords`/`normalize_query` delegation
|
||||
shims in `_query.py` (~160 lines left from the Phase-1 transition), and migrate their string-output tests in
|
||||
`test_query.py` (replace the direct `_rewrite_compact_date` test with a `translate_scalar` test).
|
||||
|
||||
---
|
||||
|
||||
## 9. Upstream tantivy-py contribution (PR-ready detail)
|
||||
|
||||
> **STATUS UPDATE (2026-06-14): already implemented upstream on `master`.** The date-value gap below is
|
||||
> closed by two merged `tantivy-py` commits that postdate the released `0.26.0` wheel we pin:
|
||||
> **#655** (`feat: support unbounded range queries via None bounds`) and **#666** (`fix: add_date loses
|
||||
tzinfo`, which added the `PyDateTime → tantivy DateTime` converter and routed both `range_query` and
|
||||
> `term_query` through it). `range_query` with `datetime` (incl. `None` open bounds) and
|
||||
> `term_query`/`term_set_query` with `datetime` on `Date` fields are verified working upstream with
|
||||
> regression tests. **The Phase 2 blocker is therefore no longer a code contribution** — it is only a
|
||||
> published `tantivy-py` release > `0.26.0` containing #655 + #666, plus bumping our pin
|
||||
> (`pyproject.toml`: `tantivy~=0.26.0`). The PR-ready detail below is retained as the historical record of
|
||||
> the gap as observed against `0.26.0`.
|
||||
|
||||
**Repo:** `quickwit-oss/tantivy-py`. **Observed version:** `0.26.0` (`tantivy v0.26.0, index_format v7`).
|
||||
|
||||
**Gap.** Python `datetime` objects cannot be passed to _any_ Query constructor for a `Date` field. Both
|
||||
`Query.range_query` and `Query.term_query` reject them:
|
||||
|
||||
```
|
||||
Expected DateTime type for field created, got datetime.datetime(2020, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)
|
||||
```
|
||||
|
||||
Int timestamps (seconds and nanoseconds) are also rejected, and there is no exposed/constructible
|
||||
`tantivy.DateTime` (`hasattr(tantivy, "DateTime") is False`). Consequently **all** date querying in paperless
|
||||
goes through `parse_query` strings; every object-mode `term_query` in the codebase is on integer fields
|
||||
(`id`, `owner_id`, `viewer_id`).
|
||||
|
||||
**Context.** PR #655 (merged 2026-04-27) added unbounded (`None`) bounds to `range_query`. That solved open
|
||||
_bounds_ but left the date _value_ path unusable from Python, so the open-range feature can't actually be used
|
||||
on date fields from Python yet.
|
||||
|
||||
**Reproduction** (against installed 0.26.0):
|
||||
|
||||
```python
|
||||
import tantivy
|
||||
from datetime import datetime, UTC
|
||||
schema = build_schema() # any schema with a date field "created"
|
||||
dt1, dt2 = datetime(2020,1,1,tzinfo=UTC), datetime(2021,1,1,tzinfo=UTC)
|
||||
|
||||
tantivy.Query.range_query(schema, "created", tantivy.FieldType.Date, lower_bound=dt1, upper_bound=dt2)
|
||||
# -> ValueError: Expected DateTime type for field created, got datetime.datetime(...)
|
||||
|
||||
tantivy.Query.range_query(schema, "created", tantivy.FieldType.Date, lower_bound=dt1, upper_bound=None)
|
||||
# -> same error (open bound is fine; the date VALUE is the problem)
|
||||
|
||||
tantivy.Query.term_query(schema, "created", dt1)
|
||||
# -> same error
|
||||
```
|
||||
|
||||
**Proposed fix (preferred):** in the Rust binding, when the target field is `Date`, accept a Python
|
||||
`datetime` and convert internally to `tantivy::DateTime` (e.g. `DateTime::from_timestamp_nanos(...)`), mirroring
|
||||
the conversion the indexing path already performs when adding date values to a document (document add-date
|
||||
already accepts `PyDateTime`). This makes `range_query`/`term_query` consistent with indexing. The value-coercion
|
||||
lives in the Query-construction value handling (the term/bound extraction in the query bindings, e.g.
|
||||
`src/query.rs`); reuse the existing `PyDateTime → tantivy DateTime` converter from the document bindings rather
|
||||
than adding a new one. Confirm exact locations against the tantivy-py source at PR time.
|
||||
|
||||
**Alternative:** expose a constructible `tantivy.DateTime` (from a Python `datetime` or an epoch-nanos int) and
|
||||
accept it in `range_query`/`term_query`. Less ergonomic; only do this if reusing the indexing converter proves
|
||||
awkward.
|
||||
|
||||
**Validation for the PR:**
|
||||
|
||||
- `range_query` on a `Date` field with two `datetime` bounds builds and returns expected hits.
|
||||
- `range_query` with one `datetime` bound and one `None` (open) works on a `Date` field.
|
||||
- `term_query` on a `Date` field with a `datetime` builds and matches.
|
||||
- Round-trip: index a doc with a known date, query it back via both closed and open ranges.
|
||||
|
||||
When this lands and we bump tantivy-py to the release containing it, Phase 2 (§8) becomes unblocked.
|
||||
|
||||
---
|
||||
|
||||
## 10. Out of scope / known separate gaps
|
||||
|
||||
- **Unknown-field 400.** `http://example.com/a,b` → `Field does not exist: 'http'`. Tantivy treats `http:` as
|
||||
a field; Whoosh's `remove_unknown=True` degraded unknown fields to text. This is the unknown-field divergence,
|
||||
not a comma or date issue. Recorded, not fixed here.
|
||||
- `>`/`<`/`>=`/`<=` comparisons — never supported in paperless-Whoosh.
|
||||
- Bare relative scalar (`created:-1week`) is P3: it "worked" in v2 but matched nothing. We only guarantee
|
||||
no-400.
|
||||
|
||||
---
|
||||
|
||||
## 11. Items to verify during implementation
|
||||
|
||||
- Exact RFC3339 **open-bound sentinels** to standardize on (`0001-01-01T00:00:00Z` / `9999-12-31T23:59:59Z`
|
||||
both parse; confirm they also behave in actual searches, not just parsing).
|
||||
- The **no-match clause** string representation for a date field (a degenerate/empty range that parses but
|
||||
matches nothing). In Phase 2 this becomes `empty_query()`.
|
||||
- ISO-dashed precision handling parity with Whoosh's separator-stripping (`-`, `.`, space).
|
||||
- Coordination with `fix/scope-comma-expansion`: either land this after that branch merges and delete its
|
||||
now-redundant regex, or absorb its narrowing directly. Do not ship both comma implementations.
|
||||
|
||||
---
|
||||
|
||||
## 12. Test plan (additive)
|
||||
|
||||
- **`test_translate.py` (new):**
|
||||
- `scan()` token-sequence tests: quotes, brackets, parens, URLs, value-lists, mixed clauses.
|
||||
- `translate_date_value` shape table: every §6 row (scalar + range), all three date fields,
|
||||
UTC/Eastern/Auckland timezones (reuse existing tz test patterns).
|
||||
- comma resolution: value-list (`tag`/`tag_id`/`viewer_id`), clause-sep (after `]`/`"`, before known
|
||||
`field:`), literal (text fields, URLs, `title:10,20`).
|
||||
- `translate_query()` golden cases: the full §3 / report-§5b ground-truth matrix.
|
||||
- **Parse-acceptance guardrail (current tests lack this):** for every golden case assert
|
||||
`index.parse_query(translate_query(q))` does not raise, against a real index.
|
||||
- **End-to-end:** a `views.py` search test asserting previously-400 v2 queries (`created:2020`,
|
||||
`created:[20200101 TO 20201231]`, `title:x,created:[…]`) now return 200.
|
||||
- Existing tests stay green via delegation; on removal of the old functions, migrate any unique assertions
|
||||
into `test_translate.py`.
|
||||
|
||||
---
|
||||
|
||||
## 13. Verification harnesses (keep for regression / ground-truth regeneration)
|
||||
|
||||
**Tantivy side** (does a translated string parse?):
|
||||
|
||||
```bash
|
||||
cd src && PAPERLESS_SECRET_KEY=x uv run python -c "
|
||||
import django, os, tempfile
|
||||
os.environ.setdefault('DJANGO_SETTINGS_MODULE','paperless.settings'); django.setup()
|
||||
import tantivy
|
||||
from documents.search._schema import build_schema
|
||||
from documents.search._tokenizer import register_tokenizers
|
||||
from documents.search._query import DEFAULT_SEARCH_FIELDS, _FIELD_BOOSTS
|
||||
idx = tantivy.Index(build_schema(), path=tempfile.mkdtemp()); register_tokenizers(idx,'english')
|
||||
idx.parse_query('<translated string>', DEFAULT_SEARCH_FIELDS, field_boosts=_FIELD_BOOSTS)
|
||||
"
|
||||
```
|
||||
|
||||
**Whoosh side** (what did v2 do? — ground truth):
|
||||
|
||||
```bash
|
||||
uv run --with cached_property python3 -W ignore -c "
|
||||
import sys; sys.path.insert(0,'whoosh/src')
|
||||
from datetime import datetime
|
||||
from whoosh.fields import Schema, TEXT, DATETIME, KEYWORD
|
||||
from whoosh.qparser import MultifieldParser
|
||||
from whoosh.qparser.dateparse import DateParserPlugin
|
||||
schema = Schema(title=TEXT(), content=TEXT(), correspondent=TEXT(),
|
||||
tag=KEYWORD(commas=True, lowercase=True), tag_id=KEYWORD(commas=True), viewer_id=KEYWORD(commas=True),
|
||||
type=TEXT(), created=DATETIME(), added=DATETIME(), modified=DATETIME(), notes=TEXT(), custom_fields=TEXT())
|
||||
qp = MultifieldParser(['content','title','correspondent','tag','type','notes','custom_fields'], schema)
|
||||
qp.add_plugin(DateParserPlugin(basedate=datetime(2026,6,14,14,0,0)))
|
||||
print(qp.parse('<query>'))
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. Phased summary
|
||||
|
||||
- **Phase 1 (now):** `_translate.py` scanner + `translate_date_value`, string output, sentinel open bounds,
|
||||
delegation shims, additive tests, parse-acceptance guardrail, end-to-end 400→200 tests. Ships on tantivy
|
||||
0.26.0, no upstream dependency. Subsumes `fix/scope-comma-expansion`.
|
||||
- **Phase 2 (later, gated on §9 upstream):** build `tantivy.Query` objects for date clauses — true open ranges
|
||||
via `range_query(None)`, real no-match via `empty_query()`, no string sentinels. Requires the tantivy-py
|
||||
date-value contribution and a version bump.
|
||||
Reference in New Issue
Block a user