diff --git a/docs/superpowers/plans/2026-08-07-whoosh-compat-transition.md b/docs/superpowers/plans/2026-08-07-whoosh-compat-transition.md index 4233647a2..734834e1b 100644 --- a/docs/superpowers/plans/2026-08-07-whoosh-compat-transition.md +++ b/docs/superpowers/plans/2026-08-07-whoosh-compat-transition.md @@ -2,6 +2,25 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +> **Parts of this plan call an API that has moved.** Read the status banner at +> the top of the spec before starting; it lists what changed, what is decided +> but not yet implemented, and the one question still open. Known-stale code +> in this plan, to fix on sight rather than copy: +> +> - `registry.resolve_json(...)` no longer exists. Tasks asserting on it (the +> registry JSON-subpath tests) must use `registry.make_ref(raw)`, which +> returns a `FieldRef` or `None`, and `registry.resolve(ref)` for the spec. +> A `None` from `make_ref` is how an unknown field or unknown subpath now +> reports itself. +> - `InvalidDateQuery(d.field, ...)` passes a `FieldRef` where a name is +> expected. Use `str(d.field)` or `d.field.name`. +> - `emit(..., schema=...)` may lose its `schema` parameter; check +> [#27](https://github.com/stumpylog/whoosh-compat/issues/27) before writing +> those call sites. +> +> Verify against the whoosh-compat checkout rather than this plan wherever the +> two disagree. The library is the source of truth. + **Goal:** Replace paperless-ngx's hand-rolled query-translation layer (`_translate.py`/`_dates.py`) with whoosh-compat, a typed Whoosh-grammar parser that emits programmatic `tantivy.Query` objects directly. **Architecture:** A shared field-definition table (`_fields.py`) drives both the Tantivy schema (`_schema.py`) and a new `whoosh_compat.FieldRegistry` (`_registry.py`), eliminating drift between what's indexed and what's query-addressable. `parse_user_query()` in `_query.py` becomes `wc.parse() -> diagnostics check -> emit()`, replacing the old regex scan/render pipeline. Landed as four sequential PRs with no feature flag; safety comes from a result-level acceptance test corpus and a date-grammar parity audit against the code being deleted. diff --git a/docs/superpowers/specs/2026-08-07-whoosh-compat-transition-design.md b/docs/superpowers/specs/2026-08-07-whoosh-compat-transition-design.md index 32a2ed048..7fc2db52f 100644 --- a/docs/superpowers/specs/2026-08-07-whoosh-compat-transition-design.md +++ b/docs/superpowers/specs/2026-08-07-whoosh-compat-transition-design.md @@ -5,6 +5,45 @@ Status: approved, pending spec review Related skill: `whoosh-compat-transition` Related issue: [stumpylog/whoosh-compat#1](https://github.com/stumpylog/whoosh-compat/issues/1) +> ## Read this before implementing: parts of this spec describe an API that has moved +> +> whoosh-compat changed after this spec was written, and more changes are +> already decided. Re-check anything below against the library's own source +> and `ARCHITECTURE.md` before writing code from it. The library repo is the +> source of truth; this document is not. +> +> **Wrong today, fix on sight:** +> +> | This spec says | Reality now | +> | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +> | `FieldRegistry.resolve_json(dotted)` (ยง JSON subpath resolution) | Gone. There is one resolver: `registry.make_ref(raw) -> FieldRef \| None` interprets a dotted name, and `registry.resolve(ref) -> FieldSpec \| None` looks it up. A dot is interpreted only inside `make_ref`. | +> | `InvalidDateQuery(d.field, d.raw_value)` | `Diagnostic.field` is now a `FieldRef`, not a string. Use `str(d.field)` for the canonical dotted name, or `d.field.name`. The name is canonical, so an aliased query (`type:`) reports `document_type`. | +> | AST leaves carrying a field name string | Every field-carrying AST leaf now holds a `FieldRef`. | +> +> **Decided upstream, not yet implemented. Write toward these, they will land before this migration executes:** +> +> | Behavior | Issue | +> | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | +> | An empty or unknown `default_fields` raises at `parse()`; a `field_boosts` key naming an alias resolves, one naming nothing raises | [#20](https://github.com/stumpylog/whoosh-compat/issues/20) | +> | A naive `basedate` is rejected rather than read in the host machine's timezone | [#19](https://github.com/stumpylog/whoosh-compat/issues/19) | +> | A wildcard on a numeric field produces a diagnostic instead of failing at search time, so the error mapping gains a case | [#17](https://github.com/stumpylog/whoosh-compat/issues/17) | +> | A bare JSON field name (`notes:foo`) demotes to a text search rather than raising at emit | [#11](https://github.com/stumpylog/whoosh-compat/issues/11) | +> | Registry construction rejects exists-target cycles, empty names, duplicate aliases within a spec, and dotted canonical names | [#21](https://github.com/stumpylog/whoosh-compat/issues/21) | +> +> **Still undecided, do not guess:** whether `emit()` keeps its `schema` +> parameter ([#27](https://github.com/stumpylog/whoosh-compat/issues/27)). +> This spec calls `emit(ast, index=index, schema=schema, registry=...)` in two +> places. Check the issue before writing either call site. +> +> **Trap:** do not add `fast=True` to the `notes` or `custom_fields` JSON +> specs. Existence checks against a fast JSON field currently return inverted +> results ([#7](https://github.com/stumpylog/whoosh-compat/issues/7)), and the +> error raised for a non-fast JSON field advises marking it fast, which walks +> straight into that bug. The field table below correctly leaves them non-fast. +> +> [#28](https://github.com/stumpylog/whoosh-compat/issues/28) tracks all open +> upstream work ordered by effort. + ## Summary Replace paperless-ngx's hand-maintained query-translation layer