* Fix: index document versions as their root document
The search index and the LLM index hold root documents only, a root being
indexed with its newest version's content. Every write path therefore had to
remember to hand them the root. Several did not.
WriteBatch.add_or_update, WriteBatch.add_or_update_ids and
llm_index_add_or_update_document now resolve a version to its root themselves.
The reindex command and update_llm_index only walk root documents, and an
incremental LLM update is scoped to the roots of the given ids through
versioning.root_document_ids, which add_or_update_ids shares. more_like_id
returns the root's id for a version. The reprocess task no longer picks the root
for the indexes and only still clears the caches of both documents.
* Fix: rebuild old search indexes and look versions up as their root for similar documents
An index built before versions were indexed as their root can still hold
version entries under their own id and metadata, and only a rebuild clears
them. Bump the schema version so the index is rebuilt on upgrade.
more_like_this_ids looked the given id up in the index as is, so asking for
documents like a version, as the AI suggestions endpoint can, silently found
nothing. It now looks the root up instead.
An interrupted rebuild left an empty index stamped as current, so the next
start reported it as up to date. Mark the rebuild as in progress and only
clear the marker once it completes.
* Feature: store barcode contents, list and search them
New setting PAPERLESS_CONSUMER_STORE_BARCODE_VALUES (off by default)
stores all barcodes found during consumption with the document: page,
type and content. They are listed on the metadata tab with a copy
button, returned by the documents API and searchable with barcodes:
in the advanced search. Versions keep their own barcodes, reprocessing
reads them again. Refs #9898
* Tests: cover the remaining barcode branches
Covers unchanged and failing barcode reads on reprocessing, unsupported files and DocumentBarcode.__str__, and uses toHaveLength in the barcode list spec as suggested by SonarCloud.
* Address review: keep barcodes when they can't be read, format choices
- Reprocessing and new versions keep the stored barcodes when the file
can't be scanned or the scan fails, and replace them atomically.
- Format is a TextChoices of the zxing-cpp formats, with a test.
- Shared scan code, latest_version helper, TypedDict, serializer reuse.
- Barcodes in the split manifest, export/import tests, pytest-style tests.
* Barcode tests: TIFF reprocess, format check both ways, fixtures
- Reprocessing a TIFF with TIFF support off keeps the stored barcodes.
- The format test also fails when zxing-cpp drops a format.
- Fixtures in place of the sample dir mixin, plugin-level disabled test.
- Format labels aren't translated, OpenAPI enum named BarcodeFormatEnum.
* Review: module-level zxing reader, shorter barcode docs
- read_barcodes_zxing is a module-level function used by scan_pdf.
- Drop the trivial __str__ test, mark it no cover.
- Shorten the barcode docs and remove the duplicate in configuration.md.
* Fix header
* Use utility class
* Return barcodes from the metadata endpoint only
Drop the barcodes field and its prefetches from the document serializer,
as agreed in the review. Also remove the now empty component stylesheet.
---------
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
* feat(search): add whoosh-compat, the shared field table and the field registry
* Sonar being useful actually
* Adds the given/when/then commenting
* Trims tests I don't think cover our logic or code or are redundant
* Coverage
* test(search): add pattern normalizer stem-alternates unit tests
* build: bump whoosh-compat to 0.2.0
Trenton H
jurassicparkicecreamandshamoon
