mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-08-01 00:22:17 +00:00
350684cd6b
* Fix: unify born-digital PDF detection between archive decision and OCR should_produce_archive() and RasterisedDocumentParser.parse() each reimplemented the "does this PDF have real text" check independently, using different normalization of pdftotext output. Raw pdftotext output can be non-empty (whitespace/form-feed layout padding) even when there is no real content, so the two checks could disagree: consumer.py treated a tagged-but-textless PDF as born-digital and skipped the archive, while the parser's own (stricter, normalized) check found no text and ran OCR anyway, leaving the document with no archive despite real OCR text (GH #13387). Both call sites now share one predicate, pdf_born_digital_text() in paperless/parsers/utils.py, so they can no longer drift apart. * Fix: restore extract_text seam for born-digital detection in parse() parse() had switched to calling pdf_born_digital_text() directly for its initial text/born-digital check, bypassing the parser's own extract_text instance method. That broke test mockability (tests patch tesseract_parser.extract_text to control the born-digital decision) and caused CI failures with mismatched OCR call counts and text. Split pdf_born_digital_text() into is_born_digital_text(text, path, log) - a pure decision function - and a thin pdf_born_digital_text() wrapper for callers without text in hand (consumer.should_produce_archive). parse() now extracts via self.extract_text(None, document_path) and passes the result to is_born_digital_text(), restoring the seam with no change to production behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Cleanup: simplify born-digital detection, close #13387 test gap Simplification pass over the born-digital detection consolidation: - is_born_digital_text(): drop the has_text temp for an early return. - consumer.py: standardize the archive-decision log lines on plain hyphens (was a mix of em-dash and hyphen) and hoist the duplicated text_length computation. - Parametrize TestPdfBornDigitalText instead of four near-identical tests. Code review follow-up: the existing tests only ever exercised pdf_born_digital_text() through mocks, so the actual #13387 scenario (a tagged PDF whose only "text" is layout padding) was never checked against real pdftotext/pikepdf output - a regression in the normalize-before-decide logic itself would have gone undetected. Moved tagged_no_text_pdf_file from parsers/conftest.py up to the shared paperless/tests/conftest.py (it was previously only visible to tests under parsers/) and added a non-mocked regression test against the real sample file. Also fixed two stale comments in test_consumer.py referencing a _extract_text_for_archive_check helper that no longer exists. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
66 lines
2.0 KiB
Python
66 lines
2.0 KiB
Python
"""
|
|
Fixtures defined here are available to every test module under
|
|
src/paperless/tests/ (including sub-packages such as parsers/).
|
|
|
|
Session-scoped fixtures for the shared samples directory live here so
|
|
sub-package conftest files can reference them without duplicating path logic.
|
|
Parser-specific fixtures (concrete parser instances, format-specific sample
|
|
files) live in paperless/tests/parsers/conftest.py.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
from typing import TYPE_CHECKING
|
|
|
|
import pytest
|
|
|
|
from paperless.parsers.registry import reset_parser_registry
|
|
|
|
if TYPE_CHECKING:
|
|
from collections.abc import Generator
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
def samples_dir() -> Path:
|
|
"""Absolute path to the shared parser sample files directory.
|
|
|
|
Sub-package conftest files derive format-specific paths from this root,
|
|
e.g. ``samples_dir / "text" / "test.txt"``.
|
|
|
|
Returns
|
|
-------
|
|
Path
|
|
Directory containing all sample documents used by parser tests.
|
|
"""
|
|
return (Path(__file__).parent / "samples").resolve()
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
def tagged_no_text_pdf_file(samples_dir: Path) -> Path:
|
|
"""Path to a tagged PDF whose only "text" is pdftotext layout padding.
|
|
|
|
Reproduces GH #13387: ``/MarkInfo /Marked true`` is set, but the only
|
|
extractable content is a form-feed byte, not real text. Lives here
|
|
rather than in parsers/conftest.py so both parser tests and
|
|
paperless/tests/test_parser_utils.py can use it.
|
|
|
|
Returns
|
|
-------
|
|
Path
|
|
Absolute path to ``tesseract/tagged-but-no-text.pdf``.
|
|
"""
|
|
return samples_dir / "tesseract" / "tagged-but-no-text.pdf"
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def clean_registry() -> Generator[None, None, None]:
|
|
"""Reset the parser registry before and after every test.
|
|
|
|
This prevents registry state from leaking between tests that call
|
|
get_parser_registry() or init_builtin_parsers().
|
|
"""
|
|
reset_parser_registry()
|
|
yield
|
|
reset_parser_registry()
|