Compare commits

..
Author SHA1 Message Date
stumpylog 6b47534841 Chore: enable flake8-datetimez (DTZ) ruff rules
Full category (10/10 codes are all default in ruff 0.16). Of 54
hits, 46 were in test fixture code constructing naive datetimes for
comparison/input purposes only - added DTZ to the existing
per-file-ignores for */tests/*.py alongside E501/SIM117.

The 8 production hits:
- documents/consumer.py, documents/views.py (index_last_modified):
  suppressed with noqa - timezone.make_aware() requires a naive
  datetime, so wrapping fromtimestamp() in tz= would break it
- documents/double_sided.py (x2): switched to
  datetime.now(tz=UTC).timestamp() - behavior-identical since
  .timestamp() returns the same epoch value regardless of the
  attached tz, but now explicit
- documents/views.py (x2, upload temp file mtime): suppressed with
  noqa - mktime() requires a local time tuple, so an aware/UTC now()
  would introduce a timezone-offset bug
- documents/workflows/ai.py (AI suggested date parsing): suppressed
  with noqa - only .date() is used, time/tz is discarded
- paperless_mail/mail.py (IMAP fetch date filter): switched
  date.today() to timezone.localdate(), which respects
  settings.TIME_ZONE instead of the system clock - a real
  correctness improvement when they differ
- paperless_mail/views.py (placeholder name string): switched
  datetime.datetime.now() to timezone.now(), matching the app's
  existing aware-datetime convention
2026-08-27 11:00:12 -07:00
stumpylog c6e3f7a0c4 Chore: enable tryceratops (TRY002/004/201/203/401) ruff rules
Only the 5 default-subset codes; the rest of tryceratops is opt-in.

- 9 TRY201 (raise e -> raise) autofixed, preserving the traceback
  identically while dropping the redundant exception name
- 26 TRY401 (redundant exception object passed to logger.exception,
  which already logs it) fixed by removing the duplicate from the
  message; three sites still needed the exception object for
  something else (re-raising, or a separate logger.error call) and
  kept their binding
- 6 TRY002 (raise bare Exception): 2 production sites (documents/
  matching.py, paperless_mail/preprocessor.py) got dedicated
  exception classes, with their tests narrowed to match instead of
  asserting a blind Exception; the other 4 are deliberate generic
  failures in test doubles/fixtures, suppressed with noqa
2026-08-27 10:50:32 -07:00
stumpylog dc7b649625 Chore: enable pylint warning (PLW) ruff rules
Full category (not just the ruff-0.16 default subset). 35 hits:
- 4 PLW0108 (unnecessary lambda) autofixed
- 6 PLW2901 (loop/with variable shadowed) renamed to distinct names
- 2 PLW1510 (subprocess.run without explicit check) given check=False,
  matching existing behavior exactly
- 6 PLW0602 (global declared but never assigned) removed - these were
  all in-place mutations (.append/.insert), not reassignments, so
  `global` was already a no-op
- 7 PLW0603 (global statement) suppressed with noqa - these are
  genuine lazy-init singletons with no class to hold the state;
  refactoring them is a separate, larger change
- 6 PLW1508 (non-str/None env var default) fixed using the existing
  get_int_from_env/get_float_from_env typed helpers instead of raw
  os.getenv, which also fixes a real bug: LOGROTATE_MAX_SIZE and
  LOGROTATE_MAX_BACKUPS were never wrapped in int(), so a string env
  var value would have flowed into RotatingFileHandler as a string
- 1 PLW1641 (__eq__ without __hash__) fixed by adding __hash__ to
  PlaceholderString
2026-08-27 10:37:18 -07:00
stumpylog a4075c49b2 Chore: enable flake8-gettext (INT001/002/003) ruff rules
All 4 hits in documents/validators.py were f-strings inside gettext
_() calls, which resolves the string before translation and breaks
extraction (confirmed: locale .po files literally contain the raw
"{value}" placeholder as msgid text). Fixed by using %(name)s-style
placeholders with Django ValidationError's existing params= kwarg,
which was already being passed but silently unused.
2026-08-27 10:23:38 -07:00
stumpylog 70b1c86ad3 Chore: enable flake8-bandit S102/S110/S112 ruff rules
3 S110 (try-except-pass) hits, all fixed by adding a log call in the
except block rather than silently swallowing the exception, matching
this codebase's existing %s lazy-formatting logging convention.
Behavior is unchanged (still no re-raise) in all three spots.
2026-08-27 10:21:27 -07:00
stumpylog 8f5f577634 Chore: enable flake8-bugbear (B) default-subset ruff rules
3 B009 (getattr with a constant string, rewrite as attribute access)
hits autofixed. 7 B017 (assert blind Exception) hits: one narrowed
to the actual ValueError raised by bulk_edit.edit_pdf, the other six
suppressed with noqa since the code under test genuinely raises (or
a mock genuinely injects) a bare Exception, so a narrower assertion
would be wrong.

Only the 29 B codes ruff 0.16 enables by default; the rest of
flake8-bugbear needs a separate, deliberate decision.
2026-08-27 10:17:16 -07:00
stumpylog 65b6db7b63 Chore: enable flake8-logging-format G101/G202 ruff rules
G202 (redundant exc_info=True passed to logger.exception, which
already includes the traceback) had 2 hits in documents/views.py,
fixed manually since ruff has no autofix for it. G101 (hardcoded
password string) had zero hits.
2026-08-27 10:11:50 -07:00
stumpylog 6f843bda8d Chore: enable flake8-2020 (YTT) ruff rules
Zero current violations. Full category (10/10 codes are all part of
ruff 0.16's default rule set already, so there's no non-default
subset to defer).
2026-08-27 10:09:20 -07:00
stumpylog b7267815e3 Chore: enable flake8-debugger T100 ruff rule
Zero current violations. Only T100 (import of pdb/ipdb/etc.) is part
of ruff 0.16's default rule set.
2026-08-27 10:09:03 -07:00
stumpylog c9bf18646a Chore: enable flake8-pytest-style (PT) default-subset ruff rules
Zero current violations. Only the 6 PT codes ruff 0.16 enables by
default; the full flake8-pytest-style linter has thousands of hits
here and needs a separate, deliberate decision.
2026-08-27 10:08:45 -07:00
stumpylog 8d1af73dc4 Chore: enable pylint refactor (PLR) default-subset ruff rules
Zero current violations. Only the 13 PLR codes ruff 0.16 enables by
default; the rest of pylint-refactor (e.g. PLR2004, PLR0913) has
hundreds of hits here and needs a separate, deliberate decision.
2026-08-27 10:08:24 -07:00
stumpylog 515866a381 Chore: enable pygrep-hooks PGH005 ruff rule
Zero current violations. Only PGH005 (invalid-mock-methods) is part
of ruff 0.16's default rule set; the rest of pygrep-hooks is opt-in.
2026-08-27 10:08:03 -07:00
stumpylog a91a2ab7e6 Chore: enable pep8-naming N999 ruff rule
Zero current violations. Only N999 (invalid-module-name) is part of
ruff 0.16's default rule set; the rest of pep8-naming is opt-in.
2026-08-27 10:07:44 -07:00
stumpylog 6d49f6e283 Chore: enable flake8-logging (LOG001/002/009/014/015) ruff rules
Zero current violations. Only these five LOG codes are part of
ruff 0.16's default rule set; the rest of the linter is opt-in.
2026-08-27 10:07:27 -07:00
stumpylog ccd39118c2 Chore: enable pydocstyle D419 ruff rule
Zero current violations. Only D419 (empty-docstring) is part of
ruff 0.16's default rule set; the rest of pydocstyle is opt-in.
2026-08-27 10:07:05 -07:00
stumpylog 9c9771b5fc Chore: enable flake8-async (ASYNC) ruff rules
Zero current violations. Full category (not just the ruff-0.16
default subset) since the rest is equally applicable async-blocking
guidance for this codebase's Channels/websocket code.
2026-08-27 10:06:48 -07:00
stumpylog 73e872d5f7 Chore: enable FA, G010, and PERF101/102/402 ruff rules
All part of ruff 0.16's expanded default rule set. FA and G010 had
zero existing violations; PERF402's one occurrence needed a manual
fix since ruff can't safely autofix a multi-line call expression.
2026-08-27 09:59:41 -07:00
stumpylog d3e6ef8c02 Chore: enable refurb (FURB) ruff rules
FURB is part of ruff 0.16's expanded default rule set and is
almost entirely autofixable.
2026-08-27 09:56:55 -07:00
stumpylog 18af45959f Chore: enable flake8-comprehensions (C4) ruff rules
C4 is part of ruff 0.16's expanded default rule set and is almost
entirely autofixable, making it a low-risk first step towards
adopting the new defaults.
2026-08-27 09:55:24 -07:00
54 changed files with 323 additions and 572 deletions
+105 -23
View File
@@ -186,29 +186,110 @@ line-ending = "lf"
# https://docs.astral.sh/ruff/rules/
select = [ "E4", "E7", "E9", "F" ]
extend-select = [
"COM", # https://docs.astral.sh/ruff/rules/#flake8-commas-com
"DJ", # https://docs.astral.sh/ruff/rules/#flake8-django-dj
"EXE", # https://docs.astral.sh/ruff/rules/#flake8-executable-exe
"FBT", # https://docs.astral.sh/ruff/rules/#flake8-boolean-trap-fbt
"FLY", # https://docs.astral.sh/ruff/rules/#flynt-fly
"G201", # https://docs.astral.sh/ruff/rules/#flake8-logging-format-g
"I", # https://docs.astral.sh/ruff/rules/#isort-i
"ICN", # https://docs.astral.sh/ruff/rules/#flake8-import-conventions-icn
"INP", # https://docs.astral.sh/ruff/rules/#flake8-no-pep420-inp
"ISC", # https://docs.astral.sh/ruff/rules/#flake8-implicit-str-concat-isc
"PIE", # https://docs.astral.sh/ruff/rules/#flake8-pie-pie
"PLC", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLE", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PTH", # https://docs.astral.sh/ruff/rules/#flake8-use-pathlib-pth
"Q", # https://docs.astral.sh/ruff/rules/#flake8-quotes-q
"RSE", # https://docs.astral.sh/ruff/rules/#flake8-raise-rse
"RUF", # https://docs.astral.sh/ruff/rules/#ruff-specific-rules-ruf
"SIM", # https://docs.astral.sh/ruff/rules/#flake8-simplify-sim
"T20", # https://docs.astral.sh/ruff/rules/#flake8-print-t20
"TC", # https://docs.astral.sh/ruff/rules/#flake8-type-checking-tc
"TID", # https://docs.astral.sh/ruff/rules/#flake8-tidy-imports-tid
"UP", # https://docs.astral.sh/ruff/rules/#pyupgrade-up
"W", # https://docs.astral.sh/ruff/rules/#pycodestyle-e-w
"ASYNC", # https://docs.astral.sh/ruff/rules/#flake8-async-async
"B002", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B003", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B004", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B005", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B006", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B008", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B009", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B010", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B012", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B013", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B014", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B015", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B016", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B017", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B018", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B019", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B020", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B021", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B022", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B023", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B025", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B026", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B029", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B030", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B031", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B032", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B033", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B035", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B039", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"C4", # https://docs.astral.sh/ruff/rules/#flake8-comprehensions-c4
"COM", # https://docs.astral.sh/ruff/rules/#flake8-commas-com
"D419", # https://docs.astral.sh/ruff/rules/#pydocstyle-d
"DJ", # https://docs.astral.sh/ruff/rules/#flake8-django-dj
"DTZ", # https://docs.astral.sh/ruff/rules/#flake8-datetimez-dtz
"EXE", # https://docs.astral.sh/ruff/rules/#flake8-executable-exe
"FA", # https://docs.astral.sh/ruff/rules/#flake8-future-annotations-fa
"FBT", # https://docs.astral.sh/ruff/rules/#flake8-boolean-trap-fbt
"FLY", # https://docs.astral.sh/ruff/rules/#flynt-fly
"FURB", # https://docs.astral.sh/ruff/rules/#refurb-furb
"G010", # https://docs.astral.sh/ruff/rules/#flake8-logging-format-g
"G101", # https://docs.astral.sh/ruff/rules/#flake8-logging-format-g
"G201", # https://docs.astral.sh/ruff/rules/#flake8-logging-format-g
"G202", # https://docs.astral.sh/ruff/rules/#flake8-logging-format-g
"I", # https://docs.astral.sh/ruff/rules/#isort-i
"ICN", # https://docs.astral.sh/ruff/rules/#flake8-import-conventions-icn
"INP", # https://docs.astral.sh/ruff/rules/#flake8-no-pep420-inp
"INT001", # https://docs.astral.sh/ruff/rules/#flake8-gettext-int
"INT002", # https://docs.astral.sh/ruff/rules/#flake8-gettext-int
"INT003", # https://docs.astral.sh/ruff/rules/#flake8-gettext-int
"ISC", # https://docs.astral.sh/ruff/rules/#flake8-implicit-str-concat-isc
"LOG001", # https://docs.astral.sh/ruff/rules/#flake8-logging-log
"LOG002", # https://docs.astral.sh/ruff/rules/#flake8-logging-log
"LOG009", # https://docs.astral.sh/ruff/rules/#flake8-logging-log
"LOG014", # https://docs.astral.sh/ruff/rules/#flake8-logging-log
"LOG015", # https://docs.astral.sh/ruff/rules/#flake8-logging-log
"N999", # https://docs.astral.sh/ruff/rules/#pep8-naming-n
"PERF101", # https://docs.astral.sh/ruff/rules/#perflint-perf
"PERF102", # https://docs.astral.sh/ruff/rules/#perflint-perf
"PERF402", # https://docs.astral.sh/ruff/rules/#perflint-perf
"PGH005", # https://docs.astral.sh/ruff/rules/#pygrep-hooks-pgh
"PIE", # https://docs.astral.sh/ruff/rules/#flake8-pie-pie
"PLC", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLE", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR0124", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR0133", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR0206", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR0402", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR1704", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR1708", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR1711", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR1716", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR1722", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR1730", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR1733", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR1736", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLR2044", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLW", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PT010", # https://docs.astral.sh/ruff/rules/#flake8-pytest-style-pt
"PT014", # https://docs.astral.sh/ruff/rules/#flake8-pytest-style-pt
"PT020", # https://docs.astral.sh/ruff/rules/#flake8-pytest-style-pt
"PT025", # https://docs.astral.sh/ruff/rules/#flake8-pytest-style-pt
"PT026", # https://docs.astral.sh/ruff/rules/#flake8-pytest-style-pt
"PT031", # https://docs.astral.sh/ruff/rules/#flake8-pytest-style-pt
"PTH", # https://docs.astral.sh/ruff/rules/#flake8-use-pathlib-pth
"Q", # https://docs.astral.sh/ruff/rules/#flake8-quotes-q
"RSE", # https://docs.astral.sh/ruff/rules/#flake8-raise-rse
"RUF", # https://docs.astral.sh/ruff/rules/#ruff-specific-rules-ruf
"S102", # https://docs.astral.sh/ruff/rules/#flake8-bandit-s
"S110", # https://docs.astral.sh/ruff/rules/#flake8-bandit-s
"S112", # https://docs.astral.sh/ruff/rules/#flake8-bandit-s
"SIM", # https://docs.astral.sh/ruff/rules/#flake8-simplify-sim
"T100", # https://docs.astral.sh/ruff/rules/#flake8-debugger-t10
"T20", # https://docs.astral.sh/ruff/rules/#flake8-print-t20
"TC", # https://docs.astral.sh/ruff/rules/#flake8-type-checking-tc
"TID", # https://docs.astral.sh/ruff/rules/#flake8-tidy-imports-tid
"TRY002", # https://docs.astral.sh/ruff/rules/#tryceratops-try
"TRY004", # https://docs.astral.sh/ruff/rules/#tryceratops-try
"TRY201", # https://docs.astral.sh/ruff/rules/#tryceratops-try
"TRY203", # https://docs.astral.sh/ruff/rules/#tryceratops-try
"TRY401", # https://docs.astral.sh/ruff/rules/#tryceratops-try
"UP", # https://docs.astral.sh/ruff/rules/#pyupgrade-up
"W", # https://docs.astral.sh/ruff/rules/#pycodestyle-e-w
"YTT", # https://docs.astral.sh/ruff/rules/#flake8-2020-ytt
]
ignore = [
"DJ001",
@@ -224,6 +305,7 @@ per-file-ignores."*/migrations/*.py" = [
]
# Testing
per-file-ignores."*/tests/*.py" = [
"DTZ",
"E501",
"SIM117",
]
+10 -10
View File
@@ -507,8 +507,8 @@ def rotate(
logger.info(
f"Queued new rotated version for document {pair.root_doc.id} by {degrees} degrees",
)
except Exception as e:
logger.exception(f"Error rotating document {pair.root_doc.id}: {e}")
except Exception:
logger.exception(f"Error rotating document {pair.root_doc.id}")
return "OK"
@@ -554,9 +554,9 @@ def merge(
affected_docs.append(doc.id)
if handoff_asn is None and doc.archive_serial_number is not None:
handoff_asn = doc.archive_serial_number
except Exception as e:
except Exception:
logger.exception(
f"Error merging document {doc.id}, it will not be included in the merge: {e}",
f"Error merging document {doc.id}, it will not be included in the merge",
)
if len(affected_docs) == 0:
logger.warning("No documents were merged")
@@ -805,8 +805,8 @@ def split(
else:
group(consume_tasks).delay()
except Exception as e:
logger.exception(f"Error splitting document {doc.id}: {e}")
except Exception:
logger.exception(f"Error splitting document {doc.id}")
return "OK"
@@ -858,8 +858,8 @@ def delete_pages(
logger.info(
f"Queued new version for document {pair.root_doc.id} after deleting pages {pages}",
)
except Exception as e:
logger.exception(f"Error deleting pages from document {pair.root_doc.id}: {e}")
except Exception:
logger.exception(f"Error deleting pages from document {pair.root_doc.id}")
return "OK"
@@ -986,7 +986,7 @@ def edit_pdf(
group(consume_tasks).delay()
except Exception as e:
logger.exception(f"Error editing document {pair.root_doc.id}: {e}")
logger.exception(f"Error editing document {pair.root_doc.id}")
raise ValueError(
f"An error occurred while editing the document: {e}",
) from e
@@ -1097,7 +1097,7 @@ def remove_password(
except Exception as e:
logger.exception(
f"Error removing password from document {pair.root_doc.id}: {e}",
f"Error removing password from document {pair.root_doc.id}",
)
raise ValueError(
f"An error occurred while removing the password: {e}",
+7 -7
View File
@@ -69,8 +69,8 @@ def load_classifier(*, raise_exception: bool = False) -> DocumentClassifier | No
Path(settings.MODEL_FILE).unlink()
classifier = None
if raise_exception:
raise e
except ClassifierModelCorruptError as e:
raise
except ClassifierModelCorruptError:
# there's something wrong with the model file.
logger.exception(
"Unrecoverable error while loading document "
@@ -79,17 +79,17 @@ def load_classifier(*, raise_exception: bool = False) -> DocumentClassifier | No
Path(settings.MODEL_FILE).unlink()
classifier = None
if raise_exception:
raise e
except OSError as e:
raise
except OSError:
logger.exception("IO error while loading document classification model")
classifier = None
if raise_exception:
raise e
except Exception as e: # pragma: no cover
raise
except Exception: # pragma: no cover
logger.exception("Unknown error while loading document classification model")
classifier = None
if raise_exception:
raise e
raise
return classifier
+4 -6
View File
@@ -216,7 +216,7 @@ class ConsumerPluginMixin:
current_progress,
max_progress,
document_id=document_id,
owner_id=self.metadata.owner_id if self.metadata.owner_id else None,
owner_id=self.metadata.owner_id or None,
users_can_view=(self.metadata.view_users or [])
+ (self.metadata.change_users or []),
groups_can_view=(self.metadata.view_groups or [])
@@ -674,9 +674,7 @@ class ConsumerPlugin(
document=document,
logging_group=self.logging_group,
classifier=classifier,
original_file=self.unmodified_original
if self.unmodified_original
else self.working_copy,
original_file=self.unmodified_original or self.working_copy,
)
# After everything is in the database, copy the files into
@@ -849,7 +847,7 @@ class ConsumerPlugin(
else:
stats = Path(self.input_doc.original_file).stat()
create_date = timezone.make_aware(
datetime.datetime.fromtimestamp(stats.st_mtime),
datetime.datetime.fromtimestamp(stats.st_mtime), # noqa: DTZ006 - make_aware() requires a naive datetime
)
self.log.debug(f"Creation date from st_mtime: {create_date}")
@@ -963,7 +961,7 @@ class ConsumerPlugin(
try:
copy_basic_file_stats(source, target)
except Exception: # pragma: no cover
pass
self.log.debug("Unable to copy file stats from %s to %s", source, target)
class ConsumerPreflightPlugin(
+4 -2
View File
@@ -78,7 +78,9 @@ class CollatePlugin(NoCleanupPluginMixin, NoSetupPluginMixin, ConsumeTaskPlugin)
stats = staging.stat()
# if the file is older than the timeout, we don't consider
# it valid
if (dt.datetime.now().timestamp() - stats.st_mtime) > TIMEOUT_SECONDS:
if (
dt.datetime.now(tz=dt.UTC).timestamp() - stats.st_mtime
) > TIMEOUT_SECONDS:
logger.warning("Outdated double sided staging file exists, deleting it")
staging.unlink()
else:
@@ -134,7 +136,7 @@ class CollatePlugin(NoCleanupPluginMixin, NoSetupPluginMixin, ConsumeTaskPlugin)
shutil.move(pdf_file, staging)
# update access to modification time so we know if the file
# is outdated when another file gets uploaded
timestamp = dt.datetime.now().timestamp()
timestamp = dt.datetime.now(tz=dt.UTC).timestamp()
os.utime(staging, (timestamp, timestamp))
logger.info(
"Got scan with odd numbered pages of double-sided scan, moved it to %s",
+1 -1
View File
@@ -734,7 +734,7 @@ class CustomFieldQueryParser:
)
# Check if any of the requested IDs are missing.
missing_ids = set(value) - set(link.document_id for link in links)
missing_ids = set(value) - {link.document_id for link in links}
if missing_ids:
# The result should be an empty set in this case.
return Q(id__in=[])
@@ -631,23 +631,25 @@ class Command(BaseCommand):
):
# Process each change
for change_type, path in changes:
path = Path(path).resolve()
resolved_path = Path(path).resolve()
if change_type == Change.deleted:
# Consumed (or otherwise removed); a later file
# reusing this name must not be skipped as
# already-queued.
queued.discard(path)
if not path.is_file():
queued.discard(resolved_path)
if not resolved_path.is_file():
continue
if path in queued:
if resolved_path in queued:
# Already queued and awaiting consumption; a stray
# event (NAS metadata touch, AV scan, etc.) while
# the file sits on disk mid-consumption must not
# cause it to be queued a second time (GH #13511).
logger.debug(f"Ignoring event for queued file: {path}")
logger.debug(
f"Ignoring event for queued file: {resolved_path}",
)
continue
logger.debug(f"Event: {change_type.name} for {path}")
tracker.track(path, change_type)
logger.debug(f"Event: {change_type.name} for {resolved_path}")
tracker.track(resolved_path, change_type)
# Check for stable files
for stable_path in tracker.get_stable_files():
+7 -1
View File
@@ -30,6 +30,10 @@ if TYPE_CHECKING:
logger = logging.getLogger("paperless.matching")
class UnsupportedWorkflowTriggerTypeError(Exception):
pass
def log_reason(
matching_model: MatchingModel | WorkflowTrigger,
document: Document,
@@ -691,7 +695,9 @@ def document_matches_workflow(
)
else:
# New trigger types need to be explicitly checked above
raise Exception(f"Trigger type {trigger_type} not yet supported")
raise UnsupportedWorkflowTriggerTypeError(
f"Trigger type {trigger_type} not yet supported",
)
if trigger_matched:
logger.info(f"Document matched {trigger} from {workflow}")
+1 -15
View File
@@ -373,29 +373,15 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
If the queryset already annotated ``effective_content``, that value is used.
"""
# Here to avoid circular import
from documents.versioning import LATEST_VERSION_CONTENT_PREFETCH_ATTR
from documents.versioning import sort_versions_newest_first
from documents.versioning import versions_newest_first
if hasattr(self, "effective_content"):
return getattr(self, "effective_content")
return self.effective_content
if self.root_document_id is not None or self.pk is None:
return self.content
latest_version_prefetch = getattr(
self,
LATEST_VERSION_CONTENT_PREFETCH_ATTR,
None,
)
if latest_version_prefetch is not None:
# Empty list means prefetch ran and found no versions — use own content.
return (
latest_version_prefetch[0].content
if latest_version_prefetch
else self.content
)
prefetched_cache = getattr(self, "_prefetched_objects_cache", None)
prefetched_versions = (
prefetched_cache.get("versions")
+1 -1
View File
@@ -41,7 +41,7 @@ def get_default_file_extension(mime_type: str) -> str:
return supported[mime_type]
ext = mimetypes.guess_extension(mime_type)
return ext if ext else ""
return ext or ""
def is_file_ext_supported(ext: str) -> bool:
@@ -43,8 +43,8 @@ def _discover_parser_class() -> type[DateParserPluginBase]:
valid_plugins.append(ep)
else:
logger.warning(f"Plugin {ep.name} does not subclass DateParser.")
except Exception as e:
logger.exception(f"Unable to load date parser plugin {ep.name}: {e}")
except Exception:
logger.exception(f"Unable to load date parser plugin {ep.name}")
if not valid_plugins:
return RegexDateParserPlugin
+2 -2
View File
@@ -91,8 +91,8 @@ class DateParserPluginBase(ABC):
},
locales=self.config.languages,
)
except Exception as e:
logger.exception(f"Error while parsing date string '{date_string}': {e}")
except Exception:
logger.exception(f"Error while parsing date string '{date_string}'")
return None
def _filter_date(
+4 -6
View File
@@ -59,11 +59,10 @@ def safe_regex_match(pattern: str, text: str, *, flags: int = 0):
try:
validate_regex_pattern(pattern)
compiled = regex.compile(pattern, flags=flags)
except (regex.error, ValueError) as exc:
except (regex.error, ValueError):
logger.exception(
"Error while processing regular expression %s: %s",
"Error while processing regular expression %s",
textwrap.shorten(pattern, width=80, placeholder=""),
exc,
)
return None
@@ -86,11 +85,10 @@ def safe_regex_sub(pattern: str, repl: str, text: str, *, flags: int = 0) -> str
try:
validate_regex_pattern(pattern)
compiled = regex.compile(pattern, flags=flags)
except (regex.error, ValueError) as exc:
except (regex.error, ValueError):
logger.exception(
"Error while processing regular expression %s: %s",
"Error while processing regular expression %s",
textwrap.shorten(pattern, width=80, placeholder=""),
exc,
)
return None
+2 -2
View File
@@ -1142,7 +1142,7 @@ def get_backend() -> TantivyBackend:
Returns:
Thread-safe singleton TantivyBackend instance
"""
global _backend, _backend_path
global _backend, _backend_path # noqa: PLW0603 - module-level singleton, no class to hold this state
current_path: Path = settings.INDEX_DIR
@@ -1173,7 +1173,7 @@ def reset_backend() -> None:
Forces creation of a new backend instance on the next get_backend() call.
Used for test isolation and when switching between different index directories.
"""
global _backend, _backend_path
global _backend, _backend_path # noqa: PLW0603 - module-level singleton, no class to hold this state
with _backend_lock:
if _backend is not None:
+1 -1
View File
@@ -240,7 +240,7 @@ def parse_user_query(
DEFAULT_SEARCH_FIELDS,
field_boosts=_FIELD_BOOSTS,
# (prefix=True, distance=1, transposition_cost_one=True) — edit-distance fuzziness
fuzzy_fields={f: (True, 1, True) for f in DEFAULT_SEARCH_FIELDS},
fuzzy_fields=dict.fromkeys(DEFAULT_SEARCH_FIELDS, (True, 1, True)),
)
# 0.1 boost keeps fuzzy hits ranked below exact matches (intentional)
clauses.append((tantivy.Occur.Should, tantivy.Query.boost_query(fuzzy, 0.1)))
+12 -20
View File
@@ -88,7 +88,6 @@ from documents.templating.utils import convert_format_str_to_template_format
from documents.templating.workflows import validate_workflow_template
from documents.validators import uri_validator
from documents.validators import url_validator
from documents.versioning import has_prefetched_effective_content
from documents.versioning import sort_versions_newest_first
if TYPE_CHECKING:
@@ -434,7 +433,7 @@ class OwnedObjectSerializer(
return set()
ctype = ContentType.objects.get_for_model(first_obj)
object_pks = list(obj.pk for obj in objects)
object_pks = [obj.pk for obj in objects]
pk_type = type(first_obj.pk)
def get_pks_for_permission_type(model):
@@ -728,7 +727,7 @@ class TagSerializer(MatchingModelSerializer, OwnedObjectSerializer):
self.instance.clean()
except ValidationError as e:
logger.debug("Tag parent validation failed: %s", e)
raise e
raise
finally:
self.instance.tn_parent = original_parent
else:
@@ -738,7 +737,7 @@ class TagSerializer(MatchingModelSerializer, OwnedObjectSerializer):
temp.clean()
except ValidationError as e:
logger.debug("Tag parent validation failed: %s", e)
raise e
raise
return super().validate(attrs)
@@ -1147,14 +1146,8 @@ class DocumentSerializer(
def to_representation(self, instance):
doc = super().to_representation(instance)
if "content" in self.fields and has_prefetched_effective_content(instance):
# Only resolve version-aware content when it's cheap: an SQL
# annotation or a versions prefetch is already on the instance.
# A caller that set up neither (e.g. TrashView, GlobalSearchView,
# which build their own querysets) gets the document's own,
# unresolved content instead of paying for an extra per-instance
# query -- same as before effective_content resolution existed.
doc["content"] = instance.get_effective_content() or ""
if "content" in self.fields and hasattr(instance, "effective_content"):
doc["content"] = instance.effective_content or ""
if self.truncate_content and "content" in self.fields:
doc["content"] = doc.get("content")[0:550]
return doc
@@ -1864,8 +1857,8 @@ class BulkEditSerializer(
if isinstance(custom_fields, dict):
try:
ids = [int(i[0]) for i in custom_fields.items()]
except Exception as e:
logger.exception(f"Error validating custom fields: {e}")
except Exception:
logger.exception("Error validating custom fields")
raise serializers.ValidationError(
f"{name} must be a list of integers or a dict of id:value pairs, see the log for details",
)
@@ -2063,13 +2056,12 @@ class BulkEditSerializer(
for doc in docs:
if "-" in doc:
pages.append(
[
x
for x in range(
list(
range(
int(doc.split("-")[0]),
int(doc.split("-")[1]) + 1,
)
],
),
),
)
else:
pages.append([int(doc)])
@@ -2930,7 +2922,7 @@ class ShareLinkBundleSerializer(OwnedObjectSerializer):
return share_link_bundle
def get_document_count(self, obj: ShareLinkBundle) -> int:
return getattr(obj, "document_total") or obj.documents.count()
return obj.document_total or obj.documents.count()
class BulkEditObjectsSerializer(SerializerWithPerms, SetPermissionsMixin):
+5 -4
View File
@@ -636,7 +636,7 @@ def update_filename_and_move_files(
# so this is not the end of the world.
# B: if moving the original file failed, nothing has changed
# anyway.
pass
logger.exception("Error reverting document changes")
# restore old values on the instance
instance.filename = old_filename
@@ -1101,10 +1101,11 @@ def _extract_input_data(
if v is None or k.startswith("_"):
continue
if isinstance(v, datetime.date):
v = v.isoformat()
override_dict[k] = v.isoformat()
elif isinstance(v, Path):
v = str(v)
override_dict[k] = v
override_dict[k] = str(v)
else:
override_dict[k] = v
if override_dict:
data["overrides"] = override_dict
return data
+6 -7
View File
@@ -217,9 +217,9 @@ def consume_file(
overrides.filename or input_doc.original_file.name,
self.request.id,
) as status_mgr,
TemporaryDirectory(dir=settings.SCRATCH_DIR) as tmp_dir,
TemporaryDirectory(dir=settings.SCRATCH_DIR) as tmp_dir_name,
):
tmp_dir = Path(tmp_dir)
tmp_dir = Path(tmp_dir_name)
msg = None
for plugin_class in plugins:
plugin_name = plugin_class.NAME
@@ -261,7 +261,7 @@ def consume_file(
)
except Exception as e:
logger.exception(f"{plugin_name} failed: {e}")
logger.exception(f"{plugin_name} failed")
status_mgr.send_progress(
ProgressStatusOptions.FAILED,
f"{e}",
@@ -495,8 +495,8 @@ def empty_trash(doc_ids=None) -> None:
content_type=ContentType.objects.get_for_model(Document),
object_id__in=deleted_document_ids,
).delete()
except Exception as e: # pragma: no cover
logger.exception(f"Error while emptying trash: {e}")
except Exception: # pragma: no cover
logger.exception("Error while emptying trash")
finally:
models.signals.post_delete.disconnect(
cleanup_document_deletion,
@@ -832,9 +832,8 @@ def build_share_link_bundle(bundle_id: int) -> None:
logger.info("Built share link bundle %s", bundle.pk)
except Exception as exc:
logger.exception(
"Failed to build share link bundle %s: %s",
"Failed to build share link bundle %s",
bundle_id,
exc,
)
bundle.status = ShareLinkBundle.Status.FAILED
bundle.last_error = {
+4
View File
@@ -78,6 +78,10 @@ class PlaceholderString(str):
def __ne__(self, other) -> bool:
return not self.__eq__(other)
def __hash__(self) -> int:
# Equal to both "-none-" and "none", so hash to a single canonical value
return hash("-none-")
NO_VALUE_PLACEHOLDER = PlaceholderString("-none-")
+3 -3
View File
@@ -138,9 +138,9 @@ def parse_w_workflow_placeholders(
# We're good!
return rendered_template
except UndefinedError as e:
except UndefinedError:
# The undefined class logs this already for us
raise e
raise
except TemplateSyntaxError as e:
logger.warning(f"Template syntax error in title generation: {e}")
except SecurityError as e:
@@ -150,5 +150,5 @@ def parse_w_workflow_placeholders(
logger.warning(
f"Invalid title format '{text}', workflow not applied: {e}",
)
raise e
raise
return None
@@ -296,7 +296,7 @@ class TestRegexDateParser:
# simulate parse failure for malformed input
if "99/99/9999" in date_string or "bad date" in date_string:
raise Exception("parse failed for malformed date")
raise Exception("parse failed for malformed date") # noqa: TRY002 - simulates a generic parser failure
return None
@@ -57,13 +57,13 @@ class MultiprocessCommand(PaperlessCommand):
def handle(self, *args, **options):
items = list(range(5))
results = []
for result in self.process_parallel(
_double_value,
items,
description="Processing...",
):
results.append(result)
results = list(
self.process_parallel(
_double_value,
items,
description="Processing...",
),
)
successes = sum(1 for r in results if r.success)
self.stdout.write(f"Successes: {successes}")
@@ -66,7 +66,7 @@ class TestWriteBatchLockRetry:
)
mock_sleep = mocker.patch(
"documents.search._backend.time.sleep",
side_effect=lambda s: sleep_values.append(s),
side_effect=sleep_values.append,
)
# Should not raise — 4th attempt succeeds
@@ -111,7 +111,7 @@ class TestWriteBatchLockRetry:
sleep_values: list[float] = []
mocker.patch(
"documents.search._backend.time.sleep",
side_effect=lambda s: sleep_values.append(s),
side_effect=sleep_values.append,
)
for _ in range(50):
sleep_values.clear()
+2 -2
View File
@@ -1003,8 +1003,8 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
for correspondent in response.data[field]:
self.assertEqual(correspondent["document_count"], 0)
self.assertCountEqual(
map(lambda c: c["id"], response.data[field]),
map(lambda c: c["id"], Entity.objects.values("id")),
(c["id"] for c in response.data[field]),
(c["id"] for c in Entity.objects.values("id")),
)
def test_api_selection_data(self) -> None:
+2 -2
View File
@@ -18,8 +18,8 @@ class MockOpenIDProvider:
def get_brands(self):
default_servers = [
dict(id="yahoo", name="Yahoo", openid_url="http://me.yahoo.com"),
dict(id="hyves", name="Hyves", openid_url="http://hyves.nl"),
{"id": "yahoo", "name": "Yahoo", "openid_url": "http://me.yahoo.com"},
{"id": "hyves", "name": "Hyves", "openid_url": "http://hyves.nl"},
]
return default_servers
+2 -2
View File
@@ -205,12 +205,12 @@ class TestBarcode(
- Barcode is detected on page 1 (zero indexed)
"""
for test_file in [
for test_filename in [
"patch-code-t-middle-reverse.pdf",
"patch-code-t-middle-distorted.pdf",
"patch-code-t-middle-fuzzy.pdf",
]:
test_file = self.BARCODE_SAMPLE_DIR / test_file
test_file = self.BARCODE_SAMPLE_DIR / test_filename
with self.get_reader(test_file) as reader:
reader.detect()
+3 -3
View File
@@ -777,7 +777,7 @@ class TestPDFActions(DirectoriesMixin, TestCase):
sig.set.return_value.apply_async.side_effect = Exception("boom")
mock_consume_file.return_value = sig
with self.assertRaises(Exception):
with self.assertRaises(Exception): # noqa: B017 - mock injects a bare Exception
bulk_edit.merge(doc_ids, delete_originals=True)
self.doc1.refresh_from_db()
@@ -1318,7 +1318,7 @@ class TestPDFActions(DirectoriesMixin, TestCase):
sig.apply_async.side_effect = Exception("boom")
mock_chord.return_value = sig
with self.assertRaises(Exception):
with self.assertRaises(Exception): # noqa: B017 - mock injects a bare Exception
bulk_edit.edit_pdf(doc_ids, operations, delete_original=True)
self.doc2.refresh_from_db()
@@ -1430,7 +1430,7 @@ class TestPDFActions(DirectoriesMixin, TestCase):
{"page": 9999}, # invalid page, forces error during PDF load
]
with self.assertLogs("paperless.bulk_edit", level="ERROR"):
with self.assertRaises(Exception):
with self.assertRaises(ValueError):
bulk_edit.edit_pdf(doc_ids, operations)
mock_group.assert_not_called()
mock_consume_file.assert_not_called()
+1 -1
View File
@@ -783,7 +783,7 @@ class TestClassifier(DirectoriesMixin, TestCase):
Path(settings.MODEL_FILE).touch()
mock_load.side_effect = Exception()
with self.assertRaises(Exception):
with self.assertRaises(Exception): # noqa: B017 - mock injects a bare Exception
load_classifier(raise_exception=True)
+2 -2
View File
@@ -137,7 +137,7 @@ class FaultyParser(_BaseNewStyleParser):
class FaultyGenericExceptionParser(_BaseNewStyleParser):
def parse(self, document_path, mime_type, *, produce_archive: bool = True) -> None:
raise Exception("Generic exception.")
raise Exception("Generic exception.") # noqa: TRY002 - deliberately not a ParseError
def fake_magic_from_file(file, *, mime=False): # NOSONAR
@@ -1333,7 +1333,7 @@ class PreConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
script_calls = [
call
for call in m.call_args_list
if call.args and call.args[0] and call.args[0][0] not in ("pdftotext",)
if call.args and call.args[0] and call.args[0][0] != "pdftotext"
]
self.assertEqual(script_calls, [])
@@ -1,239 +0,0 @@
from __future__ import annotations
from types import SimpleNamespace
from typing import TYPE_CHECKING
import pytest
from django.db import connection
from django.test.utils import CaptureQueriesContext
from rest_framework import status
from documents.models import Document
from documents.tests.factories import DocumentFactory
from documents.versioning import LATEST_VERSION_CONTENT_PREFETCH_ATTR
from documents.versioning import has_prefetched_effective_content
from documents.versioning import latest_version_content_prefetch
from documents.views import DocumentViewSet
if TYPE_CHECKING:
from rest_framework.test import APIClient
class TestNeedsEffectiveContentAnnotation:
"""
DocumentViewSet._needs_effective_content_annotation() decides whether
the effective_content correlated subquery is worth attaching to the
queryset at all -- see TestDocumentListEffectiveContentAnnotation below
for why. This only checks that decision's own logic (a plain query-param
membership test), not that Django/DRF's filtering machinery works.
"""
@pytest.mark.parametrize(
("params", "expected"),
[
({}, False),
({"ordering": "-added"}, False),
({"tags__id__in": "1,2"}, False),
({"search": ""}, False),
({"search": " "}, False),
({"content__icontains": ""}, False),
({"search": "foo"}, True),
({"title_content": "foo"}, True),
({"content__istartswith": "foo"}, True),
({"content__iendswith": "foo"}, True),
({"content__icontains": "foo"}, True),
({"content__iexact": "foo"}, True),
],
)
def test_detects_content_filter_params(
self,
params: dict[str, str],
expected: bool, # noqa: FBT001
) -> None:
# GIVEN a view bound to a request carrying the given query params
view = DocumentViewSet()
view.request = SimpleNamespace(query_params=params)
# WHEN checking whether the effective_content annotation is needed
# THEN it's needed only for requests that actually filter on it
assert view._needs_effective_content_annotation() is expected
@pytest.mark.django_db
class TestDocumentListEffectiveContentAnnotation:
"""
DocumentViewSet.get_queryset() only attaches the effective_content
correlated subquery when a request actually filters on it. Attaching it
unconditionally re-executes it once per candidate row before the page's
LIMIT is applied -- fine on SQLite/Postgres, but pathological on
MariaDB's default cardinality estimation for the root_document_id
self-join once candidate counts get large (see the root_document_id /
effective_content perf investigation).
"""
def test_list_without_content_filter_skips_annotation_but_returns_latest_content(
self,
admin_client: APIClient,
) -> None:
# GIVEN a root document whose latest version has different content
root = DocumentFactory(content="old-root-content")
DocumentFactory(
root_document=root,
version_index=1,
content="new-version-content",
)
# WHEN listing documents with no search/content-filter param
with CaptureQueriesContext(connection) as ctx:
response = admin_client.get("/api/documents/?fields=id,content")
# THEN the response still reflects the latest version's content...
assert response.status_code == status.HTTP_200_OK
assert response.data["results"] == [
{"id": root.id, "content": "new-version-content"},
]
# ...without the database ever evaluating effective_content per row
assert not any(
"effective_content" in query["sql"] for query in ctx.captured_queries
)
def test_latest_version_content_prefetch_carries_only_the_newest_version(
self,
) -> None:
# GIVEN a root document with two versions
root = DocumentFactory(content="root-content")
DocumentFactory(
root_document=root,
version_index=1,
content="older-version-content",
)
DocumentFactory(
root_document=root,
version_index=2,
content="newest-version-content",
)
# WHEN fetching the root through latest_version_content_prefetch()
fetched_root = (
Document.objects.filter(pk=root.pk)
.prefetch_related(
latest_version_content_prefetch(),
)
.get()
)
# THEN the prefetch carries only the single newest version, not
# every historical version's content (the whole point of not
# reusing the metadata-only "versions" prefetch for this)
latest = getattr(fetched_root, LATEST_VERSION_CONTENT_PREFETCH_ATTR)
assert [v.content for v in latest] == ["newest-version-content"]
class TestHasPrefetchedEffectiveContent:
"""
DocumentSerializer.to_representation() only calls get_effective_content()
when has_prefetched_effective_content() says it's cheap -- otherwise a
caller that never set up an annotation or prefetch (TrashView,
GlobalSearchView, which build their own querysets and don't display
content at all) would pay for a per-instance query nobody asked for.
"""
def test_false_with_no_annotation_or_prefetch(self) -> None:
document = Document()
assert has_prefetched_effective_content(document) is False
def test_true_with_effective_content_annotation(self) -> None:
document = Document()
document.effective_content = "resolved"
assert has_prefetched_effective_content(document) is True
def test_true_with_lean_prefetch_attr_even_when_empty(self) -> None:
document = Document()
setattr(document, LATEST_VERSION_CONTENT_PREFETCH_ATTR, [])
assert has_prefetched_effective_content(document) is True
def test_true_with_metadata_versions_prefetch_cache(self) -> None:
document = Document()
document._prefetched_objects_cache = {"versions": []}
assert has_prefetched_effective_content(document) is True
def _get_effective_content_fallback_queries(
ctx: CaptureQueriesContext,
) -> list[dict[str, str]]:
"""
Document.get_effective_content()'s per-instance fallback (no annotation,
no prefetch) is a `.values_list("content", flat=True).first()` query --
a SELECT of just the content column. Distinct from get_versions()'s own,
unrelated per-instance metadata query (id/checksum/added/etc, no
content) run to build the "versions" response field, which isn't part
of what this test file covers.
"""
return [
q
for q in ctx.captured_queries
if q["sql"].startswith('SELECT "documents_document"."content" FROM')
]
@pytest.mark.django_db
class TestTrashAndGlobalSearchDoNotResolveEffectiveContent:
"""
TrashView and GlobalSearchView serialize Document instances with
DocumentSerializer too, but build their querysets independently of
DocumentViewSet.get_queryset() -- and neither actually displays
document content. They should keep showing the document's own,
unresolved content with no extra query, exactly as before
effective_content resolution existed.
"""
def test_trash_list_shows_unresolved_content_with_no_extra_query(
self,
admin_client: APIClient,
) -> None:
# GIVEN a trashed root document whose own content differs from what
# a (also trashed, since deletion cascades) version would have had
root = DocumentFactory(content="own-content")
DocumentFactory(
root_document=root,
version_index=1,
content="version-content",
)
root.delete()
# WHEN listing trash
with CaptureQueriesContext(connection) as ctx:
response = admin_client.get("/api/trash/")
# THEN the response shows the document's own content...
assert response.status_code == status.HTTP_200_OK
[result] = [r for r in response.data["results"] if r["id"] == root.id]
assert result["content"] == "own-content"
# ...without ever querying for versions to resolve it
assert _get_effective_content_fallback_queries(ctx) == []
def test_global_search_db_only_shows_unresolved_content_with_no_extra_query(
self,
admin_client: APIClient,
) -> None:
# GIVEN a root document, findable by title, whose own content
# differs from its latest version's
root = DocumentFactory(title="findme", content="own-content")
DocumentFactory(
root_document=root,
version_index=1,
content="version-content",
)
# WHEN using the global search endpoint's db_only mode
with CaptureQueriesContext(connection) as ctx:
response = admin_client.get(
"/api/search/?query=findme&db_only=true",
)
# THEN the response shows the document's own content...
assert response.status_code == status.HTTP_200_OK
[result] = [d for d in response.data["documents"] if d["id"] == root.id]
assert result["content"] == "own-content"
# ...without ever querying for versions to resolve it
assert _get_effective_content_fallback_queries(ctx) == []
+8 -1
View File
@@ -44,6 +44,7 @@ from documents import tasks
from documents.data_models import ConsumableDocument
from documents.data_models import DocumentMetadataOverrides
from documents.data_models import DocumentSource
from documents.matching import UnsupportedWorkflowTriggerTypeError
from documents.matching import document_matches_workflow
from documents.matching import existing_document_matches_workflow
from documents.matching import prefilter_documents_by_workflowtrigger
@@ -2851,7 +2852,13 @@ class TestWorkflows(
doc = Document.objects.create(
title="test",
)
self.assertRaises(Exception, document_matches_workflow, doc, w, 99)
self.assertRaises(
UnsupportedWorkflowTriggerTypeError,
document_matches_workflow,
doc,
w,
99,
)
def test_removal_action_document_updated_workflow(self) -> None:
"""
+9 -5
View File
@@ -21,28 +21,32 @@ def uri_validator(value: str, allowed_schemes: set[str] | None = None) -> None:
parts = urlparse(value)
if not parts.scheme:
raise ValidationError(
_(f"Unable to parse URI {value}, missing scheme"),
_("Unable to parse URI %(value)s, missing scheme"),
params={"value": value},
)
elif not parts.netloc and not parts.path:
raise ValidationError(
_(f"Unable to parse URI {value}, missing net location or path"),
_("Unable to parse URI %(value)s, missing net location or path"),
params={"value": value},
)
if allowed_schemes and parts.scheme not in allowed_schemes:
raise ValidationError(
_(
f"URI scheme '{parts.scheme}' is not allowed. Allowed schemes: {', '.join(allowed_schemes)}",
"URI scheme '%(scheme)s' is not allowed. Allowed schemes: %(allowed_schemes)s",
),
params={"value": value, "scheme": parts.scheme},
params={
"value": value,
"scheme": parts.scheme,
"allowed_schemes": ", ".join(allowed_schemes),
},
)
except ValidationError:
raise
except Exception as e:
raise ValidationError(
_(f"Unable to parse URI {value}"),
_("Unable to parse URI %(value)s"),
params={"value": value},
) from e
-65
View File
@@ -7,12 +7,9 @@ from typing import Any
from django.db.models import F
from django.db.models import OuterRef
from django.db.models import Prefetch
from django.db.models import QuerySet
from django.db.models import Subquery
from django.db.models import Window
from django.db.models.functions import Coalesce
from django.db.models.functions import RowNumber
from documents.models import Document
@@ -46,68 +43,6 @@ def annotate_effective_content(documents: QuerySet[Document]) -> QuerySet[Docume
)
LATEST_VERSION_CONTENT_PREFETCH_ATTR = "_latest_version_content_prefetch"
def latest_version_content_prefetch() -> Prefetch:
"""
A Prefetch for Document.versions scoped to just the newest version's
content, for get_effective_content()'s fallback when no SQL annotation
is present.
Deliberately not merged into a metadata-only "versions" prefetch (the one
used for the serialized versions list): that one fetches every historical
version of every document, and pulling full OCR content for versions
nobody will read wastes DB transfer/memory at scale. This one is windowed
down to a single row per root, then bounded by Prefetch's own IN-list to
whatever page/result set it's attached to -- one cheap bulk query total,
not one per document and not one per version.
"""
return Prefetch(
"versions",
queryset=(
Document.objects.filter(
root_document_id__isnull=False,
deleted_at__isnull=True,
)
.annotate(
rn=Window(
RowNumber(),
partition_by=F("root_document_id"),
order_by=[
F("version_index").desc(nulls_last=True),
F("id").desc(),
],
),
)
.filter(rn=1)
.only("id", "root_document_id", "content")
),
to_attr=LATEST_VERSION_CONTENT_PREFETCH_ATTR,
)
def has_prefetched_effective_content(document: Document) -> bool:
"""
True if document.get_effective_content() can answer without an extra
per-instance query -- an SQL ``effective_content`` annotation, the lean
latest_version_content_prefetch(), or the metadata-only "versions"
prefetch is already present on the instance.
Callers that haven't set any of those up (e.g. views that build their
own querysets independently of DocumentViewSet.get_queryset(), like
TrashView or GlobalSearchView) intentionally don't pay for version-aware
content resolution -- see DocumentSerializer.to_representation(), which
uses this to decide whether to call get_effective_content() at all.
"""
if hasattr(document, "effective_content"):
return True
if getattr(document, LATEST_VERSION_CONTENT_PREFETCH_ATTR, None) is not None:
return True
prefetched_cache = getattr(document, "_prefetched_objects_cache", None)
return isinstance(prefetched_cache, dict) and "versions" in prefetched_cache
def sort_versions_newest_first(documents: list[Document]) -> list[Document]:
"""
Same sorting as versions_newest_first()
+29 -72
View File
@@ -233,7 +233,6 @@ from documents.versioning import VersionResolutionError
from documents.versioning import get_latest_version_for_root
from documents.versioning import get_request_version_param
from documents.versioning import get_root_document
from documents.versioning import latest_version_content_prefetch
from documents.versioning import resolve_requested_version_for_root
from documents.versioning import versions_newest_first
from paperless import version
@@ -1073,40 +1072,12 @@ class DocumentViewSet(
],
}
# Query params whose filtering needs effective_content evaluated in SQL
# against every candidate row -- see _needs_effective_content_annotation().
_CONTENT_FILTER_PARAMS = (
"search", # DRF SearchFilter's search_fields includes effective_content
"title_content",
"content__istartswith",
"content__iendswith",
"content__icontains",
"content__iexact",
)
def _needs_effective_content_annotation(self) -> bool:
# effective_content is a per-row correlated subquery resolving each
# document's latest version. Cheap when evaluated only for the page
# that survives filtering/sorting/pagination (the common case, via
# the "versions" prefetch + Document.get_effective_content()'s
# fallback), but if anything filters *on* it, the database has to
# evaluate it for every candidate row before the LIMIT is reached --
# pathological on MariaDB specifically for the root_document_id
# self-join once real candidate counts get large. Everything on this
# list is deprecated in favor of the Tantivy-backed search endpoint
# (see filters.py's TitleContentFilter/EffectiveContentFilter docs),
# so keep paying that cost only when one is actually used. Checked as
# a stripped, non-blank value (not just key presence) to match how
# DRF's SearchFilter and TitleContentFilter/EffectiveContentFilter
# themselves no-op on a blank value -- otherwise an empty `?search=`
# or a saved view with a cleared text filter would still pay for the
# annotation despite applying no actual predicate.
params = self.request.query_params
return any(
params.get(param, "").strip() for param in self._CONTENT_FILTER_PARAMS
)
def get_queryset(self):
latest_version_content = Subquery(
versions_newest_first(
Document.objects.filter(root_document=OuterRef("pk")),
).values("content")[:1],
)
# A correlated subquery avoids the LEFT JOIN + Count() this used to
# be, which forced a GROUP BY aggregate over every matching document
# before the query could even be sorted or limited.
@@ -1126,9 +1097,10 @@ class DocumentViewSet(
# ObjectFilter.filter(). A blanket .distinct() here forces the
# database to fully sort and dedupe every visible document before
# it can apply LIMIT, which is disastrous at scale.
queryset = (
return (
Document.objects.filter(root_document__isnull=True)
.order_by("-created", "-id")
.annotate(effective_content=Coalesce(latest_version_content, F("content")))
.annotate(num_notes=Coalesce(note_count, 0))
.select_related("correspondent", "storage_path", "document_type", "owner")
.prefetch_related(
@@ -1143,7 +1115,6 @@ class DocumentViewSet(
"version_index",
),
),
latest_version_content_prefetch(),
"tags",
Prefetch(
"custom_fields",
@@ -1153,16 +1124,6 @@ class DocumentViewSet(
Prefetch("notes", queryset=Note.objects.select_related("user")),
)
)
if self._needs_effective_content_annotation():
latest_version_content = Subquery(
versions_newest_first(
Document.objects.filter(root_document=OuterRef("pk")),
).values("content")[:1],
)
queryset = queryset.annotate(
effective_content=Coalesce(latest_version_content, F("content")),
)
return queryset
def get_serializer(self, *args, **kwargs):
fields_param = self.request.query_params.get("fields", None)
@@ -1479,7 +1440,7 @@ class DocumentViewSet(
try:
lang = detect(doc.content)
except Exception:
pass
logger.debug("Unable to detect language for document %s", doc.pk)
meta["lang"] = lang
return Response(meta)
@@ -1517,13 +1478,12 @@ class DocumentViewSet(
with get_date_parser() as date_parser:
gen = date_parser.parse(doc.filename, doc.content)
dates = sorted(
{
i
for i in itertools.islice(
set(
itertools.islice(
gen,
settings.NUMBER_OF_SUGGESTED_DATES,
)
},
),
),
)
resp_data = {
@@ -1607,21 +1567,16 @@ class DocumentViewSet(
except ValueError as exc:
logger.exception(
"Invalid AI configuration while generating suggestions for "
"document %s: %s",
"document %s",
doc.pk,
exc,
exc_info=True,
)
raise ValidationError(
{"ai": [_("Invalid AI configuration.")]},
) from exc
except LLMTimeoutError as exc:
except LLMTimeoutError:
logger.exception(
"AI backend timed out while generating suggestions for "
"document %s: %s",
"AI backend timed out while generating suggestions for document %s",
doc.pk,
exc,
exc_info=True,
)
return Response(
{"ai": [_("AI backend request timed out.")]},
@@ -2094,7 +2049,7 @@ class DocumentViewSet(
doc_name, doc_data = serializer.validated_data.get("document")
version_label = serializer.validated_data.get("version_label")
t = int(mktime(datetime.now().timetuple()))
t = int(mktime(datetime.now().timetuple())) # noqa: DTZ005 - mktime() requires a local time tuple
settings.SCRATCH_DIR.mkdir(parents=True, exist_ok=True)
@@ -3364,7 +3319,7 @@ class PostDocumentView(GenericAPIView[Any]):
cf = serializer.validated_data.get("custom_fields")
from_webui = serializer.validated_data.get("from_webui")
t = int(mktime(datetime.now().timetuple()))
t = int(mktime(datetime.now().timetuple())) # noqa: DTZ005 - mktime() requires a local time tuple
settings.SCRATCH_DIR.mkdir(parents=True, exist_ok=True)
@@ -4174,7 +4129,7 @@ class UiSettingsView(GenericAPIView[Any]):
user_resp["last_name"] = user.last_name
# strip <app_label>.
roles = map(lambda perm: re.sub(r"^\w+.", "", perm), user.get_all_permissions())
roles = (re.sub(r"^\w+.", "", perm) for perm in user.get_all_permissions())
return Response(
{
"user": user_resp,
@@ -5201,11 +5156,11 @@ class SystemStatusView(PassUserMixin):
f"{m.app}.{m.name}"
for m in MigrationRecorder.Migration.objects.all().order_by("id")
]
except Exception as e: # pragma: no cover
except Exception: # pragma: no cover
applied_migrations = []
db_status = "ERROR"
logger.exception(
f"System status detected a possible problem while connecting to the database: {e}",
"System status detected a possible problem while connecting to the database",
)
db_error = "Error connecting to database, check logs for more detail."
@@ -5221,10 +5176,10 @@ class SystemStatusView(PassUserMixin):
try:
client.ping()
redis_status = "OK"
except Exception as e:
except Exception:
redis_status = "ERROR"
logger.exception(
f"System status detected a possible problem while connecting to redis: {e}",
"System status detected a possible problem while connecting to redis",
)
redis_error = "Error connecting to redis, check logs for more detail."
@@ -5254,10 +5209,10 @@ class SystemStatusView(PassUserMixin):
else:
celery_active = "WARNING"
celery_error = "Celery worker responded unexpectedly."
except Exception as e:
except Exception:
celery_active = "ERROR"
logger.exception(
f"System status detected a possible problem while connecting to celery: {e}",
"System status detected a possible problem while connecting to celery",
)
celery_error = "Error connecting to celery, check logs for more detail."
@@ -5272,13 +5227,15 @@ class SystemStatusView(PassUserMixin):
index_dir = settings.INDEX_DIR
mtimes = [p.stat().st_mtime for p in index_dir.iterdir() if p.is_file()]
index_last_modified = (
make_aware(datetime.fromtimestamp(max(mtimes))) if mtimes else None
make_aware(datetime.fromtimestamp(max(mtimes))) # noqa: DTZ006 - make_aware() requires a naive datetime
if mtimes
else None
)
except Exception as e:
except Exception:
index_status = "ERROR"
index_error = "Error opening index, check logs for more detail."
logger.exception(
f"System status detected a possible problem while opening the index: {e}",
"System status detected a possible problem while opening the index",
)
index_last_modified = None
+5 -5
View File
@@ -66,7 +66,7 @@ def build_workflow_action_context(
else None
)
filename = document.original_file if document.original_file else ""
filename = document.original_file or ""
return {
"title": overrides.title
if overrides and overrides.title
@@ -179,9 +179,9 @@ def execute_email_action(
f"Sent {n_messages} notification email(s) to {action.email.to}",
extra={"group": logging_group},
)
except Exception as e:
except Exception:
logger.exception(
f"Error occurred sending notification email: {e}",
"Error occurred sending notification email",
extra={"group": logging_group},
)
@@ -265,9 +265,9 @@ def execute_webhook_action(
f"Webhook to {action.webhook.url} queued",
extra={"group": logging_group},
)
except Exception as e:
except Exception:
logger.exception(
f"Error occurred sending webhook: {e}",
"Error occurred sending webhook",
extra={"group": logging_group},
)
+1 -1
View File
@@ -47,7 +47,7 @@ def resolve_date(dates: list[str]) -> date | None:
"""
for value in dates:
try:
return datetime.strptime(value, "%Y-%m-%d").date()
return datetime.strptime(value, "%Y-%m-%d").date() # noqa: DTZ007 - only the calendar date is used, time/tz is discarded
except (TypeError, ValueError):
logger.debug("Ignoring unparsable suggested date %s", value)
return None
+1 -1
View File
@@ -70,6 +70,6 @@ def send_webhook(
logger.error(
f"Failed attempt sending webhook to {url}: {e}",
)
raise e
raise
finally:
transport.close()
+2 -1
View File
@@ -241,7 +241,7 @@ def check_v3_minimum_upgrade_version(
return []
logger = logging.getLogger(__name__)
last_applied = sorted(applied)[-1] if applied else "(none)"
last_applied = max(applied) if applied else "(none)"
logger.error(
"V3 upgrade check failed: last applied documents migration is %r. "
"Expected '1075_workflowaction_order' (v2.20.15). "
@@ -341,6 +341,7 @@ def get_tesseract_langs():
proc = subprocess.run(
[shutil.which("tesseract"), "--list-langs"],
capture_output=True,
check=False,
)
# Decode bytes to string, split on newlines, trim out the header
+3 -3
View File
@@ -84,7 +84,7 @@ def get_parser_registry() -> ParserRegistry:
ParserRegistry
The shared registry singleton.
"""
global _registry, _discovery_complete
global _registry, _discovery_complete # noqa: PLW0603 - module-level singleton, no class to hold this state
with _lock:
if _registry is None:
@@ -113,7 +113,7 @@ def init_builtin_parsers() -> None:
-------
None
"""
global _registry
global _registry # noqa: PLW0603 - module-level singleton, no class to hold this state
with _lock:
if _registry is None:
@@ -137,7 +137,7 @@ def reset_parser_registry() -> None:
-------
None
"""
global _registry, _discovery_complete
global _registry, _discovery_complete # noqa: PLW0603 - module-level singleton, no class to hold this state
_registry = None
_discovery_complete = False
+2 -2
View File
@@ -76,7 +76,7 @@ class RemoteEngineConfig:
def engine_is_valid(self) -> bool:
"""Return True when the engine is known and fully configured."""
return (
self.engine in ("azureai",)
self.engine == "azureai"
and self.api_key is not None
and not (self.engine == "azureai" and self.endpoint is None)
)
@@ -467,7 +467,7 @@ class RemoteDocumentParser:
return result.content
except Exception as e:
logger.exception("Azure AI Vision parsing failed: %s", e)
logger.exception("Azure AI Vision parsing failed")
raise ParseError(f"Azure AI Vision parsing failed: {e}") from e
finally:
+4 -3
View File
@@ -306,8 +306,9 @@ def extract_pdf_metadata(
for key, value in meta.items():
if isinstance(value, list):
value = " ".join(str(e) for e in value)
value = str(value)
str_value = " ".join(str(e) for e in value)
else:
str_value = str(value)
try:
m = namespace_pattern.match(key)
@@ -329,7 +330,7 @@ def extract_pdf_metadata(
namespace=namespace,
prefix=meta.REVERSE_NS[namespace],
key=key_value,
value=value,
value=str_value,
),
)
except Exception as e:
+14 -9
View File
@@ -294,7 +294,7 @@ if _CHANNELS_BACKEND.startswith("channels_redis."):
###############################################################################
EMAIL_HOST: Final[str] = os.getenv("PAPERLESS_EMAIL_HOST", "localhost")
EMAIL_PORT: Final[int] = int(os.getenv("PAPERLESS_EMAIL_PORT", 25))
EMAIL_PORT: Final[int] = get_int_from_env("PAPERLESS_EMAIL_PORT", 25)
EMAIL_HOST_USER: Final[str] = os.getenv("PAPERLESS_EMAIL_HOST_USER", "")
EMAIL_HOST_PASSWORD: Final[str] = os.getenv("PAPERLESS_EMAIL_HOST_PASSWORD", "")
DEFAULT_FROM_EMAIL: Final[str] = os.getenv("PAPERLESS_EMAIL_FROM", EMAIL_HOST_USER)
@@ -381,8 +381,9 @@ ACCOUNT_SESSION_REMEMBER = get_bool_from_env(
"True",
)
SESSION_EXPIRE_AT_BROWSER_CLOSE = not ACCOUNT_SESSION_REMEMBER
SESSION_COOKIE_AGE = int(
os.getenv("PAPERLESS_SESSION_COOKIE_AGE", 60 * 60 * 24 * 7 * 3),
SESSION_COOKIE_AGE = get_int_from_env(
"PAPERLESS_SESSION_COOKIE_AGE",
60 * 60 * 24 * 7 * 3,
)
# https://docs.djangoproject.com/en/5.1/ref/settings/#std-setting-SESSION_ENGINE
SESSION_ENGINE = "django.contrib.sessions.backends.cached_db"
@@ -395,7 +396,6 @@ if AUTO_LOGIN_USERNAME:
def _parse_remote_user_settings() -> str:
global MIDDLEWARE, AUTHENTICATION_BACKENDS, REST_FRAMEWORK
enable = get_bool_from_env("PAPERLESS_ENABLE_HTTP_REMOTE_USER")
enable_api = get_bool_from_env("PAPERLESS_ENABLE_HTTP_REMOTE_USER_API")
if enable or enable_api:
@@ -454,7 +454,6 @@ if ALLOWED_HOSTS != ["*"]:
def _parse_paperless_url():
global CSRF_TRUSTED_ORIGINS, CORS_ALLOWED_ORIGINS, ALLOWED_HOSTS
url = os.getenv("PAPERLESS_URL")
if url:
CSRF_TRUSTED_ORIGINS.append(url)
@@ -614,8 +613,8 @@ USE_TZ = True
LOGGING_DIR.mkdir(parents=True, exist_ok=True)
LOGROTATE_MAX_SIZE = os.getenv("PAPERLESS_LOGROTATE_MAX_SIZE", 1024 * 1024)
LOGROTATE_MAX_BACKUPS = os.getenv("PAPERLESS_LOGROTATE_MAX_BACKUPS", 20)
LOGROTATE_MAX_SIZE = get_int_from_env("PAPERLESS_LOGROTATE_MAX_SIZE", 1024 * 1024)
LOGROTATE_MAX_BACKUPS = get_int_from_env("PAPERLESS_LOGROTATE_MAX_BACKUPS", 20)
LOGGING = {
"version": 1,
@@ -811,9 +810,15 @@ IGNORABLE_FILES: Final[list[str]] = [
"Thumbs.db",
]
CONSUMER_POLLING_INTERVAL = float(os.getenv("PAPERLESS_CONSUMER_POLLING_INTERVAL", 0))
CONSUMER_POLLING_INTERVAL = get_float_from_env(
"PAPERLESS_CONSUMER_POLLING_INTERVAL",
0.0,
)
CONSUMER_STABILITY_DELAY = float(os.getenv("PAPERLESS_CONSUMER_STABILITY_DELAY", 5))
CONSUMER_STABILITY_DELAY = get_float_from_env(
"PAPERLESS_CONSUMER_STABILITY_DELAY",
5.0,
)
CONSUMER_DELETE_DUPLICATES = get_bool_from_env("PAPERLESS_CONSUMER_DELETE_DUPLICATES")
+1 -1
View File
@@ -111,7 +111,7 @@ def parse_dict_from_str(
return False
settings: dict[str, Any] = copy.deepcopy(defaults) if defaults else {}
_type_map = type_map if type_map else {}
_type_map = type_map or {}
if not env_str:
return settings
@@ -114,17 +114,17 @@ def test_cache_hit_when_enabled() -> None:
assert settings.CACHALOT_TIMEOUT == 1
# Read a table to populate the cache
list(list(Tag.objects.values_list("id", flat=True)))
list(Tag.objects.values_list("id", flat=True))
# Invalidate the cache then read the database, there should be DB hit
invalidate_db_cache()
with CaptureQueriesContext(connection) as ctx:
list(list(Tag.objects.values_list("id", flat=True)))
list(Tag.objects.values_list("id", flat=True))
assert len(ctx)
# Doing the same request again should hit the cache, not the DB
with CaptureQueriesContext(connection) as ctx:
list(list(Tag.objects.values_list("id", flat=True)))
list(Tag.objects.values_list("id", flat=True))
assert not len(ctx)
# Wait the end of TTL
@@ -133,7 +133,7 @@ def test_cache_hit_when_enabled() -> None:
# Read the DB again. The DB should be hit because the cache has expired
with CaptureQueriesContext(connection) as ctx:
list(list(Tag.objects.values_list("id", flat=True)))
list(Tag.objects.values_list("id", flat=True))
assert len(ctx)
# Invalidate the cache at the end of test
@@ -149,7 +149,7 @@ def test_cache_is_disabled_by_default() -> None:
# Read the table multiple times: the DB should always be hit without cache
for _ in range(3):
with CaptureQueriesContext(connection) as ctx:
list(list(Tag.objects.values_list("id", flat=True)))
list(Tag.objects.values_list("id", flat=True))
assert len(ctx)
# Invalidate the cache at the end of test
+1 -1
View File
@@ -59,7 +59,7 @@ def test_ocr_to_dateparser_languages_exception(
raise RuntimeError("Simulated error")
with caplog.at_level(logging.WARNING):
monkeypatch.setattr(utils, "LocaleDataLoader", lambda: DummyLoader())
monkeypatch.setattr(utils, "LocaleDataLoader", DummyLoader)
result = utils.ocr_to_dateparser_languages("eng+fra")
assert result == []
assert (
+2 -2
View File
@@ -103,8 +103,8 @@ def stream_chat_with_documents(
documents,
output_language=output_language,
)
except Exception as e:
logger.exception("Failed to stream document chat response: %s", e)
except Exception:
logger.exception("Failed to stream document chat response")
yield CHAT_ERROR_MESSAGE
+1 -1
View File
@@ -167,7 +167,7 @@ def test_get_ai_document_classification_failure(mock_run_llm_query, mock_documen
"""
mock_run_llm_query.side_effect = Exception("LLM query failed")
with pytest.raises(Exception):
with pytest.raises(Exception): # noqa: B017 - mock injects a bare Exception
get_ai_document_classification(mock_document)
@@ -21,5 +21,6 @@ class TestLazyAiImports:
capture_output=True,
text=True,
cwd=_SRC_DIR,
check=False,
)
assert result.returncode == 0, result.stdout + result.stderr
+5 -6
View File
@@ -7,7 +7,6 @@ import ssl
import tempfile
import traceback
import unicodedata
from datetime import date
from datetime import timedelta
from fnmatch import fnmatch
from pathlib import Path
@@ -406,7 +405,7 @@ def make_criterias(rule: MailRule, *, supports_gmail_labels: bool):
Returns criteria to be applied to MailBox.fetch for the given rule.
"""
maximum_age = date.today() - timedelta(days=rule.maximum_age)
maximum_age = timezone.localdate() - timedelta(days=rule.maximum_age)
criterias = {}
if rule.maximum_age > 0:
criterias["date_gte"] = maximum_age
@@ -723,9 +722,9 @@ class MailAccountHandler(LoggingMixin):
f"Rule {rule}: Stopping processing rules due to stop_processing flag",
)
break
except Exception as e:
except Exception:
self.log.exception(
f"Rule {rule}: Error while processing rule: {e}",
f"Rule {rule}: Error while processing rule",
)
except MailError:
raise
@@ -874,9 +873,9 @@ class MailAccountHandler(LoggingMixin):
total_processed_files += processed_files
mails_processed += 1
except Exception as e:
except Exception:
self.log.exception(
f"Rule {rule}: Error while processing mail {message.uid}: {e}",
f"Rule {rule}: Error while processing mail {message.uid}",
)
self.log.debug(f"Rule {rule}: Processed {mails_processed} matching mail(s)")
+5 -1
View File
@@ -11,6 +11,10 @@ from imap_tools import MailMessage
from documents.loggers import LoggingMixin
class MailDecryptionError(Exception):
pass
class MailMessagePreprocessor(abc.ABC):
"""
Defines the interface for preprocessors that alter messages before they are handled in MailAccountHandler
@@ -69,7 +73,7 @@ class MailMessageDecryptor(MailMessagePreprocessor, LoggingMixin):
f"Message decryption failed with status message "
f"{decrypted_raw_message.status}",
)
raise Exception(
raise MailDecryptionError(
f"Decryption failed: {decrypted_raw_message.status}, {decrypted_raw_message.stderr}",
)
self.log.debug("Message decrypted successfully.")
+1 -1
View File
@@ -50,7 +50,7 @@ class ProcessedMailFactory(DjangoModelFactory[ProcessedMail]):
rule = factory.SubFactory(MailRuleFactory)
folder = "INBOX"
uid = factory.Sequence(lambda n: str(n))
uid = factory.Sequence(str)
subject = factory.Faker("sentence", nb_words=4)
received = factory.LazyFunction(timezone.now)
processed = factory.LazyFunction(timezone.now)
+1 -1
View File
@@ -214,7 +214,7 @@ class BogusMailBox(AbstractContextManager):
)
self.messages = list(filter(lambda m: m.uid not in uid_list, self.messages))
else:
raise Exception
raise Exception # noqa: TRY002 - test double simulating a generic mailbox failure
def fake_magic_from_buffer(buffer, *, mime=False):
@@ -14,6 +14,7 @@ from imap_tools import MailMessage
from paperless_mail.mail import MailAccountHandler
from paperless_mail.models import MailRule
from paperless_mail.preprocessor import MailDecryptionError
from paperless_mail.preprocessor import MailMessageDecryptor
from paperless_mail.tests.factories import MailAccountFactory
from paperless_mail.tests.test_mail import TestMail
@@ -82,7 +83,9 @@ class MessageEncryptor:
armor=True,
)
if not encrypted_data.ok:
raise Exception(f"Encryption failed: {encrypted_data.stderr}")
raise Exception( # noqa: TRY002 - test fixture setup, not production code
f"Encryption failed: {encrypted_data.stderr}",
)
encrypted_email_content = encrypted_data.data
new_email = MIMEMultipart("encrypted", protocol="application/pgp-encrypted")
@@ -184,7 +187,11 @@ class TestMailMessageGpgDecryptor(TestMail):
EMAIL_GNUPG_HOME=empty_gpg_home,
):
message_decryptor = MailMessageDecryptor()
self.assertRaises(Exception, message_decryptor.run, encrypted_message)
self.assertRaises(
MailDecryptionError,
message_decryptor.run,
encrypted_message,
)
finally:
# Clean up the temporary GPG home used only by this test
try:
+1 -2
View File
@@ -1,4 +1,3 @@
import datetime
import logging
from datetime import timedelta
from http import HTTPStatus
@@ -87,7 +86,7 @@ class MailAccountViewSet(PassUserMixin, ModelViewSet[MailAccount]):
@action(methods=["post"], detail=False)
def test(self, request):
logger = logging.getLogger("paperless_mail")
request.data["name"] = datetime.datetime.now().isoformat()
request.data["name"] = timezone.now().isoformat()
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
existing_account = None