Compare commits

..
Author SHA1 Message Date
stumpylog ee19e8dcff Remove old rules we don't actually need still 2026-09-01 15:29:53 -07:00
stumpylog b9cf877029 Enables more rules for migration dirs 2026-09-01 15:27:43 -07:00
stumpylog 82cc1016ad More linking 2026-09-01 15:23:56 -07:00
stumpylog dcedb531c6 Fixes misisng plw link 2026-09-01 15:18:43 -07:00
stumpylog 8bedb07cea Fixes doc link for PLR 2026-09-01 15:17:38 -07:00
stumpylog 5e34d566a2 Enables 'LOG' broadly 2026-09-01 15:16:29 -07:00
stumpylog 601ecce3f1 Adds 'INT' broadly 2026-09-01 15:15:47 -07:00
stumpylog d67beba9f6 Enables 'G' more broadly 2026-09-01 15:11:39 -07:00
stumpylog 3bc8ee8425 Fixes linting and does a little formatting 2026-09-01 15:09:43 -07:00
stumpylog fdef4a99a7 Chore: enable flake8-datetimez (DTZ) ruff rules
Full category (10/10 codes are all default in ruff 0.16). Of 54
hits, 46 were in test fixture code constructing naive datetimes for
comparison/input purposes only - added DTZ to the existing
per-file-ignores for */tests/*.py alongside E501/SIM117.

The 8 production hits:
- documents/consumer.py, documents/views.py (index_last_modified):
  suppressed with noqa - timezone.make_aware() requires a naive
  datetime, so wrapping fromtimestamp() in tz= would break it
- documents/double_sided.py (x2): switched to
  datetime.now(tz=UTC).timestamp() - behavior-identical since
  .timestamp() returns the same epoch value regardless of the
  attached tz, but now explicit
- documents/views.py (x2, upload temp file mtime): suppressed with
  noqa - mktime() requires a local time tuple, so an aware/UTC now()
  would introduce a timezone-offset bug
- documents/workflows/ai.py (AI suggested date parsing): suppressed
  with noqa - only .date() is used, time/tz is discarded
- paperless_mail/mail.py (IMAP fetch date filter): switched
  date.today() to timezone.localdate(), which respects
  settings.TIME_ZONE instead of the system clock - a real
  correctness improvement when they differ
- paperless_mail/views.py (placeholder name string): switched
  datetime.datetime.now() to timezone.now(), matching the app's
  existing aware-datetime convention
2026-09-01 15:03:18 -07:00
stumpylog d16d05a391 Chore: enable tryceratops (TRY002/004/201/203/401) ruff rules
Only the 5 default-subset codes; the rest of tryceratops is opt-in.

- 9 TRY201 (raise e -> raise) autofixed, preserving the traceback
  identically while dropping the redundant exception name
- 26 TRY401 (redundant exception object passed to logger.exception,
  which already logs it) fixed by removing the duplicate from the
  message; three sites still needed the exception object for
  something else (re-raising, or a separate logger.error call) and
  kept their binding
- 6 TRY002 (raise bare Exception): 2 production sites (documents/
  matching.py, paperless_mail/preprocessor.py) got dedicated
  exception classes, with their tests narrowed to match instead of
  asserting a blind Exception; the other 4 are deliberate generic
  failures in test doubles/fixtures, suppressed with noqa
2026-09-01 15:03:18 -07:00
stumpylog 10789e63cb Chore: enable pylint warning (PLW) ruff rules
Full category (not just the ruff-0.16 default subset). 35 hits:
- 4 PLW0108 (unnecessary lambda) autofixed
- 6 PLW2901 (loop/with variable shadowed) renamed to distinct names
- 2 PLW1510 (subprocess.run without explicit check) given check=False,
  matching existing behavior exactly
- 6 PLW0602 (global declared but never assigned) removed - these were
  all in-place mutations (.append/.insert), not reassignments, so
  `global` was already a no-op
- 7 PLW0603 (global statement) suppressed with noqa - these are
  genuine lazy-init singletons with no class to hold the state;
  refactoring them is a separate, larger change
- 6 PLW1508 (non-str/None env var default) fixed using the existing
  get_int_from_env/get_float_from_env typed helpers instead of raw
  os.getenv, which also fixes a real bug: LOGROTATE_MAX_SIZE and
  LOGROTATE_MAX_BACKUPS were never wrapped in int(), so a string env
  var value would have flowed into RotatingFileHandler as a string
- 1 PLW1641 (__eq__ without __hash__) fixed by adding __hash__ to
  PlaceholderString
2026-09-01 15:03:18 -07:00
stumpylog 2197781b39 Chore: enable flake8-gettext (INT001/002/003) ruff rules
All 4 hits in documents/validators.py were f-strings inside gettext
_() calls, which resolves the string before translation and breaks
extraction (confirmed: locale .po files literally contain the raw
"{value}" placeholder as msgid text). Fixed by using %(name)s-style
placeholders with Django ValidationError's existing params= kwarg,
which was already being passed but silently unused.
2026-09-01 15:03:17 -07:00
stumpylog 981492bb33 Chore: enable flake8-bandit S102/S110/S112 ruff rules
3 S110 (try-except-pass) hits, all fixed by adding a log call in the
except block rather than silently swallowing the exception, matching
this codebase's existing %s lazy-formatting logging convention.
Behavior is unchanged (still no re-raise) in all three spots.
2026-09-01 15:03:17 -07:00
stumpylog 0ef5ef5826 Chore: enable flake8-bugbear (B) default-subset ruff rules
3 B009 (getattr with a constant string, rewrite as attribute access)
hits autofixed. 7 B017 (assert blind Exception) hits: one narrowed
to the actual ValueError raised by bulk_edit.edit_pdf, the other six
suppressed with noqa since the code under test genuinely raises (or
a mock genuinely injects) a bare Exception, so a narrower assertion
would be wrong.

Only the 29 B codes ruff 0.16 enables by default; the rest of
flake8-bugbear needs a separate, deliberate decision.
2026-09-01 15:03:17 -07:00
stumpylog a6b1763149 Chore: enable flake8-logging-format G101/G202 ruff rules
G202 (redundant exc_info=True passed to logger.exception, which
already includes the traceback) had 2 hits in documents/views.py,
fixed manually since ruff has no autofix for it. G101 (hardcoded
password string) had zero hits.
2026-09-01 15:03:17 -07:00
stumpylog 90531525e2 Chore: enable flake8-2020 (YTT) ruff rules
Zero current violations. Full category (10/10 codes are all part of
ruff 0.16's default rule set already, so there's no non-default
subset to defer).
2026-09-01 15:03:17 -07:00
stumpylog 8f00bfa931 Chore: enable flake8-debugger T100 ruff rule
Zero current violations. Only T100 (import of pdb/ipdb/etc.) is part
of ruff 0.16's default rule set.
2026-09-01 15:03:17 -07:00
stumpylog a0479f1d9b Chore: enable flake8-pytest-style (PT) default-subset ruff rules
Zero current violations. Only the 6 PT codes ruff 0.16 enables by
default; the full flake8-pytest-style linter has thousands of hits
here and needs a separate, deliberate decision.
2026-09-01 15:03:17 -07:00
stumpylog 9b8bd21044 Chore: enable pylint refactor (PLR) default-subset ruff rules
Zero current violations. Only the 13 PLR codes ruff 0.16 enables by
default; the rest of pylint-refactor (e.g. PLR2004, PLR0913) has
hundreds of hits here and needs a separate, deliberate decision.
2026-09-01 15:03:17 -07:00
stumpylog a60172bc6f Chore: enable pygrep-hooks PGH005 ruff rule
Zero current violations. Only PGH005 (invalid-mock-methods) is part
of ruff 0.16's default rule set; the rest of pygrep-hooks is opt-in.
2026-09-01 15:03:17 -07:00
stumpylog 6c5bc1c0ff Chore: enable pep8-naming N999 ruff rule
Zero current violations. Only N999 (invalid-module-name) is part of
ruff 0.16's default rule set; the rest of pep8-naming is opt-in.
2026-09-01 15:03:17 -07:00
stumpylog f4a7c478a9 Chore: enable flake8-logging (LOG001/002/009/014/015) ruff rules
Zero current violations. Only these five LOG codes are part of
ruff 0.16's default rule set; the rest of the linter is opt-in.
2026-09-01 15:03:17 -07:00
stumpylog bc07c19d9b Chore: enable pydocstyle D419 ruff rule
Zero current violations. Only D419 (empty-docstring) is part of
ruff 0.16's default rule set; the rest of pydocstyle is opt-in.
2026-09-01 15:03:16 -07:00
stumpylog bfcee24572 Chore: enable flake8-async (ASYNC) ruff rules
Zero current violations. Full category (not just the ruff-0.16
default subset) since the rest is equally applicable async-blocking
guidance for this codebase's Channels/websocket code.
2026-09-01 15:03:16 -07:00
stumpylog 4fec4b0948 Chore: enable FA, G010, and PERF101/102/402 ruff rules
All part of ruff 0.16's expanded default rule set. FA and G010 had
zero existing violations; PERF402's one occurrence needed a manual
fix since ruff can't safely autofix a multi-line call expression.
2026-09-01 15:03:16 -07:00
stumpylog 3e4ffc4132 Chore: enable refurb (FURB) ruff rules
FURB is part of ruff 0.16's expanded default rule set and is
almost entirely autofixable.
2026-09-01 15:03:16 -07:00
stumpylog 6d61bcee7e Chore: enable flake8-comprehensions (C4) ruff rules
C4 is part of ruff 0.16's expanded default rule set and is almost
entirely autofixable, making it a low-risk first step towards
adopting the new defaults.
2026-09-01 15:03:16 -07:00
80 changed files with 860 additions and 1490 deletions
+1 -1
View File
@@ -61,7 +61,7 @@ def replace_with_symlinks(
total_duplicates = 0
space_saved = 0
for file_hash, file_list in duplicate_groups.items():
for file_list in duplicate_groups.values():
# Keep the first file as the original, replace others with symlinks
original_file = file_list[0]
duplicates = file_list[1:]
+100 -47
View File
@@ -186,68 +186,121 @@ line-ending = "lf"
# https://docs.astral.sh/ruff/rules/
select = [ "E4", "E7", "E9", "F" ]
extend-select = [
"COM", # https://docs.astral.sh/ruff/rules/#flake8-commas-com
"DJ", # https://docs.astral.sh/ruff/rules/#flake8-django-dj
"EXE", # https://docs.astral.sh/ruff/rules/#flake8-executable-exe
"FBT", # https://docs.astral.sh/ruff/rules/#flake8-boolean-trap-fbt
"FLY", # https://docs.astral.sh/ruff/rules/#flynt-fly
"G201", # https://docs.astral.sh/ruff/rules/#flake8-logging-format-g
"I", # https://docs.astral.sh/ruff/rules/#isort-i
"ICN", # https://docs.astral.sh/ruff/rules/#flake8-import-conventions-icn
"INP", # https://docs.astral.sh/ruff/rules/#flake8-no-pep420-inp
"ISC", # https://docs.astral.sh/ruff/rules/#flake8-implicit-str-concat-isc
"PIE", # https://docs.astral.sh/ruff/rules/#flake8-pie-pie
"PLC", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLE", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PTH", # https://docs.astral.sh/ruff/rules/#flake8-use-pathlib-pth
"Q", # https://docs.astral.sh/ruff/rules/#flake8-quotes-q
"RSE", # https://docs.astral.sh/ruff/rules/#flake8-raise-rse
"RUF", # https://docs.astral.sh/ruff/rules/#ruff-specific-rules-ruf
"SIM", # https://docs.astral.sh/ruff/rules/#flake8-simplify-sim
"T20", # https://docs.astral.sh/ruff/rules/#flake8-print-t20
"TC", # https://docs.astral.sh/ruff/rules/#flake8-type-checking-tc
"TID", # https://docs.astral.sh/ruff/rules/#flake8-tidy-imports-tid
"UP", # https://docs.astral.sh/ruff/rules/#pyupgrade-up
"W", # https://docs.astral.sh/ruff/rules/#pycodestyle-e-w
"ASYNC", # https://docs.astral.sh/ruff/rules/#flake8-async-async
"B002", # https://docs.astral.sh/ruff/rules/#flake8-bugbear-b
"B003",
"B004",
"B005",
"B006",
"B008",
"B009",
"B010",
"B012",
"B013",
"B014",
"B015",
"B016",
"B017",
"B018",
"B019",
"B020",
"B021",
"B022",
"B023",
"B025",
"B026",
"B029",
"B030",
"B031",
"B032",
"B033",
"B035",
"B039",
"C4", # https://docs.astral.sh/ruff/rules/#flake8-comprehensions-c4
"COM", # https://docs.astral.sh/ruff/rules/#flake8-commas-com
"D419", # https://docs.astral.sh/ruff/rules/#pydocstyle-d
"DJ", # https://docs.astral.sh/ruff/rules/#flake8-django-dj
"DTZ", # https://docs.astral.sh/ruff/rules/#flake8-datetimez-dtz
"EXE", # https://docs.astral.sh/ruff/rules/#flake8-executable-exe
"FA", # https://docs.astral.sh/ruff/rules/#flake8-future-annotations-fa
"FBT", # https://docs.astral.sh/ruff/rules/#flake8-boolean-trap-fbt
"FLY", # https://docs.astral.sh/ruff/rules/#flynt-fly
"FURB", # https://docs.astral.sh/ruff/rules/#refurb-furb
"G", # https://docs.astral.sh/ruff/rules/#flake8-logging-format-g
"I", # https://docs.astral.sh/ruff/rules/#isort-i
"ICN", # https://docs.astral.sh/ruff/rules/#flake8-import-conventions-icn
"INP", # https://docs.astral.sh/ruff/rules/#flake8-no-pep420-inp
"INT", # https://docs.astral.sh/ruff/rules/#flake8-gettext-int
"ISC", # https://docs.astral.sh/ruff/rules/#flake8-implicit-str-concat-isc
"LOG", # https://docs.astral.sh/ruff/rules/#flake8-logging-log
"N999", # https://docs.astral.sh/ruff/rules/#pep8-naming-n
"PERF101", # https://docs.astral.sh/ruff/rules/#perflint-perf
"PERF102",
"PERF402",
"PGH005", # https://docs.astral.sh/ruff/rules/#pygrep-hooks-pgh
"PIE", # https://docs.astral.sh/ruff/rules/#flake8-pie-pie
"PLC", # https://docs.astral.sh/ruff/rules/#pylint-pl
"PLE", # https://docs.astral.sh/ruff/rules/#error-ple
"PLR0124", # https://docs.astral.sh/ruff/rules/#refactor-plr
"PLR0133",
"PLR0206",
"PLR0402",
"PLR1704",
"PLR1708",
"PLR1711",
"PLR1716",
"PLR1722",
"PLR1730",
"PLR1733",
"PLR1736",
"PLR2044",
"PLW", # https://docs.astral.sh/ruff/rules/#warning-plw
"PT010", # https://docs.astral.sh/ruff/rules/#flake8-pytest-style-pt
"PT014",
"PT020",
"PT025",
"PT026",
"PT031",
"PTH", # https://docs.astral.sh/ruff/rules/#flake8-use-pathlib-pth
"Q", # https://docs.astral.sh/ruff/rules/#flake8-quotes-q
"RSE", # https://docs.astral.sh/ruff/rules/#flake8-raise-rse
"RUF", # https://docs.astral.sh/ruff/rules/#ruff-specific-rules-ruf
"S102", # https://docs.astral.sh/ruff/rules/#flake8-bandit-s
"S110",
"S112",
"S113",
"SIM", # https://docs.astral.sh/ruff/rules/#flake8-simplify-sim
"T100", # https://docs.astral.sh/ruff/rules/#flake8-debugger-t10
"T20", # https://docs.astral.sh/ruff/rules/#flake8-print-t20
"TC", # https://docs.astral.sh/ruff/rules/#flake8-type-checking-tc
"TID", # https://docs.astral.sh/ruff/rules/#flake8-tidy-imports-tid
"TRY002", # https://docs.astral.sh/ruff/rules/#tryceratops-try
"TRY004",
"TRY201",
"TRY203",
"TRY401",
"UP", # https://docs.astral.sh/ruff/rules/#pyupgrade-up
"W", # https://docs.astral.sh/ruff/rules/#pycodestyle-e-w
"YTT", # https://docs.astral.sh/ruff/rules/#flake8-2020-ytt
]
ignore = [
"DJ001",
"PLC0415",
"RUF012",
"SIM105",
"G004", # Logging statement uses f-string - good to do, but a large diff
]
# Migrations
per-file-ignores."*/migrations/*.py" = [
"E501",
"SIM",
"T201",
]
per-file-ignores."*/migrations/*.py" = []
# Testing
per-file-ignores."*/tests/*.py" = [
"E501",
"DTZ",
"SIM117",
]
per-file-ignores.".github/scripts/*.py" = [
"E501",
"INP001",
"SIM117",
]
# Docker specific
per-file-ignores."docker/rootfs/usr/local/bin/wait-for-redis.py" = [
"INP001",
"T201",
]
per-file-ignores."docker/wait-for-redis.py" = [
"INP001",
"T201",
]
per-file-ignores."src/documents/models.py" = [
"SIM115",
]
isort.force-single-line = true
[tool.codespell]
ignore-words-list = "criterias,afterall,valeu,ureue,equest,ure,assertIn,Oktober,commitish,NIN,nin"
ignore-words-list = "criterias,afterall,valeu,ureue,equest,ure,assertIn,Oktober,commitish"
skip = """\
src-ui/src/locale/*,src-ui/pnpm-lock.yaml,src-ui/e2e/*,src/paperless_mail/tests/samples/*,src/paperless/tests/samples\
/mail/*,src/documents/tests/samples/*,*.po,*.json\
+140 -140
View File
File diff suppressed because it is too large Load Diff
@@ -41,8 +41,6 @@ export class TrashComponent
private modalService = inject(NgbModal)
private settingsService = inject(SettingsService)
private router = inject(Router)
private readonly emptyTrashDelaySetting =
this.settingsService.getSignal<number>(SETTINGS_KEYS.EMPTY_TRASH_DELAY)
readonly documentsInTrash = signal<Document[]>([])
readonly selectedDocuments = signal<Set<number>>(new Set())
@@ -202,7 +200,8 @@ export class TrashComponent
}
getDaysRemaining(document: Document): number {
const delay = this.emptyTrashDelaySetting()
this.settingsService.trackChanges()
const delay = this.settingsService.get(SETTINGS_KEYS.EMPTY_TRASH_DELAY)
const diff = new Date().getTime() - new Date(document.deleted_at).getTime()
const days = Math.ceil(diff / (1000 * 3600 * 24))
return delay - days
@@ -193,23 +193,6 @@ describe('AppFrameComponent', () => {
expect(savedViewSpy).toHaveBeenCalled()
})
it('should update reinitialized signal-backed settings without manual change detection', async () => {
settingsService.initializeSettings().subscribe()
httpTestingController
.expectOne(`${environment.apiBaseUrl}ui_settings/`)
.flush({
settings: { app_title: 'Reactive title' },
user: {},
permissions: [],
})
await fixture.whenStable()
expect(
fixture.nativeElement.querySelector('.brand-title').textContent
).toBe('Reactive title')
})
it('should check for update if enabled', () => {
const updateCheckSpy = jest.spyOn(remoteVersionService, 'checkForUpdates')
updateCheckSpy.mockImplementation(() => {
@@ -98,29 +98,6 @@ export class AppFrameComponent
readonly isMenuCollapsed = signal(true)
readonly slimSidebarAnimating = signal(false)
readonly mobileSearchHidden = signal(false)
private readonly versionSetting = this.settingsService.getSignal<string>(
SETTINGS_KEYS.VERSION
)
private readonly appTitleSetting = this.settingsService.getSignal<string>(
SETTINGS_KEYS.APP_TITLE
)
private readonly appLogoSetting = this.settingsService.getSignal<string>(
SETTINGS_KEYS.APP_LOGO
)
private readonly slimSidebarSetting = this.settingsService.getSignal<boolean>(
SETTINGS_KEYS.SLIM_SIDEBAR
)
private readonly attributesSectionsCollapsedSetting =
this.settingsService.getSignal<CollapsibleSection[]>(
SETTINGS_KEYS.ATTRIBUTES_SECTIONS_COLLAPSED
)
private readonly aiEnabledSetting = this.settingsService.getSignal<boolean>(
SETTINGS_KEYS.AI_ENABLED
)
private readonly sidebarViewsShowCountSetting =
this.settingsService.getSignal<boolean>(
SETTINGS_KEYS.SIDEBAR_VIEWS_SHOW_COUNT
)
private lastScrollY: number = 0
constructor() {
@@ -214,23 +191,33 @@ export class AppFrameComponent
}
get versionString(): string {
return `${environment.appTitle} v${this.versionSetting()}${environment.tag === 'prod' ? '' : ` #${environment.tag}`}`
this.settingsService.trackChanges()
return `${environment.appTitle} v${this.settingsService.get(SETTINGS_KEYS.VERSION)}${environment.tag === 'prod' ? '' : ` #${environment.tag}`}`
}
get appTitle(): string {
return this.appTitleSetting() || environment.appTitle
this.settingsService.trackChanges()
return (
this.settingsService.get(SETTINGS_KEYS.APP_TITLE) || environment.appTitle
)
}
get customAppTitle(): string {
return this.appTitleSetting()
this.settingsService.trackChanges()
return this.settingsService.get(SETTINGS_KEYS.APP_TITLE)
}
get hasCustomBranding(): boolean {
return !!(this.appTitleSetting()?.length || this.appLogoSetting()?.length)
this.settingsService.trackChanges()
return !!(
this.settingsService.get(SETTINGS_KEYS.APP_TITLE)?.length ||
this.settingsService.get(SETTINGS_KEYS.APP_LOGO)?.length
)
}
get customAppLogo(): string {
const logo = this.appLogoSetting()
this.settingsService.trackChanges()
const logo = this.settingsService.get(SETTINGS_KEYS.APP_LOGO)
return logo?.length
? environment.apiBaseUrl.replace(/\/api\/$/, logo)
: null
@@ -275,7 +262,8 @@ export class AppFrameComponent
}
get slimSidebarEnabled(): boolean {
return this.slimSidebarSetting()
this.settingsService.trackChanges()
return this.settingsService.get(SETTINGS_KEYS.SLIM_SIDEBAR)
}
set slimSidebarEnabled(enabled: boolean) {
@@ -298,9 +286,10 @@ export class AppFrameComponent
}
get attributesSectionsCollapsed(): boolean {
return this.attributesSectionsCollapsedSetting()?.includes(
CollapsibleSection.ATTRIBUTES
)
this.settingsService.trackChanges()
return this.settingsService
.get(SETTINGS_KEYS.ATTRIBUTES_SECTIONS_COLLAPSED)
?.includes(CollapsibleSection.ATTRIBUTES)
}
set attributesSectionsCollapsed(collapsed: boolean) {
@@ -323,7 +312,8 @@ export class AppFrameComponent
}
get aiEnabled(): boolean {
return this.aiEnabledSetting()
this.settingsService.trackChanges()
return this.settingsService.get(SETTINGS_KEYS.AI_ENABLED)
}
@HostListener('window:resize')
@@ -490,8 +480,9 @@ export class AppFrameComponent
}
get showSidebarCounts(): boolean {
this.settingsService.trackChanges()
return (
this.sidebarViewsShowCountSetting() &&
this.settingsService.get(SETTINGS_KEYS.SIDEBAR_VIEWS_SHOW_COUNT) &&
!this.settingsService.organizingSidebarSavedViews()
)
}
@@ -81,10 +81,6 @@ export class GlobalSearchComponent implements OnInit {
private hotkeyService = inject(HotKeyService)
private settingsService = inject(SettingsService)
private locationStrategy = inject(LocationStrategy)
private readonly searchFullTypeSetting =
this.settingsService.getSignal<GlobalSearchType>(
SETTINGS_KEYS.SEARCH_FULL_TYPE
)
public DataType = DataType
readonly query = signal<string>(null)
@@ -101,7 +97,11 @@ export class GlobalSearchComponent implements OnInit {
@ViewChildren('secondaryButton') secondaryButtons: QueryList<ElementRef>
get useAdvancedForFullSearch(): boolean {
return this.searchFullTypeSetting() === GlobalSearchType.ADVANCED
this.settingsService.trackChanges()
return (
this.settingsService.get(SETTINGS_KEYS.SEARCH_FULL_TYPE) ===
GlobalSearchType.ADVANCED
)
}
constructor() {
@@ -196,16 +196,6 @@ describe('WorkflowEditDialogComponent', () => {
fixture.detectChanges()
})
function setActionSettings({
email = true,
remoteOcr = true,
ai = true,
} = {}) {
settingsService.set(SETTINGS_KEYS.EMAIL_ENABLED, email)
settingsService.set(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED, remoteOcr)
settingsService.set(SETTINGS_KEYS.AI_ENABLED, ai)
}
it('should support create and edit modes, support adding triggers and actions on new workflow', () => {
component.dialogMode.set(EditDialogMode.CREATE)
const createTitleSpy = jest.spyOn(component, 'getCreateTitle')
@@ -228,7 +218,7 @@ describe('WorkflowEditDialogComponent', () => {
})
it('should return source options, type options, type name, schedule date field options', () => {
setActionSettings()
jest.spyOn(settingsService, 'get').mockReturnValue(true)
component.ngOnInit()
expect(component.sourceOptions).toEqual(DOCUMENT_SOURCE_OPTIONS)
expect(component.triggerTypeOptions).toEqual(WORKFLOW_TYPE_OPTIONS)
@@ -252,7 +242,7 @@ describe('WorkflowEditDialogComponent', () => {
)
// Email, remote OCR and AI all disabled
setActionSettings({ email: false, remoteOcr: false, ai: false })
jest.spyOn(settingsService, 'get').mockReturnValue(false)
component.ngOnInit()
expect(component.actionTypeOptions).toEqual(
WORKFLOW_ACTION_OPTIONS.filter(
@@ -265,7 +255,7 @@ describe('WorkflowEditDialogComponent', () => {
})
it('should offer remote OCR only for consumption workflows', () => {
setActionSettings()
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// A consumption trigger makes the action reachable
component.object = {
@@ -295,7 +285,7 @@ describe('WorkflowEditDialogComponent', () => {
})
it('should offer remote OCR on a trigger added to a new workflow', () => {
setActionSettings()
jest.spyOn(settingsService, 'get').mockReturnValue(true)
component.ngOnInit()
// Nothing for the action to apply to yet
@@ -321,7 +311,7 @@ describe('WorkflowEditDialogComponent', () => {
})
it('should keep remote OCR listed when an action already uses it', () => {
setActionSettings()
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// Otherwise changing the trigger would silently blank the selection
component.object = {
@@ -339,7 +329,9 @@ describe('WorkflowEditDialogComponent', () => {
})
it('should not offer remote OCR when no engine is configured', () => {
setActionSettings({ remoteOcr: false })
jest
.spyOn(settingsService, 'get')
.mockImplementation((key) => key !== SETTINGS_KEYS.REMOTE_OCR_CONFIGURED)
component.object = {
name: 'Workflow 1',
@@ -356,7 +348,7 @@ describe('WorkflowEditDialogComponent', () => {
})
it('should offer apply AI suggestions unless every trigger is consumption', () => {
setActionSettings()
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// Consumption runs before the document has been parsed, so there would be
// no content to make suggestions from
@@ -390,7 +382,7 @@ describe('WorkflowEditDialogComponent', () => {
})
it('should keep apply AI suggestions listed when an action already uses it', () => {
setActionSettings()
jest.spyOn(settingsService, 'get').mockReturnValue(true)
// Otherwise changing the trigger would silently blank the selection
component.object = {
@@ -408,7 +400,9 @@ describe('WorkflowEditDialogComponent', () => {
})
it('should not offer apply AI suggestions when AI is disabled', () => {
setActionSettings({ ai: false })
jest
.spyOn(settingsService, 'get')
.mockImplementation((key) => key !== SETTINGS_KEYS.AI_ENABLED)
component.object = {
name: 'Workflow 1',
@@ -537,13 +537,6 @@ export class WorkflowEditDialogComponent
readonly dateCustomFields = computed(() =>
this.customFields()?.filter((f) => f.data_type === CustomFieldDataType.Date)
)
private readonly emailEnabledSetting =
this.settingsService.getSignal<boolean>(SETTINGS_KEYS.EMAIL_ENABLED)
private readonly remoteOcrConfiguredSetting =
this.settingsService.getSignal<boolean>(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED)
private readonly aiEnabledSetting = this.settingsService.getSignal<boolean>(
SETTINGS_KEYS.AI_ENABLED
)
expandedItem: number = null
@@ -596,7 +589,7 @@ export class WorkflowEditDialogComponent
private getAllowedActionTypes() {
let allowed = WORKFLOW_ACTION_OPTIONS
if (!this.emailEnabledSetting()) {
if (!this.settingsService.get(SETTINGS_KEYS.EMAIL_ENABLED)) {
allowed = allowed.filter((a) => a.id !== WorkflowActionType.Email)
}
@@ -604,7 +597,7 @@ export class WorkflowEditDialogComponent
// offered for workflows that run at consumption.
const formWorkflow: Workflow = this.objectForm?.value
const remoteOcrUsable =
this.remoteOcrConfiguredSetting() &&
this.settingsService.get(SETTINGS_KEYS.REMOTE_OCR_CONFIGURED) &&
(formWorkflow?.triggers?.some(
(trigger) => trigger.type === WorkflowTriggerType.Consumption
) ||
@@ -619,7 +612,7 @@ export class WorkflowEditDialogComponent
// once every trigger is consumption, so it stays offered on a workflow
// that has no triggers yet.
const aiSuggestionsUsable =
this.aiEnabledSetting() &&
this.settingsService.get(SETTINGS_KEYS.AI_ENABLED) &&
(!formWorkflow?.triggers?.length ||
formWorkflow.triggers.some(
(trigger) => trigger.type !== WorkflowTriggerType.Consumption
@@ -1369,6 +1362,7 @@ export class WorkflowEditDialogComponent
}
get actionTypeOptions() {
this.settingsService.trackChanges()
// Computed on read rather than cached
return this.getAllowedActionTypes()
}
@@ -839,9 +839,7 @@ describe('FilterableDropdownComponent & FilterableDropdownSelectionModel', () =>
selectionModel.items = [memoRoot]
selectionModel.documentCounts = [{ id: memoRoot.id, document_count: 9 }]
const getRootDocCount = (selectionModel as any).createRootDocCounter(
selectionModel.items
)
const getRootDocCount = (selectionModel as any).createRootDocCounter()
expect(getRootDocCount(memoRoot.id)).toEqual(9)
selectionModel.documentCounts = []
@@ -857,9 +855,7 @@ describe('FilterableDropdownComponent & FilterableDropdownSelectionModel', () =>
selectionModel.items = [rootWithoutSelection]
selectionModel.documentCounts = []
const getRootDocCount = (selectionModel as any).createRootDocCounter(
selectionModel.items
)
const getRootDocCount = (selectionModel as any).createRootDocCounter()
expect(getRootDocCount(rootWithoutSelection.id)).toEqual(4)
})
@@ -869,9 +865,7 @@ describe('FilterableDropdownComponent & FilterableDropdownSelectionModel', () =>
selectionModel.items = [rootWithoutCounts]
selectionModel.documentCounts = []
const getRootDocCount = (selectionModel as any).createRootDocCounter(
selectionModel.items
)
const getRootDocCount = (selectionModel as any).createRootDocCounter()
expect(getRootDocCount(rootWithoutCounts.id)).toEqual(0)
})
@@ -972,7 +966,7 @@ describe('FilterableDropdownComponent & FilterableDropdownSelectionModel', () =>
component.selectionModel['temporarySelectionStates'].set(id, state)
const changedSpy = jest.spyOn(component.selectionModel.changed, 'next')
component.selectionModel.exclude(id)
expect(component.selectionModel.temporaryLogicalOperator()).toBe(
expect(component.selectionModel.temporaryLogicalOperator).toBe(
LogicalOperator.And
)
expect(component.selectionModel['temporarySelectionStates'].get(id)).toBe(
@@ -64,56 +64,43 @@ export class FilterableDropdownSelectionModel {
manyToOne = false
singleSelect = false
private _logicalOperator: LogicalOperator = LogicalOperator.And
temporaryLogicalOperator: LogicalOperator = this._logicalOperator
private _intersection: Intersection = Intersection.Include
temporaryIntersection: Intersection = this._intersection
private readonly _logicalOperator = signal(LogicalOperator.And)
readonly temporaryLogicalOperator = signal(LogicalOperator.And)
private readonly _intersection = signal(Intersection.Include)
readonly temporaryIntersection = signal(Intersection.Include)
private readonly _documentCounts = signal<SelectionDataItem[]>([])
private readonly _items = signal<MatchingModel[]>([])
private readonly _selectionStates = signal(
new Map<number, ToggleableItemState>()
)
private readonly _temporarySelectionStates = signal(
new Map<number, ToggleableItemState>()
)
private _documentCounts: SelectionDataItem[] = []
public documentCountSortingEnabled = false
private get selectionStates(): ReadonlyMap<number, ToggleableItemState> {
return this._selectionStates()
}
private get temporarySelectionStates(): ReadonlyMap<
number,
ToggleableItemState
> {
return this._temporarySelectionStates()
}
public set documentCounts(counts: SelectionDataItem[]) {
this._documentCounts.set(counts)
this._documentCounts = counts
if (this.documentCountSortingEnabled) {
this._items.set(this.sortItems(this.items))
this.sortItems()
}
}
private _items: MatchingModel[] = []
get items(): MatchingModel[] {
return this._items()
return this._items
}
set items(items: MatchingModel[]) {
if (items) {
this._items.set(this.withNullItem(this.sortItems(Array.from(items))))
this._items = Array.from(items)
this.sortItems()
this.setNullItem()
}
}
private withNullItem(items: MatchingModel[]): MatchingModel[] {
private setNullItem() {
if (this.manyToOne && this.logicalOperator === LogicalOperator.Or) {
return items[0]?.id === null ? items.slice(1) : items
if (this._items[0]?.id === null) {
this._items.shift()
}
return
}
const nullItem = {
const item = {
name: $localize`:Filter drop down element to filter for documents with no correspondent/type/tag assigned:Not assigned`,
id:
this.manyToOne || this.intersection === Intersection.Include
@@ -121,17 +108,22 @@ export class FilterableDropdownSelectionModel {
: NEGATIVE_NULL_FILTER_VALUE,
}
return items[0]?.id === null || items[0]?.id === NEGATIVE_NULL_FILTER_VALUE
? [nullItem, ...items.slice(1)]
: [nullItem, ...items]
if (
this._items[0]?.id === null ||
this._items[0]?.id === NEGATIVE_NULL_FILTER_VALUE
) {
this._items[0] = item
} else if (this._items) {
this._items.unshift(item)
}
}
constructor(manyToOne: boolean = false) {
this.manyToOne = manyToOne
}
private sortItems(items: MatchingModel[]): MatchingModel[] {
const sorted = [...items].sort((a, b) => {
private sortItems() {
this._items.sort((a, b) => {
if (
(a.id == null && b.id != null) ||
(a.id == NEGATIVE_NULL_FILTER_VALUE &&
@@ -162,13 +154,13 @@ export class FilterableDropdownSelectionModel {
) {
return -1
} else if (
this._documentCounts().length &&
this._documentCounts.length &&
this.getDocumentCount(b.id) === 0 &&
this.getDocumentCount(a.id) > this.getDocumentCount(b.id)
) {
return -1
} else if (
this._documentCounts().length &&
this._documentCounts.length &&
this.getDocumentCount(a.id) === 0 &&
this.getDocumentCount(a.id) < this.getDocumentCount(b.id)
) {
@@ -178,11 +170,15 @@ export class FilterableDropdownSelectionModel {
}
})
return this._documentCounts().length
? this.promoteBranchesWithDocumentCounts(sorted)
: sorted
if (this._documentCounts.length) {
this.promoteBranchesWithDocumentCounts()
}
}
private selectionStates = new Map<number, ToggleableItemState>()
private temporarySelectionStates = new Map<number, ToggleableItemState>()
getSelectedItems() {
return this.items.filter(
(i) =>
@@ -198,33 +194,30 @@ export class FilterableDropdownSelectionModel {
}
set(id: number, state: ToggleableItemState, fireEvent = true) {
const states = new Map(this.temporarySelectionStates)
if (state == ToggleableItemState.NotSelected) {
states.delete(id)
this.temporarySelectionStates.delete(id)
} else {
states.set(id, state)
this.temporarySelectionStates.set(id, state)
}
this._temporarySelectionStates.set(states)
if (fireEvent) {
this.changed.next(this)
}
}
toggle(id: number, fireEvent = true) {
const states = new Map(this.temporarySelectionStates)
let state = states.get(id)
let state = this.temporarySelectionStates.get(id)
if (
state == undefined ||
(state != ToggleableItemState.Selected &&
state != ToggleableItemState.Excluded)
) {
if (this.manyToOne || this.singleSelect) {
states.set(id, ToggleableItemState.Selected)
this.temporarySelectionStates.set(id, ToggleableItemState.Selected)
if (this.singleSelect) {
for (let key of states.keys()) {
for (let key of this.temporarySelectionStates.keys()) {
if (key != id) {
states.delete(key)
this.temporarySelectionStates.delete(key)
}
}
}
@@ -240,26 +233,25 @@ export class FilterableDropdownSelectionModel {
) {
newState = ToggleableItemState.NotSelected
}
states.set(id, newState)
this.temporarySelectionStates.set(id, newState)
}
} else if (
state == ToggleableItemState.Selected ||
state == ToggleableItemState.Excluded
) {
states.delete(id)
this.clearDescendantSelections(states, id)
this.temporarySelectionStates.delete(id)
this.clearDescendantSelections(id)
}
if (!id) {
for (let key of states.keys()) {
for (let key of this.temporarySelectionStates.keys()) {
if (key) {
states.delete(key)
this.temporarySelectionStates.delete(key)
}
}
} else {
states.delete(null)
this.temporarySelectionStates.delete(null)
}
this._temporarySelectionStates.set(states)
if (fireEvent) {
this.changed.next(this)
@@ -267,21 +259,20 @@ export class FilterableDropdownSelectionModel {
}
exclude(id: number, fireEvent: boolean = true) {
const states = new Map(this.temporarySelectionStates)
let state = states.get(id)
let state = this.temporarySelectionStates.get(id)
if (id && (state == null || state != ToggleableItemState.Excluded)) {
const operator = this.manyToOne ? LogicalOperator.And : LogicalOperator.Or
this.temporaryLogicalOperator.set(operator)
this._logicalOperator.set(operator)
this.temporaryLogicalOperator = this._logicalOperator = this.manyToOne
? LogicalOperator.And
: LogicalOperator.Or
if (this.manyToOne || this.singleSelect) {
states.set(id, ToggleableItemState.Excluded)
this.clearDescendantSelections(states, id)
this.temporarySelectionStates.set(id, ToggleableItemState.Excluded)
this.clearDescendantSelections(id)
if (this.singleSelect) {
for (let key of states.keys()) {
for (let key of this.temporarySelectionStates.keys()) {
if (key != id) {
states.delete(key)
this.temporarySelectionStates.delete(key)
}
}
}
@@ -296,18 +287,17 @@ export class FilterableDropdownSelectionModel {
) {
newState = ToggleableItemState.NotSelected
}
states.set(id, newState)
this.temporarySelectionStates.set(id, newState)
if (newState == ToggleableItemState.Excluded) {
this.clearDescendantSelections(states, id)
this.clearDescendantSelections(id)
}
}
} else if (!id || state == ToggleableItemState.Excluded) {
states.delete(id)
this.temporarySelectionStates.delete(id)
if (id) {
this.clearDescendantSelections(states, id)
this.clearDescendantSelections(id)
}
}
this._temporarySelectionStates.set(states)
if (fireEvent) {
this.changed.next(this)
@@ -318,12 +308,9 @@ export class FilterableDropdownSelectionModel {
return this.selectionStates.get(id) || ToggleableItemState.NotSelected
}
private clearDescendantSelections(
states: Map<number, ToggleableItemState>,
id: number
) {
private clearDescendantSelections(id: number) {
for (const descendantID of this.getDescendantIDs(id)) {
states.delete(descendantID)
this.temporarySelectionStates.delete(descendantID)
}
}
@@ -333,7 +320,7 @@ export class FilterableDropdownSelectionModel {
while (queue.length) {
const parentID = queue.shift()
for (const item of this.items) {
for (const item of this._items) {
if (
typeof item?.id === 'number' &&
typeof (item as any)['parent'] === 'number' &&
@@ -349,12 +336,12 @@ export class FilterableDropdownSelectionModel {
}
get logicalOperator(): LogicalOperator {
return this.temporaryLogicalOperator()
return this.temporaryLogicalOperator
}
set logicalOperator(operator: LogicalOperator) {
this.temporaryLogicalOperator.set(operator)
this._items.set(this.withNullItem(this.items))
this.temporaryLogicalOperator = operator
this.setNullItem()
}
toggleOperator() {
@@ -362,12 +349,12 @@ export class FilterableDropdownSelectionModel {
}
get intersection(): Intersection {
return this.temporaryIntersection()
return this.temporaryIntersection
}
set intersection(intersection: Intersection) {
this.temporaryIntersection.set(intersection)
this._items.set(this.withNullItem(this.items))
this.temporaryIntersection = intersection
this.setNullItem()
}
toggleIntersection() {
@@ -377,20 +364,18 @@ export class FilterableDropdownSelectionModel {
? ToggleableItemState.Selected
: ToggleableItemState.Excluded
const states = new Map(this.temporarySelectionStates)
states.forEach((state, key) => {
this.temporarySelectionStates.forEach((state, key) => {
if (key === null && this.intersection === Intersection.Exclude) {
states.set(NEGATIVE_NULL_FILTER_VALUE, newState)
this.temporarySelectionStates.set(NEGATIVE_NULL_FILTER_VALUE, newState)
} else if (
key === NEGATIVE_NULL_FILTER_VALUE &&
this.intersection === Intersection.Include
) {
states.set(null, newState)
this.temporarySelectionStates.set(null, newState)
} else {
states.set(key, newState)
this.temporarySelectionStates.set(key, newState)
}
})
this._temporarySelectionStates.set(states)
this.changed.next(this)
}
@@ -410,12 +395,10 @@ export class FilterableDropdownSelectionModel {
}
clear(fireEvent = true) {
this._temporarySelectionStates.set(new Map())
this.temporaryLogicalOperator.set(LogicalOperator.And)
this._logicalOperator.set(LogicalOperator.And)
this.temporaryIntersection.set(Intersection.Include)
this._intersection.set(Intersection.Include)
this._items.set(this.withNullItem(this.items))
this.temporarySelectionStates.clear()
this.temporaryLogicalOperator = this._logicalOperator = LogicalOperator.And
this.temporaryIntersection = this._intersection = Intersection.Include
this.setNullItem()
if (fireEvent) {
this.changed.next(this)
}
@@ -436,9 +419,9 @@ export class FilterableDropdownSelectionModel {
)
) {
return true
} else if (this.temporaryLogicalOperator() !== this._logicalOperator()) {
} else if (this.temporaryLogicalOperator !== this._logicalOperator) {
return true
} else if (this.temporaryIntersection() !== this._intersection()) {
} else if (this.temporaryIntersection !== this._intersection) {
return true
} else {
return false
@@ -455,29 +438,23 @@ export class FilterableDropdownSelectionModel {
}
getDocumentCount(id: number) {
return this._documentCounts().find((c) => c.id === id)?.document_count
return this._documentCounts.find((c) => c.id === id)?.document_count
}
private promoteBranchesWithDocumentCounts(
items: MatchingModel[]
): MatchingModel[] {
const parentById = this.buildParentById(items)
private promoteBranchesWithDocumentCounts() {
const parentById = this.buildParentById()
const findRootId = this.createRootFinder(parentById)
const getRootDocCount = this.createRootDocCounter(items)
const summaries = this.buildBranchSummaries(
items,
findRootId,
getRootDocCount
)
const getRootDocCount = this.createRootDocCounter()
const summaries = this.buildBranchSummaries(findRootId, getRootDocCount)
const orderedBranches = this.orderBranchesByPriority(summaries)
return orderedBranches.flatMap((summary) => summary.items)
this._items = orderedBranches.flatMap((summary) => summary.items)
}
private buildParentById(items: MatchingModel[]): Map<number, number | null> {
private buildParentById(): Map<number, number | null> {
const parentById = new Map<number, number | null>()
for (const item of items) {
for (const item of this._items) {
if (typeof item?.id === 'number') {
const parentValue = (item as any)['parent']
parentById.set(
@@ -515,9 +492,7 @@ export class FilterableDropdownSelectionModel {
return findRootId
}
private createRootDocCounter(
items: MatchingModel[]
): (rootId: number) => number {
private createRootDocCounter(): (rootId: number) => number {
const docCountMemo = new Map<number, number>()
return (rootId: number): number => {
@@ -532,7 +507,7 @@ export class FilterableDropdownSelectionModel {
return explicit
}
const rootItem = items.find((i) => i.id === rootId)
const rootItem = this._items.find((i) => i.id === rootId)
const fallback =
typeof (rootItem as any)?.['document_count'] === 'number'
? (rootItem as any)['document_count']
@@ -544,13 +519,12 @@ export class FilterableDropdownSelectionModel {
}
private buildBranchSummaries(
items: MatchingModel[],
findRootId: (id: number) => number,
getRootDocCount: (rootId: number) => number
): Map<string, BranchSummary> {
const summaries = new Map<string, BranchSummary>()
for (const [index, item] of items.entries()) {
for (const [index, item] of this._items.entries()) {
const { key, special, rootId } = this.describeBranchItem(
item,
index,
@@ -642,23 +616,28 @@ export class FilterableDropdownSelectionModel {
}
init(map: Map<number, ToggleableItemState>) {
this._temporarySelectionStates.set(new Map(map))
this.temporarySelectionStates = map
this.apply()
}
apply() {
this._selectionStates.set(new Map(this.temporarySelectionStates))
this._logicalOperator.set(this.temporaryLogicalOperator())
this._intersection.set(this.temporaryIntersection())
this._items.set(this.sortItems(this.items))
this.selectionStates.clear()
this.temporarySelectionStates.forEach((value, key) => {
this.selectionStates.set(key, value)
})
this._logicalOperator = this.temporaryLogicalOperator
this._intersection = this.temporaryIntersection
this.sortItems()
}
reset(complete: boolean = false) {
this.temporarySelectionStates.clear()
if (complete) {
this._selectionStates.set(new Map())
this._temporarySelectionStates.set(new Map())
this.selectionStates.clear()
} else {
this._temporarySelectionStates.set(new Map(this.selectionStates))
this.selectionStates.forEach((value, key) => {
this.temporarySelectionStates.set(key, value)
})
}
}
@@ -7,7 +7,7 @@
<div class="list-group list-group-flush">
<button class="list-group-item list-group-item-action d-flex align-items-center p-2 border-top-0 border-start-0 border-end-0 border-bottom" role="menuitem" (click)="setFilter(OwnerFilterType.NONE)" [disabled]="disabled">
<div class="selected-icon me-1">
@if (selectionModel.ownerFilter() === OwnerFilterType.NONE) {
@if (selectionModel.ownerFilter === OwnerFilterType.NONE) {
<i-bs width="1em" height="1em" name="check"></i-bs>
}
</div>
@@ -17,7 +17,7 @@
</button>
<button class="list-group-item list-group-item-action d-flex align-items-center p-2 border-top-0 border-start-0 border-end-0 border-bottom" role="menuitem" (click)="setFilter(OwnerFilterType.SELF)" [disabled]="disabled">
<div class="selected-icon me-1">
@if (selectionModel.ownerFilter() === OwnerFilterType.SELF) {
@if (selectionModel.ownerFilter === OwnerFilterType.SELF) {
<i-bs width="1em" height="1em" name="check"></i-bs>
}
</div>
@@ -27,7 +27,7 @@
</button>
<button class="list-group-item list-group-item-action d-flex align-items-center p-2 border-top-0 border-start-0 border-end-0 border-bottom" role="menuitem" (click)="setFilter(OwnerFilterType.NOT_SELF)" [disabled]="disabled">
<div class="selected-icon me-1">
@if (selectionModel.ownerFilter() === OwnerFilterType.NOT_SELF) {
@if (selectionModel.ownerFilter === OwnerFilterType.NOT_SELF) {
<i-bs width="1em" height="1em" name="check"></i-bs>
}
</div>
@@ -37,7 +37,7 @@
</button>
<button class="list-group-item list-group-item-action d-flex align-items-center p-2 border-top-0 border-start-0 border-end-0 border-bottom" role="menuitem" (click)="setFilter(OwnerFilterType.SHARED_BY_ME)" [disabled]="disabled">
<div class="selected-icon me-1">
@if (selectionModel.ownerFilter() === OwnerFilterType.SHARED_BY_ME) {
@if (selectionModel.ownerFilter === OwnerFilterType.SHARED_BY_ME) {
<i-bs width="1em" height="1em" name="check"></i-bs>
}
</div>
@@ -47,7 +47,7 @@
</button>
<button class="list-group-item list-group-item-action d-flex align-items-center p-2 border-top-0 border-start-0 border-end-0 border-bottom" role="menuitem" (click)="setFilter(OwnerFilterType.UNOWNED)" [disabled]="disabled">
<div class="selected-icon me-1">
@if (selectionModel.ownerFilter() === OwnerFilterType.UNOWNED) {
@if (selectionModel.ownerFilter === OwnerFilterType.UNOWNED) {
<i-bs width="1em" height="1em" name="check"></i-bs>
}
</div>
@@ -57,7 +57,7 @@
</button>
<button *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.User }" class="list-group-item list-group-item-action d-flex align-items-center p-2 border-top-0 border-start-0 border-end-0 border-bottom" role="menuitem" [disabled]="disabled">
<div class="selected-icon me-1">
@if (selectionModel.ownerFilter() === OwnerFilterType.OTHERS) {
@if (selectionModel.ownerFilter === OwnerFilterType.OTHERS) {
<i-bs width="1em" height="1em" name="check"></i-bs>
}
</div>
@@ -65,8 +65,7 @@
<ng-select
name="user"
class="user-select small"
[ngModel]="selectionModel.includeUsers()"
(ngModelChange)="selectionModel.includeUsers.set($event)"
[(ngModel)]="selectionModel.includeUsers"
[disabled]="disabled"
[clearable]="false"
[items]="users()"
@@ -79,10 +78,10 @@
</ng-select>
</div>
</button>
@if (selectionModel.ownerFilter() === OwnerFilterType.NONE || selectionModel.ownerFilter() === OwnerFilterType.NOT_SELF) {
@if (selectionModel.ownerFilter === OwnerFilterType.NONE || selectionModel.ownerFilter === OwnerFilterType.NOT_SELF) {
<div class="list-group-item list-group-item-action d-flex align-items-center p-2 ps-3 border-bottom-0 border-start-0 border-end-0">
<div class="form-check form-switch w-100">
<input type="checkbox" class="form-check-input" id="hideUnowned" [ngModel]="selectionModel.hideUnowned()" (ngModelChange)="selectionModel.hideUnowned.set($event)" (change)="onChange()" [disabled]="disabled">
<input type="checkbox" class="form-check-input" id="hideUnowned" [(ngModel)]="this.selectionModel.hideUnowned" (change)="onChange()" [disabled]="disabled">
<label class="form-check-label w-100" for="hideUnowned"><small i18n>Hide unowned</small></label>
</div>
</div>
@@ -90,56 +90,56 @@ describe('PermissionsFilterDropdownComponent', () => {
component.setFilter(OwnerFilterType.OTHERS)
expect(component.isActive).toBeTruthy()
component.setFilter(OwnerFilterType.NONE)
component.selectionModel.hideUnowned.set(true)
component.selectionModel.hideUnowned = true
expect(component.isActive).toBeTruthy()
})
it('should describe concrete user filters honestly', () => {
component.selectionModel.ownerFilter.set(OwnerFilterType.SELF)
component.selectionModel.userID.set(1)
component.selectionModel.ownerFilter = OwnerFilterType.SELF
component.selectionModel.userID = 1
expect(component.ownerFilterLabel).toEqual('Owned by user1')
component.selectionModel.ownerFilter.set(OwnerFilterType.NOT_SELF)
component.selectionModel.excludeUsers.set([1])
component.selectionModel.ownerFilter = OwnerFilterType.NOT_SELF
component.selectionModel.excludeUsers = [1]
expect(component.ownerExclusionFilterLabel).toEqual('Not owned by user1')
component.selectionModel.ownerFilter.set(OwnerFilterType.SHARED_BY_ME)
component.selectionModel.userID.set(1)
component.selectionModel.ownerFilter = OwnerFilterType.SHARED_BY_ME
component.selectionModel.userID = 1
expect(component.sharedByFilterLabel).toEqual('Shared by user1')
})
it('should describe concrete filters when usernames are unavailable', () => {
component.selectionModel.ownerFilter.set(OwnerFilterType.SELF)
component.selectionModel.userID.set(99)
component.selectionModel.ownerFilter = OwnerFilterType.SELF
component.selectionModel.userID = 99
expect(component.ownerFilterLabel).toEqual('Owned by another user')
component.selectionModel.ownerFilter.set(OwnerFilterType.NOT_SELF)
component.selectionModel.excludeUsers.set([99])
component.selectionModel.ownerFilter = OwnerFilterType.NOT_SELF
component.selectionModel.excludeUsers = [99]
expect(component.ownerExclusionFilterLabel).toEqual(
'Not owned by another user'
)
component.selectionModel.excludeUsers.set([98, 99])
component.selectionModel.excludeUsers = [98, 99]
expect(component.ownerExclusionFilterLabel).toEqual(
'Not owned by selected users'
)
component.selectionModel.ownerFilter.set(OwnerFilterType.SHARED_BY_ME)
component.selectionModel.userID.set(99)
component.selectionModel.ownerFilter = OwnerFilterType.SHARED_BY_ME
component.selectionModel.userID = 99
expect(component.sharedByFilterLabel).toEqual('Shared by another user')
})
it('should retain relative labels for filters bound to the current user', () => {
component.selectionModel.userID.set(currentUserID)
component.selectionModel.userID = currentUserID
expect(component.ownerFilterLabel).toEqual('My documents')
expect(component.sharedByFilterLabel).toEqual('Shared by me')
component.selectionModel.excludeUsers.set([currentUserID])
component.selectionModel.excludeUsers = [currentUserID]
expect(component.ownerExclusionFilterLabel).toEqual('Shared with me')
})
it('should retain relative labels for inactive filter choices', () => {
component.selectionModel.ownerFilter.set(OwnerFilterType.NONE)
component.selectionModel.ownerFilter = OwnerFilterType.NONE
expect(component.ownerFilterLabel).toEqual('My documents')
expect(component.ownerExclusionFilterLabel).toEqual('Shared with me')
@@ -148,41 +148,32 @@ describe('PermissionsFilterDropdownComponent', () => {
it('should support reset', () => {
component.setFilter(OwnerFilterType.OTHERS)
expect(component.selectionModel.ownerFilter()).not.toEqual(
expect(component.selectionModel.ownerFilter).not.toEqual(
OwnerFilterType.NONE
)
component.reset()
expect(component.selectionModel.ownerFilter()).toEqual(OwnerFilterType.NONE)
expect(component.selectionModel.ownerFilter).toEqual(OwnerFilterType.NONE)
})
it('should toggle owner filter type when users selected', () => {
component.selectionModel.ownerFilter.set(OwnerFilterType.NONE)
component.selectionModel.ownerFilter = OwnerFilterType.NONE
// this would normally be done by select component
component.selectionModel.includeUsers.set([12])
component.selectionModel.includeUsers = [12]
component.onUserSelect()
expect(component.selectionModel.ownerFilter()).toEqual(
OwnerFilterType.OTHERS
)
expect(component.selectionModel.ownerFilter).toEqual(OwnerFilterType.OTHERS)
// this would normally be done by select component
component.selectionModel.includeUsers.set(null)
component.selectionModel.includeUsers = null
component.onUserSelect()
expect(component.selectionModel.ownerFilter()).toEqual(OwnerFilterType.NONE)
expect(component.selectionModel.ownerFilter).toEqual(OwnerFilterType.NONE)
})
it('should emit a selection model depending on the type of owner filter set', () => {
const emitted = () => ({
excludeUsers: ownerFilterSetResult.excludeUsers(),
hideUnowned: ownerFilterSetResult.hideUnowned(),
includeUsers: ownerFilterSetResult.includeUsers(),
ownerFilter: ownerFilterSetResult.ownerFilter(),
userID: ownerFilterSetResult.userID(),
})
component.selectionModel.ownerFilter.set(OwnerFilterType.NONE)
component.selectionModel.ownerFilter = OwnerFilterType.NONE
component.setFilter(OwnerFilterType.SELF)
expect(emitted()).toEqual({
expect(ownerFilterSetResult).toEqual({
excludeUsers: [],
hideUnowned: false,
includeUsers: [],
@@ -191,7 +182,7 @@ describe('PermissionsFilterDropdownComponent', () => {
})
component.setFilter(OwnerFilterType.NOT_SELF)
expect(emitted()).toEqual({
expect(ownerFilterSetResult).toEqual({
excludeUsers: [currentUserID],
hideUnowned: false,
includeUsers: [],
@@ -200,7 +191,7 @@ describe('PermissionsFilterDropdownComponent', () => {
})
component.setFilter(OwnerFilterType.NONE)
expect(emitted()).toEqual({
expect(ownerFilterSetResult).toEqual({
excludeUsers: [],
hideUnowned: false,
includeUsers: [],
@@ -209,7 +200,7 @@ describe('PermissionsFilterDropdownComponent', () => {
})
component.setFilter(OwnerFilterType.SHARED_BY_ME)
expect(emitted()).toEqual({
expect(ownerFilterSetResult).toEqual({
excludeUsers: [],
hideUnowned: false,
includeUsers: [],
@@ -218,7 +209,7 @@ describe('PermissionsFilterDropdownComponent', () => {
})
component.setFilter(OwnerFilterType.UNOWNED)
expect(emitted()).toEqual({
expect(ownerFilterSetResult).toEqual({
excludeUsers: [],
hideUnowned: false,
includeUsers: [],
@@ -25,18 +25,18 @@ import { ComponentWithPermissions } from '../../with-permissions/with-permission
import { ClearableBadgeComponent } from '../clearable-badge/clearable-badge.component'
export class PermissionsSelectionModel {
readonly ownerFilter = signal(OwnerFilterType.NONE)
readonly hideUnowned = signal(false)
readonly userID = signal<number>(null)
readonly includeUsers = signal<number[]>([])
readonly excludeUsers = signal<number[]>([])
ownerFilter: OwnerFilterType
hideUnowned: boolean
userID: number
includeUsers: number[]
excludeUsers: number[]
clear() {
this.ownerFilter.set(OwnerFilterType.NONE)
this.userID.set(null)
this.hideUnowned.set(false)
this.includeUsers.set([])
this.excludeUsers.set([])
this.ownerFilter = OwnerFilterType.NONE
this.userID = null
this.hideUnowned = false
this.includeUsers = []
this.excludeUsers = []
}
}
@@ -84,31 +84,33 @@ export class PermissionsFilterDropdownComponent extends ComponentWithPermissions
readonly users = signal<User[]>([])
hideUnowned: boolean
get isActive(): boolean {
return (
this.selectionModel.ownerFilter() !== OwnerFilterType.NONE ||
this.selectionModel.hideUnowned()
this.selectionModel.ownerFilter !== OwnerFilterType.NONE ||
this.selectionModel.hideUnowned
)
}
get ownerFilterLabel(): string {
if (
this.selectionModel?.ownerFilter() !== OwnerFilterType.SELF ||
this.selectionModel?.userID() === this.settingsService.currentUser()?.id
this.selectionModel?.ownerFilter !== OwnerFilterType.SELF ||
this.selectionModel?.userID === this.settingsService.currentUser()?.id
) {
return $localize`My documents`
}
const username = this.getUsername(this.selectionModel?.userID())
const username = this.getUsername(this.selectionModel?.userID)
return username
? $localize`Owned by ${username}`
: $localize`Owned by another user`
}
get ownerExclusionFilterLabel(): string {
const excludedUsers = this.selectionModel?.excludeUsers() ?? []
const excludedUsers = this.selectionModel?.excludeUsers ?? []
if (
this.selectionModel?.ownerFilter() !== OwnerFilterType.NOT_SELF ||
this.selectionModel?.ownerFilter !== OwnerFilterType.NOT_SELF ||
(excludedUsers.length === 1 &&
excludedUsers[0] === this.settingsService.currentUser()?.id)
) {
@@ -128,13 +130,13 @@ export class PermissionsFilterDropdownComponent extends ComponentWithPermissions
get sharedByFilterLabel(): string {
if (
this.selectionModel?.ownerFilter() !== OwnerFilterType.SHARED_BY_ME ||
this.selectionModel?.userID() === this.settingsService.currentUser()?.id
this.selectionModel?.ownerFilter !== OwnerFilterType.SHARED_BY_ME ||
this.selectionModel?.userID === this.settingsService.currentUser()?.id
) {
return $localize`Shared by me`
}
const username = this.getUsername(this.selectionModel?.userID())
const username = this.getUsername(this.selectionModel?.userID)
return username
? $localize`Shared by ${username}`
: $localize`Shared by another user`
@@ -167,36 +169,34 @@ export class PermissionsFilterDropdownComponent extends ComponentWithPermissions
}
setFilter(type: OwnerFilterType) {
this.selectionModel.ownerFilter.set(type)
if (this.selectionModel.ownerFilter() === OwnerFilterType.SELF) {
this.selectionModel.includeUsers.set([])
this.selectionModel.excludeUsers.set([])
this.selectionModel.userID.set(this.settingsService.currentUser().id)
this.selectionModel.hideUnowned.set(false)
} else if (this.selectionModel.ownerFilter() === OwnerFilterType.NOT_SELF) {
this.selectionModel.userID.set(null)
this.selectionModel.includeUsers.set([])
this.selectionModel.excludeUsers.set([
this.settingsService.currentUser().id,
])
this.selectionModel.hideUnowned.set(false)
} else if (this.selectionModel.ownerFilter() === OwnerFilterType.NONE) {
this.selectionModel.userID.set(null)
this.selectionModel.includeUsers.set([])
this.selectionModel.excludeUsers.set([])
this.selectionModel.hideUnowned.set(false)
this.selectionModel.ownerFilter = type
if (this.selectionModel.ownerFilter === OwnerFilterType.SELF) {
this.selectionModel.includeUsers = []
this.selectionModel.excludeUsers = []
this.selectionModel.userID = this.settingsService.currentUser().id
this.selectionModel.hideUnowned = false
} else if (this.selectionModel.ownerFilter === OwnerFilterType.NOT_SELF) {
this.selectionModel.userID = null
this.selectionModel.includeUsers = []
this.selectionModel.excludeUsers = [this.settingsService.currentUser().id]
this.selectionModel.hideUnowned = false
} else if (this.selectionModel.ownerFilter === OwnerFilterType.NONE) {
this.selectionModel.userID = null
this.selectionModel.includeUsers = []
this.selectionModel.excludeUsers = []
this.selectionModel.hideUnowned = false
} else if (
this.selectionModel.ownerFilter() === OwnerFilterType.SHARED_BY_ME
this.selectionModel.ownerFilter === OwnerFilterType.SHARED_BY_ME
) {
this.selectionModel.userID.set(this.settingsService.currentUser()?.id)
this.selectionModel.includeUsers.set([])
this.selectionModel.excludeUsers.set([])
this.selectionModel.hideUnowned.set(false)
} else if (this.selectionModel.ownerFilter() === OwnerFilterType.UNOWNED) {
this.selectionModel.userID.set(null)
this.selectionModel.includeUsers.set([])
this.selectionModel.excludeUsers.set([])
this.selectionModel.hideUnowned.set(false)
this.selectionModel.userID = this.settingsService.currentUser()?.id
this.selectionModel.includeUsers = []
this.selectionModel.excludeUsers = []
this.selectionModel.hideUnowned = false
} else if (this.selectionModel.ownerFilter === OwnerFilterType.UNOWNED) {
this.selectionModel.userID = null
this.selectionModel.includeUsers = []
this.selectionModel.excludeUsers = []
this.selectionModel.hideUnowned = false
}
this.onChange()
}
@@ -206,11 +206,11 @@ export class PermissionsFilterDropdownComponent extends ComponentWithPermissions
}
onUserSelect() {
this.selectionModel.ownerFilter.set(
this.selectionModel.includeUsers()?.length
? OwnerFilterType.OTHERS
: OwnerFilterType.NONE
)
if (this.selectionModel.includeUsers?.length) {
this.selectionModel.ownerFilter = OwnerFilterType.OTHERS
} else {
this.selectionModel.ownerFilter = OwnerFilterType.NONE
}
this.onChange()
}
@@ -1209,53 +1209,24 @@ describe('DocumentDetailComponent', () => {
expect(fixture.debugElement.queryAll(By.css('textarea.rtl'))).not.toBeNull()
})
it('should display built-in pdf viewer if not disabled', async () => {
it('should display built-in pdf viewer if not disabled', () => {
initNormally()
component.document.update((document) => ({
...document,
archived_file_name: 'file.pdf',
}))
component.document().archived_file_name = 'file.pdf'
settingsService.set(SETTINGS_KEYS.USE_NATIVE_PDF_VIEWER, false)
expect(component.useNativePdfViewer).toBeFalsy()
await fixture.whenStable()
fixture.detectChanges()
expect(fixture.debugElement.query(By.css('pngx-pdf-viewer'))).not.toBeNull()
})
it('should display native pdf viewer if enabled', () => {
initNormally()
component.document.update((document) => ({
...document,
archived_file_name: 'file.pdf',
}))
component.document().archived_file_name = 'file.pdf'
settingsService.set(SETTINGS_KEYS.USE_NATIVE_PDF_VIEWER, true)
expect(component.useNativePdfViewer).toBeTruthy()
fixture.detectChanges()
expect(fixture.debugElement.query(By.css('object'))).not.toBeNull()
})
it('should reflect signal-backed document detail display settings', () => {
settingsService.set(SETTINGS_KEYS.DOCUMENT_EDITING_OVERLAY_THUMBNAIL, false)
settingsService.set(SETTINGS_KEYS.DOCUMENT_DETAILS_HIDDEN_FIELDS, [
component.DocumentDetailFieldID.Correspondent,
])
expect(component.showThumbnailOverlay).toBeFalsy()
expect(
component.isFieldHidden(component.DocumentDetailFieldID.Correspondent)
).toBeTruthy()
expect(
component.isFieldHidden(component.DocumentDetailFieldID.DocumentType)
).toBeFalsy()
settingsService.set(SETTINGS_KEYS.DOCUMENT_EDITING_OVERLAY_THUMBNAIL, true)
settingsService.set(SETTINGS_KEYS.DOCUMENT_DETAILS_HIDDEN_FIELDS, [])
expect(component.showThumbnailOverlay).toBeTruthy()
expect(
component.isFieldHidden(component.DocumentDetailFieldID.Correspondent)
).toBeFalsy()
})
it('should attempt to retrieve metadata', () => {
const metadataSpy = jest.spyOn(documentService, 'getMetadata')
metadataSpy.mockReturnValue(of({ has_archive_version: true }))
@@ -1714,10 +1685,7 @@ describe('DocumentDetailComponent', () => {
it('should change preview element by render type', () => {
initNormally()
component.document.update((document) => ({
...document,
archived_file_name: 'file.pdf',
}))
component.document().archived_file_name = 'file.pdf'
fixture.detectChanges()
expect(component.archiveContentRenderType).toEqual(
component.ContentRenderType.PDF
@@ -1726,11 +1694,8 @@ describe('DocumentDetailComponent', () => {
fixture.debugElement.query(By.css('pdf-viewer-container'))
).not.toBeUndefined()
component.document.update((document) => ({
...document,
archived_file_name: undefined,
mime_type: 'text/plain',
}))
component.document().archived_file_name = undefined
component.document().mime_type = 'text/plain'
fixture.detectChanges()
expect(component.archiveContentRenderType).toEqual(
component.ContentRenderType.Text
@@ -1739,10 +1704,7 @@ describe('DocumentDetailComponent', () => {
fixture.debugElement.query(By.css('div.preview-sticky'))
).not.toBeUndefined()
component.document.update((document) => ({
...document,
mime_type: 'image/jpeg',
}))
component.document().mime_type = 'image/jpeg'
fixture.detectChanges()
expect(component.archiveContentRenderType).toEqual(
component.ContentRenderType.Image
@@ -1750,12 +1712,9 @@ describe('DocumentDetailComponent', () => {
expect(
fixture.debugElement.query(By.css('.preview-sticky img'))
).not.toBeUndefined()
component.document.update((document) => ({
...document,
mime_type:
'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
}))
fixture.detectChanges()
;((component.document().mime_type =
'application/vnd.openxmlformats-officedocument.wordprocessingml.document'),
fixture.detectChanges())
expect(component.archiveContentRenderType).toEqual(
component.ContentRenderType.Other
)
@@ -227,19 +227,6 @@ export class DocumentDetailComponent
private deviceDetectorService = inject(DeviceDetectorService)
private savedViewService = inject(SavedViewService)
private readonly websocketStatusService = inject(WebsocketStatusService)
private readonly useNativePdfViewerSetting = this.settings.getSignal<boolean>(
SETTINGS_KEYS.USE_NATIVE_PDF_VIEWER
)
private readonly aiEnabledSetting = this.settings.getSignal<boolean>(
SETTINGS_KEYS.AI_ENABLED
)
private readonly showThumbnailOverlaySetting =
this.settings.getSignal<boolean>(
SETTINGS_KEYS.DOCUMENT_EDITING_OVERLAY_THUMBNAIL
)
private readonly hiddenFieldsSetting = this.settings.getSignal<
DocumentDetailFieldID[]
>(SETTINGS_KEYS.DOCUMENT_DETAILS_HIDDEN_FIELDS)
@ViewChild('inputTitle')
titleInput: TextComponent
@@ -346,7 +333,8 @@ export class DocumentDetailComponent
}
get useNativePdfViewer(): boolean {
return this.useNativePdfViewerSetting()
this.settings.trackChanges()
return this.settings.get(SETTINGS_KEYS.USE_NATIVE_PDF_VIEWER)
}
get isMobile(): boolean {
@@ -354,10 +342,12 @@ export class DocumentDetailComponent
}
get aiEnabled(): boolean {
return this.aiEnabledSetting()
this.settings.trackChanges()
return this.settings.get(SETTINGS_KEYS.AI_ENABLED)
}
get archiveContentRenderType(): ContentRenderType {
this.settings.trackChanges()
const hasArchiveVersion =
this.metadata()?.has_archive_version ??
!!this.document()?.archived_file_name
@@ -369,17 +359,22 @@ export class DocumentDetailComponent
}
get originalContentRenderType(): ContentRenderType {
this.settings.trackChanges()
return this.getRenderType(
this.metadata()?.original_mime_type || this.document()?.mime_type
)
}
get showThumbnailOverlay(): boolean {
return this.showThumbnailOverlaySetting()
this.settings.trackChanges()
return this.settings.get(SETTINGS_KEYS.DOCUMENT_EDITING_OVERLAY_THUMBNAIL)
}
isFieldHidden(fieldId: DocumentDetailFieldID): boolean {
return this.hiddenFieldsSetting().includes(fieldId)
this.settings.trackChanges()
return this.settings
.get(SETTINGS_KEYS.DOCUMENT_DETAILS_HIDDEN_FIELDS)
.includes(fieldId)
}
private getRenderType(mimeType: string): ContentRenderType {
@@ -121,8 +121,6 @@ export class DocumentListComponent
settingsService = inject(SettingsService)
private hotKeyService = inject(HotKeyService)
permissionService = inject(PermissionsService)
private readonly notesEnabledSetting =
this.settingsService.getSignal<boolean>(SETTINGS_KEYS.NOTES_ENABLED)
DisplayField = DisplayField
DisplayMode = DisplayMode
@@ -576,7 +574,8 @@ export class DocumentListComponent
}
get notesEnabled(): boolean {
return this.notesEnabledSetting()
this.settingsService.trackChanges()
return this.settingsService.get(SETTINGS_KEYS.NOTES_ENABLED)
}
resetFilters() {
@@ -621,43 +621,6 @@ describe('FilterEditorComponent', () => {
component.toggleTag(2) // coverage
})
it('should reflect ingested tag filter rules in the dropdown toggle', () => {
const dropdown = fixture.debugElement.query(
By.css('pngx-filterable-dropdown')
)
const toggle = dropdown.nativeElement.querySelector('#dropdown_tags')
expect(toggle.classList.contains('btn-primary')).toBeFalsy()
expect(
dropdown.nativeElement.querySelector('pngx-clearable-badge')
).toBeNull()
// switching to a view with a tag filter
component.filterRules = [
{
rule_type: FILTER_HAS_TAGS_ALL,
value: '2',
},
]
fixture.detectChanges()
expect(toggle.classList.contains('btn-primary')).toBeTruthy()
expect(
dropdown.nativeElement.querySelector('pngx-clearable-badge')
).not.toBeNull()
// and back to a view without one
component.filterRules = [
{
rule_type: FILTER_HAS_CORRESPONDENT_ANY,
value: '12',
},
]
fixture.detectChanges()
expect(toggle.classList.contains('btn-primary')).toBeFalsy()
expect(
dropdown.nativeElement.querySelector('pngx-clearable-badge')
).toBeNull()
})
it('should ingest filter rules for has any tags', () => {
expect(component.tagSelectionModel.getSelectedItems()).toHaveLength(0)
component.filterRules = [
@@ -1115,7 +1078,7 @@ describe('FilterEditorComponent', () => {
})
it('should ingest filter rules for owner', () => {
expect(component.permissionsSelectionModel.ownerFilter()).toEqual(
expect(component.permissionsSelectionModel.ownerFilter).toEqual(
OwnerFilterType.NONE
)
component.filterRules = [
@@ -1124,38 +1087,15 @@ describe('FilterEditorComponent', () => {
value: '100',
},
]
expect(component.permissionsSelectionModel.ownerFilter()).toEqual(
expect(component.permissionsSelectionModel.ownerFilter).toEqual(
OwnerFilterType.SELF
)
expect(component.permissionsSelectionModel.hideUnowned()).toBeFalsy()
expect(component.permissionsSelectionModel.userID()).toEqual(100)
})
it('should reflect ingested owner filter rules in the dropdown toggle', () => {
const dropdown = fixture.debugElement.query(
By.css('pngx-permissions-filter-dropdown')
)
const toggle = dropdown.nativeElement.querySelector('button')
expect(toggle.classList.contains('btn-primary')).toBeFalsy()
// switching to a view with an owner filter
component.filterRules = [
{
rule_type: FILTER_OWNER,
value: '100',
},
]
fixture.detectChanges()
expect(toggle.classList.contains('btn-primary')).toBeTruthy()
// and back to a view without one
component.filterRules = []
fixture.detectChanges()
expect(toggle.classList.contains('btn-primary')).toBeFalsy()
expect(component.permissionsSelectionModel.hideUnowned).toBeFalsy()
expect(component.permissionsSelectionModel.userID).toEqual(100)
})
it('should ingest filter rules for owner is others', () => {
expect(component.permissionsSelectionModel.ownerFilter()).toEqual(
expect(component.permissionsSelectionModel.ownerFilter).toEqual(
OwnerFilterType.NONE
)
component.filterRules = [
@@ -1164,14 +1104,14 @@ describe('FilterEditorComponent', () => {
value: '50',
},
]
expect(component.permissionsSelectionModel.ownerFilter()).toEqual(
expect(component.permissionsSelectionModel.ownerFilter).toEqual(
OwnerFilterType.OTHERS
)
expect(component.permissionsSelectionModel.includeUsers()).toContain(50)
expect(component.permissionsSelectionModel.includeUsers).toContain(50)
})
it('should ingest filter rules for owner does not include others', () => {
expect(component.permissionsSelectionModel.ownerFilter()).toEqual(
expect(component.permissionsSelectionModel.ownerFilter).toEqual(
OwnerFilterType.NONE
)
component.filterRules = [
@@ -1180,14 +1120,14 @@ describe('FilterEditorComponent', () => {
value: '50',
},
]
expect(component.permissionsSelectionModel.ownerFilter()).toEqual(
expect(component.permissionsSelectionModel.ownerFilter).toEqual(
OwnerFilterType.NOT_SELF
)
expect(component.permissionsSelectionModel.excludeUsers()).toContain(50)
expect(component.permissionsSelectionModel.excludeUsers).toContain(50)
})
it('should ingest filter rules for owner is null', () => {
expect(component.permissionsSelectionModel.ownerFilter()).toEqual(
expect(component.permissionsSelectionModel.ownerFilter).toEqual(
OwnerFilterType.NONE
)
component.filterRules = [
@@ -1196,10 +1136,10 @@ describe('FilterEditorComponent', () => {
value: 'true',
},
]
expect(component.permissionsSelectionModel.ownerFilter()).toEqual(
expect(component.permissionsSelectionModel.ownerFilter).toEqual(
OwnerFilterType.UNOWNED
)
expect(component.permissionsSelectionModel.hideUnowned()).toBeFalsy()
expect(component.permissionsSelectionModel.hideUnowned).toBeFalsy()
})
it('should ingest filter rules for owner is not null', () => {
@@ -1209,14 +1149,14 @@ describe('FilterEditorComponent', () => {
value: 'false',
},
]
expect(component.permissionsSelectionModel.hideUnowned()).toBeTruthy()
expect(component.permissionsSelectionModel.hideUnowned).toBeTruthy()
component.filterRules = [
{
rule_type: FILTER_OWNER_ISNULL,
value: '0',
},
]
expect(component.permissionsSelectionModel.hideUnowned()).toBeTruthy()
expect(component.permissionsSelectionModel.hideUnowned).toBeTruthy()
})
it('should ingest filter rules for shared by me', () => {
@@ -1226,7 +1166,7 @@ describe('FilterEditorComponent', () => {
value: '2',
},
]
expect(component.permissionsSelectionModel.userID()).toEqual(2)
expect(component.permissionsSelectionModel.userID).toEqual(2)
})
// GET filterRules
@@ -1992,10 +1932,7 @@ describe('FilterEditorComponent', () => {
value: '1',
},
])
component.permissionsSelectionModel.excludeUsers.update((users) => [
...users,
2,
])
component.permissionsSelectionModel.excludeUsers.push(2)
fixture.detectChanges()
expect(component.filterRules).toEqual([
{
@@ -2045,11 +1982,8 @@ describe('FilterEditorComponent', () => {
// TODO: mock input in code
// userSelect.query(By.css('input')).nativeElement.value = '3'
// userSelect.triggerEventHandler('change')
component.permissionsSelectionModel.ownerFilter.set(OwnerFilterType.OTHERS)
component.permissionsSelectionModel.includeUsers.update((users) => [
...users,
3,
])
component.permissionsSelectionModel.ownerFilter = OwnerFilterType.OTHERS
component.permissionsSelectionModel.includeUsers.push(3)
fixture.detectChanges()
expect(component.filterRules).toEqual([
{
@@ -2069,7 +2003,7 @@ describe('FilterEditorComponent', () => {
ownerToggle.nativeElement.checked = true
// ownerToggle.triggerEventHandler('change')
// TODO: ngModel isn't doing this here
component.permissionsSelectionModel.hideUnowned.set(true)
component.permissionsSelectionModel.hideUnowned = true
fixture.detectChanges()
expect(component.filterRules).toEqual([
{
@@ -735,50 +735,38 @@ export class FilterEditorComponent
this._textFilter = rule.value
break
case FILTER_OWNER:
this.permissionsSelectionModel.ownerFilter.set(OwnerFilterType.SELF)
this.permissionsSelectionModel.hideUnowned.set(false)
this.permissionsSelectionModel.ownerFilter = OwnerFilterType.SELF
this.permissionsSelectionModel.hideUnowned = false
if (rule.value)
this.permissionsSelectionModel.userID.set(
Number.parseInt(rule.value, 10)
)
this.permissionsSelectionModel.userID = parseInt(rule.value, 10)
break
case FILTER_OWNER_ANY:
this.permissionsSelectionModel.ownerFilter.set(OwnerFilterType.OTHERS)
this.permissionsSelectionModel.ownerFilter = OwnerFilterType.OTHERS
if (rule.value)
this.permissionsSelectionModel.includeUsers.update((users) => [
...users,
Number.parseInt(rule.value, 10),
])
this.permissionsSelectionModel.includeUsers.push(
parseInt(rule.value, 10)
)
break
case FILTER_OWNER_DOES_NOT_INCLUDE:
this.permissionsSelectionModel.ownerFilter.set(
OwnerFilterType.NOT_SELF
)
this.permissionsSelectionModel.ownerFilter = OwnerFilterType.NOT_SELF
if (rule.value)
this.permissionsSelectionModel.excludeUsers.update((users) => [
...users,
Number.parseInt(rule.value, 10),
])
this.permissionsSelectionModel.excludeUsers.push(
parseInt(rule.value, 10)
)
break
case FILTER_SHARED_BY_USER:
this.permissionsSelectionModel.ownerFilter.set(
this.permissionsSelectionModel.ownerFilter =
OwnerFilterType.SHARED_BY_ME
)
if (rule.value)
this.permissionsSelectionModel.userID.set(
Number.parseInt(rule.value, 10)
)
this.permissionsSelectionModel.userID = parseInt(rule.value, 10)
break
case FILTER_OWNER_ISNULL:
if (rule.value === 'true' || rule.value === '1') {
this.permissionsSelectionModel.hideUnowned.set(false)
this.permissionsSelectionModel.ownerFilter.set(
OwnerFilterType.UNOWNED
)
this.permissionsSelectionModel.hideUnowned = false
this.permissionsSelectionModel.ownerFilter = OwnerFilterType.UNOWNED
} else {
this.permissionsSelectionModel.hideUnowned.set(
this.permissionsSelectionModel.hideUnowned =
rule.value === 'false' || rule.value === '0'
)
break
}
}
@@ -1086,35 +1074,34 @@ export class FilterEditorComponent
})
}
}
if (this.permissionsSelectionModel.ownerFilter() == OwnerFilterType.SELF) {
if (this.permissionsSelectionModel.ownerFilter == OwnerFilterType.SELF) {
filterRules.push({
rule_type: FILTER_OWNER,
value: this.permissionsSelectionModel.userID().toString(),
value: this.permissionsSelectionModel.userID.toString(),
})
} else if (
this.permissionsSelectionModel.ownerFilter() == OwnerFilterType.NOT_SELF
this.permissionsSelectionModel.ownerFilter == OwnerFilterType.NOT_SELF
) {
filterRules.push({
rule_type: FILTER_OWNER_DOES_NOT_INCLUDE,
value: this.permissionsSelectionModel.excludeUsers()?.join(','),
value: this.permissionsSelectionModel.excludeUsers?.join(','),
})
} else if (
this.permissionsSelectionModel.ownerFilter() == OwnerFilterType.OTHERS
this.permissionsSelectionModel.ownerFilter == OwnerFilterType.OTHERS
) {
filterRules.push({
rule_type: FILTER_OWNER_ANY,
value: this.permissionsSelectionModel.includeUsers()?.join(','),
value: this.permissionsSelectionModel.includeUsers?.join(','),
})
} else if (
this.permissionsSelectionModel.ownerFilter() ==
OwnerFilterType.SHARED_BY_ME
this.permissionsSelectionModel.ownerFilter == OwnerFilterType.SHARED_BY_ME
) {
filterRules.push({
rule_type: FILTER_SHARED_BY_USER,
value: this.permissionsSelectionModel.userID().toString(),
value: this.permissionsSelectionModel.userID.toString(),
})
} else if (
this.permissionsSelectionModel.ownerFilter() == OwnerFilterType.UNOWNED
this.permissionsSelectionModel.ownerFilter == OwnerFilterType.UNOWNED
) {
filterRules.push({
rule_type: FILTER_OWNER_ISNULL,
@@ -1122,7 +1109,7 @@ export class FilterEditorComponent
})
}
if (this.permissionsSelectionModel.hideUnowned()) {
if (this.permissionsSelectionModel.hideUnowned) {
filterRules.push({
rule_type: FILTER_OWNER_ISNULL,
value: 'false',
@@ -210,48 +210,6 @@ describe('SettingsService', () => {
expect(settingsService.get(SETTINGS_KEYS.THEME_COLOR)).toEqual('#000000')
})
it('provides stable signals that update when settings change', () => {
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}ui_settings/`
)
req.flush(ui_settings)
const notesEnabled = settingsService.getSignal<boolean>(
SETTINGS_KEYS.NOTES_ENABLED
)
expect(notesEnabled()).toBeTruthy()
expect(
settingsService.getSignal<boolean>(SETTINGS_KEYS.NOTES_ENABLED)
).toBe(notesEnabled)
settingsService.set(SETTINGS_KEYS.NOTES_ENABLED, false)
expect(notesEnabled()).toBeFalsy()
})
it('updates setting signals when settings are reinitialized', () => {
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}ui_settings/`
)
req.flush(ui_settings)
const appTitle = settingsService.getSignal<string>(SETTINGS_KEYS.APP_TITLE)
settingsService.initializeSettings().subscribe()
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}ui_settings/`
)
req.flush({
...ui_settings,
settings: {
...ui_settings.settings,
app_title: 'Updated title',
},
})
expect(appTitle()).toBe('Updated title')
})
it('sets django cookie for languages', () => {
httpTestingController
.expectOne(`${environment.apiBaseUrl}ui_settings/`)
+4 -16
View File
@@ -2,8 +2,6 @@ import { HttpClient } from '@angular/common/http'
import {
DOCUMENT,
EventEmitter,
Signal,
computed,
inject,
Injectable,
LOCALE_ID,
@@ -299,7 +297,6 @@ export class SettingsService {
private settings: Record<string, any> = {}
private readonly settingsVersion = signal(0)
private readonly settingSignals = new Map<string, Signal<unknown>>()
readonly currentUser = signal<User>(undefined)
public settingsSaved: EventEmitter<any> = new EventEmitter()
@@ -329,6 +326,10 @@ export class SettingsService {
return !UNSAFE_OBJECT_KEYS.has(key)
}
public trackChanges(): void {
this.settingsVersion()
}
private assignSafeSettings(source: Record<string, any>) {
if (!source || typeof source !== 'object' || Array.isArray(source)) {
return
@@ -338,7 +339,6 @@ export class SettingsService {
if (!this.isSafeObjectKey(key)) continue
this.settings[key] = source[key]
}
this.settingsVersion.update((version) => version + 1)
}
// this is called by the app initializer in app.module
@@ -594,18 +594,6 @@ export class SettingsService {
}
}
getSignal<T = any>(key: string): Signal<T> {
let settingSignal = this.settingSignals.get(key)
if (!settingSignal) {
settingSignal = computed(() => {
this.settingsVersion()
return this.get(key)
})
this.settingSignals.set(key, settingSignal)
}
return settingSignal as Signal<T>
}
set(key: string, value: any) {
// parse key:key:key into nested object
let settingObj = this.settings
+10 -10
View File
@@ -507,8 +507,8 @@ def rotate(
logger.info(
f"Queued new rotated version for document {pair.root_doc.id} by {degrees} degrees",
)
except Exception as e:
logger.exception(f"Error rotating document {pair.root_doc.id}: {e}")
except Exception:
logger.exception(f"Error rotating document {pair.root_doc.id}")
return "OK"
@@ -554,9 +554,9 @@ def merge(
affected_docs.append(doc.id)
if handoff_asn is None and doc.archive_serial_number is not None:
handoff_asn = doc.archive_serial_number
except Exception as e:
except Exception:
logger.exception(
f"Error merging document {doc.id}, it will not be included in the merge: {e}",
f"Error merging document {doc.id}, it will not be included in the merge",
)
if len(affected_docs) == 0:
logger.warning("No documents were merged")
@@ -805,8 +805,8 @@ def split(
else:
group(consume_tasks).delay()
except Exception as e:
logger.exception(f"Error splitting document {doc.id}: {e}")
except Exception:
logger.exception(f"Error splitting document {doc.id}")
return "OK"
@@ -858,8 +858,8 @@ def delete_pages(
logger.info(
f"Queued new version for document {pair.root_doc.id} after deleting pages {pages}",
)
except Exception as e:
logger.exception(f"Error deleting pages from document {pair.root_doc.id}: {e}")
except Exception:
logger.exception(f"Error deleting pages from document {pair.root_doc.id}")
return "OK"
@@ -986,7 +986,7 @@ def edit_pdf(
group(consume_tasks).delay()
except Exception as e:
logger.exception(f"Error editing document {pair.root_doc.id}: {e}")
logger.exception(f"Error editing document {pair.root_doc.id}")
raise ValueError(
f"An error occurred while editing the document: {e}",
) from e
@@ -1097,7 +1097,7 @@ def remove_password(
except Exception as e:
logger.exception(
f"Error removing password from document {pair.root_doc.id}: {e}",
f"Error removing password from document {pair.root_doc.id}",
)
raise ValueError(
f"An error occurred while removing the password: {e}",
+7 -7
View File
@@ -72,8 +72,8 @@ def load_classifier(*, raise_exception: bool = False) -> DocumentClassifier | No
Path(settings.MODEL_FILE).unlink()
classifier = None
if raise_exception:
raise e
except ClassifierModelCorruptError as e:
raise
except ClassifierModelCorruptError:
# there's something wrong with the model file.
logger.exception(
"Unrecoverable error while loading document "
@@ -82,17 +82,17 @@ def load_classifier(*, raise_exception: bool = False) -> DocumentClassifier | No
Path(settings.MODEL_FILE).unlink()
classifier = None
if raise_exception:
raise e
except OSError as e:
raise
except OSError:
logger.exception("IO error while loading document classification model")
classifier = None
if raise_exception:
raise e
except Exception as e: # pragma: no cover
raise
except Exception: # pragma: no cover
logger.exception("Unknown error while loading document classification model")
classifier = None
if raise_exception:
raise e
raise
return classifier
+4 -6
View File
@@ -217,7 +217,7 @@ class ConsumerPluginMixin:
current_progress,
max_progress,
document_id=document_id,
owner_id=self.metadata.owner_id if self.metadata.owner_id else None,
owner_id=self.metadata.owner_id or None,
users_can_view=(self.metadata.view_users or [])
+ (self.metadata.change_users or []),
groups_can_view=(self.metadata.view_groups or [])
@@ -675,9 +675,7 @@ class ConsumerPlugin(
document=document,
logging_group=self.logging_group,
classifier=classifier,
original_file=self.unmodified_original
if self.unmodified_original
else self.working_copy,
original_file=self.unmodified_original or self.working_copy,
)
# After everything is in the database, copy the files into
@@ -858,7 +856,7 @@ class ConsumerPlugin(
else:
stats = Path(self.input_doc.original_file).stat()
create_date = timezone.make_aware(
datetime.datetime.fromtimestamp(stats.st_mtime),
datetime.datetime.fromtimestamp(stats.st_mtime), # noqa: DTZ006 - make_aware() requires a naive datetime
)
self.log.debug(f"Creation date from st_mtime: {create_date}")
@@ -972,7 +970,7 @@ class ConsumerPlugin(
try:
copy_basic_file_stats(source, target)
except Exception: # pragma: no cover
pass
self.log.debug("Unable to copy file stats from %s to %s", source, target)
class ConsumerPreflightPlugin(
+4 -2
View File
@@ -78,7 +78,9 @@ class CollatePlugin(NoCleanupPluginMixin, NoSetupPluginMixin, ConsumeTaskPlugin)
stats = staging.stat()
# if the file is older than the timeout, we don't consider
# it valid
if (dt.datetime.now().timestamp() - stats.st_mtime) > TIMEOUT_SECONDS:
if (
dt.datetime.now(tz=dt.UTC).timestamp() - stats.st_mtime
) > TIMEOUT_SECONDS:
logger.warning("Outdated double sided staging file exists, deleting it")
staging.unlink()
else:
@@ -134,7 +136,7 @@ class CollatePlugin(NoCleanupPluginMixin, NoSetupPluginMixin, ConsumeTaskPlugin)
shutil.move(pdf_file, staging)
# update access to modification time so we know if the file
# is outdated when another file gets uploaded
timestamp = dt.datetime.now().timestamp()
timestamp = dt.datetime.now(tz=dt.UTC).timestamp()
os.utime(staging, (timestamp, timestamp))
logger.info(
"Got scan with odd numbered pages of double-sided scan, moved it to %s",
+1 -1
View File
@@ -734,7 +734,7 @@ class CustomFieldQueryParser:
)
# Check if any of the requested IDs are missing.
missing_ids = set(value) - set(link.document_id for link in links)
missing_ids = set(value) - {link.document_id for link in links}
if missing_ids:
# The result should be an empty set in this case.
return Q(id__in=[])
@@ -631,23 +631,25 @@ class Command(BaseCommand):
):
# Process each change
for change_type, path in changes:
path = Path(path).resolve()
resolved_path = Path(path).resolve()
if change_type == Change.deleted:
# Consumed (or otherwise removed); a later file
# reusing this name must not be skipped as
# already-queued.
queued.discard(path)
if not path.is_file():
queued.discard(resolved_path)
if not resolved_path.is_file():
continue
if path in queued:
if resolved_path in queued:
# Already queued and awaiting consumption; a stray
# event (NAS metadata touch, AV scan, etc.) while
# the file sits on disk mid-consumption must not
# cause it to be queued a second time (GH #13511).
logger.debug(f"Ignoring event for queued file: {path}")
logger.debug(
f"Ignoring event for queued file: {resolved_path}",
)
continue
logger.debug(f"Event: {change_type.name} for {path}")
tracker.track(path, change_type)
logger.debug(f"Event: {change_type.name} for {resolved_path}")
tracker.track(resolved_path, change_type)
# Check for stable files
for stable_path in tracker.get_stable_files():
+7 -1
View File
@@ -30,6 +30,10 @@ if TYPE_CHECKING:
logger = logging.getLogger("paperless.matching")
class UnsupportedWorkflowTriggerTypeError(Exception):
pass
def log_reason(
matching_model: MatchingModel | WorkflowTrigger,
document: Document,
@@ -691,7 +695,9 @@ def document_matches_workflow(
)
else:
# New trigger types need to be explicitly checked above
raise Exception(f"Trigger type {trigger_type} not yet supported")
raise UnsupportedWorkflowTriggerTypeError(
f"Trigger type {trigger_type} not yet supported",
)
if trigger_matched:
logger.info(f"Document matched {trigger} from {workflow}")
@@ -75,7 +75,7 @@ def recompute_checksums(apps, schema_editor):
if updated_fields:
batch.append(doc)
processed += 1
processed += 1 # noqa: SIM113
if len(batch) >= _BATCH_SIZE:
Document.objects.bulk_update(batch, ["checksum", "archive_checksum"])
+1 -1
View File
@@ -377,7 +377,7 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
from documents.versioning import versions_newest_first
if hasattr(self, "effective_content"):
return getattr(self, "effective_content")
return self.effective_content
if self.root_document_id is not None or self.pk is None:
return self.content
+2 -2
View File
@@ -41,7 +41,7 @@ def get_default_file_extension(mime_type: str) -> str:
return supported[mime_type]
ext = mimetypes.guess_extension(mime_type)
return ext if ext else ""
return ext or ""
def is_file_ext_supported(ext: str) -> bool:
@@ -110,7 +110,7 @@ def run_convert(
args += ["-define", "pdf:use-cropbox=true"] if use_cropbox else []
args += [str(input_file), str(output_file)]
logger.debug("Execute: " + " ".join(args), extra={"group": logging_group})
logger.debug("Execute: %s", " ".join(args), extra={"group": logging_group})
try:
run_subprocess(args, environment, logger)
@@ -43,8 +43,8 @@ def _discover_parser_class() -> type[DateParserPluginBase]:
valid_plugins.append(ep)
else:
logger.warning(f"Plugin {ep.name} does not subclass DateParser.")
except Exception as e:
logger.exception(f"Unable to load date parser plugin {ep.name}: {e}")
except Exception:
logger.exception(f"Unable to load date parser plugin {ep.name}")
if not valid_plugins:
return RegexDateParserPlugin
+2 -2
View File
@@ -91,8 +91,8 @@ class DateParserPluginBase(ABC):
},
locales=self.config.languages,
)
except Exception as e:
logger.exception(f"Error while parsing date string '{date_string}': {e}")
except Exception:
logger.exception(f"Error while parsing date string '{date_string}'")
return None
def _filter_date(
+4 -6
View File
@@ -59,11 +59,10 @@ def safe_regex_match(pattern: str, text: str, *, flags: int = 0):
try:
validate_regex_pattern(pattern)
compiled = regex.compile(pattern, flags=flags)
except (regex.error, ValueError) as exc:
except (regex.error, ValueError):
logger.exception(
"Error while processing regular expression %s: %s",
"Error while processing regular expression %s",
textwrap.shorten(pattern, width=80, placeholder=""),
exc,
)
return None
@@ -86,11 +85,10 @@ def safe_regex_sub(pattern: str, repl: str, text: str, *, flags: int = 0) -> str
try:
validate_regex_pattern(pattern)
compiled = regex.compile(pattern, flags=flags)
except (regex.error, ValueError) as exc:
except (regex.error, ValueError):
logger.exception(
"Error while processing regular expression %s: %s",
"Error while processing regular expression %s",
textwrap.shorten(pattern, width=80, placeholder=""),
exc,
)
return None
+2 -2
View File
@@ -1142,7 +1142,7 @@ def get_backend() -> TantivyBackend:
Returns:
Thread-safe singleton TantivyBackend instance
"""
global _backend, _backend_path
global _backend, _backend_path # noqa: PLW0603 - module-level singleton, no class to hold this state
current_path: Path = settings.INDEX_DIR
@@ -1173,7 +1173,7 @@ def reset_backend() -> None:
Forces creation of a new backend instance on the next get_backend() call.
Used for test isolation and when switching between different index directories.
"""
global _backend, _backend_path
global _backend, _backend_path # noqa: PLW0603 - module-level singleton, no class to hold this state
with _backend_lock:
if _backend is not None:
+1 -1
View File
@@ -240,7 +240,7 @@ def parse_user_query(
DEFAULT_SEARCH_FIELDS,
field_boosts=_FIELD_BOOSTS,
# (prefix=True, distance=1, transposition_cost_one=True) — edit-distance fuzziness
fuzzy_fields={f: (True, 1, True) for f in DEFAULT_SEARCH_FIELDS},
fuzzy_fields=dict.fromkeys(DEFAULT_SEARCH_FIELDS, (True, 1, True)),
)
# 0.1 boost keeps fuzzy hits ranked below exact matches (intentional)
clauses.append((tantivy.Occur.Should, tantivy.Query.boost_query(fuzzy, 0.1)))
+11 -12
View File
@@ -434,7 +434,7 @@ class OwnedObjectSerializer(
return set()
ctype = ContentType.objects.get_for_model(first_obj)
object_pks = list(obj.pk for obj in objects)
object_pks = [obj.pk for obj in objects]
pk_type = type(first_obj.pk)
def get_pks_for_permission_type(model):
@@ -730,7 +730,7 @@ class TagSerializer(MatchingModelSerializer, OwnedObjectSerializer):
self.instance.clean()
except ValidationError as e:
logger.debug("Tag parent validation failed: %s", e)
raise e
raise
finally:
self.instance.tn_parent = original_parent
else:
@@ -740,7 +740,7 @@ class TagSerializer(MatchingModelSerializer, OwnedObjectSerializer):
temp.clean()
except ValidationError as e:
logger.debug("Tag parent validation failed: %s", e)
raise e
raise
return super().validate(attrs)
@@ -1150,7 +1150,7 @@ class DocumentSerializer(
def to_representation(self, instance):
doc = super().to_representation(instance)
if "content" in self.fields and hasattr(instance, "effective_content"):
doc["content"] = getattr(instance, "effective_content") or ""
doc["content"] = instance.effective_content or ""
if self.truncate_content and "content" in self.fields:
doc["content"] = doc.get("content")[0:550]
return doc
@@ -1860,8 +1860,8 @@ class BulkEditSerializer(
if isinstance(custom_fields, dict):
try:
ids = [int(i[0]) for i in custom_fields.items()]
except Exception as e:
logger.exception(f"Error validating custom fields: {e}")
except Exception:
logger.exception("Error validating custom fields")
raise serializers.ValidationError(
f"{name} must be a list of integers or a dict of id:value pairs, see the log for details",
)
@@ -2059,13 +2059,12 @@ class BulkEditSerializer(
for doc in docs:
if "-" in doc:
pages.append(
[
x
for x in range(
list(
range(
int(doc.split("-")[0]),
int(doc.split("-")[1]) + 1,
)
],
),
),
)
else:
pages.append([int(doc)])
@@ -2926,7 +2925,7 @@ class ShareLinkBundleSerializer(OwnedObjectSerializer):
return share_link_bundle
def get_document_count(self, obj: ShareLinkBundle) -> int:
return getattr(obj, "document_total") or obj.documents.count()
return obj.document_total or obj.documents.count()
class BulkEditObjectsSerializer(SerializerWithPerms, SetPermissionsMixin):
+5 -4
View File
@@ -637,7 +637,7 @@ def update_filename_and_move_files(
# so this is not the end of the world.
# B: if moving the original file failed, nothing has changed
# anyway.
pass
logger.exception("Error reverting document changes")
# restore old values on the instance
instance.filename = old_filename
@@ -1102,10 +1102,11 @@ def _extract_input_data(
if v is None or k.startswith("_"):
continue
if isinstance(v, datetime.date):
v = v.isoformat()
override_dict[k] = v.isoformat()
elif isinstance(v, Path):
v = str(v)
override_dict[k] = v
override_dict[k] = str(v)
else:
override_dict[k] = v
if override_dict:
data["overrides"] = override_dict
return data
+6 -7
View File
@@ -217,9 +217,9 @@ def consume_file(
overrides.filename or input_doc.original_file.name,
self.request.id,
) as status_mgr,
TemporaryDirectory(dir=settings.SCRATCH_DIR) as tmp_dir,
TemporaryDirectory(dir=settings.SCRATCH_DIR) as tmp_dir_name,
):
tmp_dir = Path(tmp_dir)
tmp_dir = Path(tmp_dir_name)
msg = None
for plugin_class in plugins:
plugin_name = plugin_class.NAME
@@ -261,7 +261,7 @@ def consume_file(
)
except Exception as e:
logger.exception(f"{plugin_name} failed: {e}")
logger.exception(f"{plugin_name} failed")
status_mgr.send_progress(
ProgressStatusOptions.FAILED,
f"{e}",
@@ -495,8 +495,8 @@ def empty_trash(doc_ids=None) -> None:
content_type=ContentType.objects.get_for_model(Document),
object_id__in=deleted_document_ids,
).delete()
except Exception as e: # pragma: no cover
logger.exception(f"Error while emptying trash: {e}")
except Exception: # pragma: no cover
logger.exception("Error while emptying trash")
finally:
models.signals.post_delete.disconnect(
cleanup_document_deletion,
@@ -832,9 +832,8 @@ def build_share_link_bundle(bundle_id: int) -> None:
logger.info("Built share link bundle %s", bundle.pk)
except Exception as exc:
logger.exception(
"Failed to build share link bundle %s: %s",
"Failed to build share link bundle %s",
bundle_id,
exc,
)
bundle.status = ShareLinkBundle.Status.FAILED
bundle.last_error = {
+4
View File
@@ -78,6 +78,10 @@ class PlaceholderString(str):
def __ne__(self, other) -> bool:
return not self.__eq__(other)
def __hash__(self) -> int:
# Equal to both "-none-" and "none", so hash to a single canonical value
return hash("-none-")
NO_VALUE_PLACEHOLDER = PlaceholderString("-none-")
+3 -3
View File
@@ -138,9 +138,9 @@ def parse_w_workflow_placeholders(
# We're good!
return rendered_template
except UndefinedError as e:
except UndefinedError:
# The undefined class logs this already for us
raise e
raise
except TemplateSyntaxError as e:
logger.warning(f"Template syntax error in title generation: {e}")
except SecurityError as e:
@@ -150,5 +150,5 @@ def parse_w_workflow_placeholders(
logger.warning(
f"Invalid title format '{text}', workflow not applied: {e}",
)
raise e
raise
return None
@@ -296,7 +296,7 @@ class TestRegexDateParser:
# simulate parse failure for malformed input
if "99/99/9999" in date_string or "bad date" in date_string:
raise Exception("parse failed for malformed date")
raise Exception("parse failed for malformed date") # noqa: TRY002 - simulates a generic parser failure
return None
@@ -57,13 +57,13 @@ class MultiprocessCommand(PaperlessCommand):
def handle(self, *args, **options):
items = list(range(5))
results = []
for result in self.process_parallel(
_double_value,
items,
description="Processing...",
):
results.append(result)
results = list(
self.process_parallel(
_double_value,
items,
description="Processing...",
),
)
successes = sum(1 for r in results if r.success)
self.stdout.write(f"Successes: {successes}")
@@ -66,7 +66,7 @@ class TestWriteBatchLockRetry:
)
mock_sleep = mocker.patch(
"documents.search._backend.time.sleep",
side_effect=lambda s: sleep_values.append(s),
side_effect=sleep_values.append,
)
# Should not raise — 4th attempt succeeds
@@ -111,7 +111,7 @@ class TestWriteBatchLockRetry:
sleep_values: list[float] = []
mocker.patch(
"documents.search._backend.time.sleep",
side_effect=lambda s: sleep_values.append(s),
side_effect=sleep_values.append,
)
for _ in range(50):
sleep_values.clear()
+2 -2
View File
@@ -1003,8 +1003,8 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
for correspondent in response.data[field]:
self.assertEqual(correspondent["document_count"], 0)
self.assertCountEqual(
map(lambda c: c["id"], response.data[field]),
map(lambda c: c["id"], Entity.objects.values("id")),
(c["id"] for c in response.data[field]),
(c["id"] for c in Entity.objects.values("id")),
)
def test_api_selection_data(self) -> None:
-77
View File
@@ -1,18 +1,11 @@
from __future__ import annotations
from typing import TYPE_CHECKING
from unittest import mock
import pytest
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from rest_framework import status
from rest_framework.test import APIClient
from rest_framework.test import APITestCase
if TYPE_CHECKING:
from pytest_mock import MockerFixture
class TestChatStreamingViewInputValidation(APITestCase):
def setUp(self) -> None:
@@ -49,73 +42,3 @@ class TestChatStreamingViewInputValidation(APITestCase):
format="json",
)
assert resp.status_code == status.HTTP_400_BAD_REQUEST
@pytest.mark.django_db
class TestChatStreamingViewUnrestrictedFlag:
"""The document id filter may only be skipped (``unrestricted=True``) for
an active superuser, never for a regular user -- regardless of what
permissions that user holds.
"""
@pytest.fixture
def mocked_stream_chat(self, mocker: MockerFixture) -> mock.MagicMock:
"""AI enabled, with stream_chat_with_documents patched so the view
never touches the real vector store; returns the patched callable so
tests can inspect how it was called.
"""
mocker.patch("documents.views.AIConfig").return_value.ai_enabled = True
return mocker.patch(
"documents.views.stream_chat_with_documents",
return_value=iter(()),
)
@pytest.fixture
def viewer_client(self, user_client: APIClient, regular_user: User) -> APIClient:
"""The conftest regular-user client, granted the global
view_document permission -- the minimum ViewDocumentsPermissions
needs to reach the view at all. Model-level only: says nothing
about which documents (if any) this user can actually see.
"""
regular_user.user_permissions.add(
*Permission.objects.filter(codename="view_document"),
)
return user_client
@pytest.mark.parametrize(
("client_fixture", "expected_unrestricted"),
[
pytest.param("admin_client", True, id="superuser_is_unrestricted"),
pytest.param("viewer_client", False, id="regular_user_is_restricted"),
],
)
def test_unrestricted_only_for_superuser(
self,
request: pytest.FixtureRequest,
mocked_stream_chat: mock.MagicMock,
client_fixture: str,
*,
expected_unrestricted: bool,
) -> None:
"""
GIVEN:
- A superuser, or a regular user holding the global
view_document permission (but no object-level document access)
WHEN:
- They post a chat question with no document_id
THEN:
- stream_chat_with_documents is called with unrestricted=True
only for the superuser; the regular user is always
unrestricted=False, regardless of their permissions
"""
client: APIClient = request.getfixturevalue(client_fixture)
client.post(
"/api/documents/chat/",
data={"q": "What's in these documents?"},
format="json",
)
assert (
mocked_stream_chat.call_args.kwargs["unrestricted"] is expected_unrestricted
)
+2 -2
View File
@@ -18,8 +18,8 @@ class MockOpenIDProvider:
def get_brands(self):
default_servers = [
dict(id="yahoo", name="Yahoo", openid_url="http://me.yahoo.com"),
dict(id="hyves", name="Hyves", openid_url="http://hyves.nl"),
{"id": "yahoo", "name": "Yahoo", "openid_url": "http://me.yahoo.com"},
{"id": "hyves", "name": "Hyves", "openid_url": "http://hyves.nl"},
]
return default_servers
+2 -2
View File
@@ -205,12 +205,12 @@ class TestBarcode(
- Barcode is detected on page 1 (zero indexed)
"""
for test_file in [
for test_filename in [
"patch-code-t-middle-reverse.pdf",
"patch-code-t-middle-distorted.pdf",
"patch-code-t-middle-fuzzy.pdf",
]:
test_file = self.BARCODE_SAMPLE_DIR / test_file
test_file = self.BARCODE_SAMPLE_DIR / test_filename
with self.get_reader(test_file) as reader:
reader.detect()
+3 -3
View File
@@ -777,7 +777,7 @@ class TestPDFActions(DirectoriesMixin, TestCase):
sig.set.return_value.apply_async.side_effect = Exception("boom")
mock_consume_file.return_value = sig
with self.assertRaises(Exception):
with self.assertRaises(Exception): # noqa: B017 - mock injects a bare Exception
bulk_edit.merge(doc_ids, delete_originals=True)
self.doc1.refresh_from_db()
@@ -1318,7 +1318,7 @@ class TestPDFActions(DirectoriesMixin, TestCase):
sig.apply_async.side_effect = Exception("boom")
mock_chord.return_value = sig
with self.assertRaises(Exception):
with self.assertRaises(Exception): # noqa: B017 - mock injects a bare Exception
bulk_edit.edit_pdf(doc_ids, operations, delete_original=True)
self.doc2.refresh_from_db()
@@ -1430,7 +1430,7 @@ class TestPDFActions(DirectoriesMixin, TestCase):
{"page": 9999}, # invalid page, forces error during PDF load
]
with self.assertLogs("paperless.bulk_edit", level="ERROR"):
with self.assertRaises(Exception):
with self.assertRaises(ValueError):
bulk_edit.edit_pdf(doc_ids, operations)
mock_group.assert_not_called()
mock_consume_file.assert_not_called()
+1 -1
View File
@@ -806,7 +806,7 @@ class TestClassifier(DirectoriesMixin, TestCase):
Path(settings.MODEL_FILE).touch()
mock_load.side_effect = Exception()
with self.assertRaises(Exception):
with self.assertRaises(Exception): # noqa: B017 - mock injects a bare Exception
load_classifier(raise_exception=True)
+2 -2
View File
@@ -137,7 +137,7 @@ class FaultyParser(_BaseNewStyleParser):
class FaultyGenericExceptionParser(_BaseNewStyleParser):
def parse(self, document_path, mime_type, *, produce_archive: bool = True) -> None:
raise Exception("Generic exception.")
raise Exception("Generic exception.") # noqa: TRY002 - deliberately not a ParseError
def fake_magic_from_file(file, *, mime=False): # NOSONAR
@@ -1356,7 +1356,7 @@ class PreConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
script_calls = [
call
for call in m.call_args_list
if call.args and call.args[0] and call.args[0][0] not in ("pdftotext",)
if call.args and call.args[0] and call.args[0][0] != "pdftotext"
]
self.assertEqual(script_calls, [])
+8 -1
View File
@@ -44,6 +44,7 @@ from documents import tasks
from documents.data_models import ConsumableDocument
from documents.data_models import DocumentMetadataOverrides
from documents.data_models import DocumentSource
from documents.matching import UnsupportedWorkflowTriggerTypeError
from documents.matching import document_matches_workflow
from documents.matching import existing_document_matches_workflow
from documents.matching import prefilter_documents_by_workflowtrigger
@@ -2851,7 +2852,13 @@ class TestWorkflows(
doc = Document.objects.create(
title="test",
)
self.assertRaises(Exception, document_matches_workflow, doc, w, 99)
self.assertRaises(
UnsupportedWorkflowTriggerTypeError,
document_matches_workflow,
doc,
w,
99,
)
def test_removal_action_document_updated_workflow(self) -> None:
"""
+9 -5
View File
@@ -21,28 +21,32 @@ def uri_validator(value: str, allowed_schemes: set[str] | None = None) -> None:
parts = urlparse(value)
if not parts.scheme:
raise ValidationError(
_(f"Unable to parse URI {value}, missing scheme"),
_("Unable to parse URI %(value)s, missing scheme"),
params={"value": value},
)
elif not parts.netloc and not parts.path:
raise ValidationError(
_(f"Unable to parse URI {value}, missing net location or path"),
_("Unable to parse URI %(value)s, missing net location or path"),
params={"value": value},
)
if allowed_schemes and parts.scheme not in allowed_schemes:
raise ValidationError(
_(
f"URI scheme '{parts.scheme}' is not allowed. Allowed schemes: {', '.join(allowed_schemes)}",
"URI scheme '%(scheme)s' is not allowed. Allowed schemes: %(allowed_schemes)s",
),
params={"value": value, "scheme": parts.scheme},
params={
"value": value,
"scheme": parts.scheme,
"allowed_schemes": ", ".join(allowed_schemes),
},
)
except ValidationError:
raise
except Exception as e:
raise ValidationError(
_(f"Unable to parse URI {value}"),
_("Unable to parse URI %(value)s"),
params={"value": value},
) from e
+22 -30
View File
@@ -180,7 +180,6 @@ from documents.permissions import has_system_status_permission
from documents.permissions import permitted_document_ids
from documents.permissions import permitted_object_ids
from documents.permissions import set_permissions_for_object
from documents.permissions import user_is_unrestricted
from documents.plugins.date_parsing import get_date_parser
from documents.schema import generate_object_with_permissions_schema
from documents.search import SearchHit
@@ -1450,7 +1449,7 @@ class DocumentViewSet(
try:
lang = detect(doc.content)
except Exception:
pass
logger.debug("Unable to detect language for document %s", doc.pk)
meta["lang"] = lang
return Response(meta)
@@ -1488,13 +1487,12 @@ class DocumentViewSet(
with get_date_parser() as date_parser:
gen = date_parser.parse(doc.filename, doc.content)
dates = sorted(
{
i
for i in itertools.islice(
set(
itertools.islice(
gen,
settings.NUMBER_OF_SUGGESTED_DATES,
)
},
),
),
)
resp_data = {
@@ -1582,21 +1580,16 @@ class DocumentViewSet(
except ValueError as exc:
logger.exception(
"Invalid AI configuration while generating suggestions for "
"document %s: %s",
"document %s",
doc.pk,
exc,
exc_info=True,
)
raise ValidationError(
{"ai": [_("Invalid AI configuration.")]},
) from exc
except LLMTimeoutError as exc:
except LLMTimeoutError:
logger.exception(
"AI backend timed out while generating suggestions for "
"document %s: %s",
"AI backend timed out while generating suggestions for document %s",
doc.pk,
exc,
exc_info=True,
)
return Response(
{"ai": [_("AI backend request timed out.")]},
@@ -2069,7 +2062,7 @@ class DocumentViewSet(
doc_name, doc_data = serializer.validated_data.get("document")
version_label = serializer.validated_data.get("version_label")
t = int(mktime(datetime.now().timetuple()))
t = int(mktime(datetime.now().timetuple())) # noqa: DTZ005 - mktime() requires a local time tuple
settings.SCRATCH_DIR.mkdir(parents=True, exist_ok=True)
@@ -2330,12 +2323,10 @@ class ChatStreamingView(GenericAPIView[Any]):
return HttpResponseForbidden("Insufficient permissions")
documents = Document.objects.filter(pk=document.pk)
unrestricted = False
else:
documents = Document.objects.filter(
id__in=permitted_document_ids(request.user),
)
unrestricted = user_is_unrestricted(request.user)
output_language = get_llm_output_language(
ai_config=ai_config,
@@ -2346,7 +2337,6 @@ class ChatStreamingView(GenericAPIView[Any]):
stream_chat_with_documents(
query_str=question,
documents=documents,
unrestricted=unrestricted,
output_language=output_language,
),
content_type="text/event-stream",
@@ -3342,7 +3332,7 @@ class PostDocumentView(GenericAPIView[Any]):
cf = serializer.validated_data.get("custom_fields")
from_webui = serializer.validated_data.get("from_webui")
t = int(mktime(datetime.now().timetuple()))
t = int(mktime(datetime.now().timetuple())) # noqa: DTZ005 - mktime() requires a local time tuple
settings.SCRATCH_DIR.mkdir(parents=True, exist_ok=True)
@@ -4152,7 +4142,7 @@ class UiSettingsView(GenericAPIView[Any]):
user_resp["last_name"] = user.last_name
# strip <app_label>.
roles = map(lambda perm: re.sub(r"^\w+.", "", perm), user.get_all_permissions())
roles = (re.sub(r"^\w+.", "", perm) for perm in user.get_all_permissions())
return Response(
{
"user": user_resp,
@@ -5190,11 +5180,11 @@ class SystemStatusView(PassUserMixin):
f"{m.app}.{m.name}"
for m in MigrationRecorder.Migration.objects.all().order_by("id")
]
except Exception as e: # pragma: no cover
except Exception: # pragma: no cover
applied_migrations = []
db_status = "ERROR"
logger.exception(
f"System status detected a possible problem while connecting to the database: {e}",
"System status detected a possible problem while connecting to the database",
)
db_error = "Error connecting to database, check logs for more detail."
@@ -5210,10 +5200,10 @@ class SystemStatusView(PassUserMixin):
try:
client.ping()
redis_status = "OK"
except Exception as e:
except Exception:
redis_status = "ERROR"
logger.exception(
f"System status detected a possible problem while connecting to redis: {e}",
"System status detected a possible problem while connecting to redis",
)
redis_error = "Error connecting to redis, check logs for more detail."
@@ -5243,10 +5233,10 @@ class SystemStatusView(PassUserMixin):
else:
celery_active = "WARNING"
celery_error = "Celery worker responded unexpectedly."
except Exception as e:
except Exception:
celery_active = "ERROR"
logger.exception(
f"System status detected a possible problem while connecting to celery: {e}",
"System status detected a possible problem while connecting to celery",
)
celery_error = "Error connecting to celery, check logs for more detail."
@@ -5261,13 +5251,15 @@ class SystemStatusView(PassUserMixin):
index_dir = settings.INDEX_DIR
mtimes = [p.stat().st_mtime for p in index_dir.iterdir() if p.is_file()]
index_last_modified = (
make_aware(datetime.fromtimestamp(max(mtimes))) if mtimes else None
make_aware(datetime.fromtimestamp(max(mtimes))) # noqa: DTZ006 - make_aware() requires a naive datetime
if mtimes
else None
)
except Exception as e:
except Exception:
index_status = "ERROR"
index_error = "Error opening index, check logs for more detail."
logger.exception(
f"System status detected a possible problem while opening the index: {e}",
"System status detected a possible problem while opening the index",
)
index_last_modified = None
+5 -5
View File
@@ -66,7 +66,7 @@ def build_workflow_action_context(
else None
)
filename = document.original_file if document.original_file else ""
filename = document.original_file or ""
return {
"title": overrides.title
if overrides and overrides.title
@@ -179,9 +179,9 @@ def execute_email_action(
f"Sent {n_messages} notification email(s) to {action.email.to}",
extra={"group": logging_group},
)
except Exception as e:
except Exception:
logger.exception(
f"Error occurred sending notification email: {e}",
"Error occurred sending notification email",
extra={"group": logging_group},
)
@@ -265,9 +265,9 @@ def execute_webhook_action(
f"Webhook to {action.webhook.url} queued",
extra={"group": logging_group},
)
except Exception as e:
except Exception:
logger.exception(
f"Error occurred sending webhook: {e}",
"Error occurred sending webhook",
extra={"group": logging_group},
)
+1 -1
View File
@@ -47,7 +47,7 @@ def resolve_date(dates: list[str]) -> date | None:
"""
for value in dates:
try:
return datetime.strptime(value, "%Y-%m-%d").date()
return datetime.strptime(value, "%Y-%m-%d").date() # noqa: DTZ007 - only the calendar date is used, time/tz is discarded
except (TypeError, ValueError):
logger.debug("Ignoring unparsable suggested date %s", value)
return None
+1 -1
View File
@@ -70,6 +70,6 @@ def send_webhook(
logger.error(
f"Failed attempt sending webhook to {url}: {e}",
)
raise e
raise
finally:
transport.close()
+2 -1
View File
@@ -241,7 +241,7 @@ def check_v3_minimum_upgrade_version(
return []
logger = logging.getLogger(__name__)
last_applied = sorted(applied)[-1] if applied else "(none)"
last_applied = max(applied) if applied else "(none)"
logger.error(
"V3 upgrade check failed: last applied documents migration is %r. "
"Expected '1075_workflowaction_order' (v2.20.15). "
@@ -341,6 +341,7 @@ def get_tesseract_langs():
proc = subprocess.run(
[shutil.which("tesseract"), "--list-langs"],
capture_output=True,
check=False,
)
# Decode bytes to string, split on newlines, trim out the header
+3 -3
View File
@@ -84,7 +84,7 @@ def get_parser_registry() -> ParserRegistry:
ParserRegistry
The shared registry singleton.
"""
global _registry, _discovery_complete
global _registry, _discovery_complete # noqa: PLW0603 - module-level singleton, no class to hold this state
with _lock:
if _registry is None:
@@ -113,7 +113,7 @@ def init_builtin_parsers() -> None:
-------
None
"""
global _registry
global _registry # noqa: PLW0603 - module-level singleton, no class to hold this state
with _lock:
if _registry is None:
@@ -137,7 +137,7 @@ def reset_parser_registry() -> None:
-------
None
"""
global _registry, _discovery_complete
global _registry, _discovery_complete # noqa: PLW0603 - module-level singleton, no class to hold this state
_registry = None
_discovery_complete = False
+2 -2
View File
@@ -78,7 +78,7 @@ class RemoteEngineConfig:
def engine_is_valid(self) -> bool:
"""Return True when the engine is known and fully configured."""
return (
self.engine in ("azureai",)
self.engine == "azureai"
and self.api_key is not None
and not (self.engine == "azureai" and self.endpoint is None)
)
@@ -505,7 +505,7 @@ class RemoteDocumentParser:
return result.content
except Exception as e:
logger.exception("Azure AI Vision parsing failed: %s", e)
logger.exception("Azure AI Vision parsing failed")
raise ParseError(f"Azure AI Vision parsing failed: {e}") from e
finally:
+4 -3
View File
@@ -306,8 +306,9 @@ def extract_pdf_metadata(
for key, value in meta.items():
if isinstance(value, list):
value = " ".join(str(e) for e in value)
value = str(value)
str_value = " ".join(str(e) for e in value)
else:
str_value = str(value)
try:
m = namespace_pattern.match(key)
@@ -329,7 +330,7 @@ def extract_pdf_metadata(
namespace=namespace,
prefix=meta.REVERSE_NS[namespace],
key=key_value,
value=value,
value=str_value,
),
)
except Exception as e:
+14 -9
View File
@@ -294,7 +294,7 @@ if _CHANNELS_BACKEND.startswith("channels_redis."):
###############################################################################
EMAIL_HOST: Final[str] = os.getenv("PAPERLESS_EMAIL_HOST", "localhost")
EMAIL_PORT: Final[int] = int(os.getenv("PAPERLESS_EMAIL_PORT", 25))
EMAIL_PORT: Final[int] = get_int_from_env("PAPERLESS_EMAIL_PORT", 25)
EMAIL_HOST_USER: Final[str] = os.getenv("PAPERLESS_EMAIL_HOST_USER", "")
EMAIL_HOST_PASSWORD: Final[str] = os.getenv("PAPERLESS_EMAIL_HOST_PASSWORD", "")
DEFAULT_FROM_EMAIL: Final[str] = os.getenv("PAPERLESS_EMAIL_FROM", EMAIL_HOST_USER)
@@ -381,8 +381,9 @@ ACCOUNT_SESSION_REMEMBER = get_bool_from_env(
"True",
)
SESSION_EXPIRE_AT_BROWSER_CLOSE = not ACCOUNT_SESSION_REMEMBER
SESSION_COOKIE_AGE = int(
os.getenv("PAPERLESS_SESSION_COOKIE_AGE", 60 * 60 * 24 * 7 * 3),
SESSION_COOKIE_AGE = get_int_from_env(
"PAPERLESS_SESSION_COOKIE_AGE",
60 * 60 * 24 * 7 * 3,
)
# https://docs.djangoproject.com/en/5.1/ref/settings/#std-setting-SESSION_ENGINE
SESSION_ENGINE = "django.contrib.sessions.backends.cached_db"
@@ -395,7 +396,6 @@ if AUTO_LOGIN_USERNAME:
def _parse_remote_user_settings() -> str:
global MIDDLEWARE, AUTHENTICATION_BACKENDS, REST_FRAMEWORK
enable = get_bool_from_env("PAPERLESS_ENABLE_HTTP_REMOTE_USER")
enable_api = get_bool_from_env("PAPERLESS_ENABLE_HTTP_REMOTE_USER_API")
if enable or enable_api:
@@ -454,7 +454,6 @@ if ALLOWED_HOSTS != ["*"]:
def _parse_paperless_url():
global CSRF_TRUSTED_ORIGINS, CORS_ALLOWED_ORIGINS, ALLOWED_HOSTS
url = os.getenv("PAPERLESS_URL")
if url:
CSRF_TRUSTED_ORIGINS.append(url)
@@ -614,8 +613,8 @@ USE_TZ = True
LOGGING_DIR.mkdir(parents=True, exist_ok=True)
LOGROTATE_MAX_SIZE = os.getenv("PAPERLESS_LOGROTATE_MAX_SIZE", 1024 * 1024)
LOGROTATE_MAX_BACKUPS = os.getenv("PAPERLESS_LOGROTATE_MAX_BACKUPS", 20)
LOGROTATE_MAX_SIZE = get_int_from_env("PAPERLESS_LOGROTATE_MAX_SIZE", 1024 * 1024)
LOGROTATE_MAX_BACKUPS = get_int_from_env("PAPERLESS_LOGROTATE_MAX_BACKUPS", 20)
LOGGING = {
"version": 1,
@@ -811,9 +810,15 @@ IGNORABLE_FILES: Final[list[str]] = [
"Thumbs.db",
]
CONSUMER_POLLING_INTERVAL = float(os.getenv("PAPERLESS_CONSUMER_POLLING_INTERVAL", 0))
CONSUMER_POLLING_INTERVAL = get_float_from_env(
"PAPERLESS_CONSUMER_POLLING_INTERVAL",
0.0,
)
CONSUMER_STABILITY_DELAY = float(os.getenv("PAPERLESS_CONSUMER_STABILITY_DELAY", 5))
CONSUMER_STABILITY_DELAY = get_float_from_env(
"PAPERLESS_CONSUMER_STABILITY_DELAY",
5.0,
)
CONSUMER_DELETE_DUPLICATES = get_bool_from_env("PAPERLESS_CONSUMER_DELETE_DUPLICATES")
+1 -1
View File
@@ -111,7 +111,7 @@ def parse_dict_from_str(
return False
settings: dict[str, Any] = copy.deepcopy(defaults) if defaults else {}
_type_map = type_map if type_map else {}
_type_map = type_map or {}
if not env_str:
return settings
@@ -114,17 +114,17 @@ def test_cache_hit_when_enabled() -> None:
assert settings.CACHALOT_TIMEOUT == 1
# Read a table to populate the cache
list(list(Tag.objects.values_list("id", flat=True)))
list(Tag.objects.values_list("id", flat=True))
# Invalidate the cache then read the database, there should be DB hit
invalidate_db_cache()
with CaptureQueriesContext(connection) as ctx:
list(list(Tag.objects.values_list("id", flat=True)))
list(Tag.objects.values_list("id", flat=True))
assert len(ctx)
# Doing the same request again should hit the cache, not the DB
with CaptureQueriesContext(connection) as ctx:
list(list(Tag.objects.values_list("id", flat=True)))
list(Tag.objects.values_list("id", flat=True))
assert not len(ctx)
# Wait the end of TTL
@@ -133,7 +133,7 @@ def test_cache_hit_when_enabled() -> None:
# Read the DB again. The DB should be hit because the cache has expired
with CaptureQueriesContext(connection) as ctx:
list(list(Tag.objects.values_list("id", flat=True)))
list(Tag.objects.values_list("id", flat=True))
assert len(ctx)
# Invalidate the cache at the end of test
@@ -149,7 +149,7 @@ def test_cache_is_disabled_by_default() -> None:
# Read the table multiple times: the DB should always be hit without cache
for _ in range(3):
with CaptureQueriesContext(connection) as ctx:
list(list(Tag.objects.values_list("id", flat=True)))
list(Tag.objects.values_list("id", flat=True))
assert len(ctx)
# Invalidate the cache at the end of test
+1 -1
View File
@@ -59,7 +59,7 @@ def test_ocr_to_dateparser_languages_exception(
raise RuntimeError("Simulated error")
with caplog.at_level(logging.WARNING):
monkeypatch.setattr(utils, "LocaleDataLoader", lambda: DummyLoader())
monkeypatch.setattr(utils, "LocaleDataLoader", DummyLoader)
result = utils.ocr_to_dateparser_languages("eng+fra")
assert result == []
assert (
+6 -23
View File
@@ -8,8 +8,7 @@ from documents.models import Document
from paperless.config import AIConfig
from paperless_ai.client import AIClient
from paperless_ai.db import db_connection_released
from paperless_ai.indexing import document_id_filters
from paperless_ai.indexing import exclude_document_ids_filter
from paperless_ai.indexing import _document_id_filters
from paperless_ai.indexing import get_rag_prompt_helper
from paperless_ai.indexing import load_or_build_index
from paperless_ai.indexing import read_store
@@ -96,27 +95,22 @@ def _format_chat_metadata_trailer(references: list[dict[str, int | str]]) -> str
def stream_chat_with_documents(
query_str: str,
documents: QuerySet[Document],
*,
unrestricted: bool = False,
output_language: str | None = None,
):
try:
yield from _stream_chat_with_documents(
query_str,
documents,
unrestricted=unrestricted,
output_language=output_language,
)
except Exception as e:
logger.exception("Failed to stream document chat response: %s", e)
except Exception:
logger.exception("Failed to stream document chat response")
yield CHAT_ERROR_MESSAGE
def _stream_chat_with_documents(
query_str: str,
documents: QuerySet[Document],
*,
unrestricted: bool = False,
output_language: str | None = None,
):
if not documents.exists():
@@ -129,20 +123,9 @@ def _stream_chat_with_documents(
from llama_index.core.retrievers import VectorIndexRetriever
config = AIConfig()
if unrestricted:
# Exclude trashed ids (usually few) instead of an IN filter over the
# full permitted set, which risks the vector store's bound parameter
# limit (_MAX_IN_VALUES) on large installs. Trashed documents stay
# indexed until permanent deletion (delete_document_from_llm_index
# hangs off post_delete, not trash), so must be excluded explicitly.
trashed_ids = Document.global_objects.filter(
deleted_at__isnull=False,
).values_list("pk", flat=True)
filters = exclude_document_ids_filter(str(pk) for pk in trashed_ids)
else:
filters = document_id_filters(
str(pk) for pk in documents.values_list("pk", flat=True)
)
filters = _document_id_filters(
str(pk) for pk in documents.values_list("pk", flat=True)
)
# Hold the shared read lock for the whole operation: the query engine
# retrieves from the vector store again during synthesis, so the connection
+2 -19
View File
@@ -362,7 +362,7 @@ def _embed_nodes(nodes: list["BaseNode"], embed_model) -> None:
node.embedding = emb
def document_id_filters(doc_ids):
def _document_id_filters(doc_ids):
"""Return a MetadataFilters IN filter scoped to ``doc_ids``."""
from llama_index.core.vector_stores.types import FilterOperator
from llama_index.core.vector_stores.types import MetadataFilter
@@ -396,23 +396,6 @@ def _exclude_document_id_filter(document_id: int | str):
)
def exclude_document_ids_filter(doc_ids):
"""Return a MetadataFilters NIN filter excluding every id in ``doc_ids``."""
from llama_index.core.vector_stores.types import FilterOperator
from llama_index.core.vector_stores.types import MetadataFilter
from llama_index.core.vector_stores.types import MetadataFilters
return MetadataFilters(
filters=[
MetadataFilter(
key="document_id",
operator=FilterOperator.NIN,
value=list(doc_ids),
),
],
)
def update_llm_index(
*,
iter_wrapper: IterWrapper[Document] = identity,
@@ -677,7 +660,7 @@ def retrieve_similar_nodes(
filter_parts = []
if allowed_document_ids is not None:
filter_parts.extend(document_id_filters(allowed_document_ids).filters)
filter_parts.extend(_document_id_filters(allowed_document_ids).filters)
if document.pk is not None:
filter_parts.extend(_exclude_document_id_filter(document.pk).filters)
-29
View File
@@ -154,35 +154,6 @@ class DocumentMetaTable:
}
class PermittedIdsTable:
"""Per-connection scratch space for an oversized IN-filter id list.
A literal ``IN (?,?,...)`` list binds one SQL parameter per id, capped by
SQLite's own SQLITE_MAX_VARIABLE_NUMBER (see _MAX_IN_VALUES in
vector_store.py). Loading the ids into a TEMP TABLE and filtering via a
subquery instead has no such limit. TEMP tables live in a
connection-private namespace -- never visible to another connection,
even under this identical name -- so this is safe under the vector
store's one-connection-per-request model without any extra locking or
per-call naming scheme.
"""
TABLE_NAME = "permitted_document_ids"
@staticmethod
def load(conn: sqlite3.Connection, ids: Iterable[int]) -> None:
"""Replace this connection's scratch table with ``ids``."""
conn.execute(f"DROP TABLE IF EXISTS temp.{PermittedIdsTable.TABLE_NAME}")
conn.execute(
f"CREATE TEMP TABLE {PermittedIdsTable.TABLE_NAME} "
"(id INTEGER PRIMARY KEY)",
)
conn.executemany(
f"INSERT INTO {PermittedIdsTable.TABLE_NAME} (id) VALUES (?)",
((i,) for i in ids),
)
class IndexMetaTable:
"""Typed accessors over index_meta's key/value rows -- replaces
PaperlessSqliteVecVectorStore._meta_get_on/_meta_set_on, which returned
+1 -1
View File
@@ -164,7 +164,7 @@ def test_get_ai_document_classification_failure(mock_run_llm_query, mock_documen
"""
mock_run_llm_query.side_effect = Exception("LLM query failed")
with pytest.raises(Exception):
with pytest.raises(Exception): # noqa: B017 - mock injects a bare Exception
get_ai_document_classification(mock_document)
+23 -114
View File
@@ -1,14 +1,9 @@
from __future__ import annotations
import json
from typing import TYPE_CHECKING
from typing import Any
from unittest.mock import MagicMock
from unittest.mock import patch
import pytest
from django.db.models.signals import post_init
from django.utils import timezone
from llama_index.core import settings as llama_settings
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from llama_index.core.schema import TextNode
@@ -23,11 +18,6 @@ from paperless_ai.chat import _build_chat_prompt
from paperless_ai.chat import _build_refine_prompt
from paperless_ai.chat import stream_chat_with_documents
if TYPE_CHECKING:
from pathlib import Path
import pytest_mock
@pytest.fixture(autouse=True)
def patch_embed_model():
@@ -320,40 +310,8 @@ def test_stream_chat_unexpected_failure_returns_generic_error(caplog) -> None:
assert "private provider detail" in caplog.text
def _retriever_filter_values(captured_filters: list[Any]) -> list[str]:
"""The value list of the single MetadataFilter the retriever received."""
assert captured_filters, "VectorIndexRetriever was never constructed"
filt = captured_filters[0]
assert filt is not None, "Retriever must receive a MetadataFilters"
return filt.filters[0].value
@pytest.mark.django_db
class TestStreamChatRetrieval:
@pytest.fixture
def captured_filters(self, mocker: pytest_mock.MockerFixture) -> list[Any]:
"""Stub out the AI client and the retriever, capturing the ``filters``
kwarg of every VectorIndexRetriever construction.
VectorIndexRetriever is imported inside _stream_chat_with_documents,
so it is patched at the llama_index source for the lazy import to
pick it up.
"""
captured: list[Any] = []
retriever = mocker.MagicMock()
retriever.retrieve.return_value = []
def capture_retriever(*args, **kwargs) -> pytest_mock.MockType:
captured.append(kwargs.get("filters"))
return retriever
mocker.patch("paperless_ai.chat.AIClient")
mocker.patch(
"llama_index.core.retrievers.VectorIndexRetriever",
side_effect=capture_retriever,
)
return captured
def test_no_nodes_yields_no_content_message(
self,
temp_llm_index_dir,
@@ -371,9 +329,9 @@ class TestStreamChatRetrieval:
def test_chat_filter_contains_only_requested_document_ids(
self,
temp_llm_index_dir: Path,
mock_embed_model: pytest_mock.MockType,
captured_filters: list[Any],
temp_llm_index_dir,
mock_embed_model,
mocker,
) -> None:
"""The MetadataFilter passed to the retriever must be scoped to the
requested documents only content from other indexed documents must
@@ -384,6 +342,22 @@ class TestStreamChatRetrieval:
indexing.llm_index_add_or_update_document(included)
indexing.llm_index_add_or_update_document(excluded)
# VectorIndexRetriever is imported inside _stream_chat_with_documents;
# patch it at the llama_index source so the lazy import picks it up.
captured_filters = []
mock_retriever = mocker.MagicMock()
mock_retriever.retrieve.return_value = []
def capture_retriever(*args, **kwargs):
captured_filters.append(kwargs.get("filters"))
return mock_retriever
mocker.patch("paperless_ai.chat.AIClient")
mocker.patch(
"llama_index.core.retrievers.VectorIndexRetriever",
side_effect=capture_retriever,
)
list(
chat.stream_chat_with_documents(
"question?",
@@ -391,78 +365,13 @@ class TestStreamChatRetrieval:
),
)
filter_values = _retriever_filter_values(captured_filters)
assert captured_filters, "VectorIndexRetriever was never constructed"
filt = captured_filters[0]
assert filt is not None, "Retriever must receive a MetadataFilters"
filter_values = filt.filters[0].value
assert str(included.pk) in filter_values
assert str(excluded.pk) not in filter_values
def test_unrestricted_chat_excludes_nothing_when_no_documents_are_trashed(
self,
temp_llm_index_dir: Path,
mock_embed_model: pytest_mock.MockType,
captured_filters: list[Any],
) -> None:
"""
GIVEN:
- A document indexed in the vector store, nothing trashed
WHEN:
- stream_chat_with_documents is called with unrestricted=True
THEN:
- The retriever receives a NOT IN filter excluding zero ids, so
the whole index is effectively searched -- and no IN-list is
built from the full permitted set, which is what risks the
vector store's safety limit on large installs
"""
document = DocumentFactory.create(content="indexed document content")
indexing.llm_index_add_or_update_document(document)
list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=document.pk),
unrestricted=True,
),
)
assert _retriever_filter_values(captured_filters) == []
def test_unrestricted_chat_excludes_trashed_documents(
self,
temp_llm_index_dir: Path,
mock_embed_model: pytest_mock.MockType,
captured_filters: list[Any],
) -> None:
"""
GIVEN:
- Two indexed documents, one of them trashed -- trashed documents
stay in the vector index until permanently deleted, since
delete_document_from_llm_index is wired to post_delete
WHEN:
- stream_chat_with_documents is called with unrestricted=True
THEN:
- The retriever receives a NOT IN filter excluding the trashed
document's id, so an unrestricted caller (e.g. a superuser)
never has trashed content surfaced in a chat answer
"""
kept = DocumentFactory.create(content="kept document content")
trashed = DocumentFactory.create(content="trashed document content")
indexing.llm_index_add_or_update_document(kept)
indexing.llm_index_add_or_update_document(trashed)
Document.global_objects.filter(pk=trashed.pk).update(
deleted_at=timezone.now(),
)
list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=kept.pk),
unrestricted=True,
),
)
filter_values = _retriever_filter_values(captured_filters)
assert str(trashed.pk) in filter_values
assert str(kept.pk) not in filter_values
@pytest.mark.django_db
def test_get_document_references_only_queries_referenced_documents(
self,
@@ -21,5 +21,6 @@ class TestLazyAiImports:
capture_output=True,
text=True,
cwd=_SRC_DIR,
check=False,
)
assert result.returncode == 0, result.stdout + result.stderr
-83
View File
@@ -9,7 +9,6 @@ from paperless_ai.tables import DocumentChunksTable
from paperless_ai.tables import DocumentMetaRow
from paperless_ai.tables import DocumentMetaTable
from paperless_ai.tables import IndexMetaTable
from paperless_ai.tables import PermittedIdsTable
@pytest.fixture
@@ -339,85 +338,3 @@ class TestIndexMetaTable:
IndexMetaTable.increment_total_inserts(conn, 100)
IndexMetaTable.reset_total_inserts(conn, 7)
assert IndexMetaTable.get_total_inserts(conn) == 7
class TestPermittedIdsTable:
def _loaded_ids(self, conn: sqlite3.Connection) -> list[int]:
return [
row["id"]
for row in conn.execute(
f"SELECT id FROM {PermittedIdsTable.TABLE_NAME} ORDER BY id",
)
]
def test_load_then_read_back_all_ids(self, conn: sqlite3.Connection) -> None:
"""
GIVEN:
- A bare sqlite3 connection
WHEN:
- load() is called with a set of ids
THEN:
- Every id is present in the TEMP TABLE, and only those ids
"""
PermittedIdsTable.load(conn, [3, 1, 2])
assert self._loaded_ids(conn) == [1, 2, 3]
def test_load_replaces_previous_contents(self, conn: sqlite3.Connection) -> None:
"""
GIVEN:
- A connection whose PermittedIdsTable already holds one id set
WHEN:
- load() is called again with a different id set
THEN:
- Only the new ids are present -- a connection reused across
multiple queries in one request never leaks a stale filter
"""
PermittedIdsTable.load(conn, [1, 2, 3])
PermittedIdsTable.load(conn, [4, 5])
assert self._loaded_ids(conn) == [4, 5]
def test_load_is_connection_private(self) -> None:
"""
GIVEN:
- Two separate connections
WHEN:
- Each loads PermittedIdsTable with a different id set, under
the identical TABLE_NAME
THEN:
- Each connection sees only its own ids -- TEMP TABLE is
connection-private, so concurrent requests never collide or
cross-contaminate despite sharing the same table name (the
vector store opens one connection per request; see
PaperlessSqliteVecVectorStore)
"""
conn_a = sqlite3.connect(":memory:")
conn_a.row_factory = sqlite3.Row
conn_b = sqlite3.connect(":memory:")
conn_b.row_factory = sqlite3.Row
try:
PermittedIdsTable.load(conn_a, [1, 2, 3])
PermittedIdsTable.load(conn_b, [4, 5, 6])
assert self._loaded_ids(conn_a) == [1, 2, 3]
assert self._loaded_ids(conn_b) == [4, 5, 6]
finally:
conn_a.close()
conn_b.close()
def test_load_handles_more_ids_than_a_bound_parameter_list_could(
self,
conn: sqlite3.Connection,
) -> None:
"""
GIVEN:
- An id count over SQLite's own bound-parameter limit
(SQLITE_MAX_VARIABLE_NUMBER, 32766 by default) -- more than a
literal IN(?,?,...) list could ever bind in one statement
WHEN:
- load() is called with that many ids
THEN:
- Every id is loaded without error, since executemany() binds
one row at a time rather than one statement with N parameters
"""
ids = list(range(40_000))
PermittedIdsTable.load(conn, ids)
assert self._loaded_ids(conn) == ids
+18 -153
View File
@@ -1,6 +1,5 @@
import inspect
import sqlite3
from collections.abc import Callable
from collections.abc import Generator
from pathlib import Path
@@ -18,7 +17,6 @@ from paperless_ai.migrations import Migration
from paperless_ai.migrations import m0001_v1_to_v2
from paperless_ai.tables import DocumentChunksTable
from paperless_ai.tables import DocumentMetaTable
from paperless_ai.tables import PermittedIdsTable
from paperless_ai.vector_store import _MAX_IN_VALUES
from paperless_ai.vector_store import DB_FILENAME
from paperless_ai.vector_store import DEFAULT_TABLE_NAME
@@ -99,18 +97,6 @@ def _ne_filter(document_id: int):
)
def _nin_filter(document_ids: list[int]):
return MetadataFilters(
filters=[
MetadataFilter(
key="document_id",
operator=FilterOperator.NIN,
value=document_ids,
),
],
)
class TestCrud:
def test_add_then_query_returns_node(self, store) -> None:
node = make_node("n1", 1)
@@ -281,23 +267,8 @@ class TestCrud:
class TestBuildWhere:
@pytest.fixture
def conn(self) -> Generator[sqlite3.Connection, None, None]:
"""A bare connection, sufficient for _build_where(): it only ever
touches the connection via PermittedIdsTable, which needs no vec0
extension loaded.
"""
connection = sqlite3.connect(":memory:")
try:
yield connection
finally:
connection.close()
def test_ne_filter_translates_to_not_equal_clause(
self,
conn: sqlite3.Connection,
) -> None:
where, params = _build_where(conn, _ne_filter(1))
def test_ne_filter_translates_to_not_equal_clause(self) -> None:
where, params = _build_where(_ne_filter(1))
assert where == "(document_id != ?)"
assert params == [1]
@@ -309,60 +280,7 @@ class TestBuildWhere:
"b1",
]
def test_nin_filter_translates_to_not_in_clause(
self,
conn: sqlite3.Connection,
) -> None:
where, params = _build_where(conn, _nin_filter([1, 2]))
assert where == "(document_id NOT IN (?,?))"
assert params == [1, 2]
def test_query_with_nin_filter_excludes_matching_documents(self, store) -> None:
store.add([make_node("a1", 1), make_node("b1", 2), make_node("c1", 3)])
assert sorted(
_query(store, [0.0] * DIM, top_k=5, filters=_nin_filter([1, 2])).ids,
) == ["c1"]
def test_empty_in_filter_excludes_everything(
self,
conn: sqlite3.Connection,
) -> None:
"""
GIVEN:
- An IN filter with an empty value list
WHEN:
- _build_where() translates it to SQL
THEN:
- It excludes everything (the opposite of an empty NOT IN
filter) -- an empty inclusion list must never widen results
"""
where, params = _build_where(conn, _in_filter([]))
assert where == "(1 = 0)"
assert params == []
def test_empty_nin_filter_excludes_nothing(
self,
conn: sqlite3.Connection,
) -> None:
"""
GIVEN:
- A NOT IN filter with an empty value list -- e.g. an
unrestricted chat caller when nothing is currently trashed
WHEN:
- _build_where() translates it to SQL
THEN:
- It excludes nothing (unlike an empty IN filter, which
excludes everything) -- an empty exclusion list must never
narrow results
"""
where, params = _build_where(conn, _nin_filter([]))
assert where == "(1 = 1)"
assert params == []
def test_fails_closed_when_no_filter_is_translatable(
self,
conn: sqlite3.Connection,
) -> None:
def test_fails_closed_when_no_filter_is_translatable(self) -> None:
# A nested MetadataFilters is not a MetadataFilter, so it is skipped.
# With no translatable clauses, the function must fail closed rather
# than emit "()" (invalid SQL) and never widen document access.
@@ -375,88 +293,35 @@ class TestBuildWhere:
),
],
)
where, params = _build_where(conn, MetadataFilters(filters=[nested]))
where, params = _build_where(MetadataFilters(filters=[nested]))
assert where == "1 = 0"
assert params == []
@pytest.mark.parametrize(
("build_filter", "sql_op"),
[(_in_filter, "IN"), (_nin_filter, "NOT IN")],
ids=["in", "nin"],
)
def test_filter_over_max_values_uses_permitted_ids_table(
def test_fails_closed_when_in_filter_exceeds_max_values(
self,
conn: sqlite3.Connection,
build_filter: Callable[[list[str]], MetadataFilters],
sql_op: str,
caplog: pytest.LogCaptureFixture,
) -> None:
"""
GIVEN:
- An IN or NOT IN filter with more values than _MAX_IN_VALUES
(SQLite's own bound-parameter limit is 32766; this threshold
sits below that with headroom for the query's other bound
parameters)
- An IN filter with more values than _MAX_IN_VALUES (SQLite's
own bound-parameter limit is 32766; this guard sits below
that with headroom for the query's other bound parameters)
WHEN:
- _build_where() translates it to SQL
THEN:
- It builds a subquery against PermittedIdsTable's TEMP TABLE,
loaded with every id, instead of a literal list SQLite would
reject past its own limit -- true for NOT IN too (e.g. an
install with an enormous trash), not just IN
- It fails closed ("1 = 0", no params) instead of building an
IN clause SQLite would reject, and logs a warning -- this
filter scopes document access, so refusing to build it must
never widen the scope to "everything" by accident
"""
ids = list(range(_MAX_IN_VALUES + 1))
oversized = build_filter([str(i) for i in ids])
oversized = _in_filter([str(i) for i in range(_MAX_IN_VALUES + 1)])
where, params = _build_where(conn, oversized)
with caplog.at_level("WARNING"):
where, params = _build_where(oversized)
assert where == (
f"(document_id {sql_op} (SELECT id FROM {PermittedIdsTable.TABLE_NAME}))"
)
assert where == "(1 = 0)"
assert params == []
loaded = [
row[0]
for row in conn.execute(
f"SELECT id FROM {PermittedIdsTable.TABLE_NAME} ORDER BY id",
)
]
assert loaded == ids
@pytest.mark.parametrize(
("build_filter", "expected_ids"),
[(_in_filter, ["b1", "c1"]), (_nin_filter, ["a1"])],
ids=["in", "nin"],
)
def test_query_and_get_nodes_scope_correctly_when_filter_exceeds_max_values(
self,
store: PaperlessSqliteVecVectorStore,
mocker: MockerFixture,
build_filter: Callable[[list[int]], MetadataFilters],
expected_ids: list[str],
) -> None:
"""
GIVEN:
- _MAX_IN_VALUES lowered so a small IN/NOT IN filter exceeds it
WHEN:
- query() and get_nodes() are called with that filter
THEN:
- Both still correctly scope results -- the PermittedIdsTable
temp-table path behaves identically to the literal
IN(...)/NOT IN(...) path it replaces above the threshold
"""
mocker.patch("paperless_ai.vector_store._MAX_IN_VALUES", 1)
store.add(
[
make_node("a1", 1, seed=0.0),
make_node("b1", 2, seed=1.0),
make_node("c1", 3, seed=2.0),
],
)
result = _query(store, [0.0] * DIM, top_k=10, filters=build_filter([2, 3]))
nodes = store.get_nodes(filters=build_filter([2, 3]))
assert sorted(result.ids) == expected_ids
assert sorted(n.node_id for n in nodes) == expected_ids
assert "document_id" in caplog.text
def test_query_with_untranslatable_filter_returns_no_rows(
self,
+34 -41
View File
@@ -30,7 +30,6 @@ from paperless_ai.tables import DocumentChunksTable
from paperless_ai.tables import DocumentMetaRow
from paperless_ai.tables import DocumentMetaTable
from paperless_ai.tables import IndexMetaTable
from paperless_ai.tables import PermittedIdsTable
logger = logging.getLogger("paperless_ai.vector_store")
@@ -76,12 +75,14 @@ class _Row(NamedTuple):
embedding: bytes
# _build_where(): the largest IN value list translated into a literal
# IN (?,?,...) clause. SQLite's own hard limit (SQLITE_MAX_VARIABLE_NUMBER)
# is 32766 by default; this leaves headroom below that for the query's other
# bound parameters (the embedding blob, k, and any NE clause) and for the
# limit itself to move. Above this threshold _build_where() switches to
# PermittedIdsTable instead of failing closed -- see its docstring.
# _build_where(): the largest IN value list translated into bound SQL
# parameters. SQLite's own hard limit (SQLITE_MAX_VARIABLE_NUMBER) is 32766
# by default; this leaves headroom below that for the query's other bound
# parameters (the embedding blob, k, and any NE clause) and for the limit
# itself to move. An IN filter this large should not happen in practice --
# callers are expected to pass None (no filter) rather than every id when
# the filter would not actually narrow anything -- so this is a guard
# against a future regression, not a normal code path.
_MAX_IN_VALUES = 32700
@@ -105,21 +106,13 @@ def _vec0_params(rows: list[_Row]) -> list[tuple[str, int, str, bytes]]:
return [(r.chunk_id, r.document_id, r.node_content, r.embedding) for r in rows]
def _build_where(
conn: sqlite3.Connection,
filters: MetadataFilters | None,
) -> tuple[str, list[int]]:
"""Translate the EQ / IN / NIN / NE filters we use into a parameterized
SQL clause on vec0 metadata columns. Returns ("", []) when there is
nothing to filter. document_id is vec0's only filterable column and is
INTEGER; every value is coerced via int() here so callers (which today
still pass strings in places, e.g. indexing.py's MetadataFilter
construction) don't have to be individually correct -- vec0 doesn't
coerce types itself.
``conn`` is only used for an IN/NOT IN filter over _MAX_IN_VALUES: it
loads the ids into PermittedIdsTable's TEMP TABLE on that connection
rather than binding them as SQL parameters.
def _build_where(filters: MetadataFilters | None) -> tuple[str, list[int]]:
"""Translate the EQ / IN / NE filters we use into a parameterized SQL
clause on vec0 metadata columns. Returns ("", []) when there is nothing
to filter. document_id is vec0's only filterable column and is INTEGER;
every value is coerced via int() here so callers (which today still pass
strings in places, e.g. indexing.py's MetadataFilter construction) don't
have to be individually correct -- vec0 doesn't coerce types itself.
"""
if filters is None or not filters.filters:
return "", []
@@ -132,28 +125,28 @@ def _build_where(
continue
if f.key not in _FILTER_COLUMNS: # pragma: no cover - we build the keys
raise NotImplementedError(f"Unsupported filter column: {f.key}")
if f.operator in (FilterOperator.IN, FilterOperator.NIN):
is_in = f.operator == FilterOperator.IN
sql_op = "IN" if is_in else "NOT IN"
if f.operator == FilterOperator.IN:
values = [int(v) for v in f.value] # type: ignore[union-attr]
if not values:
# An empty IN list matches nothing; an empty NOT IN list
# excludes nothing, so it matches everything.
clauses.append("1 = 0" if is_in else "1 = 1")
if not values: # pragma: no cover
clauses.append("1 = 0")
continue
if len(values) > _MAX_IN_VALUES:
# A literal list this large would exceed SQLite's own
# bound-parameter limit. Load the ids into a TEMP TABLE on
# this connection instead and filter via subquery, which has
# no such limit -- see PermittedIdsTable. Applies to NOT IN
# too (e.g. an install with an enormous trash), not just IN.
PermittedIdsTable.load(conn, values)
clauses.append(
f"{f.key} {sql_op} (SELECT id FROM {PermittedIdsTable.TABLE_NAME})",
# Fail closed (see the empty-clauses case below) rather than
# let SQLite raise "too many SQL variables" past its own
# limit: this filter scopes document access, so an IN list
# too large to safely bind must match no rows, never widen
# the scope to "everything" by accident.
logger.warning(
"Refusing to build an IN filter on %r with %d values "
"(over the %d-value safety limit); returning no rows.",
f.key,
len(values),
_MAX_IN_VALUES,
)
clauses.append("1 = 0")
continue
placeholders = ",".join("?" for _ in values)
clauses.append(f"{f.key} {sql_op} ({placeholders})")
clauses.append(f"{f.key} IN ({placeholders})")
params.extend(values)
elif f.operator == FilterOperator.EQ:
clauses.append(f"{f.key} = ?")
@@ -161,7 +154,7 @@ def _build_where(
elif f.operator == FilterOperator.NE:
clauses.append(f"{f.key} != ?")
params.append(int(f.value))
else: # pragma: no cover - we only ever build EQ/IN/NIN/NE filters
else: # pragma: no cover - we only ever build EQ/IN/NE filters
raise NotImplementedError(f"Unsupported filter operator: {f.operator}")
if not clauses:
# Filters were requested but none could be translated. Fail closed
@@ -489,7 +482,7 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
)
if not self.table_exists():
return []
where, params = _build_where(self._conn, filters)
where, params = _build_where(filters)
sql = "SELECT node_content, embedding FROM " + DEFAULT_TABLE_NAME
if where:
sql += " WHERE " + where
@@ -505,7 +498,7 @@ class PaperlessSqliteVecVectorStore(BasePydanticVectorStore):
if query.query_embedding is None: # pragma: no cover
return VectorStoreQueryResult(nodes=[], similarities=[], ids=[])
top_k = query.similarity_top_k if query.similarity_top_k is not None else 10
where, params = _build_where(self._conn, query.filters)
where, params = _build_where(query.filters)
sql = (
"SELECT id, node_content, embedding, distance FROM "
+ DEFAULT_TABLE_NAME
+7 -8
View File
@@ -7,7 +7,6 @@ import ssl
import tempfile
import traceback
import unicodedata
from datetime import date
from datetime import timedelta
from fnmatch import fnmatch
from pathlib import Path
@@ -406,7 +405,7 @@ def make_criterias(rule: MailRule, *, supports_gmail_labels: bool):
Returns criteria to be applied to MailBox.fetch for the given rule.
"""
maximum_age = date.today() - timedelta(days=rule.maximum_age)
maximum_age = timezone.localdate() - timedelta(days=rule.maximum_age)
criterias = {}
if rule.maximum_age > 0:
criterias["date_gte"] = maximum_age
@@ -723,9 +722,9 @@ class MailAccountHandler(LoggingMixin):
f"Rule {rule}: Stopping processing rules due to stop_processing flag",
)
break
except Exception as e:
except Exception:
self.log.exception(
f"Rule {rule}: Error while processing rule: {e}",
f"Rule {rule}: Error while processing rule",
)
except MailError:
raise
@@ -767,8 +766,8 @@ class MailAccountHandler(LoggingMixin):
self.log.info(f"Located folder: {folder_info.name}")
except Exception as e:
self.log.error(
"Exception during folder listing, unable to provide list folders: "
+ str(e),
"Exception during folder listing, unable to provide list folders: %s",
str(e),
)
raise MailError(
@@ -874,9 +873,9 @@ class MailAccountHandler(LoggingMixin):
total_processed_files += processed_files
mails_processed += 1
except Exception as e:
except Exception:
self.log.exception(
f"Rule {rule}: Error while processing mail {message.uid}: {e}",
f"Rule {rule}: Error while processing mail {message.uid}",
)
self.log.debug(f"Rule {rule}: Processed {mails_processed} matching mail(s)")
+5 -1
View File
@@ -11,6 +11,10 @@ from imap_tools import MailMessage
from documents.loggers import LoggingMixin
class MailDecryptionError(Exception):
pass
class MailMessagePreprocessor(abc.ABC):
"""
Defines the interface for preprocessors that alter messages before they are handled in MailAccountHandler
@@ -69,7 +73,7 @@ class MailMessageDecryptor(MailMessagePreprocessor, LoggingMixin):
f"Message decryption failed with status message "
f"{decrypted_raw_message.status}",
)
raise Exception(
raise MailDecryptionError(
f"Decryption failed: {decrypted_raw_message.status}, {decrypted_raw_message.stderr}",
)
self.log.debug("Message decrypted successfully.")
+1 -1
View File
@@ -50,7 +50,7 @@ class ProcessedMailFactory(DjangoModelFactory[ProcessedMail]):
rule = factory.SubFactory(MailRuleFactory)
folder = "INBOX"
uid = factory.Sequence(lambda n: str(n))
uid = factory.Sequence(str)
subject = factory.Faker("sentence", nb_words=4)
received = factory.LazyFunction(timezone.now)
processed = factory.LazyFunction(timezone.now)
+1 -1
View File
@@ -214,7 +214,7 @@ class BogusMailBox(AbstractContextManager):
)
self.messages = list(filter(lambda m: m.uid not in uid_list, self.messages))
else:
raise Exception
raise Exception # noqa: TRY002 - test double simulating a generic mailbox failure
def fake_magic_from_buffer(buffer, *, mime=False):
@@ -14,6 +14,7 @@ from imap_tools import MailMessage
from paperless_mail.mail import MailAccountHandler
from paperless_mail.models import MailRule
from paperless_mail.preprocessor import MailDecryptionError
from paperless_mail.preprocessor import MailMessageDecryptor
from paperless_mail.tests.factories import MailAccountFactory
from paperless_mail.tests.test_mail import TestMail
@@ -82,7 +83,9 @@ class MessageEncryptor:
armor=True,
)
if not encrypted_data.ok:
raise Exception(f"Encryption failed: {encrypted_data.stderr}")
raise Exception( # noqa: TRY002 - test fixture setup, not production code
f"Encryption failed: {encrypted_data.stderr}",
)
encrypted_email_content = encrypted_data.data
new_email = MIMEMultipart("encrypted", protocol="application/pgp-encrypted")
@@ -184,7 +187,11 @@ class TestMailMessageGpgDecryptor(TestMail):
EMAIL_GNUPG_HOME=empty_gpg_home,
):
message_decryptor = MailMessageDecryptor()
self.assertRaises(Exception, message_decryptor.run, encrypted_message)
self.assertRaises(
MailDecryptionError,
message_decryptor.run,
encrypted_message,
)
finally:
# Clean up the temporary GPG home used only by this test
try:
+1 -2
View File
@@ -1,4 +1,3 @@
import datetime
import logging
from datetime import timedelta
from http import HTTPStatus
@@ -87,7 +86,7 @@ class MailAccountViewSet(PassUserMixin, ModelViewSet[MailAccount]):
@action(methods=["post"], detail=False)
def test(self, request):
logger = logging.getLogger("paperless_mail")
request.data["name"] = datetime.datetime.now().isoformat()
request.data["name"] = timezone.now().isoformat()
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
existing_account = None