Reduces the complexity of this loop now that we know it is hashable

Improves the test to actually check the behaviour of the cache, rather than bypassing the serializer
Test: fold the DocumentMetadataOverrides batching test into test_api_custom_fields.py instead of a new file
2026-09-13 13:17:59 +00:00 · 2026-09-10 14:51:30 -07:00 · 2026-09-10 14:23:17 -07:00 · 2026-09-10 13:10:10 -07:00 · 2026-09-10 13:10:10 -07:00 · 2026-09-10 13:10:10 -07:00
18 changed files with 432 additions and 597 deletions
@@ -1209,14 +1209,6 @@ left unassigned, preventing low-confidence guesses from being applied.

    Defaults to 0.6.

-#### [`PAPERLESS_MATCH_REGEX_TIMEOUT_SECONDS=<float>`](#PAPERLESS_MATCH_REGEX_TIMEOUT_SECONDS) {#PAPERLESS_MATCH_REGEX_TIMEOUT_SECONDS}
-
-: Sets the timeout, in seconds, for regular expression matching. Increase this
-value if date parsing or user-defined matching rules time out when processing
-long documents, especially on slower hardware.
-
-    Defaults to 0.1 seconds.
-
 #### [`PAPERLESS_DATE_PARSER_LANGUAGES=<lang>`](#PAPERLESS_DATE_PARSER_LANGUAGES) {#PAPERLESS_DATE_PARSER_LANGUAGES}

 : Specifies which language Paperless should use when parsing dates from documents.
@@ -1345,7 +1345,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1865</context>
+          <context context-type="linenumber">1808</context>
        </context-group>
      </trans-unit>
      <trans-unit id="1577733187050997705" datatype="html">
@@ -2423,7 +2423,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">690</context>
+          <context context-type="linenumber">673</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-version-dropdown/document-version-dropdown.component.html</context>
@@ -3330,11 +3330,11 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1479</context>
+          <context context-type="linenumber">1422</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1866</context>
+          <context context-type="linenumber">1809</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -3935,7 +3935,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1432</context>
+          <context context-type="linenumber">1375</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -4079,7 +4079,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1919</context>
+          <context context-type="linenumber">1862</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6661109599266152398" datatype="html">
@@ -4090,7 +4090,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1920</context>
+          <context context-type="linenumber">1863</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5162686434580248853" datatype="html">
@@ -4101,7 +4101,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1921</context>
+          <context context-type="linenumber">1864</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6665634854532231106" datatype="html">
@@ -6238,7 +6238,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1436</context>
+          <context context-type="linenumber">1379</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -8662,203 +8662,152 @@
        <source>Error retrieving metadata</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">442</context>
+          <context context-type="linenumber">440</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2218903673684131427" datatype="html">
        <source>An error occurred loading content: <x id="PH" equiv-text="err.message ?? err.toString()"/></source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">545,547</context>
+          <context context-type="linenumber">542,544</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1024,1026</context>
+          <context context-type="linenumber">1000,1002</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6357361810318120957" datatype="html">
        <source>Document was updated</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">685</context>
+          <context context-type="linenumber">668</context>
        </context-group>
      </trans-unit>
      <trans-unit id="5154064822428631306" datatype="html">
        <source>Document was updated at <x id="PH" equiv-text="formattedModified"/>.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">686</context>
+          <context context-type="linenumber">669</context>
        </context-group>
      </trans-unit>
      <trans-unit id="8462497568316256794" datatype="html">
        <source>Reload to discard your local unsaved edits and load the latest remote version.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">687</context>
+          <context context-type="linenumber">670</context>
        </context-group>
      </trans-unit>
      <trans-unit id="7967484035994732534" datatype="html">
        <source>Reload</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">689</context>
+          <context context-type="linenumber">672</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2907037627372942104" datatype="html">
        <source>Document reloaded with latest changes.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">745</context>
+          <context context-type="linenumber">728</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6435639868943916539" datatype="html">
        <source>Document reloaded.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">756</context>
+          <context context-type="linenumber">739</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6142395741265832184" datatype="html">
        <source>Next document</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">858</context>
+          <context context-type="linenumber">841</context>
        </context-group>
      </trans-unit>
      <trans-unit id="651985345816518480" datatype="html">
        <source>Previous document</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">868</context>
+          <context context-type="linenumber">851</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2885986061416655600" datatype="html">
        <source>Close document</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">876</context>
+          <context context-type="linenumber">859</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
-          <context context-type="linenumber">151</context>
+          <context context-type="linenumber">143</context>
        </context-group>
      </trans-unit>
      <trans-unit id="8229691481345614469" datatype="html">
        <source>Save document</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">883</context>
+          <context context-type="linenumber">866</context>
        </context-group>
      </trans-unit>
      <trans-unit id="1784543155727940353" datatype="html">
        <source>Save and close / next</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">892</context>
+          <context context-type="linenumber">875</context>
        </context-group>
      </trans-unit>
      <trans-unit id="7427704425579737895" datatype="html">
        <source>Error retrieving version content</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1006</context>
-        </context-group>
-      </trans-unit>
-      <trans-unit id="159901853873315050" datatype="html">
-        <source>Unsaved Changes</source>
-        <context-group purpose="location">
-          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1050</context>
-        </context-group>
-        <context-group purpose="location">
-          <context context-type="sourcefile">src/app/guards/dirty-form.guard.ts</context>
-          <context context-type="linenumber">15</context>
-        </context-group>
-        <context-group purpose="location">
-          <context context-type="sourcefile">src/app/guards/dirty-saved-view.guard.ts</context>
-          <context context-type="linenumber">27</context>
-        </context-group>
-        <context-group purpose="location">
-          <context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
-          <context context-type="linenumber">143</context>
-        </context-group>
-        <context-group purpose="location">
-          <context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
-          <context context-type="linenumber">170</context>
-        </context-group>
-      </trans-unit>
-      <trans-unit id="5905048729613762185" datatype="html">
-        <source>You have unsaved changes to the content of this version.</source>
-        <context-group purpose="location">
-          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1051</context>
-        </context-group>
-      </trans-unit>
-      <trans-unit id="85184271222513014" datatype="html">
-        <source>Switching versions will discard them.</source>
-        <context-group purpose="location">
-          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1052</context>
-        </context-group>
-      </trans-unit>
-      <trans-unit id="2565707334844767610" datatype="html">
-        <source>Discard and switch</source>
-        <context-group purpose="location">
-          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1054</context>
-        </context-group>
-      </trans-unit>
-      <trans-unit id="2109314380040637387" datatype="html">
-        <source>Save and switch</source>
-        <context-group purpose="location">
-          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1056</context>
+          <context context-type="linenumber">983</context>
        </context-group>
      </trans-unit>
      <trans-unit id="3456881259945295697" datatype="html">
        <source>Error retrieving suggestions.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1099</context>
+          <context context-type="linenumber">1043</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2194092841814123758" datatype="html">
        <source>Document &quot;<x id="PH" equiv-text="newValues.title"/>&quot; saved successfully.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1311</context>
+          <context context-type="linenumber">1255</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1339</context>
+          <context context-type="linenumber">1282</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6626387786259219838" datatype="html">
        <source>Error saving document &quot;<x id="PH" equiv-text="this.document().title"/>&quot;</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1345</context>
+          <context context-type="linenumber">1288</context>
        </context-group>
      </trans-unit>
      <trans-unit id="448882439049417053" datatype="html">
        <source>Error saving document</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1400</context>
+          <context context-type="linenumber">1343</context>
        </context-group>
      </trans-unit>
      <trans-unit id="8410796510716511826" datatype="html">
        <source>Do you really want to move the document &quot;<x id="PH" equiv-text="this.document().title"/>&quot; to the trash?</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1433</context>
+          <context context-type="linenumber">1376</context>
        </context-group>
      </trans-unit>
      <trans-unit id="282586936710748252" datatype="html">
        <source>Documents can be restored prior to permanent deletion.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1434</context>
+          <context context-type="linenumber">1377</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -8869,14 +8818,14 @@
        <source>Error deleting document</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1455</context>
+          <context context-type="linenumber">1398</context>
        </context-group>
      </trans-unit>
      <trans-unit id="619486176823357521" datatype="html">
        <source>Reprocess confirm</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1475</context>
+          <context context-type="linenumber">1418</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.ts</context>
@@ -8887,102 +8836,102 @@
        <source>This operation will permanently recreate the archive file for this document.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1476</context>
+          <context context-type="linenumber">1419</context>
        </context-group>
      </trans-unit>
      <trans-unit id="302054111564709516" datatype="html">
        <source>The archive file will be re-generated with the current settings.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1477</context>
+          <context context-type="linenumber">1420</context>
        </context-group>
      </trans-unit>
      <trans-unit id="4700389117298802932" datatype="html">
        <source>Reprocess operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1490</context>
+          <context context-type="linenumber">1433</context>
        </context-group>
      </trans-unit>
      <trans-unit id="4409560272830824468" datatype="html">
        <source>Error executing operation</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1501</context>
+          <context context-type="linenumber">1444</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6030453331794586802" datatype="html">
        <source>Error downloading document</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1565</context>
+          <context context-type="linenumber">1508</context>
        </context-group>
      </trans-unit>
      <trans-unit id="4458954481601077369" datatype="html">
        <source>Page Fit</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1643</context>
+          <context context-type="linenumber">1586</context>
        </context-group>
      </trans-unit>
      <trans-unit id="4663705961777238777" datatype="html">
        <source>PDF edit operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1886</context>
+          <context context-type="linenumber">1829</context>
        </context-group>
      </trans-unit>
      <trans-unit id="9043972994040261999" datatype="html">
        <source>Error executing PDF edit operation</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1898</context>
+          <context context-type="linenumber">1841</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6172690334763056188" datatype="html">
        <source>Please enter the current password before attempting to remove it.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1909</context>
+          <context context-type="linenumber">1852</context>
        </context-group>
      </trans-unit>
      <trans-unit id="968660764814228922" datatype="html">
        <source>Password removal operation for &quot;<x id="PH" equiv-text="this.document().title"/>&quot; will begin in the background.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1943</context>
+          <context context-type="linenumber">1886</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2282118435712883014" datatype="html">
        <source>Error executing password removal operation</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">1957</context>
+          <context context-type="linenumber">1900</context>
        </context-group>
      </trans-unit>
      <trans-unit id="3740891324955700797" datatype="html">
        <source>Print failed.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">2007</context>
+          <context context-type="linenumber">1950</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6457245677384603573" datatype="html">
        <source>Error loading document for printing.</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">2016</context>
+          <context context-type="linenumber">1959</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6085793215710522488" datatype="html">
        <source>An error occurred loading tiff: <x id="PH" equiv-text="err.toString()"/></source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">2099</context>
+          <context context-type="linenumber">2042</context>
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/components/document-detail/document-detail.component.ts</context>
-          <context context-type="linenumber">2105</context>
+          <context context-type="linenumber">2048</context>
        </context-group>
      </trans-unit>
      <trans-unit id="4958946940233632319" datatype="html">
@@ -12341,6 +12290,25 @@
          <context context-type="linenumber">16</context>
        </context-group>
      </trans-unit>
+      <trans-unit id="159901853873315050" datatype="html">
+        <source>Unsaved Changes</source>
+        <context-group purpose="location">
+          <context context-type="sourcefile">src/app/guards/dirty-form.guard.ts</context>
+          <context context-type="linenumber">15</context>
+        </context-group>
+        <context-group purpose="location">
+          <context context-type="sourcefile">src/app/guards/dirty-saved-view.guard.ts</context>
+          <context context-type="linenumber">27</context>
+        </context-group>
+        <context-group purpose="location">
+          <context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
+          <context context-type="linenumber">135</context>
+        </context-group>
+        <context-group purpose="location">
+          <context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
+          <context context-type="linenumber">162</context>
+        </context-group>
+      </trans-unit>
      <trans-unit id="2573823578527613511" datatype="html">
        <source>You have unsaved changes.</source>
        <context-group purpose="location">
@@ -12349,7 +12317,7 @@
        </context-group>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
-          <context context-type="linenumber">171</context>
+          <context context-type="linenumber">163</context>
        </context-group>
      </trans-unit>
      <trans-unit id="3305084982600522070" datatype="html">
@@ -12489,28 +12457,28 @@
        <source>You have unsaved changes to the document</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
-          <context context-type="linenumber">145</context>
+          <context context-type="linenumber">137</context>
        </context-group>
      </trans-unit>
      <trans-unit id="2089045849587358256" datatype="html">
        <source>Are you sure you want to close this document?</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
-          <context context-type="linenumber">149</context>
+          <context context-type="linenumber">141</context>
        </context-group>
      </trans-unit>
      <trans-unit id="6755718693176327396" datatype="html">
        <source>Are you sure you want to close all documents?</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
-          <context context-type="linenumber">172</context>
+          <context context-type="linenumber">164</context>
        </context-group>
      </trans-unit>
      <trans-unit id="4215561719980781894" datatype="html">
        <source>Close documents</source>
        <context-group purpose="location">
          <context context-type="sourcefile">src/app/services/open-documents.service.ts</context>
-          <context context-type="linenumber">174</context>
+          <context context-type="linenumber">166</context>
        </context-group>
      </trans-unit>
      <trans-unit id="1206520795340730278" datatype="html">
@@ -28,9 +28,8 @@ import { Subject, of, throwError } from 'rxjs'
 import { routes } from 'src/app/app-routing.module'
 import { Correspondent } from 'src/app/data/correspondent'
 import { CustomFieldDataType } from 'src/app/data/custom-field'
-import { CustomFieldInstance } from 'src/app/data/custom-field-instance'
 import { DataType } from 'src/app/data/datatype'
-import { Document, DocumentVersionInfo } from 'src/app/data/document'
+import { Document } from 'src/app/data/document'
 import { DocumentType } from 'src/app/data/document-type'
 import {
  FILTER_CORRESPONDENT,
@@ -101,18 +100,13 @@ const doc: Document = {
  custom_fields: [
    {
      field: 0,
+      document: 3,
+      created: new Date(),
      value: 'custom foo bar',
    },
-  ] as CustomFieldInstance[],
+  ],
 }

-// Newest first, as the API returns them: 12 is the latest, 3 is the root
-const docVersions: DocumentVersionInfo[] = [
-  { id: 12, is_root: false },
-  { id: 10, is_root: false },
-  { id: doc.id, is_root: true },
-]
-
 const customFields = [
  {
    id: 0,
@@ -2051,208 +2045,6 @@ describe('DocumentDetailComponent', () => {
    expect(saveSpy).toHaveBeenCalled()
  })

-  it('selectVersion should use the version content as the baseline and ignore stale responses', () => {
-    initNormally()
-    const version10Content = new Subject<Document>()
-    jest
-      .spyOn(documentService, 'get')
-      .mockReturnValueOnce(version10Content)
-      .mockReturnValueOnce(of({ content: 'version 12 content' } as Document))
-    const version10Metadata = new Subject<any>()
-    jest
-      .spyOn(documentService, 'getMetadata')
-      .mockReturnValueOnce(version10Metadata)
-      .mockReturnValueOnce(of({ lang: 'de' }))
-
-    component.selectVersion(10)
-    component.selectVersion(12)
-    version10Content.next({ content: 'version 10 content' } as Document)
-    version10Metadata.next({ lang: 'en' })
-
-    expect(component.documentForm.get('content').value).toEqual(
-      'version 12 content'
-    )
-    expect(component.store.value.content).toEqual('version 12 content')
-    expect(component.metadata().lang).toEqual('de')
-    expect(
-      httpTestingController.expectOne(component.previewUrl()).cancelled
-    ).toBeFalsy()
-    expect(
-      httpTestingController.match((req) => req.url.includes('version=10'))[0]
-        ?.cancelled
-    ).toBeTruthy()
-  })
-
-  it('should confirm before discarding unsaved content edits when switching versions', () => {
-    initNormally()
-    component.document().versions = docVersions
-    jest
-      .spyOn(documentService, 'get')
-      .mockImplementation((id, versionID) =>
-        of({ content: `version ${versionID} content` } as Document)
-      )
-    let openModal: NgbModalRef
-    modalService.activeInstances.subscribe((modals) => (openModal = modals[0]))
-    const modalSpy = jest.spyOn(modalService, 'open')
-
-    // shared fields carry over between versions, so no confirmation
-    component.documentForm.get('title').setValue('Edited title')
-    component.documentForm.get('title').markAsDirty()
-    component.documentForm.get('content').markAsDirty()
-    component.onVersionSelected(12)
-    expect(modalSpy).not.toHaveBeenCalled()
-    expect(component.selectedVersionId()).toEqual(12)
-
-    component.documentForm.get('content').setValue('edited content')
-    component.documentForm.get('content').markAsDirty()
-    component.onVersionSelected(12) // already selected, nothing to do
-    expect(modalSpy).not.toHaveBeenCalled()
-    component.onVersionSelected(10)
-    expect(modalSpy).toHaveBeenCalledWith(
-      ConfirmDialogComponent,
-      expect.anything()
-    )
-    openModal.componentInstance.cancel()
-    expect(component.selectedVersionId()).toEqual(12)
-    expect(component.documentForm.get('content').value).toEqual(
-      'edited content'
-    )
-
-    component.onVersionSelected(10)
-    openModal.componentInstance.confirmClicked.emit()
-    expect(component.selectedVersionId()).toEqual(10)
-    expect(component.documentForm.get('content').value).toEqual(
-      'version 10 content'
-    )
-    expect(component.documentForm.get('content').dirty).toBeFalsy()
-    expect(component.documentForm.get('title').value).toEqual('Edited title')
-  })
-
-  it('should save unsaved content edits to the current version before switching, and stay if that fails', () => {
-    initNormally()
-    component.document().versions = docVersions
-    component.selectedVersionId.set(12)
-    jest
-      .spyOn(documentService, 'get')
-      .mockReturnValue(of({ content: 'version 10 content' } as Document))
-    const savedDoc = new Subject<Document>()
-    const patchSpy = jest
-      .spyOn(documentService, 'patch')
-      .mockReturnValueOnce(throwError(() => new Error('failed to save')))
-      .mockReturnValueOnce(savedDoc)
-    const modalSpy = jest.spyOn(modalService, 'open')
-    component.documentForm.get('content').setValue('edited content')
-    component.documentForm.get('content').markAsDirty()
-
-    component.onVersionSelected(10)
-    let modal: NgbModalRef = modalSpy.mock.results[0].value
-    const closeSpy = jest.spyOn(modal, 'close')
-    modal.componentInstance.alternativeClicked.emit()
-    expect(closeSpy).toHaveBeenCalled()
-    expect(component.selectedVersionId()).toEqual(12)
-    expect(component.documentForm.get('content').value).toEqual(
-      'edited content'
-    )
-
-    component.onVersionSelected(10)
-    modal = modalSpy.mock.results[1].value
-    modal.componentInstance.alternativeClicked.emit()
-    expect(patchSpy).toHaveBeenLastCalledWith(
-      expect.objectContaining({ content: 'edited content' }),
-      12
-    )
-    component.onVersionSelected(doc.id) // ignored while saving
-    expect(modalSpy).toHaveBeenCalledTimes(2)
-    savedDoc.next(doc)
-    expect(component.selectedVersionId()).toEqual(10)
-    expect(component.documentForm.get('content').value).toEqual(
-      'version 10 content'
-    )
-  })
-
-  it('should switch without confirmation when the selected version was deleted, even while saving', () => {
-    initNormally()
-    component.document().versions = docVersions
-    component.selectedVersionId.set(10)
-    jest
-      .spyOn(documentService, 'get')
-      .mockReturnValue(of({ content: 'version 12 content' } as Document))
-    const modalSpy = jest.spyOn(modalService, 'open')
-    component.documentForm.get('content').setValue('edited content')
-    component.documentForm.get('content').markAsDirty()
-    component.networkActive.set(true)
-
-    // the version dropdown emits this after deleting the selected version
-    component.onVersionsUpdated(docVersions.filter((v) => v.id !== 10))
-    component.onVersionSelected(12)
-
-    expect(modalSpy).not.toHaveBeenCalled()
-    expect(component.selectedVersionId()).toEqual(12)
-    expect(component.documentForm.get('content').value).toEqual(
-      'version 12 content'
-    )
-  })
-
-  it('should restore the selected version and its unsaved content when returning to a document', () => {
-    initNormally()
-    const openDoc = component.document()
-    openDoc.versions = docVersions
-    jest.spyOn(openDocumentsService, 'getOpenDocument').mockReturnValue(openDoc)
-    jest
-      .spyOn(documentService, 'get')
-      .mockImplementation((id, versionID) =>
-        of(
-          (versionID
-            ? { content: `version ${versionID} content` }
-            : { ...doc, versions: docVersions }) as Document
-        )
-      )
-    component.selectVersion(10)
-    // an edit that happens to match the latest version's content
-    component.documentForm.get('content').setValue(doc.content)
-    openDoc.__changedFields = ['content']
-
-    component['loadDocument'](doc.id)
-
-    expect(component.selectedVersionId()).toEqual(10)
-    expect(component.documentForm.get('content').value).toEqual(doc.content)
-    expect(openDocumentsService.isDirty(openDoc)).toBeTruthy()
-    const patchSpy = jest
-      .spyOn(documentService, 'patch')
-      .mockReturnValue(of(doc))
-    component.save()
-    expect(patchSpy).toHaveBeenCalledWith(
-      expect.objectContaining({ content: doc.content }),
-      10
-    )
-  })
-
-  it('should fall back to the latest version when the remembered version no longer exists', () => {
-    initNormally()
-    const openDoc = component.document()
-    openDoc.versions = docVersions
-    jest.spyOn(openDocumentsService, 'getOpenDocument').mockReturnValue(openDoc)
-    jest.spyOn(documentService, 'get').mockImplementation((id, versionID) =>
-      of(
-        (versionID
-          ? { content: `version ${versionID} content` }
-          : {
-              ...doc,
-              content: 'version 12 content',
-              versions: docVersions.filter((v) => v.id !== 10),
-            }) as Document
-      )
-    )
-    component.selectVersion(10)
-
-    component['loadDocument'](doc.id)
-
-    expect(component.selectedVersionId()).toEqual(12)
-    expect(component.documentForm.get('content').value).toEqual(
-      'version 12 content'
-    )
-  })
-
  it('createDisabled should return true if the user does not have permission to add the specified data type', () => {
    currentUserCan = false
    expect(component.createDisabled(DataType.Correspondent)).toBeTruthy()
@@ -98,8 +98,8 @@ import { ISODateAdapter } from 'src/app/utils/ngb-iso-date-adapter'
 import * as UTIF from 'utif'
 import { DocumentDetailFieldID } from '../admin/settings/settings.component'
 import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
-import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
 import { ReprocessConfirmDialogComponent } from '../common/confirm-dialog/reprocess-confirm-dialog/reprocess-confirm-dialog.component'
+import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
 import { CustomFieldsDropdownComponent } from '../common/custom-fields-dropdown/custom-fields-dropdown.component'
 import { CorrespondentEditDialogComponent } from '../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
 import { DocumentTypeEditDialogComponent } from '../common/edit-dialog/document-type-edit-dialog/document-type-edit-dialog.component'
@@ -304,7 +304,6 @@ export class DocumentDetailComponent
  isDirty$: Observable<boolean>
  unsubscribeNotifier: Subject<any> = new Subject()
  docChangeNotifier: Subject<any> = new Subject()
-  versionChangeNotifier: Subject<void> = new Subject()
  private incomingUpdateModal: NgbModalRef
  private pendingIncomingUpdate: IncomingDocumentUpdate
  private lastLocalSaveModified: string | null = null
@@ -418,8 +417,7 @@ export class DocumentDetailComponent
      .pipe(
        first(),
        takeUntil(this.unsubscribeNotifier),
-        takeUntil(this.docChangeNotifier),
-        takeUntil(this.versionChangeNotifier)
+        takeUntil(this.docChangeNotifier)
      )
      .subscribe({
        next: (result) => {
@@ -535,8 +533,7 @@ export class DocumentDetailComponent
      .pipe(
        first(),
        takeUntil(this.unsubscribeNotifier),
-        takeUntil(this.docChangeNotifier),
-        takeUntil(this.versionChangeNotifier)
+        takeUntil(this.docChangeNotifier)
      )
      .subscribe({
        next: (res) => this.previewText.set(res.toString()),
@@ -598,13 +595,6 @@ export class DocumentDetailComponent
            openDocument.duplicate_documents = doc.duplicate_documents
            this.openDocumentService.save()
          }
-          // use server versions
-          if (openDocument) {
-            openDocument.versions = doc.versions
-            if (!openDocument.__changedFields?.includes('content')) {
-              openDocument.content = doc.content
-            }
-          }
          let useDoc = openDocument || doc
          if (openDocument && forceRemote) {
            Object.assign(openDocument, doc)
@@ -652,14 +642,7 @@ export class DocumentDetailComponent
              this.documentForm.patchValue({ title: titleValue })
              this.documentForm.get('title').markAsDirty()
            })
-          const keepContentEdits =
-            useDoc.__selectedVersionId === this.selectedVersionId() &&
-            !!useDoc.__changedFields?.includes('content')
          this.setupDirtyTracking(useDoc, doc)
-          // Maybe load the stored version
-          if (useDoc.__selectedVersionId) {
-            this.selectVersion(this.selectedVersionId(), keepContentEdits)
-          }
        },
      })
  }
@@ -920,11 +903,9 @@ export class DocumentDetailComponent

  updateComponent(doc: Document) {
    this.document.set(doc)
-    // Load the selected version, or default to API first (newest)
+    // Default selected version is the newest version, which the API returns first
    const versions = doc.versions ?? []
-    const selectedVersion =
-      versions.find((v) => v.id === doc.__selectedVersionId) ?? versions[0]
-    this.selectedVersionId.set(selectedVersion?.id ?? doc.id)
+    this.selectedVersionId.set(versions.length ? versions[0].id : doc.id)
    this.previewLoaded.set(false)
    this.requiresPassword = false
    this.updateFormForCustomFields()
@@ -959,12 +940,8 @@ export class DocumentDetailComponent
  }

  // Update file preview and download target to a specific version (by document id)
-  selectVersion(versionId: number, keepContentEdits: boolean = false) {
-    this.versionChangeNotifier.next()
+  selectVersion(versionId: number) {
    this.selectedVersionId.set(versionId)
-    // remember so the version can be restored when returning to the document
-    this.document().__selectedVersionId = versionId
-    this.openDocumentService.save()
    this.previewLoaded.set(false)
    this.previewUrl.set(
      this.documentsService.getPreviewUrl(
@@ -986,20 +963,20 @@ export class DocumentDetailComponent
      .pipe(
        first(),
        takeUntil(this.unsubscribeNotifier),
-        takeUntil(this.docChangeNotifier),
-        takeUntil(this.versionChangeNotifier)
+        takeUntil(this.docChangeNotifier)
      )
      .subscribe({
        next: (doc) => {
          const content = doc?.content ?? ''
-          if (keepContentEdits) {
-            this.store.next({ ...this.store.value, content })
-          } else {
-            // Update in-place and avoid the debounce wait
-            this.store.value.content = content
-            this.documentForm.patchValue({ content })
-            this.documentForm.get('content').markAsPristine()
-          }
+          this.document().content = content
+          this.documentForm.patchValue(
+            {
+              content,
+            },
+            {
+              emitEvent: false,
+            }
+          )
        },
        error: (error) => {
          this.toastService.showError(
@@ -1014,8 +991,7 @@ export class DocumentDetailComponent
      .pipe(
        first(),
        takeUntil(this.unsubscribeNotifier),
-        takeUntil(this.docChangeNotifier),
-        takeUntil(this.versionChangeNotifier)
+        takeUntil(this.docChangeNotifier)
      )
      .subscribe({
        next: (res) => this.previewText.set(res.toString()),
@@ -1029,39 +1005,7 @@ export class DocumentDetailComponent
  }

  onVersionSelected(versionId: number) {
-    if (versionId === this.selectedVersionId()) return
-    // Bail if the selected version was just deleted.
-    const selectedVersionExists = this.document()?.versions?.some(
-      (v) => v.id === this.selectedVersionId()
-    )
-    if (this.networkActive() && selectedVersionExists) return
-    if (
-      !selectedVersionExists ||
-      this.documentForm.get('content').value === this.store.value.content
-    ) {
-      this.selectVersion(versionId)
-      return
-    }
-
-    // Confirm any unsaved content changes
-    const modal = this.modalService.open(ConfirmDialogComponent, {
-      backdrop: 'static',
-    })
-    modal.componentInstance.title = $localize`Unsaved Changes`
-    modal.componentInstance.messageBold = $localize`You have unsaved changes to the content of this version.`
-    modal.componentInstance.message = $localize`Switching versions will discard them.`
-    modal.componentInstance.btnClass = 'btn-secondary'
-    modal.componentInstance.btnCaption = $localize`Discard and switch`
-    modal.componentInstance.alternativeBtnClass = 'btn-primary'
-    modal.componentInstance.alternativeBtnCaption = $localize`Save and switch`
-    modal.componentInstance.confirmClicked.pipe(first()).subscribe(() => {
-      modal.close()
-      this.selectVersion(versionId)
-    })
-    modal.componentInstance.alternativeClicked.pipe(first()).subscribe(() => {
-      modal.close()
-      this.save(false, () => this.selectVersion(versionId))
-    })
+    this.selectVersion(versionId)
  }

  onVersionsUpdated(versions: DocumentVersionInfo[]) {
@@ -1289,7 +1233,7 @@ export class DocumentDetailComponent
    return changes
  }

-  save(close: boolean = false, savedCallback: () => void = null) {
+  save(close: boolean = false) {
    this.networkActive.set(true)
    ;(document.activeElement as HTMLElement)?.dispatchEvent(new Event('change'))
    this.documentsService
@@ -1322,7 +1266,6 @@ export class DocumentDetailComponent
            this.flushPendingIncomingUpdate()
          }
          this.savedViewService.maybeRefreshDocumentCounts()
-          savedCallback?.()
        },
        error: (error) => {
          this.networkActive.set(false)
@@ -167,7 +167,6 @@ export interface Document extends ObjectWithPermissions {

  // Frontend only
  __changedFields?: string[]
-  __selectedVersionId?: number
 }

 export interface DocumentVersionInfo {
@@ -221,25 +221,6 @@ describe('OpenDocumentsService', () => {
    expect(openDocumentsService.getOpenDocuments()).toHaveLength(1)
  })

-  it('should refresh documents in place and keep unsaved edits', () => {
-    const openDoc = { ...documents[0] }
-    subscriptions.push(openDocumentsService.openDocument(openDoc).subscribe())
-    openDoc.title = 'Unsaved title'
-    openDocumentsService.setDirty(openDoc, true, { title: openDoc.title })
-
-    openDocumentsService.refreshDocument(openDoc.id)
-    httpTestingController
-      .expectOne(
-        `${environment.apiBaseUrl}documents/${openDoc.id}/?full_perms=true`
-      )
-      .flush({ ...documents[0], tags: [4] })
-
-    const refreshed = openDocumentsService.getOpenDocument(openDoc.id)
-    expect(refreshed).toBe(openDoc)
-    expect(refreshed.title).toEqual('Unsaved title')
-    expect(refreshed.tags).toEqual([4])
-  })
-
  it('should handle error on refresh documents', () => {
    subscriptions.push(
      openDocumentsService.openDocument(documents[1]).subscribe()
@@ -50,15 +50,7 @@ export class OpenDocumentsService {
    if (index > -1) {
      this.documentService.get(id).subscribe({
        next: (doc) => {
-          const openDoc = this.openDocuments.find((d) => d.id == id)
-          if (!openDoc) return
-          const unsavedEdits = Object.fromEntries(
-            (openDoc.__changedFields ?? []).map((field) => [
-              field,
-              openDoc[field],
-            ])
-          )
-          Object.assign(openDoc, doc, unsavedEdits)
+          this.openDocuments[index] = doc
          this.save()
        },
        error: () => {
@@ -129,7 +129,7 @@ class DocumentMetadataOverrides:
        )
        overrides.custom_fields = {
            custom_field.field.id: custom_field.value
-            for custom_field in doc.custom_fields.all()
+            for custom_field in doc.custom_fields.select_related("field").all()
        }

        groups_with_perms = get_groups_with_perms(
@@ -3,6 +3,7 @@ from __future__ import annotations
 import logging
 import math
 import re
+from collections.abc import Iterable
 from datetime import datetime
 from datetime import timedelta
 from decimal import Decimal
@@ -883,8 +884,86 @@ def validate_documentlink_targets(user, doc_ids):
        )


+# drf-writable-nested revalidates a document's custom_fields more than once
+# per request: once as the ordinary nested list, then again per-item while
+# matching existing vs. new CustomFieldInstance rows during save() -- and
+# that second pass builds a brand new serializer (and field) instance per
+# item (see its update_or_create_reverse_relations / _get_serializer_for_field),
+# so a cache on the field instance alone only helps the first pass. It does,
+# however, explicitly pass `context=self.context` to every one of those
+# fresh serializers -- the *same* dict object the outer DocumentSerializer
+# is using, not a copy. That context dict is already request-scoped (DRF
+# builds it fresh per request via get_serializer_context()), so stashing the
+# resolved CustomField objects there -- rather than in some new global/
+# thread-local cache -- lets every later pass reuse them for free while
+# staying entirely within DRF's existing, already-request-scoped machinery.
+_CUSTOM_FIELD_CONTEXT_CACHE_KEY = "_custom_field_lookup_cache"
+
+
+class _CachingCustomFieldPrimaryKeyField(serializers.PrimaryKeyRelatedField):
+    """
+    Resolves CustomField ids via a cache on the serializer context, so
+    later, separately-instantiated fields for the same request (drf-writable-
+    nested rebuilds one per item during save) reuse what was already
+    resolved instead of re-querying.
+    """
+
+    def _cache(self) -> dict[int, CustomField]:
+        return self.context.setdefault(_CUSTOM_FIELD_CONTEXT_CACHE_KEY, {})
+
+    @staticmethod
+    def _normalize_pk(data: Any) -> int | None:
+        """
+        Returns `data` coerced to the int a valid CustomField pk would be,
+        or None if `data` isn't a plausible pk (wrong type, unhashable,
+        non-numeric, or a bool -- DRF itself rejects bools as pks since
+        `True == 1` would otherwise silently match). None tells callers to
+        leave `data` alone and let `super().to_internal_value()` report the
+        normal validation error instead of touching the cache/queryset with
+        it directly.
+        """
+        if isinstance(data, bool):
+            return None
+        try:
+            return int(data)
+        except (TypeError, ValueError):
+            return None
+
+    def prefetch(self, ids: Iterable[Any]) -> None:
+        cache = self._cache()
+        candidates = {pk for i in ids if (pk := self._normalize_pk(i)) is not None}
+        missing = {i for i in candidates if i not in cache}
+        if missing:
+            for obj in self.get_queryset().filter(pk__in=missing):
+                cache[obj.pk] = obj
+
+    def to_internal_value(self, data: Any) -> CustomField:
+        pk = self._normalize_pk(data)
+        if pk is None:
+            return super().to_internal_value(data)
+        cache = self._cache()
+        if pk in cache:
+            return cache[pk]
+        obj: CustomField = super().to_internal_value(data)
+        cache[obj.pk] = obj
+        return obj
+
+
+class CustomFieldInstanceListSerializer(serializers.ListSerializer):
+    def to_internal_value(self, data: Any) -> list[Any]:
+        if isinstance(data, list):
+            field_ids = [
+                item["field"]
+                for item in data
+                if isinstance(item, dict) and "field" in item
+            ]
+            if field_ids:
+                self.child.fields["field"].prefetch(field_ids)
+        return super().to_internal_value(data)
+
+
 class CustomFieldInstanceSerializer(serializers.ModelSerializer[CustomFieldInstance]):
-    field = serializers.PrimaryKeyRelatedField(queryset=CustomField.objects.all())
+    field = _CachingCustomFieldPrimaryKeyField(queryset=CustomField.objects.all())
    value = ReadWriteSerializerMethodField(allow_null=True)

    def create(self, validated_data):
@@ -985,6 +1064,7 @@ class CustomFieldInstanceSerializer(serializers.ModelSerializer[CustomFieldInsta

    class Meta:
        model = CustomFieldInstance
+        list_serializer_class = CustomFieldInstanceListSerializer
        fields = [
            "value",
            "field",
@@ -38,42 +38,6 @@ class TestChatStreamingViewInputValidation(APITestCase):
            )
        assert resp.status_code == status.HTTP_400_BAD_REQUEST

-    def test_answer_is_not_compressed(self) -> None:
-        """
-        GIVEN:
-            - A client that accepts compressed responses
-        WHEN:
-            - It asks the chat endpoint a question
-        THEN:
-            - The answer is streamed unencoded, chunk for chunk
-
-        The stream compressors buffer, so a compressed answer arrives in one
-        piece. The view cannot opt out by flagging the request: DRF's request
-        wrapper proxies reads but keeps writes to itself, so the flag never
-        reaches the Django request the middleware sees.
-        """
-        chunks = [f"token{i} " for i in range(40)]
-        with (
-            mock.patch(
-                "documents.views.AIConfig",
-                return_value=self._mock_ai_enabled(),
-            ),
-            mock.patch(
-                "documents.views.stream_chat_with_documents",
-                return_value=iter(chunks),
-            ),
-        ):
-            resp = self.client.post(
-                "/api/documents/chat/",
-                {"q": "What is in my archive?"},
-                format="json",
-                HTTP_ACCEPT_ENCODING="gzip, deflate, br, zstd",
-            )
-
-        assert resp.status_code == status.HTTP_200_OK
-        assert not resp.has_header("Content-Encoding")
-        assert list(resp.streaming_content) == [c.encode() for c in chunks]
-
    def test_missing_question_is_rejected(self) -> None:
        with mock.patch(
            "documents.views.AIConfig",
@@ -5,14 +5,19 @@ from unittest.mock import ANY

 from django.contrib.auth.models import Permission
 from django.contrib.auth.models import User
+from django.db import connection
 from django.test import override_settings
+from django.test.utils import CaptureQueriesContext
 from guardian.shortcuts import assign_perm
 from rest_framework import status
 from rest_framework.test import APITestCase

+from documents.data_models import DocumentMetadataOverrides
 from documents.models import CustomField
 from documents.models import CustomFieldInstance
 from documents.models import Document
+from documents.serialisers import DocumentSerializer
+from documents.tests.factories import DocumentFactory
 from documents.tests.utils import DirectoriesMixin


@@ -530,6 +535,190 @@ class TestCustomFieldsAPI(DirectoriesMixin, APITestCase):
        doc.refresh_from_db()
        self.assertEqual(len(doc.custom_fields.all()), 10)

+    def test_document_serializer_custom_fields_validation_batches_field_lookup(
+        self,
+    ) -> None:
+        """
+        GIVEN:
+            - A document is being validated with several custom field values
+              at once (as happens on every PATCH/PUT/POST)
+        WHEN:
+            - The serializer is validated
+        THEN:
+            - The referenced CustomField objects are resolved with a single
+              query, not one query per custom field
+        """
+        doc = DocumentFactory(mime_type="application/pdf")
+        custom_fields = [
+            CustomField.objects.create(
+                name=f"Test Custom Field {i}",
+                data_type=CustomField.FieldDataType.STRING,
+            )
+            for i in range(5)
+        ]
+
+        serializer = DocumentSerializer(
+            doc,
+            data={
+                "custom_fields": [
+                    {"field": custom_field.id, "value": "test value"}
+                    for custom_field in custom_fields
+                ],
+            },
+            partial=True,
+        )
+
+        with CaptureQueriesContext(connection) as ctx:
+            self.assertTrue(serializer.is_valid(), serializer.errors)
+
+        custom_field_lookups = [
+            query
+            for query in ctx.captured_queries
+            if 'FROM "documents_customfield" WHERE "documents_customfield"."id"'
+            in query["sql"]
+        ]
+        self.assertEqual(
+            len(custom_field_lookups),
+            1,
+            "Expected a single batched query to resolve the custom fields, "
+            f"got {len(custom_field_lookups)}: {custom_field_lookups}",
+        )
+
+    def test_document_serializer_save_reuses_cached_custom_fields(self) -> None:
+        """
+        GIVEN:
+            - A document is being saved with several custom field values via
+              DocumentSerializer, which drives drf-writable-nested's real
+              update_or_create_reverse_relations path -- rebuilding a fresh
+              CustomFieldInstanceSerializer per item during save(), the exact
+              mechanism the shared-context cache exists to optimize
+        WHEN:
+            - The serializer, already validated, is saved
+        THEN:
+            - No further CustomField queries are issued: each per-item
+              nested serializer reuses the CustomField objects resolved
+              during is_valid(), instead of re-resolving them during save()
+        """
+        doc = DocumentFactory(mime_type="application/pdf")
+        custom_fields = [
+            CustomField.objects.create(
+                name=f"Test Custom Field {i}",
+                data_type=CustomField.FieldDataType.STRING,
+            )
+            for i in range(5)
+        ]
+
+        serializer = DocumentSerializer(
+            doc,
+            data={
+                "custom_fields": [
+                    {"field": custom_field.id, "value": "test value"}
+                    for custom_field in custom_fields
+                ],
+            },
+            partial=True,
+        )
+        self.assertTrue(serializer.is_valid(), serializer.errors)
+
+        with CaptureQueriesContext(connection) as ctx:
+            serializer.save()
+
+        custom_field_lookups = [
+            query
+            for query in ctx.captured_queries
+            if 'FROM "documents_customfield" WHERE "documents_customfield"."id"'
+            in query["sql"]
+        ]
+        self.assertEqual(
+            custom_field_lookups,
+            [],
+            "Expected save() to reuse CustomField objects resolved during "
+            f"is_valid(), got: {custom_field_lookups}",
+        )
+        self.assertEqual(doc.custom_fields.count(), 5)
+
+    def test_custom_field_validation_rejects_malformed_field_value(self) -> None:
+        """
+        GIVEN:
+            - A document is being validated with a malformed custom_fields
+              entry whose "field" value is neither a valid CustomField id
+              nor a type DRF's own PrimaryKeyRelatedField can safely reject
+              on its own (unhashable, or a non-numeric scalar)
+        WHEN:
+            - The serializer is validated
+        THEN:
+            - A normal validation error is raised, not an unhandled
+              TypeError/ValueError escaping past DRF's validation layer
+        """
+        doc = DocumentFactory(mime_type="application/pdf")
+
+        bad_field_values = {
+            "unhashable-list": [],
+            "unhashable-dict": {},
+            "non-numeric-scalar": "abc",
+        }
+        for case_id, bad_field_value in bad_field_values.items():
+            with self.subTest(case_id):
+                serializer = DocumentSerializer(
+                    doc,
+                    data={
+                        "custom_fields": [
+                            {"field": bad_field_value, "value": "test value"},
+                        ],
+                    },
+                    partial=True,
+                )
+
+                self.assertFalse(serializer.is_valid())
+                self.assertIn("custom_fields", serializer.errors)
+
+    def test_document_metadata_overrides_from_document_batches_field_lookup(
+        self,
+    ) -> None:
+        """
+        GIVEN:
+            - A document has several custom field values
+            - The document instance has just been refreshed from the database,
+              which drops any prefetched related objects (as
+              send_websocket_document_updated does before building overrides)
+        WHEN:
+            - DocumentMetadataOverrides.from_document() reads the document's
+              custom field values
+        THEN:
+            - The referenced CustomField objects are resolved with a single
+              query, not one query per custom field
+        """
+        doc = DocumentFactory(mime_type="application/pdf")
+        for i in range(5):
+            CustomFieldInstance.objects.create(
+                document=doc,
+                field=CustomField.objects.create(
+                    name=f"Test Custom Field {i}",
+                    data_type=CustomField.FieldDataType.STRING,
+                ),
+                value_text="value",
+            )
+
+        doc.refresh_from_db()
+
+        with CaptureQueriesContext(connection) as ctx:
+            overrides = DocumentMetadataOverrides.from_document(doc)
+
+        self.assertEqual(len(overrides.custom_fields), 5)
+        unbatched_field_lookups = [
+            query
+            for query in ctx.captured_queries
+            if 'FROM "documents_customfield" WHERE "documents_customfield"."id"'
+            in query["sql"]
+        ]
+        self.assertEqual(
+            unbatched_field_lookups,
+            [],
+            "Expected CustomField data to come from the CustomFieldInstance "
+            "join, not a separate per-instance lookup, "
+            f"got: {unbatched_field_lookups}",
+        )
+
    def test_change_custom_field_instance_value(self) -> None:
        """
        GIVEN:
@@ -1,6 +1,5 @@
 import pytest
 import regex
-from django.conf import settings
 from pytest_mock import MockerFixture

 from documents.regex import safe_regex_finditer
@@ -10,12 +9,6 @@ from documents.regex import safe_regex_sub
 from documents.regex import validate_regex_pattern


-def test_regex_timeout_uses_configured_setting() -> None:
-    from documents.regex import REGEX_TIMEOUT_SECONDS
-
-    assert REGEX_TIMEOUT_SECONDS == settings.MATCH_REGEX_TIMEOUT_SECONDS
-
-
 class TestValidateRegexPattern:
    def test_valid_pattern(self) -> None:
        validate_regex_pattern(r"\d+")
@@ -2380,6 +2380,7 @@ class ChatStreamingView(GenericAPIView[Any]):
    serializer_class = ChatStreamingSerializer

    def post(self, request, *args, **kwargs):
+        request.compress_exempt = True
        ai_config = AIConfig()
        if not ai_config.ai_enabled:
            return HttpResponseBadRequest("AI is required for this feature")
@@ -2,7 +2,7 @@ msgid ""
 msgstr ""
 "Project-Id-Version: paperless-ngx\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2026-09-11 15:47+0000\n"
+"POT-Creation-Date: 2026-09-10 19:39+0000\n"
 "PO-Revision-Date: 2022-02-17 04:17\n"
 "Last-Translator: \n"
 "Language-Team: English\n"
@@ -1632,7 +1632,7 @@ msgid "workflow runs"
 msgstr ""

 #: documents/serialisers.py:525 documents/serialisers.py:882
-#: documents/serialisers.py:2849 documents/views.py:318 documents/views.py:2693
+#: documents/serialisers.py:2849 documents/views.py:318 documents/views.py:2694
 #: paperless_mail/serialisers.py:156
 msgid "Insufficient permissions."
 msgstr ""
@@ -1673,7 +1673,7 @@ msgstr ""
 msgid "Duplicate document identifiers are not allowed."
 msgstr ""

-#: documents/serialisers.py:2935 documents/views.py:4700
+#: documents/serialisers.py:2935 documents/views.py:4701
 #, python-format
 msgid "Documents not found: %(ids)s"
 msgstr ""
@@ -1941,7 +1941,7 @@ msgstr ""
 msgid "Unable to parse URI {value}"
 msgstr ""

-#: documents/views.py:311 documents/views.py:2690
+#: documents/views.py:311 documents/views.py:2691
 msgid "Invalid more_like_id"
 msgstr ""

@@ -1957,24 +1957,24 @@ msgstr ""
 msgid "AI backend rejected the request. Check logs for details."
 msgstr ""

-#: documents/views.py:2515 documents/views.py:2836
+#: documents/views.py:2516 documents/views.py:2837
 msgid "Specify only one of text, title_search, query, or more_like_id."
 msgstr ""

-#: documents/views.py:4713
+#: documents/views.py:4714
 #, python-format
 msgid "Insufficient permissions to share document %(id)s."
 msgstr ""

-#: documents/views.py:4759
+#: documents/views.py:4760
 msgid "Bundle is already being processed."
 msgstr ""

-#: documents/views.py:4823
+#: documents/views.py:4824
 msgid "The share link bundle is still being prepared. Please try again later."
 msgstr ""

-#: documents/views.py:4837
+#: documents/views.py:4838
 msgid "The share link bundle is unavailable."
 msgstr ""

@@ -2258,151 +2258,151 @@ msgstr ""
 msgid "paperless application settings"
 msgstr ""

-#: paperless/settings/__init__.py:556
+#: paperless/settings/__init__.py:569
 msgid "English (US)"
 msgstr ""

-#: paperless/settings/__init__.py:557
+#: paperless/settings/__init__.py:570
 msgid "Arabic"
 msgstr ""

-#: paperless/settings/__init__.py:558
+#: paperless/settings/__init__.py:571
 msgid "Afrikaans"
 msgstr ""

-#: paperless/settings/__init__.py:559
+#: paperless/settings/__init__.py:572
 msgid "Belarusian"
 msgstr ""

-#: paperless/settings/__init__.py:560
+#: paperless/settings/__init__.py:573
 msgid "Bulgarian"
 msgstr ""

-#: paperless/settings/__init__.py:561
+#: paperless/settings/__init__.py:574
 msgid "Catalan"
 msgstr ""

-#: paperless/settings/__init__.py:562
+#: paperless/settings/__init__.py:575
 msgid "Czech"
 msgstr ""

-#: paperless/settings/__init__.py:563
+#: paperless/settings/__init__.py:576
 msgid "Danish"
 msgstr ""

-#: paperless/settings/__init__.py:564
+#: paperless/settings/__init__.py:577
 msgid "German"
 msgstr ""

-#: paperless/settings/__init__.py:565
+#: paperless/settings/__init__.py:578
 msgid "Greek"
 msgstr ""

-#: paperless/settings/__init__.py:566
+#: paperless/settings/__init__.py:579
 msgid "English (GB)"
 msgstr ""

-#: paperless/settings/__init__.py:567
+#: paperless/settings/__init__.py:580
 msgid "Spanish"
 msgstr ""

-#: paperless/settings/__init__.py:568
+#: paperless/settings/__init__.py:581
 msgid "Persian"
 msgstr ""

-#: paperless/settings/__init__.py:569
+#: paperless/settings/__init__.py:582
 msgid "Finnish"
 msgstr ""

-#: paperless/settings/__init__.py:570
+#: paperless/settings/__init__.py:583
 msgid "French"
 msgstr ""

-#: paperless/settings/__init__.py:571
+#: paperless/settings/__init__.py:584
 msgid "Hungarian"
 msgstr ""

-#: paperless/settings/__init__.py:572
+#: paperless/settings/__init__.py:585
 msgid "Indonesian"
 msgstr ""

-#: paperless/settings/__init__.py:573
+#: paperless/settings/__init__.py:586
 msgid "Italian"
 msgstr ""

-#: paperless/settings/__init__.py:574
+#: paperless/settings/__init__.py:587
 msgid "Japanese"
 msgstr ""

-#: paperless/settings/__init__.py:575
+#: paperless/settings/__init__.py:588
 msgid "Korean"
 msgstr ""

-#: paperless/settings/__init__.py:576
+#: paperless/settings/__init__.py:589
 msgid "Luxembourgish"
 msgstr ""

-#: paperless/settings/__init__.py:577
+#: paperless/settings/__init__.py:590
 msgid "Norwegian"
 msgstr ""

-#: paperless/settings/__init__.py:578
+#: paperless/settings/__init__.py:591
 msgid "Dutch"
 msgstr ""

-#: paperless/settings/__init__.py:579
+#: paperless/settings/__init__.py:592
 msgid "Polish"
 msgstr ""

-#: paperless/settings/__init__.py:580
+#: paperless/settings/__init__.py:593
 msgid "Portuguese (Brazil)"
 msgstr ""

-#: paperless/settings/__init__.py:581
+#: paperless/settings/__init__.py:594
 msgid "Portuguese"
 msgstr ""

-#: paperless/settings/__init__.py:582
+#: paperless/settings/__init__.py:595
 msgid "Romanian"
 msgstr ""

-#: paperless/settings/__init__.py:583
+#: paperless/settings/__init__.py:596
 msgid "Russian"
 msgstr ""

-#: paperless/settings/__init__.py:584
+#: paperless/settings/__init__.py:597
 msgid "Slovak"
 msgstr ""

-#: paperless/settings/__init__.py:585
+#: paperless/settings/__init__.py:598
 msgid "Slovenian"
 msgstr ""

-#: paperless/settings/__init__.py:586
+#: paperless/settings/__init__.py:599
 msgid "Serbian"
 msgstr ""

-#: paperless/settings/__init__.py:587
+#: paperless/settings/__init__.py:600
 msgid "Swedish"
 msgstr ""

-#: paperless/settings/__init__.py:588
+#: paperless/settings/__init__.py:601
 msgid "Turkish"
 msgstr ""

-#: paperless/settings/__init__.py:589
+#: paperless/settings/__init__.py:602
 msgid "Ukrainian"
 msgstr ""

-#: paperless/settings/__init__.py:590
+#: paperless/settings/__init__.py:603
 msgid "Vietnamese"
 msgstr ""

-#: paperless/settings/__init__.py:591
+#: paperless/settings/__init__.py:604
 msgid "Chinese Simplified"
 msgstr ""

-#: paperless/settings/__init__.py:592
+#: paperless/settings/__init__.py:605
 msgid "Chinese Traditional"
 msgstr ""

@@ -1,23 +1,8 @@
-from compression_middleware.middleware import CompressionMiddleware
 from django.conf import settings

 from paperless import version


-class StreamAwareCompressionMiddleware(CompressionMiddleware):
-    """
-    Bypasses compression for server-sent streams (text/event-stream).
-
-    See https://github.com/friedelwolff/django-compression-middleware/pull/7
-    """
-
-    def process_response(self, request, response):
-        content_type = response.headers.get("Content-Type", "")
-        if content_type.startswith("text/event-stream"):
-            return response
-        return super().process_response(request, response)
-
-
 class ApiVersionMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response
@@ -10,6 +10,7 @@ from pathlib import Path
 from typing import Final
 from urllib.parse import urlparse

+from compression_middleware.middleware import CompressionMiddleware
 from django.core.exceptions import ImproperlyConfigured
 from django.utils.translation import gettext_lazy as _
 from dotenv import load_dotenv
@@ -102,10 +103,6 @@ CLASSIFIER_MATCH_THRESHOLD: Final[float] = get_float_from_env(
    "PAPERLESS_CLASSIFIER_MATCH_THRESHOLD",
    0.6,
 )
-MATCH_REGEX_TIMEOUT_SECONDS: Final[float] = get_float_from_env(
-    "PAPERLESS_MATCH_REGEX_TIMEOUT_SECONDS",
-    0.1,
-)
 LLM_INDEX_DIR = DATA_DIR / "llm_index"
 LLM_INDEX_LOCK = LLM_INDEX_DIR / "index.lock"
 # Cross-process read/write lock guarding the LLM index compaction/migration
@@ -204,10 +201,22 @@ MIDDLEWARE = [
    "allauth.account.middleware.AccountMiddleware",
 ]

-# Optional to enable compression. The subclass leaves server-sent events
-# uncompressed; see paperless.middleware.StreamAwareCompressionMiddleware.
+# Optional to enable compression
 if get_bool_from_env("PAPERLESS_ENABLE_COMPRESSION", "yes"):  # pragma: no cover
-    MIDDLEWARE.insert(0, "paperless.middleware.StreamAwareCompressionMiddleware")
+    MIDDLEWARE.insert(0, "compression_middleware.middleware.CompressionMiddleware")
+
+# Workaround to not compress streaming responses (e.g. chat).
+# See https://github.com/friedelwolff/django-compression-middleware/pull/7
+original_process_response = CompressionMiddleware.process_response
+
+
+def patched_process_response(self, request, response):
+    if getattr(request, "compress_exempt", False):
+        return response
+    return original_process_response(self, request, response)
+
+
+CompressionMiddleware.process_response = patched_process_response

 ROOT_URLCONF = "paperless.urls"

@@ -1,54 +0,0 @@
-from django.http import HttpResponse
-from django.http import StreamingHttpResponse
-from django.test import RequestFactory
-from django.test import TestCase
-
-from paperless.middleware import StreamAwareCompressionMiddleware
-
-
-class TestStreamAwareCompressionMiddleware(TestCase):
-    def setUp(self) -> None:
-        super().setUp()
-        self.factory = RequestFactory()
-        self.middleware = StreamAwareCompressionMiddleware(lambda request: None)
-
-    def _request(self):
-        return self.factory.get(
-            "/api/documents/chat/",
-            HTTP_ACCEPT_ENCODING="gzip, deflate, br, zstd",
-        )
-
-    def test_event_stream_is_not_compressed(self) -> None:
-        """
-        GIVEN:
-            - A server-sent event response produced chunk by chunk
-        WHEN:
-            - The compression middleware processes it
-        THEN:
-            - It is passed through unencoded, one wire chunk per source chunk
-        """
-        chunks = [f"token{i} ".encode() for i in range(40)]
-        response = StreamingHttpResponse(
-            iter(chunks),
-            content_type="text/event-stream",
-        )
-
-        response = self.middleware.process_response(self._request(), response)
-
-        assert not response.has_header("Content-Encoding")
-        assert list(response.streaming_content) == chunks
-
-    def test_regular_response_is_still_compressed(self) -> None:
-        """
-        GIVEN:
-            - An ordinary response large enough to be worth compressing
-        WHEN:
-            - The compression middleware processes it
-        THEN:
-            - It is compressed as before
-        """
-        response = HttpResponse(b"a" * 5000, content_type="application/json")
-
-        response = self.middleware.process_response(self._request(), response)
-
-        assert response.has_header("Content-Encoding")
@@ -40,6 +40,7 @@ LLM_SYSTEM_PROMPT = (

 # openai-python rejects empty keys since 2.34.0, "fake" is the stand-in from
 # llama-index's own OpenAILike docs https://docs.llamaindex.ai/en/stable/api_reference/llms/openai_like/
+# TODO: remove pending resolution of https://github.com/openai/openai-python/issues/3224
 PLACEHOLDER_API_KEY: Final = "fake"
Author	SHA1	Message	Date
stumpylog	fa60ab7f31	Reduces the complexity of this loop now that we know it is hashable	2026-09-10 14:51:30 -07:00
stumpylog	77194fe74e	Improves the test to actually check the behaviour of the cache, rather than bypassing the serializer	2026-09-10 14:23:17 -07:00
stumpylog	e46e384b80	Test: fold the DocumentMetadataOverrides batching test into test_api_custom_fields.py instead of a new file	2026-09-10 13:10:10 -07:00
stumpylog	3a7666400b	Perf: collapse the per-field and shared context CustomField caches into a single context-level cache	2026-09-10 13:10:10 -07:00
stumpylog	8d03b70d9b	Handles a bad client sending malformed JSON or non-int primary keys	2026-09-10 13:10:10 -07:00
Trenton Holmes	abf16f6af4	Perf: avoid per-instance CustomField reload in DocumentMetadataOverrides send_websocket_document_updated calls document.refresh_from_db() before building overrides, which drops the custom_fields prefetch (and its select_related("field")) set up by the view's queryset. DocumentMetadataOverrides.from_document() then lazily reloads field once per custom field instance. Since from_document() can't rely on the caller having a prefetched document, select_related explicitly at the point of use instead.	2026-09-10 13:10:10 -07:00
Trenton Holmes	6d3891ea84	Perf: reuse resolved CustomField objects across drf-writable-nested's per-item revalidation drf-writable-nested's update_or_create_reverse_relations rebuilds a fresh serializer -- and fresh field instances -- per custom_fields item while matching existing vs. new instances during save(), so the per-instance lookup cache alone only helped the first validation pass. It passes the same context dict (by reference) to every one of those serializers, so stash resolved CustomField objects there instead: later passes reuse them for free rather than re-querying.	2026-09-10 13:10:10 -07:00
Trenton Holmes	93c9d5a6d9	Perf: batch CustomField lookups when validating a document's custom_fields DocumentSerializer.custom_fields validates each item's field id via a plain PrimaryKeyRelatedField, which issues one SELECT per custom field per validation pass (discussion #13690). Batch-resolve all field ids in one query and cache them on the field instance so per-item validation is free instead of re-querying.	2026-09-10 13:10:10 -07:00