Document tools/macro_builder and tidy up serve_header.py

tools/macro_builder generates the NLOHMANN_JSON_EXPAND,
NLOHMANN_JSON_GET_MACRO and NLOHMANN_JSON_PASTE* macros in
macro_scope.hpp, but nothing referred to it. Add a README that explains
what it generates, how to run it and where the output goes, and which
dependent tables (NLOHMANN_JSON_DOUBLE_PASTE, NLOHMANN_JSON_TYPE_BODY)
are maintained by hand. Point to it from a comment above
NLOHMANN_JSON_EXPAND. The generator itself is unchanged; following the
README reproduces the header byte for byte.

In serve_header.py, drop the LGTM suppression (LGTM.com shut down in
2022), replace the """.""" placeholder docstrings with real ones, and
import socket and ssl at module level. DualStackServer.server_bind uses
socket, which was only imported under __main__; when the module was
imported instead, the NameError was swallowed and IPV6_V6ONLY was not
cleared.

The header change is a comment only; behavior, API and ABI are
unchanged.

Part of #5717

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

Makefile

@@ -34,7 +34,6 @@ all:
 	@echo "ChangeLog.md - generate ChangeLog file"
 	@echo "check-amalgamation - check whether sources have been amalgamated and BUILD.bazel is up to date"
 	@echo "clean - remove built files"
-	@echo "doctest - compile example files and check their output"
 	@echo "fuzz_testing - prepare fuzz testing of the JSON parser"
 	@echo "fuzz_testing_bon8 - prepare fuzz testing of the BON8 parser"
 	@echo "fuzz_testing_bson - prepare fuzz testing of the BSON parser"
@@ -156,10 +155,6 @@ install_astyle:
 pretty: install_astyle
 	$(ASTYLE) --project=tools/astyle/.astylerc $(SRCS) $(TESTS_SRCS) $(AMALGAMATED_FILE) $(AMALGAMATED_FWD_FILE) docs/mkdocs/docs/examples/*.cpp
 
-# call the Clang-Format on all source files
-pretty_format:
-	for FILE in $(SRCS) $(TESTS_SRCS) $(AMALGAMATED_FILE) docs/mkdocs/docs/examples/*.cpp; do echo $$FILE; clang-format -i $$FILE; done
-
 # create single header files and pretty print
 amalgamate: $(AMALGAMATED_FILE) $(AMALGAMATED_FWD_FILE)
 	$(MAKE) pretty
@@ -245,7 +240,6 @@ release: include.zip json.tar.xz
 # clean up
 clean:
 	rm -fr fuzz fuzz-testing *.dSYM tests/*.dSYM
-	rm -fr benchmarks/files/numbers/*.json
 	rm -fr cmake-build-benchmarks fuzz-testing cmake-build-pvs-studio release_files
 	$(MAKE) clean -Cdocs
 

include/nlohmann/detail/macro_scope.hpp

@@ -359,6 +359,7 @@ void templated_json_throw(ExceptionType exception)
 
 // Macros to simplify conversion from/to types
 
+// NLOHMANN_JSON_EXPAND to NLOHMANN_JSON_PASTE64 are generated by tools/macro_builder (see its README.md)
 #define NLOHMANN_JSON_EXPAND( x ) x
 #define NLOHMANN_JSON_GET_MACRO(_1, _2, _3, _4, _5, _6, _7, _8, _9, _10, _11, _12, _13, _14, _15, _16, _17, _18, _19, _20, _21, _22, _23, _24, _25, _26, _27, _28, _29, _30, _31, _32, _33, _34, _35, _36, _37, _38, _39, _40, _41, _42, _43, _44, _45, _46, _47, _48, _49, _50, _51, _52, _53, _54, _55, _56, _57, _58, _59, _60, _61, _62, _63, _64, NAME,...) NAME
 #define NLOHMANN_JSON_PASTE(...) NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_GET_MACRO(__VA_ARGS__, \

single_include/nlohmann/json.hpp

@@ -2770,6 +2770,7 @@ void templated_json_throw(ExceptionType exception)
 
 // Macros to simplify conversion from/to types
 
+// NLOHMANN_JSON_EXPAND to NLOHMANN_JSON_PASTE64 are generated by tools/macro_builder (see its README.md)
 #define NLOHMANN_JSON_EXPAND( x ) x
 #define NLOHMANN_JSON_GET_MACRO(_1, _2, _3, _4, _5, _6, _7, _8, _9, _10, _11, _12, _13, _14, _15, _16, _17, _18, _19, _20, _21, _22, _23, _24, _25, _26, _27, _28, _29, _30, _31, _32, _33, _34, _35, _36, _37, _38, _39, _40, _41, _42, _43, _44, _45, _46, _47, _48, _49, _50, _51, _52, _53, _54, _55, _56, _57, _58, _59, _60, _61, _62, _63, _64, NAME,...) NAME
 #define NLOHMANN_JSON_PASTE(...) NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_GET_MACRO(__VA_ARGS__, \

tools/macro_builder/README.md

@@ -0,0 +1,43 @@
+# macro_builder
+
+Generates the argument-counting macros behind the `NLOHMANN_DEFINE_TYPE_*` and `NLOHMANN_DEFINE_DERIVED_TYPE_*`
+macros in [`include/nlohmann/detail/macro_scope.hpp`](../../include/nlohmann/detail/macro_scope.hpp):
+
+- `NLOHMANN_JSON_EXPAND`
+- `NLOHMANN_JSON_GET_MACRO`, which selects a macro by the number of its arguments (64 slots)
+- `NLOHMANN_JSON_PASTE`, which calls a function-like macro for each member, and its helpers `NLOHMANN_JSON_PASTE2`
+  to `NLOHMANN_JSON_PASTE64`
+
+The number of slots (`max_args` in [`main.cpp`](main.cpp)) sets the member limit of these macros.
+`NLOHMANN_JSON_PASTE` takes the function as its first argument, so 64 slots allow 63 members.
+
+## Usage
+
+From the project root:
+
+```shell
+c++ -std=c++11 tools/macro_builder/main.cpp -o macro_builder
+./macro_builder
+```
+
+1. In `include/nlohmann/detail/macro_scope.hpp`, replace the lines from `#define NLOHMANN_JSON_EXPAND( x ) x` to the
+   `#define NLOHMANN_JSON_PASTE64(...)` line with the output, without its trailing empty line.
+2. Run `make amalgamate`. It updates `single_include/nlohmann/json.hpp` and runs `make pretty`, which indents the
+   continuation lines of `NLOHMANN_JSON_PASTE` that the tool writes unindented.
+
+With an unchanged `main.cpp`, these steps reproduce the header byte for byte.
+
+## Maintained by hand
+
+The tool does not generate everything that depends on the number of slots. When changing `max_args`, also update:
+
+- the `NLOHMANN_JSON_DOUBLE_PASTE` table right after the generated block (`NLOHMANN_JSON_DOUBLE_PASTE` and
+  `NLOHMANN_JSON_DOUBLE_PASTE3`, `NLOHMANN_JSON_DOUBLE_PASTE5`, ..., `NLOHMANN_JSON_DOUBLE_PASTE63`), which the
+  `*_WITH_NAMES` macros use
+- the slot table of `NLOHMANN_JSON_TYPE_BODY`, which chooses between the implementations for zero members and for one
+  or more members
+- the documented limit of 63 members in `docs/mkdocs/docs` and the tests at that limit in
+  `tests/src/unit-udt_macro.cpp`
+
+Both tables pass one macro name per slot to `NLOHMANN_JSON_GET_MACRO`, so they need exactly as many entries as it has
+slots.

tools/serve_header/serve_header.py

@@ -5,6 +5,8 @@ import logging
 import os
 import re
 import shutil
+import socket
+import ssl
 import sys
 import subprocess
 
@@ -34,7 +36,7 @@ JSON_VERSION_RE = re.compile(r'\s*#\s*define\s+NLOHMANN_JSON_VERSION_MAJOR\s+')
 
 class ExitHandler(logging.StreamHandler):
     def __init__(self, level):
-        """."""
+        """Exit the process on log records at or above level."""
         super().__init__()
         self.level = level
 
@@ -54,7 +56,7 @@ def is_project_root(test_dir='.'):
 
 class DirectoryEventBucket:
     def __init__(self, callback, delay=1.2, threshold=0.8):
-        """."""
+        """Batch directory events and pass their common path to callback."""
         self.delay = delay
         self.threshold = timedelta(seconds=threshold)
         self.callback = callback
@@ -99,7 +101,7 @@ class WorkTree:
     make_command = 'make'
 
     def __init__(self, root_dir, tree_dir):
-        """."""
+        """Track the working tree at tree_dir and its amalgamated header."""
         self.root_dir = root_dir
         self.tree_dir = tree_dir
         self.rel_dir = os.path.relpath(tree_dir, root_dir)
@@ -114,11 +116,11 @@ class WorkTree:
         self.build_time = t.strftime(DATETIME_FORMAT)
 
     def __hash__(self):
-        """."""
+        """Hash by working tree directory."""
         return hash((self.tree_dir))
 
     def __eq__(self, other):
-        """."""
+        """Compare by working tree directory."""
         if not isinstance(other, type(self)):
             return NotImplemented
         return self.tree_dir == other.tree_dir
@@ -150,7 +152,7 @@ class WorkTree:
 
 class WorkTrees(FileSystemEventHandler):
     def __init__(self, root_dir):
-        """."""
+        """Find the working trees below root_dir and watch it for changes."""
         super().__init__()
         self.root_dir = root_dir
         self.trees = set([])
@@ -250,11 +252,11 @@ class WorkTrees(FileSystemEventHandler):
         self.observer.stop()
         self.observer.join()
 
-class HeaderRequestHandler(SimpleHTTPRequestHandler): # lgtm[py/missing-call-to-init]
+class HeaderRequestHandler(SimpleHTTPRequestHandler):
     cors_origins = DEFAULT_CORS_ORIGINS
 
     def __init__(self, request, client_address, server):
-        """."""
+        """Handle a request for a header below the working trees' root directory."""
         self.worktrees = server.worktrees
         self.worktree = None
         try:
@@ -336,7 +338,7 @@ class HeaderRequestHandler(SimpleHTTPRequestHandler): # lgtm[py/missing-call-to-
 
 class DualStackServer(ThreadingHTTPServer):
     def __init__(self, addr, worktrees):
-        """."""
+        """Serve the headers of worktrees on addr."""
         self.worktrees = worktrees
         super().__init__(addr, HeaderRequestHandler)
 
@@ -349,8 +351,6 @@ class DualStackServer(ThreadingHTTPServer):
 
 if __name__ == '__main__':
     import argparse
-    import ssl
-    import socket
     import yaml
 
     # exit code