Generate the docset index and pages with Python (#5799)

Replace the hand-maintained docs/docset/docSet.sql and the bash/sed
Makefile recipe with docs/docset/generate_docset.py:

- The search index is generated from the mkdocs.yml nav and each page's
  H1 and declaration. Pages documenting several entities (e.g.
  JSON_HAS_CPP_11..26) get one entry per name; all non-API pages become
  guides. New API pages no longer need a manual entry.
- Page titles are set from the index names.
- The docset is self-contained: the mermaid loader no longer points to
  https://json.nlohmann.me/assets/..., the repository widget no longer
  queries api.github.com, remaining remote images are downloaded, and
  the build fails if any remote resource load remains.
- The CSS that hides the site navigation now uses Material's classes;
  the element selectors lost against them, so the header was shown.

check_structure.py's docset check is replaced by running the generator
in `make style_check`.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/Makefile

@@ -17,6 +17,7 @@ build: style_check
 
 style_check:
 	@cd docs ; ../venv/bin/python3 ../scripts/check_structure.py
+	@venv/bin/python3 ../docset/generate_docset.py list > /dev/null
 
 # check that all Mermaid diagrams parse (needs Node.js)
 # This target is used in the CI (ci_test_documentation_mermaid).

docs/mkdocs/scripts/check_structure.py

@@ -288,36 +288,6 @@ def check_header_links() -> None:
                                f'link to "{match.group(0)}" does not point to a documentation page')
 
 
-def check_docset() -> None:
-    """Every API page and every macro has an entry in the docset index; no entry points to a missing page."""
-    entry_re = re.compile(r"VALUES \('((?:[^']|'')*)', '(\w+)', '([^']*)'\);")
-    names_by_path = {}
-    with open("../../docset/docSet.sql", encoding="utf-8") as sql:
-        for name, _, path in entry_re.findall(sql.read()):
-            names_by_path.setdefault(path, set()).add(name.replace("''", "'"))
-
-    def to_path(page):
-        if os.path.basename(page) == "index.md":
-            return page[:-len("index.md")] + "index.html"
-        return page[:-len(".md")] + "/index.html"
-
-    pages = sorted(glob.glob("**/*.md", recursive=True))
-    for path in sorted(set(names_by_path) - {to_path(p) for p in pages}):
-        report("docset/stale_entry", "../../docset/docSet.sql", f'entry "{path}" has no documentation page')
-    for page in (p for p in pages if p.startswith("api/")):
-        names = names_by_path.get(to_path(page))
-        if not names:
-            report("docset/missing_entry", page, "page has no entry in docs/docset/docSet.sql")
-        elif page.startswith("api/macros/") and os.path.basename(page) != "index.md":
-            with open(page, encoding="utf-8") as content:
-                text = content.read()
-            match = re.search(r"^# (.+)$", text, re.MULTILINE) or re.search(r"<h1>(.*?)</h1>", text, re.DOTALL)
-            title = re.sub(r"<[^>]+>|\s+", " ", match.group(1))
-            for macro in filter(None, (x.strip() for x in re.split(r"[,/]", title))):
-                if macro not in names:
-                    report("docset/missing_macro", page, f'macro "{macro}" has no entry in docs/docset/docSet.sql')
-
-
 if __name__ == "__main__":
     print(120 * "-")
     check_structure()
@@ -327,7 +297,6 @@ if __name__ == "__main__":
     check_heading_levels()
     check_image_alt_text()
     check_header_links()
-    check_docset()
     print(120 * "-")
 
     if warnings > 0: