Compare commits

..
Author SHA1 Message Date
Niels Lohmann c92d9195a5 Merge develop
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 20:32:24 +01:00
Niels Lohmann 84c5421644 Fix regression1 operator[](json_pointer) test under JSON_NO_THREAD_LOCAL (#5952)
The test relies on a copied fifo_map restarting its timestamps. Copying a
basic_json does that only on the recursive path; with JSON_NO_THREAD_LOCAL
the copy is built member by member and keeps correct timestamps, so the
premise was missing and the test failed (ci_test_no_thread_local, MinGW
clang). Copy the object_t explicitly to create the state on every path.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 20:31:53 +01:00
Niels Lohmann b22ba482fd Silence misc-const-correctness for patch_aliases_this
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 20:23:11 +01:00
Niels Lohmann f45e65cee5 Merge develop
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 20:21:24 +01:00
Niels Lohmann 95519dbbcb Detect a patch stored inside the patched value with JSON_DIAGNOSTICS
patch_inplace applied a copy only when the patch was the value itself. With JSON_DIAGNOSTICS the parent pointers make it cheap to also detect a patch nested inside the value, so that case is handled the same way. Without diagnostics it stays documented undefined behavior.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 17:18:36 +01:00
Niels Lohmann 359c889ed2 Fix JSON_DIAGNOSTICS test crash and clang-tidy hit in patch_inplace self-aliasing
The spare-capacity test reallocated non-empty elements through get_ref(), which leaves stale parent pointers under JSON_DIAGNOSTICS; reserve while the array is empty instead. Suppress performance-unnecessary-copy-initialization for the copy that is needed because *this is modified.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 14:36:57 +01:00
Niels Lohmann 16793d6884 Merge develop
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 14:34:36 +01:00
Niels Lohmann 9edff5292d Merge branch 'develop' into fix-self-aliasing-pointer-patch
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 07:29:17 +01:00
Niels Lohmann 8dd8f5ce8e Merge origin/develop into fix-self-aliasing-pointer-patch
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-26 21:39:59 +01:00
Niels Lohmann 0babaa1a4e Fix self-aliasing in json_pointer::operator/= and patch_inplace
json_pointer::operator/=(const json_pointer&) passed iterators into its
own token vector to insert() when a pointer was appended to itself
(p /= p), which is undefined behavior; libstdc++ produced "/a/b//" for
"/a/b". The pointer now appends a copy in that case.

basic_json::patch_inplace iterated the patch while changing *this, so
j.patch_inplace(j) read freed memory. It now applies a copy of the
patch when the patch is the value itself. A patch stored inside the
value remains undefined behavior and is documented as such; copying the
patch unconditionally would double the time of a large patch.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-26 12:10:55 +01:00
15 changed files with 263 additions and 448 deletions

No files matched your search

-5
View File
@@ -154,11 +154,6 @@ and each page's title (H1) and declaration by
[`docs/docset/generate_docset.py`](https://github.com/nlohmann/json/blob/develop/docs/docset/generate_docset.py);
`make build` reports API pages that cannot be classified.
A pull request that changes only the documentation, Markdown files, GitHub metadata, or license files skips the
compiler workflows (Ubuntu, macOS, Windows); the documentation and license checks still run. See
[`tools/check_path_filters`](https://github.com/nlohmann/json/tree/develop/tools/check_path_filters) for the list of
skipped files and what to do when a compiler job starts reading one of them.
### Amalgamate the source code
The single-header files
-19
View File
@@ -7,25 +7,6 @@ on:
- master
- release/*
pull_request:
# Skip pull requests that change only files no job here reads. The list is checked by
# tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
paths:
- '**'
- '!docs/**'
- '!**.md'
- '!.github/*'
- '!.github/ISSUE_TEMPLATE/**'
- '!.github/external_ci/**'
- '!.github/workflows/**'
- '!.reuse/**'
- '!LICENSES/**'
- '!CITATION.cff'
- '!REUSE.toml'
# read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
- 'docs/mkdocs/docs/integration/package_managers.md'
# built by the swiftpm job (macos.yml)
- 'docs/mkdocs/docs/integration/swift/**'
- '.github/workflows/macos.yml'
workflow_dispatch:
concurrency:
-78
View File
@@ -1,78 +0,0 @@
name: Repository checks
# Checks that concern files the compiler workflows (ubuntu.yml, macos.yml,
# windows.yml) skip with their path filter: the documentation, the license
# information, and the path filter itself. This workflow has no path filter.
on:
push:
branches:
- develop
- master
- release/*
pull_request:
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
cancel-in-progress: true
jobs:
check_path_filters:
runs-on: ubuntu-26.04
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check the path filters of the compiler workflows
run: python3 tools/check_path_filters/check_path_filters.py
ci_reuse_compliance:
runs-on: ubuntu-26.04
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_reuse_compliance
ci_test_documentation:
runs-on: ubuntu-26.04
strategy:
matrix:
target: [ci_test_examples, ci_test_build_documentation, ci_test_documentation_mermaid]
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# the git-revision-date-localized plugin needs the history; a shallow clone makes the strict build fail
fetch-depth: ${{ matrix.target == 'ci_test_build_documentation' && '0' || '1' }}
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
if: matrix.target == 'ci_test_documentation_mermaid'
with:
node-version: 24
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
+26 -20
View File
@@ -7,25 +7,6 @@ on:
- master
- release/*
pull_request:
# Skip pull requests that change only files no job here reads. The list is checked by
# tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
paths:
- '**'
- '!docs/**'
- '!**.md'
- '!.github/*'
- '!.github/ISSUE_TEMPLATE/**'
- '!.github/external_ci/**'
- '!.github/workflows/**'
- '!.reuse/**'
- '!LICENSES/**'
- '!CITATION.cff'
- '!REUSE.toml'
# read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
- 'docs/mkdocs/docs/integration/package_managers.md'
# built by the swiftpm job (macos.yml)
- 'docs/mkdocs/docs/integration/swift/**'
- '.github/workflows/ubuntu.yml'
workflow_dispatch:
permissions:
@@ -128,7 +109,7 @@ jobs:
runs-on: ubuntu-26.04
strategy:
matrix:
target: [ci_test_single_header, ci_cppcheck, ci_cpplint, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata, ci_test_valgrind]
target: [ci_test_single_header, ci_cppcheck, ci_cpplint, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata, ci_reuse_compliance, ci_test_valgrind]
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
@@ -468,6 +449,31 @@ jobs:
- name: Build
run: cmake --build build
ci_test_documentation:
runs-on: ubuntu-26.04
strategy:
matrix:
target: [ci_test_examples, ci_test_build_documentation, ci_test_documentation_mermaid]
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# the git-revision-date-localized plugin needs the history; a shallow clone makes the strict build fail
fetch-depth: ${{ matrix.target == 'ci_test_build_documentation' && '0' || '1' }}
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
if: matrix.target == 'ci_test_documentation_mermaid'
with:
node-version: 24
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
# Native Linux ARM64 runner with the latest GCC. The ubuntu-24.04-arm label is
# only available for public repositories.
ci_test_arm64:
-19
View File
@@ -7,25 +7,6 @@ on:
- master
- release/*
pull_request:
# Skip pull requests that change only files no job here reads. The list is checked by
# tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
paths:
- '**'
- '!docs/**'
- '!**.md'
- '!.github/*'
- '!.github/ISSUE_TEMPLATE/**'
- '!.github/external_ci/**'
- '!.github/workflows/**'
- '!.reuse/**'
- '!LICENSES/**'
- '!CITATION.cff'
- '!REUSE.toml'
# read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
- 'docs/mkdocs/docs/integration/package_managers.md'
# built by the swiftpm job (macos.yml)
- 'docs/mkdocs/docs/integration/swift/**'
- '.github/workflows/windows.yml'
workflow_dispatch:
permissions:
@@ -69,6 +69,19 @@ Duplicate member names inside an operation object are resolved by the parser bef
is kept, see [`object_t`](object_t.md#behavior)), so `#!json {"op": "add", "op": "remove", "path": "/a"}` is a
`remove` operation.
The JSON patch may be the JSON value itself (`#!cpp j.patch_inplace(j)`): the operations are then read from a copy of
the patch made before the first operation is applied, so the result is the same as that of
`#!cpp j = j.patch(j)`, and the exception safety is the same as above.
If [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) is enabled, the same holds for a JSON patch stored inside the
JSON value (e.g., `#!cpp j.patch_inplace(j["patch"])`): it is detected via the parent pointers.
!!! danger "Undefined behavior"
Without [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md), the JSON patch must not be a value stored inside the
JSON value (e.g., `#!cpp j.patch_inplace(j["patch"])`), as the operations may change or destroy it while it is
read. Pass a copy (`#!cpp j.patch_inplace(json(j["patch"]))`) or use [`patch`](patch.md) instead.
## Examples
??? example "Example: apply a JSON patch in place"
@@ -122,3 +135,5 @@ is kept, see [`object_t`](object_t.md#behavior)), so `#!json {"op": "add", "op":
- Throws [`parse_error.109`](../../home/exceptions.md#jsonexceptionparse_error109) instead of
[`out_of_range.404`](../../home/exceptions.md#jsonexceptionout_of_range404) for a one-character array index that is
not a digit (e.g., `/x`) in version 3.13.0, as it already did for longer ones.
- Applying a JSON value as patch to itself (or, with `JSON_DIAGNOSTICS`, to a value stored inside it) no longer results
in undefined behavior in version 3.13.0.
@@ -18,7 +18,8 @@ json_pointer& operator/=(std::size_t array_idx)
## Parameters
`ptr` (in)
: JSON pointer to append
: JSON pointer to append; may be the JSON pointer itself (`#!cpp p /= p`), which then appends a copy of its
reference tokens
`token` (in)
: reference token to append
@@ -69,6 +70,7 @@ json_pointer& operator/=(std::size_t array_idx)
## Version history
1. Added in version 3.6.0.
1. Added in version 3.6.0. Appending a JSON pointer to itself no longer results in undefined behavior in version
3.13.0.
2. Added in version 3.6.0. Changed type of `token` to `string_t` in version 3.11.0.
3. Added in version 3.6.0.
+7
View File
@@ -266,6 +266,13 @@ class json_pointer
/// @sa https://json.nlohmann.me/api/json_pointer/operator_slasheq/
json_pointer& operator/=(const json_pointer& ptr)
{
// insert() must not be given iterators into the vector it inserts
// into: append a copy when a pointer is appended to itself
if (JSON_HEDLEY_UNLIKELY(&ptr == this))
{
return *this /= json_pointer(ptr);
}
reference_tokens.insert(reference_tokens.end(),
ptr.reference_tokens.begin(),
ptr.reference_tokens.end());
+19
View File
@@ -6950,6 +6950,25 @@ public:
/// @sa https://json.nlohmann.me/api/basic_json/patch/
void patch_inplace(const basic_json& json_patch)
{
// the operations below change *this while iterating json_patch: apply
// a copy when a value is applied as patch to itself or, with
// JSON_DIAGNOSTICS, to a value stored inside it
// NOLINTNEXTLINE(misc-const-correctness): only the JSON_DIAGNOSTICS build modifies it
bool patch_aliases_this = (&json_patch == this);
#if JSON_DIAGNOSTICS
for (const basic_json* p = json_patch.m_parent; !patch_aliases_this && p != nullptr; p = p->m_parent)
{
patch_aliases_this = (p == this);
}
#endif
if (JSON_HEDLEY_UNLIKELY(patch_aliases_this))
{
// NOLINTNEXTLINE(performance-unnecessary-copy-initialization): *this is modified below
const basic_json json_patch_copy = json_patch;
patch_inplace(json_patch_copy);
return;
}
basic_json& result = *this;
// the valid JSON Patch operations
enum class patch_operations {add, remove, replace, move, copy, test, invalid};
+26
View File
@@ -24331,6 +24331,13 @@ class json_pointer
/// @sa https://json.nlohmann.me/api/json_pointer/operator_slasheq/
json_pointer& operator/=(const json_pointer& ptr)
{
// insert() must not be given iterators into the vector it inserts
// into: append a copy when a pointer is appended to itself
if (JSON_HEDLEY_UNLIKELY(&ptr == this))
{
return *this /= json_pointer(ptr);
}
reference_tokens.insert(reference_tokens.end(),
ptr.reference_tokens.begin(),
ptr.reference_tokens.end());
@@ -43701,6 +43708,25 @@ public:
/// @sa https://json.nlohmann.me/api/basic_json/patch/
void patch_inplace(const basic_json& json_patch)
{
// the operations below change *this while iterating json_patch: apply
// a copy when a value is applied as patch to itself or, with
// JSON_DIAGNOSTICS, to a value stored inside it
// NOLINTNEXTLINE(misc-const-correctness): only the JSON_DIAGNOSTICS build modifies it
bool patch_aliases_this = (&json_patch == this);
#if JSON_DIAGNOSTICS
for (const basic_json* p = json_patch.m_parent; !patch_aliases_this && p != nullptr; p = p->m_parent)
{
patch_aliases_this = (p == this);
}
#endif
if (JSON_HEDLEY_UNLIKELY(patch_aliases_this))
{
// NOLINTNEXTLINE(performance-unnecessary-copy-initialization): *this is modified below
const basic_json json_patch_copy = json_patch;
patch_inplace(json_patch_copy);
return;
}
basic_json& result = *this;
// the valid JSON Patch operations
enum class patch_operations {add, remove, replace, move, copy, test, invalid};
+123
View File
@@ -826,6 +826,129 @@ TEST_CASE("JSON patch")
CHECK(doc.at("foo") == "bar");
}
#endif // !defined(JSON_NOEXCEPTION)
SECTION("applying a value as patch to itself")
{
// a JSON patch is an array, so an array of operations can be
// applied to itself; the operations are read from the patch as
// it was before the first operation, as patch() does it
const json copy_op = {{"op", "copy"}, {"from", "/0"}, {"path", "/-"}};
json copy_ops = json::array();
for (int i = 0; i < 5; ++i)
{
copy_ops.push_back(copy_op);
}
json expected_copies = copy_ops;
for (int i = 0; i < 5; ++i)
{
expected_copies.push_back(copy_op);
}
SECTION("copy operations without spare capacity")
{
// a copy has exactly as much capacity as elements
json doc = copy_ops;
CHECK(doc.patch(doc) == expected_copies);
doc.patch_inplace(doc);
CHECK(doc == expected_copies);
}
#if JSON_DIAGNOSTICS
SECTION("patch stored inside the patched value")
{
// the parent pointers of JSON_DIAGNOSTICS make the aliasing detectable
const json op = {{"op", "replace"}, {"path", "/patch"}, {"value", 1}};
json doc = {{"a", 1}, {"patch", json::array({op})}};
const json expected = doc.patch(doc["patch"]);
doc.patch_inplace(doc["patch"]);
CHECK(doc == expected);
}
#endif
SECTION("copy operations with spare capacity")
{
// reserve while the array is empty: reallocating non-empty
// elements behind the library's back leaves stale parent
// pointers with JSON_DIAGNOSTICS
json doc = json::array();
doc.get_ref<json::array_t&>().reserve(100);
for (int i = 0; i < 5; ++i)
{
doc.push_back(copy_op);
}
CHECK(doc.patch(doc) == expected_copies);
doc.patch_inplace(doc);
CHECK(doc == expected_copies);
}
SECTION("remove operations")
{
json doc = R"([
{"op": "remove", "path": "/0"},
{"op": "remove", "path": "/0"},
{"op": "remove", "path": "/0"}
])"_json;
CHECK(doc.patch(doc) == json::array());
doc.patch_inplace(doc);
CHECK(doc == json::array());
}
SECTION("operations that read a changed element")
{
// the second operation is still a "test" operation, although
// the first one replaced it in the document
json doc = R"([
{"op": "replace", "path": "/1", "value": 1},
{"op": "test", "path": "/1", "value": 1},
{"op": "add", "path": "/-", "value": 2}
])"_json;
const json expected = R"([
{"op": "replace", "path": "/1", "value": 1},
1,
{"op": "add", "path": "/-", "value": 2},
2
])"_json;
CHECK(doc.patch(doc) == expected);
doc.patch_inplace(doc);
CHECK(doc == expected);
}
SECTION("replacing the root")
{
json doc = R"([
{"op": "add", "path": "", "value": [1, 2]},
{"op": "add", "path": "/-", "value": 3},
{"op": "replace", "path": "", "value": {"a": 4}}
])"_json;
const json expected = {{"a", 4}};
CHECK(doc.patch(doc) == expected);
doc.patch_inplace(doc);
CHECK(doc == expected);
}
#if !defined(JSON_NOEXCEPTION)
SECTION("failing operation")
{
// the operations before the failing one remain applied
json doc = R"([
{"op": "add", "path": "/-", "value": 1},
{"op": "test", "path": "/0/op", "value": "remove"}
])"_json;
CHECK_THROWS_AS(doc.patch(doc), json::other_error&);
CHECK_THROWS_AS(doc.patch_inplace(doc), json::other_error&);
CHECK(doc.size() == 3);
CHECK(doc.back() == 1);
}
#endif // !defined(JSON_NOEXCEPTION)
}
}
SECTION("errors")
+38
View File
@@ -917,6 +917,44 @@ TEST_CASE("JSON pointers")
CHECK(ptr.to_string() == "/object/~1");
}
SECTION("appending a pointer to itself")
{
// operator/= must not pass iterators into its own token vector to
// insert(); each pointer is appended to itself and must yield the
// pointer whose string representation is the original one twice
const std::vector<std::string> pointers =
{
"",
"/",
"/a",
"/a/b",
"/foo/0/-",
"/a~1b/~0c/~01",
"/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17"
};
for (const auto& s : pointers)
{
CAPTURE(s)
json::json_pointer ptr(s);
const json::json_pointer expected(s + s);
json::json_pointer& ret = (ptr /= ptr);
CHECK(&ret == &ptr);
CHECK(ptr == expected);
CHECK(ptr.to_string() == s + s);
// appending to itself again doubles the pointer once more
ptr /= ptr;
CHECK(ptr.to_string() == s + s + s + s);
// operator/ appends to a copy and leaves the operand unchanged
const json::json_pointer original(s);
CHECK((original / original) == expected);
CHECK(original.to_string() == s);
}
}
SECTION("equality comparison")
{
std::string ptr_string{"/foo/bar"};
+5 -2
View File
@@ -1504,13 +1504,16 @@ TEST_CASE("regression tests 1")
// a copied fifo_map restarts its timestamps, so a new key "k" gets
// the timestamp of "x": the lookup does not find "k", but emplace()
// finds "x" and inserts nothing; the pointer then continues with the
// member of "x", as basic_json::operator[] does
// member of "x", as basic_json::operator[] does. The object_t is
// copied explicitly: copying the my_json would build the object
// member by member (with fresh timestamps) when built with
// JSON_NO_THREAD_LOCAL.
const my_json orig = my_json::parse(R"({"x":1})");
const std::array<const char*, 3> pointers = {{"/k/0", "/k/a/b", "/k/-"}};
for (const char* pointer : pointers)
{
CAPTURE(pointer)
my_json j = orig;
my_json j(orig.get_ref<const my_json::object_t&>());
CHECK_THROWS_AS(j[my_json::json_pointer(pointer)], my_json::out_of_range&);
CHECK(j.dump() == R"({"x":1})");
}
-34
View File
@@ -1,34 +0,0 @@
# check_path_filters
The compiler workflows [`ubuntu.yml`](../../.github/workflows/ubuntu.yml),
[`macos.yml`](../../.github/workflows/macos.yml), and [`windows.yml`](../../.github/workflows/windows.yml) skip pull
requests that change only files no compiler job reads: the documentation, Markdown files, GitHub metadata, other
workflows, and the license files. Pushes to `develop`, `master`, and `release/*` always run them. The documentation,
license, and path-filter checks run in [`repository_checks.yml`](../../.github/workflows/repository_checks.yml), which
has no path filter.
The filter starts with `'**'` and excludes these files explicitly, so a new file runs the compiler workflows unless it
is excluded on purpose. [`check_path_filters.py`](check_path_filters.py) holds the filter (`PATHS`) and checks that
- each of the three workflows has exactly this filter, with its own workflow file included,
- the CI targets that read excluded files (`UNFILTERED_TARGETS`) do not run in these workflows, and
- no file the filter keeps (sources, tests, build files, tools, workflows) mentions an excluded file or directory
outside a comment line, unless the mention is listed in `KNOWN_REFERENCES` with a reason.
When the check reports a mention, decide whether a compiler job reads that file:
- If it does, include the file in `PATHS` after the exclusions (as `package_managers.md` and the Swift example are) and
copy the new filter into the three workflows.
- If it does not, add the mention to `KNOWN_REFERENCES` with the reason.
Run the check with
```shell
python3 tools/check_path_filters/check_path_filters.py
```
To see which workflows a pull request would run, pass the changed files:
```shell
python3 tools/check_path_filters/check_path_filters.py --changed $(git diff --name-only origin/develop)
```
@@ -1,269 +0,0 @@
#!/usr/bin/env python3
"""Check the path filters of the compiler workflows.
ubuntu.yml, macos.yml and windows.yml skip pull requests that change only files
no compiler job reads (documentation, Markdown files, GitHub metadata, license
files). The filter starts with '**' and excludes these files explicitly, so a
new file runs the workflows unless it is excluded on purpose.
The script checks that
- each of the three workflows has exactly the filter defined in PATHS below,
with its own workflow file included, and
- no tracked file that the filter keeps (sources, tests, build files, tools,
workflows) mentions a file the filter excludes, unless the mention is listed
in KNOWN_REFERENCES with the reason why it does not matter.
A mention found this way must either be added to PATHS as an included path
(the file is read by a compiler job) or to KNOWN_REFERENCES (it is not).
With --changed FILE..., the script prints instead which workflows a pull
request changing these files would run.
"""
import argparse
import os
import re
import subprocess # nosec B404
import sys
REPO_ROOT = os.path.normpath(os.path.join(sys.path[0], '..', '..'))
SELF = 'tools/check_path_filters/check_path_filters.py'
WORKFLOWS = ['.github/workflows/ubuntu.yml', '.github/workflows/macos.yml', '.github/workflows/windows.yml']
# The pull_request paths filter of the workflows above; '{workflow}' is replaced
# with the workflow's own file. GitHub applies the patterns in order: a file
# matching a later pattern is included ('pattern') or excluded ('!pattern').
PATHS = [
'**',
'!docs/**',
'!**.md',
'!.github/*',
'!.github/ISSUE_TEMPLATE/**',
'!.github/external_ci/**',
'!.github/workflows/**',
'!.reuse/**',
'!LICENSES/**',
'!CITATION.cff',
'!REUSE.toml',
# read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
'docs/mkdocs/docs/integration/package_managers.md',
# built by the swiftpm job (macos.yml)
'docs/mkdocs/docs/integration/swift/**',
'{workflow}',
]
# Mentions of excluded files that no job of the three workflows depends on:
# (file that mentions it, the mention, reason)
KNOWN_REFERENCES = [
('cmake/ci.cmake', 'docs/', 'ci_test_examples; runs in repository_checks.yml, which has no path filter'),
('cmake/ci.cmake', 'docs/mkdocs/', 'ci_test_build_documentation and ci_test_documentation_mermaid; they run in '
'repository_checks.yml, which has no path filter'),
('cmake/ci.cmake', 'docs/mkdocs/docs/examples/', 'indentation check of ci_test_amalgamation; the same check runs in '
'check_amalgamation.yml, which has no path filter'),
('Makefile', 'docs/mkdocs/docs/examples/', 'make pretty; runs in check_amalgamation.yml, which has no path filter'),
('Makefile', 'ChangeLog.md', 'make ChangeLog.md writes the file; no CI job runs it'),
('Makefile', 'tests/fuzzing.md', 'help text'),
('Makefile', 'tools/generate_natvis/README.md', 'help text'),
('tests/src/make_test_data_available.hpp', 'README.md', 'a file of the external test data repository'),
('tools/check_build_options/check_build_options.py', 'docs/', 'builds the path of package_managers.md, which '
'PATHS includes'),
]
# CI targets that read excluded files; they must run in a workflow without a path filter
UNFILTERED_TARGETS = ['ci_test_examples', 'ci_test_build_documentation', 'ci_test_documentation_mermaid',
'ci_test_amalgamation', 'ci_reuse_compliance']
def git_files(root):
# fixed command, no user input; includes new files not yet added, so a local run
# sees what CI sees after the commit
out = subprocess.check_output(['git', 'ls-files', '-z', '--cached', '--others', '--exclude-standard'], cwd=root) # nosec B603, B607
return [f for f in out.decode('utf-8').split('\0') if f]
def glob_to_regex(pattern):
"""Translate a GitHub Actions path pattern to a regular expression."""
regex = ''
i = 0
while i < len(pattern):
if pattern.startswith('**', i):
regex += '.*'
i += 2
elif pattern[i] == '*':
regex += '[^/]*'
i += 1
elif pattern[i] == '?':
regex += '[^/]'
i += 1
else:
regex += re.escape(pattern[i])
i += 1
return re.compile(regex + r'\Z')
def compile_filter(paths, workflow):
return [(p.startswith('!'), glob_to_regex(p.lstrip('!').replace('{workflow}', workflow))) for p in paths]
def included(compiled, path):
result = False
for negated, regex in compiled:
if regex.match(path):
result = not negated
return result
def read_filter(root, workflow):
"""Return the pull_request paths list of a workflow file."""
with open(os.path.join(root, workflow), encoding='utf-8') as f:
lines = f.read().splitlines()
paths = None
in_pull_request = False
for line in lines:
if re.match(r'^ pull_request:', line):
in_pull_request = True
continue
if in_pull_request:
if re.match(r'^ \S', line) or re.match(r'^\S', line):
break
if re.match(r'^ paths:\s*$', line):
paths = []
continue
m = re.match(r"^ - '(.*)'\s*(#.*)?$", line)
if paths is not None and m:
paths.append(m.group(1))
elif paths is not None and line.strip() and not line.strip().startswith('#'):
break
return paths
TOKEN = re.compile(r'(["\'/]?)([\w.-]+(?:/[\w.-]+)*)')
COMMENT_LINE = re.compile(r'^[ \t]*(//|#|/?\*|--).*$', re.MULTILINE)
class Mentions:
"""Finds the words and paths in a text that name an excluded file or directory."""
def __init__(self, files, excluded, is_excluded):
self.files = set(files)
self.excluded = excluded
self.dirs = {'/'.join(f.split('/')[:n]) for f in files for n in range(1, f.count('/') + 1)}
# an excluded directory: a new file in it would be excluded
self.excluded_dirs = {d for d in self.dirs if is_excluded(d + '/new_file')}
# base names that only excluded files have, e.g. 'ChangeLog.md' but not 'CMakeLists.txt'
self.names = {os.path.basename(f) for f in excluded} - {os.path.basename(f) for f in files if f not in excluded}
def find(self, text):
"""Yield (offset, mention) for every word or path in text that names an excluded file or directory."""
for m in TOKEN.finditer(text):
parts = m.group(2).rstrip('/').split('/')
# a single word names a directory only in a path context, e.g. 'docs' in os.path.join('docs', ...)
path_context = len(parts) > 1 or m.group(1)
for i in range(len(parts)):
path = '/'.join(parts[i:])
if path in self.files:
if path in self.excluded:
yield m.start(2), path
break
if path in self.dirs and path_context:
if path in self.excluded_dirs:
yield m.start(2), path + '/'
break
else:
if parts[-1] in self.names and '.' in parts[-1]:
yield m.start(2), parts[-1]
def known(path, mention):
return any(path == file and mention == known_mention for file, known_mention, _ in KNOWN_REFERENCES)
def check_workflows(root):
"""Return the errors in the filters of the workflows and in the targets they run."""
errors = []
for workflow in WORKFLOWS:
expected = [p.replace('{workflow}', workflow) for p in PATHS]
actual = read_filter(root, workflow)
if actual != expected:
errors.append('{}: the pull_request paths filter differs from tools/check_path_filters/check_path_filters.py\n'
' expected: {}\n actual: {}'.format(workflow, expected, actual))
with open(os.path.join(root, workflow), encoding='utf-8') as f:
text = f.read()
errors += ['{}: runs {}, which reads excluded files; run it in repository_checks.yml'.format(workflow, target)
for target in UNFILTERED_TARGETS if re.search(r'\b' + target + r'\b', text)]
return errors
def scanned_text(root, path):
"""Return the text of a kept file without comment lines and without the filter itself, or None to skip it."""
# third-party code and Markdown files read nothing; this script lists the excluded files
if path.startswith('tests/thirdparty/') or path.endswith('.md') or path == SELF:
return None
try:
with open(os.path.join(root, path), encoding='utf-8') as f:
# a mention in a comment line reads nothing; keep the line numbers
text = COMMENT_LINE.sub('', f.read())
except (UnicodeDecodeError, OSError):
return None
if path in WORKFLOWS:
# the filter itself lists the excluded files
for p in PATHS:
text = text.replace(" - '{}'".format(p.replace('{workflow}', path)), '')
return text
def check_mentions(root, files, excluded, finder):
"""Return an error for every mention of an excluded file in a kept file that KNOWN_REFERENCES does not list."""
errors = []
for path in sorted(set(files) - excluded):
text = scanned_text(root, path)
reported = set()
for offset, mention in finder.find(text or ''):
if mention not in reported and not known(path, mention):
reported.add(mention)
line = text.count('\n', 0, offset) + 1
errors.append('{}:{}: mentions excluded {}; include it in PATHS if a compiler job reads it, '
'or add it to KNOWN_REFERENCES'.format(path, line, mention))
return errors
def check(root):
files = git_files(root)
union = [compile_filter(PATHS, w) for w in WORKFLOWS]
def is_excluded(path):
return not any(included(c, path) for c in union)
excluded = {f for f in files if is_excluded(f)}
errors = check_workflows(root) + check_mentions(root, files, excluded, Mentions(files, excluded, is_excluded))
for e in errors:
print(e)
if not errors:
print('path filters OK: {} of {} tracked files skip the compiler workflows'.format(len(excluded), len(files)))
return 1 if errors else 0
def explain(changed):
for workflow in WORKFLOWS:
compiled = compile_filter(PATHS, workflow)
hits = [f for f in changed if included(compiled, f)]
print('{}: {}'.format(workflow, 'runs ({})'.format(', '.join(hits[:3]) + (' ...' if len(hits) > 3 else ''))
if hits else 'skipped'))
return 0
def main():
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument('--root', default=REPO_ROOT, help='repository root')
parser.add_argument('--changed', nargs='+', metavar='FILE', help='show which workflows these changed files run')
args = parser.parse_args()
return explain(args.changed) if args.changed else check(args.root)
if __name__ == '__main__':
sys.exit(main())