Pleae add comments so we don’t undo this in a code refactor

Might just put Python.h up top, that way it will stay there after running clang-format.

Could also just run clang-format on the file after, it needs a cleanup.

I added an explicit Python.h include to those two files, with comments to explain why.

I then went a bit overboard and added those comments to every other Python.h include in the repository, moving it higher in the file if necessary.
EDIT: Those changes are in separate commits, to make it easier to pull those into separate PRs if needed.

I guess this all makes sense. What do others think?

Just a drive-by thought: maybe we could do this using a lint check instead of adding identical comments all over the codebase?

Unfortunately I let #28634 sit, otherwise we'd already have a C linter you could add to....

Just a drive-by thought: maybe we could do this using a lint check instead of adding identical comments all over the codebase?

There's the Python script to check this I wrote for SciPy. Should I revert the last two comments, add that as a check, and adjust the files it flags? If so, should I move existing headers to the top of the files or add an explicit Python.h include at the top?

Should I revert the last two comments, add that as a check, and adjust the files it flags?

Yes, I think that would be better. We have a linter check on azure

should I move existing headers to the top of the files

I think that would be a less noisy change.

I am not sure this works. I reverted the change to numpy/_core/src/common/blas_utils.h and ran the script, it did not complain about the bad include order. Did I do something wrong?

I am not sure this works. I reverted the change to numpy/_core/src/common/blas_utils.h and ran the script, it did not complain about the bad include order. Did I do something wrong?

It looks like blas_utils.h doesn't include Python.h, so that's fine. Running the same test with blas_utils.c produces the same results, and that is a problem, since numpy/npy_math.h includes Python.h. It looks like the #include regex has problems with trailing comments. I will try to fix that.

CI is failing with

File "/home/vsts/work/1/s/tools/check_python_h_first.py", line 119, in check_python_h_included_first
    LEAF_HEADERS.append(this_header)
                        ^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'this_header' where it is not associated with a value

Note that if Python.h is present, clang-format will put it first.

I wrote the initial version of check_python_h_first.py, but there's additions from other people in SciPy; the get_submodule_paths.py module is from someone else. Should I add comments to that effect to both files, or revert to my initial version from there and add my changes from here?

I dunno if it matters much, since I guess you mostly wrote it either way. It seems to me we should just get it in, or does anyone of the original reviewers want to have another look?

I didn't try to understand the logic in the script. It seems pretty complicated?

A short version of the core function: for each file in the list, it checks each line in that file for #include and variants (space before #, space between # and include, double quotes or angle brackets). If it finds #include:

1. If the include is of a file known not to include any other files, continue to the next line
2. If the include is not of Python.h, note it's including a non-Python header
3. If the include is of Python.h, or a file known to include Python.h, mark this file as one known to include Python.h and return a list of the lines on which the file already included of non-python files earlier in the file
   If it does not see a #include, it looks for things that imply Python.h should have been included (mostly the py:: C++ namespace; the npy_ C namespace isn't as useful here, though the Py C namespace might be) and returns the line where that happens.

If it reaches the end of the file without including any files, that file is added to the list of files known not to include any other file.

The rest of it is basically a wrapper to exclude files in submodules or vendored projects from the list of files it should check, and to try to sort the list so headers appear before the files that include them, then to count the number of files that first include a non-Python.h file and exit with that status. I think someone over in SciPy added a fancy bit so you can tell it to check only the files changed since a specific commit, which I might have dropped here.

But if SciPy is using it, it's probably reasonably battle-tested, so I think I'm ok with including it.

I think I added it to SciPy a year or so back (after some searching, as scipy/scipy#20536), as a cheaper alternative to a Cygwin CI run to catch the most common failures: I think SciPy takes an hour to build, and likely longer to run the tests.