cmake: Make build reproducible across different build directories #730

EchterAgo · 2024-02-17T07:29:36Z

Without this, the usage of the FILE macro leaves the build directory in the binary. When building the Python extension with build isolation enabled this would lead to random paths in the binary.

EchterAgo · 2024-02-17T07:33:02Z

This mostly works, but the path of ZXAlgorithms.h still remains in the binary:

$ strings wrappers/python/build/lib.linux-x86_64-cpython-311/zxingcpp.cpython-311-x86_64-linux-gnu.so | grep $HOME
/home/ago/src/zxing-cpp/core/src/ZXAlgorithms.h

This seems to be from the precompiled headers, which somehow include the file from zxing-cpp/core/src/ZXAlgorithms.h instead of zxing-cpp/wrappers/python/core/src/ZXAlgorithms.h.

Anyone have an idea why this is? I checked the include paths and it has only the wrapper path. However when I copy the root core directory to the wrapper and delete ZXAlgorithms.h in the wrapper directory, the correct path ends up in the binary, so it does seem like some path issue to me.

EchterAgo · 2024-02-17T07:38:32Z

When inspecting wrappers/python/build/temp.linux-x86_64-cpython-311/ZXing/CMakeFiles/ZXing.dir/cmake_pch.hxx.gch.d you can see that it seems to mix include paths:

 /home/ago/src/zxing-cpp/wrappers/python/core/src/ByteArray.h \
 /home/ago/src/zxing-cpp/wrappers/python/core/src/CharacterSet.h \
 /home/ago/src/zxing-cpp/wrappers/python/core/src/GTIN.h \
 /home/ago/src/zxing-cpp/core/src/ZXAlgorithms.h \
 /home/ago/src/zxing-cpp/core/src/Error.h /usr/include/c++/13/algorithm \
...
 /home/ago/src/zxing-cpp/wrappers/python/core/src/TextUtfEncoding.h \
 /home/ago/src/zxing-cpp/wrappers/python/core/src/ZXConfig.h \
 /home/ago/src/zxing-cpp/wrappers/python/core/src/Content.h \
 /home/ago/src/zxing-cpp/core/src/ReaderOptions.h \
 /home/ago/src/zxing-cpp/wrappers/python/core/src/DecodeHints.h \
 /home/ago/src/zxing-cpp/wrappers/python/core/src/ImageView.h \
 /home/ago/src/zxing-cpp/wrappers/python/core/src/Point.h \

EchterAgo · 2024-02-17T08:52:00Z

This seems to only happen with target_precompile_headers. The cmake_pch.hxx and cmake_pch.h files both only contain the absolute paths to the wrapper directory, so how do the other paths end up in the PCH?

EchterAgo · 2024-02-17T09:38:20Z

Ah, I get it, PCH starts including BarcodeFormat.h from the wrapper directory, that file has a #include "Flags.h", which in turn incldues BitHacks.h.

This matches the cmake_pch.hxx.gch file:

ZXing/CMakeFiles/ZXing.dir/cmake_pch.hxx.gch: \
 /home/ago/src/zxing-cpp/wrappers/python/build/temp.linux-x86_64-cpython-311/ZXing/CMakeFiles/ZXing.dir/cmake_pch.hxx.cxx \
 /usr/include/stdc-predef.h \
 /home/ago/src/zxing-cpp/wrappers/python/build/temp.linux-x86_64-cpython-311/ZXing/CMakeFiles/ZXing.dir/cmake_pch.hxx \
 /home/ago/src/zxing-cpp/wrappers/python/core/src/BarcodeFormat.h \
 /home/ago/src/zxing-cpp/core/src/Flags.h \
 /home/ago/src/zxing-cpp/core/src/BitHacks.h /usr/include/c++/13/cassert \
<snip>

So it seems like the include path for PCH dependencies is wrong?

EchterAgo · 2024-02-17T10:38:25Z

GCC actually resolves the symlinks:

[pid 195830] readlink("/home", 0x7ffda37f38d0, 1023) = -1 EINVAL (Invalid argument)
[pid 195830] readlink("/home/ago", 0x7ffda37f38d0, 1023) = -1 EINVAL (Invalid argument)
[pid 195830] readlink("/home/ago/src", 0x7ffda37f38d0, 1023) = -1 EINVAL (Invalid argument)
[pid 195830] readlink("/home/ago/src/zxing-cpp", 0x7ffda37f38d0, 1023) = -1 EINVAL (Invalid argument)
[pid 195830] readlink("/home/ago/src/zxing-cpp/wrappers", 0x7ffda37f38d0, 1023) = -1 EINVAL (Invalid argument)
[pid 195830] readlink("/home/ago/src/zxing-cpp/wrappers/python", 0x7ffda37f38d0, 1023) = -1 EINVAL (Invalid argument)
[pid 195830] readlink("/home/ago/src/zxing-cpp/wrappers/python/core", "../../core", 1023) = 10
[pid 195830] readlink("/home/ago/src/zxing-cpp/core", 0x7ffda37f38d0, 1023) = -1 EINVAL (Invalid argument)
[pid 195830] readlink("/home/ago/src/zxing-cpp/core/src", 0x7ffda37f38d0, 1023) = -1 EINVAL (Invalid argument)
[pid 195830] readlink("/home/ago/src/zxing-cpp/core/src/Flags.h", 0x7ffda37f38d0, 1023) = -1 EINVAL (Invalid argument)
[pid 195830] openat(AT_FDCWD, "/home/ago/src/zxing-cpp/core/src/Flags.h", O_RDONLY|O_NOCTTY) = 5
[pid 195830] newfstatat(5, "", {st_mode=S_IFREG|0664, st_size=4452, ...}, AT_EMPTY_PATH) = 0
[pid 195830] read(5, "/*\n* Copyright 2020 Axel Waggers"..., 4452) = 4452
[pid 195830] close(5)                   = 0

I'm still not quite sure how this then winds up in the binary, but it is definitely through the cmake_pch.hxx.gch

EchterAgo · 2024-02-17T11:07:57Z

Updated to find the project root.

core/CMakeLists.txt

EchterAgo · 2024-02-17T12:25:14Z

Updated to not loop over the parent directories.

I noticed that the .editorconfig file does not match some of the files in the repo, for example for .py it specifies tab indent, but the setup.py file is space indented. I think we should change one of them to make it consistent. Should I raise an issue for that or do a PR?

axxel · 2024-02-17T12:31:07Z

I noticed that the .editorconfig file does not match some of the files in the repo, for example for .py it specifies tab indent, but the setup.py file is space indented. I think we should change one of them to make it consistent. Should I raise an issue for that or do a PR?

Please raise an issue. I'll have to look into it.

EchterAgo · 2024-02-17T14:33:38Z

Updated, sdist works again.

axxel · 2024-02-17T15:33:12Z

wrappers/python/CMakeLists.txt

@@ -17,8 +26,11 @@ if (NOT hasParent)
    option (BUILD_WRITERS "Build with writer support (encoders)" ON)
    option (BUILD_READERS "Build with reader support (decoders)" ON)
    set(BUILD_DEPENDENCIES "AUTO")
-    if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/core)


Looking at this line, it occurred to me that is does not make any sense in current HEAD because core always exists. Then I realized: your function is not necessary at all. Simply do the IS_SYMLINK check on core (which will fail in an sdist build) and if it is a symlink, then hardcode the ../../core path. Less lines of code to worry about.

Will change, but maybe not today.

Updated, I still kept the EXISTS check but made a variable CORE_PATH that is set conditionally based on whether ./core is a symlink.

Without this, the usage of the __FILE__ macro leaves the build directory in the binary. When building the Python extension with build isolation enabled this would lead to random paths in the binary.

This updates the zxing-cpp build to use a patched version of the zxing-cpp 2.2.1 release with a patch for reproducible build see zxing-cpp/zxing-cpp#730

axxel · 2024-02-27T13:35:15Z

@EchterAgo I just noticed that this change broke my debugging environment because the IDE can't find the sources anymore and displays assembly instead. A quick fix is to only apply the file-prefix-map flag in a release build. I guess this would still satisfy your requirements?

EchterAgo · 2024-02-27T15:20:12Z

Maybe try -ffile-prefix-map=${CMAKE_CURRENT_SOURCE_DIR}=. (note the trailing dot).

I think just using -fmacro-prefix-map instead of -ffile-prefix-map should also solve this. File prefix map sets the prefix map for all options, -fmacro-prefix-map, -fdebug-prefix-map, -fprofile-prefix-map and -fcanon-prefix-map. See https://gcc.gnu.org/onlinedocs/gcc/Overall-Options.html#:~:text=args%20cc1%20%E2%80%A6%E2%80%99.-,%2Dffile%2Dprefix%2Dmap,-%3Dold%3D

axxel · 2024-02-27T15:42:22Z

-fmacro-prefix-map works for me. thanks. will push later.

…ging) See #730.

EchterAgo mentioned this pull request Feb 17, 2024

Fix Appimage build files for deterministic build Electron-Cash/Electron-Cash#2790

Merged

EchterAgo force-pushed the reproducible_build branch from e1552d6 to fd360f6 Compare February 17, 2024 11:20

axxel reviewed Feb 17, 2024

View reviewed changes

core/CMakeLists.txt Outdated Show resolved Hide resolved

EchterAgo force-pushed the reproducible_build branch from 144be6b to b7ef2f1 Compare February 17, 2024 12:23

EchterAgo force-pushed the reproducible_build branch 2 times, most recently from 0ae543a to 2448126 Compare February 17, 2024 13:03

EchterAgo marked this pull request as ready for review February 17, 2024 14:32

axxel reviewed Feb 17, 2024

View reviewed changes

cmake: Make build reproducible across different build directories

1bda584

Without this, the usage of the __FILE__ macro leaves the build directory in the binary. When building the Python extension with build isolation enabled this would lead to random paths in the binary.

EchterAgo force-pushed the reproducible_build branch from 2448126 to 1bda584 Compare February 19, 2024 13:05

axxel merged commit 428aeb7 into zxing-cpp:master Feb 19, 2024

EchterAgo deleted the reproducible_build branch February 19, 2024 13:26

axxel added a commit that referenced this pull request Feb 27, 2024

cmake: replace -ffile-prefix-map -> -fmacro-prefix-map (fix IDE debug…

7b75061

…ging) See #730.

Uh oh!

cmake: Make build reproducible across different build directories #730

cmake: Make build reproducible across different build directories #730

Uh oh!

Conversation

EchterAgo commented Feb 17, 2024

Uh oh!

EchterAgo commented Feb 17, 2024

Uh oh!

EchterAgo commented Feb 17, 2024

Uh oh!

EchterAgo commented Feb 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EchterAgo commented Feb 17, 2024

Uh oh!

EchterAgo commented Feb 17, 2024

Uh oh!

EchterAgo commented Feb 17, 2024

Uh oh!

Uh oh!

EchterAgo commented Feb 17, 2024

Uh oh!

axxel commented Feb 17, 2024

Uh oh!

EchterAgo commented Feb 17, 2024

Uh oh!

axxel Feb 17, 2024

Choose a reason for hiding this comment

Uh oh!

EchterAgo Feb 17, 2024

Choose a reason for hiding this comment

Uh oh!

EchterAgo Feb 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

axxel commented Feb 27, 2024

Uh oh!

EchterAgo commented Feb 27, 2024

Uh oh!

axxel commented Feb 27, 2024

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

EchterAgo commented Feb 17, 2024 •

edited

Loading

EchterAgo Feb 19, 2024 •

edited

Loading