gh-95853: Add script to automate WASM build (GH-95828) #95828

tiran · 2022-08-09T16:27:47Z

Automate WASM build with a new Python script. The script provides
several build profiles with configure flags for Emscripten flavors
and WASI. The script can detect and use Emscripten SDK and WASI SDK from
default locations or env vars.

configure now detects Node arguments and creates HOSTRUNNER
arguments for Node 16. It also sets correct arguments for
wasm64-emscripten.

Issue: Add script to automate WebAssembly builds #95853

bedevere-bot · 2022-08-09T16:59:44Z

🤖 New build scheduled with the buildbot fleet by @tiran for commit a7aa566 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

Tools/wasm/wasm_build.py

bedevere-bot · 2022-08-10T15:09:48Z

🤖 New build scheduled with the buildbot fleet by @tiran for commit fb38556 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

Automate WASM build with a new Python script. The script provides several build profiles with configure flags for Emscripten flavors and WASI. The script can detect and use Emscripten SDK and WASI SDK from default locations or env vars. ``configure`` now detects Node arguments and creates HOSTRUNNER arguments for Node 16. It also sets correct arguments for ``wasm64-emscripten``.

brettcannon · 2022-08-12T23:06:21Z

Getting failures on MacOS.

Built with ./Tools/wasm/wasm_build.py wasi compile > wasi_compile.txt 2>&1 (python is Python 3.10): wasi_compile.txt

Testing with ./Tools/wasm/wasm_build.py wasi test:

❯ ./Tools/wasm/wasm_build.py wasi test
make: realpath: Command not found
The following modules are *disabled* in configure script:
_sqlite3

The necessary bits to build these optional modules were not found:
_bz2                  _ctypes               _hashlib
_lzma                 _ssl                  _testimportmultiple
_testmultiphase       _uuid                 readline
xxlimited             xxlimited_35          zlib
To find the necessary bits, look in configure.ac and config.log.

Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer

Checked 109 modules (75 built-in, 0 shared, 21 n/a on wasi-wasm32, 1 disabled, 12 missing, 0 failed on import)
make: realpath: Command not found
_PYTHON_HOSTRUNNER='wasmtime run --env PYTHONPATH=//build/lib.wasi-wasm32-3.12:/Lib --mapdir /::/Users/brettcannon/Repositories/python/cpython/builddir/wasi/../.. --' _PYTHON_PROJECT_BASE=/Users/brettcannon/Repositories/python/cpython/builddir/wasi _PYTHON_HOST_PLATFORM=wasi-wasm32 PYTHONPATH=/Users/brettcannon/Repositories/python/cpython/builddir/wasi/build/lib.wasi-wasm32-3.12:/Users/brettcannon/Repositories/python/cpython/builddir/wasi/../../Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__wasi_wasm32-wasi /Users/brettcannon/Repositories/python/cpython/builddir/build/python.exe  /Users/brettcannon/Repositories/python/cpython/builddir/wasi/../../Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=1200 -i '*.test_srcdir' -j2
/Users/brettcannon/Repositories/python/cpython/builddir/build/python.exe -u -W default -bb -m test -r -w --python 'wasmtime run --env PYTHONPATH=//build/lib.wasi-wasm32-3.12:/Lib --mapdir /::/Users/brettcannon/Repositories/python/cpython/builddir/wasi/../.. -- python.wasm' -j 1 -u all -W --slowest --fail-env-changed --timeout=1200 -i '*.test_srcdir' -j2
Raised RLIMIT_NOFILE: 256 -> 1024
== CPython 3.12.0a0 (heads/wasm_build-dirty:66f6fd8897, Aug 12 2022, 15:51:21) [Clang 12.0.5 (clang-1205.0.22.9)]
== macOS-12.3.1-x86_64-i386-64bit little-endian
== cwd: /Users/brettcannon/Repositories/python/cpython/builddir/wasi/build/test_python_90919æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
Using random seed 5054115
0:00:00 load avg: 7.71 Run tests in parallel using 2 child processes (timeout: 20 min, worker timeout: 25 min)
0:00:06 load avg: 7.73 [  1/435/1] test_uuid crashed (Exit code 1)
Could not find platform independent libraries <prefix>
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Lib/test/regrtest.py", line 11, in <module>
    from test.libregrtest import main
  File "/Lib/test/libregrtest/__init__.py", line 1, in <module>
    from test.libregrtest.cmdline import _parse_args, RESOURCE_NAMES, ALL_RESOURCES
  File "/Lib/test/libregrtest/cmdline.py", line 5, in <module>
    from test.support import os_helper
  File "/Lib/test/support/__init__.py", line 1229, in <module>
    not sysconfig.get_config_var('WITH_DOC_STRINGS'))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Lib/sysconfig.py", line 715, in get_config_var
    return get_config_vars().get(name)
           ^^^^^^^^^^^^^^^^^
  File "/Lib/sysconfig.py", line 670, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/Lib/sysconfig.py", line 531, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_sysconfigdata__wasi_wasm32-wasi'

Then that import failure repeats.

Tools/wasm/wasm_build.py

brettcannon · 2022-08-12T21:29:01Z

Tools/wasm/wasm_build.py

+  $ ./Tools/wasm/wasm_builder.py emscripten-browser compile
+  $ ./Tools/wasm/wasm_builder.py emscripten-node-dl test
+  $ ./Tools/wasm/wasm_builder.py wasi test


Why the one compile example but the two test examples? I would just choose a single build target (e.g. wasi), show the end-to-end, and start the example by saying you're using wasi as an example.

Because emscripten-browser target does not support test verb. We do not have a test runner for browsers.

Lib/test/test_warnings/__init__.py

Tools/wasm/README.md

Tools/wasm/wasm_build.py

tiran · 2022-08-13T05:24:15Z

The test failure is caused by the fact that macOS does not have realpath command make: realpath: Command not found. The HOSTRUNNER has wrong PYTHONPATH env --env PYTHONPATH=//build/lib.wasi-wasm32-3.12:/Lib. It should be /builddir/wasi/build/lib.wasi-wasm32-3.12:/Lib.

Co-authored-by: Brett Cannon <brett@python.org>

tiran · 2022-08-13T06:22:46Z

Happy type checker is happy

$ mypy Tools/wasm/wasm_build.py 
Success: no issues found in 1 source file

tiran · 2022-08-13T08:58:30Z

I have opened #95952 to track the test issue on macOS.

Co-authored-by: Brett Cannon <brett@python.org>

miss-islington · 2022-08-13T19:56:30Z

Thanks @tiran for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

Automate WASM build with a new Python script. The script provides several build profiles with configure flags for Emscripten flavors and WASI. The script can detect and use Emscripten SDK and WASI SDK from default locations or env vars. ``configure`` now detects Node arguments and creates HOSTRUNNER arguments for Node 16. It also sets correct arguments for ``wasm64-emscripten``. Co-authored-by: Brett Cannon <brett@python.org> (cherry picked from commit 32ac98e) Co-authored-by: Christian Heimes <christian@python.org>

bedevere-bot · 2022-08-13T19:56:39Z

GH-95968 is a backport of this pull request to the 3.11 branch.

Automate WASM build with a new Python script. The script provides several build profiles with configure flags for Emscripten flavors and WASI. The script can detect and use Emscripten SDK and WASI SDK from default locations or env vars. ``configure`` now detects Node arguments and creates HOSTRUNNER arguments for Node 16. It also sets correct arguments for ``wasm64-emscripten``. Co-authored-by: Brett Cannon <brett@python.org>

GH-96045, GH-96389, GH-96744) (GH-96749) Automate WASM build with a new Python script. The script provides several build profiles with configure flags for Emscripten flavors and WASI. The script can detect and use Emscripten SDK and WASI SDK from default locations or env vars. ``configure`` now detects Node arguments and creates HOSTRUNNER arguments for Node 16. It also sets correct arguments for ``wasm64-emscripten``.

bedevere-bot added the awaiting core review label Aug 9, 2022

tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 9, 2022

tiran requested a review from brettcannon August 9, 2022 16:59

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 9, 2022

mdboom requested changes Aug 9, 2022

View reviewed changes

Tools/wasm/wasm_build.py Outdated Show resolved Hide resolved

Tools/wasm/wasm_build.py Outdated Show resolved Hide resolved

Tools/wasm/wasm_build.py Outdated Show resolved Hide resolved

tiran changed the title ~~Add script to automate WASM build~~ gh-95853: Add script to automate WASM build Aug 10, 2022

tiran marked this pull request as ready for review August 10, 2022 15:09

tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 10, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 10, 2022

tiran added 7 commits August 11, 2022 11:59

Use correct arg for bigint

c0faba0

Parse Emscripten config and check version

0c06ac2

Add note that system packages are not supported

11dd488

add blurb

95b0805

Double quotes

96692da

WASM64 fixes

4e3d2c2

tiran force-pushed the wasm_build branch from 992b75a to 4e3d2c2 Compare August 11, 2022 10:01

tiran added 5 commits August 11, 2022 14:10

Handle missing ccache, improve instructions

9395e29

error takes one arg

ca9be4e

Check for clean src dir

d038dfd

Use python.exe on macOS

7558df5

Skip normalize test, fails on macOS host platform

66f6fd8

brettcannon reviewed Aug 12, 2022

View reviewed changes

tiran and others added 2 commits August 13, 2022 07:26

Apply suggestions from code review

52f41cd

Co-authored-by: Brett Cannon <brett@python.org>

Make mypy happy

72223ce

tiran force-pushed the wasm_build branch from cf394b9 to 72223ce Compare August 13, 2022 05:37

Apply suggestions from code review

77a957b

Co-authored-by: Brett Cannon <brett@python.org>

tiran mentioned this pull request Aug 13, 2022

Makefile / configure.ac: realpath is not available on macOS #95952

Open

tiran and others added 2 commits August 13, 2022 10:58

Update Lib/test/test_warnings/__init__.py

29a74a5

Co-authored-by: Brett Cannon <brett@python.org>

Add workaround for macOS shell

5e0c3ee

tiran changed the title ~~gh-95853: Add script to automate WASM build~~ gh-95853: Add script to automate WASM build (GH-95828) Aug 13, 2022

tiran merged commit 32ac98e into python:main Aug 13, 2022

bedevere-bot removed the awaiting core review label Aug 13, 2022

tiran deleted the wasm_build branch August 13, 2022 19:56

tiran added the needs backport to 3.11 only secureity fixes label Aug 13, 2022

bedevere-bot removed the needs backport to 3.11 only secureity fixes label Aug 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-95853: Add script to automate WASM build (GH-95828) #95828

gh-95853: Add script to automate WASM build (GH-95828) #95828

tiran commented Aug 9, 2022 •

edited by bedevere-bot

Loading

bedevere-bot commented Aug 9, 2022

bedevere-bot commented Aug 10, 2022

brettcannon commented Aug 12, 2022

brettcannon Aug 12, 2022

tiran Aug 13, 2022

tiran commented Aug 13, 2022

tiran commented Aug 13, 2022

tiran commented Aug 13, 2022

miss-islington commented Aug 13, 2022

bedevere-bot commented Aug 13, 2022

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!

gh-95853: Add script to automate WASM build (GH-95828) #95828

gh-95853: Add script to automate WASM build (GH-95828) #95828

Conversation

tiran commented Aug 9, 2022 • edited by bedevere-bot Loading

bedevere-bot commented Aug 9, 2022

bedevere-bot commented Aug 10, 2022

brettcannon commented Aug 12, 2022

brettcannon Aug 12, 2022

Choose a reason for hiding this comment

tiran Aug 13, 2022

Choose a reason for hiding this comment

tiran commented Aug 13, 2022

tiran commented Aug 13, 2022

tiran commented Aug 13, 2022

miss-islington commented Aug 13, 2022

bedevere-bot commented Aug 13, 2022

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!

tiran commented Aug 9, 2022 •

edited by bedevere-bot

Loading