bpo-40280: Add --enable-wasm-dynamic-linking (GH-32253) #32253

tiran · 2022-04-02T15:53:11Z

https://bugs.python.org/issue40280

configure

hoodmane · 2022-04-02T16:45:43Z

Maybe this should be called --enable-emscripten-dynamic-linking since it doesn't work with wasi or other wasm targets?

tiran · 2022-04-02T17:20:39Z

Maybe this should be called --enable-emscripten-dynamic-linking since it doesn't work with wasi or other wasm targets?

WASI does not support dynamic linking yet. It may get it in the future, https://helda.helsinki.fi/bitstream/handle/10138/337740/Dynamic_linking_in_WebAssembly_Architecture_and_Performance_Evaluation.pdf .

hoodmane · 2022-04-02T17:25:18Z

But probably stuff like -s SIDE_MODULE=1 will always be emscripten specific, so even when other wasm platforms gain dynamic linking it is not clear if this flag can work for both.

hoodmane · 2022-04-02T17:28:51Z

I still think we should discuss with Emscripten whether we could set some environment variable like EMSCRIPTEN_RESPECT_SHARED and then use the -shared flag. Then other wasm targets would just ignore the EMSCRIPTEN_RESPECT_SHARED variable, whereas they would probably get angry when passed nonsense like -s SIDE_MODULE=1 as a command line argument.

cf emscripten-core/emscripten#16481

@sbc100

tiran · 2022-04-02T17:50:17Z

I still think we should discuss with Emscripten whether we could set some environment variable like EMSCRIPTEN_RESPECT_SHARED and then use the -shared flag. Then other wasm targets would just ignore the EMSCRIPTEN_RESPECT_SHARED variable, whereas they would probably get angry when passed nonsense like -s SIDE_MODULE=1 as a command line argument.

WASI wouldn't see the Emscripten flags. We can easily extend our configure file later to do something like:

AS_VAR_IF([enable_wasm_dynamic_linking], [yes], [
  AS_CASE([$ac_sys_system],
    [Emscripten], [BLDSHARED='$(CC) -shared -s SIDE_MODULE=1 -s WASM=1'],
    [WASI], [BLDSHARED='$(CC) --shared --wasi-magic-linking-flag']
  )
])

hoodmane · 2022-04-02T17:52:27Z

Right, makes sense that autotools has ways of dealing with different compilers needing different flags.

erlend-aasland

AC changes look good to me!

configure.ac

tiran · 2022-04-03T18:52:17Z

The PR drops ac_cv_func_dlopen=no from our config.site override and will allow us to test dynamic linking in the future.

@hoodmane does the PR help you, or at least not cause any new problems?

hoodmane · 2022-04-03T19:04:53Z

I'm having trouble testing on tip of tree because of the changes to _sre and regex. I need a docker image like 3.11.0a6-slim-buster but with tip of tree. Or maybe I can just locate the changes to _sre and regex since alpha6 and revert them? I can still build locally against a locally built copy of tot Python but it's inconvenient not to have the CI working.

hoodmane · 2022-04-03T19:05:38Z

Or maybe I can cherry-pick this commit onto 3.11.0a6?

tiran · 2022-04-03T19:15:34Z

What's the problem with _sre module? GH-32177 converted the modules to a package. The change should be fully backwards compatible.

hoodmane · 2022-04-03T19:22:27Z

I get failures at
assert _sre.MAGIC == MAGIC, "SRE module mismatch" on line 17 of Lib/sre_compile.py

tiran · 2022-04-03T19:43:03Z

There is no line 17 in Lib/sre_compile.py any more. The file got moved and replaced with a shorter stub. Some files on your system are out of date.

hoodmane · 2022-04-03T19:45:09Z

Some files on your system are out of date.

Yeah hence I need a docker image with tip of tree Python because my v3.11.0a6 system is out of date

tiran · 2022-04-03T19:54:52Z

Are you familiar with out-of-tree builds? We compile a build Python interpreter to bootstrap Emscripten cross build from the same source check, https://github.com/ethanhs/python-wasm/blob/main/build-python-build.sh and https://github.com/ethanhs/python-wasm/blob/main/build-python-emscripten-browser.sh . The trick is ../../configure.

hoodmane · 2022-04-03T19:57:37Z

Are you familiar with out-of-tree builds?

Apparently not.

The trick is ../../configure

Is it important that there are two layers builddir/build?

hoodmane · 2022-04-03T20:00:57Z

Also, are there docs on out of tree build somewhere that I should read?

tiran · 2022-04-03T20:06:01Z

No, the paths are not important. Any directory structure will do. It's only important that you run make clean in the source directory first. And don't use build, it gets easily confused with distutils's build directory. ouf-of-tree builds (aka VPATH builds) are an autoconf feature. https://www.gnu.org/software/automake/manual/html_node/VPATH-Builds.html

hoodmane · 2022-04-03T20:08:35Z

It's only important that you run make clean in the source directory first.

Do I need to do this if I have cloned a fresh copy of python/cpython?

tiran · 2022-04-03T20:13:09Z

Do I need to do this if I have cloned a fresh copy of python/cpython?

No, a fresh clone is clean.

hoodmane · 2022-04-03T22:15:35Z

I tried to follow your directions for this:
https://github.com/hoodmane/pyodide/blob/python-tot/cpython/Makefile
But the build still fails with:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/src/cpython/installs/python-3.11.0dev0/lib/python3.11/lib2to3/pgen2/driver.py", line 21, in <module>
    import logging
    ^^^^^^^^^^^^^^
  File "/src/cpython/installs/python-3.11.0dev0/lib/python3.11/logging/__init__.py", line 26, in <module>
    import sys, os, time, io, re, traceback, warnings, weakref, collections.abc
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/cpython/installs/python-3.11.0dev0/lib/python3.11/re/__init__.py", line 125, in <module>
    from . import _compiler, _parser
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/cpython/installs/python-3.11.0dev0/lib/python3.11/re/_compiler.py", line 17, in <module>
    assert _sre.MAGIC == MAGIC, "SRE module mismatch"
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: SRE module mismatch

Maybe you can tell me what I'm doing wrong?
https://app.circleci.com/pipelines/github/hoodmane/pyodide/2654/workflows/282f60fd-a7e5-4d22-8688-09f2daddcf03/jobs/28765

hoodmane · 2022-04-03T22:30:04Z

Ah maybe the problem is just buggy make recipes.

hoodmane · 2022-04-03T22:40:45Z

I think the_sre.MAGIC crash is triggered while it is trying to log some other error, something about pybuilddir.txt.

hoodmane · 2022-04-03T22:50:31Z

Okay the problem is on Makefile.pre.in line 2032:

	-PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
		$(PYTHON_FOR_BUILD) -Wi $(DESTDIR)$(LIBDEST)/compileall.py \
		-j0 -d $(LIBDEST) -f \
		-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
		$(DESTDIR)$(LIBDEST)

Note that this is the build Python (PYTHON_FOR_BUILD) running with the target Python's standard library (-PYTHONPATH=$(DESTDIR)$(LIBDEST)).

hoodmane · 2022-04-03T22:57:07Z

Or something. I tried manually removing PYTHONPATH from that and it still broke in the same way so I don't know.

tiran · 2022-04-04T17:31:23Z

Is the error coming from libinstall target? Our python-wasm PoC does not use make install. We are building WASM port in-place and then copy files around manually. I'll investigate after next alpha is out.

I'm merging the PR now to get it into the upcoming alpha. It's unlike to cause you any problems and allows us to test dynamic linking more easily.

hoodmane · 2022-04-04T17:43:44Z

Yeah I think make libinstall has a bug where when cross compiling it uses the Python standard library of the target Python with the Python executable of the build Python. As long as the target Python and the build Python are compatible for the small number of packages it needs, it works okay. But if there are changes to certain libraries, it breaks.

tiran · 2022-04-05T07:39:05Z

Congratulations, you have discovered an annoying issue with development during Python's alpha phase. :)

Code is moving fast and breaks often. You should rebuild your build Python interpreter every time you update your checkout or switch branches. It's going to stabilize during beta and won't be an issue once Python 3.11 reaches RC and final stages. I recommend config cache (./configure -C) and ccache (export EM_COMPILER_WRAPPER=ccache + export PATH=/usr/lib/ccache:$PATH) to speed up configure and build steps.

hoodmane · 2022-04-05T18:06:47Z

Makes sense. At least Python is fast to build. The main annoyance is that if I want to get the CI running I have to set up building Python either in my Dockerfile or in my pipeline.

hoodmane · 2022-04-05T19:00:56Z

Hmm, I'm still having trouble. Even building a x86 Linux Python and passing it as --with-build-python=, the build is generating invocations to /usr/local/bin/python3.11. The issue seems to be that ./configure is generating a slightly buggy Makefile. A bunch of variables are all on the same line:

PYTHON_FOR_BUILD=_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) /src/cpython/build/Python-3.11.0dev0/builddir/build-host/python

I think this should be:

PYTHON_FOR_BUILD=/src/cpython/build/Python-3.11.0dev0/builddir/build-host/python
_PYTHON_PROJECT_BASE=$(abs_builddir) 
_PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) 
PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib 
_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH)

brettcannon · 2022-04-05T19:17:43Z

@hoodmane at this point it's probably better to open an issue, else this is liable to get lost since the PR has already been merged.

hoodmane · 2022-04-05T19:27:49Z

Okay I opened an issue here: https://bugs.python.org/issue47232

sbc100 · 2022-04-13T19:17:45Z

configure.ac

+dnl Emscripten's emconfigure sets LDSHARED. Set BLDSHARED outside the
+dnl test -z $LDSHARED block to configure BLDSHARED for side module support.
+if test "$enable_wasm_dynamic_linking" = "yes" -a "$ac_sys_system" = "Emscripten"; then
+  BLDSHARED='$(CC) -shared -sSIDE_MODULE=1 -sWASM=1'


Is there some reason why you add -sWASM=1 here. Since its the default its unlikely you need it.

Also, you can simplify this command line (and the ones elsewhere in this file) by dropping with =1 at the end of all the settings.

Python shared extensions have a .so extension. emcc creates a .so JavaScript file and a .wasm file along the .so file. We want the .so file to contain the .wasm code.

This is not what the -sWASM=1 option does though. The -sSIDE_MODULE option is enough to ensure that the output is just a wasm file.

If you are building a normal application without -sSIDE_MODULE then you always get a JS file and a wasm file side-by-side (even with -sWASM=1). The -sWASM=1 option (which is the default) simply means "don't convert the wasm file to JS using wasm2js".. which is what happens if you set -sWASM=0 (which is never the default).

-sSIDE_MODULE on its own should always build just a wasm file.

Please note this PR is merged already, so it would be best to open a new issue to track this discussion if there's something to change.

bedevere-bot added the awaiting core review label Apr 2, 2022

the-knights-who-say-ni added the CLA signed label Apr 2, 2022

tiran requested review from brettcannon and erlend-aasland April 2, 2022 15:53

tiran force-pushed the bpo-40280-wasm-dynamic-linking branch from 7cfc9d8 to ad6a484 Compare April 2, 2022 15:54

tiran mentioned this pull request Apr 2, 2022

Review CPython patches / tests and contribute upstream pyodide/pyodide#2000

Closed

pmp-p reviewed Apr 2, 2022

View reviewed changes

configure Show resolved Hide resolved

pmp-p mentioned this pull request Apr 2, 2022

Get ctypes working emmatyping/python-wasm#68

Open

pmp-p approved these changes Apr 2, 2022

View reviewed changes

erlend-aasland approved these changes Apr 2, 2022

View reviewed changes

configure.ac Outdated Show resolved Hide resolved

tiran added 5 commits April 3, 2022 18:35

bpo-40280: Add --enable-wasm-dynamic-linking

dbc74ad

Set to missing, add breadcrumbs for WASI

9d5b67a

Typo

17998ea

Tab vs. space

09e5ecf

Drop space between -s and argument

c1a4260

tiran force-pushed the bpo-40280-wasm-dynamic-linking branch from e03d117 to c1a4260 Compare April 3, 2022 17:25

tiran changed the title ~~bpo-40280: Add --enable-wasm-dynamic-linking~~ bpo-40280: Add --enable-wasm-dynamic-linking (GH-32253) Apr 4, 2022

tiran merged commit c9844cb into python:main Apr 4, 2022

tiran deleted the bpo-40280-wasm-dynamic-linking branch April 4, 2022 17:31

bedevere-bot removed the awaiting core review label Apr 4, 2022

sbc100 reviewed Apr 13, 2022

View reviewed changes

SimonBiggs mannequin mentioned this pull request Jun 7, 2022

Consider supporting emscripten/webassembly as a build target #84461

Closed

bpo-40280: Add --enable-wasm-dynamic-linking (GH-32253) #32253

bpo-40280: Add --enable-wasm-dynamic-linking (GH-32253) #32253

Conversation

tiran commented Apr 2, 2022 • edited by bedevere-bot Loading

hoodmane commented Apr 2, 2022

tiran commented Apr 2, 2022

hoodmane commented Apr 2, 2022

hoodmane commented Apr 2, 2022

tiran commented Apr 2, 2022

hoodmane commented Apr 2, 2022

erlend-aasland left a comment

Choose a reason for hiding this comment

tiran commented Apr 3, 2022

hoodmane commented Apr 3, 2022

hoodmane commented Apr 3, 2022

tiran commented Apr 3, 2022

hoodmane commented Apr 3, 2022

tiran commented Apr 3, 2022

hoodmane commented Apr 3, 2022

tiran commented Apr 3, 2022

hoodmane commented Apr 3, 2022

hoodmane commented Apr 3, 2022

tiran commented Apr 3, 2022

hoodmane commented Apr 3, 2022

tiran commented Apr 3, 2022

hoodmane commented Apr 3, 2022

hoodmane commented Apr 3, 2022

hoodmane commented Apr 3, 2022

hoodmane commented Apr 3, 2022

hoodmane commented Apr 3, 2022

tiran commented Apr 4, 2022

hoodmane commented Apr 4, 2022

tiran commented Apr 5, 2022

hoodmane commented Apr 5, 2022

hoodmane commented Apr 5, 2022 • edited Loading

brettcannon commented Apr 5, 2022

hoodmane commented Apr 5, 2022

sbc100 Apr 13, 2022

Choose a reason for hiding this comment

tiran Apr 13, 2022

Choose a reason for hiding this comment

sbc100 Apr 13, 2022

Choose a reason for hiding this comment

brettcannon Apr 14, 2022

Choose a reason for hiding this comment

tiran commented Apr 2, 2022 •

edited by bedevere-bot

Loading

hoodmane commented Apr 5, 2022 •

edited

Loading