Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wasm] Add WASILibc module and its build support #66539

Conversation

kateinoigakukun
Copy link
Member

wasi-libc is a libc implementation widely used in WASI toolchains. This patch adds a new Swift and C module for it to import and use it in Swift code. This module is intentionally separate from the existing Glibc module because wasi-libc is actually based on musl libc.

This PR changed ClangImporter also to inject the modulemap and auxiliary header for wasi-libc into VFS to recognize it as well as Glibc.

@kateinoigakukun
Copy link
Member Author

@swift-ci Please smoke test

@MaxDesiatov
Copy link
Contributor

@swift-ci build toolchain

@kateinoigakukun kateinoigakukun marked this pull request as ready for review June 11, 2023 07:35
@finagolfin
Copy link
Contributor

@kateinoigakukun, do you mind updating this to remove the SwiftWASILibc.h header file and just listing the wasi-libc headers in the modulemap, as we discussed before and @egorzhdan has already done with the libstdc++ modulemap?

@finagolfin
Copy link
Contributor

I just tried switching glibc.modulemap to listing the Bionic headers directly and most everything from the compiler validation suite kept working on my Android phone. However, it broke around 70 C++ Interop tests that worked before with the SwiftGlibc.h header, outputting this error:

<testcase classname="Swift(android-aarch64).Interop/Cxx/class" name="constructors-executable.swift" time="8.35">
  <failure><![CDATA[Script:
--
: 'RUN: at line 1';   rm -rf "/data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/test-android-aarch64/Interop/Cxx/class/Output/constructors-executable.swift.tmp" && mkdir -p "/data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/test-android-aarch64/Interop/Cxx/class/Output/constructors-executable.swift.tmp" && /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/bin/swiftc -target aarch64-unknown-linux-android -toolchain-stdlib-rpath  -module-cache-path /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/swift-test-results/aarch64-unknown-linux-android/clang-module-cache -swift-version 4  -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 9999:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.1:macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.2:macOS 10.15.4, iOS 13.4, watchOS 6.2, tvOS 13.4' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.3:macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.4:macOS 11.3, iOS 14.5, watchOS 7.4, tvOS 14.5' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.5:macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.6:macOS 12.3, iOS 15.4, watchOS 8.5, tvOS 15.4' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.7:macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.8:macOS 13.3, iOS 16.4, watchOS 9.4, tvOS 16.4' -Xfrontend -define-availability -Xfrontend 'SwiftStdlib 5.9:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999'  -module-cache-path /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/swift-test-results/aarch64-unknown-linux-android/clang-module-cache /data/data/com.termux/files/home/swift/test/Interop/Cxx/class/constructors-executable.swift -I /data/data/com.termux/files/home/swift/test/Interop/Cxx/class/Inputs/ -Xfrontend -enable-experimental-cxx-interop -o /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/test-android-aarch64/Interop/Cxx/class/Output/constructors-executable.swift.tmp/a.out -module-name main  && echo /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/test-android-aarch64/Interop/Cxx/class/Output/constructors-executable.swift.tmp/a.out && /usr/bin/env LD_LIBRARY_PATH='/data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/lib/swift/android:/data/data/com.termux/files/home/build/Ninja-Release/libdispatch-android-aarch64'  /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/test-android-aarch64/Interop/Cxx/class/Output/constructors-executable.swift.tmp/a.out
--
Exit Code: 1

Command Output (stderr):
--
<unknown>:0: warning: the '-enable-experimental-cxx-interop' flag is deprecated; please pass '-cxx-interoperability-mode=' instead
<unknown>:0: note: Swift will maintain source compatibility for imported APIs based on the selected compatibility mode, so updating the Swift compiler will not change how APIs are imported
/data/data/com.termux/files/usr/include/bits/termios_inlines.h:32:10: note: while building module 'std' imported from /data/data/com.termux/files/usr/include/bits/termios_inlines.h:32:
#include <errno.h>
         ^
<module-includes>:2:10: note: in file included from <module-includes>:2:
#include "ctype.h"
         ^
/data/data/com.termux/files/usr/include/c++/v1/ctype.h:38:15: error: cyclic dependency in module 'SwiftGlibc': SwiftGlibc -> std -> SwiftGlibc
#include_next <ctype.h>
              ^
<module-includes>:11:10: note: in file included from <module-includes>:11:
#include "pty.h"
         ^
/data/data/com.termux/files/usr/include/pty.h:38:10: note: in file included from /data/data/com.termux/files/usr/include/pty.h:38:
#include <termios.h>
         ^
/data/data/com.termux/files/usr/include/termios.h:154:10: note: in file included from /data/data/com.termux/files/usr/include/termios.h:154:
#include <android/legacy_termios_inlines.h>
         ^
/data/data/com.termux/files/usr/include/android/legacy_termios_inlines.h:43:10: note: in file included from /data/data/com.termux/files/usr/include/android/legacy_termios_inlines.h:43:
#include <bits/termios_inlines.h>
         ^
/data/data/com.termux/files/usr/include/bits/termios_inlines.h:32:10: error: could not build module 'std'
#include <errno.h>
         ^
<unknown>:0: error: could not build C module 'SwiftGlibc'
<module-includes>:1:10: note: in file included from <module-includes>:1:
#include "LibcOverlayShims.h"
         ^

--
]]></failure>
</testcase>

This has to do with the two different ctype.h headers in libc++ versus Bionic. I then tried removing the ones that forwarded to the C header with #include_next and failed from glibc.modulemap, until I got to an error that the libc++ fenv.h forwarded to the Bionic one, which ultimately included sys/types.h, causing another cyclic dependency error like above, ie having libc++ headers forwarding to libc headers alone causes build issues.

@egorzhdan, how do you work around this: don't have any libc++ headers that forward to libc headers in the C++ modulemap?

The other failure is some oddity related to stdbool.h:

<failure><![CDATA[Script:
--
: 'RUN: at line 1';   /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/bin/swift-frontend -target aarch64-unknown-linux-android  -module-cache-path /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/swift-test-results/aarch64-unknown-linux-android/clang-module-cache -swift-version 4  -define-availability 'SwiftStdlib 9999:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999' -define-availability 'SwiftStdlib 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2' -define-availability 'SwiftStdlib 5.1:macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0' -define-availability 'SwiftStdlib 5.2:macOS 10.15.4, iOS 13.4, watchOS 6.2, tvOS 13.4' -define-availability 'SwiftStdlib 5.3:macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0' -define-availability 'SwiftStdlib 5.4:macOS 11.3, iOS 14.5, watchOS 7.4, tvOS 14.5' -define-availability 'SwiftStdlib 5.5:macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0' -define-availability 'SwiftStdlib 5.6:macOS 12.3, iOS 15.4, watchOS 8.5, tvOS 15.4' -define-availability 'SwiftStdlib 5.7:macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0' -define-availability 'SwiftStdlib 5.8:macOS 13.3, iOS 16.4, watchOS 9.4, tvOS 16.4' -define-availability 'SwiftStdlib 5.9:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999'  -typo-correction-limit 10  -import-objc-header /data/data/com.termux/files/home/swift/test/IRGen/Inputs/huge_c_type.h /data/data/com.termux/files/home/swift/test/IRGen/huge_c_type.swift -disable-llvm-optzns -emit-ir | /data/data/com.termux/files/usr/bin/python3.11 /data/data/com.termux/files/home/swift/utils/PathSanitizingFileCheck --allow-unused-prefixes --sanitize BUILD_DIR=/data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64 --sanitize SOURCE_DIR=/data/data/com.termux/files/home/swift --use-filecheck /data/data/com.termux/files/home/build/Ninja-Release/llvm-android-aarch64/bin/FileCheck  /data/data/com.termux/files/home/swift/test/IRGen/huge_c_type.swift
: 'RUN: at line 4';   /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/bin/swift-frontend -target aarch64-unknown-linux-android  -module-cache-path /data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/swift-test-results/aarch64-unknown-linux-android/clang-module-cache -swift-version 4  -define-availability 'SwiftStdlib 9999:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999' -define-availability 'SwiftStdlib 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2' -define-availability 'SwiftStdlib 5.1:macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0' -define-availability 'SwiftStdlib 5.2:macOS 10.15.4, iOS 13.4, watchOS 6.2, tvOS 13.4' -define-availability 'SwiftStdlib 5.3:macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0' -define-availability 'SwiftStdlib 5.4:macOS 11.3, iOS 14.5, watchOS 7.4, tvOS 14.5' -define-availability 'SwiftStdlib 5.5:macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0' -define-availability 'SwiftStdlib 5.6:macOS 12.3, iOS 15.4, watchOS 8.5, tvOS 15.4' -define-availability 'SwiftStdlib 5.7:macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0' -define-availability 'SwiftStdlib 5.8:macOS 13.3, iOS 16.4, watchOS 9.4, tvOS 16.4' -define-availability 'SwiftStdlib 5.9:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999'  -typo-correction-limit 10  -import-objc-header /data/data/com.termux/files/home/swift/test/IRGen/Inputs/huge_c_type.h /data/data/com.termux/files/home/swift/test/IRGen/huge_c_type.swift -c
--
Exit Code: 2

Command Output (stderr):
--
/data/data/com.termux/files/home/swift/test/IRGen/Inputs/huge_c_type.h:3:17: error: cannot combine with previous 'type-name' declaration specifier
typedef uint8_t bool;
                ^
/data/data/com.termux/files/home/build/Ninja-Release/swift-android-aarch64/lib/swift/clang/include/stdbool.h:20:14: note: expanded from macro 'bool'
#define bool _Bool
             ^
/data/data/com.termux/files/home/swift/test/IRGen/Inputs/huge_c_type.h:3:1: warning: typedef requires a name
typedef uint8_t bool;
^
<unknown>:0: error: failed to import bridging header '/data/data/com.termux/files/home/swift/test/IRGen/Inputs/huge_c_type.h'
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /data/data/com.termux/files/home/build/Ninja-Release/llvm-android-aarch64/bin/FileCheck --allow-unused-prefixes /data/data/com.termux/files/home/swift/test/IRGen/huge_c_type.swift

--
]]></failure>

@kateinoigakukun, I don't know if you run these C/C++ interop tests on wasm, but everything else in the validation suite worked for me.

I can slap up my libc modulemap changes if you guys are interested in trying it with your libc's, particularly @3405691582 and @mhjacobson with the BSDs and maybe @stevapple with Windows.

@finagolfin
Copy link
Contributor

I've submitted my glibc modulemap changes in #66665, you may want to try something similar with wasi-libc and see if it works.

@MaxDesiatov MaxDesiatov added the WebAssembly Platform: WebAssembly label Sep 19, 2023
kateinoigakukun and others added 2 commits September 23, 2023 11:51
[wasi-libc](https://github.com/WebAssembly/wasi-libc) is a libc
implementation widely used in WASI toolchains. This patch adds a new
module named WASILibc. Some extra emulation libraries are also added as
wasi_emulated_xxx.
@finagolfin
Copy link
Contributor

Nice, working well with the headers listed in the module map, @kateinoigakukun?

@egorzhdan, please review.

@kateinoigakukun
Copy link
Member Author

@swift-ci Please smoke test

@MaxDesiatov
Copy link
Contributor

@swift-ci build toolchain

Copy link
Contributor

@al45tair al45tair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@kateinoigakukun kateinoigakukun merged commit a7d006e into swiftlang:main Sep 25, 2023
6 checks passed
@kateinoigakukun kateinoigakukun deleted the katei/patch-series/wasi-libc-module branch September 25, 2023 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
WebAssembly Platform: WebAssembly
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants