CLI and networking support #119

adamziel · 2023-01-25T11:33:04Z

Description

Adds support for CLI SAPI and networking in node.js:

> npm run build

> node ./build-cli/php-cli.js -r 'echo "Hello from PHP !";'
Hello from PHP !

> node ./build-cli/php-cli.js -r 'echo substr(file_get_contents("https://wordpress.org"), 0, 16);'
<!DOCTYPE html>

> node ./build-cli/php-cli.js -r 'echo phpversion();'
8.2.0-dev

> PHP=5.6 node ./build-cli/php-cli.js -r 'echo phpversion();'
5.6.40

Highlights:

Networking is supported (including MySQL and HTTPS)
Switching PHP versions is supported.
Most WordPress PHPUnit tests pass. The failures are caused by missing extensions and a few misconfigured settings
PHP Interactive mode is supported but the arrow keys don't work
wp-cli works

In broad strokes:

CLI SAPI is compiled with libedit (readline replacement) and ncurses.
Network calls are asynchronous. Emscripten's Asyncify enables calling asynchronous code from synchronous code. TCP sockets are shimmed with a WebSocket connection to a built-in proxy server running on localhost. It supports data transfer, arbitrary connection targets, and setting a few TCP socket options.
PHP's OpenSSL uses the same CA certs as Node.js
PHP 5.6 is patched to work with OpenSSL 1.1.0 and many other small patches are introduced. For more details, see patches overview, Dockerfile, and phpwasm-emscripten-library.js

Future work:

PHP Interactive server isn't supported yet. Adding support is a matter of making the incoming connection polling non-blocking using Asyncify.
Use a more recent OpenSSL version
Better support for CLI interactive mode

dmsnell · 2023-01-27T20:34:46Z

Is there a discussion of where these patches come from or why they are what they are? Is there a reason they have to be here vs. in a fork of the upstream projects? I saw patches to SSL, which you know, catches the eye.

eliot-akira · 2023-01-28T11:28:19Z

Here's some discussion started about the patches: Propose PRs for the projects that are now patched ad-hoc.

adamziel · 2023-01-30T10:19:53Z

Is there a discussion of where these patches come from or why they are what they are? Is there a reason they have to be here vs. in a fork of the upstream projects? I saw patches to SSL, which you know, catches the eye.

As @eliot-akira mentioned, I'd love to merge these patches to the upstream projects. I will open Pull Requests for actively maintained projects like Emscripten. Unfortunately, PHP 5.6 and some friends are not getting new releases anymore. The OpenSSL patch you've mentioned is actually a PHP 5.6 patch to add OpenSSL 1.1.0 support. It's been proposed for upstream PHP 5.6 and rejected. AFAIK most webhosts who still support PHP 5.6 are maintaining their own fork to keep applying security patches.

Here's the overview of all the patches:

This PR removes all sqlite3 patches originally inherited from https://github.com/seanmorris/php-wasm. These patches moved SQLite3 sources directly to php-src, updated the #include paths to bake SQLite3 into the PHP build, and prevented a sqlite3 > 3.7.4 version check during the PHP ./configure command. This PR instead builds the correct sqlite3 version as a static WASM library using Emscripten's SIDE_MODULE and lets the linker connect the dots.
ncurses.patch – PHP CLI is compiled with readline support and linked with ncurses, however building ncurses using Emscripten fails since it involved creating some database files using the make_hash and make_keys binaries that can't be compiled as WASM. The patch ensures these databases are built using the system-native versions of make_hash and make_keys. Source 1 Source 2 (also provided inline).
bison2.7.patch – Compatibility with Emscripten's version of glibc which is newer than the one supported by Bison 2.7. Source: OpenWRT, more details (also provided inline).
php5.6-openssl1.1 patch – Adds OpenSSL 1.1.0 support for PHP 5.6. Source (also provided inline).
php5.6.patch – Adds a missing header declaration, otherwise emscripten won't compile the mbstring extension. Source: me.
php7.0.patch up to php8.0.patch:
- Replaces zend_ulong with size_t, zend_off_t with long long, and ensures a pointer is used when calling php_stream_seek. It solves Esmcripten compilation errors, but I don't have a good grasp why is this even needed. Ideally this part of the patch could be removed. Source: me.
- Avoids copying empty files as that crashes Emscripten's runtime. That's an Emscripten bug, not a PHP bug, but I found it much easier to patch in PHP. An ideal solution would patch upstream Emscripten. Source: me.
Dockerfile performs string-replace patching of:
- ncurses Makefile – this could be merged into the provided .patch file and perhaps proposed for the upstream ncurses project. Source: same as above.
- PHP – Adjusts build nuances like enforcing posix readdir_r support on older PHP versions or manually disabling unitstd.h. A few of these changes could be brought into upstream PHP 8, but most target the EOL-ed PHP versions. Source: me.
- Built Emscripten's php5.6.js, php-8.0.js, etc. – these bugfixes and features are documented inline and I'd love to have every single one of them merged into upstream Emscripten repo as they're useful and will be hard to maintain separately. Source: me.
zlib – Playground uses a 10 years old Emscripten-compatible port of zlib because my initial attempt to compile the upstream version as WASM failed. I'm sure this old port can be removed, especially since Emscripten comes with -s USE_ZLIB=1 switch these days, but I didn't get there yet. PRs welcome!

adamziel · 2023-01-30T12:15:20Z

Let's merge this one and keep iterating in subsequent PRs

### Description Adds support for CLI SAPI and networking in node.js: ``` > npm run build > node ./build-cli/php-cli.js -r 'echo "Hello from PHP !";' Hello from PHP ! > node ./build-cli/php-cli.js -r 'echo substr(file_get_contents("https://wordpress.org"), 0, 16);' <!DOCTYPE html> > node ./build-cli/php-cli.js -r 'echo phpversion();' 8.2.0-dev > PHP=5.6 node ./build-cli/php-cli.js -r 'echo phpversion();' 5.6.40 ``` ### Highlights: * Networking is supported (including MySQL and HTTPS) * Switching PHP versions is supported. * [Most WordPress PHPUnit tests pass](WordPress/wordpress-playground#111). The failures are caused by missing extensions and a few misconfigured settings * PHP Interactive mode is supported but [the arrow keys don't work](WordPress/wordpress-playground#118) * `wp-cli` works ### In broad strokes: * CLI SAPI is compiled with libedit (readline replacement) and ncurses. * Network calls are asynchronous. Emscripten's Asyncify enables calling asynchronous code from synchronous code. TCP sockets are shimmed with a WebSocket connection to a built-in proxy server running on localhost. It supports data transfer, arbitrary connection targets, and setting a few TCP socket options. * PHP's OpenSSL uses the same CA certs as Node.js * PHP 5.6 is patched to work with OpenSSL 1.1.0 and many other small patches are introduced. For more details, see [patches overview](WordPress/wordpress-playground#119 (comment)), Dockerfile, and `phpwasm-emscripten-library.js` ### Future work: * PHP Interactive server isn't supported yet. Adding support is a matter of making the incoming connection polling non-blocking using Asyncify. * Use a more recent OpenSSL version * [Better support for CLI interactive mode](WordPress/wordpress-playground#118)

adamziel added 30 commits January 14, 2023 20:36

Explore a generic php-cli npm package that enables running PHP scripts

45c5c54

Skip over the shebang line in CLI scripts

dacb78a

Configurable CLI command

d431850

Compile with the actual CLI SAPI

d09743d

Support PHP-CLI interactive mode

0e5e96e

Accept arbitrary ENV variables

48dab7c

Last steps towards a working CLI build

cd7d013

Support arbitrary number of CLI args via a linkedlist

499edd7

Cleanup PHP build dockerfile

4cb34d9

Disable unsupported PHP features via initial php.ini values

1c16b4d

Remove a typo from Docekrfile

ce92698

Remove pthreads artifats

9282cf4

Make php_wasm.c compatible with php8

331f199

Fix up dev artifacts

ca16273

Reliable CLI build for PHP 5.6-8.2

5c7b792

Restrict Node.js specific changes to node runtime

c7fb1e9

Restore the init() call when dispatching web requests

88b7e3e

Remove dev artifact

00b7397

Cleanup unused imports

c6fdba4

Add a build-cli directory for the php-cli builds

47b9c29

Ignore build-cli

f115faf

Add php-cli build logic

118d041

Minify PHP CLI build

1ac2593

Add a php.ini script and bump the version

86a5766

Adjust the build script to PHP 8.2 requirements

7c7b31d

Support popen()

8ef797b

Add pre-build node binaries

594665f

Enable exec functions in php.ini and bump the version

17ac5a1

Fix typo in esbuild.js

24f543c

Add the missing php-cli script

901e009

Solve last compilation kinks – it all works now!

2048a35

adamziel mentioned this pull request Jan 27, 2023

Generic php-cli.js script to replace the PHP terminal command #114

Closed

Updated WASM files

20dda3c

adamziel changed the title ~~Networking support in CLI~~ CLI and networking support Jan 27, 2023

Remove TERMINFO override

4949e87

adamziel marked this pull request as ready for review January 27, 2023 16:13

adamziel mentioned this pull request Jan 30, 2023

Propose PRs for the projects that are now patched ad-hoc #91

Open

adamziel added 8 commits January 30, 2023 12:15

Exit the CLI node.js process once the PHP runtime exits

dc54afa

Allowlist a missing PHP 7.0 Asyncify function

ebd9479

Remove the commented php.ini options

ec39ba4

Exit the php-cli process on error

37da784

Rebuild PHP 7.0 and 7.1

1d21d5b

Add the missing newline at the end of build-cli/package.json

d4418fc

Bump the cli package version

100c5f3

Add ws as a php-cli dependency

a626d1a

adamziel merged commit 9c7d797 into trunk Jan 30, 2023

adamziel deleted the explore/networking branch January 30, 2023 12:15

adamziel mentioned this pull request Jan 30, 2023

Support MySQL in CLI #116

Closed

artemiomorales mentioned this pull request Jan 31, 2023

Add import and export logic #110

Merged

gregvolny mentioned this pull request Jan 31, 2023

Support for external database networking r-wasm/webr#129

Open

adamziel mentioned this pull request Feb 2, 2023

Network access in the browser #85

Open

This was referenced Jun 2, 2023

Gutenberg Block WordPress/playground-tools#57

Open

Feature Gallery #128

Closed

adamziel mentioned this pull request Apr 27, 2023

Tracking: Redesign Documentation #217

Closed

56 tasks

eliot-akira mentioned this pull request May 12, 2023

PHP WASM: Use Emscripten's zlib and remove bundled copy #335

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLI and networking support #119

CLI and networking support #119

adamziel commented Jan 25, 2023 •

edited

Loading

dmsnell commented Jan 27, 2023

eliot-akira commented Jan 28, 2023

adamziel commented Jan 30, 2023 •

edited

Loading

adamziel commented Jan 30, 2023

CLI and networking support #119

CLI and networking support #119

Conversation

adamziel commented Jan 25, 2023 • edited Loading

Description

Highlights:

In broad strokes:

Future work:

dmsnell commented Jan 27, 2023

eliot-akira commented Jan 28, 2023

adamziel commented Jan 30, 2023 • edited Loading

adamziel commented Jan 30, 2023

adamziel commented Jan 25, 2023 •

edited

Loading

adamziel commented Jan 30, 2023 •

edited

Loading