Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for aix ppc64 BE #3549

Closed
merceyz opened this issue Dec 16, 2023 · 31 comments · Fixed by #3550
Closed

Support for aix ppc64 BE #3549

merceyz opened this issue Dec 16, 2023 · 31 comments · Fixed by #3550

Comments

@merceyz
Copy link

merceyz commented Dec 16, 2023

esbuild doesn't support aix ppc64 BE at the moment and it would be great if it did so that projects that use it can be tested on that architecture in Node.js's CITGM setup.

09:34:22 warn: readable-stream npm-install:| npm ERR! code 1                                                                                                                                                        
09:34:22 warn:                             | npm ERR! path /home/iojs/tmp/citgm_tmp/cd2c8a76-4c62-439c-840f-095946746320/readable-stream/node_modules/esbuild                                                       
09:34:22 warn:                             | npm ERR! command failed                                                                                                                                                
09:34:22 warn:                             | npm ERR! command sh -c node install.js                                                                                                                                 
09:34:22 warn:                             | npm ERR! /home/iojs/tmp/citgm_tmp/cd2c8a76-4c62-439c-840f-095946746320/readable-stream/node_modules/esbuild/install.js:69                                              
09:34:22 warn:                             | npm ERR!     throw new Error(`Unsupported platform: ${platformKey}`);                                                                                                  
09:34:22 warn:                             | npm ERR!           ^                                                                                                                                                   
09:34:22 warn:                             | npm ERR!                                                                                                                                                               
09:34:22 warn:                             | npm ERR! Error: Unsupported platform: aix ppc64 BE                                                                                                                     
09:34:22 warn:                             | npm ERR!     at pkgAndSubpathForCurrentPlatform (/home/iojs/tmp/citgm_tmp/cd2c8a76-4c62-439c-840f-095946746320/readable-stream/node_modules/esbuild/install.js:69:11)  
09:34:22 warn:                             | npm ERR!     at checkAndPreparePackage (/home/iojs/tmp/citgm_tmp/cd2c8a76-4c62-439c-840f-095946746320/readable-stream/node_modules/esbuild/install.js:211:28)          
09:34:22 warn:                             | npm ERR!     at Object.<anonymous> (/home/iojs/tmp/citgm_tmp/cd2c8a76-4c62-439c-840f-095946746320/readable-stream/node_modules/esbuild/install.js:238:1)               
09:34:22 warn:                             | npm ERR!     at Module._compile (node:internal/modules/cjs/loader:1375:14)                                                                                             
09:34:22 warn:                             | npm ERR!     at Module._extensions..js (node:internal/modules/cjs/loader:1434:10)                                                                                      
09:34:22 warn:                             | npm ERR!     at Module.load (node:internal/modules/cjs/loader:1206:32)                                                                                                 
09:34:22 warn:                             | npm ERR!     at Module._load (node:internal/modules/cjs/loader:1022:12)                                                                                                
09:34:22 warn:                             | npm ERR!     at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:142:12)                                                                    
09:34:22 warn:                             | npm ERR!     at node:internal/main/run_main_module:28:49                                                                                                               
09:34:22 warn:                             | npm ERR!                                                                                                                                                               
09:34:22 warn:                             | npm ERR! Node.js v22.0.0-pre                                                                                                                                           
09:34:22 warn:                             |                                                                                                                                                                        
09:34:22 warn:                             | npm ERR! A complete log of this run can be found in: /home/iojs/tmp/citgm_tmp/cd2c8a76-4c62-439c-840f-095946746320/home/.npm/_logs/2023-12-15T09_44_41_045Z-debug-0.log

Ref nodejs/citgm#1039 (comment)

@evanw
Copy link
Owner

evanw commented Dec 16, 2023

Do you have access to such a system and can conform that esbuild can build and run on that system (e.g. installing go and running make test-go in this repo passes all tests)? I don't have access to such a system myself.

@merceyz
Copy link
Author

merceyz commented Dec 16, 2023

I don't but https://github.com/nodejs/citgm does, maybe they'd be willing to test it, if not a WebAssembly fallback package should presumably work and resolve the issue in the comment I linked to.

@richardlau
Copy link

Do you have access to such a system and can conform that esbuild can build and run on that system (e.g. installing go and running make test-go in this repo passes all tests)? I don't have access to such a system myself.

I've asked @abmusse from IBM to take a look. We think it should be okay, based on https://go.dev/doc/install/source#environment listing $GOOS=AIX and $GOARCH=ppc64 as a valid combination but @abmusse will try to verify the tests pass on an AIX system.

@evanw
Copy link
Owner

evanw commented Dec 19, 2023

I just published a version of esbuild that works on aix ppc64 BE in theory. Please try it out in that environment and let me know if it works or not. Something like npm i esbuild@0.19.10 && npx esbuild --version should be working now.

@abmusse
Copy link

abmusse commented Dec 20, 2023

@evanw
Just tested this on AIX 7.2 system and confirm install is successful!

$ uname -vrs
AIX 2 7

$ npm i esbuild@0.19.10

added 2 packages, and audited 3 packages in 7s

found 0 vulnerabilities

$ npx esbuild --version
0.19.10

@bskimball
Copy link

bskimball commented Jan 15, 2024

Can you do the same for os400? It runs PASE so if esbuild runs on AIX it will also run in the pase environment. I do have access to the server
$GOOS=os400 and $GOARCH=ppc64

@evanw
Copy link
Owner

evanw commented Jan 15, 2024

This issue came about from someone trying to install esbuild using node/npm on aix, which is possible because aix is one of the operating systems that node supports: https://nodejs.org/api/process.html#processplatform. However, os400 is not on this list, which might mean that running an official build of node on os400 is not possible (although there still might still be an unofficial build of node/npm for os400).

If you just want to run the esbuild binary executable on os400 (or any other platform that the Go compiler toolchain supports) without using node, then you should already be able to compile esbuild for yourself for that platform. You don't need any changes to esbuild to do that.

If you want to use esbuild with node via the package that's published to npm, then it needs to be possible to run npm on that platform (so you can run the installer to download that prebuilt version of esbuild). If this is possible, then I'm assuming that means there's an unofficial build of node/npm for os400 since os400 is not one of node's officially-supported operating systems. In that case I'd need to know what values that unofficial build uses to identify the platform (specifically what node -p '[process.platform, os.arch(), os.endianness()]' says, which is how esbuild's installer identifies the current platform). This is what the error message in the original post provided, which is what made it possible for me to know how to add support for this particular platform without being able to use it myself.

@bskimball
Copy link

os400 more commonly rreferred to as IBMi has an @ibm-release maintained version of Node that runs in the shell environment and is installed via yum (https://ibmi-oss-docs.readthedocs.io/en/latest/nodejs/README.html) .

node -p '[process.platform, os.arch(), os.endianness()]' results in the following -> ['os400','ppc64','BE']

@evanw
Copy link
Owner

evanw commented Jan 16, 2024

Thanks. That should be enough info for me to proceed.

@evanw
Copy link
Owner

evanw commented Jan 16, 2024

Wait actually Go doesn't appear to support building for os400 even on the latest release:

$ go version
go version go1.21.6 linux/amd64

$ GOOS=os400 GOARCH=ppc64 go build ./cmd/esbuild
go: unsupported GOOS/GOARCH pair os400/ppc64

$ go tool dist list | grep os400
(no output)

So I'm not able to build esbuild for this OS because Go doesn't support it.

$GOOS=os400 and $GOARCH=ppc64

How are you able to build esbuild using these parameters? Do you also have a custom Go compiler?

@bskimball
Copy link

bskimball commented Jan 16, 2024

I haven't tried any go programs on there personally. I have read that go compiled for aix will run in the PASE environment, but I've never tried it.

This particular issue was from me trying to run vite ssr on IBMi. install.js throws the unsupported platform error.

This may fix my issue by adding to nodePlatform.ts

export const knownUnixlikePackages: Record<string, string> = {
   'aix ppc64 BE': '@esbuild/aix-ppc64',
+  'os400 ppc64 BE': '@esbuild/aix-ppc64',   
   'android arm64 LE': '@esbuild/android-arm64',

PASE on IBMi is a unix shell similar to aix. If I get time tomorrow I may try and fork to test.

@evanw
Copy link
Owner

evanw commented Jan 16, 2024

To save you some forking work: I believe the only thing you would need to confirm is whether the binary executable in https://registry.npmjs.org/@esbuild/aix-ppc64/-/android-arm-0.19.11.tgz works correctly on os400 or not. You probably don't need to fork esbuild. I assume that node will run correctly on os400 so you shouldn't need to test that, but I need to know whether the executable will work or not. It may not work correctly (e.g. this issue is open).

@bskimball
Copy link

It does not run unfortunately. Due to the mmap pointers. Doesn't look like IBM has any plans on supporting Go either. Thanks for your help with this issue.

@abmusse
Copy link

abmusse commented Jan 16, 2024

@bskimball

Yes AIX complied go programs won't work properly on IBM i due to many issues including the mmap one you mentioned.

For now the best solution would be to run esbuild to bundle your code on another machine then serve the files on your IBM i.

@merceyz
Copy link
Author

merceyz commented Jan 16, 2024

If Node.js runs on the machine you could always try esbuild-wasm.

@bskimball
Copy link

@bskimball

Yes AIX complied go programs won't work properly on IBM i due to many issues including the mmap one you mentioned.

For now the best solution would be to run esbuild to bundle your code on another machine then serve the files on your IBM i.

That's what I do now. I was hoping to eventually run ssr on IBMi with esbuild

@bskimball
Copy link

If Node.js runs on the machine you could always try esbuild-wasm.

I might try that, thanks.

@abmusse
Copy link

abmusse commented Jan 17, 2024

If Node.js runs on the machine you could always try esbuild-wasm.

Yes Node.js is supported on IBM i.

I think giving esbuild-wasm a shot is worth it, will try to test it on IBM i.

@bskimball
Copy link

bskimball commented Jan 17, 2024

Looks like the wasm packages do run on IBMi. I'm using vite so my package.json lists the following overrides for esbuild and rollup.

  "overrides": {
    "esbuild": "npm:esbuild-wasm@latest",
    "rollup": "npm:@rollup/wasm-node"
  },

** update I may need to check some configs with vite. The npm installs work but vite throws an error.

@abmusse
Copy link

abmusse commented Jan 22, 2024

** update I may need to check some configs with vite. The npm installs work but vite throws an error.

@bskimball

Interesting, Is there a simple example project that uses esbuild? Would like to use it to test using esbuild-wasm on IBM i.

@bskimball
Copy link

bskimball commented Jan 22, 2024

** update I may need to check some configs with vite. The npm installs work but vite throws an error.

@bskimball

Interesting, Is there a simple example project that uses esbuild? Would like to use it to test using esbuild-wasm on IBM i.

I was using @fastify/vite but I think remix with the vite package may be easier to set up to test SSR with vite and is here.

I haven't dug into much but it looks like even with the overrides in package.json vite tries to pull esbuild instead of esbuild-wasm. We host a bunch of IBMi servers, so I was going to screw around with it when I got a chance.

Probably related to this issue

@abmusse
Copy link

abmusse commented Feb 19, 2024

@evanw

Would it make sense to install esbuild-wasm as an optional dep when the os is IBM i (os400)?

"optionalDependencies": {

Node.js is available and could be used to run the wasm binaries.

This would install esbuild-wasm when packages require esbuild e.g. Vite in this case.

@merceyz
Copy link
Author

merceyz commented Feb 19, 2024

esbuild has WASM fallbacks for certain platforms, for example https://github.com/evanw/esbuild/tree/40711afe0baa545012b813c5c7788225be9ef74c/npm/%40esbuild/android-x64, so that is an option.

@abmusse
Copy link

abmusse commented Feb 19, 2024

Yes I think having that for IBM i would be good. That way we can fallback to esbuild-wasm when esbuild is installed. Before doing that though are there tests for esbuild-wasm that I can run to verify things are working properly?

@merceyz
Copy link
Author

merceyz commented Feb 19, 2024

Yes, see

esbuild/Makefile

Lines 55 to 62 in 40711af

# This uses "env -i" to run in a clean environment with no environment
# variables. It then adds some environment variables back as needed.
# This is a hack to avoid a problem with the WebAssembly support in Go
# 1.17.2, which will crash when run in an environment with over 4096
# bytes of environment variable data such as GitHub Actions.
test-wasm-node: esbuild
env -i $(shell go env) PATH="$(shell go env GOROOT)/misc/wasm:$(PATH)" GOOS=js GOARCH=wasm go test ./internal/...
node scripts/wasm-tests.js
.

@abmusse
Copy link

abmusse commented Feb 20, 2024

I'll see about running the tests and updating with the results!

@abmusse
Copy link

abmusse commented Feb 20, 2024

I cloned the repo and I tried to run the wasm-tests:

node scripts/wasm-tests.js

Error: spawn go ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:284:19)
    at onErrorNT (node:internal/child_process:477:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn go',
  path: 'go',
  spawnargs: [
    'build',
    '-o',
    '~/esbuild/npm/esbuild-wasm/esbuild.wasm',
    '-ldflags=-s -w',
    '-trimpath',
    '~/esbuild/cmd/esbuild'
  ],
  cmd: 'go build -o ~/esbuild/npm/esbuild-wasm/esbuild.wasm -ldflags=-s -w -trimpath ~/esbuild/cmd/esbuild'
}

Looks like that failed trying to build the wasm lib with go.

async function main() {
// Generate the WebAssembly module
const esbuildPathNative = path.join(__dirname, '..', process.platform === 'win32' ? 'esbuild.exe' : 'esbuild');
await buildWasmLib(esbuildPathNative);

esbuild/scripts/esbuild.js

Lines 129 to 141 in 40711af

exports.buildWasmLib = async (esbuildPath) => {
// Asynchronously start building the WebAssembly module
const npmWasmDir = path.join(repoDir, 'npm', 'esbuild-wasm')
const goBuildPromise = new Promise((resolve, reject) => childProcess.execFile('go',
[
'build',
'-o', path.join(npmWasmDir, 'esbuild.wasm'),
'-ldflags=-s -w', // This removes ~0.14mb of unnecessary WebAssembly code
'-trimpath',
path.join(repoDir, 'cmd', 'esbuild'),
],
{ cwd: repoDir, stdio: 'inherit', env: { ...process.env, GOOS: 'js', GOARCH: 'wasm' } },
err => err ? reject(err) : resolve()))

We don't have go available to build the wasm lib from.

Are there any alternatives of using a pre-built wasm lib?

@bskimball
Copy link

@abmusse I got back around to this and set up a test remix app that uses vite. The overrides in package.json does in fact use the wasm version. However, there are other errors showing in the wasm-function:


RuntimeError: float unrepresentable in integer range
    at wasm://wasm/02bc523a:wasm-function[1066]:0xe5a2c
    at wasm://wasm/02bc523a:wasm-function[1476]:0x147f82
    at wasm://wasm/02bc523a:wasm-function[743]:0xa740a
    at wasm://wasm/02bc523a:wasm-function[743]:0xa7277
    at wasm://wasm/02bc523a:wasm-function[743]:0xa7277
    at wasm://wasm/02bc523a:wasm-function[743]:0xa7277
    at wasm://wasm/02bc523a:wasm-function[743]:0xa7277
    at wasm://wasm/02bc523a:wasm-function[1061]:0xe594e
    at wasm://wasm/02bc523a:wasm-function[1059]:0xe5921
    at globalThis.Go.run (/var/apps/remix-test/my-remix-app/node_modules/esbuild/wasm_exec.js:527:23)
failed to load config from /var/apps/remix-test/my-remix-app/vite.config.js
Error: The service was stopped
    at /var/apps/remix-test/my-remix-app/node_modules/esbuild/lib/main.js:1074:25
    at responseCallbacks.<computed> (/var/apps/remix-test/my-remix-app/node_modules/esbuild/lib/main.js:694:9)
    at Socket.afterClose (/var/apps/remix-test/my-remix-app/node_modules/esbuild/lib/main.js:684:28)
    at Socket.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1344:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

I don't know that falling back to esbuild-wasm would resolve anything.

side note: I hope IBM figures out a way to allow us to run Go on the IBMi platform.

@bskimball
Copy link

@abmusse this library https://github.com/onlysumitg/ibmigo/tree/main does run and compile Go on IBMi

@abmusse
Copy link

abmusse commented Apr 23, 2024

@abmusse I got back around to this and set up a test remix app that uses vite. The overrides in package.json does in fact use the wasm version. However, there are other errors showing in the wasm-function:


RuntimeError: float unrepresentable in integer range
    at wasm://wasm/02bc523a:wasm-function[1066]:0xe5a2c
    at wasm://wasm/02bc523a:wasm-function[1476]:0x147f82
    at wasm://wasm/02bc523a:wasm-function[743]:0xa740a
    at wasm://wasm/02bc523a:wasm-function[743]:0xa7277
    at wasm://wasm/02bc523a:wasm-function[743]:0xa7277
    at wasm://wasm/02bc523a:wasm-function[743]:0xa7277
    at wasm://wasm/02bc523a:wasm-function[743]:0xa7277
    at wasm://wasm/02bc523a:wasm-function[1061]:0xe594e
    at wasm://wasm/02bc523a:wasm-function[1059]:0xe5921
    at globalThis.Go.run (/var/apps/remix-test/my-remix-app/node_modules/esbuild/wasm_exec.js:527:23)
failed to load config from /var/apps/remix-test/my-remix-app/vite.config.js
Error: The service was stopped
    at /var/apps/remix-test/my-remix-app/node_modules/esbuild/lib/main.js:1074:25
    at responseCallbacks.<computed> (/var/apps/remix-test/my-remix-app/node_modules/esbuild/lib/main.js:694:9)
    at Socket.afterClose (/var/apps/remix-test/my-remix-app/node_modules/esbuild/lib/main.js:684:28)
    at Socket.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1344:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

I don't know that falling back to esbuild-wasm would resolve anything.

side note: I hope IBM figures out a way to allow us to run Go on the IBMi platform.

Ah so falling back to esbuild-wasm is still not ideal 😞

@phpdave
Copy link

phpdave commented Apr 24, 2024

I'm also having troubles using Vite which is using esbuild dependency on IBM i PASE Environment (os400).

When i run this

npm create vite@latest
cd vite-project/

modify the package.json to use wasm

"overrides": {
    "vite": {
      "rollup": "npm:@rollup/wasm-node@*"
    },
    "rollup": "npm:@rollup/wasm-node@*",
    "esbuild": "npm:esbuild-wasm@latest"
  }

I'm getting this panic: assignment to entry in nil map

npm install
npm run dev
bash-5.2$ npm run dev

> vite-project@0.0.0 dev
> vite --port 8765


  VITE v5.2.10  ready in 1082 ms
bash-5.2$ panic: assignment to entry in nil map

goroutine 9 [running]:
github.com/evanw/esbuild/internal/config.ProcessDefines(0x891288)
        github.com/evanw/esbuild/internal/config/globals.go:967 +0x52
github.com/evanw/esbuild/pkg/api.validateDefines({0x8c0d20, 0x80a330, 0x80a348, 0x80a360, 0x3, 0x8c0b40}, 0x8c0540, {0x0, 0x0, 0x0}, ...)
        github.com/evanw/esbuild/pkg/api/api_impl.go:668 +0xba
github.com/evanw/esbuild/pkg/api.validateBuildOptions({0x2, 0x0, 0x0, 0x8c05d0, 0x0, {0x0, 0x0}, 0x0, 0x0, {0x0, ...}, ...}, ...)
        github.com/evanw/esbuild/pkg/api/api_impl.go:1212 +0x18
github.com/evanw/esbuild/pkg/api.contextImpl({0x2, 0x0, 0x0, 0x8c05d0, 0x0, {0x0, 0x0}, 0x0, 0x0, {0x0, ...}, ...})
        github.com/evanw/esbuild/pkg/api/api_impl.go:879 +0x53
github.com/evanw/esbuild/pkg/api.Context({0x2, 0x0, 0x0, 0x8c05d0, 0x0, {0x0, 0x0}, 0x0, 0x0, {0x0, ...}, ...})
        github.com/evanw/esbuild/pkg/api/api.go:525 +0x2
main.(*serviceType).handleBuildRequest(0x8c00c0, 0x0, 0x8c01b0)
        github.com/evanw/esbuild/cmd/esbuild/service.go:739 +0xa0
main.(*serviceType).handleIncomingPacket.func2()
        github.com/evanw/esbuild/cmd/esbuild/service.go:235 +0x4
created by main.(*serviceType).handleIncomingPacket
        github.com/evanw/esbuild/cmd/esbuild/service.go:233 +0x27panic: assignment to entry in nil mapnpm ls esbuild

which is referencing

result.IdentifierDefines["NaN"] = DefineData{
DefineExpr: &DefineExpr{Constant: &js_ast.ENumber{Value: math.NaN()}},
}

Is this also something related to WASM causing the issue?

node -p '[process.platform, os.arch(), os.endianness()]'
[ 'os400', 'ppc64', 'BE' ]

I followed the stack trace but still cant figure it out
image
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants