-
Notifications
You must be signed in to change notification settings - Fork 252
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to bundle query-planner-wasm dependecy for use on lambdas #255
Comments
hey @MarcelCutts! thanks so much for the detailed issue. i've opened up #270 and run it successfully with your example repo. if you'd like please give it a try and let me know if it meets your needs! |
I'm running into a similar problem. After updating to @apollo/gateway 0.20.1 or higher I cannot get our webpack build script to work with the new wasm query planner. When using @apollo/gateway 0.20.4 the created/"preinstalled" version does not throw compile errors but does not start the application mentioning that the module "util" cannot be found. I tried running wasm-pack (npm script monorepo-prepare) with the branch mentioned in #270 and tried out both the created files in dist and module folder. When using module folder webpack tells me I set the experiments option and type as proposed but the gateway does not seem to find the was query planer telling me Does anyone have an idea how to get webpack working again with @apollo/gateway 0.20.x or 0.21.x? |
Just ran into this problem today. |
Ran into this as well, when updating to ContextVersions
Webpack confconst path = require('path')
const webpack_config = {
entry: './src/index.ts',
module: {
rules: [
{ test: /\.m?js/, resolve: { fullySpecified: false } },
{
test: /\.ts$/,
loader: 'ts-loader',
exclude: /node_modules/,
resolve: { fullySpecified: false }
}
]
},
resolve: {
roots: [ path.resolve(process.cwd(), 'src') ],
mainFiles: [ 'index' ],
extensions: [ '.ts', '.json', '.js', '.mjs' ],
alias: { '@': path.resolve(process.cwd(), 'src') }
},
target: 'node',
output: {
path: path.resolve(process.cwd(), 'build', 'dev'),
filename: 'index.js',
devtoolModuleFilenameTemplate: '[absolute-resource-path]',
devtoolFallbackModuleFilenameTemplate: '[absolute-resource-path]?[hash]'
},
ignoreWarnings: [ [Function] ], // Custom ignore, not relevant
devtool: 'source-map',
mode: 'development',
optimization: { nodeEnv: false }
} FindingsPoked around for a bit and manually editing the package to get an idea of what's going wrong.
const { TextDecoder } = require(String.raw`util`); Seems to cause the initial issue which is that it cannot find the module Error: Cannot find module 'util'
at webpackEmptyContext (<project>/build/dev/index.js:7708:10)
at Object../node_modules/@apollo/query-planner-wasm/dist/index.js (<project>/build/dev/index.js:7545:118)
at __webpack_require__ (<project>/build/dev/index.js:224349:42)
at Object../node_modules/@apollo/gateway/dist/buildQueryPlan.js (<project>/build/dev/index.js:3870:30)
at __webpack_require__ (<project>/build/dev/index.js:224349:42)
at Object../node_modules/@apollo/gateway/dist/index.js (<project>/build/dev/index.js:4542:26)
at __webpack_require__ (<project>/build/dev/index.js:224349:42)
at Module../src/index.ts (<project>/build/dev/index.js:218996:73)
at __webpack_require__ (<project>/build/dev/index.js:224349:42)
at <project>/build/dev/index.js:224426:11 {
code: 'MODULE_NOT_FOUND'
}
Changing this to Error: ENOENT: no such file or directory, open '<project>/build/dev/index_bg.wasm'
at Object.openSync (fs.js:462:3)
at Object.readFileSync (fs.js:364:35)
at Object../node_modules/@apollo/query-planner-wasm/dist/index.js (<project>/build/dev/index.js:7687:51)
at __webpack_require__ (<project>/build/dev/index.js:224328:42)
at Object../node_modules/@apollo/gateway/dist/buildQueryPlan.js (<project>/build/dev/index.js:3870:30)
at __webpack_require__ (<project>/build/dev/index.js:224328:42)
at Object../node_modules/@apollo/gateway/dist/index.js (<project>/build/dev/index.js:4542:26)
at __webpack_require__ (<project>/build/dev/index.js:224328:42)
at Module../src/index.ts (<project>/build/dev/index.js:218975:73)
at __webpack_require__ (<project>/build/dev/index.js:224328:42) {
errno: -2,
syscall: 'open',
code: 'ENOENT',
path: '<project>/build/dev/index_bg.wasm'
} A quick look at the const path = require('path').join(__dirname, 'index_bg.wasm');
const bytes = require('fs').readFileSync(path); When bundled With this in mind I tested adding a const webpackConfig = {
<...>
module: {
rules: [
{
test: /\.wasm/,
loader: 'file-loader',
options: {
name: '[name].[ext]'
}
},
<...>
]
},
<...>
} In the end the changes to -const { TextDecoder } = require(String.raw`util`);
+const { TextDecoder } = require('util');
+require('./index_bg.wasm') DisclaimerIt's far from pretty, but I'm not familiar with WebAssembly at all, and feel quite out of my depth. But hopefully it might help someone. I'll probably end up using patch-package as a temporary solution. |
any update on this? it's still failing with webpack, should I just downgrade for the time being? |
Downgrading works for now
|
any update on this? |
As the original poster I feel I should post what our workaround ended up being shortly after opening the issue. We have a custom version of the October 2020 dated To ensure calls to new webpack.NormalModuleReplacementPlugin(
new RegExp('@apollo/query-planner-wasm'),
path.resolve(__dirname, 'src/query-planner-wasm')
), From there we use a zipping tool to package it as a dependency that can be used on lambdas. This has meant our internal version of apollo federation has stagnated due to the fragility of upgrading. While I deeply appreciated Ashley's work in resolving this, it would mean doing a custom build of the WASM locally from what I could interpret, which is one option - but ideally it would not add a new step to our build pipeline that would also have to be maintained by the team. As we are federating a substantially large business with an ecosystem stretching from the cutting edge to the venerable, we are going to stick to our current approach for stability right at this moment. I have taken occasional looks at this repo but can be difficult to tackle or experiment with deeper issues while being external to the apollo organisation. For example, some of the package build arcana hidden so one cannot build all packages locally. However I do also take note that the CONTRIBUTING.md is honest about at most wanting small fixes (20 lines or less) and as such this project's governance is closer to apollo. As a semi-British person I feel the need to clarify this is not a passive aggressive slight, I appreciate the free software as-is and owners' management of their repo should be fully up to their needs and wants. We will seek to solve this issue a few times more in the coming months, I suspect. Since there's a smattering of interest I will post updates here if there are successes. |
@MarcelCutts Hi just to clarify: would merging and releasing #270 resolve your issue? |
I just gave #270 a final review and a merge; it looks good to get out into a release that will hopefully validate that this fix works and get folks who are pinned to an older version (e.g., @MarcelCutts) onto a published version. |
Please try |
Hi @abernix I tried the latest versions and i still have an error Cannot find module 'util'. Downgrading to 0.19.1 works |
Failed to compile. ./src/cardano/serialization-lib/@emurgo/cardano-serialization-lib-browser/cardano_serialization_lib_bg.wasm
|
Context
Hi folks!
We've been working quite extensively with Federation and all the tooling in this lovely repository. One thing we do is bundle these packages with webpack so we can run a a federated gateway in lambdas on various clouds. It's great for things like spinning up a federation instance for each new PR. We even have a version running in-browser as a service worker for doing tests! It was pretty neat.
Previously, this was pretty simplen as all depenencies were
.{j|t}s
and these artifacts could be combined with little issue through a standard webpack setup.With the introduction the query planner being created in WASM, this caused issues for bundling which we have been working around in an in-elegant manner that would be better solved here.
Behaviour details
Technical analaysis
With the query planner now being written in Rust and using the
wasm-pack
toolchain, a WASM binary and a series of bindings are generated. This in itself isn't insurmountable. It is possible to inline rust or have lambdas where there is a co-locatedindex_bg.wasm
file that the service can reference.However, as the output of the WASM in this repository (and I one assumes the background building/packaging system used here) is set to
--target node
we run into some difficulties when bundling. The bindings require string interpolation in require statements that break webpack, e.g.For clarity, this require statement when bundled also fails when run on node, where the
util
package is defintely availableThis can be fixed by having the
wasm-pack
target to be set to--target bundler
. Currently our work-around is to have a copy of the@apollo/query-planner-wasm
library that we modified by hand and then have webpack re-routerequire()
statements from@apollo/gateway
to our locally modified query planner instead. Then the packages run as expected on our node lambdas - however this is not ideal, as I am sure you can appreciate!Here is a version of the
apollo/federation-demo
repository that shows this. If you runnpm run build
followed bynode dist/main.js
on thebundle_woes
branch you can see the errors that occur.We can't be the only folks to desire bundling these dependencies, so I have been thinking about how to resolve the issue in this repository. Potentially having two outputs to satisfy bundler and node needs - although the bundled output will work on node!
Thank you for patience in reading through this issue. Here's a .gif in thanks 👍
The text was updated successfully, but these errors were encountered: