Unable to bundle query-planner-wasm dependecy for use on lambdas

[MarcelCutts]

Context

Hi folks!

We've been working quite extensively with Federation and all the tooling in this lovely repository. One thing we do is bundle these packages with webpack so we can run a a federated gateway in lambdas on various clouds. It's great for things like spinning up a federation instance for each new PR. We even have a version running in-browser as a service worker for doing tests! It was pretty neat.

Previously, this was pretty simplen as all depenencies were .{j|t}s and these artifacts could be combined with little issue through a standard webpack setup.

With the introduction the query planner being created in WASM, this caused issues for bundling which we have been working around in an in-elegant manner that would be better solved here.

Behaviour details

• Last working version: @apollo/gateway: 20.* (using the NPM package repository)
• Expected behaviour: Can bundle @apollo/gateway and its dependency, @apollo/query-planner-wasm without issue
• Actual behaviour: Bundling results in a stream of difficult to circumvent errors

Technical analaysis

With the query planner now being written in Rust and using the wasm-pack toolchain, a WASM binary and a series of bindings are generated. This in itself isn't insurmountable. It is possible to inline rust or have lambdas where there is a co-located index_bg.wasm file that the service can reference.

However, as the output of the WASM in this repository (and I one assumes the background building/packaging system used here) is set to --target node we run into some difficulties when bundling. The bindings require string interpolation in require statements that break webpack, e.g.

// @apollo/query-planner-wasm/dist/index.js

let imports = {};
imports['__wbindgen_placeholder__'] = module.exports;
let wasm;
const { TextDecoder } = require(String.raw`util`);

For clarity, this require statement when bundled also fails when run on node, where the util package is defintely available

This can be fixed by having the wasm-pack target to be set to --target bundler. Currently our work-around is to have a copy of the @apollo/query-planner-wasm library that we modified by hand and then have webpack re-route require() statements from @apollo/gateway to our locally modified query planner instead. Then the packages run as expected on our node lambdas - however this is not ideal, as I am sure you can appreciate!

Here is a version of the apollo/federation-demo repository that shows this. If you run npm run build followed by node dist/main.js on the bundle_woes branch you can see the errors that occur.

We can't be the only folks to desire bundling these dependencies, so I have been thinking about how to resolve the issue in this repository. Potentially having two outputs to satisfy bundler and node needs - although the bundled output will work on node!

Thank you for patience in reading through this issue. Here's a .gif in thanks 👍

[ashleygwilliams]

hey @MarcelCutts! thanks so much for the detailed issue. i've opened up #270 and run it successfully with your example repo. if you'd like please give it a try and let me know if it meets your needs!

[sgohlke]

I'm running into a similar problem. After updating to @apollo/gateway 0.20.1 or higher I cannot get our webpack build script to work with the new wasm query planner. When using @apollo/gateway 0.20.4 the created/"preinstalled" version does not throw compile errors but does not start the application mentioning that the module "util" cannot be found.

I tried running wasm-pack (npm script monorepo-prepare) with the branch mentioned in #270 and tried out both the created files in dist and module folder. When using module folder webpack tells me
ERROR in ./node_modules/@apollo/query-planner-wasm/dist/index_bg.wasm 1:0 Module parse failed: Unexpected character '' (1:0) The module seem to be a WebAssembly module, but module is not flagged as WebAssembly module for webpack. BREAKING CHANGE: Since webpack 5 WebAssembly is not enabled by default and flagged as experimental feature. You need to enable one of the WebAssembly experiments via 'experiments.asyncWebAssembly: true' (based on async modules) or 'experiments.syncWebAssembly: true' (like webpack 4, deprecated). For files that transpile to WebAssembly, make sure to set the module type in the 'module.rules' section of the config (e. g. 'type: "webassembly/async"').

I set the experiments option and type as proposed but the gateway does not seem to find the was query planer telling me
This data graph is missing a valid configuration. query_planner_wasm_1.getQueryPlanner is not a function.

Does anyone have an idea how to get webpack working again with @apollo/gateway 0.20.x or 0.21.x?

[suparngp]

Just ran into this problem today.

[Otard95]

Ran into this as well, when updating to @apollo/gateway 0.21.4.

Context

Versions

• @apollo/gateway @ ^0.21.4
• webpack @ ^5.9.0
• webpack-cli @ ^4.2.0

Webpack conf

const path = require('path')
const webpack_config = {
  entry: './src/index.ts',
  module: {
    rules: [
      { test: /\.m?js/, resolve: { fullySpecified: false } },
      {
        test: /\.ts$/,
        loader: 'ts-loader',
        exclude: /node_modules/,
        resolve: { fullySpecified: false }
      }
    ]
  },
  resolve: {
    roots: [ path.resolve(process.cwd(), 'src') ],
    mainFiles: [ 'index' ],
    extensions: [ '.ts', '.json', '.js', '.mjs' ],
    alias: { '@': path.resolve(process.cwd(), 'src') }
  },
  target: 'node',
  output: {
    path: path.resolve(process.cwd(), 'build', 'dev'),
    filename: 'index.js',
    devtoolModuleFilenameTemplate: '[absolute-resource-path]',
    devtoolFallbackModuleFilenameTemplate: '[absolute-resource-path]?[hash]'
  },
  ignoreWarnings: [ [Function] ], // Custom ignore, not relevant
  devtool: 'source-map',
  mode: 'development',
  optimization: { nodeEnv: false }
}

Findings

Poked around for a bit and manually editing the package to get an idea of what's going wrong.

<project>/node_modules/@apollo/query-planner-wasm/dist/index.js:4

const { TextDecoder } = require(String.raw`util`);

Seems to cause the initial issue which is that it cannot find the module utils

Error: Cannot find module 'util'
    at webpackEmptyContext (<project>/build/dev/index.js:7708:10)
    at Object../node_modules/@apollo/query-planner-wasm/dist/index.js (<project>/build/dev/index.js:7545:118)
    at __webpack_require__ (<project>/build/dev/index.js:224349:42)
    at Object../node_modules/@apollo/gateway/dist/buildQueryPlan.js (<project>/build/dev/index.js:3870:30)
    at __webpack_require__ (<project>/build/dev/index.js:224349:42)
    at Object../node_modules/@apollo/gateway/dist/index.js (<project>/build/dev/index.js:4542:26)
    at __webpack_require__ (<project>/build/dev/index.js:224349:42)
    at Module../src/index.ts (<project>/build/dev/index.js:218996:73)
    at __webpack_require__ (<project>/build/dev/index.js:224349:42)
    at <project>/build/dev/index.js:224426:11 {
  code: 'MODULE_NOT_FOUND'
}

Changing this to const { TextDecoder } = require('util'); resolves the issue, but leaves you with this new error.

Error: ENOENT: no such file or directory, open '<project>/build/dev/index_bg.wasm'
    at Object.openSync (fs.js:462:3)
    at Object.readFileSync (fs.js:364:35)
    at Object../node_modules/@apollo/query-planner-wasm/dist/index.js (<project>/build/dev/index.js:7687:51)
    at __webpack_require__ (<project>/build/dev/index.js:224328:42)
    at Object../node_modules/@apollo/gateway/dist/buildQueryPlan.js (<project>/build/dev/index.js:3870:30)
    at __webpack_require__ (<project>/build/dev/index.js:224328:42)
    at Object../node_modules/@apollo/gateway/dist/index.js (<project>/build/dev/index.js:4542:26)
    at __webpack_require__ (<project>/build/dev/index.js:224328:42)
    at Module../src/index.ts (<project>/build/dev/index.js:218975:73)
    at __webpack_require__ (<project>/build/dev/index.js:224328:42) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '<project>/build/dev/index_bg.wasm'
}

A quick look at the index.js reveals
<project>/node_modules/@apollo/query-planner-wasm/dist/index.js:145

const path = require('path').join(__dirname, 'index_bg.wasm');
const bytes = require('fs').readFileSync(path);

When bundled __dirname becomes <project>/build/dev, but since the index_bg.wasm is not a required module, webpack has not included it in the bundle. As a test I copied the index_bg.wasm to you output directory and with no rebuild the app runs fine.

With this in mind I tested adding a require statement for the .wasm file which would allow me to use webpack's file-loader like this:

const webpackConfig = {
  <...>
  module: {
    rules: [
      {
        test: /\.wasm/,
        loader: 'file-loader',
        options: {
          name: '[name].[ext]'
        }
      },
      <...>
    ]
  },
  <...>
}

In the end the changes to <project>/node_modules/@apollo/query-planner-wasm/dist/index.js are:

-const { TextDecoder } = require(String.raw`util`);
+const { TextDecoder } = require('util');
+require('./index_bg.wasm')

Disclaimer

It's far from pretty, but I'm not familiar with WebAssembly at all, and feel quite out of my depth. But hopefully it might help someone. I'll probably end up using patch-package as a temporary solution.

[karanpratapsingh]

any update on this? it's still failing with webpack, should I just downgrade for the time being?

[karanpratapsingh]

Downgrading works for now

"@apollo/gateway": "0.19.1"

[bryantbiggs]

any update on this?

[MarcelCutts]

As the original poster I feel I should post what our workaround ended up being shortly after opening the issue. We have a custom version of the October 2020 dated query-planner node module wherein we change the require hook for text decoder by hand, and then use the copy-webpack-plugin to colocate the WASM to the webpack output bundle.

To ensure calls to query-planner go to our modified version, we also employ this piece of moderate webpack shamanism.

    new webpack.NormalModuleReplacementPlugin(
      new RegExp('@apollo/query-planner-wasm'),
      path.resolve(__dirname, 'src/query-planner-wasm')
    ),

From there we use a zipping tool to package it as a dependency that can be used on lambdas.

This has meant our internal version of apollo federation has stagnated due to the fragility of upgrading.

While I deeply appreciated Ashley's work in resolving this, it would mean doing a custom build of the WASM locally from what I could interpret, which is one option - but ideally it would not add a new step to our build pipeline that would also have to be maintained by the team. As we are federating a substantially large business with an ecosystem stretching from the cutting edge to the venerable, we are going to stick to our current approach for stability right at this moment.

I have taken occasional looks at this repo but can be difficult to tackle or experiment with deeper issues while being external to the apollo organisation. For example, some of the package build arcana hidden so one cannot build all packages locally. However I do also take note that the CONTRIBUTING.md is honest about at most wanting small fixes (20 lines or less) and as such this project's governance is closer to apollo. As a semi-British person I feel the need to clarify this is not a passive aggressive slight, I appreciate the free software as-is and owners' management of their repo should be fully up to their needs and wants.

We will seek to solve this issue a few times more in the coming months, I suspect. Since there's a smattering of interest I will post updates here if there are successes.

[glasser]

@MarcelCutts Hi just to clarify: would merging and releasing #270 resolve your issue?

[abernix]

I just gave #270 a final review and a merge; it looks good to get out into a release that will hopefully validate that this fix works and get folks who are pinned to an older version (e.g., @MarcelCutts) onto a published version.

[abernix]

Please try @apollo/gateway@0.24.4, which (via @apollo/query-planner@0.0.13) includes @apollo/query-planner-wasm@0.2.3 that shipped with the fix (and a couple follow-up commits) provided in #270. Would be happy to hear feedback about if this worked!

[vasicvuk]

Hi @abernix I tried the latest versions and i still have an error Cannot find module 'util'. Downgrading to 0.19.1 works

[Innocent-Akim]

Failed to compile.

./src/cardano/serialization-lib/@emurgo/cardano-serialization-lib-browser/cardano_serialization_lib_bg.wasm
Module parse failed: magic header not detected
File was processed with these loaders:

• ./node_modules/wasm-loader/index.js
  You may need an additional loader to handle the result of these loaders.
  Error: magic header not detected