Support 64 bit NodeJS executable

[mhoeger]

From @AshleyGrant on November 19, 2018 17:15

Switching to the 64 bit runtime does not switch to a 64 bit version of Node. Unfortunately, this causes issues as there are some NPM packages that only support running on 64 bit Node. One in particular is "sharp." This is an image manipulation library that is orders of magnitude faster than alternatives, but only supports running on 64 bit Node due to it calling to DLLs that are compiled to run on x64.

As it currently stands, Node developers have no way to utilize 64 bit Node on a consumption plan. Switching to the 64 bit Node executable when running the function app in 64 bit mode would solve this problem.

Copied from original issue: Azure/Azure-Functions#1041

We cannot deploy 64 bit nodejs and make breaking changes in underlying infrastructure of this behavior, especially because app service customers will be affected. However, we can introduce this change with the introduction of ~ targets for WEBSITE_NODE_DEFAULT_VERSION.

Involves:

• deploy 64 bit nodejs version for each major version: Deploy 64-bit versions of node #201
• update dwas logic to search for 64-bit version if ~ targeted by WEBSITE_NODE_DEFAULT_VERSION: Support tilda targeting of latest major version for WEBSITE_NODE_DEFAULT_VERSION  Azure-Functions#1137

[alexisbg]

It is really boring. I need to find an alternative to Azure Functions. Is there any update about this issue?

[mhoeger]

Sorry for the delay. We don't fully support a 64 bit Node.js executable. However, we do have a workaround to "bring your own node executable" and then point to it.

Bringing your own node.exe

• In Kudu, which will be at [sitename].scm.azurewebsites.net, go to Debug Console.
• Drop the node.exe you want to use in D:\home (can be in root, can be nested)
• Take note of the path (ex: D:\home\node)

Pointing to a custom node.exe

• Add the AppSetting languageWorkers:node:defaultExecutablePath and set it to the path (from before) to your node.exe (ex: D:\home\node)

Again, this is a not ideal workaround, but let me know if you are not unblocked by this.

[alexisbg]

Thank you for your answer. It is also required to switch Platform to 64-bit in Application Settings for this to work.

[ChrisSmith5]

@mhoeger I followed your instructions and loaded from a x64 node.exe. However, I am still getting the following error when loading the Sharp node module:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.postProcessMedia ---> Microsoft.Azure.WebJobs.Script.Rpc.RpcException: Result: Failure
Exception: Worker was unable to load function postProcessMedia: 'Error: \?\D:\home\site\wwwroot\postProcessMedia\node_modules\sharp\build\Release\sharp.node is not a valid Win32 application.
\?\D:\home\site\wwwroot\postProcessMedia\node_modules\sharp\build\Release\sharp.node'
Stack: Error: \?\D:\home\site\wwwroot\postProcessMedia\node_modules\sharp\build\Release\sharp.node is not a valid Win32 application.
\?\D:\home\site\wwwroot\postProcessMedia\node_modules\sharp\build\Release\sharp.node
at Object.Module._extensions..node (module.js:681:18)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at bindings (D:\home\site\wwwroot\postProcessMedia\node_modules\bindings\bindings.js:112:48)
at Object. (D:\home\site\wwwroot\postProcessMedia\node_modules\sharp\lib\constructor.js:10:34)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object. (D:\home\site\wwwroot\postProcessMedia\node_modules\sharp\lib\index.js:3:15)
at Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\Rpc\WorkerLanguageInvoker.cs:line 74
at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionInvokerBase.cs:line 84
at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task1 src) in C:\azure-webjobs-sdk-script\src\WebJobs.Script\Description\FunctionGenerator.cs:line 225 at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2.InvokeAsync(Object instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 63
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 556
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstance instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 503
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstance instance, ParameterHelper parameterHelper, IFunctionOutputDefinition outputDefinition, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 439
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstance instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 249
--- End of inner exception stack trace ---
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstance instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 293

[alexisbg]

\D:\home\site\wwwroot\postProcessMedia\node_modules\sharp\build\Release\sharp.node is not a valid Win32 application.
Are you sure your function runs on the 64-bit node.exe?
If you print a process.arch in your code, do you see 'x64' and not 'ia32'?

[AshleyGrant]

I was not able to get past running npm install It doesn't seem to use the 64 bit version of Node when running npm install during deployment. This stops me from successfully deploying.

[alexisbg]

Sadly while Git/CI npm install, a deprecated Node.js 6.x is called instead of your 64-bit node.exe.

The only way is to upload a ZIP file containing your compiled prod dependencies to a blob container. This can be achieved with WEBSITE_RUN_FROM_PACKAGE app setting together with "bundledDependencies" and "files" properties in your package.json.

[mhoeger]

@ahmelsayed - I'm guessing that choosing the bitness of node.exe will be much easier to support on the Linux offering. Is that right?

[AshleyGrant]

I have to say this is extremely frustrating. IMHO, FaaS offerings should do their best to abstract away the underlying infrastructure. This issue lays bare the underlying infrastructure that Azure Functions is built on. Frankly, the underlying OS shouldn't matter to a developer building on a FaaS platform.

I find it hard to understand how Azure Functions, a product that was first announced less than 3 years ago, would have an architecture that makes implementing support for a 64 bit platform so difficult. It's not like x64 support was yet to be implemented in Node for Windows. It has been supported since at least v4.0.0 released in September of 2015.

Why is changing the bitness of the .NET runtime a simple toggle switch in the UI, but changing which NodeJS executable is used a Herculean effort? Is it not possible to have a different VM image to use that has Node x64 installed?

And frankly, I would venture to guess that the vast majority of Windows-based developers are developing on machines that run Node x64, given that it is the default download on the NodeJS website:

So for Azure Functions to run the 32 bit version of Node is a real problem for Node developers who run windows and deploy to Azure functions.

I look forward to discussing this issue at the MVP Summit in a few weeks @eduardolaureano.

[mhoeger]

@AshleyGrant - I am sorry about the frustrating experience. I think you are primarily asking for all functions to run on a 64 bit host and a 64 bit node.exe by default. We have changed the default on the Linux offering to run on 64 bit node (Linux consumption apps are currently in preview). It's a bit trickier for the Windows offering. To provide some context, the changes we made to have JavaScript code run directly on a Node.js process instead of through the Edge.js .NET library were pretty recent. With this, we've uncovered some gaps in the underlying infrastructure such as a lack of deployed 64 bit node executables on worker machines. Azure Functions relies on spare compute from App Service, and this is an App Service limitation that we have run into. I've started talking to @paulbatum about finding the appropriate owner to update the App Service machines with 64 bit node versions. It's a more niche scenario so they haven't heard all too much noise about it, but it is a change that's long overdue. Although our goal is abstraction, as you've said, I do hope you understand that getting to that point can require changes that have a broad scope and impacts beyond the immediate situation. The toggle not applying to node.exe is absolutely a gap in expectation / experience, and we appreciate your patience as we fix that.

[AshleyGrant]

I'm not necessarily asking for all functions to run on a 64 bit host w/64 bit Node by default. I do think that 64 bit Node should be used when running on a 64 bit host, though.

Thank you for the explanation of the underlying constraints. Hopefully this can be resolved sooner rather than later. Let's please keep this issue open until then, though.

[zavr-1]

@AshleyGrant @ChrisSmith5 when using git deployment, you can install sharp by setting npm_config_arch = x64 to allow that old npm to install x64 dependencies, and then use the solution by @mhoeger to run x64 node env.

[AshleyGrant]

I've come up with a workaround that allows me to bring a custom Node executable along for the ride when I "build" my application using Azure Devops. I'm mostly using the strategy specified by @mhoeger, but distributing node.exe as part of my application zip file. Details here: https://aurelia.ninja/2019/04/04/a-strategy-for-deploying-a-custom-node-executable-to-azure-functions-on-windows/

[AshleyGrant]

@zavr-1 I've confirmed that your solution works, but I figure if I'm going far enough to include my own node executable in my demo code, I might as well use a CI/CD tool (Azure Devops in my case) to orchestrate the build. My solution also eliminates the need for manually copying node.exe. In a pinch, I would use your solution, though!

[nzthiago]

@mhoeger should this still be open? It seems the two work items you mentioned needed closing to fix this are done? Or is Azure/Azure-Functions#1216 a dependency still?

[mhoeger]

@nzthiago - It's almost out!!

In its final form, we'll let people target 64-bit node by choosing to run their site as 64-bit AND setting WEBSITE_NODE_DEFAULT_VERSION to a value that targets a major version with a tilda (ex: ~8 and ~10). This will default them into using the latest major version 64-bit node without extra configuration needed. The change is being deployed on our underlying infrastructure, but won't be available til the end of July or so until later. Keeping it open until it's fully supported. Thanks for the issue cleanup search btw :)

[nzthiago]

Great - fully agree with not closing until it's done ;) Thank you for the update

[mhoeger]

Update: rolling out this feature has been delayed. However, there is an easier temporary workaround. You can add the app setting languageWorkers:node:defaultExecutablePath with the value D:\Program Files\nodejs\10.15.2\node to target a 64-bit version of 10.15.2. This is a version that is already deployed on machines and does not require you to manually deploy node.exe.

Thanks for your patience all!

[krankin]

@mhoeger -- I've set up a function all with the settings listed above. When I go the console and
node -p "process.arch" I still see ia32.
node -v gives me v10.15.2 .
which node gives /d/Program Files (x86)/nodejs/10.15.2/node

I have my setting as you mentioned above, platform set to 64 bit. Any idea what I might be missing?

[YannickRe]

@krankin try setting WEBSITE_NODE_DEFAULT_VERSION to D:\Program Files\nodejs\10.15.2
I believe that worked for me.

[krankin]

@YannickRe -- 👍 That worked!

[mhoeger]

@krankin - try doing process.arch from inside a function app, this is a bit hacky so it won't propagate to the kudu/scm/console environment. @YannickRe - never thought about that's an interesting workaround too! Clever!

[YannickRe]

@mhoeger It's the only way I consistently get the 64-bit binaries across the platform (running the code, testing in the Kudu Console, when doing Kudu CI with deployment center, etc).
Got the idea/solution from this thread (which could also use a follow up): projectkudu/kudu#1914

Only difference is with the languageWorkers:node:defaultExecutablePath setting: in this issue you propose D:\Program Files\nodejs\10.15.2\node and in the other issue you suggested D:\Program Files\nodejs\10.15.2.
Should both work? Or just one, and which one is correct then?

Looking forward to having it easier in the future ;)

[mawtex]

I get 64-bit in the function app and 32-bit in Kudu with the below:

languageWorkers:node:defaultExecutablePath = D:\Program Files\nodejs\10.15.2\node
WEBSITE_NODE_DEFAULT_VERSION = ~10

Setting WEBSITE_NODE_DEFAULT_VERSION to D:\Program Files\nodejs\10.15.2 instead give me 64 bit both places.

So, I do not need 64 bit default in Kudu, but it did confuse me for a long time, since I used Kudu to try to verify the architecture. But for all purposes the workaround from @mhoeger works for the function app.

Thanks!

[emily-curry]

Hi @mhoeger, thank you for the workaround! This appears to be exactly what I needed. Can we expect D:\Program Files\nodejs\10.15.2\node to continue to contain the 64bit version of node 10.15.2 going forward, or might that be changed unexpectedly (causing an outage for my application)?

[mhoeger]

@emily-curry - yes, you can expect that to keep working going forward! It will be good to switch over to ~10 when that is available, but even then we don't have any plans of removing 10.15.2 any time soon.

[Cpcrook]

Only difference is with the languageWorkers:node:defaultExecutablePath setting: in this issue you propose D:\Program Files\nodejs\10.15.2\node and in the other issue you suggested D:\Program Files\nodejs\10.15.2.
Should both work? Or just one, and which one is correct then?

Looking forward to having it easier in the future ;)

@YannickRe one of my devs was struggling with this today. It appears that
WEBSITE_NODE_DEFAULT_VERSION should be D:\Program Files\nodejs\10.15.2 (absolute path to the node folder) and languageWorkers:node:defaultExecutablePath should be the absolute path to the actual node executable.

[mhoeger]

Two updates to close this issue:

1. For Function Apps deployed on Linux, a 64 bit Node.js executable is default.
2. For Function Apps deployed on Windows, you can change a few settings to get a 64 bit executable.
   • First, go to "Configuration" in the Azure Portal
     
   • Then, either change or ensure that your WEBSITE_NODE_DEFAULT_VERSION is set to ~12. Note: we support major version targeting like this for even-numbered 10+ versions that are LTS
     
   • Then, go to "General settings" and change the platform bitness.
     

A few additional considerations if you are choosing between Windows vs. Linux:

• We have perf improvements coming up, but for default scenarios. A 64 bit platform on Windows is not the default and won't benefit from those improvements.
• Linux function apps for Node.js currently have better performance, but some limitations such as no portal editing and probably most notably, no CORS support

[janpio]

This is very nice.

Is there a way to set these platform settings via CLI when creating a function as well or doe we have to use the web UI for that? Was hoping to find something on https://docs.microsoft.com/en-us/azure/azure-functions/functions-app-settings, but that is not the case.