Support mono-repos in deployment

[laurenzlong]

See: firebase/firebase-functions#172

Version info

3.17.3

Steps to reproduce

Expected behavior

Actual behavior

[dinvlad]

This seems to work in my setup, i.e. deploy picks up packages from the root node_modules, even though package.json for that is located under the api/ workspace (I've used a different folder name instead of functions/). Is there anything else that needs fixing here?

EDIT: Moreover, I copy package.json into api/dist to be used by deploy.

// firebase.json
  ...
  "functions": {
    "source": "api/dist"
  },
  ...

So, 2 levels of nesting still resolve the root node_modules successfully.

[orouz]

@dinvlad could you share a repo?

[dinvlad]

@orouz unfortunately not yet, it's closed source for now.

[audkar]

Does anyone managed to tackle this problem? Sharing simple example project would be very useful.

[jthegedus]

@audkar Currently I just use lerna.js.org and it's run command to execute an npm script in each subfolder with this folder structure:

- service1/
|  - .firebaserc
|  - firebase.json
- service2/
|  - .firebaserc
|  - firebase.json
- app1/
|  - .firebaserc
|  - firebase.json
- app2/
|  - .firebaserc
|  - firebase.json
- firestore/
|  - firestore.rules
|  - firestore.indexes.json
- etc...

Ensuring the firebase.json files for each service don't stomp on one another is left up to the user. Simple conventions of using function groups and multi-site name targeting means this is solved for Cloud Functions and Hosting. Still haven't got a solution for Firestore/GCS rules yet, though splitting them up may not be ideal...

discussed here previously - #1116

[audkar]

@jthegedus thank you for your reply. But I think issue of this ticket is different. I am trying to use yarn workspaces. And seems that firebase tools doesn't pickup symlink dependencies when uploading functions

[jthegedus]

Ah fair enough, I've avoided that rabbit hole myself

[dinvlad]

Could you elaborate what the issue is? As mentioned above, I just use bare Yarn with api and app workspaces in it, and I build them using yarn workspace api build && yarn workspace app build (with build script specific to each workspace). The build scripts

1. compile TS code with outDir into api/dist and app/dist respectively
2. copy the corresponding package.json files into dist directories
3. copy yarn.lock from the root folder, into dist directories

Then I just run yarn firebase deploy from the root folder, and it picks up both api/dist and app/dist without any hiccups. My firebase.json looks like

  "functions": {
    "source": "api/dist"
  },
  "hosting": {
    "public": "app/dist",

Unfortunately, I still can’t share the full code, but this setup is all that matters, afaik.

[dinvlad]

Also, I might be wrong but I think the firebase deploy script doesn’t actually use your node_modules directory. I think it just picks up the code, package.json, and yarn.lock from the dist directories, and does the rest.

[samtstern]

That's true. The default value of "functions.ignore" in firebase.json is ["node_modules"] so it's not uploaded. I believe you can override that though if you want to ship up some local modules.

…

On Mon, Jun 17, 2019, 6:58 PM Denis Loginov ***@***.***> wrote: Also, I might be wrong but I think the firebase deploy script doesn’t actually use your node_modules directory. I think it just picks up the cod, package.json, and yarn.lock from the dist directories, and does the rest. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#653?email_source=notifications&email_token=ACATB2U73VS2KIILUVRFFB3P3A6NPA5CNFSM4EOR24GKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX46ABQ#issuecomment-502915078>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACATB2U3Q2TLVBICRJ3B5OLP3A6NPANCNFSM4EOR24GA> .

[jthegedus]

@dinvlad Yes, it requires the package.json and whichever lock file you use as it installs deps in the cloud post deployment.

I believe the scenario originally outlined in the other issue was using a shared package within the workspace and some issues with scope-hoisting. As I was not using yarn this way I can only speculate from what I have read there.

[dinvlad]

@samtstern @jthegedus thanks, good to know!

[audkar]

Seems we all talk about different problems. I will try to describe yarn workspaces problem.

Problematic project

project layout

- utilities/
|  - package.json
- functions/
|  - package.json
- package.json

./package.json

{
  "private": true,
  "workspaces": ["functions", "utilities"]
}

functions/package.json

{
  <...>
  "dependencies": {
    "utilities": "1.0.0",
    <...>
  }
}

Problem

Error during function deployment:

Deployment error.
Build failed: {"error": {"canonicalCode": "INVALID_ARGUMENT", "errorMessage": "`gen_package_lock` had stderr output:\nnpm WARN deprecated left-pad@1.3.0: use String.prototype.padStart()\nnpm ERR! code E404\nnpm ERR! 404 Not Found: utilities@1.0.0\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /builder/home/.npm/_logs/2019-06-18T07_10_42_472Z-debug.log\n\nerror: `gen_package_lock` returned code: 1", "errorType": "InternalError", "errorId": "1971BEF9"}}

Functions works fine locally on emulator.

Solutions tried

Uploading node_modules (using functions.ignore in firebase.json). Result is same.

My guess that it is because utilities is created as syslink in node-modules node_modules/utilities -> ../../utilities

Could it be that firebase-tools doesn't include content of symlink'ed modules when uploading (no dereferencing)?

[dinvlad]

Sorry, could you clarify which folder your firebase.json lives in (and show its configuration section for functions)?

[audkar]

firebase.json was in root folder. Configuration was standard. Smth like this:

  "functions": {
    "predeploy": [
      "yarn --cwd \"$RESOURCE_DIR\" run lint",
      "yarn --cwd \"$RESOURCE_DIR\" run build"
    ],
    "source": "functions",
    "ignore": []
  },
  <...>

everything was deployed as expected (including node_modules) except node_modules/utilities which is symlink.

---

I manage to workaround this issue by writing few scripts which:

• create packages for each workspace (yarn pack). e.g. this creates utilities.tgz.
• moving all output to some specific dir.
• modifying package.json to use tgz files for workspace dependencies. e.g. dependencies { "utilities": "1.0.0" -> dependencies { "utilities": "file:./utilities.tgz"
• deploying that dir to firebase

output dir content before upload:

- dist
|  - lib
|  | -index.js
|  - utilities.tgz
|  - package.json <---------- This is modified to use *.tgz for workspaces

[0x80]

@audkar Today I ran into the same issue as you.

I am new to both Lerna and Yarn workspaces. As I understand it you can also just use Lerna. Would that help in any way?

Your workaround seems a bit complicated for me 🤔

Also wondering, what is `--cwd "$RESOURCE_DIR" for?

[audkar]

--cwd stands for "current working directory" and $RESOURCE_DIR holds value for source dir (functions in this case). Adding this flag will make yarn to be executed in functions dir instead of root

[0x80]

@audkar Ah I see. So you could do the same with yarn workspace functions lint and yarn workspace functions build

[0x80]

@dinvlad It is unclear to me why you are targeting the dist folder and copying things over there. If you build to dist, but leave the package.json where it is and point main to dist/index.js then things should work the same no? You should then set source to api instead of api/dist.

[0x80]

@dinvlad I learned the yarn workspace command from your comments, but can't seem to make it work for some reason. See here. Any idea?

Sorry for going a bit off-topic here. Maybe comment in SO, to minimize the noise.

[dinvlad]

@0x80 I copy package.jsonto api/dist and point firebase.json to api/dist so only the "built" files are packaged inside the cloud function. I'm not sure what will happen if I point firebase.json to api - perhaps it will still be smart enough to only package what's inside api/dist (based on main attribute in package.json). But I thought it was cleaner to just point to api/dist.

Re yarn workspace, I responded on SO ;)

[0x80]

@dinvlad it will bundle the root of what you point it to, but you can put everything that you don't want included in the firebase.json ignore list.

I've now used a similar workaround to @audkar.

{
  "functions": {
    "source": "packages/cloud-functions",
    "predeploy": ["./scripts/pre-deploy-cloud-functions"],
    "ignore": [
      "src",
	  "node_modules"
    ]
  }
}

Then the pre-deploy-cloud-functions script is:

#!/usr/bin/env bash

set -e

yarn workspace @gemini/common lint
yarn workspace @gemini/common build

cd packages/common
yarn pack --filename gemini-common.tgz
mv gemini-common.tgz ../cloud-functions/
cd -

cp yarn.lock packages/cloud-functions/

yarn workspace @gemini/cloud-functions lint
yarn workspace @gemini/cloud-functions build

And packages/cloud-functions has an extra gitignore file:

yarn.lock
*.tgz

[kaminskypavel]

here's what worked for me

- root/
|  - .firebaserc
|  - firebase.json
- packages/
  | - package1/
  | - functions/
    | - dist/
    | - src/
    | packages.json

and in the root/firebase.json :

{
	"functions": {
		"predeploy": "npm --prefix \"$RESOURCE_DIR\" run build",
		"source": "packages/functions"
	}
}

[0x80]

@kaminskypavel is your packages/functions depending on packages/package1 (or some other sibling package)?

[kaminskypavel]

@0x80 positive.

[0x80]

I think there was something fundamental I misunderstood about monorepos. I assumed you can share a package and deploy an app using that package without actually publishing the shared package to NPM.

It seems that this is not possible, because deployments like Firebase or Now.sh will usually upload the code and then in the cloud do an install and build. Am I correct?

@kaminskypavel I tried your approach and it works, but only after publishing my package to NPM first. Because in my case the package is private I initially got a "not found" error, so I also had to add my .npmrc file to the root of the cloud functions package as described here

@audkar Are you publishing your common package to NPM, or are you like me trying to deploy with shared code which is not published?

[StephenHaney]

@0x80 I'm with you on this understanding - I think Firebase Function deployments are just (erroneously) assuming that all packages named in package.json will be available on npm, in the name of speeding up deployments.

As yarn workspace setups are becoming more popular, I imagine more folks are going to be surprised that they can't use symlinked packages in Firebase Functions – especially since they work fine until you deploy.

[kollein]

Thanks bro! @0x80 the isolate-package works like a charm for me.

[BartSpangenberg]

Solution: Similar to this solutions: #653 (comment) from @johanneskares

I believe it is quite elegant and it works fine. Just create a postdeploy.js and predeploy.js file and modify firebase.json

Npm workspaces are used in the project (but will work with any workspace).

---

File structure

/root
--/functions/package.json
--/functions/predeploy+postdeploy.js
--src/packages/packageNames
--firebase.json

---

file: /firebase.json

  "functions": {
    "predeploy": ["npm --prefix \"$RESOURCE_DIR\" run build", "node \"$RESOURCE_DIR/predeploy.js\""],
    "postdeploy": ["node \"$RESOURCE_DIR/postdeploy.js\""],
    "source": "functions"
  },

---

file: /functions/predeploy.js

const { execSync } = require('child_process');
const path = require('path');
const fs = require('fs');

// The tarball files are the building blocks of npm packages
const createTarballs = (packagePath) => {
    // Navigate to the folder in which the package is located
    process.chdir(path.join(__dirname, packagePath));

    // Compile the package into JS and then pack it
    // This iwill create a .tgz (tarball) file inside the package folder
    execSync('npm run build && npm pack', { stdio: 'inherit' });

    // Move the .tgz file to the funcitons folder
    const tgzFile = fs.readdirSync('./').find(file => file.endsWith('.tgz'));
    if (tgzFile) {
      fs.renameSync(tgzFile, path.join(__dirname, '/', tgzFile));
    }
};

  let packagePaths = ["../src/packages/entities", "../src/packages/utility"]

  packagePaths.forEach(p => createTarballs(p));

  // Navigate back to the functions directory to install packages
  process.chdir(__dirname);

  // Install the .tgz packages using npm
  const tgzFiles = fs.readdirSync('./').filter(file => file.endsWith('.tgz'));
  for (const tgzFile of tgzFiles) {
    execSync(`npm install ./${tgzFile}`, { stdio: 'inherit' });
  }

---

file: /functions/postdeploy.js

const fs = require('fs');
const path = require('path');

// Define the path to the package.json file in the functions folder
const packageJsonPath = path.join(__dirname, 'package.json');

// Define the names of the packages you want to remove
const packagesToRemove = ['@org/packagename', '@org/packagename2'];

  // Read and parse the package.json file
  const packageJsonData = JSON.parse(fs.readFileSync(packageJsonPath, 'utf-8'));

  // Remove the specified packages from dependencies
  packagesToRemove.forEach((pkg) => {
    if (packageJsonData.dependencies && packageJsonData.dependencies[pkg]) {
      delete packageJsonData.dependencies[pkg];
    }
  });

  // Write the modified package.json back to disk
  fs.writeFileSync(packageJsonPath, JSON.stringify(packageJsonData, null, 2));

  // Delete the .tgz files from the functions folder
  fs.readdirSync(__dirname).forEach((file) => {
    if (file.endsWith('.tgz')) {
      fs.unlinkSync(path.join(__dirname, file));
    }
  });

[Psycarlo]

+1. Would be awesome to support turborepo + pnpm

[0x80]

I have been busy lately trying to overcome the last major hurdles and I have some exciting news to share!

• isolate-package now generates a compatible PNPM lockfile for deterministic deployments. The isolate process is now also exported as a function to embed in other code.
• I have forked firebase-tools to integrate isolate-package as part of the deploy command. As it then only isolates during deploy, it preserves live code updates when running the local emulators.
• I have worked on a monorepo example that includes multiple Firebase deployments using the isolate process. It also showcases how to use different internal package strategies including linking directly to TS code. This allows for live code updates from your internal packages without running build watch tasks.

I hope to publish an article about it later this week, but the monorepo readme and code should already contain all the information you need.

The main thing left to do is to also generate lockfiles for NPM and Yarn. For NPM there is arborist, and for Yarn I think there is some user-land code... I tried arborist briefly but NPM was giving me issues, and my focus was on PNPM because it is my preferred package manager.

I plan to give arborist another try later this week.

The integration with the firebase-tools could be improved, and I would love for the Firebase team to help me out on this, because it seems that some refactoring is needed to get all the log messages to display 100% correct information. But I think this works fine for now and because I don't expect it to be part of firebase-tools any time soon, I made the fork available on NPM.

Enjoy! 🌈

[gabyshev]

@BartSpangenberg have you seen this error? I tried to use your setup in monorepo (pnpm + turbo).

✔  functions: apps/api folder uploaded successfully
i  functions: updating Node.js 18 (1st Gen) function api:myFunc(region1)...
Build failed: npm ERR! code EUNSUPPORTEDPROTOCOL
npm ERR! Unsupported URL Type "workspace:": workspace:*

npm ERR! A complete log of this run can be found in:

However, in api/package.json i saw that script changed a reference from workspace:* to file:...tgz

[0x80]

🔥 I have now added support for NPM lockfiles in
isolate-package@1.6.0 and firebase-tools-with-isolate@12.9.1-10.

It turned out to be much easier than figuring this out for PNPM. It's also much slower than PNPM though, but that's maybe not surprising.

The latter is released as a pre-release number because I did not want to mess up the version match with the original firebase-tools. When 12.9.2 comes out I will likely release an updated version 12.9.2 of firebase-tools-with-isolate, and so on, to keep it in sync.

It would obviously be nice to also support Yarn lockfiles, but honestly, for me, it's not a priority at all, so I might wait for someone to show me how to do it or make a PR.

Have fun!

[0x80]

Sorry, I seem to have messed something up... Please hold off with using NPM lockfiles until further notice 🙏 I think it's bedtime for me.

NPM lockfiles are deploying again with isolate-package@1.6.2 and firebase-tools-with-isolate@12.9.1-11

[0x80]

Some updates:

isolate-package

• NPM lockfiles are now correct (it wasn't before) and fast to generate
• Lockfiles for Yarn v1 are now supported

firebase-tools-with-isolate has been synced with upstream firebase-tools to match to latest version 13.0.2

The mono-ts boilerplate, showcasing both, now has two alternative branches, one for NPM and one for Yarn classic.

I haven't figured out how to generate lockfiles for modern Yarn versions yet, but as a fallback it could just generate an NPM lockfile if you use a package manager that is not supported. I don't think it matters what package manager is used in your cloud deployment, as long as the isolated lockfile versions match your original lockfile.

[0x80]

🔥 More happy news from version 1.9.x

• Lockfile output is now available for all package managers, incl Yarn v4
• Modern versions of Yarn fall back to generating an NPM lockfile, which matches the Yarn lockfile versions.
• With forceNpm you can opt to always output in NPM format. So if you work with PNPM or Yarn for your monorepo setup, but you want to deploy your isolate output to an environment that only supports NPM, now you can do so.
• Spaces in monorepo paths are now handled correctly
• There are configuration options to pick or omit scripts from the manifest
• All outstanding issues have been resolved ✨

The mono-ts boilerplate now contains 4 parallel branches:

• main (using pnpm)
• use-npm (using npm)
• use-yarn-classic (using yarn v1)
• use-yarn-modern (using yarn v4)

Enjoy!

[Crocsx]

I bumped into this thread after long research because I am unable to deploy a nextJs app using Nx monorepo.

Might be my bad for not searching priorly the tools properly, but supporting a monorepo nowadays should be a must for such a big service that is firebase. I was planning to go full firebase/firestore/firebase-auth but I am stuck at this step that I thought would be the easiest ^^'

[SvenBudak]

Are there any updates here? My product owner/client is slowly forcing me to host elsewhere... but I don't want to. ^^"

[0x80]

@SvenBudak Sounds dramatic :) Are the known workarounds and solutions not sufficient for your client?

The isolate-package approach seems to be solid. Some people have also been using it for other types of deployments like google cloud functions (outside of firebase) and cloud run.

The only caveat is that using NPM is problematic if you want to deploy with pruned lockfiles to multiple platforms in parallel, because the lockfile pruning (based on npmcli Arborist) currently can not execute in parallel To get around it you could deploy in series, use PNPM or Yarn instead of NPM, or omit the lockfile (not recommended)

[0x80]

Has anyone tried the Turborepo prune command for Firebase deployments? It seems very similar to what isolate-package is doing.

Also, Nx has an option to generate a pruned lockfile. I remember looking at Nx and somehow concluding that it was not what I wanted, but I can't recall why. So I'm also curious to hear if anyone tried that.

[0x80]

Hi all,

Just a small update to tell you I have released isolate-package@1.13.0 and firebase-tools-with-isolate@13.6.2 and this includes support for Rush monorepos. It has only been tested with PNPM but I am fairly confident that NPM and Yarn will work too.

Notable recent changes:

• Add Rush monorepo support (zero-config, only tested with PNPM)
• Adopt PNPM overrides from root package manifest
• Prune PNPM lockfile packages list (this didn't affect installs, but bloated lockfile size)

[AndyOooh]

Hi @0x80

I just gave firebase-tools-with-isolate a go and I'm impressed.
It Took me 5-10 mins to switch my Turborepo project from yarn to pnpm (I was planning to anyway), install the package and get it up and running.
So far everything works:

• Emulators with live changes (using plain tsc --watch in a separate terminal).
• Emulators with vscode debugger.
• Deployments with local package (internal) dependency.

Thanks for you efforts on this!

[e-CORS]

Hey everyone,

I'm encountering an issue while using a monorepo with Bun. When I run my setup, I receive the following error:

npm ERR! Unsupported URL Type "workspace": ../../../shared

Our setup includes TypeScript and tsc-alias for handling custom paths in tsconfig.json, where we map custom-shared/* to shared/*.

Has anyone faced a similar issue or have any suggestions on how to resolve this?

Thanks in advance!

[0x80]

@e-CORS I don't think it has to do with your code. It's probably the cloud build complaining about the workspace paths in your package.json file. PNPM also uses these but the build pipeline will switch to using PNPM as the package manager when it is detected.

In your case you are using Bun as the package manager which I don't think would be supported by a cloud infrastructure that is based on Node.js.

That said, I have no experience with Bun, so I don't know what your options are.

[e-CORS]

Hey @0x80 by running pnpm install and adding @google-cloud/functions-framework ( since it was a requirement ) we were able to deploy without an issue, so your suggestion of PNPM really helped us, thanks!

[KalebKloppe]

When I deploy a Next.Js app in my pnpm monorepo using firebase-tools-with-isolate

• ✅ Firebase Cloud Functions succeeds
• ✅ Next.Js without Cloud Functions succeeds
• ❌ Next.Js with Cloud Functions (image optimization, SSR, etc.) fails

Building a Cloud Function to run this application. This is needed due to:
• non-static route /[routerUrl]

npm error code EUNSUPPORTEDPROTOCOL
npm error Unsupported URL Type "workspace:": workspace:*

[0x80]

@KalebKloppe This is not the place to report issues for that package, please do that in its own repo. It looks like your build pipeline is using npm, which doesn't understand the pnpm format.

[staticshock]

There are some calls to npm harcoded into the firebase-tools repo, so I don't think this is necessarily about @KalebKloppe's build pipeline.

I think the line I linked is one of the things @KalebKloppe is running into.

I say this specifically because it tripped me up, too: Firebase Functions advertise some compatibility with pnpm, and yet Firebase Tools attempts to use npm when deploying the Next.js backend function.

[0x80]

@staticshock Indeed, that was also my conclusion. Firebase-tools is not detecting the used package manager. In that sense @KalebKloppe was reporting in the right repo after all, just not the right issue :)

isolate-package had a setting to force its output to use npm, but unfortunately I had removed the feature since there was no clear use-case, and it complicated the code. I've now restored the feature and published it under isolate-package@next. It'll make it into firebase-tools-with-isolate once I get the confirmation that it works.