Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

With webpack 5 and once deployed, local files are not readable from APIs anymore #24700

Closed
QuentinRoy opened this issue May 2, 2021 · 35 comments
Assignees
Labels
Output (export/standalone) Related to the the output option in `next.config.js`.

Comments

@QuentinRoy
Copy link

QuentinRoy commented May 2, 2021

What version of Next.js are you using?

10.2.0

What version of Node.js are you using?

14.0.0

What browser are you using?

Chrome,Firefox,Safari

What operating system are you using?

macOS

How are you deploying your application?

Vercel

Describe the Bug

Since updating to 10.2.0, local files are not accessible from apis anymore once deployed on Vercel. I do not know if it was on purpose, but it used to work fine before upgrading.

In both case, an internal server error is returned. After closer inspection of the logs, the error message is as follows:

[Error: ENOENT: no such file or directory, open '/app/data/metadata.yml'] {

errno: -2,

code: 'ENOENT',

syscall: 'open',

path: '/app/data/metadata.yml'

}

It went back to normal after downgrading to next 10.1.3.

EDIT: It seems the problem comes from webpack 5, it is working fine if webpack5 is disabled in next.config.js.

EDIT: It does not fail if there is only one API trying to read only one local file. It is possible the problem occurs when multiple files have to be available for reading.

EDIT: According to @jacksonhardaker, this also fails when reading files from getServerSideProps.

Expected Behavior

Local files should be readable from apis. If these are not embedded by default anymore, maybe an option could be used to indicate what files should be available?

To Reproduce

Here is an example showing APIs failing: https://github.com/QuentinRoy/vercel-local-min-example deployed at https://vercel-local-min-example.vercel.app.

Here is another example from @jacksonhardaker showing getServerSideProps failing: https://github.com/jacksonhardaker/next-webpack-5-fs-bug deployed at https://next-webpack-5-fs-bug-jwegk8q1o-jacksonhardaker1.vercel.app/.

@QuentinRoy QuentinRoy added the bug Issue was opened via the bug report template. label May 2, 2021
@QuentinRoy
Copy link
Author

QuentinRoy commented May 2, 2021

Following @mmazzarolo suggestion in #8251, I can attest the problem actually comes with webpack 5. Disabling webpack 5 after updating fixes it:

next.config.js

module.exports = {
  future: {
    webpack5: false,
  },
};

EDIT:
With newer Next versions, switching back to webpack 4 is:

module.exports = {
  webpack5: false,
};

@QuentinRoy QuentinRoy changed the title Local files not readable from APIs anymore With webpack 5 and once deployed, local files not readable from APIs anymore May 2, 2021
@QuentinRoy QuentinRoy changed the title With webpack 5 and once deployed, local files not readable from APIs anymore With webpack 5 and once deployed, local files are not readable from APIs anymore May 2, 2021
@joe-bell
Copy link
Contributor

joe-bell commented May 2, 2021

I manually opted into webpack5 and can confirm I'm also seeing this behaviour inside API routes

Example

API Route

const CONTENT_DIR = path.join(process.cwd(), "src", "content");

export const getContentByDir = async (
  dir: string,
  options?: IGetContentOptions
) => {
  const srcDir = path.join(CONTENT_DIR, dir);

  if (!fs.lstatSync(srcDir)?.isDirectory()) {
    // Error thrown to Vercel
    throw Error(`${srcDir} is not a valid directory`);
  }
  // other unrelated code
};

Vercel Logs

Error: ENOENT: no such file or directory, lstat '/var/task/src/content/blog'
    at Object.lstatSync (fs.js:1077:3)
    at getContentByDir (/var/task/.next/server/pages/api/sitemap.js:283:49)
    at getBlogPaths (/var/task/.next/server/pages/api/sitemap.js:292:23)
    at getAllPaths (/var/task/.next/server/pages/api/sitemap.js:313:67)
    at getSitemapPaths (/var/task/.next/server/pages/api/sitemap.js:314:43)
    at exports.modules.3692.__webpack_exports__.default (/var/task/.next/server/pages/api/sitemap.js:331:96)
    at apiResolver (/var/task/node_modules/next/dist/next-server/server/api-utils.js:8:7)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Server.handleApiRequest (/var/task/node_modules/next/dist/next-server/server/next-server.js:66:462)
    at async Object.fn (/var/task/node_modules/next/dist/next-server/server/next-server.js:58:580) {
  errno: -2,
  syscall: 'lstat',
  code: 'ENOENT',
  path: '/var/task/src/content/blog'
}

Additional Context

I tried adding this to my next.config.js, but had no luck:

  webpack: (config, { isServer }) => {
    if (!isServer) {
      config.resolve.fallback.fs = false;
    }

    return config;
  },

This behaviour doesn't occur when I opt-out of webpack5

@timneutkens timneutkens added kind: bug and removed bug Issue was opened via the bug report template. labels May 5, 2021
@timneutkens timneutkens added this to the Iteration 20 milestone May 5, 2021
@ijjk
Copy link
Member

ijjk commented May 5, 2021

Hi, can someone who's run into this provide a repo with a minimal reproduction, we aren't able to reproduce this with the above provided information

@ijjk ijjk added the please add a complete reproduction Please add a complete reproduction. label May 5, 2021
@QuentinRoy
Copy link
Author

QuentinRoy commented May 5, 2021

Hi @ijjk, thank you for looking into this.
I investigated the problem a bit more. It seems it is working fine when there is only one api reading one local file. However, if there are two apis reading two local files, it breaks.
Here is a minimal reproduction repository: https://github.com/QuentinRoy/vercel-local-min-example.
This is deployed here: https://vercel-local-min-example.vercel.app

@jacksonhardaker
Copy link
Contributor

jacksonhardaker commented May 5, 2021

Good tip @QuentinRoy I can confirm that the addition of a second read appears to reveal the issue.
Repo: https://github.com/jacksonhardaker/next-webpack-5-fs-bug
Deployment: https://next-webpack-5-fs-bug-mfyukza5m-jacksonhardaker1.vercel.app/ https://next-webpack-5-fs-bug-jwegk8q1o-jacksonhardaker1.vercel.app/

edit: updated the deployment url

@wworrall
Copy link

wworrall commented May 6, 2021

This is a major issue causing Vercel deployments to break silently (build completing successfully). Rolling back to webpack 4 for now - please fix asap, Vercel 😍. Also, perhaps add something to your tests as to which projects are automatically compatible with webpack5 https://nextjs.org/docs/messages/webpack5

@joe-bell
Copy link
Contributor

joe-bell commented May 6, 2021

@wworrall your tone is super unproductive and contributes nothing to this issue

@QuentinRoy
Copy link
Author

QuentinRoy commented May 6, 2021

@jacksonhardaker, does your deployment matches the code in your repo? The deployed app appears functional.

@jacksonhardaker
Copy link
Contributor

@QuentinRoy Thanks for the ping, I must have pasted the incorrect url. Updated.

@marbiano
Copy link
Contributor

marbiano commented May 25, 2021

We have a slightly similar case. We can read files but only if one API endpoint is doing the fs.readFile. The moment we add a second endpoint it fails!

Thanks for taking a look and providing help, Vercel crew. 🙏

EDIT: just realized it's the same issue as @QuentinRoy described above. Good to know we are not the only ones!

@carl0s
Copy link

carl0s commented Oct 2, 2021

Any update on this?

@ijjk
Copy link
Member

ijjk commented Oct 2, 2021

@carl0s did you try the experimental.nftTracing config already and experienced issues still? If so you can try the latest version of Next.js v11.1.3-canary.45 with the now renamed experimental.outputFileTracing.

@carl0s
Copy link

carl0s commented Oct 5, 2021

I will totally try that!

@carl0s
Copy link

carl0s commented Oct 6, 2021

I tried that with the mentioned version of Next.js, v11.1.3-canary.45.
I'm trying to have a next build with next-18n and I have this error.

Error occurred prerendering page "/de/article/test-article". Read more: https://nextjs.org/docs/messages/prerender-error Error: Cannot find module '[redacted]/next-i18next.config.js' at webpackEmptyContext ([redacted]/.next/serverless/pages/[...slug].js:11:10) at [redacted]/.next/serverless/chunks/709.js:245:72 at processTicksAndRejections (node:internal/process/task_queues:96:5)

I removed some reference to local paths, as they might be confusing.
Without I18n works fine, and with webpack 4 it builds fine. Though we need webpack 5 to work correctly.

@adrai
Copy link

adrai commented Oct 6, 2021

I tried that with the mentioned version of Next.js, v11.1.3-canary.45. I'm trying to have a next build with next-18n and I have this error.

Error occurred prerendering page "/de/article/test-article". Read more: https://nextjs.org/docs/messages/prerender-error Error: Cannot find module '[redacted]/next-i18next.config.js' at webpackEmptyContext ([redacted]/.next/serverless/pages/[...slug].js:11:10) at [redacted]/.next/serverless/chunks/709.js:245:72 at processTicksAndRejections (node:internal/process/task_queues:96:5)

I removed some reference to local paths, as they might be confusing. Without I18n works fine, and with webpack 4 it builds fine. Though we need webpack 5 to work correctly.

@carl0s This is a "known" issue... read about it here: i18next/next-i18next#1202 (comment) and here: i18next/next-i18next#1384

@eric-burel
Copy link
Contributor

eric-burel commented Oct 15, 2021

The update to 11.3 with experimental.outputFileTracing seems to stop this error message from happening but leads to new issues with Material UI v5 / Emotion, I get this:

/node/index.js',
    '/var/task/node_modules/@mui/system/index.js',
    '/var/task/node_modules/@mui/material/node/styles/adaptV4Theme.js',
    '/var/task/node_modules/@mui/material/node/styles/index.js',
    '/var/task/.next/server/pages/_document.js',
    '/var/task/node_modules/next/dist/server/require.js',
    '/var/task/node_modules/next/dist/server/load-components.js',
    '/var/task/node_modules/next/dist/server/api-utils.js',
    '/var/task/node_modules/next/dist/server/next-server.js',
    '/var/task/___next_launcher.js',
    '/var/runtime/UserFunction.js',
    '/var/runtime/index.js'
  ]
}
2021-10-15T07:57:31.238Z	f5ecb81a-7cb9-49fa-9f66-4595e6f427d0	ERROR	Error: Cannot find module 'stylis'
Require stack:
- /var/task/node_modules/@emotion/cache/dist/emotion-cache.cjs.prod.js
- /var/task/node_modules/@emotion/cache/dist/emotion-cache.cjs.js
... // etc.

Seems similar to this ticket: emotion-js/emotion#850 and this old SO question (https://stackoverflow.com/questions/62526318/nextjs-emotionjs-and-vercel-error-err-require-esm-must-use-import-to-load-es). Installing stylis explicitely doesn't fix the issue.

Only one route of my app is affected I don't really get why specifically, probably because it's an MDX file, with @next/mdx.
It also affects API routes despite them not using @emotion at all.

I can reproduce only on Vercel so far, not locally. I can share a non-minimal repro if needed, this affect an open source project.

@ijjk
Copy link
Member

ijjk commented Oct 19, 2021

@eric-burel the experimental.outputFileTracing has been updated in the latest version of Next.js v11.1.3-canary.86, please update and give it a try! If you are still seeing issues please provide the mentioned repo for further investigation.

@adrai
Copy link

adrai commented Oct 20, 2021

v11.1.3-canary.86 behaves still the same for next-i18next issue: #24700 (comment)

@eric-burel
Copy link
Contributor

@ijjk First try after this update canary.86 gives me ModuleNotFoundError: Module not found: Error: Can't resolve 'querystring' in '/vercel/path0/node_modules/next/dist/compiled/native-url' during Vercel build and locally as well.
It's probably just an unrelated regression in this version, it's fixed by installing querystring, explicitely.

Then after a second try, it seems to partially fix the issue: I don't have trouble with API routes anymore.

But my MDX page still doesn't work.

2021-10-20T15:24:05.039Z	2f2bc57a-9453-45f1-8b7c-785d883c2e14	ERROR	TypeError: Cannot destructure property 'components' of 'object null' as it is null.
    at Server.renderToResponseWithComponents (/var/task/node_modules/next/dist/server/next-server.js:985:90)
    at Server.renderErrorToResponse (/var/task/node_modules/next/dist/server/next-server.js:1428:35)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Server.pipe (/var/task/node_modules/next/dist/server/next-server.js:866:25)
    at async Object.fn (/var/task/node_modules/next/dist/server/next-server.js:671:21)
    at async Router.execute (/var/task/node_modules/next/dist/server/router.js:205:32)
    at async Server.run (/var/task/node_modules/next/dist/server/next-server.js:843:29)
    at async Server.handleRequest (/var/task/node_modules/next/dist/server/next-server.js:293:20)
    at async Server.<anonymous> (/var/task/___next_launcher.js:32:9)

This page works locally but still not in Vercel.

You can get the code here: https://github.com/eric-burel/vulcan-next/tree/main
Faulty page is there: https://vulcan-next.vercel.app/learn

@ijjk
Copy link
Member

ijjk commented Oct 20, 2021

@adrai it looks like the config is being included correctly in the latest canary while deploying the example, test deployment done here it seems there is a different issue which is responded about on the initial issue here.

@ijjk
Copy link
Member

ijjk commented Oct 20, 2021

@eric-burel it looks like your issue is related to next-i18next as well so the above comment is relevant for your case as well.

@adrai
Copy link

adrai commented Oct 20, 2021

@adrai it looks like the config is being included correctly in the latest canary while deploying the example, test deployment done here it seems there is a different issue which is responded about on the initial issue here.

@ijjk I still get: error - Error: Cannot find module '/Users/usr/Projects/next-i18next/examples/simple/next-i18next.config.js'

steps to reproduce:

  1. clone https://github.com/isaachinman/next-i18next
  2. add experimental flag here: https://github.com/isaachinman/next-i18next/blob/master/examples/simple/next.config.js
  3. use new canary version of next here: https://github.com/isaachinman/next-i18next/blob/master/examples/simple/package.json#L12
  4. npm run run-example and visit http://localhost:3000

@ijjk
Copy link
Member

ijjk commented Oct 20, 2021

@adrai that appears to be an issue with running the package from the monorepo, the package.json for the example should look like this:

{
  "name": "v10",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "scripts": {
    "dev": "next",
    "build": "next build",
    "start": "next start"
  },
  "dependencies": {
    "next": "canary",
    "next-i18next": "latest",
    "react": "^17.0.2",
    "react-dom": "^17.0.2"
  },
  "devDependencies": {
    "prop-types": "^15.7.2"
  }
}

and the modules should be installed specific to the example along with running yarn next build inside of the example folder. The reason it isn't finding the config is because an incorrect current working directory is being used.

@adrai
Copy link

adrai commented Oct 20, 2021

@ijjk you're right

@eric-burel
Copy link
Contributor

eric-burel commented Oct 20, 2021

Much thanks for the help @ijjk I've made some progress.

I ended up with the config described there in the relevant next-i18next ticket i18next/next-i18next#661 (comment)

However, I have a new issue :/

2021-10-20T18:55:03.846Z	63b14614-4efe-45fb-877f-d362013ab373	ERROR	Error: Cannot find module '@mdx-js/react'
Require stack:
- /var/task/.next/server/pages/learn/intro-online.js
- /var/task/node_modules/next/dist/server/require.js
- /var/task/node_modules/next/dist/server/load-components.js
- /var/task/node_modules/next/dist/server/api-utils.js
- /var/task/node_modules/next/dist/server/next-server.js
- /var/task/___next_launcher.js
- /var/runtime/UserFunction.js
- /var/runtime/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
    at Function.Module._load (internal/modules/cjs/loader.js:745:27)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at Object.7425 (/var/task/.next/server/pages/learn/intro-online.js:144:18)
    at __webpack_require__ (/var/task/.next/server/webpack-runtime.js:25:42)
    at Object.8373 (/var/task/.next/server/pages/learn/intro-online.js:19:71)
    at __webpack_require__ (/var/task/.next/server/webpack-runtime.js:25:42)
    at __webpack_exec__ (/var/task/.next/server/pages/learn/intro-online.js:399:39)
    at /var/task/.next/server/pages/learn/intro-online.js:400:91 {
  code: 'MODULE_NOT_FOUND',
  • this happens if I refresh the page https://vulcan-next.vercel.app/learn/intro-online
  • this doesn't happen if I go to another page, and just add /learn in the URL => as far as I can tell Next is doing a kind of client page transition in this case. So basically only the server render fails.
  • no repro locally, it works ok

The package is correctly listed in the dependencies.

Sorry I'd like to help more but I don't fully understand what happens, and the error message is kinda vague here. That's maybe a different issue. Could there be some server-rendering happening? This is an MDX page using the row @next/mdx setup and MDXProvider from @mdx-js/react to pass some components.

@ijjk
Copy link
Member

ijjk commented Oct 25, 2021

@eric-burel you are adding getInitialProps in _app here so any page that doesn't use getStaticProps will be server rendered for every request. It seems the above shared URL is working correctly currently, did you need to change something related to mdx for this to work?

@ijjk
Copy link
Member

ijjk commented Oct 25, 2021

For anyone else who ran into the initial issue, please try upgrading to v11.1.3-canary.104 of Next.js and remove the experimental.outputFileTracing as it is no longer needed and confirm the issue is resolved fully 🙏

@eric-burel
Copy link
Contributor

@ijjk Nope it doesn't actually work, if you reload the page again, you'll get a 500 error. After updating to canary 104 I still get the server error Cannot find module '@mdx-js/react'. Very weird, I could not reproduce locally ever, it's really specific to the Vercel deployment.

Thanks for the tip though, I think getting rid of getInitialProps will fix it since it will prevent a rerender, I need to update next-i18next to the latest version. However it also means I'll lose next export, but I can live without it.
I'll do that and open a separate issue if I'm still stuck.

@styfle styfle modified the milestones: 11.1.x, 12.0.4 Nov 5, 2021
@timneutkens timneutkens added the Output (export/standalone) Related to the the output option in `next.config.js`. label Nov 16, 2021
@timneutkens timneutkens removed this from the 12.0.5 milestone Nov 17, 2021
@ijjk
Copy link
Member

ijjk commented Nov 17, 2021

I'm going to close this as this should be working properly in the latest version of Next.js v12.0.4, if you are still encountering issues with this in the latest version please open a fresh issue with reproduction.

@ijjk ijjk closed this as completed Nov 17, 2021
@balazsorban44
Copy link
Member

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.

@vercel vercel locked as resolved and limited conversation to collaborators Jan 27, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Output (export/standalone) Related to the the output option in `next.config.js`.
Projects
None yet
Development

No branches or pull requests