Support locating non-page js/jsx files with page files

[zenflow]

Original title: Support having non-page js/jsx files in pages dir

This is an often requested feature. See issues #3508 #1914 #1689 #1545 #988 and there are likely more. edit: Also #3183.

This issue was fixed in PR #3195 but somewhere along the way the pagesGlobPattern config became broken, and was cleaned out in PR #3578.

This feature would allow us to place files that support specific pages, with the pages that they support. These "support" files could be tests, components, utilities, and more. This would improve the application's file organization, especially for large projects.

[zenflow]

For clarification

This is the folder structure we want to avoid, especially if it is to scale to have more than 3 pages in a hierarchy more than 1 deep:

• components/
  • chat/
  • timeline/
  • (... index components)
• helpers/
  • chat/
  • timeline/
  • (... index helpers)
• pages/
  • chat.js
  • timeline.js
  • index.js
• tests/
  • chat/
  • timeline/
    (... index tests)

This would be more maintainable at scale, since we don't duplicate the (root->(chat, timeline)) structure for each file type:

• pages/
  • __tests__/
  • lib/
    • components/
    • helpers/
  • chat/
    • __tests__/
    • lib/
      • components/
      • helpers/
    • index.js
  • timeline/
    • __tests__/
    • lib/
      • components/
      • helpers/
    • index.js
  • index.js

[zenflow]

Don't see any harm in adding this 👍 I do want to keep this as 'internal' (undocumented) cause it's not a practice we endorse. @timneutkens in PR #3195

@timneutkens Why would you not endorse this practice?

[zenflow]

I see two solutions:

1. Re-implement pagesGlobPattern config, but this time:
   • remove the 'pages/' bit from the default, and just prefix it before using, since this cannot be changed (pages must be in the pages/ dir, otherwise this solution becomes more complicated)
   • add tests to make sure there is not a regression
2. Take a convention-over-configuration approach, and make all lib/, __tests__/, and __mocks__/ folders ignored. __test__/ and __mock__/ (singular) too

[zenflow]

@timneutkens @sergiodxa Would you be open to a PR for either of the above solutions? Or perhaps a different solution?

[moaxaca]

This really is a deal breaker for people designing large more enterprise applications.

Some notable solutions that allow similar functionality:
https://www.gatsbyjs.org/ Static Generation
https://github.com/jaredpalmer/after.js SSR - React Router

If I had the community support, I would consider making my fork something more modularized & flexible then pushing it to npm something like flext.

[timneutkens]

You could actually try to do this:

1. Make a directory, eg components
2. Put all the code that you want to put into pages into components
3. Use symlinks to link the actual pages to the pages directory

This gives more granular control over routes that are accessible to the outside world. And is safer and clearer than providing exclude/includes.

but somewhere along the way the pagesGlobPattern config became broken

To be clear, pagesGlobPattern was always broken. It only accounted for next build scenario. And I removed it because it was obsolete because of that.

This really is a deal breaker for people designing large more enterprise applications.

We're continuously adding features/improvements to face bigger scale, as our own app grows (currently over a thousand components), and as the community gives feedback.

[moaxaca]

@timneutkens That's clever, I will give it a shot. Hopefully, the windows guys on my team don't hate me for committing a symlink if it works!

This being said, I would prefer to see a roadmap or a consensus from the maintainers on pivotal features.

[rauchg]

@moaxaca the only reason we're hesitant to support the glob is precisely because we are concerned that at scale it could open critical security concerns.

We basically would be encouraging people to put into pages/ things that are not meant to be routable, that they have to carefully exclude (as opposed to opt into and include explicitly).

For example, a side effect of not excluding __tests__ in the example above would be to execute code that results in denial of service, because it's not meant to be run in the context of a HTTP request.

I don't consider it "enterprise ready" to go back on what's otherwise a very clear and secure pattern, but we're always listening to the community for different approaches, in case one is suggested here.

In fact, it suffices to do some light research into the vast literature of PHP vulnerabilities to find many that fall under "I exposed a .php file to the end user that was not meant to be world-accessible".

[zenflow]

@timneutkens I am a Windows guy (don't judge) so the symlink approach will not work for me, and besides, there should be a portable solution. But thank you very much for the suggestion!

[zenflow]

@rauchg

but we're always listening to the community for different approaches, in case one is suggested here.

What about the second solution that I proposed above?

Take a convention-over-configuration approach, and make all lib/, __tests__/, and __mocks__/ folders ignored. __test__/ and __mock__/ (singular) too

It seems that this would be less error-prone; users would not have to carefully exclude the files (i.e. write a glob pattern), they just have to make sure that they don't misspell lib or __tests__ or __test__.

This would also be a very clear pattern I think, no?

[zenflow]

If this is still too error-prone, maybe we could still work with this idea, and extend it with something like "print a warning if a dir like lbi is detected", or something else

[gihrig]

I worked around this issue as follows:

Think of ./pages as "routes", move existing pages content to another directory.

Create ./containers directory
Move contents of ./pages into ./containers
Create 'link files' in pages. e.g. ./pages/index.js:

import Home from '../containers/Home'

export default Home

This allows for any complex folder structure in ./pages for matching complex url's and does not risk unwanted exposure of content that should not be public.

[zenflow]

Beautiful solution @gihrig! 🎉 This solves the problem!

Though I still would prefer not to have to write modules like:

import Foo from '../../../../../path/to/deeply/nested/container/Foo'

export default Foo

and if it was approved I would still make a PR implementing what I proposed above.

[rauchg]

By the way, I also think we do need a solution to not having to write so many ../../, but I think that can be done with babel? It's nice to be able to register a "root", and be able to use it anywhere like so:

import "containers/…"

That's also generally useful throughout the codebase, so maybe @gihrig's solution has to be extended with some babel plugin in case you have too many nested paths.

I believe @hanford is using such a plugin successfully already.

tl;DR: @gihrig's solution is sufficient. It can be combined with the "root" import solution which is orthogonal and generally useful beyond pages/

[hanford]

@rauchg @zenflow - We're using babel-plugin-module-resolver at Eaze, and I'm a huge fan

[zenflow]

In the same way that writing so many ../../ is irritating and less-than-elegant, so is this

// pages/some/deeply/nested/Foo.js
import Foo from "containers/some/deeply/nested/Foo"
export default Foo

note the repetition of "some/deeply/nested/Foo"

Also

1. This type of Babel plugin breaks code completion in most (if not all) IDEs (as well as other code intelligence features like ctrl+clicking the import path to open the imported file)
2. The path is ambiguous. It could also refer to <project root>/node_modules/containers/some/deeply/nested/Foo.js. And this is what a casual observer would probably assume it's referring to.

@rauchg Why not make some particular folder names ignored, as I proposed? Even if it is only a single folder name (like lib) that would be enough since you could place everything (tests, components, utilities, etc.) there.

[zenflow]

@hanford Can you confirm # 1 in my above comment?

edit: I see here that with configuration the code completion in the popular IDEs can still work with babel-plugin-module-resolver https://github.com/tleunen/babel-plugin-module-resolver#editors-autocompletion

[gihrig]

Though I still would prefer not to have to write modules like:

import Foo from '../../../../../path/to/deeply/nested/container/Foo'

export default Foo

Add module-resolver to .babelrc and it becomes

import Foo from 'containers/Foo'

export default Foo

.babelrc:

...
  "plugins": [
...
	[
	  "module-resolver",
	  {
		"root": ["./"],
		"alias": {
		  "components": "./src/components",
		  "containers": "./src/containers",
		  "graphql": "./src/graphql",
		  "lib": "./src/lib",
		  "pages": "./src/pages",
		  "src": "./src"
		}
	  }
	]
  ]

I keep my 'source' under ./src, make adjustments as appropriate to your configuration.

@hanford beat me by a few, but the details may still be helpful... 😄

[gihrig]

@zenflow I'm working with VS Code:

Re: 1. I have not noticed any problems. I have a pretty strict ESLint setup an it has always told me when I have an unresolved module.

Re: 2. I avoid module nesting more that two level below components or containers. I usually match nesting between containers and pages.

And yeah, it would be super awesome if Next could provide a way to avoid the need for the boilerplate in pages 😄

[zenflow]

Sorry @gihrig could you clarify your comment about # 2? I'm not sure I understand what you mean, or if I do I don't see how it relates to that issue.

[gihrig]

@zenflow Yeah, I was commenting on problems with deeply nested folders (I try to avoid that).

On closer reading, I see you were referring to an 'absolute' import being expected to be under node_modules and this possibly confusing those unfamiliar with the setup.

That's a valid point, but since import Foo from 'folder/Foo' under node_modules is less common (though not unheard of) I don't see it as a big issue. Once learned, I don't think devs will have an issue with it, but there is a learning curve, to be sure.

[moaxaca]

@rauchg that makes sense and I completely understand how this could be a loaded gun handed to the masses. I have a ton of empathy for library maintainers that protect their users from themselves.

But, (and there's always a butt) the current implementation is very similar to page controller pattern (Talk about some ancient PHP!), and assuming stick with the file system delegating routes, issues like this will always arise.

This being said, A NextServer Router that allows route definitions could be optimal.

I currently use a Koa instance to handle all this dynamic routing, and while its excellent on the initial hit, it quickly falls apart on the frontend router. I relish the idea of having a router could resolve against my koa definitions while allowing me to structure my app in a modular fashion.

Also just to mention, we use Lerna and would love to package pieces our interface in different bundles. A router that could be composed would be amazing. We currently use a similar pattern to the one mentioned above where we import container then just manually link them to root pages.

Also if you guys ever need help. I would be happy to contribute some of these features.

[zenflow]

@gihrig Yeah, it's just a minor issue. # 1 is a minor issue as well, since we will not be working in these "link files" often, but I think it's still an issue, since people use a variety of IDE's, each with it's own way of supporting code completion with babel-plugin-module-resolver.

[zenflow]

@moaxaca Let's try to keep this issue concentrated on "Support locating non-page js/jsx files with page files".. I might suggest opening a separate issue for "file-system based routing is not flexible". (btw I think the resolution of issue #257 might help with your modularization issue since you will be able to run multiple apps with common dependencies on the same domain)

[moaxaca]

@zenflow Ok, but I am going to leave my comment. I genuinely feel the root of this issue is in the routing mechanics.

Also ty for the PR I didn't consider multiple apps 👍.

[zenflow]

The only downside I see with my proposal is that you can no longer have routes containing /lib/, /__tests__/, etc. (unless you are using one of the router extensions), but I think that is a reasonable sacrifice considering how often those strings are used in routes.

@moaxaca @gihrig @rauchg Do you see any other downsides or problems?

[moaxaca]

@zenflow I agree with @timneutkens I feel like this may create a false sense of security and be prone to exposing too much accidentally.

If I were to take a convention over configuration approach, I would prefer a solution where I explicitly define my pages through a .pages.js|jsx|ts extension. They share a common naming convention and don't require blacklisted words like __tests__ __mocks__ lib.

[arunoda]

@zenflow I don't think we don't need any huge customization for what you wanted.
Our pages directory is something specially. It's the entry point of for a page.
You don't need to put your all the files inside this.

Try using the following approach:

• pages

  • home.js
  • app.js
  • index.js

• modules

  • home
    • components
    • containers
    • helpers
    • actions
  • app
    • components
    • containers
    • helpers
    • actions

[zenflow]

@moaxaca Do you think people would be prone to misspelling lib? Yes/no?... Like I said: "Even if it is only a single folder name (like lib) that would be enough since you could place everything (tests, components, utilities, etc.) there."

btw it was @rauchg that said that and I believe (I could be wrong) he was talking only about the pagesGlobPattern approach, hence "For example, a side effect of not excluding __tests__ in the example above" which does not apply to the convention over configuration (2nd) approach.

I would personally be OK with a .page.js extension, but I don't think (edit) I'm not sure it's a good idea because of the reasons here. Particularly, it would be a breaking change for every single Next.js app out there.

[ziserman]

Just want to add to solution with ./containers directory.

Instead of ./containers folder you can use ./src/pages directory to build any structure (components, tests etc.) and import pages to root ./pages for routing since:

1. we have another usage for ./containers folder time to time (I mean React Container Component)
2. src/pages will be ignored if pages is present in the root directory (Next.js docs).

[blujedis]

@rauchg I don't see this request (it's not an issue) going away any time soon. While I largely agree as to the security related issues, I would argue that if Next is smart enough to look for "pages" in root or "src/pages" and it's smart enough to discriminate or ignore other files/dirs it stands to reason this could be solved with a convention, even if a bit opinionated. While there are workarounds, I'll be honest this really does drive me bonkers at times ;)

[johot]

Thank you for your suggestion @gihrig

My current go to solution is very simple and becomes very nice to work with, this is ofc a matter of personal taste but instead of creating a folder called features, modules, container etc I simply use the following sample structure:

// files in pages just render a component from the `src` folder and are basically empty/very think except for data fetching (which can also be moved into `src` folder if you want to)
pages\...
src\common\...
src\start\...
src\profile\...
src\contact\...

It might feel odd to see the pages folder outside the src directory but it really makes it easier to work with as you don't mix them up as easily. I almost see the pages directory as a configuration folder and the real meat of the application lives inside the src folder. By doing this I could also avoid coming up with a good name for the "features" root folder 😄

[granmoe]

In addition to the pageExtensions idea (which would be awesome), you could allow optionally setting a "page component prefix" in next.config.js that would cause next to ignore any files with this prefix. Say you set it to "_", then you could have :

pages/
  about/
    index.tsx
    _child-component.tsx
    _complex-child-component/
      this-is-already-ignored-due-to-the-folder-name.tsx
    whoa-here-is-a-child-route.tsx

Kind of like the ESLint no-unused-vars prefixIgnorePattern option.

[OzzieOrca]

Seems like using Custom Page Extensions is a good solution. See:
#8454 (comment)
#8617 (comment)
#3728 (comment)

[mihaisavezi]

I worked around this issue as follows:

Think of ./pages as "routes", move existing pages content to another directory.

Create ./containers directory
Move contents of ./pages into ./containers
Create 'link files' in pages. e.g. ./pages/index.js:

import Home from '../containers/Home'

export default Home

This allows for any complex folder structure in ./pages for matching complex url's and does not risk unwanted exposure of content that should not be public.

Does the global css load for you if using this approach?

my setup is:

// pages/items/new
import ModulesNewItem from "@modules/items/new";

export default function newItem({ ...props }) {
  return <ModulesNewItem {...props} />;
}

OR

export default ModuleNewItem

// src/modules/items/new
export default function ModulesNewItem() {return(<>...content</>)}

[protometa]

Citing possible user error as a reason not to implement filters on page files is not well founded. I guarantee there are nextjs sites out there that have test and supporting files in pages/ exposed because the devs didn't realize that __tests__, .tests.js, .stories.js, etc. aren't automatically filtered out. Allowing an override filter actually provides an additional security check.

[sethbattin]

Credit to stackoverflow, I think this is the clearest option to maintain the __tests__ adjacency convention that is typical in react apps with jest et al. No new dependencies (you already have webpack), no special-case config, no filename or symlink shenanigans. Just write your test files for pages in the same place as for everything else.

change webpack settings direct in next.config.js

module.exports = {
  webpack: (config, { webpack }) => {
    config.plugins.push(new webpack.IgnorePlugin(/\/__tests__\//))
    return config
  }
}

We're going for principle of least surprise, right? Make the pages folder permit the normal test code convention. 🤷

This doesn't solve the general "but i really want to" reasoning for putting other components in the pages directory. I guess do this if you want actively break conventions instead of following them.

---

Edit: actual credit to this undescribed code snippet from the config docs. 😂

[coopbri]

The solution that @sethbattin suggested is great, it worked for me using webpack 4. However, I am using webpack 5 in my Next.js projects. To adapt the solution to work with webpack 5, I tried the following configuration (Next.js 10.2.3):

module.exports = {
  future: {
    webpack5: true,
  },
  webpack: (config, { webpack }) => {
    config.plugins.push(
      new webpack.IgnorePlugin({ resourceRegExp: /\/__test__\// })
    );
    return config;
  },
};

See https://webpack.js.org/plugins/ignore-plugin for details on this change. Essentially, the webpack team changed the implicit regex parameters to now be passed within an object.

Unfortunately, builds error out with HookWebpackError: Cannot read property 'slice' of undefined. For now, I switched back to webpack 4, and will update this comment if I find a solution.

[CopyJosh]

I started collocating tests to the /pages directory and did not give it a second thought that Next could pick these up as routes, until I was recently logging the generated routes and noticed this. Thankfully @sethbattin's solution works!

Just feedback to the other ideas, the page extension solution looks plausible for new projects, and I would do this in the future, but changing the extensions on hundreds of pages on an existing project is not reasonable. Not sure why an include/exclude or other pattern matcher options are not supported.

[qafoori]

@austincondiff
maybe we can use a config like this, and rename index.tsx to index.page.tsx to ignore files which is not a page, it work at Next.js version ^9.0.4

module.exports = {
  pageExtensions: ["page.tsx"],
}

It's working well in "next": "11.0.1" and its very helpful

[ScottyMJacobson]

@qafoori - thank you for the suggestion! I just tried it and all my pages gave me 404s, though 🙁. If anyone has a guess as to why, I'd love to hear it! I'd much prefer to leave my components/tests/pages colocated, but for now I've gone for the "two folders" solution with one of those folders called src/

[qafoori]

@qafoori - thank you for the suggestion! I just tried it and all my pages gave me 404s, though 🙁. If anyone has a guess as to why, I'd love to hear it! I'd much prefer to leave my components/tests/pages colocated, but for now I've gone for the "two folders" solution with one of those folders called src/

The two folder is not the best practice at all.
well, can you explain how did you config your next? (next.config.js)
what schema of folders are you looking for?

[OzzieOrca]

I solved this for our project by using Custom Page Extensions.

next.config.js

module.exports = {
  // Force .page prefix on page files (ex. index.page.tsx) so generated files can be included in /pages directory without Next.js throwing build errors
  pageExtensions: ['page.tsx', 'page.ts', 'page.jsx', 'page.js'],
}

It seems like this has been mentioned a few times but this issue is still filling up my inbox with new users trying to figure it out 😄

My initial discovery of how to do this in a comment above: #3728 (comment)
My PR to improve the docs for this: #22740
Relevant StackOverflow question: https://stackoverflow.com/questions/57838908/build-error-occurred-referenceerror-describe-is-not-defined-during-now-sh-dep/68718957#68718957

If this works for you, maybe you can upvote the docs PR or suggest things that would make this solution more discoverable.

[ScottyMJacobson]

I solved this for our project by using Custom Page Extensions.

next.config.js

module.exports = {
  // Force .page prefix on page files (ex. index.page.tsx) so generated files can be included in /pages directory without Next.js throwing build errors
  pageExtensions: ['page.tsx', 'page.ts', 'page.jsx', 'page.js'],
}

It seems like this has been mentioned a few times but this issue is still filling up my inbox with new users trying to figure it out 😄

My initial discovery of how to do this in a comment above: #3728 (comment)
My PR to improve the docs for this: #22740
Relevant StackOverflow question: https://stackoverflow.com/questions/57838908/build-error-occurred-referenceerror-describe-is-not-defined-during-now-sh-dep/68718957#68718957

If this works for you, maybe you can upvote the docs PR or suggest things that would make this solution more discoverable.

It worked! Sorry I don't/didn't have more details about what went wrong when I tried a few days ago - i was at the end of lots and lots of reading the relevant GitHub issues about it, and was just trying to get something to work as quickly as possible.

Thanks Scotty / @OzzieOrca - and thanks again @qafoori

Also, for what it's worth, I actually think this solution is better (explicit opt-in) than an explicit opt-out solution that others have advocated for. Much less room for user error and it doesn't really prevent you from any dev workflows. Hey, maybe even the default should be *.page.[jt]sx? 😃

-Scotty

[jldillman]

I solved this for our project by using Custom Page Extensions.

next.config.js

module.exports = {
  // Force .page prefix on page files (ex. index.page.tsx) so generated files can be included in /pages directory without Next.js throwing build errors
  pageExtensions: ['page.tsx', 'page.ts', 'page.jsx', 'page.js'],
}

It seems like this has been mentioned a few times but this issue is still filling up my inbox with new users trying to figure it out 😄

My initial discovery of how to do this in a comment above: #3728 (comment)
My PR to improve the docs for this: #22740
Relevant StackOverflow question: https://stackoverflow.com/questions/57838908/build-error-occurred-referenceerror-describe-is-not-defined-during-now-sh-dep/68718957#68718957

If this works for you, maybe you can upvote the docs PR or suggest things that would make this solution more discoverable.

This is a great suggestion, but it still introduces a new rule that others will have to follow if you have this enabled in your project.

I still vote for the addition of an ignore pattern option in next.config.js.

[jldillman]

The solution that @sethbattin suggested is great, it worked for me using webpack 4. However, I am using webpack 5 in my Next.js projects. To adapt the solution to work with webpack 5, I tried the following configuration (Next.js 10.2.3):

module.exports = {
  future: {
    webpack5: true,
  },
  webpack: (config, { webpack }) => {
    config.plugins.push(
      new webpack.IgnorePlugin({ resourceRegExp: /\/__test__\// })
    );
    return config;
  },
};

See https://webpack.js.org/plugins/ignore-plugin for details on this change. Essentially, the webpack team changed the implicit regex parameters to now be passed within an object.

Unfortunately, builds error out with HookWebpackError: Cannot read property 'slice' of undefined. For now, I switched back to webpack 4, and will update this comment if I find a solution.

I also ran into this same error trying to use IgnorePlugin with Next 11.0.0.

[Jeffrey-FLS]

Is there a way to pass styles to src/pages ? It doesn't work with TailwindCSS v3 if pages is within src on nextjs v12. But works if placed within root

[balazsorban44]

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.