'yarn pack' should not process .gitignore if .npmignore is present

[donpark]

Do you want to request a feature or report a bug?

Report a bug.

What is the current behavior?

yarn pack ignores files specified in .gitignore while npm pack does not.

When .js files are generated from another language or flavor of javascript, generated files are often not checked-in and thus *.js or **/*.js entry is added to .gitignore. These generated files should be included in the package at runtime.

If the current behavior is a bug, please provide the steps to reproduce.

1. create index.js file at project root.
2. create .gitignore file at project root with entry *.js.
3. run yarn pack.
4. decompress package .tgz file.
5. index.js file will not be in the decompressed package, only package.json and node_modules.

What is the expected behavior?

index.js should be there.

Please mention your node.js, yarn and operating system version.

Node.js v6.7
Yarn v0.15.1
macOS Sierra

[tony19]

Seems related to #685

[donpark]

Appears so except .npmignore is already in IGNORE_FILENAMES list used to create ignores so its bug is likely regex related.

This issue needs .gitignore removed from the IGNORE_FILENAMES list to be fixed.

But then it's not clear if npm honoring .gitignore is accidental or not. I don't believe so but I'm long past expecting common sense from npm.

[Jessidhia]

npm will honor .gitignore as long as there is no .npmignore. https://docs.npmjs.com/misc/developers#keeping-files-out-of-your-package

[donpark]

oy. I wasn't expecting that. Likely npm started ignoring files in .gitignore then added .npmignore to resolve complications.

I'll retitle this issue again.

[ghost]

(Typing this out here as future reference or for anyone willing to take this up)

What the problem here seems to be is that pack goes ahead and fetches (src) all the files during a walk (src). What this basically does is it reads the directory and it pushes the files in a list, if the file is a directory it does the walk again recursively. Note that the result is not a tree, files will appear in an alphabetical order.

Afterwards pack goes ahead and iterates over all these files (src) and will add any line from a file that is defined in IGNORE_FILENAMES (yarn/npm/git-ignore).

Some (basic) possible solutions for this are:

• Replace the iteration logic with logic that will map IGNORE_FILENAMES per directory, after this has been done, resolve the highest priority one (yarn > npm > git, I would guess). As a final step you can now read these files from your map and append them in the filters list (src). (Easier, but quite some steps)
• Extend utils/fs with a walk-like function that returns a tree instead of list. Traverse the nodes, get the highest priority IGNORE_FILENAME per directory and append them in the filter list.

Note that you need to keep the iteration and check all directories, as deeper directory ignore files overrule higher ones.

[donpark]

I'd just do the following:

1. maintain a queue of directories to process where directories are discovered through breadth-first search and each directory entry is linked to its parent directory's queue entry.

2. start processing with module root as first queue entry.

3. for each unprocessed directory entry,

   3a. fetch its immediate children.
   3b. build a list of filters from .yarnignore or .npmignore or .gitignore (only one xxxignore, selected based on the listed check priority, should be used).
   3c. throw each child through the gauntlet of filters (check parent's filter, then its parent's filter, so on).
   3d. for each file that makes it through, add to pack list of files.
   3e. for each directory that makes it through, add to queue of directories to process.

[ghost]

Yes @donpark! A bit the way I had in mind with the second one, but I quickly listed thought-flows around the problem, trying to help/kickstart possible future readers.

Yours absolutely makes more sense, and would possible solve any complexities with higher/deeper listed ignore files as-well. Thanks for chipping in!

[caesarsol]

It's quite surprising, are you saying that an empty .npmignore does not prevent yarn to ignore all files in the .gitignore?

That is what I expect since it's what npm does...

[jamesamcl]

I had the problem of yarn ignoring my output folder when publishing because it was in .gitignore, and creating an empty .npmignore fixed it.

yarn 1.12.3