proposal: cmd/go: ignore directories containing .goignore in file scans

[tj]

@bradfitz mentioned go doc could special-case node_modules for exclusion.

If you have many client/server projects in $GOPATH, node_modules can quickly bloat and cause go doc to resolve slowly—this is especially noticeable in editors which use go doc to display inline documentation (see vscode issue).

I recursively removed my node_modules (15gb, crazy I know), here are the results before and after:

$ time go doc something

real    2m25.068s
user    0m17.772s
sys    1m45.039s

$ time go doc something

real    0m19.955s
user    0m2.499s
sys    0m9.343s

[agnivade]

/cc @robpike

[bradfitz]

@bradfitz mentioned go doc could special-case node_modules for exclusion.

That was done as a result of #16417, which you opened in 2016 and is still open.

[mvdan]

I presume this would extend the following bit of go help packages:

Directory and file names that begin with "." or "_" are ignored by the go tool, as are directories named "testdata".

Or is it fine for go doc to skip certain directories, while other commands like go build and go test wouldn't?

[tj]

Now that I think about it, gorename took ~5-10 minutes before as well, I'll have to try it now. It does seem a little hacky to special-case directories specific to some other language/framework, but I can't say I've ever wanted to operate against anything in node_modules either so it probably wouldn't be harmful

[agnivade]

To generalize this behavior, should we also ignore top-level directories under .gitignore ? That should take care of node_modules as well.

[rsc]

This can't be done in just cmd/doc. If we're going to do it, it should be globally, in cmd/go, go/build, etc. (And we can't start parsing .gitignore.)

It's unfortunate to have the definition of Go packages depending on or working around problems specific to other languages. On the other hand 15 GB is pretty big.

[rsc]

If we do this, we have to write:

Directory and file names that begin with "." or "_" are ignored by the go tool,
as are directories named "node_modules" or "testdata".

That's a bit strange, to call out Node explicitly like that. Are there any other giant directories to avoid too?

We could also make it user-configurable in some way, but then the meaning of things like 'go list foo/...' or 'go install foo/...' becomes configuration-specific, which we've tried hard to avoid.

[snikch]

I've been watching this as I've been affected heavily in the past with this via goimports and more recently go doc.

I agree that it is unfortunate to have to call out a specific language (and all derivatives of), however in the scope of web development as a practice today with node_modules being ubiquitous for front end development, it doesn't seem unreasonable.

[josharian]

I've also had this problem with Carthage. Looking at my .goimportsignore, I also see a few packages that are Go wrappers around large C projects that have lots of unrelated code and that do in-place builds, leading to large numbers of build artifacts.

[agnivade]

Some more discussion here - #16427. I agree that we should make it user-configurable. Hardcoding node_modules just seems odd to me.

At the expense of adding new knobs, I think having a .goignore file will help a lot (given that we already have .goimportsignore). And that file can serve as a common base which can be inspected by all other tools (go doc, gorename, goimports).

[ianlancetaylor]

I agree that adding .goignore would be better than special-casing node_modules.

[robpike]

Please don't create more dot files. If the tools need configuration, use a single file.

[ianlancetaylor]

The suggestion is basically to rename .goimportsignore to .goignore so there would be a single file.

It occurs to me that if we adopt #30411 then another possible approach would be to use an environment variable.

[rsc]

What about

go env -w GOIGNORE=

(as Ian suggested)? This would apply to go doc and so on but not to the extraction of modules from your Git repos. If the files go into your git repos, they will go into the module zips.

We've been circling various options here for a while. This seems like the least bad.

[josharian]

go env -w GOIGNORE=

Ignore lists like this tend to grow slowly over time. This is a nice fit for a text file, which is easy to edit and append to. Env vars are much less edit-friendly. Every time I type export PATH=“$PATH”: I experience a Rob-like twinge of exasperation. (Or anyway, I imagine it is Rob-like.)

[rsc]

Ian suggests that we might also consider a special file in the directory to be ignored. For example, we could say that when the go command walks into a directory and finds a file named .goignore or .goskipdir or .gorobots.txt, it turns around and walks back out immediately. So if you needed a node_modules inside your Go module working directory, you'd just put this magic file in there too.

Thoughts? Name bikesheds?

[jimmyfrasche]

That can be a bit annoying to work with for directories that sometimes get removed and then replaced with newer versions or for directories owned by a tool that then needs to be told to ignore the sentinel file.

It semi-makes sense to include the blacklist in go.mod since it would be listing directories that, though they happen to be contained in the module root, are not actually part of the module.

[agnivade]

Ian suggests that we might also consider a special file in the directory to be ignored.

A major problem that I see is that it will need .gitignore'd directories to be in the filesystem. For eg. node_modules are not checked in to a repo. But in this case, we would need an empty node_modules to be present, just for that special file to reside. Does not feel very elegant.

Moreover, this will create multiple files peppered throughout the repo compared to a single file. Also, now it becomes harder to know what directories are ignored at once.

What is the advantage of this solution over having a single dot file, since both are using a special magic file anyways (which I thought was the main point of contention) ?

[ianlancetaylor]

Typically the directories that should be ignored don't have to live in a Go repo at all. After all, by definition they don't contain any Go code. So the advantage of a file in a directory is that only people who create that directory will need to put the file in it. If we use a single dot file, it's hard to avoid putting that file into the Go repo, but now people who create a directory that doesn't contain any Go code have to maintain a local patch to the Go repo. (That argument does not apply to .gitignore since git supports putting ignore patterns outside of the repo.)

(I'm not saying we have to do it that way, I'm just answering the question about the advantage of this approach over a single dot file.)

[mvdan]

Android has a .nomedia empty file and I think it works nicely. I think it would be a cleaner solution than adding more config files or more env vars. If anyone wishes to mix Go with other tools/languages in a single repository which produce tons of files, I assume they can make sure to also add such a file as necessary.

[agnivade]

Thanks for the clarification Ian.

The only issue I see is that new users cloning a node repo have to traverse inside the node_modules folder and create the special file everytime, as opposed to the folder ignore path being already there in the single-file solution. But your point on maintaining local Go patches is valid as well. I am fine with both solutions.

[josharian]

So the advantage of a file in a directory is that only people who create that directory will need to put the file in it.

I'm concerned that this won't play well with external dependencies, either managed via git submodules or via some package manager. To get the go tool to ignore them, we'd have to add a Go-specific file to a completely non-Go repository; I can't imagine other projects being happy about that. The alternative is to add the file yourself after downloading the code, but dirtying the repo will also be an annoyance; it'll cause git submodule pain, any package manager that checks directory contents for security purposes may fail, etc.

[amery]

is there a better solution than the empty go.mod? npm will delete such file from node_modules/

[burdiyan]

I think node_modules is hardcoded and is always ignored by the Go tooling (please correct me if I'm wrong).

[amery]

I think node_modules is hardcoded and is always ignored by the Go tooling (please correct me if I'm wrong).

Hi @burdiyan,
go get ./... complains about a symlink within node_modules unless I touch node_modules/go.mod every time after using npm to install something

go1.17.3 linux/amd64

[Jamesits]

As #45713 has landed, does anybody know how to ignore a go.work file in a random subdirectory (it is part of the documentation) from stopping the whole project from building? (Cannot disable the whole workspace feature as there are genuine go.work files working elsewhere in the project)

[jabalsad]

I think node_modules is hardcoded and is always ignored by the Go tooling (please correct me if I'm wrong).

When I run go mod tidy I still get dependencies added to go.mod that include directories in node_modules.

Putting empty go.mod everywhere is not something that can be replicated across different members of the team.

I echo this sentiment: the node_modules directory is created and managed by npm. We cannot reliably have a go.mod file in there across the entire team.

We cannot use the . or _ method either, since the node_modules directory name is not configurable.

What would be ideal is a way of telling go mod tidy to ignore the node_modules directory?

[paralin]

I'm also running into this, is there a way to get it to ignore node_modules?

[amery]

@paralin it's pretty obvious by now they don't care at all. Their so-called solution is BS.
I'm handling it at the moment including the following before every call to go on my Makefiles.

find * -type d -name "node_modules" -exec touch {}/go.mod \;

I also get dist related problems, but I can't safely inject things there. fortunately, in my case they don't have .go files.

[paralin]

Ended up resolving it by ensuring that "go.mod" and "go.sum" were published with the npm package I was importing. However, it's a bit concerning to me that potentially some other npm module added could have some Go code in it which ends up causing the Go compiler to fetch random packages & modify my project's go.mod.

[paralin]

Note: This bug was effectively moved to #42965

@amery

[mvdan]

Discussion and ideas continue at #42965, as mentioned by @paralin. I'm going to lock this thread for now, because this effectively duplicate thread is causing parallel discussions and leading to some confusion. Please direct any comments or ideas to 42965, which is an open proposal.