cmd/go: should default to use mod=vendor flag if the vendor folder exists

[JeremyLoy]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"

What did you do?

• Given a fresh install of go, there are no files downloaded in $GOPATH/pkg/mod.
• I cloned a repo that uses the new Go 1.11 modules with vendored dependencies
• I executed go run main.go to run the application.

What did you expect to see?

• the application ran without downloading external dependencies.

What did you see instead?

• The go tool downloaded all dependencies to $GOPATH/pkg/mod, despite the fact that the dependencies were vendored

I know this can easily be fixed by instead go run -mod=vendor main.go, but because the go tool is now being advertised as "context aware", I think this would be a good QoL change.

[thepudds]

@JeremyLoy note that while the default is to ignore the vendor directory, you can relatively easily change that behavior with an environmental setting GOFLAGS=-mod=vendor. You could use that in CI, or could set it in a .bashrc or similar for day-to-day use if you want.

This is a new environmental variable in Go 1.11, where one of the justifications for introducing it was specifically to make it easier to opt in to vendoring given some people would want that regularly (e.g., see #26585 or https://go-review.googlesource.com/c/go/+/126656).

From the doc:

GOFLAGS
	A space-separated list of -flag=value settings to apply
	to go commands by default, when the given flag is known by
	the current command. Flags listed on the command-line
	are applied after this list and therefore override it.

One other quick note is that those downloads end up in a shared cached (so even if you don't opt-in to vendoring, it cuts down on how many times you would end up downloading dependencies).

[rasky]

I strongly agree with this proposal. If a project committed a vendor folder, this is a strong indicator that vendoring should be used by default. I can’t think of a workflow in which a developer would commit a vendor folder but prefer other developers to ignore it by default.

In other words, if a project decides to use vendoring, then the whole go tool should default to use vendor first, and download/cache only if not there.

[FiloSottile]

By the way, this is part of what GOFLAGS was added for: https://go-review.googlesource.com/c/go/+/126656

[rasky]

GOFLAGS isn’t the right solution here. This is not a per-user / per-computer decision like “add debug flags”. This is a per-project decision that should affect all users building it. Currently, if a project uses vendoring, there is no way to let all builders use vendoring by default.

[JeremyLoy]

WhileGOFLAGS is a decent immediate workaround, I do want to stress the importance of the vendor folder as a part of this issue.

To put it bluntly, Github is not an artifact repository. At any given time, a repository owner can either

1. delete their account
2. delete a repository
3. force push deletions to master
4. remove a tag
5. change a tag

To avoid any of the above damaging your project, vendoring is a necessity, and should be treated as a first class citizen in the go toolchain.

There seems to be a rush to remove vendor from the go toolchain. It is still referenced as an "experiment" and there was a bit of swirl in even getting modules to support vendoring in its first release. Until Project Athens or equivalent becomes mainstream, vendoring should be fully supported (but never required).

EDIT: coincidentally, I found out the official Heroku build pack for Go uses this very workaround. It seems like this really should be the default case.

[myitcv]

@rasky

Currently, if a project uses vendoring, there is no way to let all builders use vendoring by default.

But we're talking about working in module mode here, correct? Or by "currently" do you mean GOPATH-mode? Because vendoring as a "first class citizen" continues to work in GOPATH mode, it's just that in module mode it is not the default.

@JeremyLoy

To avoid any of the above damaging your project, vendoring is a necessity, and should be treated as a first class citizen in the go toolchain.

There was a similar discussion over at https://twitter.com/_myitcv/status/1033824806930534400. Vendoring (as it was and still is in GOPATH mode) is one way of dealing with this problem. As I suggested, having a separate repo as a module download cache is another, which actually have a number of derivative benefits.

As you say, Athens (or indeed anything else that talks the published module protocol) are also solutions, solutions that in some cases try to solve even bigger problems.

There seems to be a rush to remove vendor from the go toolchain

I don't think this is quite accurate. In GOPATH mode (the default), vendoring works exactly as it has done up until now. In module mode (the opt in) the behaviour changes.

So for the record, in module mode I am against what is being proposed here, not least because of what I wrote above, but also because it confuses/complicates the rule that "vendor directories are totally ignored in module mode" which is particularly important when it to dependencies.

[rasky]

Because vendoring as a "first class citizen" continues to work in GOPATH mode, it's just that in module mode it is not the default.

Yes, I'm speaking about module mode. The point is that "not being the default" for projects that uses vendoring (that is, projects in which developers went through the process of calling go mod vendor and commit the vendor directory) is the wrong default. If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used. People that don't use vendoring don't need to have vendoring explicitly disabled; they simply won't have a vendor directory in their project. By switching the vendoring support off in module mode, go 1.11 is making using vendoring more difficult for no reasons I can understand, without making things simpler for those who don't need or want vendoring.

There was a similar discussion over at https://twitter.com/_myitcv/status/1033824806930534400. Vendoring (as it was and still is in GOPATH mode) is one way of dealing with this problem. As I suggested, having a separate repo as a module download cache is another, which actually have a number of derivative benefits.

Thanks, but that is largely suboptimal. It requires a custom installation step to be repeated for each user, and also handle and commit binary files into the SCM which are harder to handle (especially with git) and review.

As you say, Athens (or indeed anything else that talks the published module protocol) are also solutions, solutions that in some cases try to solve even bigger problems.

I fail to see how Athens solves the needs of people using vendoring. AFAICT Athens helps with the issue that people can remove/sell GitHub repositories to other parties, and this can break builds and/or even cause security problems. This is not what vendoring is after. Vendoring is required in workflows that need 100% of the source code for a project committed under a single SCM, so that it can be fully audited, versioned, and built completely offline, without any dependency on any third-party service or installation. I deliver code to enterprises that want the code to be complete, not depend on third party services, and fully versioned with their existing code tracking tools. Vendoring solves this problem; Athens doesn't seem to.

So for the record, in module mode I am against what is being proposed here, not least because of what I wrote above, but also because it confuses/complicates the rule that "vendor directories are totally ignored in module mode" which is particularly important when it to dependencies.

You seem against adding a feature for people that need it for the sake of orthogonality. If you don't want or need vendoring, there's nothing in vendoring support that creates problems to you. Just use modules, your download cache, Athens or whatever you prefer. You will never have a vendor directory in your repos and nothing will break or be suboptimal for you. But if I need vendoring support and commit a vendor directory, why should vendoring support be disabled by default for me, causing more difficult builds for all involved developers?

[bcmills]

If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used.

That's not at all “obvious” to me. In fact, the current structure (in 1.11 module mode) is exactly the opposite: if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.

Giving their users an option is not at all the same thing as providing an explicit recommendation.

Moreover, vendor directories can only reasonably work at the topmost level. If a developer makes edits within the vendor folder (intentionally or otherwise), they would effectively perform an implicit replace operation without surfacing that fact in the rest of the UI (such as in the output of go list).

[rasky]

If a developers explicitly goes through the steps required to commit vendored dependencies, it is pretty obvious that they want those vendored dependencies to be used.

That's not at all “obvious” to me. In fact, the current structure (in 1.11 module mode) is exactly the opposite: if a developer goes through the steps to commit vendored dependencies in module mode, it could just as easily be because they want to give their users the option to build using those vendored dependencies as an alternative to using modules.

I'm not sure that we can deduce anything from what Go 1.11. It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures. What we ended up with is basically non working for many vendoring use cases, hence this bug (and others).

Rather than discussing hypothesis made by people that don't vendor, I'd rather somebody that does use vendoring in production shows up and say that they absolutely don't want vendored modules to be used by default unless explicitly opted in (like Go 1.11 does), and explain us why with a small experience report. I for sure know this doesn't work for my use case as already explained in mine, and others have already chimed in.

Moreover, vendor directories can only reasonably work at the topmost level. If a developer makes edits within the vendor folder (intentionally or otherwise), they would effectively perform an implicit replace operation without surfacing that fact in the rest of the UI (such as in the output of go list).

That's perfectly fine for me at least. In fact, in my use case, vendoring support could go as far as completely subsuming the module download cache and/or go.sum. That is, it would be great that once you opt-in to "vendoring mode", all new modules are automatically expanded into the vendor directory, used by default, removed when they're not used, etc.

[bcmills]

I'd rather somebody that does use vendoring in production shows up and say that they absolutely don't want vendored modules to be used by default unless explicitly opted in (like Go 1.11 does), and explain us why with a small experience report.

That's kind of my point, though: you're requesting feedback from one particular subset (package maintainers that choose to use vendoring) and ignoring another (package consumers, who have no choice in the matter in GOPATH mode, but do now have a choice in module mode).

We need experience reports from everyone — including package maintainers and consumers, and including both open-source and enterprise codebases.

[bcmills]

@JeremyLoy, could you give some more information about your use-case?

The go tool downloaded all dependencies to $GOPATH/pkg/mod, despite the fact that the dependencies were vendored

• Did you run go mod download after cloning the repository? Why or why not?
• What is the impact of downloading the dependencies?
  • Do you intentionally want to bypass go.sum verification, or do you have in mind that the vendored dependencies remain unmodified?

Until [module proxies become] mainstream, vendoring should be fully supported (but never required).

Vendoring is still supported in Go 1.11, both in GOPATH mode and with -mod=vendor.
I am not aware of any proposal to remove it in 1.12, and would be surprised to see one at this point.

[bcmills]

@rasky

It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures.

Please remember that among our gopher values is to “be charitable”.

The approach to vendoring in Go 1.11 does seem to have evolved with the implementation, but that doesn't mean it was “retrofitted”. A flag very similar to -mod=vendor was floated in #19234, and the ability to disable vendoring was discussed in #16562; as far as I am aware, both of those significantly predate the current module design.

[JeremyLoy]

@bcmills I'm currently using Go for enterprise application development, so I can give an experience report for that. You'll have to find someone else for the package maintainer report.

We currently vendor our dependencies using dep. We looked into converting to go modules, but found that the differences were too great to warrant a change right now, especially as modules are still in development.

We vendor for a variety of reasons, many of which were already stated here, but I will reiterate them for clarity.

• a single deliverable - the client is able to take the codebase and build the application from source without internet access.
• guaranteed stability - the leftpad problem simply cannot happen with vendoring. It is still an issue with both GOPATH and module development
• security audits - the client is able to audit not just our source code, but our dependencies in a reproducible manner.

we also vendor tools such as gomock, goimports, swagger etc.

Now to answer your questions:

Did you run go mod download after cloning the repository? Why or why not?

no. because with the vendor folder, I should not have to.

Do you intentionally want to bypass go.sum verification

no, quite the opposite in fact. I imagined that go.sum would be a drop-in replacement for dep's Gopkg.lock

do you have in mind that the vendored dependencies remain unmodified?

yes.

Having to set an environment variable or constantly add a flag is a bit of a nuisance, but not a game breaker. The biggest issue to me is that I expected go modules + vendoring to behave exactly like dep with vendoring (algorithmic changes not included)

[rasky]

It's pretty obvious that vendoring wasn't even plan of the original module plan, and has been retrofitted at some point later in the process given different pressures.

Please remember that among our gopher values is to “be charitable”.
The approach to vendoring in Go 1.11 does seem to have evolved with the implementation, but that doesn't mean it was “retrofitted”.

I'm sorry if it was borderline harsh, but anything related to "proposals" in Go often leads to a very frustrating experience, so much that I tend to steer away as much as possible from it and focus on where the community experience is vastly superior (eg: compiler development).

This said, I still think the quoted sentence is factually true. It was clearly stated in Russ' first blogpost:

This proposal keeps the best parts of go get, adds reproducible builds, adopts semantic versioning, eliminates vendoring, deprecates GOPATH in favor of a project-based workflow, and provides for a smooth migration from dep and its predecessors.

and has been repeated other times elsewhere. The vgo design didn't include vendoring; at some point, it was re-added to the picture, but there wasn't any discussion or attempt (at least that I could find) to rethink vendoring in the Go modules world. The fact that a default that doesn't work for people using vendoring was chosen, or that there are large overlaps between the vendor directory and the download cache is probably a consequence of not having fitted vendoring in the picture from day one.

[thepudds]

@rasky I think you are right that the initial set of vgo blog posts initially proposed dropping support for vendoring.

However, there was then a fair amount of feedback from the community about the importance of vendoring. As far as I could see, that community feedback was clearly heard by the core Go team, which then resulted in the proposed plan being updated to actually retain support for vendoring with modules. The feedback was discussed in multiple venues, but this might have been the longest thread:

https://groups.google.com/d/msg/golang-dev/FTMScX1fsYk/uEUSjBAHAwAJ

...where within a week or so of the initial vgo blog post, the core Go team had switched to proposing retaining vendoring support with vgo/modules based on that community feedback.

In any event, sorry if my historical comments are not very interesting. A much more interesting question of course is what should the behavior be in 1.12+.

[bcmills]

The vgo design didn't include vendoring; at some point, it was re-added to the picture, but there wasn't any discussion or attempt (at least that I could find) to rethink vendoring in the Go modules world.

Yeah, I can see that. Bear in mind that the module support in Go 1.11 is still preliminary: we know that it has some rough edges, and vendoring is one of them.

I think it's important that we figure out a path forward, but it's not clear to me whether that will be to improve vendoring and better integrate it into module mode, or to improve the available module operations and deprecate vendoring entirely. I expect we'll try something in at least one of those directions for 1.12, but it's important that we have enough time to try out the improvements and make sure they work for those use-cases before they become the default.

In particular, because vendoring support in module mode is pretty awkward in 1.11, I think it's important not to try to push it as a default: I don't think it's the right design point yet, and I don't want folks to start relying on a design that really needs more thought.

[ondrej-fabry]

• a single deliverable - the client is able to take the codebase and build the application from source without internet access.
• guaranteed stability - the leftpad problem simply cannot happen with vendoring. It is still an issue with both GOPATH and module development
• security audits - the client is able to audit not just our source code, but our dependencies in a reproducible manner.

@JeremyLoy has made some great points here why vendoring is really needed in the software we build.

Without vendoring how are you planning to solve these points? Force everyone to host some caching/proxy thingy that would serve the dependencies transparently?
This would most likely compel developers to create various different solutions just like with vendor did before (Godeps, govendor..) and thus split the community once again while still not solve all of the points above.

What if some of the dependencies disappears and you don't have it in the proxy cache or anywhere else? This would force developers to fix/replace the dependency in their repository. However what if this is related to multiple versions with ongoing support? It would have to be fixed for all of them..

I'm really curious about your future solutions without the vendoring support that would satisfy the points above while at the same time not force us to manage extra things outside of the repository to successfully build our Go programs.

[bcmills]

@ondrej-fabry

Without vendoring how are you planning to solve these points?

That is a matter for a separate issue (perhaps #27618, and perhaps others as well). If you have specific requirements, please contribute an experience report describing both your use-case and how you address it today. (As I've said, it's not clear to me whether the solution will be to improve vendoring or to replace it, but experience reports will help either way.)

The matter at hand for this issue is whether -mod=vendor should be the default, in module mode, when the module's root directory has a vendor subdirectory.

[rasky]

Is there a single experience report, email or comment on GitHub issue that describes a real-world workflow in which the current default is correct? I have not seen any.

[bcmills]

@rasky I don't think we have any experience reports for vendoring with module mode at all. As I've said repeatedly, I think it needs work before we can recommend any particular workflow. For now, if the module support in 1.11 doesn't work for you, don't use it. GOPATH mode is still supported and shouldn't get in your way.

If you want to influence the future direction of module mode, you need to share your actual use-case instead of just arguing that what we have today doesn't work: your current criticism is not constructive. We already know that what we have today doesn't work for you: we need to know how to make that better.

[JeremyLoy]

@bcmills I added a link to my comments (and by proxy, this issue) to the experience report page you referenced.

If you want me to modify the experience report/submit it in a different way etc just let me know!

[driusan]

My experience with the current vendor behaviour has been as follows:

I've written a (partial) drop in replacement for git in Go, primarily so that I can use it on Plan 9 where there's no access to the real git client. A user requested that I vendor my dependencies so that it can be bootstrapped without @0intro's rc script hack, which seemed like a simple enough enough request so I decided to oblige.

I copied the files to the vendor directory as I did to vendor with previous versions of Go and it didn't work. I ran "go mod vendor" which appeared to generate some ancilliary files for Go 1.11 like vendor/modules.txt. It appeared to work (I was inside my $GOPATH), so I committed and pushed the results. The same user then filed a GitHub issue saying that it was still trying to pull the code that was supposed to be vendored code from bitbucket instead of using the vendored library when doing go build -x. After some investigation, I found the -mod vendor flag and it resolved his problems. (Most of this took place at driusan/dgit#142)

I support this proposal because the current Go 1.11 behaviour:

1. Isn't how previous versions of Go worked.
2. Isn't what package maintainers expect.
3. Isn't what users expect.
4. Makes it difficult to build offline, even when all dependencies are vendored.

[bcmills]

I think mod=vendor is too coarse: ideally the vendor folder should function more like an implicit set of replace directives, and we should automatically use whatever modules are found there (and fetch the remaining modules as usual).

That's certainly not happening for 1.12, but it's still on my radar.

[myitcv]

Forcing everyone

Of those people using modules (because nobody is forced to use modules; modules will only become the default in 1.14) it is demonstrably the case that not everyone has to use -mod=vendor.

As was stated before, for those people whose workflow has been broken/frustrated by a move to modules, yes, that is very unfortunate, but there are competing priorities here and limited resources.

A large part of the reason the default will only be switched in 1.14 is:

• this issue of "vendoring" has not been fully addressed
• documentation/examples/FAQs/etc are still lacking in places, including this one

Go is not a community-driven language

It is extremely unfortunate at best that that the term "community" continues to be used as a stick by (non-Google) people who feel like they haven't been listened to. Combined with the hyperbole of "forcing everyone", "spending thousands of hours", "wait 6 months for modules to be fully baked": these arguments/points simply become less rather than more convincing to my mind.

Contributing specific, constructive feedback/points is the way forward here. Please, let's all move forward on that basis.

[komuw]

@y0ssar1an

see #31857

[klische]

I support the notion that having a vendor folder would look for dependencies there first, with a resulting error message to the developer that "hey, we checked your vendor folder, but your vendor folder is out of date/missing certain dependencies in go.mod file, press Y to update them to your system $GOPATH/src directory or C to update your vendor directory, or N to do nothing. I think this makes it more user friendly, especially to someone who is new to Go.

Also, there does not seem to be a way to share the vendor folder across several packages in the same repository. For example, instead of github.com/username/foo/vendorIBM and github.com/username/bar/vendorIBM containing the same vendor folders, having a vendor folder at the github.com/username/vendor level would provide dependencies for both foo and bar

[provPaulBrousseau]

Also, there does not seem to be a way to share the vendor folder across several packages in the same repository. For example, instead of github.com/username/foo/vendorIBM and github.com/username/bar/vendorIBM containing the same vendor folders, having a vendor folder at the github.com/username/vendor level would provide dependencies for both foo and bar

The way that we deal with that is to create a mono-repo for any projects that are expected to interoperate and share vendor.

[leitzler]

@kliche the "shared vendor folder" you describe sounds just like a module cache, except for the fact that a module cache also know the exact version of you dependency. When you use "Go Modules" this is what you get today already. I can't see why you should add third party dependencies to your repo at all with the existing tooling around modules.

[owais]

I work on a project that uses a custom source code re-writer to re-write some of the dependencies before using them in a build and vendoring was the easiest way to achieve this. I don't even commit the vendor directory into VCS. It is re-created with go mod vendor before every build. The project has a Makefile that makes it easy to add the -mod=vendor to all relevant commands so it is not as bad a solution but it would have been nicer if everyone could reliably just use go build without having to worry about where to read the dependencies from.

(To elaborate on the re-writer, I use a tool to replace all references to protobuf package with gogoproto allowing me to use gogoproto with dependencies that use protobuf internally)

[amacneil]

Agree with the comments about this needing to be a project-wide setting, not a per-user configuration.

Is there a reason that -mod=vendor is not just the default? If people don't want to commit their vendor directory to git, that's their choice. But it seems that there would be no harm in always populating that directory by default (similar to node.js node_modules directory).

[andreygrehov]

I stumbled upon this issue while googling and trying to understand why go build ignores the vendor directory even after adding it. I'm a huge fan of Go's explicitness and for this use case I tend to agree that having an explicit order of where Go looks for dependencies would be great. There is no need for -mod=vendor in my opinion, unless I'm missing something (didn't read the entire thread).

[mvdan]

@amacneil @andreygrehov please see the very long thread above your comments; this is not a simple issue by any means. If you want to follow progress, see #27227. Asking the same basic questions asked in the original post again is just making the thread more unbearably long for everyone :)

[amacneil]

@mvdan thank you, I re-read this thread and to be fair it still does come across as if the Go team doesn't really understand this use case. However, after some digging through links I came across #33848, where it appears there is a concrete proposal to support automatic vendoring, which is great news. Hopefully others coming across this issue can also follow this new proposal.

[bcmills]

Closing in favor of #33848, which is a concrete proposal intended to implement the requested default behavior. (That proposal explicitly addresses the specific details and interactions that defaulting to -mod=vendor would imply.)

[bcmills]

Duplicate of #33848

[bcmills]

Note that #33848 now has a prototype implementation for review.