installing development tools

[nathany]

One thing the import graph doesn't provide is an easy way to install the tools used to work with a particular project.

Maybe this is worth considering as a future enhancement to dep. A list of dependencies in the manifest that aren't imported, but have binaries that the project uses.

[tools]
"github.com/golang/lint/golint" = "^v1.0.0"
"github.com/go-swagger/go-swagger/cmd/swagger" = "^v1.0.0"
"github.com/hirokidaichi/goviz" = "^v1.0.0"
"github.com/vektra/mockery" = "^v1.0.0"
"github.com/jstemmer/go-junit-report" = "^v1.0.0"
"github.com/haya14busa/goverage" = "^v1.0.0"

In projects I've worked on, this is just a sequence of go get commands in a Makefile, which certainly is good enough if something like this isn't warranted.

slightly related to #220

[groob]

I've been thinking about this one lately. We had a CI dependency which was being pulled with go get in our Makefile, and it eventually lead to a broken build because the tool changed upstream to depend on 1.8 beta features.

[nathany]

This could solve that, but in the longterm there could be other solutions. Assuming go get sticks around, maybe it would eventually allow go get -u github.com/jstemmer/go-junit-report@^0.8.0 or even just have the same "latest stable version" logic as dep.

The thing I find odd about my suggestion here is that [tools] all get installed to a common bin folder. That doesn't feel quite right, but per-project bin folders doesn't seem great either. I don't really know which I'd expect.

At least with go get, the behaviour is global and always has been.

[sdboyer]

First, there's two cases here to delineate:

• Tools used for e.g. code generation like github.com/tinylib/msgp, where both the supporting binary and your code need to import some library, and need to use the same version of the lib
• Fully external tooling, like a linter, that needn't be entwined with your project's depgraph

(I talked in greater detail about this difference in this glide issue: Masterminds/glide#418 (comment))

We have to separate these cases because only the former demands some way of signaling to the solver that a main package be incorporated when computing the depgraph. (Being main packages, we can't do it with an import). This is one of the main use cases that required packages in the manifest are intended to cover.

So, that's the first bit. The second is what you're directly interested in - fetching and building a supporting tool.

I do believe this is an important feature to include, but it also significantly expands scope. Right now, dep deals strictly with the management of source code - it does not touch compilation or binaries, at all.

Also, you've noted, this is basically just a more version-aware reimplementation of go get. IMO, that's something we should be looking at (as I alluded to in #183 (comment)), but that's stepping directly into modifications of the go toolchain. That's not in our purview to deal with directly here, and if we want to keep dep on a sane trajectory towards official-ness, we need to kick some of those decisions down the road.

Basically, while discussion now is ofc fine, before really moving on any implementation we need to wait for #154 :) And we're working on it!

[mh-cbon]

The thing I find odd about my suggestion here is that [tools] all get installed to a common bin folder. That doesn't feel quite right, but per-project bin folders doesn't seem great either. I don't really know which I'd expect.

npm does put a .bin folder under its vendor folder. Its great thing because if a project rely on some binaries, it matters that the version remains consistent across all the builds.
Now, it means that somehow for all OS, it is possible to update PATH with ./vendor/.bin (assuming the dest folder is ./vendor/.bin). I did not dig in npm to see how they manage that, i believe it is not trivial.
It also matters for binary tools that would rely upon such versioned binary, they will need a sort of library which knows how to locate those binaries over the global one, or the one in the path.

[nkovacs]

I did not dig in npm to see how they manage that, i believe it is not trivial.

Npm adds node_modules/.bin to PATH when running scripts in package.json (https://docs.npmjs.com/misc/scripts#path) using npm run.

The other common solution is to install a tool globally and also install it locally as a dependency of the package. The two versions might be wildly different, so if you invoke the global version (which is in your PATH), it'll complain but still run (gulp does this), or the global version is smart enough to load the local version instead. There's a helper library for the latter: https://github.com/js-cli/js-liftoff

[mh-cbon]

right! https://github.com/timoxley/npm-path/blob/master/index.js#L131

[nathany]

Fyi, there are some concerns over security with a global bin path.

https://groups.google.com/forum/#!topic/golang-dev/fZ_maFYY9nU

golang/go#18690 (comment)

[mh-cbon]

lets summarize,

q1

I am also concerned that this

go get github.com/hacker/ping

might install some random program on my computer.

I can t really understand that.
If you use go get then it means you do use Go,
you should understand what you are doing.
If not, better to read the doc or to do something else.
It s a reasonable behavior, and hygienic, to always double check the repo
you are installing before doing it,
i can only think of heavy windows users of the get-blind-then-click method.

Should unreasonable behaviors be a stop to the creation and delivery
of an integrated environment for the benefit of all ?

Said differently, cars can kill, but people still uses it, irl the benefit is larger than the risk.

q2

Adding %GOPATH%\bin into %PATH% is dangerous in fact. Especially, go install github.com/xxxxx/yyyyy/...
This blind what executable will be installed. If the %GOPATH% is prepended to ahead of %PATH%, it can overwrite go command.

This is a pure problem of setup, i don t see what prevent
to implement and distribute a tool to check about that,
or integrate a kind of self check to avoid that situation.

I might be missing enough vision about those two Qs,
i just don t feel like they should prevent such change.

---

A real question, to me, is how to avoid naming collision.
the current go package naming convention is great for that purpose
as it always includes the triplet provider, user name, project name.
It automatically helps to avoid any collisions.

Regarding this thread, if no special strategy is applied to
avoid collisions, i see only two advantages,
save some keystrokes when typing a command,
perpetuate a well known behavior.

I m sure and certain that collisions will happen,
I wonder about the extent of such situation on the end user
and its capability to fix the situation.

On the other side of the coin, if we d include the
whole triplet to locate a bin program,
well that s a long name to type.
I also wonder if that will support auto-completion.
It is not very user friendly.
The result will not meet user expectation of what the system should produce,
in his idea.

One idea is to not install those binaries into an accessible folder to PATH,
but install one program in PATH which would be a wrapper to the
invocation of such binary.

gobin yourprgoram arg blah
gobin that-user/yourprgoram arg blah
gobin provider/that-user/yourprgoram arg blah

Where gobin would locate the program
given hints (bin name, user name, projet name)
provided by the end user, its cwd ect,
invoke and reproduce IO of the dest program.

• Somehow, it might be able to include and integrate an
  autocompletion which might be equal on all systems.
• As the program won t be too stupid, it can provide the
  expected user friendliness.
• As its not adding any new value on PATH (might be a go bin),
  it fixes some concern raised previously.
• As its a program, it can help to prevent distribution binaries to be overridden.
• But, it won t make coffee, it will still need to double check the source repo before running go get.

[nathany]

Interesting: https://github.com/twitchtv/retool

[JelteF]

Just wanted to post the tool here that I've been working on to solve this at my company which integrates with dep: https://github.com/GetStream/vg

After installation you can run vg ensure and it will install all the packages in the required list.

[vektah]

I wrote something that helps with this: https://github.com/vektah/gorunpkg

The idea is rather than installing you just check the code out in vendor, then use

gorunpkg github.com/whatever/package -args to package

gorunpkg will resolve the package, first looking in vendor, then in your gopath, build, and run, without installing globally. This is closer to how npm/composer/etc work by being able to run binaries out of ./node-modules/.bin/ in the version defined in the lock.

The problem ive hit is because go dep only looks at imports there is no way to add github.com/whatever/package to the lock.

[JelteF]

@vektah looks like a cool alternative to my vg solution! Regarding your problem, if you want to add a package that is not an import to the lock file you should add it to the required list in Gopkg.toml. This is why vg reuses that list as well.

[cyberhck]

So if I want to install testify (but only on dev version, since I don't test it on production), is there a way to do that with dep (like npm's devDependencies)? Is this issue being worked on? Can I help with anything?

[mcandre]

retool, gorunpkg, vg are neat and all, but this is something that a modern package manager is reasonably expected to handle, without requiring yet another external tool to accomplish this, and of course yet another set of shell commands to actually get retool/gorunpkg/vg installed themselves!

Cabal has solved this. Ruby has solved this with Bundler. NPM has mostly solved this with npm bin. I think dep ensure should loop through required items, compile them, and install them to the normal $GOPATH/bin directory, where $PATH normally includes them. OR, dep can select an official retool/gorunpkg/vg champion and use it like bundle exec (dep exec?) to run per-project Go tools. I don't care which style is chosen, I just want it automated, hey!

I wouldn't even mind global collisions between required versions, between different Go projects or even transitive dependencies. I just want my tools to do their best to do the lion's share of the low-level work involved in developing modern applications, and if that means forcing a project tree to the latest version of a linter, so be it. But I want dep to be able to identify the conflicting versions and to install the version of my choice on my behalf.

[gyurobenjamin]

Now we only need a postinstall or something like that.

[jszwedko]

@kevinburke why was this closed? Has it been decided that dep won't support this?

We use code generation in a few places and I agree that it would be nice to vendor and manage these with dep rather than introducing a new tool.