gx use <repo>

[whyrusleeping]

Alright, so heres something moderately interesting.

I want to be able to have a gx repo (see the readme, nobody uses them yet) full of all my dependencies, that I keep updated, and run all sorts of checks on, etc.

Then, I want to be able to just tell my package to use deps from it. When it does this, it should mark the exact hash of the repo at the point its using it. From there, I can easily update all the deps in the repo at once. This also opens the door to having less strict versioning that i'm okay with. If the deps in your repo get updated, then your package could automatically use the latest. This is a strictly better model because you are explicitly trusting the source of the packages (via the configured repo)

I'm gonna keep thinking about this, and fill it out with more info later. Comments and questions welcome.

[hsanjuan]

This would be essentially like using the same package.json file for all projects?

[whyrusleeping]

Yeah, essentially.

[chriscool]

What if someone uses a repo (maybe managed by someone else) and then later finds out that a dependency that is outside the repo is also needed?

Is it possible to add another dependency that is outside the repo? What happens if the dependency is on a package already in the repo but another version?

[whyrusleeping]

Yeah, I would imagine stuff from the repo would serve as a base, and then you can import more on top of that. Multiple versions becomes an issue though, you would want to ensure the repo contains no conflicts, and the extra packages you import don't cause any

[chriscool]

Yeah, perhaps the multiple version situation can just be forbidden for now.

[whyrusleeping]

Some thoughts on UX (definitely not final):

gx repo use <repo>

Would set a field in the package.json pointing to that repo, and note the latest hash of that repo. Then, when running gx install, for each non-stdlib dependency, gx would search for it in the repo, at the hash noted in the package.json (this will have to be figured out, we would have to search through each package.json for dvcs-imports). And rewrite the paths to those hashes.

You can still import packages directly, but packages from the repo will take priority in the even of a conflict.

gx repo update <repo>

Would re-resolve the repo, note the new hashes, and update things as needed.

[jvsteiner]

moving my comments from #160 here for deduplication:

having worked with tool now for a month or so, as a user - one thing that stands out is that I don't have a convenient way to decide which versions of the dependancies I want: the package owner has to do a release with a named version (sure I can publish custom versions on my own, but it's freaking hard to then navigate the tree and republish all the parents that would be required to depend on those customs cuts, and their parents etc.). If the dependency tree is deep and wide (as we see in libp2p projects) this results in a lot of constraints between compatible versions. Recently, I've been building the un-gxified version of my project without any dependency management, and its more reliable in the sense that I can usually get it to build, whereas if I add a dep or update something i'm in deep trouble with gx. One idea I have is that there could be a command to "freeze" and "update" deps.

In this model, when you "freeze", it cuts a brand new version for each dependency - exactly as it is at that moment in your $GOPATH - by following all the package.jsons from the top down, recursively to the bottom (lib with no deps using gx), and then back from leaves to root. As you walk back up, the deps need to each be published for this to work - but that seems doable, and if you have to make actual changes in the files that imply causality. Maybe it can be avoided if you -f publish and don't permanently mess with the package.json files. Such changes could maybe be undone afterwards in order to avoid mass chaos in your $GOPATH.

The payoff is, I as a user, can adjust my regular un-rewritten libs the way I want for my deps, caring only about the golang compatibility of interfaces, not the full version locking stuff, then run a single command to freeze them, or unfreeze and refreeze as required without any worries about when the package owners/maintainers cut releases. I don't care about interdependencies at all - and I don't need a crummy vendor directory with a full copy of everything and can rely on ipfs to deduplicate content. Every release can be a custom release.

One possible catch is that, as a user, I have to run ipfs and take responsibility for making sure my custom versions of code are well replicated for the team I am working with - which I do anyway.

[whyrusleeping]

@jvsteiner Thanks a lot for your input! I think we're pretty close to being on the same page here.

Could you provide maybe an example or two of the workflow (in terms of hypothetical commands to run) that you think would be good?

Semi-related, have you seen gx-workspace? The docs in the readme are out of date, but the tool allows you to fairly easily update a dependency throughout an entire tree of gx deps (it handles all the gx updates and commits and publishes). Currently it iterates through the tree, updates things, runs tests, and prompts the user if they want to stop and investigate or continue to the next step. If that tooling was really fast and reliable, would that solve the issues you run into?

[jvsteiner]

@whyrusleeping - I took a look at gx-workspace. I didn't manage to get it to work right off the bat, but I'm going to spend some more time investigating it. It does look like it aims to solve the same problem domain, and if it were fast and reliable, I agree it would be really useful, and would solve a number of issues I've had.
That said, the approach I was thinking about above is slightly different one, and I'm still thinking what I describe might be useful in other situations. Perhaps, it could be a different command under gx-workspace even.
As I understand it, gx-workspace attempts to resolve the duplicates in dependancies, so in the end you get a top level package.json file that refers to released versions of your deps.
My proposal is that instead of trying to rely on released versions of packages, the tool looks in the regular directories for each of the packages, runs gx publish (-f) then takes the hash and uses it everywhere that dependency is used. In case the dependency repositories happen to be on released commits, then the hashed used will end up being released hashes, but it's not required. In fact, one would be free to modify a dependency, commit on a local branch, and the resulting package would use that instead. So there could be custom, non-official releases baked in, which wouldn't be replicated anywhere except the users machine, at least until someone else installed the package.json file.