Proposal: split drivers pacakge into its own library

[jeffmendoza]

Python has libcloud, Java jclouds, Ruby fog, etc. Go ? There is a current gap here for go, in that we don't have a generic cloud abstraction library for creating VMs (among other things: storage, networking, etc.).

But wait, we have 'github.com/docker/machine/drivers' which is almost exactly perfect. Outside of machine it is relatively simple to drivers.NewDriver and driver.Create. I think this is a great opportunity to start something very useful.

Here is a gist of a quick program that creates a VM with the current drivers package.

Firstly, it would make sense for the drivers package to be in a separate repo. Where? I don't know. I imagine it would need to be incubated for a bit before being used directly by machine.

Next, a few changes would be helpful to generalize it, so that it is useful for the greater go community as the library for creating VMs on clouds.

The main change I think would be needed is:

• The drivers open up the 'docker port' witch is usually stored as a constant, and returned as a part of GetURL. This could be pulled out as a separate function.

A few changes, while not blockers, would be helpful to make it universally useful:

• Currently the drivers store the docker cert and key information, though it's not used by the driver. This could be pulled out of the driver itself, and be a part of machine consuming the driver.
• docker/machine/state cloud be a part of the library.
• codegangsta/cli is tightly coupled. A more generic options interface would help for clients that are not a cli.
• It is debatable weather docker/machine/ssh should be a part of the library, or be something that clients implement, and stay in machine.

Thoughts? Feedback?

[ehazlett]

Thanks Jeff! This is great timing as we are getting ready to plan for 0.4.0. One of the topics we've been discussing is making libmachine first class. I think this plays in nicely.

This could be a great way to include more drivers but perhaps the official Docker Machine only brings in the ones verified and long term supported, etc. This way, it would be easy to use the drivers from different apps without limiting to what we merge while still maintaining the experience we wish to have in Machine.

/cc @bfirsh @nathanleclaire @hairyhenderson @sthulb

[jeffmendoza]

Cool! I'm glad I posted this. I'm happy to start hacking things up, but didn't want to go at it in a vacuum.

[nathanleclaire]

Yep, really apropos as we've been chatting about this quite a bit lately. We plan on putting in some more formal proposals / specs for where we intend to go with it pretty soon. Appreciate you filing an issue first before going straight to hacking :)

This could be a great way to include more drivers but perhaps the official Docker Machine only brings in the ones verified and long term supported, etc.

+1

[thaJeztah]

For reference, an earlier proposal discussion can be found here; #500

[ehazlett]

Thanks @thaJeztah. I think the big difference is that this would become a Go library instead of separate binaries. Still relevant -- thx for linking!

[thaJeztah]

Yeah, I was aware of the difference, the general discussion around it applies though, so I thought it was worth a mention.

[ehazlett]

@thaJeztah -- absolutely -- thanks!

[zchee]

Would you add this discuss log in CONTRIBUTING.md or ROADMAP.md ?

[nathanleclaire]

@zchee We will eventually be updating ROADMAP.md and/or the project wiki with the high-level information, and filing issues for the specific, technical nitty-gritty discussion around the proposed / plan features.

[hairyhenderson]

+1 on this.

[hairyhenderson]

So, with talk of splitting bits into different repos (which I think overall is a good thing), this does introduce some considerations with dependency management. For at least the first little while we're going to be constantly updating Godeps/Godeps.json.

Not really a big deal, but there's going to be hiccups here, and I wonder if maybe it could be smoothed over by getting the build to issue a mild warning if we build with a stale version of the drivers...

[ehazlett]

@hairyhenderson good point. the engine does something similar with libcontainer. Perhaps we could use what they do.

@crosbymichael how do you handle updates with libcontainer in the engine?

[thaJeztah]

@ehazlett afaik, it's still a manual process, see, e.g. moby/moby#13196

[ehazlett]

@thaJeztah thx -- do you know if they just vendor everything or use a subtree or other git wizardry?

[nathanleclaire]

@ehazlett They have vendor.sh - https://github.com/docker/docker/blob/master/hack/vendor.sh

[sthulb]

Sorry for the delay in replying, I've been enjoying things away from computers for a little bit! But I'm back and refreshed!

I like the idea of allowing people to use machine's internal to control hypervisors away from machine, we've done a pretty good job at creating support for them :)

@hairyhenderson #500 would help with this, since we wouldn't necessarily build against driver versions.

[jeffmendoza]

Looks like libmachine #1211 #1239 will be a library to achieve the current functionality of Machine. Specifically to create a host with Docker installed and security certificates configured.

My intention with this proposal is to create a generic cloud abstraction library for creating VMs. One that could be used by all go cloud developers. Theoretically, Machine could be built on top of it, only adding the code to install and configure Docker.

I'm going to create a PoC based on the current machine drivers, not sure where yet.

[ehazlett]

@jeffmendoza I think they could potentially work together. I don't want the scope / maintenance to blow up by enabling generic access for everything but I think we could find a middleground where the drivers could be used to just access a provider and then hand off to do the rest of the machine work.

thoughts @bfirsh @nathanleclaire

[bfirsh]

You're right Jeff, we do have a good opportunity here to make something widely reusable. If it's not a massive amount of work to draw the line of abstraction where there's no Docker, then I'd say it'd be worth it. If its usage picked up, then the community would do loads of maintenance of it for us. We could get the cloud maintainers on board to maintain their drivers too.

Maybe it needs a different name to avoid confusion with Machine? I'm not sure...

If people wanted to use Machine as a library (which is less likely than wanting a generic cloud VM library) then they could just use the code that machine uses with import "github.com/docker/machine/...".

[nathanleclaire]

Yep, @ehazlett @bfirsh pretty much share the exact same thoughts. If it's no maintenance burden to decouple, I'm all for it, but I don't intend for it to become the project's primary focus, or to interfere with ongoing work with the Docker-centric aspects.

[dgageot]

Duplicate of #2815