ContainerPilot v3.0 Roadmap

[tgross]

RFD86 for ContainerPilot 3 has a lot of moving parts, so this issue is an attempt to set up an order-of-operations on what needs to be done. This includes parallel testing of each phase of work using the Consul, MySQL, and Nginx Autopilot Pattern blueprints.

• Tag ContainerPilot 2.x and update README to indicate that master is ContainerPilot 3.x work only (Add README notice that ContainerPilot v3 work has started #280)
• Remove all etcd-related code and tests (Remove etcd #281)
• Make some development environment updates:
  • remove live Consul server dependency for unit tests (mock it out except for integration tests) (Not entirely removed but the situation is improved: Use Hashicorp testutil for testing vs Consul API #286)
  • make buildable directly on MacOS w/o Docker (speed up code/test cycles, make go get-able) (Improve development environment #282)
  • update to go1.8 (Upgrade toolchain to golang 1.8 #285)
• Remove deprecated features like config for onStart, special chars in service names, etc. (Remove deprecated features #284)
• Update command runner to use Contexts
  • under-the-hood update to RunWithTimeout to simplify and reduce test surface for remaining work
  • update Nginx, Consul, and MySQL blueprints to use this (no config changes, just testing)
• Remove signal handlers in lieu of socket control plane
  • update Nginx, Consul, and MySQL blueprints to use this in their hooks
• Add config syntax to allow service to run its own command rather than shim the "main" one
  • update Nginx, Consul, and MySQL blueprints to use this in their configs
• Update command runner to support multiple-processes
  • split TTL and heartbeat from health check under the hood
  • keep existing config syntax except for adding command to service configuration (this keeps the change buried till config work is done)
  • update Nginx, Consul, and MySQL blueprints to use this in their configs (moving main app into a service stanza)
• Update config parser to use JSON5
  • change the configuration syntax to JSON5
  • dropping coprocesses and tasks from config
  • support multiple health checks by surfacing split TTL and heartbeat
  • update Nginx, Consul, and MySQL blueprints to use this in their configs

The Mariposa status endpoint can be added at any point along the way once the spec for that is better-defined.

We'll want to ship pre-release versions of this so we can test it out with Nginx, Consul, and MySQL blueprints without having to bump the major version of ContainerPilot n times along the way as we add new major backwards-incompatible features as per the RFD.

So we will tag releases as 3.0.0-dev (and ultimately 3.0.0-rc1, 3.0.0-rc2, etc.) until we make the big config syntax change at the end of this window. And we can have a similar split for the Nginx, Consul, and MySQL repos during this window.

[jwreagor]

@tgross Is the new unix socket based control plane still in need of dev? I thought your spec was extremely straight forward and would like to take a crack at it.

[tgross]

@cheapRoc I'd love to have a contribution from you on this. I'll be able to work on some of the configuration work in parallel to that.

I'd recommend opening a PR early so we can chat about the work and you can get any questions answered quickly. I have an internal timeline that I'm working under which has plenty of room in it but if things are going to get stalled as other work comes up please feel free to reach out and I'll carry the PR if we need to.

[tgross]

First pre-release with current work-in-progress https://github.com/joyent/containerpilot/releases/tag/3.0.0-dev

[jwreagor]

Sounds good to me. I'll try and init the PR later today.

[tgross]

• Add config syntax to allow service to run its own command rather than shim the "main" one
  update Nginx, Consul, and MySQL blueprints to use this in their configs
• Update command runner to support multiple-processes

I've been working thru the implementation details and there's a good chance this will have to be atomic if we want to do it sanely. That's my next major task.

[jwreagor]

Work on the control plane/socket has initiated over at #289.

[tgross]

#290 has been merged, which messed up the order of operations a bit with this roadmap.

I was really hoping to wait to refactor how the config stuff is tested until we did the big config update, but swapping out the underlying mechanism for running services and health checks made this too hairy -- we'd end up having to do the work twice. Coming up next:

• @cheapRoc is going to rebase WiP: Initial implementation of HTTP control plane #289 onto that work
• I'll continue on by first sanity-checking our current state vs Nginx/Consul/MySQL blueprints
• I'll then move on to the config update (which probably won't start until the first week of April because of internal meetings with our team).

[jwreagor]

It makes perfect sense to iron out and refactor the work from #290 then continue pursuing extra 3.0 features like control plane. I've started integrating my first PR and will open a second now that I'm based on top of an updated master.

[tgross]

@cheapRoc I want to make sure you know we'd be happy to take a PR that implements only a subset of the control plane and leaves "dummy" handlers for the rest. Your call, but please don't feel obligated to take it all on at once! 😀

[tgross]

Another note: I've split the v3 issues label into v3.0.0 and v3.x and reassigned issues as required. The idea is that items labeled v3.0.0 are blockers for the release of v3, whereas v3.x are items we want to do in v3 at some point but aren't blockers for the initial release.

[jwreagor]

@tgross Fair enough. I realized the spec was still in draft so I also thought of releasing this iteratively. I'll wrap up the server init again, cut a new PR, and we can move forward with that.

[jwreagor]

Posted the latest rework into PR #300, closed the old PR.

[tgross]

I'm going to close this as all the checklist items here are done. I'm in the process of testing and debugging and documenting now.