Feature: Add dependency spec to Chart.yaml (was Kubernetesfile)

[osterman]

what

• Introduce the concept of a Kubernetesfile that can be included with applications to express how the service is deployed. It would contain the YAML or JSON of kubernetes resources.
• Support remote dependencies expressed via a git URL to a repo containing a top-level Kubernetesfile

why

As applications change or evolve, the way they are instantiated may change. For example, new environment variables may be introduced or removed, new command line arguments may be accepted, additional ports exposed, etc. From a maintenance perspective, it would be convenient if the author of the app could keep the Kubernetesfile updated as the code evolves rather than try to keep multiple repositories (e.g. external Helm Charts) current.

This is similar to what the Dockerfile expresses for building an application, the Jenkinsfile for integrating it with CI/CD pipelines and (once-upon-a-time) the Vagrantfile for setting up a local environment.

our use-case

We have a product that is composed of a dozen proprietary micro-services as well as other off-the-shelf components like statsd. We want to use a repository that maps out the Chart for how to deploy our product in our Kubernetes cluster (the way it works now), but we want service owners to "own" the definitions of how to deploy their service much the way they "own" their Dockerfiles for building them.

We like the way it works now, but would like this alternative approach for including services that we own.

[technosophos]

So that we avoid any potential rabbit trails: Currently, charts do not have to be in their own repos -- they can live alongside source. We got rid of the Helm Classic requirement that charts all exist in one repo.

I'd love to see some mocked up examples of what this might look like. The goals stated above are definitely inline with what we want to accomplish with Helm. But I admit that I'm having a hard time mentally mapping between a Dockerfile and Kubernetes manifests or Helm charts. Is this sort of like a special case of a values.yaml file?

[osterman]

I will write this up as an additional comment. I think another way to look at is to support charts of charts (aka charts with dependencies that are other charts)

[osterman]

Sure! Also, I am not a "helm" expert - so if this is already possible, please let me know.

NOTE I think what I was calling the Kubernetesfile just makes it more confusing from the helm perspective. What I'm really just saying is I want a chart to pull down other charts.

Let's start by creating a fictitious company called Foobar. Foobar is community site with the following architecture. We'll also call this the "foobar" product.

• frontend (NodeJS) - in-house/proprietary service in dedicated private repo
• backend API (Go) - in-house/proprietary service in dedicated private repo
• batch-jobs (Python) - in-house/proprietary service in dedicated private repo
• statsd (external) - uses some charts from public repo
• memcache - uses some charts from public repo
• redis - uses some charts from public repo

Our objective is to deploy the entire product in one fell swoop. With one command (e.g. helm install --values=foobar-prod.yaml foobar), we should to be able to setup the complete application stack above.

This is possible today, but since charts are downward facing there needs to be one repo that specifies the entire environment. The "frontend" could specify everything that it needs (e.g. backend), but the "frontend" should not be defining the kubernetes resources for the "backend" because the backend service is the right place for it. In a micro-service architecture, there will be dozens if not more projects.

What I would propose is creating a new repository for our "product" called "foobar". This repo has a chart that says to install the "redis" chart, "statsd" chart and "memcache" chart by pulling them down from (https://github.com/helm/charts). But it would also be able to specify git://github.com/foobar/frontend.git and git://github.com/foobar/backend.git. Each one of those would have their own charts/ directory that define just how to manage that one service.

[osterman]

The helm fetch command is essentially what I'm referring to, but the problem is that it explicitly copies the dependency charts instead of referring to them. This feels like we then have two places we need to update them any time one change is made.

[osterman]

I think git submodules will get me most of the way there.

[technosophos]

There is a reason for including the charts statically (in charts/), and a reason why we went back from resolving by reference to statically including them. In a nutshell, the reason is that involving the Tiller process in resolving dependencies prevents us from building immutable packages. If you're really interested in it, I'll type out the full explanation with all of the use cases and stuff. But in a nutshell, it led us to the decision that a packaged Helm chart should contain all of its dependencies and never require additional resolution steps.

But you've really got me thinking. Here's an idea.

Probably, though, a suitable middle ground would be to come up with a file format for a file that goes inside of charts/ and specifies dependencies. Then we could add some commands to manage that.

So I'm thinking that helm chart add $REPO/$CHART-$VERSION (or something like that) would add an entry to, say, charts/requirements.yaml and helm fetch that dependency.

Then a command like helm chart update would update all of the charts in charts/requirements.txt to their current version.

The chart repo spec means that we probably wouldn't need to build GitHub-specific logic. @michelleN was working on figuring out how to use GitHub to manage chart releases, but I think that got back-burnered for some other Alpha.2/3 tasks.

All of this would give a workflow like this (drawing on your example):

$ cd foobar
$ helm chart add myrepo/frontend-1.2.3 myrepo/backend-2.3.4 //...
$ helm install .
fluffy-bunny

Then, to update your deps:

$ cd foobar
$ helm chart up
Updating to myrepo/backend-2.3.5
$ helm upgrade fluffy-bunny .
Upgraded release fluffy-bunny

Does that get closer to meeting the use case you are elaborating above? Does it sound viable? I guess we should mock out a requirements.yaml file (or maybe discuss putting that info in Chart.yaml again, which is where it originally was).

[osterman]

@technosophos: wow, yes that looks beautiful. It reminds me a little bit of the interface to bower
https://bower.io/docs/api/#update

[osterman]

Just to be extra clear, the helm chart add should accept a repo URL like:

helm chart add https://github.com/supercorp/frontend.git#release-1.2.3 \
    https://github.com/supercorp/backend.git#release-2.3.4

Where everything after the # is a tag, branch or git hash. If #<something> is omitted, it should default to #master

I am not clear on how to express a non-standard charts/ folder. Bower does not support this (from what I can tell).

[technosophos]

So @prydonius just suggested that if we added dependency information to Chart.yaml instead of a separate file, we'd actually have the entire dependency graph inside of a chart repository's index.yaml file. This could be an incredibly powerful feature.

Chart.yaml might then look like this:

name: wordpress
#...
dependencies:
  - name: percona-mysql
    version: ">5.7.0"     # Human-generated
    lock: "5.7.1+eff45" # Computed by Helm, version actually used.
  - name: memcached
    version: 1.2.3

Chart developers would be able to write the chart.yaml file and then just do something like helm chart build (name TBD) and have those charts vendored into wordpress/charts for them.

And @osterman's helm chart add command could then write that info right into the chart.yaml file as it installed things.

Among the other benefits:

• It would be relatively straightforward to be able to report on which packages were out of date.
• It would be easy to display chart relationships and dependencies
• The dev experience would be much better than today's copy-paste mechanism

[migmartri]

My 2 cents regarding the dependencies topic.

Dependencies definition

• I think that there is value on representing the dependencies and its versions in a declarative way (a la Gemfile, package.json, etc)
• I am not in favor of allowing any program to modify files that are supposed to be modified by the user, I prefer more a Gemfile, Gemfile.lock model.

I like @technosophos proposal and share the long term benefits exposed. But as initial version, I would simplify it forcing the user to add an specific version (and maybe latest) of the dependency that she wants to add.

name: wordpress
#...
dependencies:
  - name: percona-mysql
    version: "5.7.0"
  - name: memcached
    version: "1.2.3"

This will not require helm to resolve dependencies nor modify the chart.yaml file while still we get the declarative benefits of it.

Installing/ updating dependencies

• Updating a dependent chart should be possible to be done changing the dependencies definition in the chart.yaml and calling for example help chart update. I would skip the implementation of helm charts add for now.
• helm chart update will juts download the resolved dependencies into /charts (We might want to think of renaming charts directory into something else, dependencies for example.)
• The /charts directory could be vendorized (added to the repository) or just ignored. In the case of being ignored, helm chart update will be required before packaging or installing a chart, or maybe these commands will download the dependencies for you. Similar to node_modules.

In my opinion, this will put as in the right track. Acting as base for future helper functions (helm chart add) and more advanced dependency resolution.

[philips]

@technosophos I will file another issue but I would like to see something like:

name: wordpress
#...
dependencies:
  - name: percona-mysql
    digest: sha256:eff45

But, I think to do this we would need to do some work in the provenance specification.

[technosophos]

Elsewhere there has been some confusion over the dependencies: section of Chart.yaml that makes me think we do in fact need to move this to a dedicated charts/requirements.yaml file. It is unclear what that section is conveying about a chart.

tl;dr

The dependencies section expresses developer intent, not chart state. As such, it should not be included in Charts.yaml. The better place is in charts/requirements.yaml.

In detail

If you have not read @sdboyer's excellent blog post on package management, please read it now: https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527#.vma29reij

In that post, Sam points out the difference between capturing developer intent and capturing the relevant state. What we're working on as part of this issue is capturing developer intent.

So the anticipated workflow that this issue describes goes like this:

• I create a chart
• I indicate that this newly created chart has a dependency on some other chart, and that it should work fine with version (or version range) X.
• I run a command (perhaps helm charts update) and Helm fetches a version of the dependency that satisfies my intent. That version is placed in charts/.
• At some point later, I revise the version range, and then re-run helm charts update. Helm then ensures that the version I now have in charts/ satisfies that condition.
• At some point later, once dev is complete, I run helm package. At that point the chart and its dependencies become immutable in virtue of the fact that they are now packaged together as a bundle (and presumably signed).

Charts can be vendored. But one request has been that for those developing charts, they need not check the dependencies into their VCS tree. The mechanism above is a first step toward supporting that model. The second step (which is not strictly necessary, but is definitely a best practice) is to introduce a lock (either as a lock file requirements.lock or as a program-controlled field on requirements.yaml). A lock file would definitely use a strong identifier (of which a hash is one example) to validate that the version requested matches the version fetched during a helm charts update.

I'm suggesting that we remove the dependencies: section from the Chart.yaml file specifically because people are already assuming that the presence of that information in the Chart.yaml is a guarantee that a specific version of the dependency is present in the chart.

Instead, we should put it in a place where it clearly displays itself for what it is: a statement of developer intent. I think we can follow the Python model and put it in charts/requirements.yaml.

Then we end up with a fairly clean separation of concerns like this:

• Chart.yaml and the .prov file (and a .lock file in the future) reflect the state of the chart
• requirements.yaml expresses developer intent

Pros

• Clearer expression of separation of concerns
• Tiller does not need to understand these directives, and can simply ignore them

Cons

• It is no longer as easy for indexing/UI tools to give contextual information about the relation of a chart to its dependencies. (To which I would point out that using the existing dependencies: for this is not sufficient anyway.)

[philips]

I agree with the proposal. I do think that you want to be able to point to a particular dependency by digest in both requirements.yaml and requirements.lock though. By offering both I have the option of dev'ing with loose dependencies or very specific pinned dependencies.

[osterman]

This is kind of related to what I was originally thinking but could not articulate: https://github.com/roboll/helmfile