Add manifest-query commandline utility for parsing JSON or YAML.

[timstclair]

Add a commandline utility for parsing arbitrary JSON or YAML, to make operations with these formats more feasible in bash scripts.

Resolves: #9937

Note: I'm not sure about the name, and am open to suggestions.

Examples

$ manifest-query -f examples/guestbook/frontend-controller.yaml '{{ .metadata.name }}'
frontend
$ manifest-query -f examples/guestbook/frontend-controller.yaml '{{ range .spec.template.spec.containers }}{{ .name }}{{ end }}'
php-redis
$ echo '{{ .foo }}' > query.txt
$ echo '{ "foo": "bar" }' | manifest-query -t foo -o result.txt
$ cat result.txt
bar

Keys with non-alpha numeric characters must be indexed with the index function:

$ manifest-query -f cluster/addons/fluentd-elasticsearch/kibana-controller.yaml '{{ index ..metadata.labels "kubernetes.io/cluster-service"}}'
true
$ manifest-query -f cluster/addons/fluentd-elasticsearch/kibana-controller.yaml '{{ index . "spec"  "template" "metadata" "labels" "k8s-app"}}'
kibana-logging

$ manifest-query --help
Parse JSON or YAML input and format it with the given template.
The template is interpreted via the go "text/template" package.
For usage and syntax, see: http://golang.org/pkg/text/template

Usage: 
  manifest-query [template] [flags]
Flags:
  -h, --help=false: help for manifest-query
  -f, --input="": File to read data from. Defaults to stdin
      --log_flush_frequency=5s: Maximum number of seconds between log flushes
  -o, --output="": File to write output to. Defaults to stdout.
  -t, --template="": File with query template. By Default, template is the first positional argument.

[mbforbes]

nit: 2015

[mbforbes]

@davidopp for triage and, if applicable, pointing to two TL reviewers?

[bgrant0607]

See also #4348

[timstclair]

Thanks for the pointer @bgrant0607. I'll merge those changes in once that issue is resolved (maybe in a follow-up PR depending on when it's merged).

One thing I was considering was whether the command should just be part of the kubectl binary, perhaps as another sub-command (kubectl parse?), or maybe even kubectl config view with a flag to point at a specific file.

[davidopp]

@davidopp for triage and, if applicable, pointing to two TL reviewers?

The issue this PR addresses was marked v1.0-post. This PR is very low risk since it just adds a new binary, but in theory we're only accepting PRs that address 1.0 issues...

[bgrant0607]

Seems like a poor-man's jq. Could also just put jq into a docker container, like https://github.com/Gonzih/jq-docker

We could put this into contrib for now and figure out what to do with it later.

Also similar to #4471 and #7576.

[timstclair]

Ok, sounds like this needs some more discussion (post v1.0). If you want me to move this to contrib I'm happy to do so, or I can just archive the change until further notice.

Thanks!

[bgrant0607]

Please just archive for now. I don't think this is critical for 1.0

[timstclair]

Closing until we have a better idea of what we want out of this.

[zmerlynn]

I'd like to propose reopening this, or something like this. See #17241 (comment). We have chunks of deployment code that, for good or ill, need to query YAML files for specific fields, and no easy way to do it. In the case of #17241, the right way forward is honestly #9470, but that's going to take a while longer.

I searched for something like jq for YAML and didn't see anything (ETA: that I'd hang my hat on as better than what was proposed in this PR). We've been strongly encouraging YAML for so long now that it would be a shame to move all of our add-on config back to JSON just to be able to query it from shell, so jq itself seems possibly incorrect (though I could settle for that solution here).

[timstclair]

BTW, here's the recommended approach from jq for yaml: https://github.com/stedolan/jq/wiki/FAQ#projects

Q: How can I use jq for YAML instead of JSON?

A: Use y2j. It provides a wrapper called yq that uses Python to convert YAML to JSON, runs a jq filter on the JSON, and converts the result JSON back to YAML.

It sounds like built in yaml support in jq is not going to happen for a long time.

As something for use by our internal scripts (as opposed to something we encourage the use of) I think the utility in this PR would probably be better suited to the task than y2j

[zmerlynn]

@timstclair: I saw that, but then if we Dockerized y2j, we'd basically be cheating and Dockerizing the python interpretter. Which might be a fine solution for #17241 as well. :/

[yifan-gu]

@zmerlynn @timstclair Maybe dockerized python is an acceptable temporal solution. I am only concerned the size of the image is not small (~700mb)
Btw, I am not sure if the python codes will still be needed if we replace the whole updator script with Deployment API?

[yifan-gu]

@zmerlynn
Do you think I can try to experiment with Dockerized python? (e.g. if there's no python on the machine, fetch a python image)

[zmerlynn]

@yifan-gu: That seems fine. If we did that, would you just use Dockerized y2j and jq as well? That seems like the right solution.

[yifan-gu]

@yifan-gu: That seems fine. If we did that, would you just use Dockerized y2j and jq as well? That seems like the right solution.

Ack.