consul validate only accepts .json or .hcl files => impossible to roll out + validate configs e.g. with puppet

[jenshadlich]

consul version for both Client and Server

Client: v1.0.0
Server: v1.0.0

consul info for both Client and Server

Client:

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 0
	services = 0
build:
	prerelease = 
	revision = 51ea240
	version = 1.0.0
consul:
	known_servers = X
	server = false
runtime:
	arch = amd64
	cpu_count = 2
	goroutines = 44
	max_procs = 2
	os = linux
	version = go1.9.1

Operating system and Environment details

Linux / Ubuntu 14.04

Description of the Issue (and unexpected/desired result)

My use puppet to roll out consul configurations. Puppet appends a random string to the filename e.g. my-service.json20171026-6787-5on04gwhich breaks the validation. This makes the validation pointless because we now have to disable this step. It would be good to control the filename check e.g. by command line argument.

Execution of '/usr/local/bin/consul validate /etc/consul.d/client/my-service.json20171026-6787-5on04g' returned 1: Config validation failed: Missing or invalid file extension for "/etc/consul.d/client/my-service.json20171026-6787-5on04g". Please use ".json" or ".hcl".

Reproduction steps

Create a config file with a file extension other than .json or .hcl and validate.

[jenshadlich]

I guess this behaviour came in with #3480

[magiconair]

@jenshadlich this is not a mistake and was always working like that inside a config directory. Only for the main config file the .json extension was not required. You can change your puppet roll-out rules to put the random id before the extension, e.g. my-service-20171026-6787-5on04g.json

[magiconair]

and yes, this was added as part of #3480 also to make it unambiguous to the parser whether it should parse a JSON or a HCL file.

[jenshadlich]

@magiconair thanks for the feedback. Can you please tell me what I need to do make puppet do that? (has to work with puppt 3.7 btw)

[magiconair]

No, unfortunately not. However, my guess is that you can customize the format of the random string that is being added and if all fails you could just add another .json extension so that you would get my-service.json20171026-6787-5on04g.json

[magiconair]

A better option is to validate the entire config directory instead of the file since validating incomplete config fragments is also no longer supported. You would probably get a data_dir missing error or something similar.

[jenshadlich]

@magiconair I'm afraid that it's not possible to simply tell puppet to add a .json extension when using the validate_cmd (it creates a temp file). I found really awkward ways to work around it.

It would be great if you could provide a way to disable the file extension check or allow forcing it via --json or something similar.

In theory the puppet code is very straight forward like this (if not validate would be so strict):

file { "${consul_clientconfig_dir}/${name}.json":
    ...
    content      => template("${module_name}/template.json.erb"),
    validate_cmd => "${consul_binary_dir}/consul validate %",
    notify       => Exec['consul_reload']
    ...
}

All workarounds will create a mess and are harder to read and maintain.

[magiconair]

@jenshadlich This will break as soon as someone wants to use .hcl files since the only backwards compatible way of doing this would be to assume that files w/o extension are JSON. Lets try to find a way that works here since this will solve this for everybody. This should have never worked like it did before in the first place IMO.

[magiconair]

We could loosen the requirement on where the extension has to be. Instead of at the and we could just accept it anywhere. This makes it more ambiguous but for all practical purposes we should be fine unless someone creates a JSON file and calls it foo.hcl.json.

[magiconair]

The question about -json and -hcl flags is whether they should apply to all files, just the main file or just the ones without extensions.

[jenshadlich]

@magiconair I understand your concerns. In my case the validation shall take place for exactly one file. So -json or -hcl flags would work perfectly.

Validating the extension anywhere feels a bit strange even though it would also do the job. About the scope of the flags: I personally think the most logical thing is that it applies to all files. I would assume that the client knows what is is doing when setting those flags.

Another option / a little tweak could to fallback to consul's preferred config file format (I don't really care if it's .json or .hcl unless it's deterministic) if no matching extension is available.

[martinseener]

Yeah, i also think that relying on a filename's extension is so DOS-era. The "validate" command should instead try to determine the type from the content (like it's done for most pictures for example).
Nevertheless, we use the following validate_cmd string for validation at the moment. It basically creates a symlink to the temporary file which is invalidated afterwards automatically, since puppet only creates a temporary file, thus it's a bit more secure: "${::consul::bin_dir}/consul validate $(ln -s % /tmp/consul-config.json && echo \"/tmp/consul-config.json\") > /dev/null"

But you can see it in the puppet module which my colleague referenced already

[magiconair]

@martinseener I need to think about that. IMHO, config files should have only one format but if they can be in multiple formats you need to have a way of telling them apart unambiguously. Usually, I prefer things to be explicit so that operators know what they are dealing with and don't have to guess.

I've written a PR which adds a -config-format (hcl|json) option which forces the the format independent of the extension. We could also return to the 'it's JSON unless .hcl' approach but then the people who want to use only HCL would have a problem.

I'll discuss this with the team internally to see what they think.

[martinseener]

@magiconair sure! i think the PR your wrote is a good approach and should ease some things. Thanks alot for that!

[jenshadlich]

@magiconair great! many thanks!