s3 provider not parsing JSON

[pimguilherme]

Hi dear maintainers!

It looks like the s3 provider does not parse JSON as described on the docs.. I have successfully parsed a YAML file but not a JSON file using the vals eval command

browsing the code we can see that there doesn't seem to be any json unmarshalling other than in google sheets provider

Could you please confirm if this is really the case?

Cheers!

[mumoshu]

@pimguilherme YAML is a superset of JSON so even though there isn't a specific JSON unmarshalling code in vals, our YAML unmarhsalling code should unmarshal your JSON data just fine!
Could you give us a full reproduction example?

[mumoshu]

Here's the part of the s3 provider that does json/yaml parsing

vals/pkg/providers/s3/s3.go

Lines 72 to 85 in 332f488

	func (p *provider) GetStringMap(key string) (map[string]interface{}, error) {
	yamlData, err := p.GetString(key)
	if err != nil {
	return nil, err
	}
	m := map[string]interface{}{}
	if err := yaml.Unmarshal([]byte(yamlData), &m); err != nil {
	return nil, err
	}
	return m, nil
	}

[pimguilherme]

thanks for reaching out @mumoshu !

I was trying to pull a list of strings from a tfstate file in AWS S3.. I did some tests now and realized we can only pull values that are text, no numbers or anything else:

$ echo 'test_plain_yaml: ref+s3://teste-s3fs/test.yml?profile=homolog' | vals eval -f -
s3: successfully retrieved object for key=teste-s3fs/test.yml
test_plain_yaml: |
  key: 456

$ echo 'test_plain_json: ref+s3://teste-s3fs/test.json?profile=homolog' | vals eval -f -
s3: successfully retrieved object for key=teste-s3fs/test.json
test_plain_json: |
  {
  	"key": 123
  }

$ echo 'test_json_key: ref+s3://teste-s3fs/test.json?profile=homolog#/key' | vals eval -f -
s3: successfully retrieved object for key=teste-s3fs/test.json
expand s3://teste-s3fs/test.json?profile=homolog#/key: no value found for key key

$ echo 'test_yaml_key: ref+s3://teste-s3fs/test.yml?profile=homolog#/key' | vals eval -f -
s3: successfully retrieved object for key=teste-s3fs/test.yml
expand s3://teste-s3fs/test.yml?profile=homolog#/key: no value found for key key

# changed the values to text

$ echo 'test_yaml_key: ref+s3://teste-s3fs/test.yml?profile=homolog#key' | vals eval -f -
s3: successfully retrieved object for key=teste-s3fs/test.yml
test_yaml_key: "456"
$ echo 'test_json_key: ref+s3://teste-s3fs/test.json?profile=homolog#/key' | vals eval -f -
s3: successfully retrieved object for key=teste-s3fs/test.json
test_json_key: "123"

In my example back when this ticket was open, I was looking to get the JSON version of whatever was in the key..

I guess this is just not a feature, as desgined.. but would it make sense? 😅

thanks!!

[mumoshu]

Triaging an issue almost an year later 🤦
Thanks for your detailed report @pimguilherme! I consider this as a bug- not sure what's causing it but it isn't what I expected.

If anyone is still reading this issue and interested in contributing a patch- PR is always welcomed!