aws_secretsmanager() function fails to fetch JSON secrets

[durnik-ivo]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Overview of the Issue

I am trying to fetch a JSON secret stored in the AWS SecretsManager. I have found out, that the aws_secretsmanager() function is able to get either text secrets, or JSON secrets in key:value format - only by individual keys.

However, I need to fetch JSONs, that cannot be converted to key:value format and I need to fetch them "as is". Would it be possible to include something like raw=false/true flag, that would not attempt to convert JSONs into key:value format?

Additionally, it would be great if aws_secretsmanager() printed, which secret failed. Currently, it prints only the line of code, which can be problematic, when one is iterating over multiple secrets within a dynamic block with let's say file priovisioner.

Thank you for considering this issue.

Reproduction Steps

1. Load following JSON into the AWS SecretsManager:

   {
     "foo":{
       "bar":"baz"
     }
   }
   

2. Modify included Packer template to fetch your secret
3. Attempt to fetch the secret using provided Packer template by running packer validate .

Packer version

From packer v1.11.0 (Ubuntu 20.04 LTS), packer v1.11.1 (MacOS 14.5 Sonoma)

Simplified Packer Template

packer {
  required_plugins {
    amazon = {
      source  = "github.com/hashicorp/amazon"
      version = "~> 1"
    }
  }
}

locals {
  secret = aws_secretsmanager("/XXX/test.json", null)
}

Operating system and Environment details

Ubuntu 20.04 LTS, MacOS 14.5 Sonoma

Log Fragments and crash.log files

$ PACKER_LOG=1 packer validate .
2024/07/22 11:01:19 [INFO] Packer version: 1.11.0 [go1.21.10 linux amd64]
2024/07/22 11:01:19 [INFO] PACKER_CONFIG env var not set; checking the default config file path
2024/07/22 11:01:19 [INFO] PACKER_CONFIG env var set; attempting to open config file: /home/XXX/.packerconfig
2024/07/22 11:01:19 [WARN] Config file doesn't exist: /home/XXX/.packerconfig
2024/07/22 11:01:19 [INFO] Setting cache directory: /home/XXX/.cache/packer
2024/07/22 11:01:19 [TRACE] listing potential installations for "github.com/hashicorp/amazon" that match "~> 1". plugingetter.ListInstallationsOptions{PluginDirectory:"/home/XXX/.config/packer/plugins", BinaryInstallationOptions:plugingetter.BinaryInstallationOptions{APIVersionMajor:"5", APIVersionMinor:"0", OS:"linux", ARCH:"amd64", Ext:"", Checksummers:[]plugingetter.Checksummer{plugingetter.Checksummer{Type:"sha256", Hash:(*sha256.digest)(0xc0001c4280)}}, ReleasesOnly:false}}
2024/07/22 11:01:20 [TRACE] Found the following "github.com/hashicorp/amazon" installations: [{/home/XXX/.config/packer/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v1.3.0_x5.0_linux_amd64 v1.3.0 x5.0},{/home/XXX/.config/packer/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v1.3.2_x5.0_linux_amd64 v1.3.2 x5.0}]
2024/07/22 11:01:20 [INFO] found external [chroot ebs ebssurrogate ebsvolume instance] builders from amazon plugin
2024/07/22 11:01:20 [INFO] found external [import] post-processors from amazon plugin
2024/07/22 11:01:20 found external [ami parameterstore secretsmanager] datasource from amazon plugin
2024/07/22 11:01:20 [TRACE] listing potential installations for <nil> that match "". plugingetter.ListInstallationsOptions{PluginDirectory:"/home/XXX/.config/packer/plugins", BinaryInstallationOptions:plugingetter.BinaryInstallationOptions{APIVersionMajor:"5", APIVersionMinor:"0", OS:"linux", ARCH:"amd64", Ext:"", Checksummers:[]plugingetter.Checksummer{plugingetter.Checksummer{Type:"sha256", Hash:(*sha256.digest)(0xc000822300)}}, ReleasesOnly:false}}
2024/07/22 11:01:20 [INFO] found external [chroot ebs ebssurrogate ebsvolume instance] builders from amazon plugin
2024/07/22 11:01:20 [INFO] found external [import] post-processors from amazon plugin
2024/07/22 11:01:20 found external [ami parameterstore secretsmanager] datasource from amazon plugin
2024/07/22 11:01:20 [INFO] found external [arm chroot dtl] builders from azure plugin
2024/07/22 11:01:20 found external [dtlartifact] provisioner from azure plugin

Error: Error in function call

  on secrets.pkr.hcl line 11:
  (source code not available)

  on secrets.pkr.hcl line 11:
Call to function "aws_secretsmanager" failed: Unsupported secret value type:
map[string]interface {}.

2024/07/22 11:01:21 [INFO] (telemetry) Finalizing.
  (source code not available)

Call to function "aws_secretsmanager" failed: Unsupported secret value type:
map[string]interface {}.


2024/07/22 11:01:21 waiting for all plugin processes to complete...

[durnik-ivo]

Hi,

I would like to add one more comment to using aws_secretsmanager() function in Packer. Currently, the user experience is really bad as soon as you encounter a secret that cannot be fetched (incorrect name, marked for deletion,...).

The only error you receive is:

Error: Failed preparing provisioner-block "file" ""

  on foo.pkr.hcl line 134:
  (source code not available)

foo.pkr.hcl:138,17-36: Error in function call; Call to function
"aws_secretsmanager" failed: InvalidRequestException: You can't perform this
operation on the secret because it was marked for deletion..

However, in my case, the referenced code block in the output is dynamic with for_each iterating over tens of secrets. The only solution I am left with is to write a custom check or to manually search for the incorrect secret. If the function provided information about which secret ID was problematic, I would greatly appreciate it.

EDIT: You can convert your HCl file into JSON, select your secret IDs and check them using AWS CLI. Something like:

brew intall yj
cat foo.hcl | yj -cj | jq -r '.path.to.secrets.[].id' | while read SM; do echo $SM; aws secretsmanager get-secret-value --secret-id $SM > /dev/null; done