Error: invalid character '<' looking for beginning of value

[zzarkouna]

Hi there,

Terraform Version

Terraform v0.12.13

Affected Resource(s)

provider "datadog"

Error Output

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

Error: invalid character '<' looking for beginning of value

on provider.tf line 1, in provider "datadog":
1: provider "datadog" {

Expected Behavior

message error: APP_KEY or API_KEY are wrong
or connection fail to datadog

Actual Behavior

Error: invalid character '<' looking for beginning of value

Steps to Reproduce

1. export wrong DATADOG_API_KEY or DATADOG_APP_KEY or both values are wrong
2. terraform plan

[dohoangkhiem]

I have exactly the same error, but with very different circumstance, in my case terraform apply already executed successfully, but when destroying, the error occurred.

Terraform 0.12.13

data "aws_ssm_parameter" "dd_api_key" {
  name     = var.ssm_dd_api_key_name
  provider = aws.platform
}

data "aws_ssm_parameter" "dd_app_key" {
  name     = var.ssm_dd_app_key_name
  provider = aws.platform
}

provider "datadog" {
  api_key = data.aws_ssm_parameter.dd_api_key.value
  app_key = data.aws_ssm_parameter.dd_app_key.value
}

This happened if there's no Datadog resources get created with the datadog provider (as we have some conditions regarding DD usage), if we have at least 1 DD resource then destroy works.

[dohoangkhiem]

Added:
This does not happen with TF 0.11.14, with the same DD provider version v2.5.0, I'm wondering if this is because of changes in Terraform core?

[bkabrda]

As noted in #364, the part of this issue that's happening with invalid API key has been solved by zorkian/go-datadog-api#282, so we just need to tackle the second part as reported by @dohoangkhiem.

[lakshmigk01]

When is the next provider update. My datadog key are right but I am still getting this issue.

[lakshmigk01]

+1

[lakshmigk01]

When is the next release for the provider ?

[bkabrda]

@lakshmigk01 we did a release 2.6.0 on 20th January, feel free to give that a go.

[johnnyplaydrums]

@bkabrda I can create a separate issue for this but just tested with 2.6.0 and I get an error Invalid or missing credentials provided to the Datadog Provider. Please confirm your API and APP keys are valid and see https://terraform.io/docs/providers/datadog/index.html for more information on providing credentials for the Datadog Provider. My app and api keys are pulled in via an ssm data resource. If I put the keys as strings, the error goes away. If I bump back down to 1.9.0, the error also goes away, but then the Error: invalid character '<' looking for beginning of value error comes back (if no resources are created as mentioned above).

[bkabrda]

@johnnyplaydrums this might be related, so let's keep it in this issue for now. I'd like to ask you to try one thing. Could you set TF_LOG=debug, TF_LOG_PATH=/tmp/tflog and run again while pulling api and app keys from ssm? Then go to /tmp/tflog and search for GET /api/v1/validate. Could you confirm that the api and app keys in headers Dd-Api-Key and Dd-Application-Key are what they're expected to be?

Edit: Additionally, if you could provide the whole RESPONSE block (will be located below the REQUEST block with headers), that would also be helpful. Thanks!

[johnnyplaydrums]

@bkabrda so in doing a terraform destroy on a statefile that has no datadog resources (due to a conditional), and with the keys being pulled in from SSM, the Dd-Api-Key and Dd-Application-Key are unset:

2020-02-05T09:58:41.599-0500 [DEBUG] plugin.terraform-provider-datadog_v2.6.0_x4: ---[ REQUEST ]---------------------------------------
2020-02-05T09:58:41.600-0500 [DEBUG] plugin.terraform-provider-datadog_v2.6.0_x4: GET /api/v1/validate HTTP/1.1
2020-02-05T09:58:41.600-0500 [DEBUG] plugin.terraform-provider-datadog_v2.6.0_x4: Host: api.datadoghq.com
2020-02-05T09:58:41.600-0500 [DEBUG] plugin.terraform-provider-datadog_v2.6.0_x4: User-Agent: Go-http-client/1.1
2020-02-05T09:58:41.600-0500 [DEBUG] plugin.terraform-provider-datadog_v2.6.0_x4: Dd-Api-Key: 
2020-02-05T09:58:41.600-0500 [DEBUG] plugin.terraform-provider-datadog_v2.6.0_x4: Dd-Application-Key: 
2020-02-05T09:58:41.600-0500 [DEBUG] plugin.terraform-provider-datadog_v2.6.0_x4: Accept-Encoding: gzip

With the keys as hard coded strings, I see the headers are set correctly. Any ideas?

[bkabrda]

@johnnyplaydrums that's interesting. I was able to reproduce this using local_file data source and it turns out that this is very likely a Terraform issue - TF doesn't seem to initialize the provider correctly if:

• it's using data source for key values
• terraform state is empty
• there is at least one DD resource defined in the TF config

I opened a bug for this for Terraform core: hashicorp/terraform#24055

I'm going to close this issue now, since the original report was fixed and the followup report is not an issue in Datadog Provider. Please watch the above mentioned issue to see if this gets fixed in Terraform.

The only workaround that I see right now is hardcoding the keys in the TF config file.

[johnnyplaydrums]

@bkabrda thanks for helping me debug and for creating the issue upstream 👍