Terraform not using STS regional endpoint

[ghost]

This issue was originally opened by @sanvipy as hashicorp/terraform#24592. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.12.20
+ provider.aws v2.35.0

Terraform Configuration Files

provider "aws" {
  region      = "ap-southeast-1"

 assume_role {
   role_arn = "role_arn"
}
  endpoints {
    sts = "sts.ap-southeast-1.amazonaws.com"
  }
}

~/.aws/config
[default]
region      = ap-southeast-1
sts_regional_endpoints = regional

>echo $AWS_STS_REGIONAL_ENDPOINT
>regional

Debug Output

2020/04/08 09:28:39 [INFO] Terraform version: 0.12.24
2020/04/08 09:28:39 [INFO] Go runtime version: go1.12.13
2020/04/08 09:28:39 [INFO] CLI args: []string{"/usr/bin/terraform", "plan", "--var-file=uat.tfvars"}
2020/04/08 09:28:39 [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2020/04/08 09:28:39 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/04/08 09:28:39 [INFO] CLI command args: []string{"plan", "--var-file=uat.tfvars"}
2020/04/08 09:28:39 [TRACE] Meta.Backend: built configuration for "s3" backend with hash value 704415183
2020/04/08 09:28:39 [TRACE] Preserving existing state lineage "b273c1fd-f9bd-3803-0889-a7d64efe1dcf"
2020/04/08 09:28:39 [TRACE] Preserving existing state lineage "b273c1fd-f9bd-3803-0889-a7d64efe1dcf"
2020/04/08 09:28:39 [TRACE] Meta.Backend: working directory was previously initialized for "s3" backend
2020/04/08 09:28:39 [TRACE] Meta.Backend: using already-initialized, unchanged "s3" backend configuration
2020/04/08 09:28:39 [INFO] Setting AWS metadata API timeout to 100ms
2020/04/08 09:28:39 [INFO] AWS EC2 instance detected via default metadata API endpoint, EC2RoleProvider added to the auth chain
2020/04/08 09:28:39 [INFO] AWS Auth provider used: "EC2RoleProvider"
2020/04/08 09:28:39 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2020/04/08 09:28:39 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: sts.amazonaws.com
User-Agent: aws-sdk-go/1.25.3 (go1.12.13; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.24
Content-Length: 43
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXX/20200408/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20200408T012839Z
X-Amz-Security-Token: XXXXXXXXXXXXXXXXXXX//////////XXXXXXXXXXXXXXXXXXX//////////XXXXXXXXXXXXXXXXXXX==
Accept-Encoding: gzip

Action=GetCallerIdentity&Version=2011-06-15

2020/04/08 09:28:42 [ERR] Checkpoint error: Get https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=linux&signature=47f21672-90d8-81e4-0605-5c077c2c8166&version=0.12.24: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020/04/08 09:29:09 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, attempt 0/5, error RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: dial tcp 52.46.134.192:443: i/o timeout
2020/04/08 09:29:09 [DEBUG] [aws-sdk-go] DEBUG: Retrying Request sts/GetCallerIdentity, attempt 1
2020/04/08 09:29:09 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: sts.amazonaws.com
User-Agent: aws-sdk-go/1.25.3 (go1.12.13; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.24
Content-Length: 43
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXX/20200408/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20200408T012909Z
X-Amz-Security-Token: XXXXXXXXXXXXXXXXXXX//////////XXXXXXXXXXXXXXXXXXX//////////XXXXXXXXXXXXXXXXXXX==
Accept-Encoding: gzip

Action=GetCallerIdentity&Version=2011-06-15

2020/04/08 09:29:39 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, attempt 1/5, error RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: dial tcp 54.239.29.25:443: i/o timeout
2020/04/08 09:29:39 [DEBUG] [aws-sdk-go] DEBUG: Retrying Request sts/GetCallerIdentity, attempt 2
2020/04/08 09:29:39 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:

Crash Output

Expected Behavior

Attempt to connect to STS regional endpoint- https://sts.ap-southeast-1.amazonaws.com which resolve to private IP of STS VPC endpoint

Actual Behavior

Attempt to connect to STS global endpoint- https://sts.amazonaws.com and its public IP

Steps to Reproduce

terraform plan

Additional Context

• Machine has no internet access
• STS VPC Endpoint created
• aws --region ap-southeast-1 sts get-caller-identity --endpoint-url https://sts.ap-southeast-1.amazonaws.com is successfull from the machine

References

[ewbankkit]

Related:

• Allow configuring token version for global STS endpoint #10707

@sanvipy Thanks for opening this. Could you try setting AWS_STS_REGIONAL_ENDPOINTS=regional in your environment (if possible)?

[sanvipy]

Thanks @ewbankkit , I did try setting both AWS_STS_REGIONAL_ENDPOINTS environment variable and sts_regional_endpoints configuration file options with no luck

[root@xxxxxxxxx xxxxxxxxx]# terraform plan --var-file=uat.tfvars
Error: error using credentials to get account ID: error calling sts:GetCallerIdentity: RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: dial tcp 54.239.29.25:443: i/o timeout

[root@xxxxxxxxx xxxxxxxxx]# echo $AWS_STS_REGIONAL_ENDPOINTS
regional

[joelthompson]

Hey @sanvipy -- can you try setting the AWS_SDK_LOAD_CONFIG=true in your environment?

[ewbankkit]

Many issues in this area will be fixed with #13608. Revisit once that PR is merged.

[bflad]

Hi @sanvipy 👋 Are you also using the Terraform S3 Backend? The debug logs posted seem to indicate that you are. Please note that it has a separate configuration for STS endpoints (if explicitly configuring them) and that many of the unexpected behaviors were resolved in Terraform 0.13.0-beta2 (the ones that will be fixed in the AWS Provider in version 3.0.0). Are you able to see if running that version or later with terraform plan is successful?

[bflad]

The authentication changes applied to the Terraform S3 Backend as part of Terraform CLI 0.13.0-beta2 which should resolve this issue have also been merged into the Terraform AWS Provider and will release with version 3.0.0, likely in the next two weeks. Please follow the v3.0.0 milestone for tracking the progress of that release. If you are still having trouble after updating when its released, please file a new issue. Thanks!

[ghost]

This has been released in version 3.0.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!