TerraformCloud private registry integration not working using terraformrc

[nalshamaajc]

What happened?

I'm trying to deploy a workspace resource from a module hosted on terraform cloud

How can we reproduce it?

Create terraform secret to be used by the ProviderConfig

kubectl -n crossplane-system create secret generic terraformrc --from-file=.terraformrc
## File Content
credentials "app.terraform.io" {
  token = "XXXXXXXTerraformCloudTOKENXXXXXXXX"
}

Create ProviderConfig

cat <<EOF | kubectl replace --force -f -
apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
    name: terraform
spec:
  credentials:
  - filename: .terraformrc 
    source: Secret
    secretRef:
      namespace: crossplane-system 
      name: terraformrc
      key: .terraformrc
  configuration: |
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4"
        }
      }

      cloud {
        organization = "ORGNAME"
        workspaces {
          tags = ["tags"]
        }
      }
    }
    provider "aws" {
      region = region
    
      assume_role {
        role_arn = arn value
      }

      default_tags {
        tags = {
          passing tags = tags
          ...
        }
      }
    } 
EOF

Create the configmap that holds the values for the variables needed by the remote module

kubectl create configmap terraform --from-file=val-cm-dev-p01-crossplane.tfvars

Create the workspace

cat <<EOF | kubectl replace --force -f -
apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  annotations:
    crossplane.io/external-name: tf_crossplane
    meta.upbound.io/example-id: tf/v1beta1/workspace
  name: test-remote
spec:
  deletionPolicy: Delete 
  providerConfigRef:
    name: terraform
  forProvider:
    module: app.terraform.io/ORG/module/aws # I also tried adding https:// at the beginning
    source: Remote
    varFiles:
      - configMapKeyRef:
          key: val-cm-dev-p01-crossplane.tfvars
          name: terraform
          namespace: default
        source: ConfigMapKey
  writeConnectionSecretToRef:
    name: terraform-workspace-00-test-remote
    namespace: default
EOF

What environment did it happen in?

• Crossplane Version: v1.14.0
• Provider Version: provider-terraform:v0.11.0
• Kubernetes Version: Client Version: v1.28.3 / Server Version: v1.25.15-eks-4f4795d
• Kubernetes Distribution: EKS

Expected Behavior

Pull the module and deploy it as a workspace resource.

Current Behavior

getting errors like the below

  Warning  CannotConnectToProvider  102s (x12 over 12m)  managed/workspace.tf.upbound.io  cannot get remote Terraform module: error downloading 'file:///tf/d08392f5-038b-4639-b7de-3458439bc57d/app.terraform.io/org/module/aws?ref=v3.3.0': source path error: stat /tf/d08392f5-038b-4639-b7de-3458439bc57d/app.terraform.io/org/module/aws: no such file or directory

Adding https:// to the module URL returned the below error

    Module:      https://app.terraform.io/ORG/MODULE/aws/3.3.0'
    Source:      Remote
and got this error

Status:
  At Provider:
  Conditions:
    Last Transition Time:  2023-11-21T10:17:09Z
    Message:               connect failed: cannot get remote Terraform module: error downloading 'https://app.terraform.io/ORG/MODULE/aws/3.3.0': bad response code: 404
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
Events:
  Type     Reason                   Age              From                             Message
  ----     ------                   ----             ----                             -------
  Warning  CannotConnectToProvider  1s (x4 over 9s)  managed/workspace.tf.

[nalshamaajc]

Is this the right way to use terraform cloud as a backend?

[nalshamaajc]

I switched to using Github as the source of the module while using terraform cloud workspaces as a backend. I passed the token credentials in a .terraformrc secret.

    Namespace:  default
Status:
  At Provider:
  Conditions:
    Last Transition Time:  2023-11-24T18:02:24Z
    Message:               connect failed: cannot select Terraform workspace: Terraform encountered an error. Summary: Required token could not be found. To see the full error run: echo "H4sIAAAAAAAA/zyMwa3DMAzF7priTZABcv8LZAP9WHaN2nqpIKPrFwGKHkmAlL8Ixo7DXquHFSSf5ji5RoEz8W+oXF5EjuXIx41j8N294eSc6neDZm6hadDvoTKg17WlRWhlzK1zFwD4GQy27iIfAAAA//8BAAD//8xxXZSEAAAA" | base64 -d | gunzip
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
Events:
  Type     Reason                   Age   From                             Message
  ----     ------                   ----  ----                             -------
  Warning  CannotConnectToProvider  9s    managed/workspace.tf.upbound.io  cannot select Terraform workspace: Terraform encountered an error. Summary: Required token could not be found. To see the full error run: echo "H4sIAAAAAAAA/zyMwa3DMAzF7priTZABcv8LZAP9WHaN2nqpIKPrFwGKHkmAlL8Ixo7DXquHFSSf5ji5RoEz8W+oXF5EjuXIx41j8N294eSc6neDZm6hadDvoTKg17WlRWhlzK1zFwD4GQy27iIfAAAA//8BAAD//8xxXZSEAAAA" | base64 -d | gunzip

$ echo "H4sIAAAAAAAA/zyMwa3DMAzF7priTZABcv8LZAP9WHaN2nqpIKPrFwGKHkmAlL8Ixo7DXquHFSSf5ji5RoEz8W+oXF5EjuXIx41j8N294eSc6neDZm6hadDvoTKg17WlRWhlzK1zFwD4GQy27iIfAAAA//8BAAD//8xxXZSEAAAA" | base64 -d | gunzip

Error: Required token could not be found

Run the following command to generate a token for app.terraform.io:
    terraform login

• Updated ProviderConfig *

apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
    name: terraform
spec:
  credentials:
  - filename: .terraformrc
    source: Secret
    secretRef:
      namespace: crossplane-system 
      name: terraformrc
      key: .terraformrc
  - filename: .git-credentials
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: git-credentials
      key: .git-credentials
  configuration: |
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4"
        }
      }
      backend "remote" {
        organization = "ORG"
        hostname     = "app.terraform.io"
        workspaces {
          name = "dev-us-west-2-p01-crossplane"
        }
      }
    }
    provider "aws" {
      region = us-west-2
    
      assume_role {
        role_arn = format("arn:aws:iam::%s:role/crossplane", xxxxxxxxxxxxx)
      }

      default_tags {
        tags = {
          Source             = format("ORG/infra/terraform/%s/%s/%s/foundation", dev, us-west-2, p01-crossplane)
          Team               = "devops"
          CostOrg            = "engineering"
          ProductLine        = "shared"
          CrossplaneResource = "True"
          Env                = dev
        }
      }
    } 

[ytsarev]

I never tested terraform cloud integration with provider-terraform, but it looks like we need to propagate the token in some supported manner, e.g. https://developer.hashicorp.com/terraform/cli/config/config-file#environment-variable-credentials

[nalshamaajc]

Thanks @ytsarev

I managed to make it work by adding the token in the providerconfig, but it seems that terraform workspaces as a backend are not supported right?

  configuration: |
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4"
        }
      }
      backend "remote" {
        organization = "ORG"
        hostname     = "app.terraform.io"
        token = "XXXXXXXXXXXXXXTOKENXXXXXXXXXXXXXXX"
        workspaces {
          name = "p01-crossplane-foundation"
        }
      }

  Type     Reason                   Age                   From                             Message
  ----     ------                   ----                  ----                             -------
  Warning  CannotConnectToProvider  3m56s (x45 over 54m)  managed/workspace.tf.upbound.io  cannot select Terraform workspace: Terraform encountered an error. Summary: . To see the full error run: echo "H4sIAAAAAAAA/wTA0Q2EMAwD0P+bwnPcAOwRNaaqgKZKXLE+77Bx06FAp9BinqPvpGPaQ0fJxPrjjbxqWWNhhlB7rUjRfx8AAAD//wEAAP//Bc+iCkAAAAA=" | base64 -d | gunzip

$echo "H4sIAAAAAAAA/wTA0Q2EMAwD0P+bwnPcAOwRNaaqgKZKXLE+77Bx06FAp9BinqPvpGPaQ0fJxPrjjbxqWWNhhlB7rUjRfx8AAAD//wEAAP//Bc+iCkAAAAA=" | base64 -d | gunzip
Failed to get configured named states: workspaces not supported

[ytsarev]

Everything that is supported by the standard open source terraform CLI(current version included https://github.com/upbound/provider-terraform/blob/main/cluster/images/provider-terraform/Dockerfile#L6) , should be supported as well.

I would investigate standard terraform issues like hashicorp/terraform#23076 (comment) to proceed.

[nalshamaajc]

@ytsarev the Cloud block should be supported, but even after doing more modifications to narrow it down I still got the below result workspaces not supported. could I be missing something here? Or is it that workspaces as a state backend are not supported.

PS: the workspace was already created on terraform cloud

spec:
  credentials:
  - filename: .git-credentials 
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: git-credentials
      key: .git-credentials
  configuration: |
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4"
        }
      }
      cloud {
        organization = "ORG"
        hostname     = "app.terraform.io"
        token = "XXXXXTOKENXXXX"
        workspaces {
          name = "crossplane-workspace"
        }
      }
    }

I think this this comment is related to what you posted earlier but gives a bit more context.

Next I tried using workspacetags and I'm stuck at making it use the actual workspace instead of the defualt. Where I'm trying to pass the value of the workspace as an environment variable TF_WORKSPACE, but I don't think the Workspace object supports this.

ProviderConfig

  configuration: |
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4"
        }
      }
      cloud {
        organization = "ORG"
        hostname     = "app.terraform.io"
        token = "XXXXXXXXXXXXXXXXXXXXTOKENXXXXXXXXXXXXXXXXXXXXXXX"
        workspaces {
           tags = ["tag1", "tag2", "tag3", "tag4", "tag5"]
        }
      }
    }

Error Message

Status:
  At Provider:
  Conditions:
    Last Transition Time:  2023-11-26T11:08:48Z
    Message:               connect failed: cannot initialize Terraform configuration: Terraform encountered an error. Summary: Currently selected workspace "default" does not exist. To see the full error run: echo "H4sIAAAAAAAA/2zMMU7DQBCF4X5O8eQDmN5tRIFEGUS9sd/GK9Yz1uxYgZYzkKMg7uOTIFIC/f/9x5lOJCeaLcTqdqpcGi4lZsRMHB4fMJrmct48RTEdZL9+yf7xjufkWvQ84EnTqRJhsJX6lyCXyh9yY8f/rrcEKdD1d33QPWXzxccOk7FBLcDX0qKX/fopIvfu5gMOmzs16hsaK8fghIv5S1vTSHQTc9pq/H6IyDcAAAD//wEAAP//7j2vjfcAAAA=" | base64 -d | gunzip
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
Events:
  Type     Reason                         Age                From                             Message
  ----     ------                         ----               ----                             -------
  Warning  CannotObserveExternalResource  97s                managed/workspace.tf.upbound.io  cannot diff (i.e. plan) Terraform configuration: Terraform encountered an error. Summary: Variables not allowed. To see the full error run: echo "H4sIAAAAAAAA/7zQQWrDMBCF4b1O8S6gEDuLgMDLXqHbMrFfWhdZY2Zkh96+VBSyLhS/A8z/MeHFTC3hVWyWW6ajaIXkrA9OIQBaMJq6r1kK42q6zxMtVprJXW2J51O972KOPBeiSwFAl2B8n7UAAzaPD3qNfQjPyiJfrXQjNueEDxpP4b81fdP0CSw7fjZg4n6849Icl4TV9JNjxYD13MXnieNJ10a6JgBj3rzS3oosxO/ao+Jfkd8AAAD//wEAAP///MTLMk4CAAA=" | base64 -d | gunzip
  Warning  CannotConnectToProvider        29s (x5 over 89s)  managed/workspace.tf.upbound.io  cannot initialize Terraform configuration: Terraform encountered an error. Summary: Currently selected workspace "default" does not exist. To see the full error run: echo "H4sIAAAAAAAA/2zMMU7DQBCF4X5O8eQDmN5tRIFEGUS9sd/GK9Yz1uxYgZYzkKMg7uOTIFIC/f/9x5lOJCeaLcTqdqpcGi4lZsRMHB4fMJrmct48RTEdZL9+yf7xjufkWvQ84EnTqRJhsJX6lyCXyh9yY8f/rrcEKdD1d33QPWXzxccOk7FBLcDX0qKX/fopIvfu5gMOmzs16hsaK8fghIv5S1vTSHQTc9pq/H6IyDcAAAD//wEAAP//7j2vjfcAAAA=" | base64 -d | gunzip
echo "H4sIAAAAAAAA/2zMMU7DQBCF4X5O8eQDmN5tRIFEGUS9sd/GK9Yz1uxYgZYzkKMg7uOTIFIC/f/9x5lOJCeaLcTqdqpcGi4lZsRMHB4fMJrmct48RTEdZL9+yf7xjufkWvQ84EnTqRJhsJX6lyCXyh9yY8f/rrcEKdD1d33QPWXzxccOk7FBLcDX0qKX/fopIvfu5gMOmzs16hsaK8fghIv5S1vTSHQTc9pq/H6IyDcAAAD//wEAAP//7j2vjfcAAAA=" | base64 -d | gunzip
There are some problems with the CLI configuration:
╷
│ Warning: Unable to open CLI configuration file
│
│ The CLI configuration file at "./.terraformrc" does not exist.
╵


Error: Currently selected workspace "default" does not exist

[nalshamaajc]

@ytsarev I tried to use a S3 bucket as the backend and it seems that the provider wasn't using the role I passed to the runtimeConfig noting that I tried the same configuration for the provider-aws-ec2 and it was able to use the IRSA arn I passed and create the needed aws resources.

ProviderConfig

  configuration: |
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4"
        }
      }
      backend "s3" {
        bucket         = "bucket-dev-terraform-state"
        key            = "crossplane/foundation/foundation.tfstate"
        region         = "us-west-2"
        dynamodb_table = "crossplane-terraform-lock"
      }  

Provider / DeploymentRuntimeConfig

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: terraform-runtimeconfig
spec:
  serviceAccountTemplate:
    metadata:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXX:role/crossplane
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-terraform
spec:
  package: xpkg.upbound.io/upbound/provider-terraform:v0.11.0
  runtimeConfigRef:
    name: terraform-runtimeconfig

Yet I still get this ERROR

Status:
  At Provider:
  Conditions:
    Last Transition Time:  2023-11-26T13:23:52Z
    Message:               connect failed: cannot initialize Terraform configuration: Terraform encountered an error. Summary: error configuring S3 Backend: no valid credential sources for S3 Backend found.. To see the full error run: echo "H4sIAAAAAAAA/0yPTW7jMAyF16NT8ADjn7EzcUa7DJpFN0XRLrKWJdoWaospSSVIT1+4QdGsCBD8+L5nDszEFnAd4CkNccwc0wivLfx3/g1TsJAIzm6OATxjwKTRzSCU2aPAQHx3CwPlFEpjnmd0giCIMKmexFbV5XIpFZndQLyUkapAXqrZpTG7EStB1ZhGqfrbK6mkLSddZrNGLMQIMa2o00gJXE9Z4cR0jmH1/VGT0nzXOmL/+LXV64HZwuDijAGUgFE54hnvOeNdFgzQXy3svUeRB0wRg4UnUnBZJ+L4ceNPyKsLiIrdi+QFX2jGY9TpLtP8EnWaBTwFtLCp29/A+J5RFGKw0IUBm3qHRfu3q4tNt8Wib5tQ/Bu2f+q+65pd0xhjzCcAAAD//wEAAP//uNz3rqYBAAA=" | base64 -d | gunzip
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
Events:
  Type     Reason                   Age                  From                             Message
  ----     ------                   ----                 ----                             -------
  Warning  CannotConnectToProvider  54s (x4 over 4m13s)  managed/workspace.tf.upbound.io  (combined from similar events): cannot initialize Terraform configuration: Terraform encountered an error. Summary: error configuring S3 Backend: no valid credential sources for S3 Backend found.. To see the full error run: echo "H4sIAAAAAAAA/0yPTW7jMAyF16NT8ADjn7EzcUa7DJpFN0XRLrKWJdoWaospSSVIT1+4QdGsCBD8+L5nDszEFnAd4CkNccwc0wivLfx3/g1TsJAIzm6OATxjwKTRzSCU2aPAQHx3CwPlFEpjnmd0giCIMKmexFbV5XIpFZndQLyUkapAXqrZpTG7EStB1ZhGqfrbK6mkLSddZrNGLMQIMa2o00gJXE9Z4cR0jmH1/VGT0nzXOmL/+LXV64HZwuDijAGUgFE54hnvOeNdFgzQXy3svUeRB0wRg4UnUnBZJ+L4ceNPyKsLiIrdi+QFX2jGY9TpLtP8EnWaBTwFtLCp29/A+J5RFGKw0IUBm3qHRfu3q4tNt8Wib5tQ/Bu2f+q+65pd0xhjzCcAAAD//wEAAP//uNz3rqYBAAA=" | base64 -d | gunzip
echo "H4sIAAAAAAAA/0yPTW7jMAyF16NT8ADjn7EzcUa7DJpFN0XRLrKWJdoWaospSSVIT1+4QdGsCBD8+L5nDszEFnAd4CkNccwc0wivLfx3/g1TsJAIzm6OATxjwKTRzSCU2aPAQHx3CwPlFEpjnmd0giCIMKmexFbV5XIpFZndQLyUkapAXqrZpTG7EStB1ZhGqfrbK6mkLSddZrNGLMQIMa2o00gJXE9Z4cR0jmH1/VGT0nzXOmL/+LXV64HZwuDijAGUgFE54hnvOeNdFgzQXy3svUeRB0wRg4UnUnBZJ+L4ceNPyKsLiIrdi+QFX2jGY9TpLtP8EnWaBTwFtLCp29/A+J5RFGKw0IUBm3qHRfu3q4tNt8Wib5tQ/Bu2f+q+65pd0xhjzCcAAAD//wEAAP//uNz3rqYBAAA=" | base64 -d | gunzip

Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.

Please see https://www.terraform.io/docs/language/settings/backends/s3.html
for more information about providing credentials.

Error: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
	status code: 403, request id: 7dfe208e-3570-476e-b32d-9f610b772822

PS: IRSA works for provider-aws-ec2

[nalshamaajc]

I tried using the kuberenetes as a backend. Yet still got an error on assuming the needed role.

spec:
  credentials:
  - filename: aws.json
    source: None
  - filename: .terraformrc # use exactly this filename by convention
    source: Secret
    secretRef:
      namespace: crossplane-system #upbound-system
      name: terraformrc
      key: .terraformrc
  - filename: .git-credentials # use exactly this filename
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: git-credentials
      key: .git-credentials
  configuration: |
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4"
        }
      }
      backend "kubernetes" {
        secret_suffix     = "providerconfig-default"
        namespace         = "crossplane-system"
        in_cluster_config = true
      }
    }

I validated that the ServiceAccount user is assigned to the right role-ARN

kubectl describe sa provider-terraform-bb61cbfc5270 -n crossplane-system
Name:                provider-terraform-bb61cbfc5270
Namespace:           crossplane-system
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/crossplane
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

I checked the OIDC trust policy and it was valid, I also tried creating a resource using the provider-aws-ec2 and it was created successfully. Below is the aws policy condition.

            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.us-west-2.amazonaws.com/id/044A643CAAABC42964BFB2DD152810D6:sub": [
                        "system:serviceaccount:crossplane-system:provider-aws-*",
                        "system:serviceaccount:crossplane-system:provider-terraform" # Also tested provider-terraform-*
                    ]
                }

echo "H4sIAAAAAAAA/4yTMW/bSBCFe/6KB1d3B1EiLYmithPOKVwkCGwFKYIUK+7IXIDc2cwsbST588HKUhwYLlKwGXK+NzPvsXgnwmLQcTj6h0l8eMCeROyRZcTu8z0+Cj96R2Jwu3uPOx4I/1gJxj6p8XY0pq2qZbuq22W9qY3wQItOWDUONlDpRe2/6GwInHAgWNVpJDcvin1PQrD5QZjGAwn4iMiq/jAQOjspaS6l3itKpJ4wsiZ0PI4ccqcpgP+w7wmdkKOQvB0Uk5KDD2BxJEh81jwB8ngnTR8e7eDdmwDHyNP29pFgYxSO4m0iRJLRq3oOb1B/k04Su7sP8HrCPOsU+ZTnW3MksSljKBdwv7832J1w+bwz9ClFhZBGDkqXr5JNk/7PjgxW1XKGO/o2kabbG4PrqrHr7aYp26Zx5cq129KututyvenW20NzXTeum8FGf2btuo5Ubyh4cgafNLt78VSTvvL07Fn5lreLelPVVb2ul9tq3TZNs63ay+Z2Sj2L/0Eu3yuS5EwZZIGXbcEhb8qTdPQyxN8FqygK4MmnHvEc0i9XQg9ek3yfp0uK554XvdXedyxxYZ/06uusQNb9A3cBlM8/wjwdMfhAqJtZztLlNa5yf85d3ZhXVfwsil8AAAD//wEAAP//xnzGSE8DAAA=" | base64 -d | gunzip

Error: configuring Terraform AWS Provider: IAM Role (arn:aws:iam::xxxxxxxxxxx:role/crossplane) cannot be assumed.

There are a number of possible causes of this - the most common are:
  * The credentials used in order to assume the role are invalid
  * The credentials do not have appropriate permission to assume the role
  * The role ARN is not valid

AWS Error: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 206a5976-866d-4d89-a495-57c59b6216dc, api error AccessDenied: User: arn:aws:sts::xxxxxxxxxxx:assumed-role/crossplane/1701015139058666908 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::xxxxxxxxxxxx:role/crossplane


  with provider["registry.terraform.io/hashicorp/aws"],
  on crossplane-provider-config.tf line 16, in provider "aws":
  16: provider "aws" {

[nalshamaajc]

It feels like something is not right, the docs here mentioned that the process is similar to what we have in provider-aws. But none of the source options provided there work to force the terraform provider to use a specific role-arn instead of a file, secret, or configMap.

[bobh66]

Have you verified that the AWS environment variables are set in the pod as a result of the annotation?

[nalshamaajc]

@bobh66 They are

$ kubectl exec -it provider-terraform-bb61cbfc5270-867bb7f685-bbs9h -n crossplane-system -- printenv | grep -i aws

AWS_STS_REGIONAL_ENDPOINTS=regional
AWS_DEFAULT_REGION=us-west-2
AWS_ROLE_ARN=arn:aws:iam::xxxxxxxxxxx:role/crossplane
AWS_REGION=us-west-2
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

[nalshamaajc]

I tested older version v0.7.0 and v0.2.0. So far all the tests failed.

It only worked after I passed a user token and used AWS s3 as a backend. This is not ideal in my case since the project that I'm working on uses IRSA and terraform cloud as a backend.

ProviderConfig

  credentials:
  - filename: .terraformrc 
    source: Secret
    secretRef:
      namespace: crossplane-system 
      name: terraformrc
      key: .terraformrc
  - filename: aws-credentials # <---------- AWS Credentials passed as secrets
    source: Secret
    secretRef:
      name: aws-credentials
      namespace: crossplane-system
      key: credentials 
  - filename: .git-credentials
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: git-credentials
      key: .git-credentials
  configuration: |
    provider "aws" {
      region = "us-west-2"
      shared_credentials_files = ["aws-credentials"] # <------------ Credentials file instead of role-arn

[bobh66]

it sounds like a problem with the Assume Role configuration - are you sure that the provider-terraform service account name is right? You might add "" to the end of the service account name in the AssumeRole definition so that any provider-terraform service account can assume the role.

[nalshamaajc]

Yeah the name is right @bobh66, seems I figured it out. Looks like the issue was caused by the assume_role block. Removing it solved the problem, yet I'm still curious why this block was causing the issue with the role used here being the same for IRSA.

    provider "aws" {
      region = "us-west-2"
   
      assume_role {
        role_arn = "arn:aws:iam::xxxxxxxx:role/crossplane"
      }

[bobh66]

I have seen problems with a role trying to assume itself - that used to be allowed by default but it was changed some time ago to require specific provisioning to allow it to work.

Glad you got it working!

[nalshamaajc]

Thank you @bobh66

I hope I get the same luck with the terraformCloud backend issue.

[nalshamaajc]

More testing was done and I noticed that the workspace was creating a new workspace on other than the one I provided which matched the external-name annotation in the workspace object.

apiVersion: tf.upbound.io/v1beta1
kind: Workspace
metadata:
  annotations:
    crossplane.io/external-name: "crossplane_test"
    meta.upbound.io/example-id: tf/v1beta1/workspace

I changed the value of the external-name annotation to match the workspace I created and the "workspace not supported" error mentioned in this comment was gone.

I was now left with a different error which was complaining about the absence of a state file in this backend.

echo "H4sIAAAAAAAA/0yOMU7FQAxE+73Cb4YD8AXiHhRAl8psvFlLWTt4vYq4PUoUpF+48cy8mdv09treDT0oGEVWxk4dxYbOTyl9nu9GSgs31kC21kjnDuefIc6gh+gdH0MRVfq/L4mCMItzDvNf7JWd8cXuVMwbKnV8Myt8KMwxOiMq4/mCrrSkMGwmGqdwcRF2NG+cpUi+JqyWKcT0fpte2nHpDwAA//8BAAD//yMXozveAAAA" | base64 -d | gunzip
No state file was found!

State management commands require a state file. Run this command
in a directory where Terraform has been run or use the -state flag
to point the command to a specific state location.

I will be discussing this error in another issue.

I will close this issue for the time being since the main problems (using IRSA, and TerraformCloud as a backend) were resolved