TFC Token not being propagated through .terraformrc

[nalshamaajc]

What happened?

Trying to use .terraformrc to use modules from terraform cloud registry

getting the below error

    Namespace:  default
Status:
  At Provider:
  Conditions:
    Last Transition Time:  2023-11-24T18:02:24Z
    Message:               connect failed: cannot select Terraform workspace: Terraform encountered an error. Summary: Required token could not be found. To see the full error run: echo "H4sIAAAAAAAA/zyMwa3DMAzF7priTZABcv8LZAP9WHaN2nqpIKPrFwGKHkmAlL8Ixo7DXquHFSSf5ji5RoEz8W+oXF5EjuXIx41j8N294eSc6neDZm6hadDvoTKg17WlRWhlzK1zFwD4GQy27iIfAAAA//8BAAD//8xxXZSEAAAA" | base64 -d | gunzip
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
Events:
  Type     Reason                   Age   From                             Message
  ----     ------                   ----  ----                             -------
  Warning  CannotConnectToProvider  9s    managed/workspace.tf.upbound.io  cannot select Terraform workspace: Terraform encountered an error. Summary: Required token could not be found. To see the full error run: echo "H4sIAAAAAAAA/zyMwa3DMAzF7priTZABcv8LZAP9WHaN2nqpIKPrFwGKHkmAlL8Ixo7DXquHFSSf5ji5RoEz8W+oXF5EjuXIx41j8N294eSc6neDZm6hadDvoTKg17WlRWhlzK1zFwD4GQy27iIfAAAA//8BAAD//8xxXZSEAAAA" | base64 -d | gunzip

$ echo "H4sIAAAAAAAA/zyMwa3DMAzF7priTZABcv8LZAP9WHaN2nqpIKPrFwGKHkmAlL8Ixo7DXquHFSSf5ji5RoEz8W+oXF5EjuXIx41j8N294eSc6neDZm6hadDvoTKg17WlRWhlzK1zFwD4GQy27iIfAAAA//8BAAD//8xxXZSEAAAA" | base64 -d | gunzip

Error: Required token could not be found

Run the following command to generate a token for app.terraform.io:
    terraform login

More details here

How can we reproduce it?

Deploy a terraform provider and try to use a module from a private registry in a provider-terraform workspace

ProviderConfig

apiVersion: tf.upbound.io/v1beta1
kind: ProviderConfig
metadata:
    name: terraform
spec:
  credentials:
  - filename: .terraformrc
    source: Secret
    secretRef:
      namespace: crossplane-system 
      name: terraformrc
      key: .terraformrc
  configuration: |
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4"
        }
      }
      backend "remote" {
        organization = "ORG"
        hostname     = "app.terraform.io"
        workspaces {
          name = "dev-us-west-2-p01-crossplane"
        }
      }
    }
    provider "aws" {
      region = us-west-2
    
      assume_role {
        role_arn = format("arn:aws:iam::%s:role/crossplane", xxxxxxxxxxxxx)
      }

      default_tags {
        tags = {
          Source             = format("ORG/infra/terraform/%s/%s/%s/foundation", dev, us-west-2, p01-crossplane)
          Team               = "tag1"
          CostOrg            = "tag2"
          ProductLine        = "tag3"
          CrossplaneResource = "True"
          Env                = dev
        }
      }
    } 

Workaround
I managed to make it work by adding the token in the providerconfig, but it seems that terraform workspaces as a backend are not supported right?

  configuration: |
    terraform {
      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 4"
        }
      }
      backend "remote" {
        organization = "ORG"
        hostname     = "app.terraform.io"
        token = "XXXXXXXXXTOKENXXXXXXXXXXXX"
        workspaces {
          name = "p01-crossplane-foundation"
        }
      }

  Type     Reason                   Age                   From                             Message
  ----     ------                   ----                  ----                             -------
  Warning  CannotConnectToProvider  3m56s (x45 over 54m)  managed/workspace.tf.upbound.io  cannot select Terraform workspace: Terraform encountered an error. Summary: . To see the full error run: echo "H4sIAAAAAAAA/wTA0Q2EMAwD0P+bwnPcAOwRNaaqgKZKXLE+77Bx06FAp9BinqPvpGPaQ0fJxPrjjbxqWWNhhlB7rUjRfx8AAAD//wEAAP//Bc+iCkAAAAA=" | base64 -d | gunzip

$echo "H4sIAAAAAAAA/wTA0Q2EMAwD0P+bwnPcAOwRNaaqgKZKXLE+77Bx06FAp9BinqPvpGPaQ0fJxPrjjbxqWWNhhlB7rUjRfx8AAAD//wEAAP//Bc+iCkAAAAA=" | base64 -d | gunzip
Failed to get configured named states: workspaces not supported

What environment did it happen in?

• Crossplane Version: v1.14.0
• Provider Version: provider-terraform:v0.11.0
• Kubernetes Version: Client Version: v1.28.3 / Server Version: v1.25.15-eks-4f4795d
• Kubernetes Distribution: EKS

[ytsarev]

Can you show the redacted contents of secret from?

   secretRef:
      namespace: crossplane-system 
      name: terraformrc
      key: .terraformrc

Is it configured according to https://developer.hashicorp.com/terraform/cli/config/config-file#credentials ?

[nalshamaajc]

@ytsarev yes and yes. I'm using this format for TF Cloud when using it via CLI and it works.

credentials "app.terraform.io" {
  token = "XXXXXXXTOKENXXXXXXXX"
}