Teleport ArgoCD Configuration

[olawale-ib]

i want to know if this ArgoCD Teleport configuration is alright.

kind: Application
metadata:
  name: teleport-cluster
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: all-clusters
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
  source:
    chart: teleport-cluster
    repoURL: https://charts.releases.teleport.dev
    targetRevision: "15.4.6"
    helm:
      values: |
        clusterName: teleport.example.com
        kubeClusterName: application-dev
        proxyListenerMode: multiplex
        proxyProtocol: "off"
        authentication:
          type: github
          connectorName: github
        auth:
          persistence:
            storageClassName: ebs-sc
          teleportConfig:
            auth_service:
              listen_addr: 0.0.0.0:3025
        storage:
          type: "dynamodb"
          region: "us-west-2"
          table_name: "teleport"
          audit_event_uri: "dynamodb://teleport-event"
          audit_session_uri: "s3://teleport-audit-log"
        sessionRecording: proxy-sync
        tls:
          existingSecretName: teleport-tls
        annotations:
          ingress:
            cert-manager.io/cluster-issuer: letsencrypt-internal-prod
            nginx.ingress.kubernetes.io/backend-protocol: HTTPS
        log:
          format: json
          level: DEBUG
        service:
          type: ClusterIP
        proxy:
          teleportConfig:
            proxy_service:
              web_listen_addr: "0.0.0.0:3080"
              https_keypairs: []
              https_keypairs_reload_interval: 0s
              acme: {}
              trust_x_forwarded_for: true
          highAvailability:
            replicaCount: 1
        ingress:
          enabled: true
        resources:
          requests:
            cpu: 50m
            memory: 300Mi
          limits:
            memory: 300Mi
  destination:
    name: application-dev
    namespace: teleport

Because it seems like there might be an issue with the storage block, there's no item showing in dynamo table.

[olawale-ib]

sessions are not getting uploaded to s3 too

[webvictim]

It looks OK, with a few issues:

1. audit_session_uri is a plural, it should be audit_sessions_uri

2. audit_event_uri is a plural, it should be audit_events_uri

3. Your storage section should be set as a config override:

spec:
  source:
    helm:
      values:
        auth:
          teleportConfig:
            teleport:
              storage:
              ...

This is why your config isn't being applied.

For DynamoDB config examples, see https://goteleport.com/docs/reference/backends/#configuring-the-dynamodb-backend.

3. I don't know why you're manually setting the auth server's listen address to 0.0.0.0:3025?

[olawale-ib]

Thank you very much.
For the auth listen address, that was the configuration I met before joining the team, and it works fine
For the storage section, that was actually the config I did but I got an error and changed to what I posted above

ERROR REPORT: 
Original Error: *awserr.requestError AccessDeniedException: User: arn:aws:sts::<acct id>:assumed-role/eks-node-group-***/i-*** is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-2:<acct id>.:table/teleport because no identity-based policy allows the dynamodb:DescribeTable action status code: 400
Stack Trace:
github.com/gravitational/teleport/lib/backend/dynamo/dynamodbbk.go:750 github.com/gravitational/teleport/lib/backend/dynamo.(*Backend).getTableStatus
github.com/gravitational/teleport/lib/backend/dynamo/dynamodbbk.go:301 github.com/gravitational/teleport/lib/backend/dynamo.New
github.com/gravitational/teleport/lib/backend/dynamo/dynamodbbk.go:56 github.com/gravitational/teleport/lib/backend/dynamo.init.0.func1
github.com/gravitational/teleport/lib/backend/backend.go:125 github.com/gravitational/teleport/lib/backend.New
github.com/gravitational/teleport/lib/service/service.go:5918 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initAuthStorage
github.com/gravitational/teleport/lib/service/service.go:1786 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initAuthService
github.com/gravitational/teleport/lib/service/service.go:1247 github.com/gravitational/teleport/lib/service.NewTeleport
github.com/gravitational/teleport/lib/service/service.go:745 github.com/gravitational/teleport/lib/service.newTeleportProcess
github.com/gravitational/teleport/lib/service/service.go:761 github.com/gravitational/teleport/lib/service.Run
github.com/gravitational/teleport/tool/teleport/common/teleport.go:726
github.com/gravitational/teleport/tool/teleport/common.OnStart
github.com/gravitational/teleport/tool/teleport/common/teleport.go:613 github.com/gravitational/teleport/tool/teleport/common.Run
github.com/gravitational/teleport/tool/teleport/main.go:33 main.main   runtime/proc.go:271 runtime.main
runtime/asm_amd64.s:1695 runtime.goexit
User Message: initialization failed  AccessDeniedException: User: arn:aws:sts::<acct id>:assumed-role/eks-node-group-***/i-*** is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-2:<acct id>:table/teleport because no identity-based policy allows the dynamodb:DescribeTable action status code: 400

I can't attach s3 or dynamo policy to that role (eks-node-group-***) but there is a role (teleport-role) that has the necessary policy attached, if I add the role under serviceAccount, is that what teleport will assume when it tries to access the bucket or table

        auth:
          persistence:
            storageClassName: ebs-sc
          teleportConfig:
            auth_service:
              listen_addr: 0.0.0.0:3025
            storage:
              type: "dynamodb"
              region: "us-west-2"
              table_name: "teleport"
              audit_events_uri: "dynamodb://teleport-event"
              audit_sessions_uri: "s3://teleport-audit-log"
        sessionRecording: proxy-sync
        tls:
          existingSecretName: teleport-cluster-tls
        annotations:
          ingress:
            cert-manager.io/cluster-issuer: letsencrypt-internal-prod
            nginx.ingress.kubernetes.io/backend-protocol: HTTPS
          serviceAccount:
            eks.amazonaws.com/role-arn: "arn:aws:iam::<acct-id>:role/teleport-role"
        log:
          format: json
          level: DEBUG

[webvictim]

if I add the role under serviceAccount, is that what teleport will assume when it tries to access the bucket or table

Yes.

Bear in mind that there are separate Kubernetes ServiceAccounts for the auth and proxy - you need to add the IRSA annotation on the ServiceAccount used by the auth service.

auth:
  annotations:
    serviceAccount:
      eks.amazonaws.com/role-arn: "arn:aws:iam::<acct-id>:role/teleport-role"

You'll also need an appropriate trust policy for AssumeRoleWithWebIdentity set up on the IAM role.

[olawale-ib]

Thanks for the reply.
So there is only 3 ServiceAccount in the teleport namespace: teleport-cluster, teleport-cluster-proxy and default.

tsh kubectl describe serviceaccount teleport-cluster -n teleport
Name:                teleport-cluster
Namespace:           teleport
Labels:              app.kubernetes.io/component=auth
                     app.kubernetes.io/instance=teleport-cluster
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=teleport-cluster
                     app.kubernetes.io/version=15.4.6
                     argocd.argoproj.io/instance=teleport-cluster
                     helm.sh/chart=teleport-cluster-15.4.6
                     teleport.dev/majorVersion=15
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

It was the same when I checked on default and teleport-cluster-proxy with the only diff being component=proxy for teleport-cluster-proxy

About the IAM role, the "teleport-role" has AssumeRoleWithWebIdentity, it was initially created using this doc, then I attached s3 and dynamo policy to it

[webvictim]

teleport-cluster is the correct SA for DynamoDB/S3 IAM policies.

[olawale-ib]

Thank you very much, it worked.
But now I'm facing the same issue as #42814

{"caller":"service/supervisor.go:452","component":"proc:1","level":"debug","message":"Teleport not yet ready: still waiting for InstanceReady","pid":"7.1","timestamp":"2024-08-02T19:46:41Z"}
{"caller":"loglimit/loglimit.go:162","component":"mx:1","dst_addr{"IP":"1***","Port":**,"Zone":""},"level":"warning",
"message":"\nERROR REPORT:\nOriginal Error: x509.UnknownAuthorityError x509:
certificate signed by unknown authority\nStack Trace:\n\tgithub.com/gravitational/teleport/lib/multiplexer/proxyline.go:511
github.com/gravitational/teleport/lib/multiplexer(*ProxyLine).VerifySignature\n\tgithub.com/gravitational/teleport/lib/multiplexer/multipexer.go:565 github.com/gravitational/teleport/lib/multiplexer
(*Mux).detect\n\tgithub.com/gravitational/teleport/lib/multiplexer/multiplexer.go:339 github.com/gravitational/teleport/lib/multiplexer(*Mux).detectAndForward\n\truntime/asm_amd64.s:1695 runtime.goexit\nUser
Message: could not verify PROXY signature for connection 10.23.31.144:57460 -\u0026gt; 10.23.3.115:3025\n\tx509: certificate signed by unknown authority","src_addr" {"IP":"****","Port":****,"Zone":""},"timestamp":"2024-08-02T19:46:44Z"}
{"caller":"multiplexer/tls.go:152","component":"mxtls:1","dst_addr":{"IP":"****","Port":3025,"Zone":""},
"error":"tls: failed to verify certificate: x509: certificate signed by unknown authority","level":"warning","message":"Handshake failed.","src_addr" {"IP":"****","Port":***,"Zone":""},"timestamp":"2024-08-02T19:46:56Z"}

My teleport is running on EKS and the organization doesn't allow running kubectl exec on pods so I can't remove the folder /var/lib/teleport
I'm thinking of just removing the persistence storage in the configuration.
I want to know what I will have redo after removing it, cause honestly Teleport documentation is not the best.
Thanks

[webvictim]

Since your Teleport backend/audit logs/session recordings are stored in DynamoDB/S3, you shouldn't have to redo anything.

You should be able to delete the chart, delete the persistent volume claim and then redeploy the chart again and it should work.

[olawale-ib]

I've deleted the PVC, redeployed the chart, everything is all synced up but i get this from cluster-proxy

{"caller":"services/watcher.go:534","component":"proxy:1","level":"debug","message":"List of known proxies updated: [\"52f71e3b-87d5-467a-a30d-1043a646eb99\"].","pid":"7.1",
"timestamp":"2024-08-09T22:11:53Z"}{"timestamp":"2024-08-09T22:13:59Z","level":"warning","caller":"web/conn_upgrade.go:193","message":
"Failed to write ping message.","component":"web","error":"write tcp 10.23.22.197:3080->10.23.13.93:48838: write: broken pipe"}

~$ tsh login
ERROR: Teleport proxy not available at teleport.example.com:443.
        proxy service returned unsuccessful ping response; Teleport cluster auth may be misconfigured
                invalid connector name: github; valid options:

mind you I have removed the github authentication from my config

authentication:
  type: github
     connectorName: github

running tsh status still shows my previous profile

I thought redeploying means I will be starting from scratch with all previous resources deleted because the admin user is no longer on the team

[olawale-ib]

Everything seems to be working fine now
ran tsh logout to logout of the previous session
kubectl --namespace teleport exec deployment/teleport-auth -- tctl users add test --roles=access,editor to create a new admin user and was able to create role and etc
Thank you