Issue with teleport-cluster-proxy

[olawale-ib]

I have Teleport installed on EKS. The cluster-auth works fine, I can ssh into pods, audit logs and record sessions. But there something wrong with cluster-proxy, it is giving the logs below which is preventing me from joining RDS access to my cluster (I have tried both kubernetes and iam join method)

Teleport configuration is here and its generated deployment yaml is here

{"level":"warning","caller":"web/conn_upgrade.go:193","message":"Failed to write ping message.","component":"web","error":"write tcp x.x.x.x:3080->x.x.x.x:58026: write: broken pipe"}
{"caller":"sshutils/server.go:553","component":"ssh:proxy:server","level":"debug","message":"Incoming connection 104.202.x.x:39452 -\u003e x.x.x.x:3080 version: SSH-2.0-Go, certtype: \"user\"","pid":"6.1"}
{"caller":"reversetunnel/srv.go:673","component":"proxy:1","level":"debug","message":"Received transport request.","pid":"6.1"}
{"caller":"sshutils/server.go:558","component":"ssh:proxy:server","level":"debug","message":"Closed connection 104.202.x.x:39452.","pid":"6.1"}
{"level":"warning","caller":"web/conn_upgrade.go:193","message":"Failed to write ping message.","component":"web","error":"write tcp x.x.x.x:3080->x.x.x.x:34532: write: broken pipe"}
{"caller":"sshutils/server.go:553","component":"ssh:proxy:server","level":"debug","message":"Incoming connection 104.202.x.x:57470 -\u003e x.x.x.x:3080 version: SSH-2.0-Go, certtype: \"user\"","pid":"6.1"}
{"caller":"reversetunnel/srv.go:673","component":"proxy:1","level":"debug","message":"Received transport request.","pid":"6.1"}
{"caller":"sshutils/server.go:558","component":"ssh:proxy:server","level":"debug","message":"Closed connection 104.202.x.x:57470.","pid":"6.1"}
{"caller":"sshutils/server.go:553","component":"ssh:proxy:server","level":"debug","message":"Incoming connection 104.202.x.x:38840 -\u003e x.x.x.x:3080 version: SSH-2.0-Go, certtype: \"user\"","pid":"6.1"}
{"caller":"reversetunnel/srv.go:673","component":"proxy:1","level":"debug","message":"Received transport request.","pid":"6.1","2024-08-22T15:47:38Z"}
{"caller":"sshutils/server.go:558","component":"ssh:proxy:server","level":"debug","message":"Closed connection 104.202.x.x:38840.","pid":"6.1"}
{"level":"warning","caller":"web/conn_upgrade.go:193","message":"Failed to write ping message.","component":"web","error":"write tcp x.x.x.x:3080->x.x.x.x.x:56560: write: broken pipe"}
{"caller":"sshutils/server.go:553","component":"ssh:proxy:server","level":"debug","message":"Incoming connection 104.202.x.x:56010 -\u003e x.x.x.x:3080 version: SSH-2.0-Go, certtype: \"user\"","pid":"6.1"}
{"caller":"reversetunnel/srv.go:673","component":"proxy:1","level":"debug","message":"Received transport request.","pid":"6.1"}
{"caller":"sshutils/server.go:558","component":"ssh:proxy:server","level":"debug","message":"Closed connection 104.202.x.x:56010.","pid":"6.1"}

[webvictim]

This doesn't look like an issue with hosts joining to me, it looks like a user is trying to connect to the cluster and their connection isn't finishing cleanly. Do you recognise the 104.202.x.x IP?

• What errors are you getting when trying to join RDS to the cluster?
• Are you trying to use the teleport-kube-agent chart? If so, what are the Helm values?

[olawale-ib]

yes, I'm using the teleport-kube-agent
Was previously using the kubernetes-token join method. Currently using the iam join method, so that may be the user trying to connect??

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: teleport-agent
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: all-clusters
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
  source:
    chart: teleport-kube-agent
    repoURL: https://charts.releases.teleport.dev
    targetRevision: "15.4.6"
    helm:
      values: |
        roles: db
        proxyAddr: teleport.example.com:443
        joinParams:
          method: iam
          tokenName: iam-token
        awsDatabases:
          - types:
              - rds
            regions:
              - us-west-2
            tags:
              "*": "*"
        annotations:
          serviceAccount:
            eks.amazonaws.com/role-arn: "arn:aws:iam::XXXXX:role/teleport-role"
        resources:
          requests:
            cpu: 50m
            memory: 200Mi
          limits:
            memory: 200Mi
        log:
          format: json
          level: DEBUG
  destination:
    name: application-dev
    namespace: teleport-agent

Logs from the kube-agent

{"level":"error","caller":"service/connect.go:248","message":"Was this node already registered to a different cluster? To join this node to a new cluster, remove the data_dir and try again","component":"proc:1","pid":"7.1","data_dir":"/var/lib/teleport"}
{"level":"error","caller":"service/connect.go:97","message":"Failed to establish connection to cluster.","component":"proc:1","pid":"7.1","identity":"Instance","error":"Failed to connect to Proxy Server through tunnel: connection error: desc = \"transport: Error while dialing: failed to dial: tls: failed to verify certificate: x509: certificate signed by unknown authority\""}
{"level":"debug","caller":"service/connect.go:114","message":"Retrying connection to auth server.","component":"proc:1","pid":"7.1","identity":"Instance","backoff":12283118830}
{"level":"debug","caller":"service/connect.go:218","message":"Got connected state.","component":"proc:1","pid":"7.1","rotation_state":"never updated"}
{"level":"info","caller":"service/connect.go:238","message":"Connecting to the cluster with TLS client certificate.","component":"proc:1","pid":"7.1","cluster":"teleport.example.com"}
{"level":"debug","caller":"service/connect.go:1222","message":"Attempting to connect to Auth Server through tunnel.","component":"proc:1","pid":"7.1","proxy-server":"teleport.example.com:443"}
{"caller":"webclient/webclient.go:130","component":null,"level":"debug","message":"Attempting GET teleport.example.com:443/webapi/find"}
{"caller":"client/alpn_conn_upgrade.go:76","component":null,"level":"debug","message":"ALPN connection upgrade required for \"teleport.example.com:443\": true. No ALPN protocol is negotiated by the server."}
{"caller":"proxy/proxy.go:197","component":"http:proxy","level":"debug","message":"No proxy set in environment, returning direct dialer."}
{"caller":"client/alpn_conn_upgrade.go:278","component":null,"hostname":"teleport.example.com:443","level":"debug","message":"Performing ALPN WebSocket connection upgrade."}

[webvictim]

It looks like this agent was previously joined to a different cluster. You should delete the <release-name>-0-state secret in the teleport-kube-agent chart's namespace and restart the pod.

(this is the equivalent of removing the /var/lib/teleport/proc directory on a Linux server-based agent)

[olawale-ib]

Thanks a lot 😃