TiDB Operator - Cannot connect to TiDB Service for TiDB Cluster External IP

[ttadeo]

Bug Report

What version of Kubernetes are you using?

AWS EKS 1.18

What version of TiDB Operator are you using?

TiDB Operator Version: version.Info{GitVersion:"v1.1.12", GitCommit:"267b0d44f8c66f41a3a02685d7100c55d4697715", GitTreeState:"clean", BuildDate:"2021-04-15T12:21:20Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

PROBLEM:

Trying to connect to EXTERNAL-IP for TiDB-Cluster Service and connection times out. Using 127.0.0.1 with port 4000 forwarded works fine. I can connect to the Grafana Service using it's EXTERNAL IP just fine. Both services are of type LoadBalancer, however EXTERNAL TRAFFIC POLICY for TiDB-Cluster Service specifies "Local" and Grafana Service specifies "Cluster".

Question:

I understand the need for security(Access to TiDB Cluster SVC only accessible within Cluster) but for remote access outside the cluster having "Local" set prevents remote access using EXTERNAL IP. Is this spec correct ? If so, why ?

[DanielZhangQD]

@ttadeo Local does not block access to the TiDB service, could you please check the LB on your AWS console and see if it has available targets? And you may also check with the AWS support for the NLB issues.

[ttadeo]

@DanielZhangQD Thank you I will spin up the cluster again and check AWS NLB. I will update when I am done with my results.

[dveeden]

@ttadeo maybe have a look at pingcap/docs#5555 and leave a comment if anything special needs to be added there.

[dveeden]

#879 seems related

[dveeden]

The problem might be the "Scheme". This is visible in the web UI for the ELB. This seems to be "internal" while it should be "internet-facing".

I think this is caused by this line:

tidb-operator/examples/aws/tidb-cluster.yaml

Line 50 in d90f424

service.beta.kubernetes.io/aws-load-balancer-internal: '0.0.0.0/0'

https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/ suggests that this setting is a boolean. So the IP range here doesn't seem to be a correct value.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 15 days