Multiple clusters management in EKS #616

aylei · 2019-06-29T14:55:01Z

What problem does this PR solve?

Support multiple TiDB clusters management in EKS with some updates per #575

What is changed and how does it work?

The control plane (Kubernetes master and tidb-operator) and data plane (node pools and helm release) are separate into two different modules: tidb-operator and tidb-cluster, a top-level module compose one tidb-operator module and one or more tidb-cluster modules to support multiple cluster management.

See the README of top-level module, README of tidb-operator module and README of tidb-cluster module for detail.

Check List

Tests

Manual test (add detailed scripts or steps below)
- 1 EKS + 1 TiDB cluster
- 1 EKS + 2 TiDB cluster
- 1 EKS + 3 TiDB cluster

Code changes

Has Terraform scripts change
Has documents change

Related changes

Need to update the documentation

Does this PR introduce a user-facing change?:

The terraform scripts support manage multiple TiDB clusters in one EKS cluster.

Limitations

Terraform do not support loop for modules, I cannot figure out how to define multiple TiDB clusters in .tfvars, so it is inevitable to edit the cluster.tf directly for multiple cluster management now;

@tennix @jlerche @gregwebs PTAL, most of the works are done by @tennix in bd2343f, I did some clean up and re-organization of code in the follow up commits.

@jlerche I changed the aws-tutorial.tfvars to keep compatibility, hope this won't break your tutorial😅

Signed-off-by: Aylei <rayingecho@gmail.com>

deploy/aws/README.md

deploy/aws/eks/local.tf

gregwebs · 2019-07-01T14:19:19Z

deploy/aws/eks/manifests/crd.yaml

@@ -0,0 +1,103 @@
+apiVersion: apiextensions.k8s.io/v1beta1


Can we symlink this instead of copying?

No, this module is supposed to be workable outside this repository. But it is definitely not a good idea to keep a copy here.

How about kubectl appy -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.0.0-beta.3/manifests/crd.yaml?

Why does this need to work outside this repository? I have been referencing manifests with symlinks in the GCP terraform. The deployment doesn't work without a good way to reference our charts and manifests.

The suggested update is problematic because it will fall out of sync. Better would be to have a variable "manifests" which could be set to ../../manifests or https://raw.githubusercontent.com/pingcap/tidb-operator/v1.0.0-beta.3.

Does it help to move the manifests directory under the deploy directory?

Why does this need to work outside this repository? I have been referencing manifests with symlinks in the GCP terraform. The deployment doesn't work without a good way to reference our charts and manifests.

Cases that use these modules outside this repository:

terraform store state in the working dir, users may copy this module to manage different EKS instances (one of our colleagues have tried this and got hurt by the symlink)

for advanced users, it is possible to compose the tidb-operator module and tidb-cluster module into their own terraform scripts. A self-contained module will be easier to be composed in this case

Does it help to move the manifests directory under the deploy directory?

Yes, to some extent. But the manifest directory is still necessary because the ebs-gp2 storage class and local-volume-provisioner.yaml are dedicated to AWS. I've considered use a small overlay yaml to customize the base local-volume-provisioner.yaml via kusomize, but this has a low priority.

I used kustomize for GCP, it was quite easy: https://github.com/pingcap/tidb-operator/tree/master/deploy/gcp/manifests/local-ssd

With respect to state, it seems that users are solving a problem by copying and in response we are copying :)
I want to get away from copying as a solution to anything. The way I solve this is by having users instantiate a module, rather than just changing a terraform variable files. A user instantiates the module in a directory representing their environment (staging, production). The usability is about the same for one instantiation, but when you do multiple everything is much better.

I think in our case we are missing a nice mechanism to depend on our manifest files. The only way I can see how to do it is to use the file function to read the file in (and we can write it out to a new local file to avoid memory consumption). But then I think we still end up needing the entire git repo, so it doesn't really seem better than a symlink.

deploy/aws/eks/manifests/local-volume-provisioner.yaml

deploy/aws/eks/manifests/tiller-rbac.yaml

deploy/aws/eks/templates/kubeconfig.tpl

deploy/aws/main.tf

deploy/aws/tidb-cluster/pre_userdata

Co-Authored-By: Greg Weber <greg@gregweber.info>

Signed-off-by: Aylei <rayingecho@gmail.com>

aylei · 2019-07-02T15:00:31Z

@tennix @gregwebs I've removed the customized eks module for better maintainability, and addressed all the review comments, PTAL again

Signed-off-by: Aylei <rayingecho@gmail.com>

gregwebs · 2019-07-02T17:33:12Z

deploy/aws/tidb-operator/manifests/local-volume-provisioner.yaml

@@ -38,6 +38,8 @@ spec:
        - key: dedicated
          operator: Exists
          effect: "NoSchedule"
+      nodeSelector:
+        localVolume: "true"


local-ssd would be a slightly better name. GKE adds this tag automatically now: cloud.google.com/gke-local-ssd

tennix · 2019-07-02T23:26:06Z

deploy/aws/aws-tutorial.tfvars

cluster name should not contain underscore.

tennix

Rest LGTM

gregwebs · 2019-07-03T00:47:04Z

deploy/aws/README.md

+}
+```
+
+## Multiple Cluster Management


Suggested change

## Multiple Cluster Management

## Multiple TiDB Cluster Management

Signed-off-by: Aylei <rayingecho@gmail.com>

aylei · 2019-07-03T01:16:43Z

@tennix @gregwebs I've updated this PR according to your comments, PTAL, thanks!

gregwebs · 2019-07-03T01:17:21Z

As per my comments above, I would like to demonstrate usage that is safe for multiple environments. The good news is we have already created a module, we just need to show how to use it as one!

The only change we need to make is that all the usages of file need to have path.module, so file("${path.module}.

Then the usage as module is just a new directory with a file

module "staging" {
  source = "../"
}

The problem with this is then the suggestion to create multiple TiDB clusters by editing clusters.tf. I suggest we have the user instantiate the tidb-cluster modules themselves.

We can move all our top-level terraform to a separate directory, perhaps called "vpc-setup". Then we have a file deploy/aws/staging/main.tf with contents:

module "staging" {
  source = "../vpc-setup"
}

module module "default-cluster" {
  source = "../tidb-cluster"
  ...
}

gregwebs · 2019-07-03T01:19:25Z

Great work on this, the above comment shows how easy it is to start changing things around now that things have been properly modularized :)

aylei · 2019-07-03T01:25:00Z

Yes, and (luckily) all the file() invocation and the reference of files in local-exec handles directory properly now😄

The example is great and I'm going to add it to the README👍

gregwebs · 2019-07-03T01:37:55Z

Glad that makes sense. I think if we add it to the README but also explain how to modify the vpc-setup module directly it will lead to problems where someone will have already modified a module directly. I don't know how to fix the state file at that point to move it to the multiple environment setup.

aylei · 2019-07-03T01:45:43Z

I suggest we have the user instantiate the tidb-cluster modules themselves.

IMHO, this UX is more applicable for advanced users with hands-on terraform experience, because after all a little (maybe a lot) glue codes are necessary.

I think if we add it to the README but also explain how to modify the vpc-setup module directly it will lead to problems where someone will have already modified a module directly. I don't know how to fix the state file at that point to move it to the multiple environment setup.

We should document that users should avoid reusing the top-level module because it is not modularized. Then the state of the top-level module will be stored in /deploy/aws/, and the state of custom modules will be stored in, say, /deploy/aws/staging. And there should be no tf state in /deploy/aws/tidb-operator and /deploy/aws/tidb-cluster because they are not supposed to be applied directly.

Signed-off-by: Aylei <rayingecho@gmail.com>

gregwebs · 2019-07-03T02:04:48Z

IMHO, this UX is more applicable for advanced users with hands-on terraform experience, because after all a little (maybe a lot) glue codes are necessary.

The glue code is almost entirely default variables. But those are all just used in clusters.tf so we can move the default values into the clusters module itself.

We can also provide a default environment to use. So the user only needs to cd default.

gregwebs · 2019-07-03T02:07:41Z

Then the state of the top-level module will be stored in /deploy/aws/, and the state of custom modules will be stored in, say, /deploy/aws/staging.

Yes, but the user has already made edits to the top-level module directly. This will end up affecting deploy/aws/staging, which will cause problems. The user now needs to move their top-level usage to the environment based approach, but I don't even know how to do this without destroying the existing top-level environment first.

aylei · 2019-07-03T02:27:46Z

I mean, the top-level module is indeed an example of composing the sub modules. Whether this module located in /deploy/aws or /deploy/aws/default makes no difference in usage except that the top-level may seems to be special because it is located in the ... top-level.

aylei · 2019-07-03T02:34:27Z

Let me demonstrate if I get you correct now:

Maybe a better UX is moving the VPC setup to a separate sub module, and moving the current top-level module to the ./default directory. Then we will have the following directories layout:

./tidb-cluster
./tidb-operator
./vpc
./default

Then:

if the users want to provision more TiDB clusters in the default module, they go to edit the tf scripts in this module;
if the users want to provision a new EKS cluster and TiDB clusters in it, they create a new module like staging:

./tidb-cluster
./tidb-operator
./vpc
./default
./staging

Does this make sense?

aylei · 2019-07-03T02:38:51Z

I'd like to move the discussion about UX and the recommended usage of modules to a new issue, @gregwebs @tennix how do you think?

tennix · 2019-07-03T03:13:44Z

Agreed, I think we can merge this PR now and improve the UX in following PRs.

tennix

LGTM

gregwebs · 2019-07-03T03:42:28Z

@aylei yes, that is a good module organization.

tennix and others added 3 commits June 17, 2019 20:05

separate worker node group from master control node group

bd2343f

Merge tennix aws-multi-cluster

23454ae

Signed-off-by: Aylei <rayingecho@gmail.com>

Multiple cluster management in AWS

192a3a9

Signed-off-by: Aylei <rayingecho@gmail.com>

aylei requested review from gregwebs, jlerche and tennix June 29, 2019 14:55

aylei added 2 commits June 29, 2019 22:57

Remove commented codes

fc2e262

Signed-off-by: Aylei <rayingecho@gmail.com>

Refine README me of terraform for eks

49e22c1

Signed-off-by: Aylei <rayingecho@gmail.com>

aylei added deploy area/doc labels Jun 30, 2019