create all statefulsets immediately

[gregwebs]

TODO

• run stability tests
• decide on renaming wait-for-pd

As per #469 this creates statefulsets and pending pods immediately to help autoscalers scale more quickly.

Design doc is here

A blocking init container wait-for-pd is used to prevent TiKV and TiDB from immediately starting up. This code calls the PD API directly. For TiDB it will call the new status is_initialized API if it is available.

Error handling is also generally improved in the member managers: as a side effect this may fix bugs where the controller sync loop is not re-ran during errors.

The PD API was updated to 3.0 which seems to be backwards compatible for our usage (more testing needed).

Testing notes

I have created 50+ clusters with the operator that has the changes from the first commit. It has the desired effect for autoscaling. The second commit is still being tested.

What is changed and how does it work?

• When PD is not available for TiKV, or TiKV is not available for TiDB, go ahead and create the statefulset.
• Don't break from the update function immediately when a RequeueError is seen.

Check List

Tests

• Unit test: using existing tests
• E2E test: using existing e2e tests
• Stability test: ?
• Manual test: create single clusters of various sizes (multiple cluster creation often runs into issues with the current operator). Test with pd/tikv/tidb version 3.0.0-rc1 and version 3.0.0

Side effects

Increased image size
Originally this change meant shipping another 50M binary. However, overall image size is smaller since this PR was originally started. This work was merged in other Pull requests:

• refactor so that using the PD API avoids unnecessary imports #618 Code was re-arragned so that wait-for-pd minimized its imports to just the interaction with the PD API and logging without importing all of Kubernetes. This reduced the binary size by 50%. We could do the same for the discovery service.
• build without debugger symbols #592 Removal of debugger symbols reduced the size of all the binaries

Pending Pod warnings

• May surprise existing users that pre-provision capacity and monitor for pending pods. Alerts that alert for pending pods could get triggered.

Upgrade procedure

• When upgrading the tidb-operator image from before this change one must first upgrade the tidb-operator helm chart templates. This is expected, but it could surprise someone since it often isn't necessary.

The PD API was updated to 3.0 which seems to be backwards compatible for our usage. We will probably want to see some additional testing here.

Related changes

• Need to update the documentation

Does this PR introduce a user-facing change?

Yes, to the extent users look at pod creation (which we have in our tutorials).

Create TiKV/TiDB pending pods immediately to help autoscalers scale more quickly. They wait for their dependencies (e.g. PD) to be ready with an Init container, so they will show up as in state Init while waiting.

[gregwebs]

The usage of PD APIs in wait-for-pd should be updated. For TiDB 3.0 the TiDB component should check the PD api cluster_status for the is_initialized flag as per #586

[gregwebs]

I see an issue where the operator will expand the tidb statefulset when it is waiting in init for a long time

demo-discovery-5bb58f7bb5-wklmq   1/1     Running    0          18m
demo-monitor-6f9c9fdffb-59bkp     1/1     Running    0          18m
demo-pd-0                         1/1     Running    0          18m
demo-pd-1                         1/1     Running    0          18m
demo-pd-2                         1/1     Running    1          18m
demo-tidb-0                       0/2     Init:0/1   0          18m
demo-tidb-1                       0/2     Init:0/1   0          18m
demo-tidb-2                       0/2     Init:0/1   0          9m23s
demo-tidb-3                       0/2     Init:0/1   0          9m23s
demo-tidb-4                       0/2     Init:0/1   0          4m23s
demo-tikv-0                       0/1     Pending    0          18m
demo-tikv-1                       1/1     Running    0          18m
demo-tikv-2                       1/1     Running    0          18m

[gregwebs]

When this came out of the init state the extra tidb pods did get terminated.

[gregwebs]

okay, I added some code to handle that case

[CLAassistant]

All committers have signed the CLA.

[aylei]

@gregwebs Cut of release-1.1 branch is targeted at December 31th, could you please confirm whether this feature is expected to be releases in 1.1 or should be re-targeted to 1.2? Thanks!

[gregwebs]

Probably needs to be 1.2

[aylei]

Okay, removed from v1.1 milestone for now, feel free to add it back if this is necessary to land in v1.1

[github-actions]

This pr is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 15 days