TestReversalStatefulSetRename is flaky

[barkbay]

=== RUN   TestReversalStatefulSetRename/All_expected_Pods_should_eventually_be_ready#01
Retries (30m0s timeout): 
    utils.go:84: 
        	Error Trace:	utils.go:84
        	Error:      	Received unexpected error:
        	            	invalid StatefulSets:
                            expected map[test-sset-rename-reversal-pntt-es-masterdata:1],
                                 got map[test-sset-rename-reversal-pntt-es-masterdata:1 test-sset-rename-reversal-pntt-es-other:1]

The steps of this test are:

1. Rename the current nodeSet from masterdata to other
2. Check that the configuration has been updated
3. Rename other back to masterdata

It means that the nodeSet is renamed for a vey short moment.

When test is failing the resources are:

NAME                                                                        HEALTH   NODES   VERSION   PHASE   AGE
elasticsearch.elasticsearch.k8s.elastic.co/test-sset-rename-reversal-nr6k   green    1       7.9.1     Ready   25m

NAME                                                            READY   AGE   CONTAINERS      IMAGES
statefulset.apps/test-sset-rename-reversal-nr6k-es-masterdata   1/1     25m   elasticsearch   docker.elastic.co/elasticsearch/elasticsearch:7.9.1
statefulset.apps/test-sset-rename-reversal-nr6k-es-other        0/1     23m   elasticsearch   docker.elastic.co/elasticsearch/elasticsearch:7.9.1

NAME                                                 READY   STATUS     RESTARTS   AGE   IP            NODE                                           NOMINATED NODE   READINESS GATES
pod/test-sset-rename-reversal-nr6k-es-masterdata-0   1/1     Running    0          25m   10.0.129.77   gke-michael-dev-2-default-pool-add12a04-223g   <none>           <none>
pod/test-sset-rename-reversal-nr6k-es-other-0        0/1     Init:0/1   0          23m   10.0.128.77   gke-michael-dev-2-default-pool-ce5974a4-4jt0   <none>           <none>

NAME                                                                     TYPE                                  DATA   AGE
[..]
secret/test-sset-rename-reversal-nr6k-es-masterdata-es-transport-certs   Opaque                                3      25m
secret/test-sset-rename-reversal-nr6k-es-other-es-transport-certs        Opaque                                1      23m

test-sset-rename-reversal-nr6k-es-other-0 can't start because the transport certificate is not generated (there's only ca.crt in test-sset-rename-reversal-nr6k-es-other-es-transport-certs), which seems to be expected knowing that the nodeSet other does not exist anymore in the Elasticsearch spec.

The Pod is not deleted because of the reason: Cannot remove the last running master node.
checkDownscaleInvariants is assuming that there is only one master running (which is true, the other one will never start), but hence refuse to delete test-sset-rename-reversal-nr6k-es-other-0 which is also a master according to its labels. (even if it has never really been the case because it never started).

[barkbay]

I got a 404 first time I submitted this one, I thought that creation had failed.
Closing as duplicate of #3844