add initContainer to copy config from ConfigMap

[svmaris]

What this PR does / why we need it:

ConfigMaps are mounted read-only since Kubernetes 1.9.4 (kubernetes/kubernetes#58720). The rabbitmq-ha Chart uses a ConfigMap to provision /etc/rabbitmq. The official rabbitmq Docker Image modifies these files in its docker-entrypoint.sh. This PR adds an initContainer to the StatefulSet that copies the files from the ConfigMap to a new emptyDir.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #4166

Special notes for your reviewer:

@etiennetremel

[brianwawok]

@svmaris your change is not working for me

I see

warning: /var/lib/rabbitmq/.erlang.cookie contents do not match RABBITMQ_ERLANG_COOKIE

=ERROR REPORT==== 17-Mar-2018::16:37:42 ===
** Connection attempt from disallowed node 'rabbitmqcli61@rabbitmq-ha-rabbitmq-ha-0.rabbitmq-ha-rabbitmq-ha.default.svc.cluster.local' **

[etiennetremel]

@brianwawok as mentioned in the README.md, you need to keep the same erlang cookie configuration between upgrade:

$ export ERLANGCOOKIE=$(kubectl get secrets -n <NAMESPACE> <HELM_RELEASE_NAME>-rabbitmq-ha -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 --decode)
$ helm upgrade --name <HELM_RELEASE_NAME> \
    --set rabbitmqErlangCookie=$ERLANGCOOKIE \
    stable/rabbitmq-ha

[brianwawok]

@etiennetremel okay.. I was getting even on a helm delete --purge, followed by a helm install.

I ended up just dumping everything to file with helm template, and this fix appears to work, so happy to chalk it up to user error or a bug in helm purge..

[hiddeco]

We were experiencing the same issues as described above using the same ERLANGCOOKIE as the previous deployments.

After a helm delete <name> - -purge and helm install all pods were deployed successfully. Pods do however fail to reconnect to the cluster when they're killed.

I am almost certain these issues are not the result of the changes in this PR but due to unforseen changes in Kubernetes or RabbitMQ itself.

Update (18 Mar 10PM CET): after 500 restart attempts the reconnect came through this morning. Killed the pod again and it landed almost instantly in a CrashLoopBackoff.

DIAGNOSTICS
===========

attempted to contact: ['rabbit@10.44.0.37']

rabbit@10.44.0.37:
  * connected to epmd (port 4369) on 10.44.0.37
  * epmd reports: node 'rabbit' not running at all
                  other nodes on 10.44.0.37: [rabbitmqcli99]
  * suggestion: start the node

Current node details:
 * node name: 'rabbitmqcli99@<snip>.svc.cluster.local'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: <snip>
  Warning  FailedSync  3m (x99 over 44m)  kubelet, <snip>-89ce862a-1z01  Error syncing pod

Failing pod is not logging anything to stdout and stale for +- 5 minutes before a restart happens.

[jjlakis]

Works for me

[unguiculus]

/ok-to-test

[unguiculus]

/assign

[unguiculus]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: svmaris, unguiculus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [unguiculus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment