Cannot create Kibana with the pod security policy "readOnlyRootFilesystem=true"

[masa213f]

Bug Report

What did you do?

I try to deploy Kibana using ECK 1.2.1.

What did you expect to see?

The Kibana is deployed successfully.

What did you see instead? Under which circumstances?

The initContainer of Kibana get failed due to the pod security policy of readOnlyRootFilesystem=true.

Environment

• ECK version:

  1.2.1 (Commit ID: b531623 )

• Kubernetes information:

  • On premise

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:56:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"archive", BuildDate:"2020-09-17T02:53:25Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Resource definition:

Kibana CR

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-sample
  namespace: sandbox
spec:
  version: 7.4.2
  count: 1
  elasticsearchRef:
    name: "elastic-sample"
  http:
    service:
      metadata:
        annotations:
          projectcontour.io/upstream-protocol.tls: "5601,https"
  podTemplate:
    spec:
      containers:
      - name: kibana
        volumeMounts:
          - mountPath: /usr/share/kibana/optimize
            name: kibana-optimize
          - mountPath: /usr/share/kibana/data
            name: kibana-data
        resources:
          requests:
            memory: 4Gi
            cpu: 1
          limits:
            memory: 8Gi
            cpu: 2
      securityContext:
        runAsUser: 1000
      volumes:
        - name: kibana-optimize
          emptyDir: {}
        - name: kibana-data
          emptyDir: {}

• Logs:

$ kubectl logs -n sandbox kibana-sample-kb-78c568b88-w4279 elastic-internal-init-config
+ init_config_initialized_flag=/usr/share/kibana/config/elastic-internal-init-config.ok
+ [[ -f /usr/share/kibana/config/elastic-internal-init-config.ok ]]
+ echo 'Setup Kibana configuration'
Setup Kibana configuration
+ ln -sf /mnt/elastic-internal/kibana-config/kibana.yml /mnt/elastic-internal/kibana-config/telemetry.yml /mnt/elastic-internal/kibana-config-local/
+ touch /usr/share/kibana/config/elastic-internal-init-config.ok
touch: cannot touch '/usr/share/kibana/config/elastic-internal-init-config.ok': Read-only file system

• In my survey, the following path may be wrong.

https://github.com/elastic/cloud-on-k8s/blob/1.2.1/pkg/controller/kibana/init_configuration.go#L20

The elastic-internal-init-config.ok file should be created in InitContainerConfigVolumeMountPath.

[charith-elastic]

I think you are right about the path.

I am curious to know how you manage to run Elasticsearch with readOnlyRootFileSystem enabled. Do you use a different PSP for Elasticsearch?

[masa213f]

@chingis-elastic
Thank you for your quick response!

Do you use a different PSP for Elasticsearch?

Yes.

And my colleagues(@tapih and @zoetrope) did report the problem with Elasticsearch in #2108
Could you fix this issue too?

[charith-elastic]

I think the description of the problem in #2108 is still applicable. I can't see a good way of completely avoiding the root filesystem with Elasticsearch deployments.

[tommasopozzetti]

I also ran into the same issues trying to get Elasticsearch and Kibana to run with a ReadOnlyRootFilesystem enabled.
Thank you for this fix for the Kibana side of the issue!!
For what concerns Elasticsearch, I ended up solving it by mounting an additional writable emptyDir into the initContainer in which I copy all the configs from the /usr/share/elasticsearch/config directory. Then I run the edits that the initContainer usually does in that directory instead. Finally I fix a couple of links that would otherwise misbehave due to how the initContainer script is set up and that seems to do trick, allowing me to keep Elasticsearch working with ReadOnlyRootFilesystem enabled.
Basically this is the script that the initContainer ends up running:

rm -rf /mnt/elastic-config-copy/* && cp -r /usr/share/elasticsearch/config/* /mnt/elastic-config-copy/ && sed 's$/usr/share/elasticsearch/config$/mnt/elastic-config-copy$g' /mnt/elastic-internal/scripts/prepare-fs.sh > /tmp/prepare-fs.sh && chmod 755 /tmp/prepare-fs.sh && /tmp/prepare-fs.sh && find /mnt/elastic-internal/elasticsearch-config-local -lname '/mnt/elastic-config-copy*' | while read link; do newtarget=$(readlink -n $link | sed -e 's$/mnt/elastic-config-copy$/usr/share/elasticsearch/config$g') && ln -sf $newtarget $link; done

assuming the extra mount is at /mnt/elastic-config-copy/.
I also had to mount a writable directory at /tmp to create the modified prepare-fs.sh script.
This works but it is a bit hacky, it would be nice if the operator could take care of this. If this was taken care of by the operator, it could simply add an extra initContainer running beforehand which copies the contents of /usr/share/elasticsearch/config into the mount and then mounts it in the second initContainer over the config directory, making it effectively writable. That would not require the /tmp mount anymore and also would get rid of the need to fix the broken links.