emptyDir volume is not mounted

[barkbay]

Despite what our document suggests setting an emptyDir volume for Elasticsearch data does not actually mount and rely on that emptyDir.

To be a bit more explicit, here is the resulting Pod when the example from the documentation is applied:

apiVersion: v1
kind: Pod
metadata:
  name: elasticsearch-sample-es-default-0
  namespace: default
spec:
  containers:
  - name: elasticsearch
    volumeMounts:
    - mountPath: /mnt/elastic-internal/downward-api
      name: downward-api
      readOnly: true
    - mountPath: /usr/share/elasticsearch/bin
      name: elastic-internal-elasticsearch-bin-local
    - mountPath: /mnt/elastic-internal/elasticsearch-config
      name: elastic-internal-elasticsearch-config
      readOnly: true
    - mountPath: /usr/share/elasticsearch/config
      name: elastic-internal-elasticsearch-config-local
    - mountPath: /usr/share/elasticsearch/plugins
      name: elastic-internal-elasticsearch-plugins-local
    - mountPath: /usr/share/elasticsearch/config/http-certs
      name: elastic-internal-http-certificates
      readOnly: true
    - mountPath: /mnt/elastic-internal/probe-user
      name: elastic-internal-probe-user
      readOnly: true
    - mountPath: /usr/share/elasticsearch/config/transport-remote-certs/
      name: elastic-internal-remote-certificate-authorities
      readOnly: true
    - mountPath: /mnt/elastic-internal/scripts
      name: elastic-internal-scripts
      readOnly: true
    - mountPath: /usr/share/elasticsearch/config/transport-certs
      name: elastic-internal-transport-certificates
      readOnly: true
    - mountPath: /mnt/elastic-internal/unicast-hosts
      name: elastic-internal-unicast-hosts
      readOnly: true
    - mountPath: /mnt/elastic-internal/xpack-file-realm
      name: elastic-internal-xpack-file-realm
      readOnly: true
    - mountPath: /usr/share/elasticsearch/logs
      name: elasticsearch-logs
  volumes:
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels
        path: labels
    name: downward-api
  - emptyDir: {}
    name: elastic-internal-elasticsearch-bin-local
  - name: elastic-internal-elasticsearch-config
    secret:
      defaultMode: 420
      optional: false
      secretName: elasticsearch-sample-es-default-es-config
  - emptyDir: {}
    name: elastic-internal-elasticsearch-config-local
  - emptyDir: {}
    name: elastic-internal-elasticsearch-plugins-local
  - name: elastic-internal-http-certificates
    secret:
      defaultMode: 420
      optional: false
      secretName: elasticsearch-sample-es-http-certs-internal
  - name: elastic-internal-probe-user
    secret:
      defaultMode: 420
      items:
      - key: elastic-internal-probe
        path: elastic-internal-probe
      optional: false
      secretName: elasticsearch-sample-es-internal-users
  - name: elastic-internal-remote-certificate-authorities
    secret:
      defaultMode: 420
      optional: false
      secretName: elasticsearch-sample-es-remote-ca
  - configMap:
      defaultMode: 493
      name: elasticsearch-sample-es-scripts
      optional: false
    name: elastic-internal-scripts
  - name: elastic-internal-transport-certificates
    secret:
      defaultMode: 420
      optional: false
      secretName: elasticsearch-sample-es-default-es-transport-certs
  - configMap:
      defaultMode: 420
      name: elasticsearch-sample-es-unicast-hosts
      optional: false
    name: elastic-internal-unicast-hosts
  - name: elastic-internal-xpack-file-realm
    secret:
      defaultMode: 420
      optional: false
      secretName: elasticsearch-sample-es-xpack-file-realm
  - emptyDir: {}
    name: elasticsearch-data
  - emptyDir: {}
    name: elasticsearch-logs

While the elasticsearch-data volume is created it is not mounted in the elasticsearch container. The data are therefore written in the container instance, not in the expected Pod volume.

Might be related to #3848 I'm not sure if it's a bug, if it has ever worked, or if the doc must be fixed to include the volumeMounts in the example so adding both bug and docs label until until we clarify what the expected behavior is.

[thbkrkr]

We intentionally stopped adding volumeMount {"mountPath":"/usr/share/elasticsearch/data","name":"elasticsearch-data"} in #3806 and we didn't put it back in #3848.

As soon as the elasticsearch-data volume isn't backed by a PVC, we don't configure the corresponding volumeMounts.

What's going on?

We add default PVCs to the node spec only if no user defined PVCs exist, then build the pod template spec.

cloud-on-k8s/pkg/controller/elasticsearch/nodespec/statefulset.go

Lines 74 to 82 in c3d84bf

	// add default PVCs to the node spec only if no user defined PVCs exist
	nodeSet.VolumeClaimTemplates = defaults.AppendDefaultPVCs(
	nodeSet.VolumeClaimTemplates,
	nodeSet.PodTemplate.Spec,
	esvolume.DefaultVolumeClaimTemplates...,
	)
	// build pod template
	podTemplate, err := BuildPodTemplateSpec(ctx, client, es, nodeSet, cfg, keystoreResources, setDefaultSecurityContext)

In BuildPodTemplateSpec, we prepare the volumes and volumeMounts:

cloud-on-k8s/pkg/controller/elasticsearch/nodespec/podspec.go

Line 66 in c3d84bf

volumes, volumeMounts := buildVolumes(es.Name, ver, nodeSet, keystoreResources, downwardAPIVolume)

In buildVolumes, we take into account the volumes derived from the PVC.

cloud-on-k8s/pkg/controller/elasticsearch/nodespec/volumes.go

Lines 63 to 71 in c3d84bf

	// append future volumes from PVCs (not resolved to a claim yet)
	persistentVolumes := make([]corev1.Volume, 0, len(nodeSpec.VolumeClaimTemplates))
	for _, claimTemplate := range nodeSpec.VolumeClaimTemplates {
	persistentVolumes = append(persistentVolumes, corev1.Volume{
	Name: claimTemplate.Name,
	VolumeSource: corev1.VolumeSource{
	PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSource{
	// actual claim name will be resolved and fixed right before pod creation
	ClaimName: "claim-name-placeholder",

cloud-on-k8s/pkg/controller/elasticsearch/nodespec/volumes.go

Line 77 in c3d84bf

volumes := persistentVolumes

So, if the elasticsearch-data is provided by the user via a PVC or if there is no elasticsearch-data volume and ECK sets the default 1Gi PVC, the volumeMount is added. If the elasticsearch-data is provided by the user as a volume (e.g.: emptyDir volume, hostpath volume) without PVC, the volumeMount is not added.

cloud-on-k8s/pkg/controller/elasticsearch/nodespec/volumes.go

Line 117 in c3d84bf

volumeMounts = esvolume.AppendDefaultDataVolumeMount(volumeMounts, volumes)