Regression: 8.5.3 can no longer run with random user ID

[pebrc]

APM Server version (apm-server version):
8.5.3
Description of the problem including expected versus actual behavior:

The ECK team runs automated tests against 8.5.3 (we only started doing it 4 days ago) and those tests started failing. The tests pass with 8.5.2

We run the tests with random user ID and group ID (in this case 12345). The tests now fail with

 + keystore_initialized_flag=/usr/share/apm-server/data/elastic-internal-init-keystore.ok                                                                                                                         │
│ + [[ -f /usr/share/apm-server/data/elastic-internal-init-keystore.ok ]]                                                                                                                                          │
│ + echo 'Initializing keystore.'                                                                                                                                                                                  │
│ + /usr/share/apm-server/apm-server keystore create --force                                                                                                                                                       │
│ Initializing keystore.                                                                                                                                                                                           │
│ error initializing beat: error loading config file: config file ("apm-server.yml") must be owned by the user identifier (uid=12345) or root               

I have a suspicion that e39b98a might be at fault here and it introduced a regression compared to what was changed in reaction to elastic/beats#18871 (and a breaking change compared to 8.5.2)

Steps to reproduce:

• ECK (I used 2.6.1 but any recent version should expose the problem)

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: es-apm-sample
spec:
  version: 8.5.3
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kb-apm-sample
spec:
  version: 8.5.3
  count: 1
  elasticsearchRef:
    name: "es-apm-sample"
  config:
    xpack.fleet.packages:
    - name: apm
      version: latest
---
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-apm-sample
spec:
  version: 8.5.3
  count: 1
  secureSettings:
    - secretName: test-secret
  podTemplate:
    spec:
      securityContext:
        fsGroup: 12345
        runAsNonRoot: true
        runAsUser: 12345
  elasticsearchRef:
    name: "es-apm-sample"
  kibanaRef:
    name: "kb-apm-sample"
---
apiVersion: v1
data:
  logging.verbose: dHJ1ZQ==
kind: Secret
metadata:
  name: test-secret
type: Opaque
...

[axw]

@pebrc I think this was fixed in 8.6.0 by #9818

[axw]

Given that 8.6.0 is out, there won't be any more 8.5.x's. Please open a new issue if you find there are still issues with 8.6.x.