GCS Repository plugin not loading #1349

gidesh · 2019-07-23T14:53:08Z

Bug Report

What did you do?`

Used elastic search operator docker.elastic.co/eck/eck-operator:0.9.0-rc3. And add initcontainer to install gcs plugin as per https://github.com/elastic/cloud-on-k8s/blob/master/docs/snapshots.asciidoc.

What did you expect to see?

All the efk-es pods starting up correctly.

What did you see instead? Under which circumstances?

Errors in starting the pod, because of issues in loading the GCS plugin
This issue is explained in https://discuss.elastic.co/t/error-to-setup-google-cloud-storage-service-auth/142823

Environment

Version information:

eck-operator:0.9.0-rc3
Kubernetes information:

GKE - 1.12.8-gke.10

$ kubectl version

v1.12.8-gke.10

Resource definition:

if relevant insert the resource definition

Logs:

efk-es-8th979b24m elasticsearch "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.node.Node.<init>(Node.java:307) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-8th979b24m elasticsearch "... 6 more",

The text was updated successfully, but these errors were encountered:

thbkrkr · 2019-07-23T15:31:21Z

Hello,

Can you share the resource definition you used to try to reproduce the issue and the full log error (starting by a timestamp)?

gidesh · 2019-07-23T15:59:32Z

This is the resource definition

apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
kind: Elasticsearch
metadata:
  labels:
    app: efk
  name: efk
  namespace: elastic-system
spec:
  secureSettings:
    secretName: "gcs-credentials"
  http:
    service:
      metadata:
        creationTimestamp: null
      spec: {}
    tls:
      certificate: {}
  nodes:
  - config:
      node.data: true
      node.ingest: true
      node.master: false
    nodeCount: 2
    podTemplate:
      metadata:
        creationTimestamp: null
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: cloud.google.com/gke-nodepool
                  operator: In
                  values:
                  - efk-np
                - key: app
                  operator: In
                  values:
                  - efk
        containers: null
        initContainers:
        - command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install --batch repository-gcs
          name: install-plugins
          resources: {}
    volumeClaimTemplates:
    - metadata:
        creationTimestamp: null
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        dataSource: null
        resources:
          requests:
            storage: 150Gi
        storageClassName: elasticsearch-ssd
      status: {}
  - config:
      node.data: false
      node.ingest: false
      node.master: true
    nodeCount: 2
    podTemplate:
      metadata:
        creationTimestamp: null
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: cloud.google.com/gke-nodepool
                  operator: In
                  values:
                  - efk-np
                - key: app
                  operator: In
                  values:
                  - efk
        containers: null
        initContainers:
        - command:
          - sh
          - -c
          - |
            bin/elasticsearch-plugin install --batch repository-gcs
          name: install-plugins
          resources: {}
    volumeClaimTemplates:
    - metadata:
        creationTimestamp: null
        name: master
      spec:
        accessModes:
        - ReadWriteOnce
        dataSource: null
        resources:
          requests:
            storage: 10Gi
        storageClassName: elasticsearch-ssd
      status: {}
  updateStrategy: {}
  version: 7.2.0

and the full log error

$ stern efk-es-kflgmwkjwn                 
+ efk-es-kflgmwkjwn › elastic-internal-init-keystore
efk-es-kflgmwkjwn elastic-internal-init-keystore + echo 'Initializing keystore.'
efk-es-kflgmwkjwn elastic-internal-init-keystore + /usr/share/elasticsearch/bin/elasticsearch-keystore create
efk-es-kflgmwkjwn elastic-internal-init-keystore Initializing keystore.
efk-es-kflgmwkjwn elastic-internal-init-keystore Created elasticsearch keystore in /usr/share/elasticsearch/config
efk-es-kflgmwkjwn elastic-internal-init-keystore + for filename in '/mnt/elastic-internal/secure-settings/*'
efk-es-kflgmwkjwn elastic-internal-init-keystore + [[ -e /mnt/elastic-internal/secure-settings/gcs.client.default.credentials_file ]]
efk-es-kflgmwkjwn elastic-internal-init-keystore ++ basename /mnt/elastic-internal/secure-settings/gcs.client.default.credentials_file
efk-es-kflgmwkjwn elastic-internal-init-keystore + key=gcs.client.default.credentials_file
efk-es-kflgmwkjwn elastic-internal-init-keystore + echo 'Adding gcs.client.default.credentials_file to the keystore.'
efk-es-kflgmwkjwn elastic-internal-init-keystore + /usr/share/elasticsearch/bin/elasticsearch-keystore add gcs.client.default.credentials_file --stdin
efk-es-kflgmwkjwn elastic-internal-init-keystore Adding gcs.client.default.credentials_file to the keystore.
efk-es-kflgmwkjwn elastic-internal-init-keystore + echo 'Keystore initialization successful.'
efk-es-kflgmwkjwn elastic-internal-init-keystore Keystore initialization successful.
+ efk-es-kflgmwkjwn › install-plugins
efk-es-kflgmwkjwn install-plugins -> Downloading repository-gcs from elastic
efk-es-kflgmwkjwn install-plugins WARNING: An illegal reflective access operation has occurred
efk-es-kflgmwkjwn install-plugins WARNING: Illegal reflective access by org.bouncycastle.jcajce.provider.drbg.DRBG (file:/usr/share/elasticsearch/lib/tools/plugin-cli/bcprov-jdk15on-1.61.jar) to constructor sun.security.provider.Sun()
efk-es-kflgmwkjwn install-plugins WARNING: Please consider reporting this to the maintainers of org.bouncycastle.jcajce.provider.drbg.DRBG
efk-es-kflgmwkjwn install-plugins WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
efk-es-kflgmwkjwn install-plugins WARNING: All illegal access operations will be denied in a future release
efk-es-kflgmwkjwn install-plugins @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
efk-es-kflgmwkjwn install-plugins @     WARNING: plugin requires additional permissions     @
efk-es-kflgmwkjwn install-plugins @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
efk-es-kflgmwkjwn install-plugins * java.lang.RuntimePermission accessDeclaredMembers
efk-es-kflgmwkjwn install-plugins * java.lang.RuntimePermission setFactory
efk-es-kflgmwkjwn install-plugins * java.lang.reflect.ReflectPermission suppressAccessChecks
efk-es-kflgmwkjwn install-plugins * java.net.SocketPermission * connect,resolve
efk-es-kflgmwkjwn install-plugins See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
efk-es-kflgmwkjwn install-plugins for descriptions of what these permissions allow and the associated risks.
efk-es-kflgmwkjwn install-plugins -> Installed repository-gcs
+ efk-es-kflgmwkjwn › elasticsearch
efk-es-kflgmwkjwn elasticsearch OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
efk-es-kflgmwkjwn elasticsearch {"type": "server", "timestamp": "2019-07-23T15:58:36,701+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "efk", "node.name": "efk-es-kflgmwkjwn",  "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sde)]], net usable_space [957.3mb], net total_space [975.8mb], types [ext4]"  }
efk-es-kflgmwkjwn elasticsearch {"type": "server", "timestamp": "2019-07-23T15:58:36,707+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "efk", "node.name": "efk-es-kflgmwkjwn",  "message": "heap size [1015.6mb], compressed ordinary object pointers [true]"  }
efk-es-kflgmwkjwn elasticsearch {"type": "server", "timestamp": "2019-07-23T15:58:36,714+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "efk", "node.name": "efk-es-kflgmwkjwn",  "message": "node name [efk-es-kflgmwkjwn], node ID [FhNt8F9zS0yeusRMC05QXg], cluster name [efk]"  }
efk-es-kflgmwkjwn elasticsearch {"type": "server", "timestamp": "2019-07-23T15:58:36,715+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "efk", "node.name": "efk-es-kflgmwkjwn",  "message": "version[7.2.0], pid[1], build[default/docker/508c38a/2019-06-20T15:54:18.811730Z], OS[Linux/4.14.127+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]"  }
efk-es-kflgmwkjwn elasticsearch {"type": "server", "timestamp": "2019-07-23T15:58:36,716+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "efk", "node.name": "efk-es-kflgmwkjwn",  "message": "JVM home [/usr/share/elasticsearch/jdk]"  }
efk-es-kflgmwkjwn elasticsearch {"type": "server", "timestamp": "2019-07-23T15:58:36,716+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "efk", "node.name": "efk-es-kflgmwkjwn",  "message": "JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-5602864441011217531, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Des.cgroups.hierarchy.override=/, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]"  }
efk-es-kflgmwkjwn elasticsearch {"type": "server", "timestamp": "2019-07-23T15:58:41,261+0000", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "efk", "node.name": "efk-es-kflgmwkjwn",  "message": "uncaught exception in thread [main]" , 
efk-es-kflgmwkjwn elasticsearch "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.node.Node.<init>(Node.java:307) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "... 6 more",
efk-es-kflgmwkjwn elasticsearch "Caused by: java.lang.reflect.InvocationTargetException",
efk-es-kflgmwkjwn elasticsearch "at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.node.Node.<init>(Node.java:307) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "... 6 more",
efk-es-kflgmwkjwn elasticsearch "Caused by: java.io.UncheckedIOException: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: java.io.ByteArrayInputStream@60f21960; line: 1, column: 1])",
efk-es-kflgmwkjwn elasticsearch " at [Source: java.io.ByteArrayInputStream@60f21960; line: 1, column: 3]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.loadCredential(GoogleCloudStorageClientSettings.java:203) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.getClientSettings(GoogleCloudStorageClientSettings.java:163) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.load(GoogleCloudStorageClientSettings.java:151) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.reload(GoogleCloudStoragePlugin.java:79) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.<init>(GoogleCloudStoragePlugin.java:45) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.node.Node.<init>(Node.java:307) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "... 6 more",
efk-es-kflgmwkjwn elasticsearch "Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: java.io.ByteArrayInputStream@60f21960; line: 1, column: 1])",
efk-es-kflgmwkjwn elasticsearch " at [Source: java.io.ByteArrayInputStream@60f21960; line: 1, column: 3]",
efk-es-kflgmwkjwn elasticsearch "at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:483) ~[jackson-core-2.8.11.jar:2.8.11]",
efk-es-kflgmwkjwn elasticsearch "at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:535) ~[jackson-core-2.8.11.jar:2.8.11]",
efk-es-kflgmwkjwn elasticsearch "at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:547) ~[jackson-core-2.8.11.jar:2.8.11]",
efk-es-kflgmwkjwn elasticsearch "at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:2931) ~[jackson-core-2.8.11.jar:2.8.11]",
efk-es-kflgmwkjwn elasticsearch "at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:731) ~[jackson-core-2.8.11.jar:2.8.11]",
efk-es-kflgmwkjwn elasticsearch "at com.google.api.client.json.jackson2.JacksonParser.nextToken(JacksonParser.java:55) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at com.google.api.client.json.JsonParser.startParsingObjectOrArray(JsonParser.java:243) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at com.google.api.client.json.JsonParser.parse(JsonParser.java:442) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:781) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at com.google.api.client.json.JsonParser.parse(JsonParser.java:382) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at com.google.api.client.json.JsonParser.parse(JsonParser.java:355) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:87) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:81) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:336) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at com.google.auth.oauth2.ServiceAccountCredentials.fromStream(ServiceAccountCredentials.java:316) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.lambda$loadCredential$7(GoogleCloudStorageClientSettings.java:195) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at java.security.AccessController.doPrivileged(AccessController.java:551) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.SocketAccess.doPrivilegedIOException(SocketAccess.java:44) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.loadCredential(GoogleCloudStorageClientSettings.java:194) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.getClientSettings(GoogleCloudStorageClientSettings.java:163) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.GoogleCloudStorageClientSettings.load(GoogleCloudStorageClientSettings.java:151) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.reload(GoogleCloudStoragePlugin.java:79) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin.<init>(GoogleCloudStoragePlugin.java:45) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.node.Node.<init>(Node.java:307) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]",
efk-es-kflgmwkjwn elasticsearch "... 6 more"] }

thbkrkr · 2019-07-23T16:25:30Z

I think the issue is caused because Elasticsearch expects the value of the gcs.client.default.credentials_file key in the gcs-credentials secret to be compact JSON data.

You can compact JSON using jq:

jq -cM . jsonfile > compactedjsonfile

gidesh · 2019-07-24T09:06:25Z

Is there a way elastic operator can be be correct to take in non-compacted json key as we generate the json key through terraform. If not possible let us know we will proceed with the alternative through terraform.
It worked in 0.8.0 without compacted JSON

thbkrkr · 2019-07-24T17:55:51Z

It worked in 0.8.0 without compacted JSON

Thank you for noticing this! I've investigated and found that we changed this behaviour when we changed the way the secure settings are added to the Elasticsearch keystore (#1221).

Before, the version that works with uncompacted JSON:

elasticsearch-keystore add-file key jsonfile

Now, the version that works only with compacted JSON:

elasticsearch-keystore add key --stdin < jsonfile

I have to check but I think we could go back to the previous version so that you have nothing to do on your side.

Given it's convenient to not use `--stdin` with the ES keystore to fix elastic#1349, this commit makes the keystore settings addition different for ES than for Kibana and the APM Server. Only the Elasticsearch keystore supports settings additions using a file path. Kibana and APM Server supports only to pass the value through stdin and use the --stdin flag.

Given it's convenient to not use `--stdin` with the ES keystore to fix #1349, this commit makes the keystore settings addition different for ES than for Kibana and the APM Server. Only the Elasticsearch keystore supports settings additions using a file path. Kibana and APM Server supports only to pass the value through stdin and use the --stdin flag.

Given it's convenient to not use `--stdin` with the ES keystore to fix elastic#1349, this commit makes the keystore settings addition different for ES than for Kibana and the APM Server. Only the Elasticsearch keystore supports settings additions using a file path. Kibana and APM Server supports only to pass the value through stdin and use the --stdin flag.

Given it's convenient to not use `--stdin` with the ES keystore to fix #1349, this commit makes the keystore settings addition different for ES than for Kibana and the APM Server. Only the Elasticsearch keystore supports settings additions using a file path. Kibana and APM Server supports only to pass the value through stdin and use the --stdin flag.

anyasabo mentioned this issue Jul 23, 2019

Update pod template docs #1245

Merged

thbkrkr added the >bug Something isn't working label Jul 24, 2019

thbkrkr mentioned this issue Jul 24, 2019

Update the way to add settings in the ES keystore #1377

Merged

thbkrkr closed this as completed in #1377 Jul 25, 2019

thbkrkr mentioned this issue Jul 25, 2019

Parameterize keystore settings addition #1401

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GCS Repository plugin not loading #1349

GCS Repository plugin not loading #1349

gidesh commented Jul 23, 2019

thbkrkr commented Jul 23, 2019

gidesh commented Jul 23, 2019 •

edited by thbkrkr

Loading

thbkrkr commented Jul 23, 2019

gidesh commented Jul 24, 2019 •

edited

Loading

thbkrkr commented Jul 24, 2019

GCS Repository plugin not loading #1349

GCS Repository plugin not loading #1349

Comments

gidesh commented Jul 23, 2019

Bug Report

thbkrkr commented Jul 23, 2019

gidesh commented Jul 23, 2019 • edited by thbkrkr Loading

thbkrkr commented Jul 23, 2019

gidesh commented Jul 24, 2019 • edited Loading

thbkrkr commented Jul 24, 2019

gidesh commented Jul 23, 2019 •

edited by thbkrkr

Loading

gidesh commented Jul 24, 2019 •

edited

Loading