Failing test: Jest Integration Tests.x-pack/test_utils/jest/integration_tests - example integration test with kbn server should have started new platform server correctly #32795

kibanamachine · 2019-03-08T18:55:49Z

A test failed on a tracked branch

00:13:28.049  -> Running jest integration tests
00:13:40.922      │ info starting elasticsearch
00:13:40.922      │ info Installing from snapshot
00:13:40.923          │ info version: 8.0.0
00:13:40.923          │ info install path: /var/lib/jenkins/workspace/elastic+kibana+master/JOB/x-pack-intake/node/immutable/kibana/.es/test-mdab8p7l3s
00:13:40.924          │ info license: trial
00:13:40.924          │ info downloading from https://snapshots.elastic.co/downloads/elasticsearch/elasticsearch-8.0.0-SNAPSHOT-linux-x86_64.tar.gz
00:14:08.211          │ info etags match, using cache from 2019-03-09T20:04:34.166Z
00:14:08.212          │ info extracting /var/lib/jenkins/workspace/elastic+kibana+master/JOB/x-pack-intake/node/immutable/kibana/.es/cache/elasticsearch-8.0.0-SNAPSHOT-linux-x86_64.tar.gz
00:14:15.386          │ info extracted to /var/lib/jenkins/workspace/elastic+kibana+master/JOB/x-pack-intake/node/immutable/kibana/.es/test-mdab8p7l3s
00:14:15.386          │ info setting bootstrap password to iamsuperuser
00:14:19.035      │ info Starting
00:14:19.035          │ debg bin/elasticsearch -E cluster.name=test-mdab8p7l3s -E http.port=9220 -E discovery.type=single-node
00:14:19.036          │ info warning: ignoring JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8
00:14:19.647          │ERROR Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
00:14:19.648          │      
00:14:22.334          │ info [o.e.e.NodeEnvironment] [kibana-ci-immutable-centos-1552194070705507055] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [29.7gb], net total_space [49.9gb], types [rootfs]
00:14:22.335          │ info [o.e.e.NodeEnvironment] [kibana-ci-immutable-centos-1552194070705507055] heap size [990.7mb], compressed ordinary object pointers [true]
00:14:22.336          │ info [o.e.n.Node] [kibana-ci-immutable-centos-1552194070705507055] node name [kibana-ci-immutable-centos-1552194070705507055], node ID [3n3UqNX1RVKlBzLnehn6Lw]
00:14:22.337          │ info [o.e.n.Node] [kibana-ci-immutable-centos-1552194070705507055] version[8.0.0-SNAPSHOT], pid[5594], build[default/tar/92ac431/2019-03-09T08:50:12.216901Z], OS[Linux/3.10.0-957.5.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/11.0.2/11.0.2+9-LTS]
00:14:22.339          │ info [o.e.n.Node] [kibana-ci-immutable-centos-1552194070705507055] JVM Home [/var/lib/jenkins/.java/oracle-11.0.2+9-linux]
00:14:22.340          │ info [o.e.n.Node] [kibana-ci-immutable-centos-1552194070705507055] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-14991561783714315698, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Dio.netty.allocator.type=unpooled, -Des.path.home=/var/lib/jenkins/workspace/elastic+kibana+master/JOB/x-pack-intake/node/immutable/kibana/.es/test-mdab8p7l3s, -Des.path.conf=/var/lib/jenkins/workspace/elastic+kibana+master/JOB/x-pack-intake/node/immutable/kibana/.es/test-mdab8p7l3s/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
00:14:22.346          │ info [o.e.n.Node] [kibana-ci-immutable-centos-1552194070705507055] version [8.0.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
00:14:25.078          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [aggs-matrix-stats]
00:14:25.079          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [analysis-common]
00:14:25.081          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [data-frame]
00:14:25.081          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [ingest-common]
00:14:25.082          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [ingest-geoip]
00:14:25.083          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [ingest-user-agent]
00:14:25.084          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [lang-expression]
00:14:25.085          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [lang-mustache]
00:14:25.086          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [lang-painless]
00:14:25.086          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [mapper-extras]
00:14:25.087          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [parent-join]
00:14:25.088          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [percolator]
00:14:25.089          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [rank-eval]
00:14:25.090          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [reindex]
00:14:25.091          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [repository-url]
00:14:25.092          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [transport-netty4]
00:14:25.093          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-ccr]
00:14:25.094          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-core]
00:14:25.095          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-deprecation]
00:14:25.096          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-graph]
00:14:25.097          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-ilm]
00:14:25.098          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-logstash]
00:14:25.099          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-ml]
00:14:25.100          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-monitoring]
00:14:25.101          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-rollup]
00:14:25.101          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-security]
00:14:25.102          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-sql]
00:14:25.103          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-upgrade]
00:14:25.104          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] loaded module [x-pack-watcher]
00:14:25.104          │ info [o.e.p.PluginsService] [kibana-ci-immutable-centos-1552194070705507055] no plugins loaded
00:14:29.978          │ info [o.e.x.s.a.s.FileRolesStore] [kibana-ci-immutable-centos-1552194070705507055] parsed [0] roles from file [/var/lib/jenkins/workspace/elastic+kibana+master/JOB/x-pack-intake/node/immutable/kibana/.es/test-mdab8p7l3s/config/roles.yml]
00:14:30.960          │ info [o.e.x.m.p.l.CppLogMessageHandler] [kibana-ci-immutable-centos-1552194070705507055] [controller/5674] [Main.cc@110] controller (64 bit): Version 8.0.0-SNAPSHOT (Build c63e47a5eac6bd) Copyright (c) 2019 Elasticsearch BV
00:14:31.502          │ info [o.e.a.ActionModule] [kibana-ci-immutable-centos-1552194070705507055] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
00:14:31.874          │ info [o.e.d.DiscoveryModule] [kibana-ci-immutable-centos-1552194070705507055] using discovery type [single-node] and seed hosts providers [settings]
00:14:33.058          │ info [o.e.n.Node] [kibana-ci-immutable-centos-1552194070705507055] initialized
00:14:33.059          │ info [o.e.n.Node] [kibana-ci-immutable-centos-1552194070705507055] starting ...
00:14:33.262          │ info [o.e.t.TransportService] [kibana-ci-immutable-centos-1552194070705507055] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
00:14:33.408          │ info [o.e.h.AbstractHttpServerTransport] [kibana-ci-immutable-centos-1552194070705507055] publish_address {127.0.0.1:9220}, bound_addresses {[::1]:9220}, {127.0.0.1:9220}
00:14:33.409          │ info [o.e.n.Node] [kibana-ci-immutable-centos-1552194070705507055] started
00:14:33.865          │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-centos-1552194070705507055] failed to put user [kibana_user]
00:14:33.866          │      org.elasticsearch.ElasticsearchStatusException: Cluster state has not been recovered yet, cannot write to the security index
00:14:33.867          │      	at org.elasticsearch.xpack.security.support.SecurityIndexManager.prepareIndexIfNeededThenExecute(SecurityIndexManager.java:316) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.868          │      	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.indexUser(NativeUsersStore.java:369) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.869          │      	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.putUser(NativeUsersStore.java:315) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.870          │      	at org.elasticsearch.xpack.security.action.user.TransportPutUserAction.doExecute(TransportPutUserAction.java:50) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.871          │      	at org.elasticsearch.xpack.security.action.user.TransportPutUserAction.doExecute(TransportPutUserAction.java:31) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.872          │      	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.873          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.874          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.875          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:171) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.876          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.877          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:237) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.878          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:599) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.879          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:574) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.880          │      	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.882          │      	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:140) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.882          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:239) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.884          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:223) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.885          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:189) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.886          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.887          │      	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.888          │      	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.889          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.890          │      	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:229) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.891          │      	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.892          │      	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.893          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:191) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.894          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:171) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.895          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:157) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.896          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.897          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:245) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.898          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.899          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.901          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.902          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.903          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.904          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:154) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.905          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:107) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.906          │      	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.907          │      	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.908          │      	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.910          │      	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.910          │      	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.911          │      	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.912          │      	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:70) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.913          │      	at org.elasticsearch.xpack.security.rest.action.user.RestPutUserAction.lambda$innerPrepareRequest$0(RestPutUserAction.java:67) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.914          │      	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:113) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.915          │      	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:58) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.916          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.917          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$24(AuthenticationService.java:555) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.918          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:564) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.919          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.finishAuthentication(AuthenticationService.java:545) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.920          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:495) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.921          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$16(AuthenticationService.java:402) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.922          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.923          │      	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.924          │      	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.925          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:372) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.926          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.927          │      	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$3(CachingUsernamePasswordRealm.java:175) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.928          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.929          │      	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.lambda$doAuthenticate$0(ReservedRealm.java:115) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.930          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.930          │      	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:210) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.931          │      	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:93) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.932          │      	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:166) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.933          │      	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:103) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.934          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:363) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.935          │      	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.936          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:406) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.937          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:334) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.939          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:344) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.940          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:287) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.941          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.942          │      	at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:314) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.943          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:268) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.944          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:251) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.945          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.946          │      	at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:347) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.947          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:247) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.948          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.949          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.950          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.951          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.952          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:121) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.953          │      	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:55) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.954          │      	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.955          │      	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:337) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.955          │      	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.956          │      	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:317) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.957          │      	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:367) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.958          │      	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:296) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.959          │      	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:66) [transport-netty4-client-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.960          │      	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.961          │      	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.962          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.963          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.964          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.965          │      	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:33.965          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.966          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.967          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.968          │      	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:33.969          │      	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:33.970          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.971          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.972          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.973          │      	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:33.974          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.975          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.976          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.977          │      	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:33.977          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.978          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.979          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.980          │      	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:33.981          │      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:33.982          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.983          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.983          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.984          │      	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
00:14:33.985          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.986          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.987          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.988          │      	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.989          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.990          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.991          │      	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.992          │      	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.993          │      	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.994          │      	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.995          │      	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.995          │      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:33.996          │      	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
00:14:33.997          │      	at java.lang.Thread.run(Thread.java:834) [?:?]
00:14:33.998          │ info [r.suppressed] [kibana-ci-immutable-centos-1552194070705507055] path: /_security/user/kibana_user, params: {username=kibana_user}
00:14:33.999          │      org.elasticsearch.ElasticsearchStatusException: Cluster state has not been recovered yet, cannot write to the security index
00:14:34.000          │      	at org.elasticsearch.xpack.security.support.SecurityIndexManager.prepareIndexIfNeededThenExecute(SecurityIndexManager.java:316) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.001          │      	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.indexUser(NativeUsersStore.java:369) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.002          │      	at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.putUser(NativeUsersStore.java:315) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.003          │      	at org.elasticsearch.xpack.security.action.user.TransportPutUserAction.doExecute(TransportPutUserAction.java:50) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.004          │      	at org.elasticsearch.xpack.security.action.user.TransportPutUserAction.doExecute(TransportPutUserAction.java:31) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.005          │      	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:145) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.005          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:86) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.006          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.007          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:171) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.008          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.009          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorizeAction$4(AuthorizationService.java:237) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.010          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:599) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.011          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService$AuthorizationResultListener.onResponse(AuthorizationService.java:574) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.012          │      	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.013          │      	at org.elasticsearch.xpack.security.authz.RBACEngine.authorizeClusterAction(RBACEngine.java:140) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.014          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeAction(AuthorizationService.java:239) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.015          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService.maybeAuthorizeRunAs(AuthorizationService.java:223) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.016          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$authorize$1(AuthorizationService.java:189) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.017          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.018          │      	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.019          │      	at org.elasticsearch.xpack.security.authz.RBACEngine.lambda$resolveAuthorizationInfo$1(RBACEngine.java:113) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.020          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.021          │      	at org.elasticsearch.xpack.security.authz.store.CompositeRolesStore.getRoles(CompositeRolesStore.java:229) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.022          │      	at org.elasticsearch.xpack.security.authz.RBACEngine.getRoles(RBACEngine.java:119) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.023          │      	at org.elasticsearch.xpack.security.authz.RBACEngine.resolveAuthorizationInfo(RBACEngine.java:107) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.024          │      	at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:191) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.025          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.authorizeRequest(SecurityActionFilter.java:171) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.026          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:157) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.027          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.028          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:245) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.029          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.030          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.031          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.032          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.033          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:138) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.035          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:154) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.036          │      	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:107) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.037          │      	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:143) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.038          │      	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:121) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.039          │      	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.039          │      	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.040          │      	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.041          │      	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.042          │      	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:70) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.043          │      	at org.elasticsearch.xpack.security.rest.action.user.RestPutUserAction.lambda$innerPrepareRequest$0(RestPutUserAction.java:67) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.044          │      	at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:113) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.045          │      	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.lambda$handleRequest$0(SecurityRestFilter.java:58) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.046          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.047          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$writeAuthToContext$24(AuthenticationService.java:555) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.048          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.writeAuthToContext(AuthenticationService.java:564) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.049          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.finishAuthentication(AuthenticationService.java:545) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.051          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:495) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.051          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$16(AuthenticationService.java:402) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.053          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.053          │      	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.054          │      	at org.elasticsearch.xpack.core.common.IteratingActionListener.onResponse(IteratingActionListener.java:120) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.055          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$13(AuthenticationService.java:372) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.056          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.057          │      	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$3(CachingUsernamePasswordRealm.java:175) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.058          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.059          │      	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.lambda$doAuthenticate$0(ReservedRealm.java:115) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.060          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.061          │      	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:210) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.062          │      	at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:93) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.063          │      	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:166) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.064          │      	at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:103) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.065          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$15(AuthenticationService.java:363) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.066          │      	at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102) [x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.067          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:406) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.068          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$11(AuthenticationService.java:334) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.069          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:344) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.070          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$3(AuthenticationService.java:287) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.071          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.072          │      	at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:314) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.073          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:268) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.074          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:251) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.075          │      	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.076          │      	at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:347) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.077          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:247) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.078          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:305) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.079          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:316) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.080          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:243) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.081          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:195) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.083          │      	at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:121) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.084          │      	at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:55) [x-pack-security-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.085          │      	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.086          │      	at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:337) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.087          │      	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.088          │      	at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:317) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.089          │      	at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:367) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.090          │      	at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:296) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.091          │      	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:66) [transport-netty4-client-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.091          │      	at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.092          │      	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.093          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.094          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.095          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.096          │      	at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
00:14:34.097          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.098          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.099          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.100          │      	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:34.101          │      	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:34.102          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.103          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.104          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.105          │      	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:34.106          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.107          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.108          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.109          │      	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:34.110          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.111          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.112          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.113          │      	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:34.114          │      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
00:14:34.115          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.116          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.117          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.118          │      	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
00:14:34.119          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.120          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.121          │      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.121          │      	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.122          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.123          │      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.124          │      	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.125          │      	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.126          │      	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.126          │      	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.127          │      	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.128          │      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
00:14:34.129          │      	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
00:14:34.129          │      	at java.lang.Thread.run(Thread.java:834) [?:?]
00:14:34.130          │ info [o.e.g.GatewayService] [kibana-ci-immutable-centos-1552194070705507055] recovered [0] indices into cluster_state
00:14:34.309          │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-1552194070705507055] adding template [.triggered_watches] for index patterns [.triggered_watches*]
00:14:34.310          │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-1552194070705507055] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
00:14:34.434          │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-1552194070705507055] adding template [.watches] for index patterns [.watches*]
00:14:34.435          │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-1552194070705507055] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
00:14:34.543          │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-1552194070705507055] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
00:14:34.544          │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-1552194070705507055] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
00:14:34.544          │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-1552194070705507055] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
00:14:34.670          │ info [o.e.c.m.MetaDataIndexTemplateService] [kibana-ci-immutable-centos-1552194070705507055] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
00:14:34.671          │ info [o.e.x.i.a.TransportPutLifecycleAction] [kibana-ci-immutable-centos-1552194070705507055] adding index lifecycle policy [watch-history-ilm-policy]
00:14:34.837          │ info [o.e.l.LicenseService] [kibana-ci-immutable-centos-1552194070705507055] license [f448d8cf-3ee4-491e-9ec5-27210af4659d] mode [trial] - valid
00:18:20.873  FAIL  test_utils/jest/integration_tests/example_integration.test.ts (288.987s)
00:18:20.874   example integration test with kbn server
00:18:20.875     ✕ should have started new platform server correctly (2ms)
00:18:20.875 
00:18:20.876   ● example integration test with kbn server › should have started new platform server correctly
00:18:20.876 
00:18:20.876     Timeout - Async callback was not invoked within the 280000ms timeout specified by jest.setTimeout.Error: Timeout - Async callback was not invoked within the 280000ms timeout specified by jest.setTimeout.
00:18:20.878 
00:18:20.878       at mapper (../node_modules/jest-jasmine2/build/queueRunner.js:25:45)
00:18:20.879 
00:18:20.879   ● example integration test with kbn server › should have started new platform server correctly
00:18:20.880 
00:18:20.880     TypeError: Cannot read property 'kbnServer' of null
00:18:20.880 
00:18:20.880       23 | 
00:18:20.880       24 |   it('should have started new platform server correctly', () => {
00:18:20.881     > 25 |     expect(servers.kbnServer).toBeDefined();
00:18:20.882          |                    ^
00:18:20.882       26 |     expect(servers.kbnServer.server).toBeDefined();
00:18:20.883       27 |     expect(servers.kbnServer.server.plugins).toBeDefined();
00:18:20.884       28 |   });
00:18:20.884 
00:18:20.884       at Object.it (test_utils/jest/integration_tests/example_integration.test.ts:25:20)
00:18:20.885 
00:18:20.889 Test Suites: 1 failed, 1 total
00:18:20.889 Tests:       1 failed, 1 total
00:18:20.890 Snapshots:   0 total
00:18:20.890 Time:        290.359s
00:18:20.891 Ran all test suites.
00:18:21.898 Jest did not exit one second after the test run has completed.
00:18:21.898 
00:18:21.898 This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with `--detectOpenHandles` to troubleshoot this issue.
03:00:45.386 Build timed out (after 180 minutes). Marking the build as failed.
03:00:45.410 Build was aborted

First failure: Jenkins Build

The text was updated successfully, but these errors were encountered:

elasticmachine · 2019-03-08T18:55:51Z

Pinging @elastic/kibana-test-triage

kibanamachine · 2019-03-08T19:52:56Z

New failure: Jenkins Build

spalger · 2019-03-08T19:55:35Z

Fixed by #32803

kibanamachine · 2019-03-10T08:02:31Z

New failure: Jenkins Build

kibanamachine · 2019-03-10T10:33:33Z

New failure: Jenkins Build

spalger · 2019-03-10T19:39:59Z

servers variable is undefined, so I'm assuming the beforeAll() hook timed out and the test was still run?
jest.setTimeout(t) is used to change the timeout but the jest docs say that "The method must be called after the test framework is installed in the environment and before the test runs.", not sure if that could be related to this strange behavior
Cluster state has not been recovered yet, cannot write to the security index

I wonder if that means the users are not being created properly and that's preventing the server from starting
server logging is disabled so we have no idea what's going on
Since the servers object is undefined we can't stop the servers, but it seems that they are still listening for connections or something so the process stays open and Jenkins times out the build after 180 minutes, making this a very slow failure.

njd5475 · 2019-03-11T15:23:56Z

I have been looking into some things we can do to make sure that the server does not continue to run if the startTestServer does not return.

Firstly, we can wrap the setupUsers in a try...catch and shutdown the test es cluster if that fails. Also bubble up an exception from that catch.
Secondly, I am thinking we should create a timeout function around the elasticsearch.waitUntilReady That would stop the test cluster possibly before the beforeAll times out so that we aren't waiting on the jenkins job timeout to fail the test

@spalger What do you think about these options?

spalger · 2019-03-11T20:31:27Z

Skipped

master: f71c091 + 33dc8e6
7.x/7.1: 42bb4e9 + 75dae0a
7.0: 46a35c3 + a297d9c

spalger · 2019-03-11T20:35:37Z

@njd5475 I definitely think that failures in startup should be caught by try...catch and trigger a shutdown of already started components, but the primary issue I see is that until there is success we can't trigger a shutdown if jest aborts the test due to timeout. for this we'll need something like an AbortController, or a cancelable promise, so we can shutdown the servers when they take too long to start up.

elasticmachine · 2019-04-30T23:54:15Z

Pinging @elastic/kibana-platform

spalger · 2019-05-09T19:29:50Z

Will reopen if it fails again.

kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Mar 8, 2019

spalger closed this as completed Mar 8, 2019

kibanamachine reopened this Mar 10, 2019

spalger self-assigned this Mar 10, 2019

spalger mentioned this issue Mar 11, 2019

temporarily skip flaky test #32922

Merged

elasticmachine mentioned this issue Mar 12, 2019

[Uptime] Fix flaky behavior and re-enable Uptime API tests #32819

Merged

7 tasks

cqliu1 added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Apr 30, 2019

spalger closed this as completed May 9, 2019

mshustov mentioned this issue Dec 16, 2019

[Discuss] Functional tests for NP client side #52768

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing test: Jest Integration Tests.x-pack/test_utils/jest/integration_tests - example integration test with kbn server should have started new platform server correctly #32795

Failing test: Jest Integration Tests.x-pack/test_utils/jest/integration_tests - example integration test with kbn server should have started new platform server correctly #32795

kibanamachine commented Mar 8, 2019 •

edited by spalger

Loading

elasticmachine commented Mar 8, 2019

kibanamachine commented Mar 8, 2019

spalger commented Mar 8, 2019

kibanamachine commented Mar 10, 2019

kibanamachine commented Mar 10, 2019

spalger commented Mar 10, 2019 •

edited

Loading

njd5475 commented Mar 11, 2019

spalger commented Mar 11, 2019 •

edited

Loading

spalger commented Mar 11, 2019

elasticmachine commented Apr 30, 2019

spalger commented May 9, 2019

Failing test: Jest Integration Tests.x-pack/test_utils/jest/integration_tests - example integration test with kbn server should have started new platform server correctly #32795

Failing test: Jest Integration Tests.x-pack/test_utils/jest/integration_tests - example integration test with kbn server should have started new platform server correctly #32795

Comments

kibanamachine commented Mar 8, 2019 • edited by spalger Loading

elasticmachine commented Mar 8, 2019

kibanamachine commented Mar 8, 2019

spalger commented Mar 8, 2019

kibanamachine commented Mar 10, 2019

kibanamachine commented Mar 10, 2019

spalger commented Mar 10, 2019 • edited Loading

njd5475 commented Mar 11, 2019

spalger commented Mar 11, 2019 • edited Loading

spalger commented Mar 11, 2019

elasticmachine commented Apr 30, 2019

spalger commented May 9, 2019

kibanamachine commented Mar 8, 2019 •

edited by spalger

Loading

spalger commented Mar 10, 2019 •

edited

Loading

spalger commented Mar 11, 2019 •

edited

Loading