Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kibana crashes on startup if elasticsearch is started simultaneously #33689

Closed
redbaron4 opened this issue Mar 22, 2019 · 10 comments
Closed

Kibana crashes on startup if elasticsearch is started simultaneously #33689

redbaron4 opened this issue Mar 22, 2019 · 10 comments
Labels
Team:Operations Team label for Operations Team

Comments

@redbaron4
Copy link

Kibana version: 6.6.0

Elasticsearch version: 6.6.0

Server OS version: CentOS-6.8

Browser version: firefox-52ESR

Browser OS version: Any

Original install method (e.g. download page, yum, from source, etc.): Install from RPM provided in download page

Describe the bug: Starting up Kibana and elasticsearch simultaneously causes kibana to exit with fatal error. This is almost identical to #25027 except that I faced this issue in 6.6.0.

I ran into this bug when our infrastructure was restarted after a power shutdown. Elasticsearch and Kibana are hosted on different machines. ES went into "red" (unallocated primary shards) on startup and took a while to complete recovery. Meanwhile, kibana started and crashed with the logs given below. My guess is that .kibana shards were unalloocated and the index was unusable.

Steps to reproduce:

  1. Shutdown kibana and elasticsearch services
  2. Startup elasticsearch so that it is in red (Maybe don't start the nodes that have .kibana shard)
  3. Startup kibana and see it crash

Expected behavior: Kibana should remain running even if it cannot access its own data on ES and show a similar prompt on browser.

Provide logs and/or server output (if relevant):

The logs for stdout

{"type":"log","@timestamp":"2019-03-19T03:43:33Z","tags":["plugin","warning"],"pid":1432,"path":"/usr/share/kibana/src/legacy/core_plugins/ems_util","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/ems_util"}
{"type":"log","@timestamp":"2019-03-19T03:43:39Z","tags":["warning","elasticsearch","config","deprecation"],"pid":1432,"message":"Config key \"elasticsearch.url\" is deprecated. It has been replaced with \"elasticsearch.hosts\""}
{"type":"log","@timestamp":"2019-03-19T03:43:43Z","tags":["status","plugin:kibana@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:elasticsearch@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:xpack_main@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:graph@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:monitoring@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:spaces@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["security","warning"],"pid":1432,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["security","warning"],"pid":1432,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:security@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:searchprofiler@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:ml@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:tilemap@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:watcher@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:grokdebugger@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:dashboard_mode@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:logstash@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:beats_management@6.6.0","info"],"pid":1432,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:apm@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:elasticsearch@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["license","info","xpack"],"pid":1432,"message":"Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:xpack_main@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:graph@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:searchprofiler@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:ml@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:tilemap@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:watcher@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:grokdebugger@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:logstash@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:beats_management@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["info","monitoring-ui","kibana-monitoring"],"pid":1432,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["status","plugin:security@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"error","@timestamp":"2019-03-19T03:43:44Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]","name":"Error","stack":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]] :: {\"path\":\"/.kibana/doc/config%3A6.6.0\",\"query\":{},\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]\\\"}],\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]\\\"},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]"}
{"type":"log","@timestamp":"2019-03-19T03:43:44Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from kibana_settings collector"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["license","info","xpack"],"pid":1432,"message":"Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active"}
{"type":"error","@timestamp":"2019-03-19T03:43:45Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]","name":"Error","stack":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]] :: {\"path\":\"/.kibana/doc/kql-telemetry%3Akql-telemetry\",\"query\":{},\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]\\\"}],\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]\\\"},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from kql collector"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:interpreter@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:canvas@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:license_management@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:index_management@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:console@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:console_extensions@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:notifications@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:index_lifecycle_management@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:infra@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:rollup@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:remote_clusters@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:cross_cluster_replication@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:upgrade_assistant@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:45Z","tags":["status","plugin:metrics@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:47Z","tags":["status","plugin:timelion@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["reporting","warning"],"pid":1432,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:reporting@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:xpack_main@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:graph@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:spaces@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from yellow to red - Request Timeout after 3000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:security@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:searchprofiler@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:ml@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:tilemap@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:watcher@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:grokdebugger@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:logstash@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:beats_management@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:index_management@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:index_lifecycle_management@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:rollup@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:remote_clusters@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:cross_cluster_replication@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:reporting@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:elasticsearch@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from green to red - Request Timeout after 3000ms","prevState":"green","prevMsg":"Ready"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:xpack_main@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:graph@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:searchprofiler@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:ml@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:tilemap@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:watcher@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:grokdebugger@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:logstash@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:beats_management@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:index_management@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:index_lifecycle_management@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:rollup@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:remote_clusters@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:cross_cluster_replication@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:43:52Z","tags":["status","plugin:reporting@6.6.0","info"],"pid":1432,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"error","@timestamp":"2019-03-19T03:43:54Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]","name":"Error","stack":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]] :: {\"path\":\"/.kibana/doc/kql-telemetry%3Akql-telemetry\",\"query\":{},\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]\\\"}],\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]\\\"},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]"}
{"type":"log","@timestamp":"2019-03-19T03:43:54Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from kql collector"}
{"type":"error","@timestamp":"2019-03-19T03:43:54Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]","name":"Error","stack":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]] :: {\"path\":\"/.kibana/doc/config%3A6.6.0\",\"query\":{},\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]\\\"}],\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]\\\"},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]"}
{"type":"log","@timestamp":"2019-03-19T03:43:54Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from kibana_settings collector"}
{"type":"error","@timestamp":"2019-03-19T03:43:54Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{\"ignore_unavailable\":true,\"filter_path\":\"aggregations.types.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"query\\\":{\\\"terms\\\":{\\\"type\\\":[\\\"dashboard\\\",\\\"visualization\\\",\\\"search\\\",\\\"index-pattern\\\",\\\"graph-workspace\\\",\\\"timelion-sheet\\\"]}},\\\"aggs\\\":{\\\"types\\\":{\\\"terms\\\":{\\\"field\\\":\\\"type\\\",\\\"size\\\":6}}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"error","@timestamp":"2019-03-19T03:43:54Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{\"ignore_unavailable\":true,\"filter_path\":\"aggregations.types.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"query\\\":{\\\"terms\\\":{\\\"type\\\":[\\\"dashboard\\\",\\\"visualization\\\",\\\"search\\\",\\\"index-pattern\\\",\\\"graph-workspace\\\",\\\"timelion-sheet\\\"]}},\\\"aggs\\\":{\\\"types\\\":{\\\"terms\\\":{\\\"field\\\":\\\"type\\\",\\\"size\\\":6}}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-03-19T03:43:54Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from kibana collector"}
{"type":"error","@timestamp":"2019-03-19T03:43:54Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{\"size\":1000,\"ignore_unavailable\":true,\"filter_path\":\"hits.hits._id\"},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"filter\\\":{\\\"term\\\":{\\\"index-pattern.type\\\":\\\"rollup\\\"}}}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-03-19T03:43:54Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from rollups collector"}
{"type":"error","@timestamp":"2019-03-19T03:43:54Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.reporting-*/_search\",\"query\":{\"rest_total_hits_as_int\":true,\"filter_path\":\"hits.total,aggregations.jobTypes.buckets,aggregations.objectTypes.buckets,aggregations.layoutTypes.buckets,aggregations.statusTypes.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"aggs\\\":{\\\"jobTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"jobtype\\\",\\\"size\\\":3}},\\\"objectTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"meta.objectType.keyword\\\",\\\"size\\\":3}},\\\"layoutTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"meta.layout.keyword\\\",\\\"size\\\":3}},\\\"statusTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"status\\\",\\\"size\\\":4}}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"error","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]","name":"Error","stack":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]] :: {\"path\":\"/.kibana/doc/kql-telemetry%3Akql-telemetry\",\"query\":{},\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]\\\"}],\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]\\\"},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][kql-telemetry:kql-telemetry]: routing [null]]"}
{"type":"log","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from kql collector"}
{"type":"error","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{\"ignore_unavailable\":true,\"filter_path\":\"aggregations.types.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"query\\\":{\\\"terms\\\":{\\\"type\\\":[\\\"dashboard\\\",\\\"visualization\\\",\\\"search\\\",\\\"index-pattern\\\",\\\"graph-workspace\\\",\\\"timelion-sheet\\\"]}},\\\"aggs\\\":{\\\"types\\\":{\\\"terms\\\":{\\\"field\\\":\\\"type\\\",\\\"size\\\":6}}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from kibana collector"}
{"type":"error","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{\"ignore_unavailable\":true,\"filter_path\":\"aggregations.types.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"query\\\":{\\\"terms\\\":{\\\"type\\\":[\\\"dashboard\\\",\\\"visualization\\\",\\\"search\\\",\\\"index-pattern\\\",\\\"graph-workspace\\\",\\\"timelion-sheet\\\"]}},\\\"aggs\\\":{\\\"types\\\":{\\\"terms\\\":{\\\"field\\\":\\\"type\\\",\\\"size\\\":6}}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from kibana collector"}
{"type":"error","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{\"size\":1000,\"ignore_unavailable\":true,\"filter_path\":\"hits.hits._id\"},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"filter\\\":{\\\"term\\\":{\\\"index-pattern.type\\\":\\\"rollup\\\"}}}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from rollups collector"}
{"type":"error","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.reporting-*/_search\",\"query\":{\"rest_total_hits_as_int\":true,\"filter_path\":\"hits.total,aggregations.jobTypes.buckets,aggregations.objectTypes.buckets,aggregations.layoutTypes.buckets,aggregations.statusTypes.buckets\"},\"body\":\"{\\\"size\\\":0,\\\"aggs\\\":{\\\"jobTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"jobtype\\\",\\\"size\\\":3}},\\\"objectTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"meta.objectType.keyword\\\",\\\"size\\\":3}},\\\"layoutTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"meta.layout.keyword\\\",\\\"size\\\":3}},\\\"statusTypes\\\":{\\\"terms\\\":{\\\"field\\\":\\\"status\\\",\\\"size\\\":4}}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from reporting collector"}
{"type":"error","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]","name":"Error","stack":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]] :: {\"path\":\"/.kibana/doc/config%3A6.6.0\",\"query\":{},\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]\\\"}],\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]\\\"},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]"}
{"type":"log","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from kibana_settings collector"}
{"type":"error","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{\"size\":10000,\"ignore_unavailable\":true,\"filter_path\":\"hits.hits._source.canvas-workpad\"},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"filter\\\":{\\\"term\\\":{\\\"type\\\":\\\"canvas-workpad\\\"}}}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from canvas collector"}
{"type":"log","@timestamp":"2019-03-19T03:44:05Z","tags":["status","plugin:spaces@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from red to red - all shards failed: [search_phase_execution_exception] all shards failed","prevState":"red","prevMsg":"Request Timeout after 3000ms"}
{"type":"log","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from reporting collector"}
{"type":"error","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]","name":"Error","stack":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]] :: {\"path\":\"/.kibana/doc/config%3A6.6.0\",\"query\":{},\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]\\\"}],\\\"type\\\":\\\"no_shard_available_action_exception\\\",\\\"reason\\\":\\\"No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]\\\"},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[no_shard_available_action_exception] No shard available for [get [.kibana][doc][config:6.6.0]: routing [null]]"}
{"type":"log","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from kibana_settings collector"}
{"type":"error","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/_search\",\"query\":{\"size\":10000,\"ignore_unavailable\":true,\"filter_path\":\"hits.hits._source.canvas-workpad\"},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"filter\\\":{\\\"term\\\":{\\\"type\\\":\\\"canvas-workpad\\\"}}}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:165:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:187:15)\n    at endReadableNT (_stream_readable.js:1094:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"[search_phase_execution_exception] all shards failed"}
{"type":"log","@timestamp":"2019-03-19T03:44:04Z","tags":["warning","stats-collection"],"pid":1432,"message":"Unable to fetch data from canvas collector"}
{"type":"log","@timestamp":"2019-03-19T03:44:05Z","tags":["status","plugin:spaces@6.6.0","error"],"pid":1432,"state":"red","message":"Status changed from red to red - all shards failed: [search_phase_execution_exception] all shards failed","prevState":"red","prevMsg":"Request Timeout after 3000ms"}

The logs in stderr

FATAL  [search_phase_execution_exception] all shards failed :: {"path":"/.kibana/doc/_count","query":{},"body":"{\"query\":{\"bool\":{\"should\":[{\"bool\":{\"must\":[{\"exists\":{\"field\":\"index-pattern\"}},{\"bool\":{\"must_not\":{\"term\":{\"migrationVersion.index-pattern\":\"6.5.0\"}}}}]}}]}}}","statusCode":503,"response":"{\"error\":{\"root_cause\":[],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[]},\"status\":503}"}
@sebelga
Copy link
Contributor

sebelga commented Mar 22, 2019

Hi @redbaron4
Thanks for reporting. I will try to reproduce and come back to you.

@sebelga
Copy link
Contributor

sebelga commented Mar 22, 2019

I can confirm that on 6.6.0 the issue "Kibana crashes on startup if Elasticsearch is started simultaneously" has been resolved.

What you are seeing is a cluster state failure (master is not up or there is no replica of the .kibana index). I will close the issue as it does not seem an issue specific to Kibana.

@sebelga sebelga closed this as completed Mar 22, 2019
@redbaron4
Copy link
Author

@sebelga Thanks for looking at this.

Does this mean that kibana crashing when there is a cluster state failure is expected behaviour? Would it be possible to mitigate this expected behaviour so that kibana service stays up and allows elasticsearch to recover in case such a thing ever happens in production?

@sebelga
Copy link
Contributor

sebelga commented Mar 22, 2019

Sorry, I went a bit fast closing the issue and it might be an issue on our migration plugin. @marius-dr can you have a look into it?

@sebelga sebelga reopened this Mar 22, 2019
@jbudz jbudz added the Team:Operations Team label for Operations Team label Mar 22, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations

@anx-ckreuzberger
Copy link

I'm having the same issue with the docker image for Kibana 6.7.0

@reswob10
Copy link

reswob10 commented Sep 10, 2019 •
edited
Loading

We are running into what seems to be the same issue with 6.8.1. In our case we upgraded from 6.5.4, with the intent on upgrading to 7.3.1, and while ES is running just fine, Kibana won't connect with the same errors as above. Checking the indexes in ES shows that there is a index called .kibana_1 and .monitoring-kibana-<today's date> with both a single primary and replica shard.

Is there a suggested solution? Or temp mitigation?

@jojoob
Copy link

jojoob commented Oct 15, 2019 •
edited
Loading

I've a similar problem with Kibana (version 6.8.2). Elastic needs a few minutes for starting up and changing its status from red to green. When Kibana is started at the same time as elastic it crashes with "all shards failed".

I was wondering how this issue differs from the linked one #25027 that was supposed to be fixed by #25255. And #33316 describes a very similar problem.

@rudolf
Copy link
Contributor

rudolf commented Jan 31, 2020

Might be fixed in 7.5.0 by https://github.com/elastic/kibana/pull/51324/files

@tylersmalley
Copy link
Contributor

Closing as this should be resolved by #51324. Please let us know if you see this issue come up again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Team:Operations Team label for Operations Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@tylersmalley @reswob10 @rudolf @sebelga @jbudz @jojoob @redbaron4 @elasticmachine @anx-ckreuzberger