Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry authentication and other connection failures in Saved Object migrations #51324

Merged
merged 5 commits into from
Nov 22, 2019

Conversation

rudolf
Copy link
Contributor

@rudolf rudolf commented Nov 21, 2019

Summary

Fixes #51242 by retrying authentication and other failures encountered from callCluster in the Saved Object migrator.

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

For maintainers

@rudolf rudolf added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:New Platform v8.0.0 v7.5.0 v7.6.0 labels Nov 21, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-platform (Team:Platform)

@rudolf rudolf requested review from epixa and joshdover November 21, 2019 18:59
@LeeDr
Copy link

LeeDr commented Nov 21, 2019

I changed the kibana user's password while starting up from x-pack directory and got tons of errors repeating. Then I checked the kibana user's password back to what kibana was configured to use (around timestamp [14:00:05.764]) and it instantly started working. This wasn't an exhaustive test, but a good sign.

   │ proc [kibana] server    log   [14:00:03.950] [warning][stats-collection] Unable to fetch data from canvas collector
   │ info [o.e.x.s.a.AuthenticationService] [leebuntu] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]
   │ proc [kibana] server   error  [14:00:04.029] [warning][stats-collection] [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } } :: {"path":"/.kibana/_search","query":{"size":1000,"ignore_unavailable":true,"filter_path":"hits.hits._id"},"body":"{\"query\":{\"bool\":{\"filter\":{\"term\":{\"index-pattern.type\":\"rollup\"}}}}}","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}","wwwAuthenticateDirective":"Basic realm=\"security\" charset=\"UTF-8\""}
   │ proc [kibana]     at respond (/home/leedr/git/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
   │ proc [kibana]     at checkRespForFailure (/home/leedr/git/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
   │ proc [kibana]     at HttpConnector.<anonymous> (/home/leedr/git/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
   │ proc [kibana]     at IncomingMessage.wrapper (/home/leedr/git/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
   │ proc [kibana]     at IncomingMessage.emit (events.js:194:15)
   │ proc [kibana]     at endReadableNT (_stream_readable.js:1103:12)
   │ proc [kibana]     at process._tickCallback (internal/process/next_tick.js:63:19)
   │ proc [kibana] server    log   [14:00:04.029] [warning][stats-collection] Unable to fetch data from rollups collector
   │ info [o.e.x.s.a.AuthenticationService] [leebuntu] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]
   │ info [o.e.x.s.a.AuthenticationService] [leebuntu] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]
   │ info [o.e.x.s.a.AuthenticationService] [leebuntu] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]
   │ proc [kibana] server   error  [14:00:04.258] [warning][stats-collection] [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate="Basic realm=\"security\" charset=\"UTF-8\"" } } :: {"path":"/.kibana_task_manager/_search","query":{"ignore_unavailable":true},"body":"{\"sort\":[{\"task.runAt\":\"asc\"},{\"_id\":\"desc\"}],\"query\":{\"bool\":{\"must\":[{\"term\":{\"type\":\"task\"}},{\"bool\":{\"filter\":{\"term\":{\"_id\":\"task:oss_telemetry-vis_telemetry\"}}}}]}}}","statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}","wwwAuthenticateDirective":"Basic realm=\"security\" charset=\"UTF-8\""}
   │ proc [kibana]     at respond (/home/leedr/git/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)
   │ proc [kibana]     at checkRespForFailure (/home/leedr/git/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)
   │ proc [kibana]     at HttpConnector.<anonymous> (/home/leedr/git/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
   │ proc [kibana]     at IncomingMessage.wrapper (/home/leedr/git/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
   │ proc [kibana]     at IncomingMessage.emit (events.js:194:15)
   │ proc [kibana]     at endReadableNT (_stream_readable.js:1103:12)
   │ proc [kibana]     at process._tickCallback (internal/process/next_tick.js:63:19)
   │ proc [kibana] server    log   [14:00:04.258] [warning][stats-collection] Unable to fetch data from visualization_types collector
   │ info [o.e.x.s.a.AuthenticationService] [leebuntu] Authentication of [kibana] was terminated by realm [reserved] - failed to authenticate user [kibana]
   │ proc [kibana] server    log   [14:00:05.764] [info][status][plugin:elasticsearch@8.0.0] Status changed from red to green - Ready
   │ info [o.e.c.m.MetaDataIndexTemplateService] [leebuntu] adding template [.management-beats] for index patterns [.management-beats]
   │ proc [kibana] server    log   [14:00:05.797] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: trial | status: active | expiry date: 2019-12-21T13:54:29-06:00
   │ proc [kibana] server    log   [14:00:05.798] [info][status][plugin:xpack_main@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.798] [info][status][plugin:graph@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.800] [info][status][plugin:searchprofiler@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.801] [info][status][plugin:ml@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.801] [info][status][plugin:tilemap@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.801] [info][status][plugin:watcher@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.801] [info][status][plugin:grokdebugger@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.801] [info][status][plugin:logstash@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.802] [info][status][plugin:beats_management@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.802] [info][status][plugin:index_management@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.802] [info][status][plugin:index_lifecycle_management@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.802] [info][status][plugin:rollup@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.802] [info][status][plugin:transform@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.803] [info][status][plugin:remote_clusters@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.803] [info][status][plugin:cross_cluster_replication@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.803] [info][status][plugin:file_upload@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.803] [info][status][plugin:snapshot_restore@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.803] [info][status][plugin:reporting@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.826] [info][kibana-monitoring][monitoring] Starting monitoring stats collection
   │ proc [kibana] server    log   [14:00:05.845] [info][status][plugin:maps@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.852] [info][status][plugin:spaces@8.0.0] Status changed from red to green - Ready
   │ proc [kibana] server    log   [14:00:05.886] [info][status][plugin:security@8.0.0] Status changed from red to green - Ready

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@LeeDr
Copy link

LeeDr commented Nov 21, 2019

This is interesting to me, I changed the password in elasticsearch to be wrong again and opened Kibana in my browser.
Since Kibana couldn't get license info (or some other x-pack info) using the kibana server user from elasticsearch I got the browser's basic auth login dialog.
I entered valid elastic user credentials and Kibana opened up the status page showing red status.
Seems like reasonable results for this case.

@rudolf rudolf marked this pull request as ready for review November 21, 2019 20:48
@rudolf rudolf requested a review from a team as a code owner November 21, 2019 20:48
@LeeDr
Copy link

LeeDr commented Nov 21, 2019

This is fun... while you're logged in to Kibana and testing things run this script to constantly change the kibana user's password and make sure the Kibana process never crashes;

while (true); do curl -k -XPUT --basic "http://elastic:changeme@localhost:9220/_xpack/security/user/kibana/_password?pretty" -H 'Content-Type: application/json' -d'{ "password": "changeit" }'; echo "bad password"; sleep 4; curl -k -XPUT --basic "http://elastic:changeme@localhost:9220/_xpack/security/user/kibana/_password?pretty" -H 'Content-Type: application/json' -d'{ "password": "changeme" }'; echo "good password"; sleep 8; done

@rudolf rudolf added the release_note:skip Skip the PR/issue when compiling release notes label Nov 21, 2019
Copy link
Contributor

@joshdover joshdover left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't found any issues in manual testing. @LeeDr are you good with this change as you tested?


// TODO: Replace with APICaller from './scoped_cluster_client' once #46668 is merged
export function migrationsRetryCallCluster(
apiCaller: (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional: use APICaller since it exists now

log: Logger,
delay: number = 2500
) {
const previousErrors: string[] = [];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional: use Set instead of array

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

rudolf added a commit to rudolf/kibana that referenced this pull request Nov 22, 2019
…grations (elastic#51324)

* Retry authentication and other connection failures in migrations

* Log migration ES connection errors once, retry every 2.5s

* Test migrations es connection error logging

* retry on AuthorizationExceptions during migration

* Set delay to 1 for tests
rudolf added a commit to rudolf/kibana that referenced this pull request Nov 22, 2019
…grations (elastic#51324)

* Retry authentication and other connection failures in migrations

* Log migration ES connection errors once, retry every 2.5s

* Test migrations es connection error logging

* retry on AuthorizationExceptions during migration

* Set delay to 1 for tests
rudolf added a commit that referenced this pull request Nov 22, 2019
…grations (#51324) (#51424)

* Retry authentication and other connection failures in migrations

* Log migration ES connection errors once, retry every 2.5s

* Test migrations es connection error logging

* retry on AuthorizationExceptions during migration

* Set delay to 1 for tests
rudolf added a commit that referenced this pull request Nov 22, 2019
…grations (#51324) (#51425)

* Retry authentication and other connection failures in migrations

* Log migration ES connection errors once, retry every 2.5s

* Test migrations es connection error logging

* retry on AuthorizationExceptions during migration

* Set delay to 1 for tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:New Platform release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.5.0 v7.6.0 v8.0.0
Projects
None yet
4 participants