Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Switching to new es client #74965

Merged

Conversation

jgowdyelastic
Copy link
Member

@jgowdyelastic jgowdyelastic commented Aug 13, 2020

Changes:

  • complete removal of the old ml es client.
  • removal of all uses of legacyClient
  • updated server side checks or responses and errors returned from the js client. Responses and errors now contain a body which contains the response payload or error object.
  • Updates the way we share functions to other plugins. Previously we required a legacy cluster client scoped to our es client to be passed back to us. This has now been removed and we scope the client ourselves using the request object passed to the shared services providers.
  • Updates the need for a dummy request passed to our shared services providers. It no longer requires a param with DummyKibanaRequest. Instead we check that the request is a real instance of KibanaRequest
  • Changes file data viz and data recognizer client side error reporting to behave correctly with the new error objects.
  • Additional follow up work is needed to change how we're displaying errors on the client side as the error's message now contains less information. So we'll need to dig through the raw error contained in the attributes object.

More information about the new es client can be found here https://github.com/elastic/kibana/blob/master/src/core/MIGRATION_EXAMPLES.md#elasticsearch-client

Checklist

Delete any items that are not applicable to this PR.

@jgowdyelastic
Copy link
Member Author

@elasticmachine merge upstream

@jgowdyelastic
Copy link
Member Author

@elasticmachine merge upstream

@jgowdyelastic
Copy link
Member Author

@elasticmachine merge upstream

@elasticmachine
Copy link
Contributor

merge conflict between base and head

@jgowdyelastic
Copy link
Member Author

retest

@jgowdyelastic
Copy link
Member Author

@elasticmachine merge upstream

@jgowdyelastic
Copy link
Member Author

@elasticmachine merge upstream

Copy link
Contributor

@darnautov darnautov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Would be great to centralize the ES client mocks in a follow-up 🙂

@jgowdyelastic
Copy link
Member Author

@elasticmachine merge upstream

Copy link
Contributor

@shahzad31 shahzad31 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uptime changes LGTM.

Copy link
Contributor

@ogupte ogupte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APM changes LGTM

await ml.jobServiceProvider(scopedClusterClient, ({} as unknown) as KibanaRequest)
).jobsSummary([machineLearningJobId]);
const fakeRequest = {} as KibanaRequest;
const summaryJobs = await ml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks to break ML Rules: when an ML Rule executes, summaryJobs is now undefined, while on master it is MlSummaryJob[].

I apologize for not having tests that exercise this code: we're in the process of moving to the FTR where we can leverage ML setup helpers, but until then we do not have jobs config/anomalies data to test against.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 18ccad3

@jgowdyelastic
Copy link
Member Author

@elasticmachine merge upstream

Copy link
Member

@weltenwort weltenwort left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to function as expected for the infra plugin. I only have one question below about the log verbosity.

.isFullLicense()
.hasMlCapabilities(['canGetJobs'])
.ok(async ({ scopedClient }) => {
const { body } = await scopedClient.asInternalUser.ml.getJobs<{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible that this logs errors to the Kibana log when querying for a non-existent job, which it didn't before? Is there a less noisy way of checking for a job's existence?

Copy link
Member Author

@jgowdyelastic jgowdyelastic Sep 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes it appears this is a change introduced with using the new js client. Kibana is logging all js client errors to its log.
I am able to disable the log message by passing { ignore: [404] } as an option to getJobs.
I don't know whether this is something we should be doing by default or perhaps exposing this option to consumers of our shared functions.

Copy link
Member

@weltenwort weltenwort Sep 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For an API that returns a collection I would expect the result set to just not contain the items it couldn't find. 404s/"not found" exceptions mainly make sense for single-resource APIs IMHO.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that does make sense, unfortunately this is a basic wrapper around the elasticsearch endpoint, and so behaves in the exact same way.
https://www.elastic.co/guide/en/elasticsearch/reference/master/ml-get-job.html

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

async chunks size

id value diff baseline
ml 8.2MB -174.0B 8.2MB

distributable file count

id value diff baseline
total 45537 +2 45535

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Copy link
Member

@pheyos pheyos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API integration test changes LGTM

@jgowdyelastic jgowdyelastic merged commit 4762cf5 into elastic:master Sep 2, 2020
@jgowdyelastic jgowdyelastic deleted the switching-to-new-js-client branch September 2, 2020 13:08
jgowdyelastic added a commit to jgowdyelastic/kibana that referenced this pull request Sep 2, 2020
* [ML] Switching to new es client

* further conversions

* fixing tests

* updating responses

* test commit

* refactoring shared services to removed context parameter

* removing last scoped clients

* removing ml js client

* udating file data viz errors

* fixing jest tests

* fixing types after merge with master

* error response changes

* adding default sizes to some requests

* adding custom error types for license checks

* tidying up shared function checks

* removing test data

* removing need for DummyKibanaRequest

* updating comment

* fixing functional api tests

* removing comments

* fixing types after master merge

* throw error rather than return it

* removing placeholder error

* changes based on review comments

* fixing types after merge with master

* fixing missing return

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
jgowdyelastic added a commit that referenced this pull request Sep 2, 2020
* [ML] Switching to new es client

* further conversions

* fixing tests

* updating responses

* test commit

* refactoring shared services to removed context parameter

* removing last scoped clients

* removing ml js client

* udating file data viz errors

* fixing jest tests

* fixing types after merge with master

* error response changes

* adding default sizes to some requests

* adding custom error types for license checks

* tidying up shared function checks

* removing test data

* removing need for DummyKibanaRequest

* updating comment

* fixing functional api tests

* removing comments

* fixing types after master merge

* throw error rather than return it

* removing placeholder error

* changes based on review comments

* fixing types after merge with master

* fixing missing return

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
rylnd added a commit to rylnd/kibana that referenced this pull request Oct 28, 2020
A regression was introduced in elastic#74965 that caused an error to be
thrown while collecting telemetry on ML jobs. Because such exceptions
are caught and we degrade to zeroing out those counts, this one was not
caught until manual testing of telemetry.
rylnd added a commit that referenced this pull request Oct 28, 2020
A regression was introduced in #74965 that caused an error to be
thrown while collecting telemetry on ML jobs. Because such exceptions
are caught and we degrade to zeroing out those counts, this one was not
caught until manual testing of telemetry.
rylnd added a commit to rylnd/kibana that referenced this pull request Oct 28, 2020
A regression was introduced in elastic#74965 that caused an error to be
thrown while collecting telemetry on ML jobs. Because such exceptions
are caught and we degrade to zeroing out those counts, this one was not
caught until manual testing of telemetry.
rylnd added a commit that referenced this pull request Oct 29, 2020
A regression was introduced in #74965 that caused an error to be
thrown while collecting telemetry on ML jobs. Because such exceptions
are caught and we degrade to zeroing out those counts, this one was not
caught until manual testing of telemetry.
rylnd added a commit that referenced this pull request Oct 29, 2020
A regression was introduced in #74965 that caused an error to be
thrown while collecting telemetry on ML jobs. Because such exceptions
are caught and we degrade to zeroing out those counts, this one was not
caught until manual testing of telemetry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:ml release_note:enhancement review Team:APM All issues that need APM UI Team support Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability v7.10.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.