Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Elasticsearch service. #28344

Merged
merged 46 commits into from
Feb 28, 2019

Conversation

azasypkin
Copy link
Member

@azasypkin azasypkin commented Jan 9, 2019

In the scope of this PR I'm going to introduce ElasticsearchService and related classes based on the legacy Elasticsearch plugin. We'll still keep the plugin and interface it exposes for the BWC reasons, but internally it should rely on start contract provided by ElasticsearchService.

Initially this PR will include additional changes in @kbn/config-schema and deprecation handling code that is required for this migration to happen, but I'll move them into separate PRs when everything in place and works as expected.

Quite a lot of places are relying on the custom "cluster clients" via providing "plugin" config property so we'll need to keep this escape hatch.

NOTE: One should use --xpack.reporting.enabled=false to test this PR for now.

Related to #12442
Fixes #18841

Dev Docs

Introduce Elasticsearch service

We've introduced an Elasticsearch service to the new experimental plugin platform. This service allows new platform plugins to get elasticsearch-js clients for querying Elasticsearch on the server.

To access the new Elasticsearch service from a new platform plugin:

class MyPlugin {
  start(core, plugins) {
    core.elasticsearch.dataClient$.subscribe(client => {
      // query Elasticsearch
    });
  }
}

Clients are exposed via an Observable interface to reflect the possibility that the connection configuration changes while the server is running.

@azasypkin azasypkin added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:New Platform labels Jan 9, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-platform

@azasypkin azasypkin added the WIP Work in progress label Jan 9, 2019
@azasypkin azasypkin force-pushed the issue-18841-core-es-server branch 2 times, most recently from 1fd23ee to ca86e0e Compare January 16, 2019 16:37
@azasypkin azasypkin requested a review from a team as a code owner January 16, 2019 16:37
@azasypkin azasypkin force-pushed the issue-18841-core-es-server branch 3 times, most recently from e7283b9 to d6bbdb2 Compare January 23, 2019 15:10
@@ -139,6 +140,40 @@ export class ConfigService {
);
}

if (ConfigClass.deprecations !== undefined) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: initial version of config deprecations support for new platform configs. Since this PR is presumably targeting 7.0.0 we may just drop already deprecated properties in ES plugin removing the immediate need in this and rather handle deprecation support in the follow-up.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That'd be ideal!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created PR #29496

* @param req Request the `ScopedClusterClient` instance will be scoped to.
*/
public asScoped(req: { headers?: Headers } = {}) {
// It'd have been quite expensive to create and configure client for every incoming
Copy link
Member Author

@azasypkin azasypkin Jan 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: I tried to explain in comment why scoped cluster client was implemented this way. I spent some time learning how ES plugin works currently and is being used and couldn't find a better way to achieve that, but happy to hear your suggestions.

Not very happy with this method being here though, still experimenting...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the description here, that's really helpful. I bet we could come up with a more sophisticated way to enable this, but the implementation you have here (along with the description) is pretty straightforward at least, so I'm not losing any sleep if this goes in.

* not only entries from standard `elasticsearch.*` yaml config, but also some Elasticsearch JS
* client specific options like `keepAlive` or `plugins` (that eventually will be deprecated).
*/
export type ElasticsearchClientConfig = Pick<ConfigOptions, 'keepAlive' | 'log' | 'plugins'> &
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: I'd drop log and plugins as soon as possible and provide more controlled way of extending client if we still want to support this.

| 'username'
| 'password'
> & {
pingTimeout?: ElasticsearchConfig['pingTimeout'] | ConfigOptions['pingTimeout'];
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: we use Duration type in new platform for all timeouts that we get from configs and I personally find it much nicer than plain numbers, so I'm including support for Duration as well.

requestTimeout?: ElasticsearchConfig['requestTimeout'] | ConfigOptions['requestTimeout'];
sniffInterval?: ElasticsearchConfig['sniffInterval'] | ConfigOptions['sniffInterval'];
ssl?: Partial<ElasticsearchConfig['ssl']>;
loggerContext?: string;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: the only driver behind this loggerContext is monitoring that uses logger tag (monitoring-ui) that is different from cluster type (monitoring). I don't like it, and rather always use "cluster type" as additional logger context/tag. It feels like I can break something by replacing monitoring-ui with monitoring, but I'll double check with monitoring team later and hopefully get rid of loggerContext.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elastic/stack-monitoring how important is it for monitoring to have monitoring-ui logging tag instead of just monitoring (https://github.com/elastic/kibana/blob/master/x-pack/plugins/monitoring/common/constants.js#L12) for elastic search cluster client? Right now monitoring is the only plugin that uses logging tag that is different from the cluster name it creates (via plugins.elasticsearch.createCluster('monitoring')) and we need to support additional property in ES service just for that. Would you be open to using just monitoring logging tag instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can change it? @pickypg any historical insight here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't even know why we used monitoring-ui versus monitoring. So it seems safe to me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, do you need our help with that work @azasypkin?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, do you need our help with that work @azasypkin?

@chrisronline, I'm going to just change value of LOGGING_TAG to monitoring and update tests that depend on that. If that's fine for you then I don't need anything apart from review for the monitoring part once PR is ready :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect. Sounds good!


// Original `ConfigOptions` defines `ssl: object` so we need something more specific.
/** @internal */
export type ExtendedConfigOptions = ConfigOptions &
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this will go away as soon as we have proper types from es-js package.

.toPromise();

return {
bwc: { config: defaultConfig },
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: for console plugin and healthcheck (only healthCheckDelay)

return {
bwc: { config: defaultConfig },

apiVersion: defaultConfig.apiVersion,
Copy link
Member Author

@azasypkin azasypkin Jan 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: these are for ES plugin that exposes these fields to the client side, also timelion uses shardTimeout for some reason.


createClient: (type: string, clientConfig: Partial<ElasticsearchClientConfig> = {}) => {
return this.createClusterClient(type, {
...(defaultConfig as ElasticsearchClientConfig),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: I've noticed that in most of the places where we call createCluster we pass default ES config and I don't really see any reason to not use this one to fill the gaps if consumer doesn't specify something. Thoughts?

@azasypkin azasypkin force-pushed the issue-18841-core-es-server branch from ec1eb35 to 536f7d4 Compare January 24, 2019 11:17
@@ -48,7 +48,7 @@ function numberToDuration(numberMs: number) {
return momentDuration(numberMs);

This comment was marked as resolved.


import {
ProxyConfigCollection,
getElasticsearchProxyConfig,
createProxyRoute
} from './server';

function filterHeaders(originalHeaders, headersToKeep) {
Copy link
Member Author

@azasypkin azasypkin Jan 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: since it's the only place in Kibana that consumed filterHeaders and setHeaders I just moved it here and removed from ES plugin public interface.

Ideally we shouldn't have this logic in two places though...

@@ -74,6 +74,8 @@ export default async function (kbnServer, server, config) {
return kbnServer.config;
});

server.decorate('server', 'core', kbnServer.core);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: just a temporary location, need to find a better place to expose core.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is kbnServer.core all of the capabilities we pass through via the legacy service?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I used it as catch-all place for core -> legacy stuff (easy to grep for places that rely on core), do you think we should split into let's say kbnServer.elasticsearch and friends?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think this makes sense. When we pass plugin capabilities through, I'd assume we'd pass them as a sibling property to core, so it kind of mimics the core/plugins convention in the new platform.

Copy link
Contributor

@epixa epixa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a couple questions, but I think this is really close.

'config sourced from: monitoring cluster'
]);
});
});

describe('Custom Headers Configuration', () => {
it('Adds xpack.monitoring.elasticsearch.customHeaders if connected to production cluster', () => {
it('Does not ad xpack.monitoring.elasticsearch.customHeaders if connected to production cluster', () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
it('Does not ad xpack.monitoring.elasticsearch.customHeaders if connected to production cluster', () => {
it('Does not add xpack.monitoring.elasticsearch.customHeaders if connected to production cluster', () => {

@azasypkin
Copy link
Member Author

retest

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Copy link
Contributor

@epixa epixa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I didn't manually test this with the latest master merge, and this has conflicts at the moment. Otherwise, this looks good.

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Copy link
Contributor

@chrisronline chrisronline left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for Stack Monitoring! Thanks for making these changes!

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@azasypkin azasypkin merged commit 0835cd3 into elastic:master Feb 28, 2019
@azasypkin azasypkin deleted the issue-18841-core-es-server branch February 28, 2019 15:22
azasypkin added a commit to azasypkin/kibana that referenced this pull request Feb 28, 2019
@azasypkin
Copy link
Member Author

7.x/7.1.0: 5156829

@elasticmachine
Copy link
Contributor

💔 Build Failed

@elasticmachine
Copy link
Contributor

💔 Build Failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backported Feature:New Platform release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. review Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.2.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[server] Elasticsearch service
9 participants