[Endpoint] EMT-67: add kql support for endpoint list #56328

nnamdifrankie · 2020-01-29T18:23:19Z

Summary

This query allows a kql (kibana query language) to be passed as a filter type to the endpoint list api. This query will help pare down the endpoints that are considered based on fields in the query. We do not check if the property or field in the query is valid in the schema. This is left for future improvement or a separate ticket. The field name in the kql and result returned is determined by field mapping types. The mapping honor ECS typing as a base. We will address it through https://github.com/elastic/endpoint-app-team/issues/139

https://github.com/elastic/endpoint-app-team/issues/67
Sample Request Body:

{
            paging_properties: [
              {
                page_size: 1,
              },
              {
                page_index: 1,
              },
            ],
            filters: 'not host.ip:10.101.149.26',
          }

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

~~- [ ] This was checked for cross-browser compatibility, including a check against IE11~~
~~- [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
~~- [ ] Documentation was added for features that require explanation or tutorials~~

Unit or functional tests were updated or added to match the most common scenarios
~~- [ ] This was checked for keyboard-only and screenreader accessibility~~

For maintainers

~~- [ ] This was checked for breaking API changes and was labeled appropriately~~
~~- [ ] This includes a feature addition or change that requires a release note and was labeled appropriately~~

nnamdifrankie · 2020-01-29T20:46:19Z

@elasticmachine merge upstream

pzl · 2020-01-30T18:07:57Z

x-pack/plugins/endpoint/server/services/endpoint/endpoint_query_builders.ts

+
+function buildQueryBody(request: KibanaRequest<any, any, any>): Record<string, any> {
+  if (request?.body?.filters && typeof request.body.filters === 'string') {
+    return toElasticsearchQuery(fromKueryExpression(request.body.filters));


is this the KQL translation magic here?

It is using function from the AST (Abstract Syntax Tree) Module to transform the text to an elastic query. Not magical, just classic language syntax tree transformation.

I'm going to be using this function too, for the alert list page... let's think about pulling this out into a common location?

# Conflicts: # x-pack/plugins/endpoint/server/services/endpoint/endpoint_query_builders.test.ts # x-pack/plugins/endpoint/server/services/endpoint/endpoint_query_builders.ts

….com:nnamdifrankie/kibana into EMT-67_add_kql_query_filter_to_endpoint_list

pzl

this all looks reasonable

dplumlee · 2020-01-30T21:32:47Z

I know a lot of the code is already checked in, but after spending some time with it I think the name endpoints is kind of confusing when mixed with a lot of the global endpoint plugin names and types. I know me and a few others have gotten kind of confused at times trying to figure out whether or not certain functions are for the endpoint app or endpoint list page (the api route api/endpoint/endpoints is a good example of the redundancy, and the function registerEndpointRoutes being only for endpoint management and not the entire plugin is kind of confusing as well), and it might be a good idea to name some of this stuff differently early on. @kevinlog had suggested metadata or hosts, both of which I think are fairly good options, but discussion is welcome. @nnamdifrankie @madirey @peluja1012 @paul-tavares thoughts?

nnamdifrankie · 2020-01-30T21:42:21Z

@dplumlee Well am open to discussions and we can refactor in another ticket before we get too far. I do not think hosts may be ideal. It is actually the endpoint api if we really think about it, and the data that we fetch is metadata. But let us have an agreement and we can move forward.

kevinlog · 2020-01-30T21:46:39Z

@dplumlee @nnamdifrankie I think we do need to change the naming as people are getting confused. Franklin, good point that it's not exactly hosts as the document comes from the endpoint and will eventually contain some endpoint specific information.

How about /api/endpoint/metadata ? metadata is technically plural so could still imply a list.

bkimmel · 2020-01-30T21:57:58Z

x-pack/plugins/endpoint/server/services/endpoint/endpoint_query_builders.ts

@@ -66,6 +68,15 @@ async function getPagingProperties(
  };
 }

+function buildQueryBody(request: KibanaRequest<any, any, any>): Record<string, any> {
+  if (request?.body?.filters && typeof request.body.filters === 'string') {


I'm new to the ? operator - very cool. Could this be shortened to just typeof request?.body?.filters === 'string'?

nnamdifrankie · 2020-01-31T14:04:00Z

@kevinlog I will open another PR with a full refactor up and down the code base. Let review and merge this ticket on its merit. Thanks for the input.

paul-tavares · 2020-01-31T14:16:28Z

@dplumlee (all) I'm ok with using another name and refactoring to that.

paul-tavares · 2020-01-31T14:25:31Z

x-pack/plugins/endpoint/server/routes/endpoints.ts

+            /**
+             * filter to be applied, it could be a kql expression or discrete filter to be implemented
+             */
+            filters: schema.nullable(schema.oneOf([schema.string()])),


why not just schema.string()?

Also - we have talked about having other types of filtering possible in the future. Should this param be named something else other than filter? (FYI: in Ingest, they used a param named kuery.

paul-tavares · 2020-01-31T14:28:40Z

x-pack/plugins/endpoint/server/routes/endpoints.ts

+                    page_size: schema.number({ defaultValue: 10, min: 1, max: 10000 }),
+                  }),
+                  /**
+                   * the index of the first result in the page in the total matching set


This does not seem to be the correct description based on the current implementation. page_index is the actual page number (where 0 is the first page). The actual first result index within the total matching set is calculated based on this number and the page_size.

paul-tavares · 2020-01-31T14:43:38Z

x-pack/test/api_integration/apis/endpoint/endpoints.ts

+        expect(body.total).to.eql(2);
+        expect(body.endpoints.length).to.eql(1);
+        expect(body.request_page_size).to.eql(1);
+        expect(body.request_page_index).to.eql(1);


should we have any testing around the actual set of endpoints returned and ensuring its the expected set for the requested page_index + page_size?

paul-tavares · 2020-01-31T16:59:56Z

x-pack/plugins/endpoint/server/routes/endpoints.ts

+            /**
+             * filter to be applied, it could be a kql expression or discrete filter to be implemented
+             */
+            filters: schema.nullable(schema.oneOf([schema.string()])),


Sorry - just remember another thing.
I think (during x-school) I heard that things like support for KQL is only available at certain licensing levels (like not in Free version). Do we need to do that type of validation here? (we likely have the same question to answer on the UI side 😃 )

I am sure you are right about this. But there are many licenses and license has to be configured at the plugin level using license plugin. And we also have to know what the minimum level of license, basic e.t.c. And what features should be free or licensed. Do you want us to move take offline? and not address it here?

Sound good @nnamdifrankie . I just did quick check around KQL and it seems that its available at all levels - its the AutoComplete functionality that seems to only be enabled for Basic license and above (ref: https://www.elastic.co/guide/en/kibana/current/kuery-query.html#kuery-query), so I don't think you need any checks here

madirey · 2020-01-31T20:02:59Z

x-pack/plugins/endpoint/server/routes/endpoints.ts

-                // the index of the page to return
-                schema.object({ page_index: schema.number({ defaultValue: 0, min: 0 }) }),
-              ])
+            paging_properties: schema.nullable(


We're also using paging_properties and filters for alert list, so let's consider moving this schema out to a common location...

@madirey I have a refactor ticket coming behind this and can move it then, if this is fine with you. I think we should share only things we strongly feel are really generic, such that other paths in the system does not consume non relevant changes.

Sure @nnamdifrankie , we can refactor once both endpoints are integrated too, might be easier that way.

@madirey I see your point now, since it one app we cannot have different names for the same thing. Thanks for the input. Once we merge let work together to merge some of the properties.

kevinlog · 2020-01-31T21:39:24Z

@nnamdifrankie I'm okay with renaming in another PR, can you make the new Git issue and link it here?

nnamdifrankie · 2020-02-03T00:30:02Z

@elasticmachine merge upstream

nnamdifrankie · 2020-02-03T01:15:18Z

@elasticmachine merge upstream

….com:nnamdifrankie/kibana into EMT-67_add_kql_query_filter_to_endpoint_list

paul-tavares

👍

kibanamachine · 2020-02-04T18:55:35Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: a8a8c85

History

💚 Build #24129 succeeded 6d436b6
💔 Build #24103 failed 93cc54d
💚 Build #23926 succeeded 046e0bc
💚 Build #23637 succeeded ac6590f
💚 Build #23392 succeeded 10b6194

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (23 commits) Properly handle password change for users authenticated with provider other than `basic`. (elastic#55206) Improve pull request template proposal (elastic#56756) Only change handlers as the element changes (elastic#56782) [SIEM][Detection Engine] Final final rule changes (elastic#56806) [SIEM][Detection Engine] critical blocker, wrong ilm policy, need to match beats ilm policy Move ui/agg_types in to shim data plugin (elastic#56353) [SIEM] Fixes Signals count spinner (elastic#56797) [docs] Update upgrade version path (elastic#56658) [Canvas] Use unique Id for Canvas Embeddables (elastic#56783) [Rollups] Adjust max width for job detail panel (elastic#56674) Prevent http client from converting our form data (elastic#56772) Disable creating alerts client instances when ESO plugin is using an ephemeral encryption key (elastic#56676) Bumps terser-webpack-plugin to 2.3.4 (elastic#56662) Advanced settings component registry ⇒ kibana platform plugin (elastic#55940) [Endpoint] EMT-67: add kql support for endpoint list (elastic#56328) Implement UI for Create Alert form (elastic#55232) Fix: Filter pill base coloring (elastic#56761) fix open close signal on detail page (elastic#56757) [Search service] Move loadingCount to sync search strategy (elastic#56335) Rollup TSVB integration: Add test and fix warning text (elastic#56639) ...

* Add Endpoint plugin and Resolver embeddable (#51994) * Add functional tests for plugins to x-pack (so we can do a functional test of the Resolver embeddable) * Add Endpoint plugin * Add Resolver embeddable * Test that Resolver embeddable can be rendered Conflicts: x-pack/.i18nrc.json x-pack/test/api_integration/apis/index.js * [Endpoint] Register endpoint app (#53527) * register app, create functional test * formatting * update tests * adjust test data for endpoint * add endpoint tests for testing spaces, app enabled, disabled, etc * linting * add read privileges to endpoint * rename variable since its used now * remove deprecated context * remove unused variable * fix type check * correct test suite message Co-Authored-By: Larry Gregory <lgregorydev@gmail.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Larry Gregory <lgregorydev@gmail.com> * [Endpoint] add react router to endpoint app (#53808) * add react router to endpoint app * linting * linting * linting * correct tests * change history from hash to browser, add new test util * remove default values in helper functions * fix type check, use FunctionComponent as oppsed to FC * use BrowserRouter component * use BrowserRouter component lin * add comments to test framework, change function name to include browserHistory Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> * EMT-issue-65: add endpoint list api (#53861) add endpoint list api * EMT-65:always return accurate endpoint count (#54423) EMT-65:always return accurate endpoint count, independent of paging properties * Resolver component w/ sample data (#53619) Resolver is a map. It shows processes that ran on a computer. The processes are drawn as nodes and lines connect processes with their parents. Resolver is not yet implemented in Kibana. This PR adds a 'map' type UX. The user can click and drag to pan the map and zoom using trackpad pinching (or ctrl and mousewheel.) There is no code providing actual data. Sample data is included. The sample data is used to draw a map. The fundamental info needed is: process names the parent of a process With this info we can topologically lay out the processes. The sample data isn't yet in a realistic format. We'll be fixing that soon. Related issue: elastic/endpoint-app-team#30 * Resolver test plugin not using mount context. (#54933) Mount context was deprecated. Use core.getStartServices() instead. * Resolver nonlinear zoom (#54936) * [Endpoint] add Redux saga Middleware and app Store (#53906) * Added saga library * Initialize endpoint app redux store * Resolver is overflow: hidden to prevent obscured elements from showing up (#55076) * [Endpoint] Fix saga to start only after store is created and stopped on app unmount (#55245) - added `stop()`/`start()` methods to the Saga Middleware creator factory - adjust tests based on changes - changed application `renderApp` to stop sagas when react app is unmounted * Resolver zoom, pan, and center controls (#55221) * Resolver zoom, pan, and center controls * add tests, fix north panning * fix type issue * update west and east panning to behave like google maps * [Endpoint] FIX: Increase tests `sleep` default duration back to 100ms (#55492) Revert `sleep()` default duration, in the saga tests, back to 100ms in order to prevent intermittent failures during CI runs. Fixes #55464 Fixes #55465 * [Endpoint] EMT-65: make endpoint data types common, restructure (#54772) [Endpoint] EMT-65: make endpoint data types common, use schema changes * Basic Functionality Alert List (#55800) * sets up initial grid and data type * data feeds in from backend but doesnt update * sample data feeding in correctly * Fix combineReducers issue by importing Redux type from 'redux' package * Add usePageId hook that fires action when user navigates to page * Strict typing for middleware * addresses comments and uses better types * move types to common/types.ts * Move types to endpoint/types.ts, address PR comments blah 2 Co-authored-by: Pedro Jaramillo <peluja1012@gmail.com> * [Endpoint] Add Endpoint Details route (#55746) * Add Endpoint Details route * add Endpoint Details tests * sacrifices to the Type gods * update to latest endpoint schema Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> * [Endpoint] EMT-67: add kql support for endpoint list (#56328) [Endpoint] EMT-67: add kql support for endpoint list * [Endpoint] ERT-82 ERT-83 ERT-84: Alert list API with pagination (#56538) * ERT-82 ERT-83 ERT-84 (partial): Add Alert List API with pagination * Better type safety for alert list API * Add Test to Verify Endpoint App Landing Page (#57129) Conflicts: x-pack/test/functional/page_objects/index.ts * fixes render bug in alert list (#57152) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> * Resolver: Animate camera, add sidebar (#55590) This PR adds a sidebar navigation. clicking the icons in the nav will focus the camera on the different nodes. There is an animation effect when the camera moves. Conflicts: yarn.lock * [Endpoint] Task/basic endpoint list (#55623) * Adds host management list to endpoint security plugin Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> * [Endpoint] Policy List UI route and initial view (#56918) * Initial Policy List view * Add `endpoint/policy` route and displays Policy List * test cases (both unit and functional) Does not yet interact with API (Ingest). * Add ApplicationService app status management (#50223) This was already backported, but changes to endpoint app could not be backported, since endpoint app itself hadn't been backported. Now that the endpoint app is backported, reapply the endpoint specific changes from the original commit. * Implements `getStartServices` on server-side (#55156) This was already backported, but changes to endpoint app could not be backported, since endpoint app itself hadn't been backported. Now that the endpoint app is backported, reapply the endpoint specific changes from the original commit. * [ui/utils/query_string]: Remove unused methods & migrate apps to querystring lib (#56957) This was already backported, but changes to endpoint app could not be backported, since endpoint app itself hadn't been backported. Now that the endpoint app is backported, reapply the endpoint specific changes from the original commit. Co-authored-by: Kevin Logan <56395104+kevinlog@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Larry Gregory <lgregorydev@gmail.com> Co-authored-by: nnamdifrankie <56440728+nnamdifrankie@users.noreply.github.com> Co-authored-by: Davis Plumlee <56367316+dplumlee@users.noreply.github.com> Co-authored-by: Paul Tavares <56442535+paul-tavares@users.noreply.github.com> Co-authored-by: Pedro Jaramillo <peluja1012@gmail.com> Co-authored-by: Dan Panzarella <pzl@users.noreply.github.com> Co-authored-by: Madison Caldwell <madison.rey.caldwell@gmail.com> Co-authored-by: Charlie Pichette <56399229+charlie-pichette@users.noreply.github.com> Co-authored-by: Candace Park <56409205+parkiino@users.noreply.github.com> Co-authored-by: Pierre Gayvallet <pierre.gayvallet@gmail.com> Co-authored-by: Alexey Antonov <alexwizp@gmail.com>

EMT-67: add kql support for endpoint list

bdac93c

nnamdifrankie changed the title ~~EMT-67: add kql support for endpoint list~~ [Endpoint] EMT-67: add kql support for endpoint list Jan 29, 2020

nnamdifrankie added v7.7.0 v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:Endpoint Elastic Endpoint feature Team:Endpoint Management labels Jan 29, 2020

Merge branch 'master' into EMT-67_add_kql_query_filter_to_endpoint_list

10b6194

pzl reviewed Jan 30, 2020

View reviewed changes

nnamdifrankie added 2 commits January 30, 2020 15:26

Merge branch 'master' into EMT-67_add_kql_query_filter_to_endpoint_list

0d68763

# Conflicts: # x-pack/plugins/endpoint/server/services/endpoint/endpoint_query_builders.test.ts # x-pack/plugins/endpoint/server/services/endpoint/endpoint_query_builders.ts

Merge branch 'EMT-67_add_kql_query_filter_to_endpoint_list' of github…

ac6590f

….com:nnamdifrankie/kibana into EMT-67_add_kql_query_filter_to_endpoint_list

pzl approved these changes Jan 30, 2020

View reviewed changes

bkimmel reviewed Jan 30, 2020

View reviewed changes

paul-tavares reviewed Jan 31, 2020

View reviewed changes

madirey reviewed Jan 31, 2020

View reviewed changes

Merge branch 'master' into EMT-67_add_kql_query_filter_to_endpoint_list

e255699

nnamdifrankie added 4 commits February 2, 2020 20:17

EMt-67: clean up from PR review

ac0a0f7

Merge branch 'EMT-67_add_kql_query_filter_to_endpoint_list' of github…

046e0bc

….com:nnamdifrankie/kibana into EMT-67_add_kql_query_filter_to_endpoint_list

EMT-67: improve testing

93cc54d

EMT-67: fix test type

6d436b6

paul-tavares approved these changes Feb 3, 2020

View reviewed changes

EMT-67: change import for external dependency

a8a8c85

nnamdifrankie merged commit 84fa97f into elastic:master Feb 4, 2020

nnamdifrankie deleted the EMT-67_add_kql_query_filter_to_endpoint_list branch February 4, 2020 19:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Endpoint] EMT-67: add kql support for endpoint list #56328

[Endpoint] EMT-67: add kql support for endpoint list #56328

nnamdifrankie commented Jan 29, 2020 •

edited

Loading

nnamdifrankie commented Jan 29, 2020

pzl Jan 30, 2020

nnamdifrankie Jan 30, 2020

madirey Jan 31, 2020

pzl left a comment

dplumlee commented Jan 30, 2020

nnamdifrankie commented Jan 30, 2020 •

edited

Loading

kevinlog commented Jan 30, 2020

bkimmel Jan 30, 2020

nnamdifrankie commented Jan 31, 2020 •

edited

Loading

paul-tavares commented Jan 31, 2020

paul-tavares Jan 31, 2020

paul-tavares Jan 31, 2020

paul-tavares Jan 31, 2020

nnamdifrankie Feb 3, 2020

paul-tavares Jan 31, 2020

paul-tavares Jan 31, 2020

nnamdifrankie Feb 3, 2020

paul-tavares Feb 3, 2020

madirey Jan 31, 2020 •

edited

Loading

nnamdifrankie Feb 3, 2020

madirey Feb 3, 2020

nnamdifrankie Feb 3, 2020

kevinlog commented Jan 31, 2020

nnamdifrankie commented Feb 3, 2020

nnamdifrankie commented Feb 3, 2020

paul-tavares left a comment

kibanamachine commented Feb 4, 2020

[Endpoint] EMT-67: add kql support for endpoint list #56328

[Endpoint] EMT-67: add kql support for endpoint list #56328

Conversation

nnamdifrankie commented Jan 29, 2020 • edited Loading

Summary

Checklist

For maintainers

nnamdifrankie commented Jan 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pzl left a comment

Choose a reason for hiding this comment

dplumlee commented Jan 30, 2020

nnamdifrankie commented Jan 30, 2020 • edited Loading

kevinlog commented Jan 30, 2020

Choose a reason for hiding this comment

nnamdifrankie commented Jan 31, 2020 • edited Loading

paul-tavares commented Jan 31, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

madirey Jan 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinlog commented Jan 31, 2020

nnamdifrankie commented Feb 3, 2020

nnamdifrankie commented Feb 3, 2020

paul-tavares left a comment

Choose a reason for hiding this comment

kibanamachine commented Feb 4, 2020

💚 Build Succeeded

History

nnamdifrankie commented Jan 29, 2020 •

edited

Loading

nnamdifrankie commented Jan 30, 2020 •

edited

Loading

nnamdifrankie commented Jan 31, 2020 •

edited

Loading

madirey Jan 31, 2020 •

edited

Loading