Suggest fields as keywords #10518
Suggest fields as keywords #10518
Conversation
Most field in ECS are keyword and made optional .text if needed. Search for type: text revealed a few fields which we should potentially convert.
ruflin
added
in progress
| @@ -175,6 +175,7 @@ | |||
| something like `GET /users/_search?name=test`. For MySQL, it is | |||
| something like `SELECT id from users where name=test`. | |||
|
|
|||
| #keyword(s)? | |||
There was a problem hiding this comment.
@andrewkroh Could you take a look at the fields below. Potentially these were discussed in previous PR's.
There was a problem hiding this comment.
Good catch. These should both be set to keyword, to enable aggregations and fast indexing.
If full text search is deemed necessary by us or customers, text should be added as a multi-field.
| @@ -24,6 +24,7 @@ | |||
| type: keyword | |||
There was a problem hiding this comment.
@cwurm you might know more here.
There was a problem hiding this comment.
I think you mean user.user_information? I don't have a strong opinion I think. It's filled with strings like systemd Network Management,,,, Gnats Bug-Reporting System (admin). I don't expect too many users searching directly on it rather than something more unique like user.name.
There was a problem hiding this comment.
yes, not sure how the comment added up here :-(
Is the number of different strings limited? Do you expect users to group/aggregate/filter on these?
There was a problem hiding this comment.
When in doubt, use keyword. text should be added as a multi-field if deemed necessary.
There was a problem hiding this comment.
textshould be added as a multi-field if deemed necessary.
Can Beats generate multi-fields?
There was a problem hiding this comment.
yes:
- name: thread
type: keyword
description: >
Information about the running thread where the log originate.
multi_fields:
- name: text
type: text
| @@ -19,6 +19,7 @@ | |||
| description: Set if the file has the `setgid` bit set. Omitted otherwise. | |||
|
|
|||
| - name: origin | |||
| # Should this be keyword? | |||
There was a problem hiding this comment.
file.origin is only filled on macOS. It can be a URL or a computer name, maybe more. We treat URLs as keywords in other places, so I think keyword would make sense here, too.
/cc @adriansr who wrote the code for this
There was a problem hiding this comment.
@adriansr Could you open a PR for this change?
There was a problem hiding this comment.
Strings should always be keyword. text should be added as a multi-field if deemed necessary.
When in doubt, keyword
| @@ -829,6 +829,7 @@ | |||
| type: keyword | |||
| description: This is the path associated with a unix socket. | |||
| - name: messages | |||
| # As it's the raw message should it be keyword. Should it even be indexed? | |||
There was a problem hiding this comment.
It's rarely used (it requires a config change to enable), but when it is used it's really useful to have as text so that you can search it.
Now that we have event.original, I'm thinking this would make sense to put there (even if it is a keyword). Likewise the warnings field below could go in as error.message.
There was a problem hiding this comment.
This field should be keyword. If we want to enable full text search, text should be added as a multi-field.
There was a problem hiding this comment.
@webmat If it's called message it should be text by default. So far this convention holds in ECS I think.
There was a problem hiding this comment.
@andrewkroh I assume your team will tackle these changes?
| @@ -19,6 +19,7 @@ | |||
| description: Set if the file has the `setgid` bit set. Omitted otherwise. | |||
|
|
|||
| - name: origin | |||
| # Should this be keyword? | |||
There was a problem hiding this comment.
Strings should always be keyword. text should be added as a multi-field if deemed necessary.
When in doubt, keyword
| @@ -829,6 +829,7 @@ | |||
| type: keyword | |||
| description: This is the path associated with a unix socket. | |||
| - name: messages | |||
| # As it's the raw message should it be keyword. Should it even be indexed? | |||
There was a problem hiding this comment.
This field should be keyword. If we want to enable full text search, text should be added as a multi-field.
| @@ -175,6 +175,7 @@ | |||
| something like `GET /users/_search?name=test`. For MySQL, it is | |||
| something like `SELECT id from users where name=test`. | |||
|
|
|||
| #keyword(s)? | |||
There was a problem hiding this comment.
Good catch. These should both be set to keyword, to enable aggregations and fast indexing.
If full text search is deemed necessary by us or customers, text should be added as a multi-field.
| @@ -186,6 +186,7 @@ | |||
| IP address. In a server reply (DHCPOFFER), a DHCP server uses | |||
| this option to specify the lease time it is willing to offer. | |||
|
|
|||
| # error.message? or just message? | |||
There was a problem hiding this comment.
Yeah I vote for error.message, although if there's no time, not the end of the world.
| @@ -24,6 +24,7 @@ | |||
| type: keyword | |||
There was a problem hiding this comment.
When in doubt, use keyword. text should be added as a multi-field if deemed necessary.
|
Closing this PR as this will not be merged. @cwurm @andrewkroh I assume your team will follow up with these if needed. Even if it's closed, we can keep the discussion going. |
|
I created a PR for these changes in #10577. |
Most field in ECS are keyword and made optional .text if needed. Search for type: text revealed a few fields which we should potentially convert.