[Discover] Add simple testing functionality #5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* added xy plugin. * Added expressionXY limits. * Added xy expression functions to the expression_xy plugin. * Moved xy to a separate plugin. * Small refactoring. * Fixed types. * Fixed import of scss. * Fixed imports. * Added required plugins. * Fixed import * Fixed types. * Changed expression names. * Fixed bugs, caused by the refactoring process. * Fixed lens snapshots. * Removed new line. * Fixed xy_chart tests. * Added lazy loading for xy chart. * Fixed xy chart test. * Fixed broken chart selectors. * Fixed dashboard tests. * dashboard test fixed. * Fixed heatmap vis. * Smokescreen test fixed. * more fixes. * async dashboard tests fixed. * Fixed xy smokescreen tests selectors. * fixed show_underlying_data tests. * Updated snapshots. * updated limits. * Fixed more selectors * Fixed persistent context test. * Fixed some more test at ml. * Fixed types and imports * Fixed handlers.inspectorAdapters.tables.logDatatable * Fixed logDatatable * Translations fixed. * Fixed "Visualize App ... cleans filters and query" test. cleans filters and query * Fixed "lens disable auto-apply tests" test. * Updated dashboard tests. * Fixed translations. * Expression tests fixed. * Cleaned up expression_xy. * cleaned up lens xy_visualization. * fixed more tests. * Fix of tsvb. * Fixed more tests. * Fixed xy chart limits. * Fixed new tests. * Fixed types. * Added extended layers expressions. * Added support of tables at layers. * Fixed tests. * Fixed more tests. * Fixed lens types. * Added tables to layers. * Checks fixed. * updated tests. * Fixed types. * First try to fix merge conflicts. * Fixed annotatations. * Fixed types. * Updated snapshots * Fixed tests. * Fixed dependencies. * Fixed i18n. * Moved XY state types to lens. * Fixed more types. * Update src/plugins/chart_expressions/expression_xy/README.md Co-authored-by: Marta Bondyra <marta.bondyra@gmail.com> * [CI] Auto-commit changed files from 'node scripts/build_plugin_list_docs' * Removed yConfig from *Layers types * Fixed styles. * Fixed types. * Removed not used utils and styles. * Fixed types and tests. * updated size. * Added right behavior, related to the tables, comming from the expression. * Fixed reference lines. * Fixed jsdoc. * Added annotations to layeredXyVIs. * Fixed limits. * Refactored the implementation to be reusable. * Fixed undefined layers. * Fixed empty arrays problems. * Fixed input translations and removed not used arguments. * Fixed missing required args error, and added required to arguments. * Simplified expression configuration. * Added strict to all the expressions. * refactored code, according to the nit. * Support visdimension type for accessors in data_layer and reference_line_layer * Fix checks * Moved dataLayer to the separate component. * Fixed jest tests. * Fixed tests. * Some updates * Refactored dataLayers helpers and xy_chart. * More fixes of the expression Added extendedYConfig for dataLayers. Added yConfig for referenceLineLayers. Fixed undefined id at tooltip. * Fixed tests and snapshots. * Icons at annotations and reference lines are strict. * Fix CI * axis extent validation added. * Added checks to the legend config. * fillOpacity usage validation is added. * Fixed valueLabels argument options. Removed not used. Added validation for usage. * Removed not used tests and imports. * Fixed valueLabels and added migrations. * Fixed type checks. * Added test for the migrations. * Fixed imports. * Fixed types * Fixed i18n checks. # Conflicts: # src/plugins/chart_expressions/expression_xy/public/components/xy_chart.tsx * Fixed imports and types. * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * Update src/plugins/chart_expressions/expression_xy/common/expression_functions/extended_annotation_layer.ts Co-authored-by: Marta Bondyra <marta.bondyra@gmail.com> * Removed extra extends. * Update src/plugins/chart_expressions/expression_xy/common/expression_functions/annotation_layer.ts Co-authored-by: Marta Bondyra <marta.bondyra@gmail.com> * Added guard. * Fixed the code duplication. * Removed table from the annotation layer. * Changed the `convertActiveDataFromIndexesToLayers` location. * Added tests for convertActiveDataFromIndexesToLayers * Reduced the bundle size a little bit. * Reused strings and args. * Refactored expression functions. Added asynchronous behavior. * Fixed tests. * Updated limits. * Updated the limit. * Fixed types. * fixed types. * Turned back layerIds. * Removed convertActiveData from Lens. * Added test to the layerIds generator. * Fixed types. * Fixed problems with resetting of the inspector. * Fixed migrations. * Removed types. * Removed tones of `areFormatted` calculations. * Fixed `isTimeViz` and `isHistogramViz` by replacing filteredLayers with dataLayers. * Removed referenceLineLayers from the `groupAxesByType` fn. * Added validation to the layeredXyVis. * Fixed extent validation. * Removed comments. * Reduced limit. * Added optimizations. * Fixed floatingColumns error. * Fixed types. * Updated limits. * Turned back extent validation. * Fixed stacked error. * Parallelized async import of functions. * Decreased the complexity of the algorithm. * Fixed snapshot. * Some fixes * Fix checks * Fixed types * Fix types * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * Added validation to the accessors. * Validated accessors at the referenceLineLayer and extendedReferenceLineLayer. * Fixed some style issues. * Fixed imports. * One more fix of import. * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * Move extended data layer fn in async file * Some improvements * Fix checks * Fix validation Co-authored-by: Yaroslav Kuznietsov <kuznetsov.yaroslav.yk@gmail.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Marta Bondyra <marta.bondyra@gmail.com>
* Update copy to not include system indices * Dont include system indices in ds/indices dropdown * Start working on supporting feature states * Store feature states array of options in local state * Fix up server side integration and show deets in flyout * Fix linter issues * commit using @elastic.co * Connect the dots in restore snapshot wizard * Fix linter issues * Finish up wiring up last features * Fix copy * CR * Refactor tooltip implementation * Fix tests * Fix i18n * Add tests * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * Add missing tests * Add option for selecting none of the feature states * Finish off refactoring label placement and fixing up tests * Add tests * Remove nextTick and refactor tests * Refactor feature states into its own setting * Fix docs link * Copy review * Fix tests * Fix small bug and add more tests * Fix linter issue * Address CR * Change duped locale id * Address CR changes * Copy updates * CR changes
…tics feature (elastic#131619) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
…c#132197) * fix conflicted fields modal * fix mock
…astic#131947) * [Fleet] Changes to bulk upgrade api for allowing rolling upgrades * Remove one query and some tests * Skip version check if fleet server agents are being upgraded * Fix test * Fixing tests again * Fix failing check * Fix another test * Fix another test * Fix api integration tests * Remove parameter Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
…132068) * Fix missing package-level vars in GCP policy editor Display package-level variables even if they aren't defined on the existing package policy during an upgrade, honoring default values for all variables if they exist. Fixes elastic#131251 * Fix not persisting default values for new variables * Fix tests * Address PR feedback + update tests
…lastic#131194) * [Stack Monitoring] Fix displaying ES version for external collection (elastic#126741) * Sort node versions and join into display string * Update test fixture * Update second test fixture Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Truncate tags at 20 characters * Fix kuery error in tags filter Fixes elastic#132092 * Tweak column widths + wrap out-of-date badge * Display full tags in tooltip + add filter tooltip Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
…he telemetry stats (elastic#132233) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
… from users events tab (elastic#132238) * Allow add endpoint event filter option from users events tab * Updates var name
* Fix host risk tab title * Update translations
* add collapse fn to table and xy chart * adjust documentation * bug fixes * tests and fixes * fix bug * allow color picking if collapse fn is active Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
…help performance (elastic#131363) * Batch setState calls to make sure all state updates are applied evenly * Remove sourcerer hook from useHoverActions and pass needed fields as props * Update snapshots, remove ReactDOM batching * Make row renderers aggregatable where appropriate * add pagination to details table * Fix hover actions on host/network details * Remove unneeded props * fix table pagination tests * update test * Show top n for authentications and threat indicator match rules * Fix anomaly score, entity, influence, and user id show top N * Pass props on wrapper and not data provider * Add missing row renderer draggables to use top N props * Update snapshots * Pr feedback Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co> Co-authored-by: Robert Austin <robert.austin@elastic.co>
* [SharedUX] Minor changes to no_data_views component * [SharedUX] Minor changes to no_data_views component * Fix translations * Applying Kaarinas comments * Text update * Update button text Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Maps] term join in produce help * snapshot updates * i18n clean up * API doc updates * update API docs, update copy * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * i18n cleanup Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* Add rule details page * Fix route * Fix route * Add useBreadcrumbs * Add rule summary * Complete rule data summary * Update styling * Add update rule * Add edit role * Update desgin * Add conditions * Add connectors icons * Fix more button * Remove unused FelxBox * Add fetch alerts * Move to items to components folder * Format dates * Add tabs * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * Use the shared getRuleStatusDropdown * Add permissions * Better handling errors * Fix styling * Fix tag style * Add tags component * Use tags component from triggers aciton ui * Add last24hAlerts hook * Fix last 24h alerts count hook * Fix large font * Fix font size * Fix size Actions * Fix fontsize page header * Fix conditions size * Fix text move vertically on small screen * Update style * Update alerts counts style * Cleanup * Add formatter for the interval * Add edit button on the definition section * Add delete modal * Add loader * Fix conditions panctuation * Fix size * Use the healthColor function from rule component * Add loading while deleting a rule * Use connectors name to show actions * Fix type * Fix rule page * Fix types * Use common RULES_PAGE_LINK var * Fix checks * Better error handling * Better i18n * Code review * Fix checks i18n * Use abort signal * Revert signal for loadRule as there is no tests * Fix style * Fixing tests * Reduce bundle size * Fix i18n * Bump limits
…s. (elastic#131773) * sort and cursor plumbing for alertsclient * process events route will now grab alerts for the page of events being requested. range / cursor support added to alerts client. * handling of missing event.action in some edge case process events * fixed to fake session leader overwriting original event it was based on * deduping added for children, alerts. fix to alerts route * fake process creation cleaned up, will now try and create fake parents from widened event context. this mitigates the number of potentially orphaned processes in the tree * fixed infinite loop regression :) * tests fixed * tweaks to inline alert details and test fixes * type fix * added test for new "sort" property in AlertsClient.find * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * pagination added to alerts tab * test fixes * addressed awp team comments * || -> ?? * e2e tests added for sort and search_after options * fixed test / type check * fixed import issue * restored whitespace Co-authored-by: mitodrummer <karlgodard@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* add log and sqrt scale * Update axis_settings_popover.tsx * fix types * carry over scale types * fix test * bump limit ever so slightly Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
* add flex column styles * remove whitespace * reorder
…stic#131958) * fix grouping * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * fix grouping * unit tests * fix types * update comment Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
* adding span links data generation * fixing span links synthtrace * adding span links * span links route * fixing span links new scenario * adding span links * improved tab structure * span links table * adding span links data generation * fixing span links synthtrace * adding span links * span links route * fixing span links new scenario * adding span links * span links table * improved tab structure * adjusting table * fixing ts issue * filtering data within timerange * fixing ts * fixing ci * disabling select option when no link available * adding api tests * fixing tests * e2e tests * fixing too_many_nested_clauses issue * refactoring apis * api tests * fixing e2e tests * fixing links * renaming link * fixing tests * addressing PR comments * fixing test * fixing ci * fixing ci * addressing pr comments * passing processor event to incoming links API * updating api tests * renaming incoming and outgoing * wrapping type into details property * renaming incoming/outgoing * pr comments * adding processor event to query * renaming * new API tests * import fix * renaming * adding e2e tests * addressing pr changes * changing link * Adding filter on children fetch * renaming services on test * renaming Co-authored-by: Boris Kirov <borisasenovkirov@gmail.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [ML] Adding UI for question_answering mmodel testing * translations
…lastic#132271) * Add link to installation docs in Fleet Server instructions * Use docLinks for installation docs
* [artifacts] Enable Cloud smoke tests * login selection for cloud * fix config path * fix protocol * fix url * cleanup * fix syntax * logs * fix username * update * -r * revert unused
…nd global entries (elastic#132542)
* Add warnings to system/managed policies * Fix translations, policies * Add jest tests * Add jest tests to assert new toggle behavior * Add jest tests for edit policy callout * Fix snapshot * [ML] Update jest tests with helper, rename helper for clarity * [ML] Add hook for local storage to remember toggle setting * [ML] Fix naming Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Maps] size legend * clean-up * refine spacing * clean up * more cleanup * use euiTheme for colors * fix jest test * do not show marker sizes for icons * remove lodash Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* remove human-readable automatic slug generation * make change non-breaking * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * remove test Co-authored-by: streamich <streamich@gmail.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
…ion (elastic#132586) * fix: disable drag-ability and hover actions for agent statuses The agent fields cannot be queried with ECS and therefore should not provide Filter In/Out functionality nor should users be able to add their representative fields to timeline investigations. Therefore users should not be able to add them to a timeline query by dragging them. * chore: make code more readable
* [DOCS] Remove obsolete license expiration info As of elastic/elasticsearch#79671, Elasticsearch does a more stringent license check rather than operating in a semi-degraded mode. Closes elastic#127845 Closes elastic#125702 * Update docs/management/managing-licenses.asciidoc Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
…ext (elastic#132650) Co-authored-by: mitodrummer <karlgodard@elastic.co>
* [maps] convert ESPewPewSource to typescript * move @ts-expect-error moved by fix
* Upgrade EUI to 55.1.3 backport * [Deprecation] Remove `watchedItemProps` from EuiContextMenu usage - should no longer be necessary * Update snapshots with new data-popover attr * Fix failing FTR test - Now that EuiContextMenu focus is restored correctly, there is a tooltip around the popover toggle that's blocking an above item that the test wants to click - swapping the order so that the tooltip does not block the clicked item should work * Fix 2nd maps FTR test with blocking tooltip Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
… fields that are only one level deep (elastic#132570) ## [Security Solution] Fixes sorting and tooltips on columns for non-ECS fields that are only one level deep This PR fixes <elastic#132490>, an issue where Timeline columns for non-ECS fields that are only one level deep couldn't be sorted, and displayed incomplete metadata in the column's tooltip. ### Before  _Before: The column is **not** sortable, and the tooltip displays incomplete metadata_ ### After  _After: The column is sortable, and the tooltip displays the expected metadata_ ### Desk testing See the _Steps to reproduce_ section of <elastic#132490> for testing details.
…ated Integrations, Required Fields, and Setup (elastic#132409) **Addresses partially:** elastic/security-team#2083, elastic/security-team#558, elastic/security-team#2856, elastic/security-team#1801 (internal tickets) ## Summary **TL;DR:** With this PR, it's now possible to specify `related_integrations`, `required_fields`, and `setup` fields in prebuilt rules in https://github.com/elastic/detection-rules. They are returned within rules in the API responses. This PR: - Adds 3 new fields to the model of Security detection rules. These fields are common to all of the rule types we have. - **Related Integrations**. It's a list of Fleet integrations associated with a given rule. It's assumed that if the user installs them, the rule might start to work properly because it will start receiving source events potentially matching the rule's query. - **Required Fields**. It's a list of event fields that must be present in the source indices of a given rule. - **Setup Guide**. It's any instructions for the user for setting up their environment in order to start receiving source events for a given rule. It's a text. Markdown is supported. It's similar to the Investigation Guide that we show on the Details page. - Adjusts API endpoints accordingly: - These fields are for prebuilt rules only and are supposed to be read-only in the UI. - Specifying these fields in the request parameters of the create/update/patch rule API endpoints is not supported. - These fields are returned in all responses that contain rules. If they are missing in a rule, default values are returned (empty array, empty string). - When duplicating a prebuilt rule, these fields are being reset to their default value (empty array, empty string). - Export/Import is supported. Edge case / supported hack: it's possible to specify these fields manually in a ndjson doc and import with a rule. - The fields are being copied to `kibana.alert.rule.parameters` field of an alert document, which is mapped as a flattened field type. No special handling for the new fields was needed there. - Adjusts tests accordingly. ## Related Integrations Example (part of a rule returned from the API): ```json { "related_integrations": [ { "package": "windows", "version": "1.5.x" }, { "package": "azure", "integration": "activitylogs", "version": "~1.1.6" } ], } ``` Schema: ```ts /** * Related integration is a potential dependency of a rule. It's assumed that if the user installs * one of the related integrations of a rule, the rule might start to work properly because it will * have source events (generated by this integration) potentially matching the rule's query. * * NOTE: Proper work is not guaranteed, because a related integration, if installed, can be * configured differently or generate data that is not necessarily relevant for this rule. * * Related integration is a combination of a Fleet package and (optionally) one of the * package's "integrations" that this package contains. It is represented by 3 properties: * * - `package`: name of the package (required, unique id) * - `version`: version of the package (required, semver-compatible) * - `integration`: name of the integration of this package (optional, id within the package) * * There are Fleet packages like `windows` that contain only one integration; in this case, * `integration` should be unspecified. There are also packages like `aws` and `azure` that contain * several integrations; in this case, `integration` should be specified. * * @example * const x: RelatedIntegration = { * package: 'windows', * version: '1.5.x', * }; * * @example * const x: RelatedIntegration = { * package: 'azure', * version: '~1.1.6', * integration: 'activitylogs', * }; */ export type RelatedIntegration = t.TypeOf<typeof RelatedIntegration>; export const RelatedIntegration = t.exact( t.intersection([ t.type({ package: NonEmptyString, version: NonEmptyString, }), t.partial({ integration: NonEmptyString, }), ]) ); ``` ## Required Fields Example (part of a rule returned from the API): ```json { "required_fields": [ { "name": "event.action", "type": "keyword", "ecs": true }, { "name": "event.code", "type": "keyword", "ecs": true }, { "name": "winlog.event_data.AttributeLDAPDisplayName", "type": "keyword", "ecs": false } ], } ``` Schema: ```ts /** * Almost all types of Security rules check source event documents for a match to some kind of * query or filter. If a document has certain field with certain values, then it's a match and * the rule will generate an alert. * * Required field is an event field that must be present in the source indices of a given rule. * * @example * const standardEcsField: RequiredField = { * name: 'event.action', * type: 'keyword', * ecs: true, * }; * * @example * const nonEcsField: RequiredField = { * name: 'winlog.event_data.AttributeLDAPDisplayName', * type: 'keyword', * ecs: false, * }; */ export type RequiredField = t.TypeOf<typeof RequiredField>; export const RequiredField = t.exact( t.type({ name: NonEmptyString, type: NonEmptyString, ecs: t.boolean, }) ); ``` ## Setup Guide Example (part of a rule returned from the API): ```json { "setup": "## Config\n\nThe 'PowerShell Script Block Logging' logging policy must be enabled.\nSteps to implement the logging policy with with Advanced Audit Configuration:\n\n```\nComputer Configuration > \nAdministrative Templates > \nWindows PowerShell > \nTurn on PowerShell Script Block Logging (Enable)\n```\n\nSteps to implement the logging policy via registry:\n\n```\nreg add \"hklm\\SOFTWARE\\Policies\\Microsoft\\Windows\\PowerShell\\ScriptBlockLogging\" /v EnableScriptBlockLogging /t REG_DWORD /d 1\n```\n", } ``` Schema: ```ts /** * Any instructions for the user for setting up their environment in order to start receiving * source events for a given rule. * * It's a multiline text. Markdown is supported. */ export type SetupGuide = t.TypeOf<typeof SetupGuide>; export const SetupGuide = t.string; ``` ## Details on the schema This PR adjusts all the 6 rule schemas we have: 1. Alerting Framework rule `params` schema: - `security_solution/server/lib/detection_engine/schemas/rule_schemas.ts` - `security_solution/server/lib/detection_engine/schemas/rule_converters.ts` 2. HTTP API main old schema: - `security_solution/common/detection_engine/schemas/response/rules_schema.ts` 3. HTTP API main new schema: - `security_solution/common/detection_engine/schemas/request/rule_schemas.ts` 4. Prebuilt rule schema: - `security_solution/common/detection_engine/schemas/request/add_prepackaged_rules_schema.ts` 5. Import rule schema: - `security_solution/common/detection_engine/schemas/request/import_rules_schema.ts` 6. Rule schema used on the frontend side: - `security_solution/public/detections/containers/detection_engine/rules/types.ts` Names of the fields on the HTTP API level: - `related_integrations` - `required_fields` - `setup` Names of the fields on the Alerting Framework level: - `params.relatedIntegrations` - `params.requiredFields` - `params.setup` ## Next steps - Create a new endpoint for returning installed Fleet integrations (gonna be a separate PR). - Rebase elastic#131475 on top of this PR after merge. - Cover the new fields with dedicated tests (gonna be a separate PR). - Update API docs (gonna be a separate PR). - Address the tech debt of having 6 different schemas (gonna create a ticket for that). ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Shareable rules list * Hide snooze panel in rules list * Address comments and added tests * Fix tests * Fix tests * Fix lint * Address design comments and fix tests Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
dimaanj
pushed a commit
that referenced
this pull request
Nov 9, 2022
* Updated EUI to version 67.1.2. Updated instaces of ButtonColor from EUI to EuiButtonColor. * Updated to EuiCard instances that utilize the betaBadgeProps object to return an empty string instead of undefined when the label is unavailable * Removed two instances of the deprecated internetExplorerOnly() mixin * Updated two instances of the ButtonColor import to EuiButtonColor as is was renamed in PR elastic#6150 * Updated snapshots in Jest Test Suite #1 to account for EuiButton and EuiCard Emotion conversions. Updated snapshots for EuiTooltip as it now contains the new EuiToolTipAnchor component that replaced the tooltip anchor styles * Updated snapshots in Jest Test Suite #2 to account forEuiButton, EuiDescriptionList, EuiButtonIcon, and EuiBadge Emotion conversions. * Updated snapshots in Jest Test Suite #3 to account for EuiDescriptionList, EuiButton, and EuiBadge Emotion conversions. Updated snapshots for EuiTooltip as if now contains the new EuiTooltipAnchor component that replaced the tooltop anchor styles * Updated snapshots in Jest Test Suite #4 to account for EuiButton Emotion conversion. * Updated snapshots in Jest Test Suite #5 to account for EuiButton Emotion conversion. * Updated snapshots in Jest Test Suite #8 to account for EuiButtonIcon and EuiButton Emotion conversions. Updated snapshots for EuiTooltip as it now contains the new EuiTooltipAnchor component that replaced the tooltip anchor styles. * Updated snapshots in Jest Test Suite elastic#9 to account for EuiFlyout and EuiButton Emotion conversions. * Updated snapshots in Jest Test Suite elastic#10 to account for EuiButton, EuiBadge, EuiButtonIcon, and EuiCard Emotion conversions. Updated snapshots for EuiToolTtip as it now contains the new EuiTooltipAnchor component that replaced the tooltip anchor styles * Updated instances of EuiButtonIconColor to use EuiButtonIconProps['color'] as it was removed in PR elastic#6150 * Updated tests that target EuiButton to simulate click events to target a generic button to prevent undefined click event errors * Updated snapshots in Jest Test Suite #1 to account for EuiButton and EuiCard Emotion conversions * Added the EuiFlyout mixins and variables to Lens Sass file as EuiFlyout has been converted to Emotion and the Sass styles are no longer available in EUI * Added the EuiCallOutTypes variable to Step Progress Sass file as EuiCallOut has been converted to Emotion and the Sass styles are no longer available in EUI * Updated snapshots in Jest Test Suite #2 to account for recent Emotion conversions. Updated snapshots in server_status.test.tsx to render EuiBadge before checking the snapshots to reduce the snapshot churn caused by Emotion. Updated tests that target EuiButton to simulate click events to target a generic button to prevent undefined click event errors * [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix' * Added imports for the added flyout mixin. Removed references to EuiCallOut mixin as the component has been converted to Emotion and is no longer available for use. * Updated unit tests and snapshots in Jest Test Suite elastic#10. Updated snaphshots to account for EuiBadge, EuiDescriptionList, EuiFlyout, and EuiCard Emotion conversions. Updated snapshots for EuiTooltip as it now contains the new EuiTooltipAnchor component that replaced the tooltip anchor styles. Updated tests that target EuiButton to simulate click events to target a generic button element to prevent undefined click event errors * Updated unit tests in Jest Test Suite elastic#11 that target EuiButton to simulate click events to target a generic button to prevent undefined click event errors * Updated unit tests in Jest Test Suite elastic#12 by updating tests that target EuiButton to simulate click events. Instead, these tests now target a generic button element to prevent undefined click event errors * Updated unit tests in Jest Test Suite #1 by updating tests that target EuiButton to simulate click events. Instead, these tests now target a generic button element to prevent undefined click event errors * Updated unit tests in Jest Test Suite #2 by updating tests that use EuiButton to simulate click events. Instead, these test have been updated to target a button element to prevent undefined click event errors. * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * Updated reference to mixins Sass file. Updated snapshots for Jest Test Suite #5 to account for EuiButton Emotion conversion. Updated unit tests that target EuiButton to simulate click events. These tests have been updated to target a button element to prevent undefined click event errors * Updated unit tests in Jest Test Suites 3, 7, 8, 13, and 14. Updated snapshot to account for EuiButton Emotion conversion. Updated tests that target EuiButton to simulate click events. These tests now target a generic button element to prevent undefined click event errors. Updated a few snapshots by adding .render() before checking the snapshot. This will prevent large snapshots coming from recent Emotion conversions * Updated snapshots in Jest Test Suite elastic#10 to account for the recent EuiButton Emotion conversion * Updated unit tests in Jest Test Suite #2 by editing tests that target EuiButton to simulate click events. These tests now target a button element in order to prevent undefinde click event errors * Updated snapshots in Jest Test Suite elastic#10 to account for EuiButton and EuiDescriptionList Emotion conversions * Updated test cases in Jest Test Suites 3, 7, and 8. Updated snapshots to account for EuiButton and EuiPagination Emotion conversions. Updated tests that target EuiButton to simulate click events. These tests now target a button element to prevent undefined click errors * Updated test cases in Jest Test Suite 14. Updated snapshots to account for EuiButton Emotion conversion. Opted to use .render() when updating a few snapshots to reduce the large length of snapshots caused by Emotion * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * Revised a change to betaBadgeProps to ensure that the label is available. If not, the value for the badge with be set to undefined. * Resolved two linting errors * Resolved two linting errors * Updated Jest unit tests in various suites. Updated snapshots to account for EuiButton Emotion conversion. Updated snapshots for EuiTooltip as it now contains the new EuiTooltipAnchor component that replaced the tooltip anchor styles. * Updated EuiFlyout in query_flyout.tsx to remove the onClick function from maskProps as it is no longer available. Updated this flyout to use ownFocus and not to close when the overlay mask is clicked. * Removed the use of EuiButtonIconColor in favor of EuiButtonIconProps['color'] * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * Updated Cypress test looking for strict equality on EuiPaginationButton class names to match a substring of the Emotion generated class name * Removed unneeded debugging code. Updated snapshots for various test suites to account for the recent EuiButton Emotion conversion * Updated a few EuiButton, EuiButtonEmpty, and EuiText components that set the color as ghost. The ghost color mode has been deprecated as of PR elastic#6150. These components now are wrapped in EuiThemeProvider with a dark colorMode to create the previous ghost color. * Resolved TS error with EuiCard betaBadgeProps * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * Remove references to now-removed EuiFlyout CSS classes/vars * Remove now-removed euiBadge className references - Convert directly to EuiBadge instead of using CSS - Remove confusing and now-possibly-irrelevant CSS badge overrides - left/right icons are now set via JSX and not via flex-direction * Pre-emptively fix various euiOverlayMask CSS overrides - this data attr isn't technically in yet but will be once elastic/eui#6289 merges - at the very least this isn't breaking any more than it currently already is! * Update to v67.1.3 * v67.1.4 * Resolved test failing test case in Security/Manage/Blocklist. The test did not remove focus from the last combo box in the form, which didn't allow the disbaled attribute to be removed from the flyout submit button. I've updated the mock file for Blocklist to return focus to the first form element in the flyout to allow the disabled attribute to be removed. * Updated snapshots to account for the recent EuiText Emotion conversion * Fix Log's custom tooltips relying on EuiTooltip classNames that no longer exist * Fix Vega vis custom tooltips relying on EuiTooltip classNames that no longer exist - this one is trickier than Log's as it's not using React, so we need to use Emotion's Global to set a static className * Convert remaining vega_vis.scss to Emotion - as an example of how other global + non global styles could be handled in the future * Fix references to removed `euiPaginationButton-isActive` className - use aria-current attribute instead * Added missing EuiFlyoutAnimation keyframes for EuiFlyout. This resolved test that failed because they used onAnimationEnd because the FlyoutAnimation could not be found. * Reolved Jest Tests in suites 1 and 5. Updated snapshots to account for the recent EuiButton Emotion conversion. Updated snapshots for EuiToolTip as it now contains the new EuiToolTipAnchor component that replaced the tooltip anchor styles. * iterate on rules_list.test.tsx * bump eui to v67.1.5 * Updatde snapshots for jest test suites to account for the recent EuiButton, EuiOverlayMask, EuiTooltip, and EuiBadge Emotion conversions * Resolved failing security test by updating the target element for CONNECTOR_TITLE. EuiCard has recently been converted to Emotion and the card title is no longer wrapper in a span. * Resolved failing test case in Runtime Fields. The modify runtime field test was failing because the combobox responsbible for adding and updating scripts was not appearing. The textbox did not appear because the shared setFieldScript function targets and toggles the script textbox when opening the flyout. When a runtime field is being modified, the toggle is already active and using the shared function will trigger the toggle again (losing access to the script textbox). Also resolved an issue that prevented the warning EuiCallout to appear when changing the type of a runtime field from its original type. Resolved this by adding an enter keypress at the end of setFieldType function to confirm the type selection, thus triggering the EuiCallout * Resolved two tests that were failing in Lens. These test were failing because they were checking for equality in class names that no longer exist within EuiButtonGroup as it was recently converted to Emotion. These tests were updated to check for a substring of the new and longer class name * Quick fix in test case failing because of misspelling in data-test-sub * Updated snapshot for Jest test case as EuiButton as recently been converted to Emotion * Removed console.log statement. Oops! * Resolved a failing test case in Lens. They were failing because they were checking for equality in class names that no longer exist within EuiButtonGroup as it was recently converted to Emotion. These tests were updated to check for a substring of the new and longer class name. Updated a Security test case by giving a target button the data-test-subj attribute for easier querying * Removed reference to EuiFlyout mixin as it has been converted to Emotion. Updated the reference to an interal copy of EuiFlyout styles * Corrected spelling error in EuiFlyout animation in Lens app * Update EUI with latest backport * Update button snapshots * fix another button snapshot * More snapshot fixes * [EuiButton][Security] Fix button relying on now-removed `euiButton__text` CSS - replace removed CSS with `eui-textTruncate` util instead - combine/DRY out unnecessary span - was affecting min-width of truncation util + increase screenshot diff limit - this was smaller than updating the actual baseline screenshots for whatever reason (likely render diff between local and CI) * Fix remaining Jest tests affected by Emotion conversions - because Emotion creates its own wrapper, `.first()` can no longer be used - prefer `.last()` instead * Fix Jest test affected by EuiButton Emotion conversion + removed modifier class - targeting the native DOM node + filtering by disabled true/false gets us back to the 'correct' lengths * Fix + improve flyout test - `.last()` changes to account for EuiButton Emotion conversion is needed, but the last onClose assertion still fails due to us having modified inputs, and the confirm modal being displayed - split test into two separate tests - one testing the onClose call, and the other testing the confirm modal * derpin * Skip rules_list Jest suite * Update new EuiButton snapshot * Upgraded EUI version to 67.1.7 * [EuiCard] Update snapshots * [EuiPopover] Update snapshots * [QA] Fix missing Vega warn/error message colors ;_; * [CI] Auto-commit changed files from 'node scripts/generate codeowners' * Fix Lens kbnToolbarButton regressions - Caused by flattening of EUI button CSS specificity - background-color was previously relying on isDisabled CSS specificity to override its #fff color - `text` color modifier & `!important` is no longer needed and overrides Emotion CSS flatly - isDisabled class is no longer needed - euiButton no longer sets `pointer-events: none` on disabled buttons (fixes tooltip bug in webkit as well) * Backport EUI 67.1.8 fixes * Update EuiCard snapshots * Fix EuiModal form wrapper causing overflow issues - see https://elastic.github.io/eui/#/layout/modal#forms-in-a-modal * Workaround for `.kbnOverlayMountWrapper` mount point causing overflow issues - not sure what all is using this modal service to be honest, but the wrapper is causing issues with the modal layout, this fixes overflow issues but will not fix any mask-image issues as a result * more snapshot updates * EuiButton - added textProps to EuiButton to prevent very long button names from spilling over outside of the container * EuiButton - Update EuiButton related snapshots. Updated tests that target EuiButton directly to use a data-telementary-id for more specific element querying required by Emotion * QA - Removed unnecessary comment in code * Temporary fix for EuiCard[selectable][layout=horizontal] instances on security solutions' rule page * Temporary fix for EuiCard[selectable][layout=horizontal] instances on osquery live query and canvas's datasource selector * [CI] Auto-commit changed files from 'node scripts/precommit_hook.js --ref HEAD~1..HEAD --fix' * Fix CSS specificity, where canvas's solutionToolbarButton's background-color now takes precedence over EuiButton's primary styles * Removed update to search_marker_tooltip that removed the euiTooltip styles and replaced then with Emotion styling. Added EuiTooltip Sass styles for the component to rely on to test for a styling bug that is causing the tooltip and the tooltip arrow to be out of sync with each other. * Lint Sass file * Lint Sass file * Removed overflow:hidden style from .vgaVis_view as it was causing euiScrollStyles not to present the scroll bars in Vega Vis * Remove typo from EuiButton textProps object. 'className' should not have been included in the actual class name * Revert tooltip Sass This reverts commit 20e6ead, a5cd2de, and c605cbd * Fix Emotion tooltip arrows Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Constance Chen <constance.chen@elastic.co> Co-authored-by: Chandler Prall <chandler.prall@elastic.co>
kibanamachine
added a commit
that referenced
this pull request
Nov 23, 2022
## Summary Closes elastic/ingest-dev#1261 Added a snippet to the telemetry that I added for each requirement. Please review and let me know if any changes are needed. Also asked a few questions below. @jlind23 @kpollich 6. is blocked by [elasticsearch change](elastic/elasticsearch#91701) to give kibana_system the missing privilege to read logs-elastic_agent* indices. Took inspiration for task versioning from https://github.com/elastic/kibana/pull/144494/files#diff-0c7c49bf5c55c45c19e9c42d5428e99e52c3a39dd6703633f427724d36108186 - [x] 1. Elastic Agent versions Versions of all the Elastic Agent running: `agent.version` field on `.fleet-agents` documents ``` "agent_versions": [ "8.6.0" ], ``` - [x] 2. Fleet server configuration Think we can query for `.fleet-policies` where some `input` has `type: 'fleet-server'` for this, as well as use the `Fleet Server Hosts` settings that we define via saved objects in Fleet ``` "fleet_server_config": { "policies": [ { "input_config": { "server": { "limits.max_agents": 10000 }, "server.runtime": "gc_percent:20" } } ] } ``` - [x] 3. Number of policies Count of `.fleet-policies` index To confirm, did we mean agent policies here? ``` "agent_policies": { "count": 7, ``` - [x] 4. Output type contained in those policies Collecting this from ts logic, querying from `.fleet-policies` index. The alternative would be to write a painless script (because the `outputs` are an object with dynamic keys, we can't do an aggregation directly). ``` "agent_policies": { "output_types": [ "elasticsearch" ] } ``` Did we mean to just collect the types here, or any other info? e.g. output urls - [x] 5. Average number of checkin failures We only have the most recent checkin status and timestamp on `.fleet-agents`. Do we mean here to publish the total last checkin failure count? E.g. 3 if 3 agents are in failure checkin status currently. Or do we mean to publish specific info for all agents (`last_checkin_status`, `last_checkin` time, `last_checkin_message`)? Are the only statuses `error` and `degraded` that we want to send? ``` "agent_last_checkin_status": { "error": 0, "degraded": 0 }, ``` - [ ] 6. Top 3 most common errors in the Elastic Agent logs Do we mean here elastic-agent logs only, or fleet-server logs as well (maybe separately)? I found an alternative way to query the message field using sampler and categorize text aggregation: ``` GET logs-elastic_agent*/_search { "size": 0, "query": { "bool": { "must": [ { "term": { "log.level": "error" } }, { "range": { "@timestamp": { "gte": "now-1h" } } } ] } }, "aggregations": { "message_sample": { "sampler": { "shard_size": 200 }, "aggs": { "categories": { "categorize_text": { "field": "message", "size": 10 } } } } } } ``` Example response: ``` "aggregations": { "message_sample": { "doc_count": 112, "categories": { "buckets": [ { "doc_count": 73, "key": "failed to unenroll offline agents", "regex": ".*?failed.+?to.+?unenroll.+?offline.+?agents.*?", "max_matching_length": 36 }, { "doc_count": 7, "key": """stderr panic close of closed channel n ngoroutine running Stop ngh.neting.cc/elastic/beats/v7/libbeat/cmd/instance Beat launch.func5 \n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go n ``` - [x] 7. Number of checkin failure over the past period of time I think this is almost the same as #5. The difference would be to report new failures happened only in the last hour, or report all agents in failure state. (which would be an increasing number if the agent stays in failed state). Do we want these 2 separate telemetry fields? EDIT: removed the last1hr query, instead added a new field to report agents enrolled per policy (top 10). See comments below. ``` "agent_checkin_status": { "error": 3, "degraded": 0 }, "agents_per_policy": [2, 1000], ``` - [x] 8. Number of Elastic Agent and number of fleet server This is already there in the existing telemetry: ``` "agents": { "total_enrolled": 0, "healthy": 0, "unhealthy": 0, "offline": 0, "total_all_statuses": 1, "updating": 0 }, "fleet_server": { "total_enrolled": 0, "healthy": 0, "unhealthy": 0, "offline": 0, "updating": 0, "total_all_statuses": 0, "num_host_urls": 1 }, ``` ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
dimaanj
pushed a commit
that referenced
this pull request
Nov 24, 2022
# Backport This will backport the following commits from `main` to `8.6`: - [Fleet Usage telemetry extension (elastic#145353)](elastic#145353) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Julia Bardi","email":"90178898+juliaElastic@users.noreply.github.com"},"sourceCommit":{"committedDate":"2022-11-23T09:22:20Z","message":"Fleet Usage telemetry extension (elastic#145353)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/ingest-dev/issues/1261\r\n\r\nAdded a snippet to the telemetry that I added for each requirement.\r\nPlease review and let me know if any changes are needed.\r\nAlso asked a few questions below. @jlind23 @kpollich \r\n\r\n6. is blocked by [elasticsearch\r\nchange](elastic/elasticsearch#91701) to give\r\nkibana_system the missing privilege to read logs-elastic_agent* indices.\r\n\r\nTook inspiration for task versioning from\r\nhttps://github.com/elastic/pull/144494/files#diff-0c7c49bf5c55c45c19e9c42d5428e99e52c3a39dd6703633f427724d36108186\r\n\r\n- [x] 1. Elastic Agent versions\r\nVersions of all the Elastic Agent running: `agent.version` field on\r\n`.fleet-agents` documents\r\n\r\n```\r\n\"agent_versions\": [\r\n \"8.6.0\"\r\n ],\r\n```\r\n\r\n- [x] 2. Fleet server configuration\r\nThink we can query for `.fleet-policies` where some `input` has `type:\r\n'fleet-server'` for this, as well as use the `Fleet Server Hosts`\r\nsettings that we define via saved objects in Fleet\r\n\r\n\r\n```\r\n \"fleet_server_config\": {\r\n \"policies\": [\r\n {\r\n \"input_config\": {\r\n \"server\": {\r\n \"limits.max_agents\": 10000\r\n },\r\n \"server.runtime\": \"gc_percent:20\"\r\n }\r\n }\r\n ]\r\n }\r\n```\r\n\r\n- [x] 3. Number of policies\r\nCount of `.fleet-policies` index \r\n\r\nTo confirm, did we mean agent policies here?\r\n\r\n```\r\n \"agent_policies\": {\r\n \"count\": 7,\r\n```\r\n\r\n- [x] 4. Output type contained in those policies\r\nCollecting this from ts logic, querying from `.fleet-policies` index.\r\nThe alternative would be to write a painless script (because the\r\n`outputs` are an object with dynamic keys, we can't do an aggregation\r\ndirectly).\r\n\r\n```\r\n\"agent_policies\": {\r\n \"output_types\": [\r\n \"elasticsearch\"\r\n ]\r\n }\r\n```\r\n\r\nDid we mean to just collect the types here, or any other info? e.g.\r\noutput urls\r\n\r\n- [x] 5. Average number of checkin failures\r\nWe only have the most recent checkin status and timestamp on\r\n`.fleet-agents`.\r\n\r\nDo we mean here to publish the total last checkin failure count? E.g. 3\r\nif 3 agents are in failure checkin status currently.\r\nOr do we mean to publish specific info for all agents\r\n(`last_checkin_status`, `last_checkin` time, `last_checkin_message`)?\r\nAre the only statuses `error` and `degraded` that we want to send?\r\n\r\n```\r\n \"agent_last_checkin_status\": {\r\n \"error\": 0,\r\n \"degraded\": 0\r\n },\r\n```\r\n\r\n- [ ] 6. Top 3 most common errors in the Elastic Agent logs\r\n\r\nDo we mean here elastic-agent logs only, or fleet-server logs as well\r\n(maybe separately)?\r\n\r\nI found an alternative way to query the message field using sampler and\r\ncategorize text aggregation:\r\n```\r\nGET logs-elastic_agent*/_search\r\n{\r\n \"size\": 0,\r\n \"query\": {\r\n \"bool\": {\r\n \"must\": [\r\n {\r\n \"term\": {\r\n \"log.level\": \"error\"\r\n }\r\n },\r\n {\r\n \"range\": {\r\n \"@timestamp\": {\r\n \"gte\": \"now-1h\"\r\n }\r\n }\r\n }\r\n ]\r\n }\r\n },\r\n \"aggregations\": {\r\n \"message_sample\": {\r\n \"sampler\": {\r\n \"shard_size\": 200\r\n },\r\n \"aggs\": {\r\n \"categories\": {\r\n \"categorize_text\": {\r\n \"field\": \"message\",\r\n \"size\": 10\r\n }\r\n }\r\n }\r\n }\r\n }\r\n}\r\n```\r\nExample response:\r\n```\r\n\"aggregations\": {\r\n \"message_sample\": {\r\n \"doc_count\": 112,\r\n \"categories\": {\r\n \"buckets\": [\r\n {\r\n \"doc_count\": 73,\r\n \"key\": \"failed to unenroll offline agents\",\r\n \"regex\": \".*?failed.+?to.+?unenroll.+?offline.+?agents.*?\",\r\n \"max_matching_length\": 36\r\n },\r\n {\r\n \"doc_count\": 7,\r\n \"key\": \"\"\"stderr panic close of closed channel n ngoroutine running Stop ngh.neting.cc/elastic/beats/v7/libbeat/cmd/instance Beat launch.func5 \\n\\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go n\r\n```\r\n\r\n\r\n- [x] 7. Number of checkin failure over the past period of time\r\n\r\nI think this is almost the same as #5. The difference would be to report\r\nnew failures happened only in the last hour, or report all agents in\r\nfailure state. (which would be an increasing number if the agent stays\r\nin failed state).\r\nDo we want these 2 separate telemetry fields?\r\n\r\nEDIT: removed the last1hr query, instead added a new field to report\r\nagents enrolled per policy (top 10). See comments below.\r\n\r\n```\r\n \"agent_checkin_status\": {\r\n \"error\": 3,\r\n \"degraded\": 0\r\n },\r\n \"agents_per_policy\": [2, 1000],\r\n```\r\n\r\n- [x] 8. Number of Elastic Agent and number of fleet server\r\n\r\nThis is already there in the existing telemetry:\r\n```\r\n \"agents\": {\r\n \"total_enrolled\": 0,\r\n \"healthy\": 0,\r\n \"unhealthy\": 0,\r\n \"offline\": 0,\r\n \"total_all_statuses\": 1,\r\n \"updating\": 0\r\n },\r\n \"fleet_server\": {\r\n \"total_enrolled\": 0,\r\n \"healthy\": 0,\r\n \"unhealthy\": 0,\r\n \"offline\": 0,\r\n \"updating\": 0,\r\n \"total_all_statuses\": 0,\r\n \"num_host_urls\": 1\r\n },\r\n```\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>","sha":"e00e26e86854bdbde7c14f88453b717505fed4d9","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v8.6.0","v8.7.0"],"number":145353,"url":"https://github.com/elastic/kibana/pull/145353","mergeCommit":{"message":"Fleet Usage telemetry extension (elastic#145353)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/ingest-dev/issues/1261\r\n\r\nAdded a snippet to the telemetry that I added for each requirement.\r\nPlease review and let me know if any changes are needed.\r\nAlso asked a few questions below. @jlind23 @kpollich \r\n\r\n6. is blocked by [elasticsearch\r\nchange](elastic/elasticsearch#91701) to give\r\nkibana_system the missing privilege to read logs-elastic_agent* indices.\r\n\r\nTook inspiration for task versioning from\r\nhttps://github.com/elastic/pull/144494/files#diff-0c7c49bf5c55c45c19e9c42d5428e99e52c3a39dd6703633f427724d36108186\r\n\r\n- [x] 1. Elastic Agent versions\r\nVersions of all the Elastic Agent running: `agent.version` field on\r\n`.fleet-agents` documents\r\n\r\n```\r\n\"agent_versions\": [\r\n \"8.6.0\"\r\n ],\r\n```\r\n\r\n- [x] 2. Fleet server configuration\r\nThink we can query for `.fleet-policies` where some `input` has `type:\r\n'fleet-server'` for this, as well as use the `Fleet Server Hosts`\r\nsettings that we define via saved objects in Fleet\r\n\r\n\r\n```\r\n \"fleet_server_config\": {\r\n \"policies\": [\r\n {\r\n \"input_config\": {\r\n \"server\": {\r\n \"limits.max_agents\": 10000\r\n },\r\n \"server.runtime\": \"gc_percent:20\"\r\n }\r\n }\r\n ]\r\n }\r\n```\r\n\r\n- [x] 3. Number of policies\r\nCount of `.fleet-policies` index \r\n\r\nTo confirm, did we mean agent policies here?\r\n\r\n```\r\n \"agent_policies\": {\r\n \"count\": 7,\r\n```\r\n\r\n- [x] 4. Output type contained in those policies\r\nCollecting this from ts logic, querying from `.fleet-policies` index.\r\nThe alternative would be to write a painless script (because the\r\n`outputs` are an object with dynamic keys, we can't do an aggregation\r\ndirectly).\r\n\r\n```\r\n\"agent_policies\": {\r\n \"output_types\": [\r\n \"elasticsearch\"\r\n ]\r\n }\r\n```\r\n\r\nDid we mean to just collect the types here, or any other info? e.g.\r\noutput urls\r\n\r\n- [x] 5. Average number of checkin failures\r\nWe only have the most recent checkin status and timestamp on\r\n`.fleet-agents`.\r\n\r\nDo we mean here to publish the total last checkin failure count? E.g. 3\r\nif 3 agents are in failure checkin status currently.\r\nOr do we mean to publish specific info for all agents\r\n(`last_checkin_status`, `last_checkin` time, `last_checkin_message`)?\r\nAre the only statuses `error` and `degraded` that we want to send?\r\n\r\n```\r\n \"agent_last_checkin_status\": {\r\n \"error\": 0,\r\n \"degraded\": 0\r\n },\r\n```\r\n\r\n- [ ] 6. Top 3 most common errors in the Elastic Agent logs\r\n\r\nDo we mean here elastic-agent logs only, or fleet-server logs as well\r\n(maybe separately)?\r\n\r\nI found an alternative way to query the message field using sampler and\r\ncategorize text aggregation:\r\n```\r\nGET logs-elastic_agent*/_search\r\n{\r\n \"size\": 0,\r\n \"query\": {\r\n \"bool\": {\r\n \"must\": [\r\n {\r\n \"term\": {\r\n \"log.level\": \"error\"\r\n }\r\n },\r\n {\r\n \"range\": {\r\n \"@timestamp\": {\r\n \"gte\": \"now-1h\"\r\n }\r\n }\r\n }\r\n ]\r\n }\r\n },\r\n \"aggregations\": {\r\n \"message_sample\": {\r\n \"sampler\": {\r\n \"shard_size\": 200\r\n },\r\n \"aggs\": {\r\n \"categories\": {\r\n \"categorize_text\": {\r\n \"field\": \"message\",\r\n \"size\": 10\r\n }\r\n }\r\n }\r\n }\r\n }\r\n}\r\n```\r\nExample response:\r\n```\r\n\"aggregations\": {\r\n \"message_sample\": {\r\n \"doc_count\": 112,\r\n \"categories\": {\r\n \"buckets\": [\r\n {\r\n \"doc_count\": 73,\r\n \"key\": \"failed to unenroll offline agents\",\r\n \"regex\": \".*?failed.+?to.+?unenroll.+?offline.+?agents.*?\",\r\n \"max_matching_length\": 36\r\n },\r\n {\r\n \"doc_count\": 7,\r\n \"key\": \"\"\"stderr panic close of closed channel n ngoroutine running Stop ngh.neting.cc/elastic/beats/v7/libbeat/cmd/instance Beat launch.func5 \\n\\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go n\r\n```\r\n\r\n\r\n- [x] 7. Number of checkin failure over the past period of time\r\n\r\nI think this is almost the same as #5. The difference would be to report\r\nnew failures happened only in the last hour, or report all agents in\r\nfailure state. (which would be an increasing number if the agent stays\r\nin failed state).\r\nDo we want these 2 separate telemetry fields?\r\n\r\nEDIT: removed the last1hr query, instead added a new field to report\r\nagents enrolled per policy (top 10). See comments below.\r\n\r\n```\r\n \"agent_checkin_status\": {\r\n \"error\": 3,\r\n \"degraded\": 0\r\n },\r\n \"agents_per_policy\": [2, 1000],\r\n```\r\n\r\n- [x] 8. Number of Elastic Agent and number of fleet server\r\n\r\nThis is already there in the existing telemetry:\r\n```\r\n \"agents\": {\r\n \"total_enrolled\": 0,\r\n \"healthy\": 0,\r\n \"unhealthy\": 0,\r\n \"offline\": 0,\r\n \"total_all_statuses\": 1,\r\n \"updating\": 0\r\n },\r\n \"fleet_server\": {\r\n \"total_enrolled\": 0,\r\n \"healthy\": 0,\r\n \"unhealthy\": 0,\r\n \"offline\": 0,\r\n \"updating\": 0,\r\n \"total_all_statuses\": 0,\r\n \"num_host_urls\": 1\r\n },\r\n```\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>","sha":"e00e26e86854bdbde7c14f88453b717505fed4d9"}},"sourceBranch":"main","suggestedTargetBranches":["8.6"],"targetPullRequestStates":[{"branch":"8.6","label":"v8.6.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/145353","number":145353,"mergeCommit":{"message":"Fleet Usage telemetry extension (elastic#145353)\n\n## Summary\r\n\r\nCloses https://github.com/elastic/ingest-dev/issues/1261\r\n\r\nAdded a snippet to the telemetry that I added for each requirement.\r\nPlease review and let me know if any changes are needed.\r\nAlso asked a few questions below. @jlind23 @kpollich \r\n\r\n6. is blocked by [elasticsearch\r\nchange](elastic/elasticsearch#91701) to give\r\nkibana_system the missing privilege to read logs-elastic_agent* indices.\r\n\r\nTook inspiration for task versioning from\r\nhttps://github.com/elastic/pull/144494/files#diff-0c7c49bf5c55c45c19e9c42d5428e99e52c3a39dd6703633f427724d36108186\r\n\r\n- [x] 1. Elastic Agent versions\r\nVersions of all the Elastic Agent running: `agent.version` field on\r\n`.fleet-agents` documents\r\n\r\n```\r\n\"agent_versions\": [\r\n \"8.6.0\"\r\n ],\r\n```\r\n\r\n- [x] 2. Fleet server configuration\r\nThink we can query for `.fleet-policies` where some `input` has `type:\r\n'fleet-server'` for this, as well as use the `Fleet Server Hosts`\r\nsettings that we define via saved objects in Fleet\r\n\r\n\r\n```\r\n \"fleet_server_config\": {\r\n \"policies\": [\r\n {\r\n \"input_config\": {\r\n \"server\": {\r\n \"limits.max_agents\": 10000\r\n },\r\n \"server.runtime\": \"gc_percent:20\"\r\n }\r\n }\r\n ]\r\n }\r\n```\r\n\r\n- [x] 3. Number of policies\r\nCount of `.fleet-policies` index \r\n\r\nTo confirm, did we mean agent policies here?\r\n\r\n```\r\n \"agent_policies\": {\r\n \"count\": 7,\r\n```\r\n\r\n- [x] 4. Output type contained in those policies\r\nCollecting this from ts logic, querying from `.fleet-policies` index.\r\nThe alternative would be to write a painless script (because the\r\n`outputs` are an object with dynamic keys, we can't do an aggregation\r\ndirectly).\r\n\r\n```\r\n\"agent_policies\": {\r\n \"output_types\": [\r\n \"elasticsearch\"\r\n ]\r\n }\r\n```\r\n\r\nDid we mean to just collect the types here, or any other info? e.g.\r\noutput urls\r\n\r\n- [x] 5. Average number of checkin failures\r\nWe only have the most recent checkin status and timestamp on\r\n`.fleet-agents`.\r\n\r\nDo we mean here to publish the total last checkin failure count? E.g. 3\r\nif 3 agents are in failure checkin status currently.\r\nOr do we mean to publish specific info for all agents\r\n(`last_checkin_status`, `last_checkin` time, `last_checkin_message`)?\r\nAre the only statuses `error` and `degraded` that we want to send?\r\n\r\n```\r\n \"agent_last_checkin_status\": {\r\n \"error\": 0,\r\n \"degraded\": 0\r\n },\r\n```\r\n\r\n- [ ] 6. Top 3 most common errors in the Elastic Agent logs\r\n\r\nDo we mean here elastic-agent logs only, or fleet-server logs as well\r\n(maybe separately)?\r\n\r\nI found an alternative way to query the message field using sampler and\r\ncategorize text aggregation:\r\n```\r\nGET logs-elastic_agent*/_search\r\n{\r\n \"size\": 0,\r\n \"query\": {\r\n \"bool\": {\r\n \"must\": [\r\n {\r\n \"term\": {\r\n \"log.level\": \"error\"\r\n }\r\n },\r\n {\r\n \"range\": {\r\n \"@timestamp\": {\r\n \"gte\": \"now-1h\"\r\n }\r\n }\r\n }\r\n ]\r\n }\r\n },\r\n \"aggregations\": {\r\n \"message_sample\": {\r\n \"sampler\": {\r\n \"shard_size\": 200\r\n },\r\n \"aggs\": {\r\n \"categories\": {\r\n \"categorize_text\": {\r\n \"field\": \"message\",\r\n \"size\": 10\r\n }\r\n }\r\n }\r\n }\r\n }\r\n}\r\n```\r\nExample response:\r\n```\r\n\"aggregations\": {\r\n \"message_sample\": {\r\n \"doc_count\": 112,\r\n \"categories\": {\r\n \"buckets\": [\r\n {\r\n \"doc_count\": 73,\r\n \"key\": \"failed to unenroll offline agents\",\r\n \"regex\": \".*?failed.+?to.+?unenroll.+?offline.+?agents.*?\",\r\n \"max_matching_length\": 36\r\n },\r\n {\r\n \"doc_count\": 7,\r\n \"key\": \"\"\"stderr panic close of closed channel n ngoroutine running Stop ngh.neting.cc/elastic/beats/v7/libbeat/cmd/instance Beat launch.func5 \\n\\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go n\r\n```\r\n\r\n\r\n- [x] 7. Number of checkin failure over the past period of time\r\n\r\nI think this is almost the same as #5. The difference would be to report\r\nnew failures happened only in the last hour, or report all agents in\r\nfailure state. (which would be an increasing number if the agent stays\r\nin failed state).\r\nDo we want these 2 separate telemetry fields?\r\n\r\nEDIT: removed the last1hr query, instead added a new field to report\r\nagents enrolled per policy (top 10). See comments below.\r\n\r\n```\r\n \"agent_checkin_status\": {\r\n \"error\": 3,\r\n \"degraded\": 0\r\n },\r\n \"agents_per_policy\": [2, 1000],\r\n```\r\n\r\n- [x] 8. Number of Elastic Agent and number of fleet server\r\n\r\nThis is already there in the existing telemetry:\r\n```\r\n \"agents\": {\r\n \"total_enrolled\": 0,\r\n \"healthy\": 0,\r\n \"unhealthy\": 0,\r\n \"offline\": 0,\r\n \"total_all_statuses\": 1,\r\n \"updating\": 0\r\n },\r\n \"fleet_server\": {\r\n \"total_enrolled\": 0,\r\n \"healthy\": 0,\r\n \"unhealthy\": 0,\r\n \"offline\": 0,\r\n \"updating\": 0,\r\n \"total_all_statuses\": 0,\r\n \"num_host_urls\": 1\r\n },\r\n```\r\n\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\nCo-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>","sha":"e00e26e86854bdbde7c14f88453b717505fed4d9"}}]}] BACKPORT--> Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
WIP, add testing the the editing of searchSource based alerting
Checklist
Delete any items that are not applicable to this PR.
Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.
When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:
For maintainers