[Security Solution] Enable Cypress tests in Serverless: Detection Engine area #161539
Assignees
Labels
8.12 candidate
Team:Detection Engine
Security Solution Detection Engine Area
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
test
test-coverage
issues & PRs for improving code test coverage
Comments
banderror
changed the title
banderror
added
test
test-coverage
banderror
changed the title
banderror
changed the title
This was referenced Sep 1, 2023
banderror
added a commit
that referenced
this issue
Sep 9, 2023
…rverless (#165966) **Resolves: #164441 ## Summary Skips all Cypress tests owned by @elastic/security-detection-rule-management and @elastic/security-detection-engine teams in Serverless using the new `@skipInServerless` tag. - Adds a new `@skipInServerless` tag - Updates `x-pack/test/security_solution_cypress/cypress/README.md` to explain when to use what tags - Explicitly adds missing tags to all D&R tests - Adds `// TODO:` comments to tests with links to follow-up tickets - Fixes the `x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts` script (see below) Follow-up work: - #161540 - #161539 ## Context > Serverless test failures will soon block PR merge > During the development of the serverless test infrastructure, we decided that serverless tests will only soft-fail. That means you see the test failure in your PR but you're still able to merge. We did that mainly in order to not block delivery of stateful features and bug fixes while serverless tests and pipelines were implemented and stabilized. We now have the major building blocks for the serverless test infrastructure in place and will integrate serverless tests in our regular pipelines. As part of this process, we're skipping failing and flaky serverless tests that came in during the soft-fail phase. Once this is done, a PR with serverless test failures can no longer be merged, similar to how we have it for stateful test failures. > We plan to merge this in the next few days and we'll send out another notification when it's done. ## Fixing `parallel.ts` There were two problems with the `x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts` script we use for running Cypress tests: - The script was broken in #162673 (here on [this line](https://github.com/elastic/kibana/pull/162673/files#diff-9f40ced6d29c4fc2709af881680400293d8ce1bc9ebb07b9138d6d99c83c09c9R67)) - I think it has never supported situations when all tests matching a spec pattern (such as `./cypress/e2e/!(investigations|explore)/**/*.cy.ts`) end up being skipped via Cypress tags (such as `@skipInServerless`) Both the issues are fixed in this PR. Code owners are added for this script in the CODEOWNERS file to prevent breaking this script in future PRs.
This was referenced Sep 19, 2023
maximpn
added a commit
that referenced
this issue
Sep 21, 2023
**Related to:** #161540, #161539 ## Summary Always force the last breadcrumb to be inactive. ## Details Usual UX expects the last breadcrumb to be inactive as it represents the current page. The same can be seen from EUI [examples](https://eui.elastic.co/#/navigation/breadcrumbs). It turns out Serverless Security Solution plugin does't remove `href` and `onClick` fields from the last breadcrumb and passes it to `chrome.setBreadcrumbs()` or `serverless.setBreadcrumbs()` which renders the last breadcrumb as active but clicking on it only refreshes the page. ESS Security Solution on the other hand processes breadcrumbs currently. The same behavior may be the case for the other plungs as well. As it's much simpler to strip off undesired fields at one place instead of processing them in plugins it's done in `packages/core/chrome/core-chrome-browser-internal/src/ui/header/header_breadcrumbs.tsx`. Security Solution codebase has been updated accordingly. A side effect of this PR is consistent ESS and Serverless breadcrumbs behavior and it will help to reuse ESS tests for Serverless.
maximpn
added a commit
that referenced
this issue
Oct 4, 2023
… test (#167083) **Addresses:** #161539 ## Summary This PR unskips Threshold rule creation Cypress tests. ## Details To unskip the test with minimal changes it required to refactor navigation via breadcrumbs and update rules table breadcrumb selectors to make navigation back to the rules table work seamlessly between ESS and Serverless. ## Flaky Test runner Serverless [threshold_rule.cy.ts 150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3213) 🟢
Overview of Detection Engine Cypress tests
PRs used for investigation/enabling tests
|
|
Hey @yctercero, as we discussed over zoom, @jpdjere did a great research on what tests owned by the Detection Engine team can be unskipped and enabled in ESS and Serverless, and also enabled many of them. The result of these efforts is the wonderful table above that shows what tests are still broken/flaky and thus skipped in Serverless and/or ESS. Having that, I'd like to consider this ticket addressed and close it. Please feel free to reopen it when you get back to fixing the rest of the Detection Engine tests, or open a new ticket - whichever you will find more convenient. |
|
Thanks @banderror - I created a follow up issue for our team - #171366 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Epic: #153633
Depends on: #161537, #164441
Summary
We need to create a suite of Cypress E2E tests that will run against Serverless environments.
In
x-pack/test/security_solution_cypress, we already have existing Cypress tests that run "in ESS", meaning against non-serverless environments. We need to start gradually including these Cypress tests in the Serverless suite by:BROKEN_IN_SERVERLESSand removing this tagSKIP_IN_SERVERLESSand removing this tagSERVERLESSIdeally, we'd do it gradually, in multiple PRs, starting from covering the most critical features and use cases, and proceeding with less important ones.
The text was updated successfully, but these errors were encountered: