Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing ES Promotion: Several Security Solutions Suites #125851

Closed
spalger opened this issue Feb 16, 2022 · 9 comments
Closed

Failing ES Promotion: Several Security Solutions Suites #125851

spalger opened this issue Feb 16, 2022 · 9 comments
Labels
failed-es-promotion Team:Defend Workflows “EDR Workflows” sub-team of Security Solution Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@spalger
Copy link
Contributor

spalger commented Feb 16, 2022 •
edited
Loading

Several Security Solution tests

These failures are preventing the promotion of the current Elasticsearch nightly snapshot.

For more information on the Elasticsearch snapshot promotion process including how to reproduce using the unverified nightly ES build: https://www.elastic.co/guide/en/kibana/master/development-es-snapshots.html

Failed promotion jobs:

  1. https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify/builds/850
  2. https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify/builds/853
  3. https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify/builds/854
  4. https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify/builds/858
  5. https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify/builds/860
  6. https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify/builds/861
  7. https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify/builds/863
  8. https://buildkite.com/elastic/kibana-elasticsearch-snapshot-verify/builds/864

NOTE: Turns out these are only a sample of the failures, I'm giving up trying to skip only what's necessary and I've skipped all the neighbor suites to the ones listed above. I recommend searching for all the // FAILING ES PROMOTION: https://github.com/elastic/kibana/issues/125851 comments, removing them and their associated skips, and then running the modified suites locally or in a PR once the main es snapshot is updated to get these working with the new snapshot.

Some of the Failures:
@botelastic botelastic bot added the needs-team Issues missing a team label label Feb 16, 2022
@spalger spalger mentioned this issue Feb 16, 2022
Closed
spalger pushed a commit that referenced this issue Feb 16, 2022
skip suite blocking es promotion (#125851)
45b719d
@spalger
Copy link
Contributor Author

spalger commented Feb 16, 2022 •
edited
Loading

Skipped

main/8.2: 45b719d + 3f77c5f + ba06b99 + d013548 + 0a850d8 + c115547 + f64a0d6 + 104d119 + 98a0e18

@spalger
Copy link
Contributor Author

spalger commented Feb 16, 2022 •
edited
Loading

Broke with #125848, might be related though the failures are significantly different

Merging into a one issue along with other failures.

@spalger spalger added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Feb 16, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Feb 16, 2022
@botelastic botelastic bot removed the needs-team Issues missing a team label label Feb 16, 2022
@spalger spalger changed the title Failing ES Promotion: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/create_rules·ts - detection engine api security and spaces enabled create_rules missing timestamps should create a single rule which has a timestamp override and generates two signals with a "partial failure" status Failing ES Promotion: Several Security Solutions Suites Feb 16, 2022
spalger pushed a commit that referenced this issue Feb 16, 2022
unify skipped suites for es promotion (#125851)
d013548
spalger pushed a commit that referenced this issue Feb 16, 2022
skip more suits failing es promotion (#125851)
0a850d8
spalger pushed a commit that referenced this issue Feb 16, 2022
skip another suite failing es promotion (#125851)
c115547
spalger pushed a commit that referenced this issue Feb 16, 2022
skip another security solution suite blocking es promotion (#125851)
f64a0d6
spalger pushed a commit that referenced this issue Feb 16, 2022
skip another security solution suite blocking es promotion (#125851)
104d119
spalger pushed a commit that referenced this issue Feb 17, 2022
skip another security solution suite blocking es promotion (#125851)
98a0e18
spalger pushed a commit that referenced this issue Feb 17, 2022
skip another security solution suite blocking es promotion (#125851)
889c68a
spalger pushed a commit that referenced this issue Feb 17, 2022
skip more security solution suites blocking es promotion (#125851)
de042af
@kevinlog kevinlog added the Team:Defend Workflows “EDR Workflows” sub-team of Security Solution label Feb 17, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

@kevinlog
Copy link
Contributor

kevinlog commented Feb 17, 2022 •
edited
Loading

@paul-tavares @dasansol92 - It looks like our tests are failing to install the Endpoint package at startup after the ES upgrade. A couple recurring errors I'm seeing:

EndpointDataLoadingError: Error during extraction of package: Error: end of central directory record signature not found. Assumed content type was application/zip, check if this matches the archive type.

Error: Unable to create Package Policy via Ingest!

This was referenced Feb 17, 2022
Closed
Merged
paul-tavares added a commit to paul-tavares/kibana that referenced this issue Feb 17, 2022
@paul-tavares
Unskip Endpoint FTR tests as a result of ES promotion
84bb50c 
 from: elastic#125851
paul-tavares added a commit that referenced this issue Feb 17, 2022
@paul-tavares
[Security Solution][Endpoint] Un-skip Endpoint FTR tests as a result …
5555ac7 
…of ES promotion failures (#125963)

Fixes some issues from #125851
@paul-tavares
Copy link
Contributor

The endpoint specific tests have been un-skiped: #125963

Others are being worked by the @elastic/security-detections-response team

@FrankHassanabad FrankHassanabad mentioned this issue Feb 23, 2022
Merged
1 task
@liza-mae
Copy link
Contributor

liza-mae commented Feb 24, 2022 •
edited
Loading

I have also skipped these tests from the cloud and platform runs: test/detection_engine_api_integration/security_and_spaces
https://github.com/elastic/elastic-stack-testing/pull/1139

lucasfcosta pushed a commit to lucasfcosta/kibana that referenced this issue Mar 2, 2022
@paul-tavares @lucasfcosta
[Security Solution][Endpoint] Un-skip Endpoint FTR tests as a result …
628634c 
…of ES promotion failures (elastic#125963)

Fixes some issues from elastic#125851
@FrankHassanabad
Copy link
Contributor

The tests were failing and flake because of this issue:
elastic/elasticsearch#84256

This PR is un-skipping them as that should be fixed now:
#126294

Once that builds I will merge it and we should be ok to close this ticket.

FrankHassanabad added a commit that referenced this issue Mar 15, 2022
@FrankHassanabad
[Security Solutions] Increases timeout and unskips detection tests (#…
c060b9f 
…126294)

## Summary

Increases the timeouts from 2 minutes to now 5 minutes and unskips the detection tests. If any of the tests fail consistently then I will skip just those individual tests instead of the whole suit.

See #125851 and elastic/elasticsearch#84256

This could cause issues with:
#125319

If so, then we will have to deal with the cloud based tests in a different way but in reality we need the extra time as some test cases do take a while to run on CI.

This also:
* Removes skips around code that have been in there for a while and adds documentation to the parts that are left over.

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@spalger
Copy link
Contributor Author

spalger commented Mar 15, 2022

🎉🎉🎉

maksimkovalev pushed a commit to maksimkovalev/kibana that referenced this issue Mar 18, 2022
@FrankHassanabad @maksimkovalev
[Security Solutions] Increases timeout and unskips detection tests (e…
1169c74 
…lastic#126294)

## Summary

Increases the timeouts from 2 minutes to now 5 minutes and unskips the detection tests. If any of the tests fail consistently then I will skip just those individual tests instead of the whole suit.

See elastic#125851 and elastic/elasticsearch#84256

This could cause issues with:
elastic#125319

If so, then we will have to deal with the cloud based tests in a different way but in reality we need the extra time as some test cases do take a while to run on CI.

This also:
* Removes skips around code that have been in there for a while and adds documentation to the parts that are left over.

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
failed-es-promotion Team:Defend Workflows “EDR Workflows” sub-team of Security Solution Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@FrankHassanabad @spalger @elasticmachine @liza-mae @kibanamachine @kevinlog @paul-tavares