Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Ingest Manager] Add retries for registry requests. fixes #74598 #74507

Merged
merged 1 commit into from
Aug 20, 2020
Merged

[Ingest Manager] Add retries for registry requests. fixes #74598 #74507

merged 1 commit into from
Aug 20, 2020

Conversation

jfsiii
Copy link
Contributor

@jfsiii jfsiii commented Aug 6, 2020 •
edited
Loading

Summary

Use existing Kibana dependency p-retry to retry requests to the registry which fail with a timeout or other node/system errors (vs a 4xx or 5xx). Fixes #74598

Checklist

Added tests

setupIngestManager
  fetchUrl / getResponse errors
    ✓ regular Errors do not retry. Becomes RegistryError (10ms)
    ✓ TypeErrors do not retry. Becomes RegistryError (1ms)
    only system errors retry (like ECONNRESET)
      ✓ they eventually succeed (7037ms)
      ✓ or error after 1 failure & 5 retries with RegistryError (31021ms)

Two movies with logging showing behavior during different scenarios

  1. ingest setup with good and bad registry.mp4.zip
  2. retry get epm packages with flaky registry.mp4.zip

@jfsiii jfsiii marked this pull request as ready for review August 6, 2020 20:51
@jfsiii jfsiii requested a review from a team August 6, 2020 20:51
@jfsiii jfsiii changed the title [Ingest Manager] [DRAFT] Add retries for registry requests. [Ingest Manager] Add retries for registry requests. Aug 6, 2020
@jfsiii jfsiii added the release_note:skip Skip the PR/issue when compiling release notes label Aug 7, 2020
@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 7, 2020

cc @ruflin @jonathan-buttner

@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 7, 2020

@elasticmachine merge upstream

@botelastic botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Aug 7, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Team:Ingest Management)

@ruflin ruflin added the v7.9.1 label Aug 10, 2020
@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 10, 2020

@elasticmachine merge upstream

@jfsiii jfsiii requested a review from skh August 12, 2020 21:27
@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 12, 2020

@elasticmachine merge upstream

jfsiii
jfsiii commented Aug 13, 2020
View reviewed changes
x-pack/plugins/ingest_manager/server/routes/setup/index.ts Outdated

export const registerRoutes = (router: IRouter, config: IngestManagerConfigType) => {
// Ingest manager setup
registerIngestManagerSetupRoute(router);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I split this function up and exported the individual register*Route functions for some now-removed tests. I left it because it seems useful is common in other Kibana plugins.

@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 13, 2020

@elasticmachine merge upstream

2 similar comments
@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 14, 2020

@elasticmachine merge upstream

@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 17, 2020

@elasticmachine merge upstream

@jfsiii jfsiii mentioned this pull request Aug 18, 2020
Merged
1 task
@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 19, 2020

@elasticmachine merge upstream

@jfsiii jfsiii changed the title [Ingest Manager] Add retries for registry requests. [Ingest Manager] Add retries for registry requests. fixes #74598 Aug 19, 2020
@jfsiii jfsiii self-assigned this Aug 19, 2020
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 20, 2020

going to merge this as the code matches the PR @skh approved. I re-requested a review when I added some behavior which has now been removed

@jfsiii jfsiii merged commit 0f494f4 into elastic:master Aug 20, 2020
This was referenced Aug 20, 2020
Merged
Merged
jfsiii pushed a commit that referenced this pull request Aug 21, 2020
Retry network errors on registry requests (#74507) (#75614)
dbfc6c8
thomasneirynck pushed a commit to thomasneirynck/kibana that referenced this pull request Aug 21, 2020
@thomasneirynck
Retry network errors on registry requests (elastic#74507)
989c87b
jfsiii pushed a commit that referenced this pull request Aug 22, 2020
Retry network errors on registry requests (#74507) (#75615)
9cab3aa
@jfsiii jfsiii mentioned this pull request Aug 31, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@skh skh Awaiting requested review from skh

Assignees

@jfsiii jfsiii

Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.9.1 v7.10.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Ingest Manager] Retry for failed request to Registry
5 participants
@jfsiii @elasticmachine @kibanamachine @skh @ruflin