Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[esArchiver] log when migrations complete and we're done loading data #89938

Merged

Conversation

spalger
Copy link
Contributor

@spalger spalger commented Feb 1, 2021

I'm tracking a likely upcoming source of flakiness, probably caused by #85778

I'm 50% sure this issue is in the esArchiver, and I'd like to avoid reverting that massive PR, so I'm looking to do some logging that helps us identify what is causing failures like https://kibana-ci.elastic.co/job/elastic+kibana+pipeline-pull-request/102742/execution/node/965/log/

14:56:00                 │ proc [kibana]   log   [21:56:00.450] [info][savedobjects-service] [.kibana] Migration completed after 185ms
14:56:56                 │ proc [kibana]   log   [21:56:53.930] [error][plugins][taskManager] [WorkloadAggregator]: Error: Invalid workload: {"took":0,"timed_out":false,"_shards":{"total":0,"successful":0,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0,"hits":[]}}
14:56:57                 │ proc [kibana]   log   [21:56:57.210] [warning][collector-set][plugins][usageCollection] ResponseError: index_not_found_exception
14:56:57                 │ proc [kibana]     at onBody (/dev/shm/workspace/kibana-build-xpack-4/node_modules/@elastic/elasticsearch/lib/Transport.js:333:23)
14:56:57                 │ proc [kibana]     at IncomingMessage.onEnd (/dev/shm/workspace/kibana-build-xpack-4/node_modules/@elastic/elasticsearch/lib/Transport.js:260:11)
14:56:57                 │ proc [kibana]     at IncomingMessage.emit (events.js:327:22)
14:56:57                 │ proc [kibana]     at endReadableNT (internal/streams/readable.js:1327:12)
14:56:57                 │ proc [kibana]     at processTicksAndRejections (internal/process/task_queues.js:80:21) {
14:56:57                 │ proc [kibana]   meta: {
14:56:57                 │ proc [kibana]     body: { error: [Object], status: 404 },
14:56:57                 │ proc [kibana]     statusCode: 404,
14:56:57                 │ proc [kibana]     headers: {
14:56:57                 │ proc [kibana]       'content-type': 'application/json;charset=utf-8',
14:56:57                 │ proc [kibana]       'content-length': '449'
14:56:57                 │ proc [kibana]     },
14:56:57                 │ proc [kibana]     meta: {
14:56:57                 │ proc [kibana]       context: null,
14:56:57                 │ proc [kibana]       request: [Object],
14:56:57                 │ proc [kibana]       name: 'elasticsearch-js',
14:56:57                 │ proc [kibana]       connection: [Object],
14:56:57                 │ proc [kibana]       attempts: 0,
14:56:57                 │ proc [kibana]       aborted: false
14:56:57                 │ proc [kibana]     }
14:56:57                 │ proc [kibana]   }
14:56:57                 │ proc [kibana] }
14:56:57                 │ proc [kibana]   log   [21:56:57.216] [warning][collector-set][plugins][usageCollection] Unable to fetch data from core collector
14:58:04                 │ proc [kibana]   log   [21:57:53.935] [error][plugins][taskManager] [WorkloadAggregator]: Error: Invalid workload: {"took":0,"timed_out":false,"_shards":{"total":0,"successful":0,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":0,"hits":[]}}
14:58:04                 └- ✖ fail: saved objects tagging API DELETE /api/saved_objects_tagging/tags/{id} "before each" hook for "should delete the tag"
14:58:04                 │      Error: Timeout of 360000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/dev/shm/workspace/parallel/4/kibana/x-pack/test/saved_object_tagging/api_integration/tagging_api/apis/delete.ts)
14:58:04                 │       at listOnTimeout (internal/timers.js:554:17)
14:58:04                 │       at processTimers (internal/timers.js:497:7)

@spalger spalger added Team:Operations Team label for Operations Team v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.12.0 labels Feb 1, 2021
@spalger spalger requested a review from a team as a code owner February 1, 2021 22:20
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

@spalger
Copy link
Contributor Author

spalger commented Feb 1, 2021

esArchiver is functional, this is a tiny change and we could use it for debugging now so I'm going to merge early.

@spalger spalger merged commit 6cf1951 into elastic:master Feb 1, 2021
@spalger spalger deleted the debug/es-archiver-migration-logging branch February 1, 2021 22:51
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

gmmorris added a commit to lizozom/kibana that referenced this pull request Feb 2, 2021
…om/kibana into pr/89570

* 'sessions/save-all-sessions' of https://github.com/lizozom/kibana: (44 commits)
  [ML] Functional tests - skip DFA clone tests
  [Uptime] Fix synthetics detail step count (elastic#89940)
  Fixes the permissions to require cluster.manage in order to create an index and in order to update an index (elastic#89947)
  [Security Solution] [Detections] adds log info level for logging in cloud (elastic#89941)
  [Time to Visualize] Dashboard By Value Testing Lens (elastic#89581)
  [Uptime] Expand synthetic journey step thumbnail on hover (elastic#89179)
  TS project refs: Migrates snapshot_restore to a TS Project (elastic#89653)
  docs: APM 7.11 updates (elastic#89789)
  move skip to higher level (elastic#86952)
  Revert "Migrations v2: don't auto-create indices + FTR/esArchiver support (elastic#85778)"
  Revert "Revert "Enable v2 so migrations, disable in FTR tests (elastic#89297)""
  Revert "Enable v2 so migrations, disable in FTR tests (elastic#89297)"
  [data.search] Allow search response to follow new hits format (elastic#88115)
  [Maps] Change 'create multi-layer map' title to be use-case focused (elastic#89520)
  skip flaky suite (elastic#86952)
  [Security Solution] Remove focustrap (elastic#89905)
  [Workplace Search] Add remaining i18n support for the Content Sources tree (elastic#89910)
  [esArchiver] log when migrations complete and we're done loading data (elastic#89938)
  Add --ssl flag to make resolver generator use ssl with kbn and elasticsearch clients (elastic#89873)
  TS project refs: Migrates grokdebugger (elastic#89652)
  ...
@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Feb 3, 2021
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 89938 or prevent reminders by adding the backport:skip label.

spalger added a commit to spalger/kibana that referenced this pull request Feb 3, 2021
…elastic#89938)

Co-authored-by: spalger <spalger@users.noreply.github.com>
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Feb 4, 2021
spalger added a commit that referenced this pull request Feb 4, 2021
…g data (#89938) (#90249)

Co-authored-by: spalger <spalger@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Operations Team label for Operations Team v7.12.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants