[Security Solution] [CTI] Fixes bug that caused Threshold and Indicator Match rules to ignore custom rule filters if a saved query was used in the rule definition.

[nkhristinin]

Summary

If the rule configuration has Saved Query, it's ignored specific filter logic for Indicator match and Threshold rules. Here described how to reproduce it.

In this PR, we will ignore savedQuery for Threshold and Indicator Match, and just use the custom query from rule configuration.

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[ecezalp]

hello 👋

added some labels to ensure that the PR passes all pipeline checks. Just to quickly go over those, v7.16.0 is the version that the PR is targeting (considering this is a bug, it could also be 7.15, we can figure that out before merging). auto-backport enables the automatic cascading of the PR into releases earlier than v8.0.0 from master (I believe that's why we are also including the v8.0.0 label, but I might be wrong, @FrankHassanabad can enlighten us on that one). Team: CTI and Team: SecuritySolution is our teams, and release_note:fix is for bug fixes (a release_note label is mandatory, but it can be skip, it can also be feature, those are the ones I exclusively used so far)

as for the contents of the PR - I wasn't able to find a ticket with a clear description of the desired solution / acceptance criteria, the bug ticket had a problem statement which was helpful.

When an Indicator Match rule is configured with a saved query (i.e. a saved_id), the field matching logic (i.e. field.x matches field.y) is not taken in account when processing the rule. This leads to unexpected alerts being generated.

reading this, my understanding was that the presence of the saved query bypasses the indicator matching pieces of the rule execution for the threat_match rule, which results in more alerts. In other words, some events that are grabbed by that query that do NOT have a corresponding threat match still trigger alerts when then shouldn't (as per the matching information on the indicator match rule). It's almost like the indicator match rule just becomes a custom query rule, which is not good!

it looks like the solution implemented in this PR is to remove the saved query functionality, which is arguably one way to solve this problem. That being said, I would have assumed that we still want to keep the concept of saved queries around, but need to ensure that appropriate threat matching happens for the indicator match rule even when there is a saved query present. Would love to get some confirmation from @rylnd on this one.

[ecezalp]

I was able to check out the code and confirm that the bug is gone, and the rule creation flow with a saved query selected from in the query bar works as expected for indicator match rule as well as custom query rule.

It is also mentioned here that the saved_id params are no longer necessary.

A quick look at the QueryBar ⬇️

kibana/x-pack/plugins/security_solution/public/detections/components/rules/query_bar/index.tsx

Lines 183 to 185 in 4a54188

	filters: newSavedQuery.attributes.filters ?? [],
	query: newSavedQuery.attributes.query,
	saved_id: newSavedQuery.id,

seems to show that the filters and the query is grabbed from the saved query during the rule creation step, which seems to remove the need for the usage of saved_id during the rule execution. ✅

I was able to see that with the changes in this PR, alert.rule.saved_id is present on the alert

kibana/x-pack/plugins/security_solution/common/detection_engine/schemas/request/rule_schemas.ts

Lines 212 to 221 in 4a54188

	optional: {
	index,
	filters,
	saved_id,
	threat_filters,
	threat_indicator_path,
	threat_language: t.keyof({ kuery: null, lucene: null }),
	concurrent_searches,
	items_per_search,
	},

I was wondering if these (saved_id from optional rule params) can / should be removed altogether as a part of this change as well.

[rylnd]

I was wondering if these (saved_id from optional rule params) can / should be removed altogether as a part of this change as well.

I agree that we should remove as much of saved_id as possible. The only risk with this is existing rules: we need to verify that rules with an existing saved_id parameter can be read, updated, and deleted, all without e.g. failing validation or causing UI errors.

[nkhristinin]

I agree that ideally, we should remove save_id logic, the only question I have is how deeply we should go in this PR.

save_id has tight coupling with saved_query type of rule. Currently, if create Custom Query rule and use the saved query in configuration, then it creates a rule with type == saved_query

If we want to get rid of save_id, should we also remove saved_query logic? What about existing rules with this type, can or should we migrate them?

@rylnd @ecezalp

[ecezalp]

If we want to get rid of save_id, should we also remove saved_query logic? What about existing rules with this type, can or should we migrate them?

interesting question - I am not particularly familiar with saved_query type custom query rules, I think @spong or @madirey can provide some context regarding those. As for this immediate PR, I think that it would be safe to limit the scope to the indicator match rule as the bug was originally reported in relation to indicator matching, and possibly create a new ticket to track the rest of the work (if there is any more work, as a part of the backlog for the detections team)

[rylnd]

save_id has tight coupling with saved_query type of rule. Currently, if create Custom Query rule and use the saved query in configuration, then it creates a rule with type == saved_query

There are very few differences between the two query rule types and most logic is already shared; isQueryRule, for example, already checks for either query or saved_query. The only substantial difference between a saved_query rule and a query rule are the changes that this PR deletes: i.e. how we generate the query/filters for a given rule execution.

For 7.x I think we can remove the ability to create saved_query rules, but continue to treat both types identically. In 8.0, we can simply drop support for saved_query rules and provide a simple migration to do so. As noted in the broader issue, we'll need to update documentation to make a note of this change in behavior.

Edit: to clarify, I think we should remove the ability to create saved_query rules, but continue to support saved_query rules and treat them as if they were query rules.

Second Edit: Removing support for saved_query rules and the saved_id param is a much more involved change, both in the number of files touched and in risk (backwards compatibility with existing rules). I'm proposing that we revert those changes for now, keep this PR focused on fixing the errant bugs/behavior, and leave the saved_id/saved_query cleanup as a followup chore.

[rylnd]

Requesting changes as per the PR comments; please re-request review when saved_query rules can no longer be created!

[nkhristinin]

@elasticmachine merge upstream

[nkhristinin]

@elasticmachine merge upstream

[nkhristinin]

@marshallmain @rylnd Thank you for your latest comments.

I updated this PR, and make very simple small changes, only to address bugs with Indicator Match and Threshold rule. These rules will simply ignore saved_id and just use query.

Also, I didn't update any schema or types, I think it makes sense to do when we will completely remove save_id functionality

[nkhristinin]

@elasticmachine merge upstream

[rylnd]

Reviewed the code and verified that the regression tests failed without the change.

@nkhristinin we have the release_note:fix label here already; just be sure to follow up with the docs team to ensure an appropriate note has been generated for this issue so that affected users will be aware of the fix; before merging I would update the PR title to describe the bug(s) being fixed.

[rylnd]

I think this should also go into 7.15.1; I've added the appropriate label so the backport bot should take care of the rest, but ping me if you have questions.

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x
✅	7.15

The backport PRs will be merged automatically after passing CI.

[marshallmain]

If we're holding off on this until 7.15.1 then we need to cancel the backport to the 7.15 branch for now, right? I disabled auto-merge on the 7.15 backport PR to make sure we have a chance to confirm what the desired outcome is here.

[rylnd]

@marshallmain good catch; even with bugfixes we need to be mindful of the current release status. @nkhristinin the 7.x backport should be fine, but we'll need to hold off on the backport to 7.15 until 7.15.0 has been released.

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: ba09301
• Storybooks Preview
• Pipeline Steps (look for red circles / failed steps)
• Interpreting CI Failures

Failed CI Steps

• Jest Unit Tests
• Execute xpack-kibana-ciGroup9
• Execute xpack-kibana-ciGroup6
• Execute xpack-kibana-ciGroup6

---

Test Failures

Kibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/transform/starting·ts.transform starting continuous transform with pivot configuration start transform

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:22:40]         └-: transform
[00:22:40]           └-> "before all" hook in "transform"
[00:22:40]           └-> "before all" hook in "transform"
[00:22:40]             │ debg creating role transform_source
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_source]
[00:22:40]             │ debg creating role transform_dest
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_dest]
[00:22:40]             │ debg creating role transform_dest_readonly
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_dest_readonly]
[00:22:40]             │ debg creating role transform_ui_extras
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_ui_extras]
[00:22:40]             │ debg creating user transform_poweruser
[00:22:40]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [transform_poweruser]
[00:22:40]             │ debg created user transform_poweruser
[00:22:40]             │ debg creating user transform_viewer
[00:22:40]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [transform_viewer]
[00:22:40]             │ debg created user transform_viewer
[00:34:45]           └-: starting
[00:34:45]             └-> "before all" hook in "starting"
[00:34:45]             └-> "before all" hook in "starting"
[00:34:45]               │ info [x-pack/test/functional/es_archives/ml/ecommerce] Loading "mappings.json"
[00:34:45]               │ info [x-pack/test/functional/es_archives/ml/ecommerce] Loading "data.json.gz"
[00:34:45]               │ info [x-pack/test/functional/es_archives/ml/ecommerce] Skipped restore for existing index "ft_ecommerce"
[00:34:46]               │ debg Searching for 'index-pattern' with title 'ft_ecommerce'...
[00:34:46]               │ debg  > Found '336c4200-0f17-11ec-a48a-efe998c5b5d7'
[00:34:46]               │ debg Index pattern with title 'ft_ecommerce' already exists. Nothing to create.
[00:34:46]               │ debg Creating transform with id 'ec_starting_pivot_1630933267072_batch'...
[00:34:46]               │ debg Waiting up to 5000ms for 'ec_starting_pivot_1630933267072_batch' to exist...
[00:34:46]               │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [.transform-notifications-000002] creating index, cause [auto(bulk api)], templates [.transform-notifications-000002], shards [1]/[1]
[00:34:46]               │ info [o.e.c.r.a.AllocationService] [node-01] updating number_of_replicas to [0] for indices [.transform-notifications-000002]
[00:34:46]               │ debg Creating transform with id 'ec_starting_pivot_1630933267072_cont'...
[00:34:46]               │ debg Waiting up to 5000ms for 'ec_starting_pivot_1630933267072_cont' to exist...
[00:34:46]               │ debg Creating transform with id 'ec_starting_latest_1630933267072_batch'...
[00:34:46]               │ debg Waiting up to 5000ms for 'ec_starting_latest_1630933267072_batch' to exist...
[00:34:46]               │ debg Creating transform with id 'ec_starting_latest_1630933267072_cont'...
[00:34:46]               │ debg Waiting up to 5000ms for 'ec_starting_latest_1630933267072_cont' to exist...
[00:34:46]               │ debg applying update to kibana config: {"dateFormat:tz":"UTC"}
[00:34:46]               │ debg SecurityPage.forceLogout
[00:34:46]               │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=100
[00:34:47]               │ debg --- retry.tryForTime error: .login-form is not displayed
[00:34:47]               │ debg Redirecting to /logout to force the logout
[00:34:47]               │ debg Waiting on the login form to appear
[00:34:47]               │ debg Waiting for Login Page to appear.
[00:34:47]               │ debg Waiting up to 100000ms for login page...
[00:34:47]               │ debg browser[INFO] http://localhost:61231/logout?_t=1630935355376 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:34:47]               │
[00:34:47]               │ debg browser[INFO] http://localhost:61231/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:34:47]               │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=2500
[00:34:50]               │ debg browser[INFO] http://localhost:61231/login?msg=LOGGED_OUT 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:34:50]               │
[00:34:50]               │ debg browser[INFO] http://localhost:61231/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:34:50]               │ debg --- retry.tryForTime error: .login-form is not displayed
[00:34:51]               │ERROR browser[SEVERE] http://localhost:61231/api/licensing/info - Failed to load resource: the server responded with a status of 401 (Unauthorized)
[00:34:51]               │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=2500
[00:34:51]               │ debg TestSubjects.exists(loginForm)
[00:34:51]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="loginForm"]') with timeout=2500
[00:34:51]               │ debg Waiting for Login Form to appear.
[00:34:51]               │ debg Waiting up to 100000ms for login form...
[00:34:51]               │ debg TestSubjects.exists(loginForm)
[00:34:51]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="loginForm"]') with timeout=2500
[00:34:51]               │ debg TestSubjects.setValue(loginUsername, transform_poweruser)
[00:34:51]               │ debg TestSubjects.click(loginUsername)
[00:34:51]               │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:34:51]               │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:34:52]               │ debg TestSubjects.setValue(loginPassword, tfp001)
[00:34:52]               │ debg TestSubjects.click(loginPassword)
[00:34:52]               │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:34:52]               │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:34:52]               │ debg TestSubjects.click(loginSubmit)
[00:34:52]               │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:34:52]               │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:34:52]               │ debg Waiting for login result, expected: chrome.
[00:34:52]               │ debg Find.findByCssSelector('[data-test-subj="userMenuAvatar"]') with timeout=20000
[00:34:52]               │ proc [kibana]   log   [13:36:00.481] [info][plugins][routes][security] Logging in with provider "basic" (basic)
[00:34:56]               │ debg browser[INFO] http://localhost:61231/app/home 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:34:56]               │
[00:34:56]               │ debg browser[INFO] http://localhost:61231/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:34:56]               │ debg Finished login process currentUrl = http://localhost:61231/app/home#/
[00:34:56]               │ debg Waiting up to 20000ms for logout button visible...
[00:34:56]               │ debg TestSubjects.exists(userMenuButton)
[00:34:56]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenuButton"]') with timeout=2500
[00:34:56]               │ debg TestSubjects.exists(userMenu)
[00:34:56]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=2500
[00:34:58]               │ debg --- retry.tryForTime error: [data-test-subj="userMenu"] is not displayed
[00:34:59]               │ debg TestSubjects.click(userMenuButton)
[00:34:59]               │ debg Find.clickByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:34:59]               │ debg Find.findByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:34:59]               │ debg TestSubjects.exists(userMenu)
[00:34:59]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=120000
[00:34:59]               │ debg TestSubjects.exists(userMenu > logoutLink)
[00:34:59]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"] [data-test-subj="logoutLink"]') with timeout=2500
[00:35:17]             └-: continuous transform with pivot configuration
[00:35:17]               └-> "before all" hook for "start transform"
[00:35:17]               └-> start transform
[00:35:17]                 └-> "before each" hook: global before each for "start transform"
[00:35:17]                 │ debg === TEST STEP === should load the home page
[00:35:17]                 │ debg navigating to transform url: http://localhost:61231/app/management/data/transform
[00:35:17]                 │ debg navigate to: http://localhost:61231/app/management/data/transform
[00:35:17]                 │ debg browser[INFO] http://localhost:61231/app/management/data/transform?_t=1630935385244 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:35:17]                 │
[00:35:17]                 │ debg browser[INFO] http://localhost:61231/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:35:17]                 │ debg ... sleep(700) start
[00:35:18]                 │ debg ... sleep(700) end
[00:35:18]                 │ debg returned from get, calling refresh
[00:35:20]                 │ debg browser[INFO] http://localhost:61231/app/management/data/transform?_t=1630935385244 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:35:20]                 │
[00:35:20]                 │ debg browser[INFO] http://localhost:61231/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:35:20]                 │ debg currentUrl = http://localhost:61231/app/management/data/transform
[00:35:20]                 │          appUrl = http://localhost:61231/app/management/data/transform
[00:35:20]                 │ debg TestSubjects.find(kibanaChrome)
[00:35:20]                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:35:21]                 │ debg ... sleep(501) start
[00:35:21]                 │ debg ... sleep(501) end
[00:35:21]                 │ debg in navigateTo url = http://localhost:61231/app/management/data/transform
[00:35:21]                 │ debg TestSubjects.exists(transformPageTransformList)
[00:35:21]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformPageTransformList"]') with timeout=120000
[00:35:21]                 │ debg === TEST STEP === should display the transforms table
[00:35:21]                 │ debg TestSubjects.exists(~transformListTable)
[00:35:21]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformListTable"]') with timeout=120000
[00:35:21]                 │ debg === TEST STEP === should display the original transform in the transform list
[00:35:21]                 │ debg TestSubjects.exists(~transformListTable)
[00:35:21]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformListTable"]') with timeout=60000
[00:35:21]                 │ debg TestSubjects.exists(transformListTable loaded)
[00:35:21]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformListTable loaded"]') with timeout=30000
[00:35:21]                 │ debg TestSubjects.find(transformListTableContainer)
[00:35:21]                 │ debg Find.findByCssSelector('[data-test-subj="transformListTableContainer"]') with timeout=10000
[00:35:22]                 │ debg TestSubjects.find(~transformListTable)
[00:35:22]                 │ debg Find.findByCssSelector('[data-test-subj~="transformListTable"]') with timeout=10000
[00:35:22]                 │ debg === TEST STEP === should start the transform
[00:35:22]                 │ debg TestSubjects.exists(~transformRefreshTransformListButton)
[00:35:22]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:22]                 │ debg TestSubjects.exists(transformRefreshTransformListButton loaded)
[00:35:22]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformRefreshTransformListButton loaded"]') with timeout=30000
[00:35:22]                 │ debg TestSubjects.click(~transformRefreshTransformListButton)
[00:35:22]                 │ debg Find.clickByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:22]                 │ debg Find.findByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:23]                 │ debg TestSubjects.exists(~transformRefreshTransformListButton)
[00:35:23]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:23]                 │ debg TestSubjects.exists(transformRefreshTransformListButton loaded)
[00:35:23]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformRefreshTransformListButton loaded"]') with timeout=30000
[00:35:23]                 │ debg TestSubjects.exists(~transformListTable)
[00:35:23]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformListTable"]') with timeout=60000
[00:35:23]                 │ debg TestSubjects.exists(transformListTable loaded)
[00:35:23]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformListTable loaded"]') with timeout=30000
[00:35:23]                 │ debg Find.existsByCssSelector('.euiContextMenuPanel') with timeout=1000
[00:35:24]                 │ debg Find.existsByCssSelector('.euiContextMenuPanel') with timeout=1000
[00:35:25]                 │ debg TestSubjects.click(~transformListTable > ~row-ec_starting_pivot_1630933267072_cont > euiCollapsedItemActionsButton)
[00:35:25]                 │ debg Find.clickByCssSelector('[data-test-subj~="transformListTable"] [data-test-subj~="row-ec_starting_pivot_1630933267072_cont"] [data-test-subj="euiCollapsedItemActionsButton"]') with timeout=10000
[00:35:25]                 │ debg Find.findByCssSelector('[data-test-subj~="transformListTable"] [data-test-subj~="row-ec_starting_pivot_1630933267072_cont"] [data-test-subj="euiCollapsedItemActionsButton"]') with timeout=10000
[00:35:25]                 │ debg Find.existsByCssSelector('.euiContextMenuPanel') with timeout=1000
[00:35:25]                 │ debg TestSubjects.exists(transformActionStart)
[00:35:25]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformActionStart"]') with timeout=1000
[00:35:26]                 │ debg TestSubjects.isEnabled(transformActionStart)
[00:35:26]                 │ debg TestSubjects.find(transformActionStart)
[00:35:26]                 │ debg Find.findByCssSelector('[data-test-subj="transformActionStart"]') with timeout=10000
[00:35:26]                 │ debg Find.existsByCssSelector('.euiContextMenuPanel') with timeout=1000
[00:35:26]                 │ debg --- retry.tryForTime error: Actions popover should not exist
[00:35:26]                 │ debg Find.existsByCssSelector('.euiContextMenuPanel') with timeout=1000
[00:35:26]                 │ debg --- retry.tryForTime failed again with the same message...
[00:35:27]                 │ debg Find.existsByCssSelector('.euiContextMenuPanel') with timeout=1000
[00:35:28]                 │ debg Find.existsByCssSelector('.euiContextMenuPanel') with timeout=1000
[00:35:29]                 │ debg Find.existsByCssSelector('.euiContextMenuPanel') with timeout=1000
[00:35:30]                 │ debg TestSubjects.click(~transformListTable > ~row-ec_starting_pivot_1630933267072_cont > euiCollapsedItemActionsButton)
[00:35:30]                 │ debg Find.clickByCssSelector('[data-test-subj~="transformListTable"] [data-test-subj~="row-ec_starting_pivot_1630933267072_cont"] [data-test-subj="euiCollapsedItemActionsButton"]') with timeout=10000
[00:35:30]                 │ debg Find.findByCssSelector('[data-test-subj~="transformListTable"] [data-test-subj~="row-ec_starting_pivot_1630933267072_cont"] [data-test-subj="euiCollapsedItemActionsButton"]') with timeout=10000
[00:35:30]                 │ debg Find.existsByCssSelector('.euiContextMenuPanel') with timeout=1000
[00:35:30]                 │ debg TestSubjects.click(transformActionStart)
[00:35:30]                 │ debg Find.clickByCssSelector('[data-test-subj="transformActionStart"]') with timeout=10000
[00:35:30]                 │ debg Find.findByCssSelector('[data-test-subj="transformActionStart"]') with timeout=10000
[00:35:30]                 │ debg TestSubjects.missingOrFail(transformActionStart)
[00:35:30]                 │ debg Find.waitForDeletedByCssSelector('[data-test-subj="transformActionStart"]') with timeout=2500
[00:35:31]                 │ debg TestSubjects.exists(transformStartModal)
[00:35:31]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformStartModal"]') with timeout=60000
[00:35:31]                 │ debg TestSubjects.click(transformStartModal > confirmModalConfirmButton)
[00:35:31]                 │ debg Find.clickByCssSelector('[data-test-subj="transformStartModal"] [data-test-subj="confirmModalConfirmButton"]') with timeout=10000
[00:35:31]                 │ debg Find.findByCssSelector('[data-test-subj="transformStartModal"] [data-test-subj="confirmModalConfirmButton"]') with timeout=10000
[00:35:31]                 │ debg TestSubjects.missingOrFail(transformStartModal)
[00:35:31]                 │ debg Find.waitForDeletedByCssSelector('[data-test-subj="transformStartModal"]') with timeout=60000
[00:35:31]                 │ info [o.e.x.t.t.TransformTask] [node-01] [ec_starting_pivot_1630933267072_cont] updating state for transform to [{"task_state":"started","indexer_state":"stopped","checkpoint":0,"progress":{"docs_indexed":0,"docs_processed":0},"should_stop_at_checkpoint":false}].
[00:35:31]                 │ info [o.e.x.t.t.TransformPersistentTasksExecutor] [node-01] [ec_starting_pivot_1630933267072_cont] successfully completed and scheduled task in node operation
[00:35:32]                 │ debg TestSubjects.exists(~transformListTable)
[00:35:32]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformListTable"]') with timeout=60000
[00:35:32]                 │ debg TestSubjects.exists(transformListTable loaded)
[00:35:32]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformListTable loaded"]') with timeout=30000
[00:35:32]                 │ debg TestSubjects.find(transformListTableContainer)
[00:35:32]                 │ debg Find.findByCssSelector('[data-test-subj="transformListTableContainer"]') with timeout=10000
[00:35:32]                 │ debg TestSubjects.find(~transformListTable)
[00:35:32]                 │ debg Find.findByCssSelector('[data-test-subj~="transformListTable"]') with timeout=10000
[00:35:32]                 │ debg === TEST STEP === should display the started transform
[00:35:32]                 │ debg TestSubjects.exists(~transformRefreshTransformListButton)
[00:35:32]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:32]                 │ debg TestSubjects.exists(transformRefreshTransformListButton loaded)
[00:35:32]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformRefreshTransformListButton loaded"]') with timeout=30000
[00:35:32]                 │ debg TestSubjects.click(~transformRefreshTransformListButton)
[00:35:32]                 │ debg Find.clickByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:32]                 │ debg Find.findByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:32]                 │ debg TestSubjects.exists(~transformRefreshTransformListButton)
[00:35:32]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:32]                 │ debg TestSubjects.exists(transformRefreshTransformListButton loaded)
[00:35:32]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformRefreshTransformListButton loaded"]') with timeout=30000
[00:35:33]                 │ debg TestSubjects.exists(~transformListTable)
[00:35:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformListTable"]') with timeout=60000
[00:35:33]                 │ debg TestSubjects.exists(transformListTable loaded)
[00:35:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformListTable loaded"]') with timeout=30000
[00:35:33]                 │ debg TestSubjects.find(~transformListTable)
[00:35:33]                 │ debg Find.findByCssSelector('[data-test-subj~="transformListTable"]') with timeout=10000
[00:35:33]                 │ debg TestSubjects.exists(~transformRefreshTransformListButton)
[00:35:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:33]                 │ debg TestSubjects.exists(transformRefreshTransformListButton loaded)
[00:35:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformRefreshTransformListButton loaded"]') with timeout=30000
[00:35:33]                 │ debg TestSubjects.click(~transformRefreshTransformListButton)
[00:35:33]                 │ debg Find.clickByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:33]                 │ debg Find.findByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:33]                 │ debg TestSubjects.exists(~transformRefreshTransformListButton)
[00:35:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:33]                 │ debg TestSubjects.exists(transformRefreshTransformListButton loaded)
[00:35:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformRefreshTransformListButton loaded"]') with timeout=30000
[00:35:33]                 │ info #
[00:35:33]                 │      # There is insufficient memory for the Java Runtime Environment to continue.
[00:35:33]                 │      # Native memory allocation (malloc) failed to allocate 8589934608 bytes for Chunk::new
[00:35:33]                 │      # An error report file with more information is saved as:
[00:35:33]                 │      # logs/hs_err_pid83111.log
[00:35:33]                 │ info #
[00:35:33]                 │      # Compiler replay data is saved as:
[00:35:33]                 │      # /dev/shm/workspace/parallel/23/kibana/.es/job-xpack-kibana-ciGroup9-worker-23-es-test-cluster/replay_pid83111.log
[00:35:33]                 │ debg TestSubjects.exists(~transformListTable)
[00:35:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformListTable"]') with timeout=60000
[00:35:33]                 │ debg TestSubjects.exists(transformListTable loaded)
[00:35:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformListTable loaded"]') with timeout=30000
[00:35:33]                 │ debg TestSubjects.find(~transformListTable)
[00:35:33]                 │ debg Find.findByCssSelector('[data-test-subj~="transformListTable"]') with timeout=10000
[00:35:33]                 │ debg TestSubjects.exists(~transformRefreshTransformListButton)
[00:35:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:33]                 │ debg TestSubjects.exists(transformRefreshTransformListButton loaded)
[00:35:33]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformRefreshTransformListButton loaded"]') with timeout=30000
[00:35:33]                 │ debg TestSubjects.click(~transformRefreshTransformListButton)
[00:35:33]                 │ debg Find.clickByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:33]                 │ debg Find.findByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:34]                 │ debg TestSubjects.exists(~transformRefreshTransformListButton)
[00:35:34]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="transformRefreshTransformListButton"]') with timeout=10000
[00:35:34]                 │ debg TestSubjects.exists(transformRefreshTransformListButton loaded)
[00:35:34]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="transformRefreshTransformListButton loaded"]') with timeout=30000
[00:35:36]                 │ debg --- retry.tryForTime error: [data-test-subj="transformRefreshTransformListButton loaded"] is not displayed
[00:35:39]                 │ debg --- retry.tryForTime failed again with the same message...
[00:35:42]                 │ debg --- retry.tryForTime failed again with the same message...
[00:35:45]                 │ debg --- retry.tryForTime failed again with the same message...
[00:35:46]                 │ proc [kibana]   log   [13:36:54.526] [error][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. socket hang up
[00:35:47]                 │ proc [kibana]   log   [13:36:54.869] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:62231 error
[00:35:47]                 │ proc [kibana]   log   [13:36:54.873] [info][plugins][searchprofiler] You cannot use searchprofiler because license information is not available at this time.
[00:35:47]                 │ proc [kibana]   log   [13:36:54.874] [info][painlessLab][plugins] You cannot use painlessLab because license information is not available at this time.
[00:35:47]                 │ proc [kibana]   log   [13:36:54.878] [info][plugins][transform] You cannot use transform because license information is not available at this time.
[00:35:47]                 │ proc [kibana]   log   [13:36:54.878] [info][plugins][snapshotRestore] You cannot use snapshot_restore because license information is not available at this time.
[00:35:47]                 │ proc [kibana]   log   [13:36:54.878] [info][plugins][remoteClusters] You cannot use Remote Clusters because license information is not available at this time.
[00:35:47]                 │ proc [kibana]   log   [13:36:54.878] [info][indexLifecycleManagement][plugins] You cannot use index_lifecycle_management because license information is not available at this time.
[00:35:47]                 │ proc [kibana]   log   [13:36:54.879] [info][plugins][rollup] You cannot use rollup because license information is not available at this time.
[00:35:47]                 │ proc [kibana]   log   [13:36:54.882] [info][kibana-monitoring][monitoring][monitoring][plugins] Monitoring status upload endpoint is not enabled in Elasticsearch:Monitoring stats collection is stopped
[00:35:47]                 │ proc [kibana]   log   [13:36:55.329] [warning][kibana-monitoring][monitoring][monitoring][plugins] ConnectionError: connect ECONNREFUSED 127.0.0.1:62231
[00:35:47]                 │ proc [kibana]     at ClientRequest.onError (/dev/shm/workspace/kibana-build-23/node_modules/@elastic/elasticsearch/lib/Connection.js:116:16)
[00:35:47]                 │ proc [kibana]     at ClientRequest.emit (events.js:400:28)
[00:35:47]                 │ proc [kibana]     at Socket.socketErrorListener (_http_client.js:475:9)
[00:35:47]                 │ proc [kibana]     at Socket.emit (events.js:400:28)
[00:35:47]                 │ proc [kibana]     at emitErrorNT (internal/streams/destroy.js:106:8)
[00:35:47]                 │ proc [kibana]     at emitErrorCloseNT (internal/streams/destroy.js:74:3)
[00:35:47]                 │ proc [kibana]     at processTicksAndRejections (internal/process/task_queues.js:82:21)
[00:35:47]                 │ proc [kibana]   log   [13:36:55.330] [warning][kibana-monitoring][monitoring][monitoring][plugins] Unable to bulk upload the stats payload to the local cluster
[00:35:47]                 │ proc [kibana]   log   [13:36:55.330] [error][index][plugins][security][session] Failed to retrieve session value: connect ECONNREFUSED 127.0.0.1:62231
[00:35:47]                 │ proc [kibana]   log   [13:36:55.331] [error][server][Kibana][http] ConnectionError: connect ECONNREFUSED 127.0.0.1:62231
[00:35:47]                 │ proc [kibana]     at ClientRequest.onError (/dev/shm/workspace/kibana-build-23/node_modules/@elastic/elasticsearch/lib/Connection.js:116:16)
[00:35:47]                 │ proc [kibana]     at ClientRequest.emit (events.js:400:28)
[00:35:47]                 │ proc [kibana]     at Socket.socketErrorListener (_http_client.js:475:9)
[00:35:47]                 │ proc [kibana]     at Socket.emit (events.js:400:28)
[00:35:47]                 │ proc [kibana]     at emitErrorNT (internal/streams/destroy.js:106:8)
[00:35:47]                 │ proc [kibana]     at emitErrorCloseNT (internal/streams/destroy.js:74:3)
[00:35:47]                 │ proc [kibana]     at processTicksAndRejections (internal/process/task_queues.js:82:21) {
[00:35:47]                 │ proc [kibana]   meta: {
[00:35:47]                 │ proc [kibana]     body: null,
[00:35:47]                 │ proc [kibana]     statusCode: null,
[00:35:47]                 │ proc [kibana]     headers: null,
[00:35:47]                 │ proc [kibana]     meta: {
[00:35:47]                 │ proc [kibana]       context: null,
[00:35:47]                 │ proc [kibana]       request: [Object],
[00:35:47]                 │ proc [kibana]       name: 'elasticsearch-js',
[00:35:47]                 │ proc [kibana]       connection: [Object],
[00:35:47]                 │ proc [kibana]       attempts: 3,
[00:35:47]                 │ proc [kibana]       aborted: false
[00:35:47]                 │ proc [kibana]     }
[00:35:47]                 │ proc [kibana]   }
[00:35:47]                 │ proc [kibana] }
[00:35:47]                 │ proc [kibana]   log   [13:36:55.378] [warning][plugins][securitySolution] Unable to verify endpoint policies in line with license change: failed to fetch package policies: connect ECONNREFUSED 127.0.0.1:62231
[00:35:48]                 │ proc [kibana]   log   [13:36:56.169] [error][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. connect ECONNREFUSED 127.0.0.1:62231
[00:35:48]                 │ proc [kibana]   log   [13:36:56.495] [info][status] Kibana is now critical (was available)
[00:35:48]                 │ proc [kibana]   log   [13:36:56.528] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:62231 error
[00:35:48]                 │ proc [kibana]  error  [13:36:41.854]  Error: Internal Server Error
[00:35:48]                 │ proc [kibana]     at HapiResponseAdapter.toInternalError (/dev/shm/workspace/kibana-build-23/src/core/server/http/router/response_adapter.js:59:19)
[00:35:48]                 │ proc [kibana]     at Object.interceptAuth [as authenticate] (/dev/shm/workspace/kibana-build-23/src/core/server/http/lifecycle/auth.js:130:34)
[00:35:48]                 │ proc [kibana]     at processTicksAndRejections (internal/process/task_queues.js:95:5)
[00:35:48]                 │ proc [kibana]     at exports.Manager.execute (/dev/shm/workspace/kibana-build-23/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
[00:35:48]                 │ proc [kibana]     at module.exports.internals.Auth._authenticate (/dev/shm/workspace/kibana-build-23/node_modules/@hapi/hapi/lib/auth.js:258:30)
[00:35:48]                 │ proc [kibana]     at Request._lifecycle (/dev/shm/workspace/kibana-build-23/node_modules/@hapi/hapi/lib/request.js:370:32)
[00:35:48]                 │ proc [kibana]     at Request._execute (/dev/shm/workspace/kibana-build-23/node_modules/@hapi/hapi/lib/request.js:279:9)
[00:35:48]                 │ proc [kibana]   log   [13:36:56.550] [error][authentication][plugins][security] License is not available, authentication is not possible.
[00:35:48]                 │ proc [kibana]   log   [13:36:56.556] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:62231 error
[00:35:48]                 │ proc [kibana]   log   [13:36:56.567] [error][authentication][plugins][security] License is not available, authentication is not possible.
[00:35:48]                 │ proc [kibana]   log   [13:36:56.573] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:62231 error
[00:35:48]                 │ proc [kibana]   log   [13:36:56.669] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:62231 error
[00:35:48]                 │ proc [kibana]   log   [13:36:56.670] [warning][monitoring][monitoring][plugins] X-Pack Monitoring Cluster Alerts will not be available: connect ECONNREFUSED 127.0.0.1:62231
[00:35:48]                 │ERROR browser[SEVERE] http://localhost:61231/api/transform/transforms/_nodes - Failed to load resource: the server responded with a status of 500 (Internal Server Error)
[00:35:48]                 │ERROR browser[SEVERE] http://localhost:61231/api/transform/transforms - Failed to load resource: the server responded with a status of 503 (Service Unavailable)
[00:35:48]                 │ERROR browser[SEVERE] http://localhost:61231/api/transform/transforms/_stats - Failed to load resource: the server responded with a status of 503 (Service Unavailable)
[00:35:48]                 │ debg --- retry.tryForTime failed again with the same message...
[00:35:52]                 │ debg --- retry.tryForTime failed again with the same message...
[00:35:55]                 │ debg --- retry.tryForTime failed again with the same message...
[00:35:58]                 │ debg --- retry.tryForTime failed again with the same message...
[00:36:01]                 │ debg --- retry.tryForTime failed again with the same message...
[00:36:04]                 │ debg --- retry.tryForTime failed again with the same message...
[00:36:04]                 │ debg --- retry.tryForTime error: expected testSubject(transformRefreshTransformListButton loaded) to exist
[00:36:05]                 │ info Taking screenshot "/dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/screenshots/failure/transform starting continuous transform with pivot configuration start transform.png"
[00:36:05]                 │ info Current URL is: http://localhost:61231/app/management/data/transform
[00:36:05]                 │ info Saving page source to: /dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/failure_debug/html/transform starting continuous transform with pivot configuration start transform.html
[00:36:05]                 └- ✖ fail: transform starting continuous transform with pivot configuration start transform
[00:36:05]                 │      retry.tryForTime timeout: Error: expected testSubject(transformRefreshTransformListButton loaded) to exist
[00:36:05]                 │     at TestSubjects.existOrFail (/dev/shm/workspace/parallel/23/kibana/test/functional/services/common/test_subjects.ts:45:13)
[00:36:05]                 │     at TransformTable.waitForRefreshButtonLoaded (/dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/services/transform/transform_table.ts:113:7)
[00:36:05]                 │     at TransformTable.refreshTransformList (/dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/services/transform/transform_table.ts:121:7)
[00:36:05]                 │     at /dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/services/transform/transform_table.ts:188:9
[00:36:05]                 │     at runAttempt (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry_for_success.ts:27:15)
[00:36:05]                 │     at retryForSuccess (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry_for_success.ts:66:21)
[00:36:05]                 │     at RetryService.tryForTime (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry.ts:22:12)
[00:36:05]                 │     at TransformTable.assertTransformRowStatusNotEql (/dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/services/transform/transform_table.ts:187:7)
[00:36:05]                 │     at Context.<anonymous> (/dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/apps/transform/starting.ts:98:11)
[00:36:05]                 │     at Object.apply (/dev/shm/workspace/parallel/23/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
[00:36:05]                 │   Error: retry.tryForTime timeout: Error: expected testSubject(transformRefreshTransformListButton loaded) to exist
[00:36:05]                 │       at TestSubjects.existOrFail (/dev/shm/workspace/parallel/23/kibana/test/functional/services/common/test_subjects.ts:45:13)
[00:36:05]                 │       at TransformTable.waitForRefreshButtonLoaded (test/functional/services/transform/transform_table.ts:113:7)
[00:36:05]                 │       at TransformTable.refreshTransformList (test/functional/services/transform/transform_table.ts:121:7)
[00:36:05]                 │       at /dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/services/transform/transform_table.ts:188:9
[00:36:05]                 │       at runAttempt (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry_for_success.ts:27:15)
[00:36:05]                 │       at retryForSuccess (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry_for_success.ts:66:21)
[00:36:05]                 │       at RetryService.tryForTime (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry.ts:22:12)
[00:36:05]                 │       at TransformTable.assertTransformRowStatusNotEql (test/functional/services/transform/transform_table.ts:187:7)
[00:36:05]                 │       at Context.<anonymous> (test/functional/apps/transform/starting.ts:98:11)
[00:36:05]                 │       at Object.apply (/dev/shm/workspace/parallel/23/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
[00:36:05]                 │       at onFailure (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry_for_success.ts:17:9)
[00:36:05]                 │       at retryForSuccess (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry_for_success.ts:57:13)
[00:36:05]                 │       at RetryService.tryForTime (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry.ts:22:12)
[00:36:05]                 │       at TransformTable.assertTransformRowStatusNotEql (test/functional/services/transform/transform_table.ts:187:7)
[00:36:05]                 │       at Context.<anonymous> (test/functional/apps/transform/starting.ts:98:11)
[00:36:05]                 │       at Object.apply (/dev/shm/workspace/parallel/23/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
[00:36:05]                 │ 
[00:36:05]                 │ 

Stack Trace

Error: retry.tryForTime timeout: Error: expected testSubject(transformRefreshTransformListButton loaded) to exist
    at TestSubjects.existOrFail (/dev/shm/workspace/parallel/23/kibana/test/functional/services/common/test_subjects.ts:45:13)
    at TransformTable.waitForRefreshButtonLoaded (test/functional/services/transform/transform_table.ts:113:7)
    at TransformTable.refreshTransformList (test/functional/services/transform/transform_table.ts:121:7)
    at /dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/services/transform/transform_table.ts:188:9
    at runAttempt (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry_for_success.ts:27:15)
    at retryForSuccess (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry_for_success.ts:66:21)
    at RetryService.tryForTime (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry.ts:22:12)
    at TransformTable.assertTransformRowStatusNotEql (test/functional/services/transform/transform_table.ts:187:7)
    at Context.<anonymous> (test/functional/apps/transform/starting.ts:98:11)
    at Object.apply (/dev/shm/workspace/parallel/23/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
    at onFailure (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry_for_success.ts:17:9)
    at retryForSuccess (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry_for_success.ts:57:13)
    at RetryService.tryForTime (/dev/shm/workspace/parallel/23/kibana/test/common/services/retry/retry.ts:22:12)
    at TransformTable.assertTransformRowStatusNotEql (test/functional/services/transform/transform_table.ts:187:7)
    at Context.<anonymous> (test/functional/apps/transform/starting.ts:98:11)
    at Object.apply (/dev/shm/workspace/parallel/23/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

---

Kibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/transform/starting·ts.transform starting "after all" hook in "starting"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 1 times on tracked branches: https://github.com/elastic/kibana/issues/108146

[00:00:00]       │
[00:22:40]         └-: transform
[00:22:40]           └-> "before all" hook in "transform"
[00:22:40]           └-> "before all" hook in "transform"
[00:22:40]             │ debg creating role transform_source
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_source]
[00:22:40]             │ debg creating role transform_dest
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_dest]
[00:22:40]             │ debg creating role transform_dest_readonly
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_dest_readonly]
[00:22:40]             │ debg creating role transform_ui_extras
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_ui_extras]
[00:22:40]             │ debg creating user transform_poweruser
[00:22:40]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [transform_poweruser]
[00:22:40]             │ debg created user transform_poweruser
[00:22:40]             │ debg creating user transform_viewer
[00:22:40]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [transform_viewer]
[00:22:40]             │ debg created user transform_viewer
[00:34:45]           └-: starting
[00:34:45]             └-> "before all" hook in "starting"
[00:34:45]             └-> "before all" hook in "starting"
[00:34:45]               │ info [x-pack/test/functional/es_archives/ml/ecommerce] Loading "mappings.json"
[00:34:45]               │ info [x-pack/test/functional/es_archives/ml/ecommerce] Loading "data.json.gz"
[00:34:45]               │ info [x-pack/test/functional/es_archives/ml/ecommerce] Skipped restore for existing index "ft_ecommerce"
[00:34:46]               │ debg Searching for 'index-pattern' with title 'ft_ecommerce'...
[00:34:46]               │ debg  > Found '336c4200-0f17-11ec-a48a-efe998c5b5d7'
[00:34:46]               │ debg Index pattern with title 'ft_ecommerce' already exists. Nothing to create.
[00:34:46]               │ debg Creating transform with id 'ec_starting_pivot_1630933267072_batch'...
[00:34:46]               │ debg Waiting up to 5000ms for 'ec_starting_pivot_1630933267072_batch' to exist...
[00:34:46]               │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [.transform-notifications-000002] creating index, cause [auto(bulk api)], templates [.transform-notifications-000002], shards [1]/[1]
[00:34:46]               │ info [o.e.c.r.a.AllocationService] [node-01] updating number_of_replicas to [0] for indices [.transform-notifications-000002]
[00:34:46]               │ debg Creating transform with id 'ec_starting_pivot_1630933267072_cont'...
[00:34:46]               │ debg Waiting up to 5000ms for 'ec_starting_pivot_1630933267072_cont' to exist...
[00:34:46]               │ debg Creating transform with id 'ec_starting_latest_1630933267072_batch'...
[00:34:46]               │ debg Waiting up to 5000ms for 'ec_starting_latest_1630933267072_batch' to exist...
[00:34:46]               │ debg Creating transform with id 'ec_starting_latest_1630933267072_cont'...
[00:34:46]               │ debg Waiting up to 5000ms for 'ec_starting_latest_1630933267072_cont' to exist...
[00:34:46]               │ debg applying update to kibana config: {"dateFormat:tz":"UTC"}
[00:34:46]               │ debg SecurityPage.forceLogout
[00:34:46]               │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=100
[00:34:47]               │ debg --- retry.tryForTime error: .login-form is not displayed
[00:34:47]               │ debg Redirecting to /logout to force the logout
[00:34:47]               │ debg Waiting on the login form to appear
[00:34:47]               │ debg Waiting for Login Page to appear.
[00:34:47]               │ debg Waiting up to 100000ms for login page...
[00:34:47]               │ debg browser[INFO] http://localhost:61231/logout?_t=1630935355376 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:34:47]               │
[00:34:47]               │ debg browser[INFO] http://localhost:61231/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:34:47]               │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=2500
[00:34:50]               │ debg browser[INFO] http://localhost:61231/login?msg=LOGGED_OUT 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:34:50]               │
[00:34:50]               │ debg browser[INFO] http://localhost:61231/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:34:50]               │ debg --- retry.tryForTime error: .login-form is not displayed
[00:34:51]               │ERROR browser[SEVERE] http://localhost:61231/api/licensing/info - Failed to load resource: the server responded with a status of 401 (Unauthorized)
[00:34:51]               │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=2500
[00:34:51]               │ debg TestSubjects.exists(loginForm)
[00:34:51]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="loginForm"]') with timeout=2500
[00:34:51]               │ debg Waiting for Login Form to appear.
[00:34:51]               │ debg Waiting up to 100000ms for login form...
[00:34:51]               │ debg TestSubjects.exists(loginForm)
[00:34:51]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="loginForm"]') with timeout=2500
[00:34:51]               │ debg TestSubjects.setValue(loginUsername, transform_poweruser)
[00:34:51]               │ debg TestSubjects.click(loginUsername)
[00:34:51]               │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:34:51]               │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:34:52]               │ debg TestSubjects.setValue(loginPassword, tfp001)
[00:34:52]               │ debg TestSubjects.click(loginPassword)
[00:34:52]               │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:34:52]               │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:34:52]               │ debg TestSubjects.click(loginSubmit)
[00:34:52]               │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:34:52]               │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:34:52]               │ debg Waiting for login result, expected: chrome.
[00:34:52]               │ debg Find.findByCssSelector('[data-test-subj="userMenuAvatar"]') with timeout=20000
[00:34:52]               │ proc [kibana]   log   [13:36:00.481] [info][plugins][routes][security] Logging in with provider "basic" (basic)
[00:34:56]               │ debg browser[INFO] http://localhost:61231/app/home 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:34:56]               │
[00:34:56]               │ debg browser[INFO] http://localhost:61231/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:34:56]               │ debg Finished login process currentUrl = http://localhost:61231/app/home#/
[00:34:56]               │ debg Waiting up to 20000ms for logout button visible...
[00:34:56]               │ debg TestSubjects.exists(userMenuButton)
[00:34:56]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenuButton"]') with timeout=2500
[00:34:56]               │ debg TestSubjects.exists(userMenu)
[00:34:56]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=2500
[00:34:58]               │ debg --- retry.tryForTime error: [data-test-subj="userMenu"] is not displayed
[00:34:59]               │ debg TestSubjects.click(userMenuButton)
[00:34:59]               │ debg Find.clickByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:34:59]               │ debg Find.findByCssSelector('[data-test-subj="userMenuButton"]') with timeout=10000
[00:34:59]               │ debg TestSubjects.exists(userMenu)
[00:34:59]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"]') with timeout=120000
[00:34:59]               │ debg TestSubjects.exists(userMenu > logoutLink)
[00:34:59]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="userMenu"] [data-test-subj="logoutLink"]') with timeout=2500
[00:36:05]             └-> "after all" hook in "starting"
[00:36:05]               │ debg Deleting index pattern with title 'user-ec_2_1630933267072'...
[00:36:05]               │ debg Searching for 'index-pattern' with title 'user-ec_2_1630933267072'...
[00:36:05]               │ proc [kibana]   log   [13:37:13.312] [error][authentication][plugins][security] License is not available, authentication is not possible.
[00:36:05]               │ proc [kibana]   log   [13:37:13.318] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:62231 error
[00:36:05]               │ info Taking screenshot "/dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/screenshots/failure/transform starting _after all_ hook in _starting_.png"
[00:36:05]               │ info Current URL is: http://localhost:61231/app/management/data/transform
[00:36:05]               │ info Saving page source to: /dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/failure_debug/html/transform starting _after all_ hook in _starting_.html
[00:36:05]               └- ✖ fail: transform starting "after all" hook in "starting"
[00:36:05]               │      Error: expected 200 "OK", got 503 "Service Unavailable"
[00:36:05]               │       at Test._assertStatus (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/lib/test.js:268:12)
[00:36:05]               │       at Test._assertFunction (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/lib/test.js:283:11)
[00:36:05]               │       at Test.assert (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/lib/test.js:173:18)
[00:36:05]               │       at assert (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/lib/test.js:131:12)
[00:36:05]               │       at /dev/shm/workspace/parallel/23/kibana/node_modules/supertest/lib/test.js:128:5
[00:36:05]               │       at Test.Request.callback (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:718:3)
[00:36:05]               │       at /dev/shm/workspace/parallel/23/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:906:18
[00:36:05]               │       at IncomingMessage.<anonymous> (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/node_modules/superagent/lib/node/parsers/json.js:19:7)
[00:36:05]               │       at endReadableNT (internal/streams/readable.js:1317:12)
[00:36:05]               │       at processTicksAndRejections (internal/process/task_queues.js:82:21)
[00:36:05]               │ 
[00:36:05]               │ 

Stack Trace

Error: expected 200 "OK", got 503 "Service Unavailable"
    at Test._assertStatus (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/lib/test.js:268:12)
    at Test._assertFunction (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/lib/test.js:283:11)
    at Test.assert (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/lib/test.js:173:18)
    at assert (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/lib/test.js:131:12)
    at /dev/shm/workspace/parallel/23/kibana/node_modules/supertest/lib/test.js:128:5
    at Test.Request.callback (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:718:3)
    at /dev/shm/workspace/parallel/23/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:906:18
    at IncomingMessage.<anonymous> (/dev/shm/workspace/parallel/23/kibana/node_modules/supertest/node_modules/superagent/lib/node/parsers/json.js:19:7)
    at endReadableNT (internal/streams/readable.js:1317:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)

---

Kibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/transform.transform "after all" hook in "transform"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 9 times on tracked branches: https://github.com/elastic/kibana/issues/93890

[00:00:00]       │
[00:22:40]         └-: transform
[00:22:40]           └-> "before all" hook in "transform"
[00:22:40]           └-> "before all" hook in "transform"
[00:22:40]             │ debg creating role transform_source
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_source]
[00:22:40]             │ debg creating role transform_dest
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_dest]
[00:22:40]             │ debg creating role transform_dest_readonly
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_dest_readonly]
[00:22:40]             │ debg creating role transform_ui_extras
[00:22:40]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [node-01] added role [transform_ui_extras]
[00:22:40]             │ debg creating user transform_poweruser
[00:22:40]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [transform_poweruser]
[00:22:40]             │ debg created user transform_poweruser
[00:22:40]             │ debg creating user transform_viewer
[00:22:40]             │ info [o.e.x.s.a.u.TransportPutUserAction] [node-01] added user [transform_viewer]
[00:22:40]             │ debg created user transform_viewer
[00:36:05]             └-> "after all" hook in "transform"
[00:36:05]               │ debg deleting user transform_poweruser
[00:36:05]               │ proc [kibana]   log   [13:37:13.537] [error][authentication][plugins][security] License is not available, authentication is not possible.
[00:36:05]               │ proc [kibana]   log   [13:37:13.543] [warning][licensing][plugins] License information could not be obtained from Elasticsearch due to ConnectionError: connect ECONNREFUSED 127.0.0.1:62231 error
[00:36:05]               │ info Taking screenshot "/dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/screenshots/failure/transform _after all_ hook in _transform_.png"
[00:36:05]               │ info Current URL is: http://localhost:61231/app/management/data/transform
[00:36:06]               │ info Saving page source to: /dev/shm/workspace/parallel/23/kibana/x-pack/test/functional/failure_debug/html/transform _after all_ hook in _transform_.html
[00:36:06]               └- ✖ fail: transform "after all" hook in "transform"
[00:36:06]               │      Error: Request failed with status code 503
[00:36:06]               │       at createError (/dev/shm/workspace/parallel/23/kibana/node_modules/axios/lib/core/createError.js:16:15)
[00:36:06]               │       at settle (/dev/shm/workspace/parallel/23/kibana/node_modules/axios/lib/core/settle.js:17:12)
[00:36:06]               │       at IncomingMessage.handleStreamEnd (/dev/shm/workspace/parallel/23/kibana/node_modules/axios/lib/adapters/http.js:260:11)
[00:36:06]               │       at endReadableNT (internal/streams/readable.js:1317:12)
[00:36:06]               │       at processTicksAndRejections (internal/process/task_queues.js:82:21)
[00:36:06]               │ 
[00:36:06]               │ 

Stack Trace

Error: Request failed with status code 503
    at createError (/dev/shm/workspace/parallel/23/kibana/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/dev/shm/workspace/parallel/23/kibana/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/dev/shm/workspace/parallel/23/kibana/node_modules/axios/lib/adapters/http.js:260:11)
    at endReadableNT (internal/streams/readable.js:1317:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  config: {
    url: 'http://elastic:changeme@localhost:61231/internal/security/users/transform_poweruser',
    method: 'delete',
    headers: {
      Accept: 'application/json, text/plain, */*',
      'kbn-xsrf': 'kbn-client',
      'User-Agent': 'axios/0.21.1'
    },
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    paramsSerializer: [Function: paramsSerializer],
    timeout: 0,
    adapter: [Function: httpAdapter],
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: 30000000,
    maxBodyLength: 30000000,
    httpsAgent: null,
    validateStatus: [Function: validateStatus],
    data: undefined
  },
  request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      socket: [Function (anonymous)],
      timeout: [Function (anonymous)],
      prefinish: [Function: requestOnPrefinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: true,
    _last: true,
    chunkedEncoding: false,
    shouldKeepAlive: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: false,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    _contentLength: 0,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    socket: Socket {
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'localhost',
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: null,
      _server: null,
      parser: null,
      _httpMessage: [Circular *1],
      [Symbol(async_id_symbol)]: 464788,
      [Symbol(kHandle)]: null,
      [Symbol(kSetNoDelay)]: false,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesRead)]: 552,
      [Symbol(kBytesWritten)]: 244,
      [Symbol(RequestTimeout)]: undefined
    },
    _header: 'DELETE /internal/security/users/transform_poweruser HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'kbn-xsrf: kbn-client\r\n' +
      'User-Agent: axios/0.21.1\r\n' +
      'Host: localhost:61231\r\n' +
      'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==\r\n' +
      'Connection: close\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: noopPendingOutput],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 80,
      protocol: 'http:',
      options: [Object],
      requests: {},
      sockets: {},
      freeSockets: {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 0,
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'DELETE',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    path: '/internal/security/users/transform_poweruser',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      socket: [Socket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      headers: [Object],
      rawHeaders: [Array],
      trailers: {},
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 503,
      statusMessage: 'Service Unavailable',
      client: [Socket],
      _consuming: false,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'http://elastic:changeme@localhost:61231/internal/security/users/transform_poweruser',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(RequestTimeout)]: undefined
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'localhost',
    protocol: 'http:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 0,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'http://elastic:changeme@localhost:61231/internal/security/users/transform_poweruser',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      'kbn-xsrf': [Array],
      'user-agent': [Array],
      host: [Array],
      authorization: [Array]
    }
  },
  response: {
    status: 503,
    statusText: 'Service Unavailable',
    headers: {
      'retry-after': '30',
      'x-content-type-options': 'nosniff',
      'referrer-policy': 'no-referrer-when-downgrade',
      'kbn-name': 'kibana-ci-immutable-centos-tests-xxl-1630931326479339723',
      'kbn-license-sig': 'ccba850f2621dbae912ec059069dec625d374f381da65def76fcbaabef25997f',
      'content-type': 'application/json; charset=utf-8',
      'cache-control': 'private, no-cache, no-store, must-revalidate',
      'content-length': '86',
      date: 'Mon, 06 Sep 2021 13:37:13 GMT',
      connection: 'close'
    },
    config: {
      url: 'http://elastic:changeme@localhost:61231/internal/security/users/transform_poweruser',
      method: 'delete',
      headers: [Object],
      transformRequest: [Array],
      transformResponse: [Array],
      paramsSerializer: [Function: paramsSerializer],
      timeout: 0,
      adapter: [Function: httpAdapter],
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: 30000000,
      maxBodyLength: 30000000,
      httpsAgent: null,
      validateStatus: [Function: validateStatus],
      data: undefined
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      socket: [Socket],
      _header: 'DELETE /internal/security/users/transform_poweruser HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'kbn-xsrf: kbn-client\r\n' +
        'User-Agent: axios/0.21.1\r\n' +
        'Host: localhost:61231\r\n' +
        'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: noopPendingOutput],
      agent: [Agent],
      socketPath: undefined,
      method: 'DELETE',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/internal/security/users/transform_poweruser',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'localhost',
      protocol: 'http:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    data: {
      statusCode: 503,
      error: 'Service Unavailable',
      message: 'License is not available.'
    }
  },
  isAxiosError: true,
  toJSON: [Function: toJSON]
}

---

and 2 more failures, only showing the first 3.

Metrics [docs]

✅ unchanged

History

• 💛 Build #150585 was flaky ba09301
• 💚 Build #150335 succeeded 01a14a2
• 💔 Build #150155 failed 7f45d6f
• 💛 Build #150138 was flaky 03e775051497ec4df561cdf2a54aac55ca121e1b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nkhristinin

[kibanamachine]

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

[kibanamachine]

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

[kibanamachine]

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

[rylnd]

Removing the backport missing label for now, as this isn't going to go into the 7.15 branch until 7.15.0 is released.

[kibanamachine]

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

[rylnd]

Closed the 7.15 backport for now since it appears the bot doesn't like it staying open.