[Reporting/Tests] Use reporting default settings in test server config #111626

tsullivan · 2021-09-08T18:41:02Z

Summary

This PR changes the test server configuration in x-pack/test/functional/config by removing the Reporting settings, in order to run the tests against default settings.

The presence of Reporting plugin settings in the tests server config makes it hard to run the tests against a Cloud instance. The set of config settings can be customized in the cloud instance, but the need to do so creates a gotcha for most people.

Note: other tests still set non-default settings for Reporting

Other test suites still exist that set non-default server settings for functional tests and API integration tests in Reporting:

x-pack/test/reporting_api_integration/reporting_and_security.config.ts
x-pack/test/reporting_functional/reporting_and_security.config.ts.

In order to run these tests using a Cloud instance, the kibana.yml needs to be customized with these settings:

xpack.reporting.roles.enabled: false
xpack.reporting.csv.maxSizeBytes: 6000

…test config

elasticmachine · 2021-09-09T00:54:54Z

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

elasticmachine · 2021-09-09T00:54:54Z

Pinging @elastic/kibana-app-services (Team:AppServices)

LeeDr

LGTM - once there's a green Jenkins build. Note to other reviewers, this PR just has changes in x-pack/test/, no product changes.

My thought is that we have a separate UI test suite where the non-default config setting is tested (and not run on Cloud). And then when that setting becomes the default the tests just get swapped between the suites.

jportner

Code review only, security changes LGTM.

Are there any plans to reintroduce these settings in other integration tests in the future?

tsullivan · 2021-09-09T18:07:57Z

My thought is that we have a separate UI test suite where the non-default config setting is tested (and not run on Cloud). And then when that setting becomes the default the tests just get swapped between the suites.

Are there any plans to reintroduce these settings in other integration tests in the future?

Thanks for bringing this up. I just filed: #111780. I also updated the description of this PR to explain there are still other tests that use the non-default settings for Reporting access control.

poffdeluxe · 2021-09-09T18:16:15Z

Can we backport this to 7.14 branch as well? The minimal_all causes failures for Canvas tests there too

poffdeluxe

Canvas changes look good and tests pass locally for me. I mentioned this in another comment but it'd be great if we could get this fix in there for the 7.14 branch as well since cloud tests for Canvas are failing there as well

kibanamachine · 2021-09-09T20:23:36Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💔 Build #152112 failed 56b83c4
💔 Build #152101 failed 0857d82

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

ppisljar

code LGTM

elastic#111626) * remove custom kibana server settings for reporting in default x-pack test config * have tests use the deprecated built-in role granting reporting access * restore test user default privilege for canvas * fix app privileges in tests * fix test_user not able to access canvas in the dashboard test * simplify some tests setup * update csv export timerange and snapshot * update fn tests for app privileges * fix feature controls test * Update discover_security.ts * fix reporting tests * test using defaults in the security privilege test * fix read-only privileges with url_create Permalinks * fix security api anonymous * fix anonymous capabilities tests * fix discover csv export tests * Update screenshots.ts * update discover csv fn tests * update snapshots Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # x-pack/test/functional/apps/discover/__snapshots__/reporting.snap

elastic#111626) * remove custom kibana server settings for reporting in default x-pack test config * have tests use the deprecated built-in role granting reporting access * restore test user default privilege for canvas * fix app privileges in tests * fix test_user not able to access canvas in the dashboard test * simplify some tests setup * update csv export timerange and snapshot * update fn tests for app privileges * fix feature controls test * Update discover_security.ts * fix reporting tests * test using defaults in the security privilege test * fix read-only privileges with url_create Permalinks * fix security api anonymous * fix anonymous capabilities tests * fix discover csv export tests * Update screenshots.ts * update discover csv fn tests * update snapshots Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # x-pack/test/api_integration/apis/security/privileges.ts # x-pack/test/functional/apps/canvas/reports.ts # x-pack/test/functional/apps/discover/__snapshots__/reporting.snap

#111626) (#111823) * remove custom kibana server settings for reporting in default x-pack test config * have tests use the deprecated built-in role granting reporting access * restore test user default privilege for canvas * fix app privileges in tests * fix test_user not able to access canvas in the dashboard test * simplify some tests setup * update csv export timerange and snapshot * update fn tests for app privileges * fix feature controls test * Update discover_security.ts * fix reporting tests * test using defaults in the security privilege test * fix read-only privileges with url_create Permalinks * fix security api anonymous * fix anonymous capabilities tests * fix discover csv export tests * Update screenshots.ts * update discover csv fn tests * update snapshots Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # x-pack/test/functional/apps/discover/__snapshots__/reporting.snap

#111626) (#111824) * remove custom kibana server settings for reporting in default x-pack test config * have tests use the deprecated built-in role granting reporting access * restore test user default privilege for canvas * fix app privileges in tests * fix test_user not able to access canvas in the dashboard test * simplify some tests setup * update csv export timerange and snapshot * update fn tests for app privileges * fix feature controls test * Update discover_security.ts * fix reporting tests * test using defaults in the security privilege test * fix read-only privileges with url_create Permalinks * fix security api anonymous * fix anonymous capabilities tests * fix discover csv export tests * Update screenshots.ts * update discover csv fn tests * update snapshots Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # x-pack/test/functional/apps/discover/__snapshots__/reporting.snap

…r config (#111626) (#111825) * [Reporting/Tests] Use reporting default settings in test server config (#111626) * remove custom kibana server settings for reporting in default x-pack test config * have tests use the deprecated built-in role granting reporting access * restore test user default privilege for canvas * fix app privileges in tests * fix test_user not able to access canvas in the dashboard test * simplify some tests setup * update csv export timerange and snapshot * update fn tests for app privileges * fix feature controls test * Update discover_security.ts * fix reporting tests * test using defaults in the security privilege test * fix read-only privileges with url_create Permalinks * fix security api anonymous * fix anonymous capabilities tests * fix discover csv export tests * Update screenshots.ts * update discover csv fn tests * update snapshots Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # x-pack/test/api_integration/apis/security/privileges.ts # x-pack/test/functional/apps/canvas/reports.ts # x-pack/test/functional/apps/discover/__snapshots__/reporting.snap * update test assertions

tsullivan and others added 10 commits September 8, 2021 11:40

remove custom kibana server settings for reporting in default x-pack …

a658664

…test config

have tests use the deprecated built-in role granting reporting access

2890901

restore test user default privilege for canvas

dd5ff4d

fix app privileges in tests

41a79c7

fix test_user not able to access canvas in the dashboard test

2eb6c71

simplify some tests setup

312f479

update csv export timerange and snapshot

8dd5d9f

update fn tests for app privileges

4254945

fix feature controls test

6e0bed7

Update discover_security.ts

4a0c8a1

elastic deleted a comment from kibanamachine Sep 8, 2021

tsullivan added 2 commits September 8, 2021 17:36

fix reporting tests

2431ded

test using defaults in the security privilege test

c091869

elastic deleted a comment from kibanamachine Sep 9, 2021

tsullivan changed the title ~~remove custom kibana server settings for reporting in default x-pack …~~ [Reporting/Tests] Use reporting default settings in test server config Sep 9, 2021

tsullivan marked this pull request as ready for review September 9, 2021 00:54

tsullivan requested review from a team as code owners September 9, 2021 00:54

tsullivan added release_note:skip Skip the PR/issue when compiling release notes v7.15.1 v7.16.0 v8.0.0 (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead Team:AppServices labels Sep 9, 2021

tsullivan requested a review from poffdeluxe September 9, 2021 00:55

LeeDr approved these changes Sep 9, 2021

View reviewed changes

jportner approved these changes Sep 9, 2021

View reviewed changes

tsullivan mentioned this pull request Sep 9, 2021

[Reporting] Re-introduce tests for Canvas & Dashboard's feature declarations when Reporting disables access-by-role #111780

Closed

update snapshots

3236d2f

poffdeluxe approved these changes Sep 9, 2021

View reviewed changes

tsullivan added the v7.14.2 label Sep 9, 2021

ppisljar approved these changes Sep 10, 2021

View reviewed changes

tsullivan merged commit 297e4c3 into elastic:master Sep 10, 2021

tsullivan deleted the reporting/unbreak-default-cloud-tests branch September 10, 2021 05:13

tsullivan mentioned this pull request Sep 10, 2021

[7.x] [Reporting/Tests] Use reporting default settings in test server config (#111626) #111823

Merged

tsullivan mentioned this pull request Sep 10, 2021

[7.15] [Reporting/Tests] Use reporting default settings in test server config (#111626) #111824

Merged

tsullivan mentioned this pull request Sep 10, 2021

[7.14] [Reporting/Tests] Use reporting default settings in test server config (#111626) #111825

Merged

liza-mae mentioned this pull request Sep 14, 2021

[test-failed]: Chrome X-Pack UI Functional Tests1.x-pack/test/functional/apps/dashboard/reporting/screenshots·ts - dashboard Reporting Dashboard Reporting Screenshots Print Layout downloads a PDF file #109470

Closed

jportner mentioned this pull request Sep 15, 2021

[test-failed]: X-Pack API Integration Tests1.x-pack/test/api_integration/apis/security/privileges·ts - apis security Privileges GET /api/security/privileges should return a privilege map with all known privileges, without actions #112322

Closed

cuff-links mentioned this pull request Mar 18, 2022

[Meta] FTR configurable test users for xpack #60815

Closed

68 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Reporting/Tests] Use reporting default settings in test server config #111626

[Reporting/Tests] Use reporting default settings in test server config #111626

tsullivan commented Sep 8, 2021 •

edited

Loading

elasticmachine commented Sep 9, 2021

elasticmachine commented Sep 9, 2021

LeeDr left a comment

jportner left a comment

tsullivan commented Sep 9, 2021 •

edited

Loading

poffdeluxe commented Sep 9, 2021 •

edited

Loading

poffdeluxe left a comment

kibanamachine commented Sep 9, 2021

ppisljar left a comment

[Reporting/Tests] Use reporting default settings in test server config #111626

[Reporting/Tests] Use reporting default settings in test server config #111626

Conversation

tsullivan commented Sep 8, 2021 • edited Loading

Summary

Note: other tests still set non-default settings for Reporting

elasticmachine commented Sep 9, 2021

elasticmachine commented Sep 9, 2021

LeeDr left a comment

Choose a reason for hiding this comment

jportner left a comment

Choose a reason for hiding this comment

tsullivan commented Sep 9, 2021 • edited Loading

poffdeluxe commented Sep 9, 2021 • edited Loading

poffdeluxe left a comment

Choose a reason for hiding this comment

kibanamachine commented Sep 9, 2021

💚 Build Succeeded

Metrics [docs]

History

ppisljar left a comment

Choose a reason for hiding this comment

tsullivan commented Sep 8, 2021 •

edited

Loading

tsullivan commented Sep 9, 2021 •

edited

Loading

poffdeluxe commented Sep 9, 2021 •

edited

Loading