Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed installer script to reuse an existing UCX Cluster policy if present #964

Merged
merged 4 commits into from
Feb 21, 2024

Conversation

HariGS-DB
Copy link
Contributor

Changes

This change fixes two bugs related to cluster policy setup during ucx installation

  • If a user is rerunning installation (due to some issue previously but some steps completed like policy creation), if a UCX cluster policy is found, it reuses that instead of creating one
  • If the user is upgrading UCX where the initial installation steps are skipped, but the policy ID is not found in the config.yaml (due to manually deleting or upgrading from an older version), then raise an InvalidParameterValue with a custom msg saying policy id not found and request the user to uninstall and reinstall ucx completely.

#963

Functionality

  • added relevant user documentation
  • added new CLI command
  • modified existing command: databricks labs ucx ...
  • added a new workflow
  • modified existing workflow: ...
  • added a new table
  • modified existing table: ...

Tests

  • manually tested
  • added unit tests
  • added integration tests
  • verified on staging environment (screenshot attached)

@HariGS-DB HariGS-DB requested a review from nfx February 21, 2024 01:19
@HariGS-DB HariGS-DB marked this pull request as ready for review February 21, 2024 01:19
@HariGS-DB HariGS-DB requested review from a team and andrascsillag-db February 21, 2024 01:19
Copy link

codecov bot commented Feb 21, 2024

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (c38ff72) 87.49% compared to head (f957db4) 87.53%.

Files Patch % Lines
src/databricks/labs/ucx/install.py 96.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #964      +/-   ##
==========================================
+ Coverage   87.49%   87.53%   +0.04%     
==========================================
  Files          44       44              
  Lines        5319     5337      +18     
  Branches      949      954       +5     
==========================================
+ Hits         4654     4672      +18     
  Misses        450      450              
  Partials      215      215              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/databricks/labs/ucx/install.py Outdated Show resolved Hide resolved
src/databricks/labs/ucx/install.py Outdated Show resolved Hide resolved
@HariGS-DB HariGS-DB requested review from nfx and removed request for andrascsillag-db February 21, 2024 08:00
Copy link
Collaborator

@nfx nfx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@nfx nfx changed the title Reuse existing UCX Cluster policy if present Fixed installer script to reuse an existing UCX Cluster policy if present Feb 21, 2024
@nfx nfx merged commit 9fc1778 into main Feb 21, 2024
6 of 7 checks passed
@nfx nfx deleted the bug/clusterpolicy branch February 21, 2024 10:58
nfx added a commit that referenced this pull request Feb 21, 2024
* Added secret detection logic to Azure service principal crawler ([#950](#950)).
* Create storage credentials based on instance profiles and existing roles ([#869](#869)).
* Enforced `protected-access` pylint rule ([#956](#956)).
* Enforced `pylint` on unit and integration test code ([#953](#953)).
* Enforcing `invalid-name` pylint rule ([#957](#957)).
* Fixed AzureResourcePermissions.load to call Installation.load ([#962](#962)).
* Fixed installer script to reuse an existing UCX Cluster policy if present ([#964](#964)).
* More `pylint` tuning ([#958](#958)).
* Refactor `workspace_client_mock` to have combine fixtures stored in separate JSON files ([#955](#955)).

Dependency updates:

 * Updated databricks-sdk requirement from ~=0.19.0 to ~=0.20.0 ([#961](#961)).
@nfx nfx mentioned this pull request Feb 21, 2024
nfx added a commit that referenced this pull request Feb 21, 2024
* Added secret detection logic to Azure service principal crawler
([#950](#950)).
* Create storage credentials based on instance profiles and existing
roles ([#869](#869)).
* Enforced `protected-access` pylint rule
([#956](#956)).
* Enforced `pylint` on unit and integration test code
([#953](#953)).
* Enforcing `invalid-name` pylint rule
([#957](#957)).
* Fixed AzureResourcePermissions.load to call Installation.load
([#962](#962)).
* Fixed installer script to reuse an existing UCX Cluster policy if
present ([#964](#964)).
* More `pylint` tuning
([#958](#958)).
* Refactor `workspace_client_mock` to have combine fixtures stored in
separate JSON files
([#955](#955)).

Dependency updates:

* Updated databricks-sdk requirement from ~=0.19.0 to ~=0.20.0
([#961](#961)).
dmoore247 pushed a commit that referenced this pull request Mar 23, 2024
…sent (#964)

## Changes
This change fixes two bugs related to cluster policy setup during ucx
installation
- If a user is rerunning installation (due to some issue previously but
some steps completed like policy creation), if a UCX cluster policy is
found, it reuses that instead of creating one
- If the user is upgrading UCX where the initial installation steps are
skipped, but the policy ID is not found in the config.yaml (due to
manually deleting or upgrading from an older version), then raise an
InvalidParameterValue with a custom msg saying policy id not found and
request the user to uninstall and reinstall ucx completely.

### #963 


### Functionality 

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs ucx ...`
- [ ] added a new workflow
- [ ] modified existing workflow: `...`
- [ ] added a new table
- [ ] modified existing table: `...`

### Tests

- [X] manually tested
- [X] added unit tests
- [X] added integration tests
- [ ] verified on staging environment (screenshot attached)
dmoore247 pushed a commit that referenced this pull request Mar 23, 2024
* Added secret detection logic to Azure service principal crawler
([#950](#950)).
* Create storage credentials based on instance profiles and existing
roles ([#869](#869)).
* Enforced `protected-access` pylint rule
([#956](#956)).
* Enforced `pylint` on unit and integration test code
([#953](#953)).
* Enforcing `invalid-name` pylint rule
([#957](#957)).
* Fixed AzureResourcePermissions.load to call Installation.load
([#962](#962)).
* Fixed installer script to reuse an existing UCX Cluster policy if
present ([#964](#964)).
* More `pylint` tuning
([#958](#958)).
* Refactor `workspace_client_mock` to have combine fixtures stored in
separate JSON files
([#955](#955)).

Dependency updates:

* Updated databricks-sdk requirement from ~=0.19.0 to ~=0.20.0
([#961](#961)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants