Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added CLI Command databricks labs ucx save-uc-compatible-roles #863

Merged
merged 16 commits into from
Feb 4, 2024
Merged

Added CLI Command databricks labs ucx save-uc-compatible-roles #863

merged 16 commits into from
Feb 4, 2024

Conversation

FastLee
Copy link
Contributor

@FastLee FastLee commented Jan 31, 2024 •
edited
Loading

Changes

CLI command to scan roles that are set with trust relationships with the UC master roles and the S3 buckets they have access to.
Genererates a CSV file.

The CSV File has the following format:

iam_role_arn,resource_type,privilege,resource_path
arn:aws:iam::12345:rolerole1,s3,READ_FILES,s3://bucket1
arn:aws:iam::12345:role/role1,s3,READ_FILES,s3a://bucket1
arn:aws:iam::12345:role/role1,s3,READ_FILES,s3://bucket2
arn:aws:iam::12345:role/role1,s3,READ_FILES,s3a://bucket2

The command relies on AWS CLI Command and require the user to setup and configure it.
Requires a working setup of AWS CLI.
AWS CLI
The command saves a CSV to the UCX installation folder with the mapping.

The user has to be authenticated with AWS and the have the permissions to browse the resources and iam services.
More information can be found here:
https://docs.aws.amazon.com/IAM/latest/UserGuide/access_permissions-required.html

Linked issues

closes #861

Functionality

  • added relevant user documentation
  • added new CLI command
  • modified existing command: databricks labs ucx ...
  • added a new workflow
  • modified existing workflow: ...
  • added a new table
  • modified existing table: ...

Tests

  • manually tested
  • added unit tests
  • added integration tests
  • verified on staging environment (screenshot attached)

@codecov Codecov
Copy link

codecov bot commented Jan 31, 2024 •
edited
Loading

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (e11494c) 86.48% compared to head (86d42b5) 86.54%.

Files Patch % Lines
src/databricks/labs/ucx/assessment/aws.py 94.82% 2 Missing and 1 partial ⚠️
src/databricks/labs/ucx/cli.py 84.61% 2 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #863      +/-   ##
==========================================
+ Coverage   86.48%   86.54%   +0.06%     
==========================================
  Files          41       41              
  Lines        5016     5077      +61     
  Branches      907      925      +18     
==========================================
+ Hits         4338     4394      +56     
- Misses        469      473       +4     
- Partials      209      210       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@FastLee FastLee changed the title New CLI Command databricks labs ucx uc-compatible-roles New CLI Command databricks labs ucx save-uc-compatible-roles Jan 31, 2024
nfx
nfx requested changes Jan 31, 2024
View reviewed changes
src/databricks/labs/ucx/assessment/aws.py Outdated Show resolved Hide resolved
src/databricks/labs/ucx/assessment/aws.py Outdated Show resolved Hide resolved
@FastLee FastLee changed the title New CLI Command databricks labs ucx save-uc-compatible-roles Added CLI Command databricks labs ucx save-uc-compatible-roles Jan 31, 2024
@FastLee FastLee requested a review from nfx January 31, 2024 19:49
@FastLee FastLee force-pushed the feature/uc-compatible-roles-861 branch from d918053 to 6743c1c Compare February 1, 2024 02:24
@FastLee FastLee marked this pull request as ready for review February 1, 2024 17:52
@FastLee FastLee requested review from a team and nkvuong February 1, 2024 17:52
@FastLee FastLee force-pushed the feature/uc-compatible-roles-861 branch from eafaf6e to 5d86948 Compare February 1, 2024 21:08
@FastLee FastLee force-pushed the feature/uc-compatible-roles-861 branch from ec22358 to 2790b22 Compare February 1, 2024 22:47
nfx
nfx approved these changes Feb 1, 2024
View reviewed changes
Copy link
Contributor

@nfx nfx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

william-conti
william-conti suggested changes Feb 2, 2024
View reviewed changes
Copy link
Contributor

@william-conti william-conti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some informations should be hidden

src/databricks/labs/ucx/assessment/aws.py Show resolved Hide resolved
src/databricks/labs/ucx/assessment/aws.py Show resolved Hide resolved
src/databricks/labs/ucx/cli.py Show resolved Hide resolved
@FastLee FastLee added this pull request to the merge queue Feb 4, 2024
@nfx nfx disabled auto-merge February 4, 2024 22:59
@nfx nfx removed this pull request from the merge queue due to a manual request Feb 4, 2024
@nfx nfx merged commit c99708c into main Feb 4, 2024
7 checks passed
@nfx nfx deleted the feature/uc-compatible-roles-861 branch February 4, 2024 23:00
nfx added a commit that referenced this pull request Feb 9, 2024
@nfx
Release v0.12.0
92025c0 
* Added CLI Command `databricks labs ucx save-uc-compatible-roles` ([#863](#863)).
* Added dashboard widget with table count by storage and format ([#852](#852)).
* Added verification of group permissions ([#841](#841)).
* Checking pipeline cluster config and cluster policy in 'crawl_pipelines' task ([#864](#864)).
* Created cluster policy (ucx-policy) to be used by all UCX compute. This may require customers to reinstall UCX. ([#853](#853)).
* Skip scanning objects that were removed on platform side since the last scan time, so that integration tests are less flaky ([#922](#922)).
* Updated assessment documentation ([#873](#873)).

Dependency updates:

 * Updated databricks-sdk requirement from ~=0.18.0 to ~=0.19.0 ([#930](#930)).
@nfx nfx mentioned this pull request Feb 9, 2024
Merged
nfx added a commit that referenced this pull request Feb 9, 2024
@nfx
Release v0.12.0 (#931)
158ae61 
* Added CLI Command `databricks labs ucx save-uc-compatible-roles`
([#863](#863)).
* Added dashboard widget with table count by storage and format
([#852](#852)).
* Added verification of group permissions
([#841](#841)).
* Checking pipeline cluster config and cluster policy in
'crawl_pipelines' task
([#864](#864)).
* Created cluster policy (ucx-policy) to be used by all UCX compute.
This may require customers to reinstall UCX.
([#853](#853)).
* Skip scanning objects that were removed on platform side since the
last scan time, so that integration tests are less flaky
([#922](#922)).
* Updated assessment documentation
([#873](#873)).

Dependency updates:

* Updated databricks-sdk requirement from ~=0.18.0 to ~=0.19.0
([#930](#930)).
dmoore247 pushed a commit that referenced this pull request Mar 23, 2024
@FastLee @dmoore247
Added CLI Command databricks labs ucx save-uc-compatible-roles (#863)
25c3886
dmoore247 pushed a commit that referenced this pull request Mar 23, 2024
@nfx @dmoore247
Release v0.12.0 (#931)
8ce8edf 
* Added CLI Command `databricks labs ucx save-uc-compatible-roles`
([#863](#863)).
* Added dashboard widget with table count by storage and format
([#852](#852)).
* Added verification of group permissions
([#841](#841)).
* Checking pipeline cluster config and cluster policy in
'crawl_pipelines' task
([#864](#864)).
* Created cluster policy (ucx-policy) to be used by all UCX compute.
This may require customers to reinstall UCX.
([#853](#853)).
* Skip scanning objects that were removed on platform side since the
last scan time, so that integration tests are less flaky
([#922](#922)).
* Updated assessment documentation
([#873](#873)).

Dependency updates:

* Updated databricks-sdk requirement from ~=0.18.0 to ~=0.19.0
([#930](#930)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nfx nfx nfx approved these changes

@william-conti william-conti william-conti approved these changes

@nkvuong nkvuong Awaiting requested review from nkvuong

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEATURE]: databricks labs ucx uc-compatible-roles for AWS
3 participants
@FastLee @nfx @william-conti