New CLI command for workspace mapping #678

pritishpai · 2023-12-05T23:24:40Z

The account admin can upload workspace info at account level into all the UCX installations under each workspace for the account.

Removed some code no longer required during pair programming with @nfx

Unit tests to follow.

codecov · 2023-12-05T23:27:37Z

Codecov Report

Attention: 22 lines in your changes are missing coverage. Please review.

Comparison is base (15908e2) 81.17% compared to head (a3a9017) 80.70%.

❗ Current head a3a9017 differs from pull request most recent head b5cedd8. Consider uploading reports for the commit b5cedd8 to get more accurate results

Files	Patch %	Lines
src/databricks/labs/ucx/account/workspaces.py	48.48%	15 Missing and 2 partials ⚠️
src/databricks/labs/ucx/cli.py	0.00%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #678      +/-   ##
==========================================
- Coverage   81.17%   80.70%   -0.48%     
==========================================
  Files          37       37              
  Lines        3851     3814      -37     
  Branches      719      708      -11     
==========================================
- Hits         3126     3078      -48     
- Misses        564      580      +16     
+ Partials      161      156       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nfx

and add some unit tests

src/databricks/labs/ucx/account/workspaces.py

The account admin can upload workspace info at account level into all the UCX installations under each workspace for the account.

Changed the cli command name and added indentation for the uploaded json.

The account admin can upload workspace info at account level into all the UCX installations under each workspace for the account.

Changed the cli command name and added indentation for the uploaded json.

nfx · 2023-12-12T22:19:57Z

tests/unit/account/test_workspaces.py

+    assert len(ws_clients) == 2
+
+    # test for cloud and deployment name
+    assert ws_clients[1].config.auth_type == "azure-cli"


this auth type is failing tests, don't test for it

Adding the mock back seems to allow the unit test to pass.

tests/unit/account/test_workspaces.py

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `workspace_group_name` and `account_group_name` to `make_ucx_group` fixture ([#664](#664)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added filters to run only cloud specific task ([#681](#681)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Cleanup README.md ([#695](#695)). * Cleanup installer framework and speed up test execution ([#711](#711)). * Decouple group configuration from `install.py` ([#714](#714)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed for secret scope apply task always raises ValueError ([#683](#683)). * Fixed some flaky integration tests ([#700](#700)). * More `mypy` chores ([#697](#697)). * Moved `ExternalLocations` and `Mounts` to `locations` module ([#692](#692)). * New CLI command for workspace mapping ([#678](#678)). * No more `mypy` chores after this ([#699](#699)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Some `mypy` chores ([#696](#696)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)). * Updated installation documentation to use Databricks CLI ([#686](#686)).

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `workspace_group_name` and `account_group_name` to `make_ucx_group` fixture ([#664](#664)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added filters to run only cloud specific task ([#681](#681)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Cleanup README.md ([#695](#695)). * Cleanup installer framework and speed up test execution ([#711](#711)). * Decouple group configuration from `install.py` ([#714](#714)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed for secret scope apply task always raises ValueError ([#683](#683)). * Fixed some flaky integration tests ([#700](#700)). * More `mypy` chores ([#697](#697)). * Moved `ExternalLocations` and `Mounts` to `locations` module ([#692](#692)). * New CLI command for workspace mapping ([#678](#678)). * No more `mypy` chores after this ([#699](#699)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Some `mypy` chores ([#696](#696)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated installation documentation to use Databricks CLI ([#686](#686)). Dependency updates: * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)).

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `databricks labs ucx validate-external-locations` command for cli ([#715](#715)). * Added `workspace_group_name` and `account_group_name` to `make_ucx_group` fixture ([#664](#664)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added filters to run only cloud specific task ([#681](#681)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Cleanup README.md ([#695](#695)). * Cleanup installer framework and speed up test execution ([#711](#711)). * Decouple group configuration from `install.py` ([#714](#714)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed for secret scope apply task always raises ValueError ([#683](#683)). * Fixed some flaky integration tests ([#700](#700)). * More `mypy` chores ([#697](#697)). * Moved `ExternalLocations` and `Mounts` to `locations` module ([#692](#692)). * New CLI command for workspace mapping ([#678](#678)). * No more `mypy` chores after this ([#699](#699)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Some `mypy` chores ([#696](#696)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated installation documentation to use Databricks CLI ([#686](#686)). Dependency updates: * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)).

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx ensure-assessment-run` to CLI commands ([#708](#708)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `databricks labs ucx validate-external-locations` command for cli ([#715](#715)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed secret scope apply task to raise ValueError ([#683](#683)). * Fixed legacy table ACL ownership migration and other integration testing issues ([#722](#722)). * Fixed some flaky integration tests ([#700](#700)). * New CLI command for workspace mapping ([#678](#678)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated installation documentation to use Databricks CLI ([#686](#686)). Dependency updates: * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)).

pritishpai requested review from a team and andrascsillag-db December 5, 2023 23:24

pritishpai had a problem deploying to account-admin December 5, 2023 23:24 — with GitHub Actions Failure

pritishpai had a problem deploying to account-admin December 5, 2023 23:39 — with GitHub Actions Failure

nfx requested changes Dec 6, 2023

View reviewed changes

src/databricks/labs/ucx/account/workspaces.py Outdated Show resolved Hide resolved

pritishpai linked an issue Dec 6, 2023 that may be closed by this pull request

Add databricks labs ucx sync-workspace-info command #667

Closed

pritishpai added 6 commits December 12, 2023 16:05

Add workspace-mapping to cli

401a174

The account admin can upload workspace info at account level into all the UCX installations under each workspace for the account.

cli command name change

d5f4095

Changed the cli command name and added indentation for the uploaded json.

Adding some unit tests

633afb9

Add workspace-mapping to cli

912a07d

The account admin can upload workspace info at account level into all the UCX installations under each workspace for the account.

cli command name change

b99f34c

Changed the cli command name and added indentation for the uploaded json.

Fixing errors

8bafb88

pritishpai force-pushed the feature/workspace_mapping_cli branch from 2501233 to 8bafb88 Compare December 12, 2023 21:33

pritishpai had a problem deploying to account-admin December 12, 2023 21:33 — with GitHub Actions Failure

nfx requested changes Dec 12, 2023

View reviewed changes

Added mock auth to fix test

a3a9017

pritishpai had a problem deploying to account-admin December 12, 2023 22:29 — with GitHub Actions Failure

Renaming mock fixture

b5cedd8

pritishpai had a problem deploying to account-admin December 12, 2023 22:41 — with GitHub Actions Failure

nfx approved these changes Dec 13, 2023

View reviewed changes

nfx merged commit 00c15cc into main Dec 13, 2023
4 of 5 checks passed

nfx deleted the feature/workspace_mapping_cli branch December 13, 2023 17:20

nfx mentioned this pull request Dec 21, 2023

Release v0.7.0 #719

Closed

nfx mentioned this pull request Dec 21, 2023

Release v0.7.0 #720

Closed

nfx mentioned this pull request Dec 21, 2023

Release v0.7.0 #723

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New CLI command for workspace mapping #678

New CLI command for workspace mapping #678

pritishpai commented Dec 5, 2023

codecov bot commented Dec 5, 2023 •

edited

Loading

nfx left a comment

nfx Dec 12, 2023

pritishpai Dec 12, 2023

New CLI command for workspace mapping #678

New CLI command for workspace mapping #678

Conversation

pritishpai commented Dec 5, 2023

codecov bot commented Dec 5, 2023 • edited Loading

Codecov Report

nfx left a comment

Choose a reason for hiding this comment

nfx Dec 12, 2023

Choose a reason for hiding this comment

pritishpai Dec 12, 2023

Choose a reason for hiding this comment

codecov bot commented Dec 5, 2023 •

edited

Loading