Added cluster policy selector in the installer #655

mirhendi · 2023-11-30T21:37:25Z

With this PR, we can specify a cluster policy and create job clusters based on the specified policy

CLAassistant · 2023-11-30T21:37:31Z

All committers have signed the CLA.

mirhendi · 2023-11-30T21:45:10Z

fixes issue #656

src/databricks/labs/ucx/install.py

Added test for custom policy, fixed lint

mirhendi · 2023-12-06T20:49:52Z

Fixed formats and added unit test, please re-execute the workflow.

src/databricks/labs/ucx/install.py

codecov · 2023-12-07T03:35:32Z

Codecov Report

Attention: 14 lines in your changes are missing coverage. Please review.

Comparison is base (3e19372) 81.38% compared to head (22ce87f) 81.15%.

❗ Current head 22ce87f differs from pull request most recent head 50809f8. Consider uploading reports for the commit 50809f8 to get more accurate results

Files	Patch %	Lines
src/databricks/labs/ucx/install.py	36.36%	10 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #655      +/-   ##
==========================================
- Coverage   81.38%   81.15%   -0.23%     
==========================================
  Files          37       37              
  Lines        3831     3848      +17     
  Branches      712      719       +7     
==========================================
+ Hits         3118     3123       +5     
- Misses        555      564       +9     
- Partials      158      161       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nfx · 2023-12-09T20:04:52Z

src/databricks/labs/ucx/install.py

+        policy = self._ws.cluster_policies.get(policy_id=policy_id)
+        def_json = json.loads(policy.definition)
+        instance_pool = def_json.get("instance_pool_id")
+        if instance_pool is not None:


Suggested change

if instance_pool is not None:

return instance_pool is not None

And if-else becomes redundant 😉

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `workspace_group_name` and `account_group_name` to `make_ucx_group` fixture ([#664](#664)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added filters to run only cloud specific task ([#681](#681)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Cleanup README.md ([#695](#695)). * Cleanup installer framework and speed up test execution ([#711](#711)). * Decouple group configuration from `install.py` ([#714](#714)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed for secret scope apply task always raises ValueError ([#683](#683)). * Fixed some flaky integration tests ([#700](#700)). * More `mypy` chores ([#697](#697)). * Moved `ExternalLocations` and `Mounts` to `locations` module ([#692](#692)). * New CLI command for workspace mapping ([#678](#678)). * No more `mypy` chores after this ([#699](#699)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Some `mypy` chores ([#696](#696)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)). * Updated installation documentation to use Databricks CLI ([#686](#686)).

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `workspace_group_name` and `account_group_name` to `make_ucx_group` fixture ([#664](#664)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added filters to run only cloud specific task ([#681](#681)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Cleanup README.md ([#695](#695)). * Cleanup installer framework and speed up test execution ([#711](#711)). * Decouple group configuration from `install.py` ([#714](#714)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed for secret scope apply task always raises ValueError ([#683](#683)). * Fixed some flaky integration tests ([#700](#700)). * More `mypy` chores ([#697](#697)). * Moved `ExternalLocations` and `Mounts` to `locations` module ([#692](#692)). * New CLI command for workspace mapping ([#678](#678)). * No more `mypy` chores after this ([#699](#699)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Some `mypy` chores ([#696](#696)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated installation documentation to use Databricks CLI ([#686](#686)). Dependency updates: * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)).

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `databricks labs ucx validate-external-locations` command for cli ([#715](#715)). * Added `workspace_group_name` and `account_group_name` to `make_ucx_group` fixture ([#664](#664)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added filters to run only cloud specific task ([#681](#681)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Cleanup README.md ([#695](#695)). * Cleanup installer framework and speed up test execution ([#711](#711)). * Decouple group configuration from `install.py` ([#714](#714)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed for secret scope apply task always raises ValueError ([#683](#683)). * Fixed some flaky integration tests ([#700](#700)). * More `mypy` chores ([#697](#697)). * Moved `ExternalLocations` and `Mounts` to `locations` module ([#692](#692)). * New CLI command for workspace mapping ([#678](#678)). * No more `mypy` chores after this ([#699](#699)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Some `mypy` chores ([#696](#696)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated installation documentation to use Databricks CLI ([#686](#686)). Dependency updates: * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)).

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx ensure-assessment-run` to CLI commands ([#708](#708)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `databricks labs ucx validate-external-locations` command for cli ([#715](#715)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed secret scope apply task to raise ValueError ([#683](#683)). * Fixed legacy table ACL ownership migration and other integration testing issues ([#722](#722)). * Fixed some flaky integration tests ([#700](#700)). * New CLI command for workspace mapping ([#678](#678)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated installation documentation to use Databricks CLI ([#686](#686)). Dependency updates: * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)).

Add cluster's policy support

ce3ca06

mirhendi requested review from a team and FastLee November 30, 2023 21:37

nfx reviewed Dec 1, 2023

View reviewed changes

src/databricks/labs/ucx/install.py Outdated Show resolved Hide resolved

mirhendi and others added 2 commits December 5, 2023 12:28

Adding choice list for selecting clusters policy

6d85d4b

Merge branch 'main' into main

725c125

nfx approved these changes Dec 5, 2023

View reviewed changes

nfx changed the title ~~Add cluster's policy support~~ Added cluster policy selector in the installer Dec 5, 2023

nfx linked an issue Dec 5, 2023 that may be closed by this pull request

Create clusters based on policy #656

Closed

mirhendi and others added 7 commits December 5, 2023 15:20

Fix variable name

e62f5fb

Fix linting

453fa9a

Merge branch 'databrickslabs:main' into main

a0cd565

Fix lint

515607f

Merge branch 'databrickslabs:main' into main

fd76724

Added test for custom policy, fixed lint

8c103fd

Merge pull request #1 from mirhendi/feature/policy

22ce87f

Added test for custom policy, fixed lint

dmoore247 suggested changes Dec 6, 2023

View reviewed changes

src/databricks/labs/ucx/install.py Show resolved Hide resolved

src/databricks/labs/ucx/install.py Show resolved Hide resolved

mirhendi had a problem deploying to account-admin December 8, 2023 14:25 — with GitHub Actions Failure

nfx approved these changes Dec 9, 2023

View reviewed changes

Merge branch 'databrickslabs:main' into main

50809f8

nfx merged commit 15908e2 into databrickslabs:main Dec 12, 2023
4 of 5 checks passed

pritishpai pushed a commit that referenced this pull request Dec 12, 2023

Added cluster policy selector in the installer (#655)

608a2d7

nfx mentioned this pull request Dec 21, 2023

Release v0.7.0 #719

Closed

nfx mentioned this pull request Dec 21, 2023

Release v0.7.0 #720

Closed

nfx mentioned this pull request Dec 21, 2023

Release v0.7.0 #723

Merged

mirhendi had a problem deploying to account-admin December 22, 2023 18:06 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added cluster policy selector in the installer #655

Added cluster policy selector in the installer #655

mirhendi commented Nov 30, 2023

CLAassistant commented Nov 30, 2023 •

edited

Loading

mirhendi commented Nov 30, 2023

mirhendi commented Dec 6, 2023

codecov bot commented Dec 7, 2023 •

edited

Loading

nfx Dec 9, 2023

	if instance_pool is not None:
	return instance_pool is not None

Added cluster policy selector in the installer #655

Added cluster policy selector in the installer #655

Conversation

mirhendi commented Nov 30, 2023

CLAassistant commented Nov 30, 2023 • edited Loading

mirhendi commented Nov 30, 2023

mirhendi commented Dec 6, 2023

codecov bot commented Dec 7, 2023 • edited Loading

Codecov Report

nfx Dec 9, 2023

Choose a reason for hiding this comment

CLAassistant commented Nov 30, 2023 •

edited

Loading

codecov bot commented Dec 7, 2023 •

edited

Loading