Release v0.6.0 (#598)

**Breaking changes** (existing installations need to reinstall UCX and
re-run assessment jobs)

* Switched local group migration component to rename groups instead of
creating backup groups
([#450](#450)).
* Mitigate permissions loss in Table ACLs by folding grants belonging to
the same principal, object id and object type together
([#512](#512)).

**New features**

* Added support for the experimental Databricks CLI launcher
([#517](#517)).
* Added support for external Hive Metastores including AWS Glue
([#400](#400)).
* Added more views to assessment dashboard
([#474](#474)).
* Added rate limit for creating backup group to increase stability
([#500](#500)).
* Added deduplication for mount point list
([#569](#569)).
* Added documentation to describe interaction with external Hive
Metastores ([#473](#473)).
* Added failure injection for job failure message propagation
([#591](#591)).
* Added uniqueness in the new warehouse name to avoid conflicts on
installation ([#542](#542)).
* Added a global init script to collect Hive Metastore lineage
([#513](#513)).
* Added retry set/update permissions when possible and assess the
changes in the workspace
([#519](#519)).
* Use `~/.ucx/state.json` to store the state of both dashboards and jobs
([#561](#561)).

**Bug fixes**

* Fixed handling for `OWN` table permissions
([#571](#571)).
* Fixed handling of keys with and without values.
([#514](#514)).
* Fixed integration test failures related to concurrent group delete
([#584](#584)).
* Fixed issue with workspace listing process on None type `object_type`
([#481](#481)).
* Fixed missing group entitlement migration bug
([#583](#583)).
* Fixed entitlement application for account-level groups
([#529](#529)).
* Fixed assessment throwing an error when the owner of an object is
empty ([#485](#485)).
* Fixed installer to migrate between different configuration file
versions ([#596](#596)).
* Fixed cluster policy crawler to be aware of deleted policies
([#486](#486)).
* Improved error message for not null constraints violated
([#532](#532)).
* Improved integration test resiliency
([#597](#597),
[#594](#594),
[#586](#586)).
* Introduced Safer access to workspace objects' properties.
([#530](#530)).
* Mitigated permissions loss in Table ACLs by running appliers with
single thread ([#518](#518)).
* Running apply permission task before assessment should display message
([#487](#487)).
* Split integration tests from blocking the merge queue
([#496](#496)).
* Support more than one dashboard per step
([#472](#472)).
* Update databricks-sdk requirement from ~=0.11.0 to ~=0.12.0
([#505](#505)).
* Update databricks-sdk requirement from ~=0.12.0 to ~=0.13.0
([#575](#575)).

CHANGELOG.md

@@ -1,5 +1,47 @@
 # Version changelog
 
+## 0.6.0
+
+**Breaking changes** (existing installations need to reinstall UCX and re-run assessment jobs)
+
+ * Switched local group migration component to rename groups instead of creating backup groups ([#450](https://github.com/databricks/ucx/pull/450)).
+ * Mitigate permissions loss in Table ACLs by folding grants belonging to the same principal, object id and object type together ([#512](https://github.com/databricks/ucx/pull/512)).
+
+**New features**
+
+ * Added support for the experimental Databricks CLI launcher ([#517](https://github.com/databricks/ucx/pull/517)).
+ * Added support for external Hive Metastores including AWS Glue ([#400](https://github.com/databricks/ucx/pull/400)).
+ * Added more views to assessment dashboard ([#474](https://github.com/databricks/ucx/pull/474)).
+ * Added rate limit for creating backup group to increase stability ([#500](https://github.com/databricks/ucx/pull/500)).
+ * Added deduplication for mount point list ([#569](https://github.com/databricks/ucx/pull/569)).
+ * Added documentation to describe interaction with external Hive Metastores ([#473](https://github.com/databricks/ucx/pull/473)).
+ * Added failure injection for job failure message propagation  ([#591](https://github.com/databricks/ucx/pull/591)).
+ * Added uniqueness in the new warehouse name to avoid conflicts on installation ([#542](https://github.com/databricks/ucx/pull/542)).
+ * Added a global init script to collect Hive Metastore lineage ([#513](https://github.com/databricks/ucx/pull/513)).
+ * Added retry set/update permissions when possible and assess the changes in the workspace ([#519](https://github.com/databricks/ucx/pull/519)).
+ * Use `~/.ucx/state.json` to store the state of both dashboards and jobs ([#561](https://github.com/databricks/ucx/pull/561)).
+
+**Bug fixes**
+
+ * Fixed handling for `OWN` table permissions ([#571](https://github.com/databricks/ucx/pull/571)).
+ * Fixed handling of keys with and without values. ([#514](https://github.com/databricks/ucx/pull/514)).
+ * Fixed integration test failures related to concurrent group delete ([#584](https://github.com/databricks/ucx/pull/584)).
+ * Fixed issue with workspace listing process on None type `object_type` ([#481](https://github.com/databricks/ucx/pull/481)).
+ * Fixed missing group entitlement migration bug ([#583](https://github.com/databricks/ucx/pull/583)).
+ * Fixed entitlement application for account-level groups ([#529](https://github.com/databricks/ucx/pull/529)).
+ * Fixed assessment throwing an error when the owner of an object is empty ([#485](https://github.com/databricks/ucx/pull/485)).
+ * Fixed installer to migrate between different configuration file versions ([#596](https://github.com/databricks/ucx/pull/596)).
+ * Fixed cluster policy crawler to be aware of deleted policies ([#486](https://github.com/databricks/ucx/pull/486)).
+ * Improved error message for not null constraints violated ([#532](https://github.com/databricks/ucx/pull/532)).
+ * Improved integration test resiliency ([#597](https://github.com/databricks/ucx/pull/597), [#594](https://github.com/databricks/ucx/pull/594), [#586](https://github.com/databricks/ucx/pull/586)).
+ * Introduced Safer access to workspace objects' properties. ([#530](https://github.com/databricks/ucx/pull/530)).
+ * Mitigated permissions loss in Table ACLs by running appliers with single thread ([#518](https://github.com/databricks/ucx/pull/518)).
+ * Running apply permission task before assessment should display message ([#487](https://github.com/databricks/ucx/pull/487)).
+ * Split integration tests from blocking the merge queue ([#496](https://github.com/databricks/ucx/pull/496)).
+ * Support more than one dashboard per step ([#472](https://github.com/databricks/ucx/pull/472)).
+ * Update databricks-sdk requirement from ~=0.11.0 to ~=0.12.0 ([#505](https://github.com/databricks/ucx/pull/505)).
+ * Update databricks-sdk requirement from ~=0.12.0 to ~=0.13.0 ([#575](https://github.com/databricks/ucx/pull/575)).
+
 ## 0.5.0
 
 * Added `make install-dev` and a stronger `make clean` for easier dev on-boarding and release upgrades ([#458](https://github.com/databricks/ucx/pull/458)).

src/databricks/labs/ucx/__about__.py

@@ -1,2 +1,2 @@
 # DO NOT MODIFY THIS FILE
-__version__ = "0.5.0"
+__version__ = "0.6.0"

src/databricks/labs/ucx/install.py

@@ -15,7 +15,7 @@
 
 import yaml
 from databricks.sdk import WorkspaceClient
-from databricks.sdk.errors import NotFound, OperationFailed
+from databricks.sdk.errors import InvalidParameterValue, NotFound, OperationFailed
 from databricks.sdk.mixins.compute import SemVer
 from databricks.sdk.service import compute, jobs
 from databricks.sdk.service.sql import EndpointInfoWarehouseType, SpotInstancePolicy
@@ -457,24 +457,35 @@ def _create_jobs(self):
             settings = self._job_settings(step_name, remote_wheel)
             if self._override_clusters:
                 settings = self._apply_cluster_overrides(settings, self._override_clusters, wheel_runner)
-            if step_name in self._state.jobs:
-                job_id = self._state.jobs[step_name]
-                logger.info(f"Updating configuration for step={step_name} job_id={job_id}")
-                self._ws.jobs.reset(job_id, jobs.JobSettings(**settings))
-            else:
-                logger.info(f"Creating new job configuration for step={step_name}")
-                job_id = self._ws.jobs.create(**settings).job_id
-                self._state.jobs[step_name] = job_id
+            self._deploy_workflow(step_name, settings)
 
         for step_name, job_id in self._state.jobs.items():
             if step_name not in desired_steps:
-                logger.info(f"Removing job_id={job_id}, as it is no longer needed")
-                self._ws.jobs.delete(job_id)
+                try:
+                    logger.info(f"Removing job_id={job_id}, as it is no longer needed")
+                    self._ws.jobs.delete(job_id)
+                except InvalidParameterValue:
+                    logger.warning(f"step={step_name} does not exist anymore for some reason")
+                    continue
 
         self._state.save()
         self._create_readme()
         self._create_debug(remote_wheel)
 
+    def _deploy_workflow(self, step_name: str, settings):
+        if step_name in self._state.jobs:
+            try:
+                job_id = self._state.jobs[step_name]
+                logger.info(f"Updating configuration for step={step_name} job_id={job_id}")
+                return self._ws.jobs.reset(job_id, jobs.JobSettings(**settings))
+            except InvalidParameterValue:
+                del self._state.jobs[step_name]
+                logger.warning(f"step={step_name} does not exist anymore for some reason")
+                return self._deploy_workflow(step_name, settings)
+        logger.info(f"Creating new job configuration for step={step_name}")
+        job_id = self._ws.jobs.create(**settings).job_id
+        self._state.jobs[step_name] = job_id
+
     def _deployed_steps_pre_v06(self):
         deployed_steps = {}
         logger.debug(f"Fetching all jobs to determine already deployed steps for app={self._app}")