Release v0.16.0

[nfx]

• Added AWS IAM roles support to databricks labs ucx migrate-credentials command (#973). This commit adds AWS Identity and Access Management (IAM) roles support to the databricks labs ucx migrate-credentials command, resolving issue #862 and being related to pull request #874. It includes the addition of a load function to AWSResourcePermissions to return identified instance profiles and the creation of an IamRoleMigration class under aws/credentials.py to migrate identified AWS instance profiles. Additionally, user documentation and a new CLI command databricks labs ucx migrate-credentials have been added, and the changes have been thoroughly tested with manual, unit, and integration tests. The functionality additions include new methods such as add_uc_role_policy and update_uc_trust_role, among others, designed to facilitate the migration process for AWS IAM roles.
• Added create-catalogs-schemas command to prepare destination catalogs and schemas before table migration (#1028). The Databricks Labs Unity Catalog (UCX) tool has been updated with a new create-catalogs-schemas command to facilitate the creation of destination catalogs and schemas prior to table migration. This command should be executed after the create-table-mapping command and is designed to prepare the workspace for migrating tables to UC. Additionally, a new CatalogSchema class has been added to the hive_metastore package to manage the creation of catalogs and schemas in the Hive metastore. This new functionality simplifies the process of preparing the destination Hive metastore for table migration, reducing the likelihood of user errors and ensuring that the metastore is properly configured. Unit tests have been added to the tests/unit/hive_metastore directory to verify the behavior of the CatalogSchema class and the new create-catalogs-schemas command. This command is intended for use in contexts where GCP is not supported.
• Added automated upgrade option to set up cluster policy (#1024). This commit introduces an automated upgrade option for setting up a cluster policy for older versions of UCX, separating the cluster creation policy from install.py to installer.policy.py and adding an upgrade script for older UCX versions. A new class, ClusterPolicyInstaller, is added to the policy.py file in the installer package to manage the creation and update of a Databricks cluster policy for Unity Catalog Migration. This class handles creating a new cluster policy with specific configurations, extracting external Hive Metastore configurations, and updating job policies. Additionally, the commit includes refactoring, removal of library references, and a new script, v0.15.0_added_cluster_policy.py, which contains the upgrade function. The changes are tested through manual and automated testing with unit tests and integration tests. This feature is intended for software engineers working with the project.
• Added crawling for init scripts on local files to assessment workflow (#960). This commit introduces the ability to crawl init scripts stored on local files and S3 as part of the assessment workflow, resolving issue #9
• Added database filter for the assessment workflow (#989). In this release, we have added a new configuration option, include_databases, to the assessment workflow which allows users to specify a list of databases to include for migration, rather than crawling all the databases in the Hive Metastore. This feature is implemented in the TablesCrawler, UdfsCrawler, GrantsCrawler classes and the associated functions such as _all_databases, getIncludeDatabases, _select_databases. These changes aim to improve efficiency and reduce unnecessary crawling, and are accompanied by modifications to existing functionality, as well as the addition of unit and integration tests. The changes have been manually tested and verified on a staging environment.
• Estimate migration effort based on assessment database (#1008). In this release, a new functionality has been added to estimate the migration effort for each asset in the assessment database. The estimation is presented in days and is displayed on a new estimates dashboard with a summary widget for a global estimate per object type, along with assumptions and scope for each object type. A new query parameter has been added to the SimpleQuery class to support this feature. Additional changes include the update of the _install_viz and _install_query methods, the inclusion of the data_source_id in the query metadata, and the addition of tests to ensure the proper functioning of the new feature. A new fixture, mock_installation_with_jobs, has been added to support testing of the assessment estimates dashboard.
• Explicitly write to hive_metastore from crawl_tables task (#1021). In this release, we have improved the clarity and specificity of our handling of the hive_metastore in the crawl_tables task. Previously, the df.write.saveAsTable method was used without explicitly specifying the hive_metastore database, which could result in ambiguity. To address this issue, we have updated the saveAsTable method to include the hive_metastore database, ensuring that tables are written to the correct location in the Hive metastore. These changes are confined to the src/databricks/labs/ucx/hive_metastore/tables.scala file and affect the crawl_tables task. While no new methods have been added, the existing saveAsTable method has been modified to enhance the accuracy and predictability of our interaction with the Hive metastore.
• Improved documentation for databricks labs ucx move command (#1025). The databricks labs ucx move command has been updated with new improvements to its documentation, providing enhanced clarity and ease of use for developers and administrators. This command facilitates the movement of UC tables/table(s) from one schema to another, either in the same or different catalog, during the table upgrade process. A significant enhancement is the preservation of the source table's permissions when moving to a new schema or catalog, maintaining the original table's access controls, simplifying the management of table permissions, and streamlining the migration process. These improvements aim to facilitate a more efficient table migration experience, ensuring that developers and administrators can effectively manage their UC tables while maintaining the desired level of access control and security.
• Updated databricks-sdk requirement from ~=0.20.0 to ~=0.21.0 (#1030). In this update, the databricks-sdk package requirement has been updated to version ~=0.21.0 from ~=0.20.0. This new version addresses several bugs and provides enhancements, including the fix for the get_workspace_client method in GCP, the use of the all-apis scope with the external browser, and an attempt to initialize all Databricks globals. Moreover, the API's settings nesting approach has changed, which may cause compatibility issues with previous versions. Several new services and dataclasses have been added to the API, and documentation and examples have been updated accordingly. There are no updates to the databricks-labs-blueprint and PyYAML dependencies in this commit.