Update databricks-sdk requirement from ~=0.20.0 to ~=0.21.0 #1030
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![dependabot[bot]](https://avatars.githubusercontent.com/in/29110?s=60&v=4){kind=small}
Updates the requirements on [databricks-sdk](https://github.com/databricks/databricks-sdk-py) to permit the latest version. - [Release notes](https://github.com/databricks/databricks-sdk-py/releases) - [Changelog](https://github.com/databricks/databricks-sdk-py/blob/main/CHANGELOG.md) - [Commits](databricks/databricks-sdk-py@v0.20.0...v0.21.0) --- updated-dependencies: - dependency-name: databricks-sdk dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1030 +/- ##
=======================================
Coverage 88.54% 88.54%
=======================================
Files 48 48
Lines 6302 6302
Branches 1139 1139
=======================================
Hits 5580 5580
Misses 477 477
Partials 245 245 ☔ View full report in Codecov by Sentry. |
nfx
added a commit
that referenced
this pull request
Mar 8, 2024
* Added AWS IAM roles support to `databricks labs ucx migrate-credentials` command ([#973](#973)). This commit adds AWS Identity and Access Management (IAM) roles support to the `databricks labs ucx migrate-credentials` command, resolving issue [#862](#862) and being related to pull request [#874](#874). It includes the addition of a `load` function to `AWSResourcePermissions` to return identified instance profiles and the creation of an `IamRoleMigration` class under `aws/credentials.py` to migrate identified AWS instance profiles. Additionally, user documentation and a new CLI command `databricks labs ucx migrate-credentials` have been added, and the changes have been thoroughly tested with manual, unit, and integration tests. The functionality additions include new methods such as `add_uc_role_policy` and `update_uc_trust_role`, among others, designed to facilitate the migration process for AWS IAM roles. * Added `create-catalogs-schemas` command to prepare destination catalogs and schemas before table migration ([#1028](#1028)). The Databricks Labs Unity Catalog (UCX) tool has been updated with a new `create-catalogs-schemas` command to facilitate the creation of destination catalogs and schemas prior to table migration. This command should be executed after the `create-table-mapping` command and is designed to prepare the workspace for migrating tables to UC. Additionally, a new `CatalogSchema` class has been added to the `hive_metastore` package to manage the creation of catalogs and schemas in the Hive metastore. This new functionality simplifies the process of preparing the destination Hive metastore for table migration, reducing the likelihood of user errors and ensuring that the metastore is properly configured. Unit tests have been added to the `tests/unit/hive_metastore` directory to verify the behavior of the `CatalogSchema` class and the new `create-catalogs-schemas` command. This command is intended for use in contexts where GCP is not supported. * Added automated upgrade option to set up cluster policy ([#1024](#1024)). This commit introduces an automated upgrade option for setting up a cluster policy for older versions of UCX, separating the cluster creation policy from install.py to installer.policy.py and adding an upgrade script for older UCX versions. A new class, `ClusterPolicyInstaller`, is added to the `policy.py` file in the `installer` package to manage the creation and update of a Databricks cluster policy for Unity Catalog Migration. This class handles creating a new cluster policy with specific configurations, extracting external Hive Metastore configurations, and updating job policies. Additionally, the commit includes refactoring, removal of library references, and a new script, v0.15.0_added_cluster_policy.py, which contains the upgrade function. The changes are tested through manual and automated testing with unit tests and integration tests. This feature is intended for software engineers working with the project. * Added crawling for init scripts on local files to assessment workflow ([#960](#960)). This commit introduces the ability to crawl init scripts stored on local files and S3 as part of the assessment workflow, resolving issue [#9](#9) * Added database filter for the `assessment` workflow ([#989](#989)). In this release, we have added a new configuration option, `include_databases`, to the assessment workflow which allows users to specify a list of databases to include for migration, rather than crawling all the databases in the Hive Metastore. This feature is implemented in the `TablesCrawler`, `UdfsCrawler`, `GrantsCrawler` classes and the associated functions such as `_all_databases`, `getIncludeDatabases`, `_select_databases`. These changes aim to improve efficiency and reduce unnecessary crawling, and are accompanied by modifications to existing functionality, as well as the addition of unit and integration tests. The changes have been manually tested and verified on a staging environment. * Estimate migration effort based on assessment database ([#1008](#1008)). In this release, a new functionality has been added to estimate the migration effort for each asset in the assessment database. The estimation is presented in days and is displayed on a new estimates dashboard with a summary widget for a global estimate per object type, along with assumptions and scope for each object type. A new `query` parameter has been added to the `SimpleQuery` class to support this feature. Additional changes include the update of the `_install_viz` and `_install_query` methods, the inclusion of the `data_source_id` in the query metadata, and the addition of tests to ensure the proper functioning of the new feature. A new fixture, `mock_installation_with_jobs`, has been added to support testing of the assessment estimates dashboard. * Explicitly write to `hive_metastore` from `crawl_tables` task ([#1021](#1021)). In this release, we have improved the clarity and specificity of our handling of the `hive_metastore` in the `crawl_tables` task. Previously, the `df.write.saveAsTable` method was used without explicitly specifying the `hive_metastore` database, which could result in ambiguity. To address this issue, we have updated the `saveAsTable` method to include the `hive_metastore` database, ensuring that tables are written to the correct location in the Hive metastore. These changes are confined to the `src/databricks/labs/ucx/hive_metastore/tables.scala` file and affect the `crawl_tables` task. While no new methods have been added, the existing `saveAsTable` method has been modified to enhance the accuracy and predictability of our interaction with the Hive metastore. * Improved documentation for `databricks labs ucx move` command ([#1025](#1025)). The `databricks labs ucx move` command has been updated with new improvements to its documentation, providing enhanced clarity and ease of use for developers and administrators. This command facilitates the movement of UC tables/table(s) from one schema to another, either in the same or different catalog, during the table upgrade process. A significant enhancement is the preservation of the source table's permissions when moving to a new schema or catalog, maintaining the original table's access controls, simplifying the management of table permissions, and streamlining the migration process. These improvements aim to facilitate a more efficient table migration experience, ensuring that developers and administrators can effectively manage their UC tables while maintaining the desired level of access control and security. * Updated databricks-sdk requirement from ~=0.20.0 to ~=0.21.0 ([#1030](#1030)). In this update, the `databricks-sdk` package requirement has been updated to version `~=0.21.0` from `~=0.20.0`. This new version addresses several bugs and provides enhancements, including the fix for the `get_workspace_client` method in GCP, the use of the `all-apis` scope with the external browser, and an attempt to initialize all Databricks globals. Moreover, the API's settings nesting approach has changed, which may cause compatibility issues with previous versions. Several new services and dataclasses have been added to the API, and documentation and examples have been updated accordingly. There are no updates to the `databricks-labs-blueprint` and `PyYAML` dependencies in this commit.
Merged
nfx
added a commit
that referenced
this pull request
Mar 8, 2024
* Added AWS IAM roles support to `databricks labs ucx migrate-credentials` command ([#973](#973)). This commit adds AWS Identity and Access Management (IAM) roles support to the `databricks labs ucx migrate-credentials` command, resolving issue [#862](#862) and being related to pull request [#874](#874). It includes the addition of a `load` function to `AWSResourcePermissions` to return identified instance profiles and the creation of an `IamRoleMigration` class under `aws/credentials.py` to migrate identified AWS instance profiles. Additionally, user documentation and a new CLI command `databricks labs ucx migrate-credentials` have been added, and the changes have been thoroughly tested with manual, unit, and integration tests. The functionality additions include new methods such as `add_uc_role_policy` and `update_uc_trust_role`, among others, designed to facilitate the migration process for AWS IAM roles. * Added `create-catalogs-schemas` command to prepare destination catalogs and schemas before table migration ([#1028](#1028)). The Databricks Labs Unity Catalog (UCX) tool has been updated with a new `create-catalogs-schemas` command to facilitate the creation of destination catalogs and schemas prior to table migration. This command should be executed after the `create-table-mapping` command and is designed to prepare the workspace for migrating tables to UC. Additionally, a new `CatalogSchema` class has been added to the `hive_metastore` package to manage the creation of catalogs and schemas in the Hive metastore. This new functionality simplifies the process of preparing the destination Hive metastore for table migration, reducing the likelihood of user errors and ensuring that the metastore is properly configured. Unit tests have been added to the `tests/unit/hive_metastore` directory to verify the behavior of the `CatalogSchema` class and the new `create-catalogs-schemas` command. This command is intended for use in contexts where GCP is not supported. * Added automated upgrade option to set up cluster policy ([#1024](#1024)). This commit introduces an automated upgrade option for setting up a cluster policy for older versions of UCX, separating the cluster creation policy from install.py to installer.policy.py and adding an upgrade script for older UCX versions. A new class, `ClusterPolicyInstaller`, is added to the `policy.py` file in the `installer` package to manage the creation and update of a Databricks cluster policy for Unity Catalog Migration. This class handles creating a new cluster policy with specific configurations, extracting external Hive Metastore configurations, and updating job policies. Additionally, the commit includes refactoring, removal of library references, and a new script, v0.15.0_added_cluster_policy.py, which contains the upgrade function. The changes are tested through manual and automated testing with unit tests and integration tests. This feature is intended for software engineers working with the project. * Added crawling for init scripts on local files to assessment workflow ([#960](#960)). This commit introduces the ability to crawl init scripts stored on local files and S3 as part of the assessment workflow, resolving issue [#9](#9) * Added database filter for the `assessment` workflow ([#989](#989)). In this release, we have added a new configuration option, `include_databases`, to the assessment workflow which allows users to specify a list of databases to include for migration, rather than crawling all the databases in the Hive Metastore. This feature is implemented in the `TablesCrawler`, `UdfsCrawler`, `GrantsCrawler` classes and the associated functions such as `_all_databases`, `getIncludeDatabases`, `_select_databases`. These changes aim to improve efficiency and reduce unnecessary crawling, and are accompanied by modifications to existing functionality, as well as the addition of unit and integration tests. The changes have been manually tested and verified on a staging environment. * Estimate migration effort based on assessment database ([#1008](#1008)). In this release, a new functionality has been added to estimate the migration effort for each asset in the assessment database. The estimation is presented in days and is displayed on a new estimates dashboard with a summary widget for a global estimate per object type, along with assumptions and scope for each object type. A new `query` parameter has been added to the `SimpleQuery` class to support this feature. Additional changes include the update of the `_install_viz` and `_install_query` methods, the inclusion of the `data_source_id` in the query metadata, and the addition of tests to ensure the proper functioning of the new feature. A new fixture, `mock_installation_with_jobs`, has been added to support testing of the assessment estimates dashboard. * Explicitly write to `hive_metastore` from `crawl_tables` task ([#1021](#1021)). In this release, we have improved the clarity and specificity of our handling of the `hive_metastore` in the `crawl_tables` task. Previously, the `df.write.saveAsTable` method was used without explicitly specifying the `hive_metastore` database, which could result in ambiguity. To address this issue, we have updated the `saveAsTable` method to include the `hive_metastore` database, ensuring that tables are written to the correct location in the Hive metastore. These changes are confined to the `src/databricks/labs/ucx/hive_metastore/tables.scala` file and affect the `crawl_tables` task. While no new methods have been added, the existing `saveAsTable` method has been modified to enhance the accuracy and predictability of our interaction with the Hive metastore. * Improved documentation for `databricks labs ucx move` command ([#1025](#1025)). The `databricks labs ucx move` command has been updated with new improvements to its documentation, providing enhanced clarity and ease of use for developers and administrators. This command facilitates the movement of UC tables/table(s) from one schema to another, either in the same or different catalog, during the table upgrade process. A significant enhancement is the preservation of the source table's permissions when moving to a new schema or catalog, maintaining the original table's access controls, simplifying the management of table permissions, and streamlining the migration process. These improvements aim to facilitate a more efficient table migration experience, ensuring that developers and administrators can effectively manage their UC tables while maintaining the desired level of access control and security. * Updated databricks-sdk requirement from ~=0.20.0 to ~=0.21.0 ([#1030](#1030)). In this update, the `databricks-sdk` package requirement has been updated to version `~=0.21.0` from `~=0.20.0`. This new version addresses several bugs and provides enhancements, including the fix for the `get_workspace_client` method in GCP, the use of the `all-apis` scope with the external browser, and an attempt to initialize all Databricks globals. Moreover, the API's settings nesting approach has changed, which may cause compatibility issues with previous versions. Several new services and dataclasses have been added to the API, and documentation and examples have been updated accordingly. There are no updates to the `databricks-labs-blueprint` and `PyYAML` dependencies in this commit.
dmoore247
pushed a commit
that referenced
this pull request
Mar 23, 2024
Updates the requirements on [databricks-sdk](https://github.com/databricks/databricks-sdk-py) to permit the latest version. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/databricks/databricks-sdk-py/releases">databricks-sdk's releases</a>.</em></p> <blockquote> <h2>v0.21.0</h2> <h3>New Features and Improvements</h3> <ul> <li>Fixed get_workspace_client in GCP (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/532">#532</a>).</li> <li>Use all-apis scope with external-browser (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/563">#563</a>).</li> <li>Make a best effort attempt to initialise all Databricks globals (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/562">#562</a>).</li> <li>Fixed type issue with widgets.getArgument (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/581">#581</a>)</li> <li>Note: Backwards incompatible changes - Settings are now nested, please see the API changes below.</li> </ul> <h3>Documentation</h3> <ul> <li>Added Files API docs to the SDK Documentation (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/556">#556</a>).</li> <li>Added new example to list compute resource for SUBMIT_RUN job runs (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/572">#572</a>).</li> <li>Sorted index pages by name in docs (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/560">#560</a>).</li> <li>Added back enums to docs (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/557">#557</a>).</li> </ul> <h3>API Changes</h3> <h4>Added</h4> <p>Services:</p> <ul> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/permission_migration.html">w.permission_migration</a> workspace-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/automatic_cluster_update.html">w.settings.automatic_cluster_update</a> workspace-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/csp_enablement.html">w.settings.csp_enablement</a> workspace-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/account/settings/csp_enablement_account.html">a.settings.csp_enablement_account</a> account-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/default_namespace.html">w.settings.default_namespace</a> workspace-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/esm_enablement.html">w.settings.esm_enablement</a> workspace-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/account/settings/esm_enablement_account.html">a.settings.esm_enablement_account</a> account-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/account/settings/personal_compute.html">a.settings.personal_compute</a> account-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/restrict_workspace_admins.html">w.settings.restrict_workspace_admins</a> workspace-level service.</li> </ul> <p>Dataclasses:</p> <ul> <li><code>databricks.sdk.service.settings.AutomaticClusterUpdateSetting</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessage</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageEnablementDetails</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindow</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowDayOfWeek</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWeekDayBasedSchedule</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWeekDayFrequency</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWindowStartTime</code></li> <li><code>databricks.sdk.service.settings.ComplianceStandard</code></li> <li><code>databricks.sdk.service.settings.CspEnablement</code></li> <li><code>databricks.sdk.service.settings.CspEnablementAccount</code></li> <li><code>databricks.sdk.service.settings.CspEnablementAccountSetting</code></li> <li><code>databricks.sdk.service.settings.CspEnablementSetting</code></li> <li><code>databricks.sdk.service.settings.DeleteDefaultNamespaceRequest</code></li> <li><code>databricks.sdk.service.settings.DeletePersonalComputeRequest</code></li> <li><code>databricks.sdk.service.settings.DeleteRestrictWorkspaceAdminRequest</code></li> <li><code>databricks.sdk.service.settings.EsmEnablement</code></li> <li><code>databricks.sdk.service.settings.EsmEnablementAccount</code></li> <li><code>databricks.sdk.service.settings.EsmEnablementAccountSetting</code></li> <li><code>databricks.sdk.service.settings.EsmEnablementSetting</code></li> <li><code>databricks.sdk.service.settings.GetAutomaticClusterUpdateRequest</code></li> <li><code>databricks.sdk.service.settings.GetCspEnablementAccountRequest</code></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/databricks/databricks-sdk-py/blob/main/CHANGELOG.md">databricks-sdk's changelog</a>.</em></p> <blockquote> <h2>0.21.0</h2> <h3>New Features and Improvements</h3> <ul> <li>Fixed get_workspace_client in GCP (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/532">#532</a>).</li> <li>Use all-apis scope with external-browser (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/563">#563</a>).</li> <li>Make a best effort attempt to initialise all Databricks globals (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/562">#562</a>).</li> <li>Fixed type issue with widgets.getArgument (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/581">#581</a>)</li> <li>Note: Backwards incompatible changes - Settings are now nested, please see the API changes below.</li> </ul> <h3>Documentation</h3> <ul> <li>Added Files API docs to the SDK Documentation (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/556">#556</a>).</li> <li>Added new example to list compute resource for SUBMIT_RUN job runs (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/572">#572</a>).</li> <li>Sorted index pages by name in docs (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/560">#560</a>).</li> <li>Added back enums to docs (<a href="https://redirect.github.com/databricks/databricks-sdk-py/pull/557">#557</a>).</li> </ul> <h3>API Changes</h3> <h4>Added</h4> <p>Services:</p> <ul> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/permission_migration.html">w.permission_migration</a> workspace-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/automatic_cluster_update.html">w.settings.automatic_cluster_update</a> workspace-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/csp_enablement.html">w.settings.csp_enablement</a> workspace-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/account/settings/csp_enablement_account.html">a.settings.csp_enablement_account</a> account-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/default_namespace.html">w.settings.default_namespace</a> workspace-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/esm_enablement.html">w.settings.esm_enablement</a> workspace-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/account/settings/esm_enablement_account.html">a.settings.esm_enablement_account</a> account-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/account/settings/personal_compute.html">a.settings.personal_compute</a> account-level service.</li> <li><a href="https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/restrict_workspace_admins.html">w.settings.restrict_workspace_admins</a> workspace-level service.</li> </ul> <p>Dataclasses:</p> <ul> <li><code>databricks.sdk.service.settings.AutomaticClusterUpdateSetting</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessage</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageEnablementDetails</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindow</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowDayOfWeek</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWeekDayBasedSchedule</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWeekDayFrequency</code></li> <li><code>databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWindowStartTime</code></li> <li><code>databricks.sdk.service.settings.ComplianceStandard</code></li> <li><code>databricks.sdk.service.settings.CspEnablement</code></li> <li><code>databricks.sdk.service.settings.CspEnablementAccount</code></li> <li><code>databricks.sdk.service.settings.CspEnablementAccountSetting</code></li> <li><code>databricks.sdk.service.settings.CspEnablementSetting</code></li> <li><code>databricks.sdk.service.settings.DeleteDefaultNamespaceRequest</code></li> <li><code>databricks.sdk.service.settings.DeletePersonalComputeRequest</code></li> <li><code>databricks.sdk.service.settings.DeleteRestrictWorkspaceAdminRequest</code></li> <li><code>databricks.sdk.service.settings.EsmEnablement</code></li> <li><code>databricks.sdk.service.settings.EsmEnablementAccount</code></li> <li><code>databricks.sdk.service.settings.EsmEnablementAccountSetting</code></li> <li><code>databricks.sdk.service.settings.EsmEnablementSetting</code></li> <li><code>databricks.sdk.service.settings.GetAutomaticClusterUpdateRequest</code></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/databricks/databricks-sdk-py/commit/898b57d2558a23f2f1f03b374b7687915d868a9d"><code>898b57d</code></a> Release v0.21.0 (<a href="https://redirect.github.com/databricks/databricks-sdk-py/issues/578">#578</a>)</li> <li><a href="https://github.com/databricks/databricks-sdk-py/commit/ddf64933cc4764b47626dd400ac0dd6564b62785"><code>ddf6493</code></a> Fix type issue with widgets.getArgument (<a href="https://redirect.github.com/databricks/databricks-sdk-py/issues/581">#581</a>)</li> <li><a href="https://github.com/databricks/databricks-sdk-py/commit/e861140836523d02a0f991e0aae154a80c79bf15"><code>e861140</code></a> Update SDK to latest OpenAPI spec (<a href="https://redirect.github.com/databricks/databricks-sdk-py/issues/576">#576</a>)</li> <li><a href="https://github.com/databricks/databricks-sdk-py/commit/a282eee599c3176ccf33a00429356a4627c1b977"><code>a282eee</code></a> New example to list compute resource for SUBMIT_RUN job runs (<a href="https://redirect.github.com/databricks/databricks-sdk-py/issues/572">#572</a>)</li> <li><a href="https://github.com/databricks/databricks-sdk-py/commit/9813180303a7d12bf14426da6d76dca2ff313f80"><code>9813180</code></a> Added tokei.rs badge (<a href="https://redirect.github.com/databricks/databricks-sdk-py/issues/567">#567</a>)</li> <li><a href="https://github.com/databricks/databricks-sdk-py/commit/f0fe02358e7c576857ac41ac62458165cd81b192"><code>f0fe023</code></a> Make a best effort attempt to initialise all Databricks globals (<a href="https://redirect.github.com/databricks/databricks-sdk-py/issues/562">#562</a>)</li> <li><a href="https://github.com/databricks/databricks-sdk-py/commit/525576073da90de0b98ff652ad5b6f2d2bb77871"><code>5255760</code></a> Use all-apis scope with external-browser (<a href="https://redirect.github.com/databricks/databricks-sdk-py/issues/563">#563</a>)</li> <li><a href="https://github.com/databricks/databricks-sdk-py/commit/2a91e8d0c2d61beabf996b7824835bd0b9c67bfd"><code>2a91e8d</code></a> Support subservices (<a href="https://redirect.github.com/databricks/databricks-sdk-py/issues/559">#559</a>)</li> <li><a href="https://github.com/databricks/databricks-sdk-py/commit/0cb9e42ff2282417433d4405f77e0c710b79113d"><code>0cb9e42</code></a> Distinguish between empty types and fields that can take any value (<a href="https://redirect.github.com/databricks/databricks-sdk-py/issues/561">#561</a>)</li> <li><a href="https://github.com/databricks/databricks-sdk-py/commit/3a2798f608da46ec44eebabae21b398d22ccb8f8"><code>3a2798f</code></a> Add integration tests for Files API (<a href="https://redirect.github.com/databricks/databricks-sdk-py/issues/552">#552</a>)</li> <li>Additional commits viewable in <a href="https://github.com/databricks/databricks-sdk-py/compare/v0.20.0...v0.21.0">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
dmoore247
pushed a commit
that referenced
this pull request
Mar 23, 2024
* Added AWS IAM roles support to `databricks labs ucx migrate-credentials` command ([#973](#973)). This commit adds AWS Identity and Access Management (IAM) roles support to the `databricks labs ucx migrate-credentials` command, resolving issue [#862](#862) and being related to pull request [#874](#874). It includes the addition of a `load` function to `AWSResourcePermissions` to return identified instance profiles and the creation of an `IamRoleMigration` class under `aws/credentials.py` to migrate identified AWS instance profiles. Additionally, user documentation and a new CLI command `databricks labs ucx migrate-credentials` have been added, and the changes have been thoroughly tested with manual, unit, and integration tests. The functionality additions include new methods such as `add_uc_role_policy` and `update_uc_trust_role`, among others, designed to facilitate the migration process for AWS IAM roles. * Added `create-catalogs-schemas` command to prepare destination catalogs and schemas before table migration ([#1028](#1028)). The Databricks Labs Unity Catalog (UCX) tool has been updated with a new `create-catalogs-schemas` command to facilitate the creation of destination catalogs and schemas prior to table migration. This command should be executed after the `create-table-mapping` command and is designed to prepare the workspace for migrating tables to UC. Additionally, a new `CatalogSchema` class has been added to the `hive_metastore` package to manage the creation of catalogs and schemas in the Hive metastore. This new functionality simplifies the process of preparing the destination Hive metastore for table migration, reducing the likelihood of user errors and ensuring that the metastore is properly configured. Unit tests have been added to the `tests/unit/hive_metastore` directory to verify the behavior of the `CatalogSchema` class and the new `create-catalogs-schemas` command. This command is intended for use in contexts where GCP is not supported. * Added automated upgrade option to set up cluster policy ([#1024](#1024)). This commit introduces an automated upgrade option for setting up a cluster policy for older versions of UCX, separating the cluster creation policy from install.py to installer.policy.py and adding an upgrade script for older UCX versions. A new class, `ClusterPolicyInstaller`, is added to the `policy.py` file in the `installer` package to manage the creation and update of a Databricks cluster policy for Unity Catalog Migration. This class handles creating a new cluster policy with specific configurations, extracting external Hive Metastore configurations, and updating job policies. Additionally, the commit includes refactoring, removal of library references, and a new script, v0.15.0_added_cluster_policy.py, which contains the upgrade function. The changes are tested through manual and automated testing with unit tests and integration tests. This feature is intended for software engineers working with the project. * Added crawling for init scripts on local files to assessment workflow ([#960](#960)). This commit introduces the ability to crawl init scripts stored on local files and S3 as part of the assessment workflow, resolving issue [#9](#9) * Added database filter for the `assessment` workflow ([#989](#989)). In this release, we have added a new configuration option, `include_databases`, to the assessment workflow which allows users to specify a list of databases to include for migration, rather than crawling all the databases in the Hive Metastore. This feature is implemented in the `TablesCrawler`, `UdfsCrawler`, `GrantsCrawler` classes and the associated functions such as `_all_databases`, `getIncludeDatabases`, `_select_databases`. These changes aim to improve efficiency and reduce unnecessary crawling, and are accompanied by modifications to existing functionality, as well as the addition of unit and integration tests. The changes have been manually tested and verified on a staging environment. * Estimate migration effort based on assessment database ([#1008](#1008)). In this release, a new functionality has been added to estimate the migration effort for each asset in the assessment database. The estimation is presented in days and is displayed on a new estimates dashboard with a summary widget for a global estimate per object type, along with assumptions and scope for each object type. A new `query` parameter has been added to the `SimpleQuery` class to support this feature. Additional changes include the update of the `_install_viz` and `_install_query` methods, the inclusion of the `data_source_id` in the query metadata, and the addition of tests to ensure the proper functioning of the new feature. A new fixture, `mock_installation_with_jobs`, has been added to support testing of the assessment estimates dashboard. * Explicitly write to `hive_metastore` from `crawl_tables` task ([#1021](#1021)). In this release, we have improved the clarity and specificity of our handling of the `hive_metastore` in the `crawl_tables` task. Previously, the `df.write.saveAsTable` method was used without explicitly specifying the `hive_metastore` database, which could result in ambiguity. To address this issue, we have updated the `saveAsTable` method to include the `hive_metastore` database, ensuring that tables are written to the correct location in the Hive metastore. These changes are confined to the `src/databricks/labs/ucx/hive_metastore/tables.scala` file and affect the `crawl_tables` task. While no new methods have been added, the existing `saveAsTable` method has been modified to enhance the accuracy and predictability of our interaction with the Hive metastore. * Improved documentation for `databricks labs ucx move` command ([#1025](#1025)). The `databricks labs ucx move` command has been updated with new improvements to its documentation, providing enhanced clarity and ease of use for developers and administrators. This command facilitates the movement of UC tables/table(s) from one schema to another, either in the same or different catalog, during the table upgrade process. A significant enhancement is the preservation of the source table's permissions when moving to a new schema or catalog, maintaining the original table's access controls, simplifying the management of table permissions, and streamlining the migration process. These improvements aim to facilitate a more efficient table migration experience, ensuring that developers and administrators can effectively manage their UC tables while maintaining the desired level of access control and security. * Updated databricks-sdk requirement from ~=0.20.0 to ~=0.21.0 ([#1030](#1030)). In this update, the `databricks-sdk` package requirement has been updated to version `~=0.21.0` from `~=0.20.0`. This new version addresses several bugs and provides enhancements, including the fix for the `get_workspace_client` method in GCP, the use of the `all-apis` scope with the external browser, and an attempt to initialize all Databricks globals. Moreover, the API's settings nesting approach has changed, which may cause compatibility issues with previous versions. Several new services and dataclasses have been added to the API, and documentation and examples have been updated accordingly. There are no updates to the `databricks-labs-blueprint` and `PyYAML` dependencies in this commit.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates the requirements on databricks-sdk to permit the latest version.
Release notes
Sourced from databricks-sdk's releases.
... (truncated)
Changelog
Sourced from databricks-sdk's changelog.
... (truncated)
Commits
898b57dRelease v0.21.0 (#578)ddf6493Fix type issue with widgets.getArgument (#581)e861140Update SDK to latest OpenAPI spec (#576)a282eeeNew example to list compute resource for SUBMIT_RUN job runs (#572)9813180Added tokei.rs badge (#567)f0fe023Make a best effort attempt to initialise all Databricks globals (#562)5255760Use all-apis scope with external-browser (#563)2a91e8dSupport subservices (#559)0cb9e42Distinguish between empty types and fields that can take any value (#561)3a2798fAdd integration tests for Files API (#552)Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting
@dependabot rebase.Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot show <dependency name> ignore conditionswill show all of the ignore conditions of the specified dependency@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)