Fixing the issue in workspace id flag in create-account-group command #1094
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
prajin-29
had a problem deploying
to
account-admin
GitHub Actions
Failure
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1094 +/- ##
=======================================
Coverage 88.80% 88.81%
=======================================
Files 64 64
Lines 7130 7133 +3
Branches 1282 1284 +2
=======================================
+ Hits 6332 6335 +3
Misses 531 531
Partials 267 267 ☔ View full report in Codecov by Sentry. |
|
❌ 111/113 passed, 1 flaky, 2 failed, 20 skipped, 2h13m21s total ❌ test_migrate_external_table: databricks.labs.blueprint.parallel.ManyError: Detected 1 failures: BadRequest: PERMISSION_DENIED: Failed to get credentials: Storage Credential TEST_STORAGE_CREDENTIAL is not configured correctly. Please contact the owner or your account admin to update the configuration. (28.523s)❌ test_table_migration_job_cluster_override: databricks.sdk.errors.platform.Unknown: (java.lang.SecurityException) PERMISSION_DENIED: Failed to get credentials: Storage Credential TEST_STORAGE_CREDENTIAL is not configured correctly. Please contact the owner or your account admin to update the configuration. (12m39.31s)Flaky tests:
Running from acceptance #1726 |
nfx
added a commit
that referenced
this pull request
Mar 25, 2024
* Added instance pool id to WorkspaceConfig ([#1087](#1087)). In this release, the `create` method of the `_policy_installer` object has been updated to return an additional value, `instance_pool_id`, which is then assigned and passed as an argument to the `WorkspaceConfig` object in the `_configure_new_installation` method. The `ClusterPolicyInstaller` class in the `v0.15.0_added_cluster_policy.py` file has also been updated to return a fourth value, `instance_pool_id`, from the `create` method, allowing for more flexibility in future enhancements. Additionally, the test function `test_table_migration_job` in the `test_installation.py` file has been updated to skip when the script is not being run as part of a nightly test job or in debug mode, and the test functions in the `test_policy.py` file have been updated to reflect the new return value in the `create` method. These changes enable better management and scaling of resources through instance pools, provide more granular control in the WorkspaceConfig, and improve testing efficiency. * Added more cross-linking between CLI commands ([#1091](#1091)). In this release, we have introduced several enhancements to our open-source library's Command Line Interface (CLI) and documentation. Specifically, we have added more cross-linking between CLI commands to improve navigation and usability. The documentation has been updated to include a new step in the UCX installation process, where users are required to run the assessment workflow after installing UCX. This workflow is the first step in the migration process and checks the compatibility of the user's workspace with Unity Catalog. Additionally, we have added new commands for principal-prefix-access, migrate-credentials, and migrate-locations, which are part of the table migration process. These new commands require the assessment workflow and group migration workflow to be completed before they can be executed. Overall, these changes aim to provide a more streamlined and detailed installation and migration process, improving the user experience for software engineers. * Fixed command references in README.md ([#1093](#1093)). In this release, we have made improvements to the command references in the README.md file to enhance the overall readability and usability of the documentation for software engineers. Specifically, we have updated the links for the `migrate-locations` and `validate_external_locations` commands to use the correct syntax, enclosing them in backticks to denote code. This change ensures that the links are correctly interpreted as commands and addresses any issues that may have arisen with their previous formatting. It is important to note that no new methods have been added in this release, and the existing functionality of the commands has not been changed in scope or functionality. * Fixing the issue in workspace id flag in create-account-group command ([#1094](#1094)). In this update, we have improved the `create_account_group` command related to the `workspace_ids` flag in our open-source library. The `workspace_ids` flag's type has been changed from `list[int] | None` to `str | None`, allowing for easier input of multiple workspace IDs as a string of comma-separated integers. The `create_account_level_groups` function in the `AccountWorkspaces` class has been updated to accept this string and convert it to a list of integers before proceeding. To ensure proper functioning, we added a new test case `test_create_account_groups_with_id()` to check if the command handles the case when no workspace IDs are provided in the configuration. The `create_account_groups()` method now checks for this condition and raises a `ValueError`. Furthermore, the `manual_workspace_info()` method has been updated to handle workspace name input by the user, receiving the `ws` object, along with prompts that contain the user input for the workspace name and the next workspace ID. * Rely UCX on the latest 14.3 LTS DBR instead of 15.x ([#1097](#1097)). In this release, we have implemented a quick fix to rely on the Long Term Support (LTS) version 14.3 of the Databricks Runtime (DBR) instead of 15.x for UCX, addressing issue [#1096](#1096). This change affects the `_definition` function, which has been modified to use the latest LTS DBR instead of the latest Spark version. The `latest_lts_dbr` variable is now assigned the value returned by the `select_spark_version` method with the `latest=True` and `long_term_support=True` parameters. The `spark_version` key in the `policy_definition` dictionary is set to the value returned by the `_policy_config` method with `latest_lts_dbr` as the argument. Additionally, in the `tests/unit/installer/test_policy.py` file, the `select_spark_version` method of the `clusters` object has been updated to accept any number of arguments and consistently return the string "14.2.x-scala2.12", allowing for greater flexibility. This is a temporary solution, with a more comprehensive fix being tracked in issue [#1098](#1098). Developers should be aware of how the `clusters` object is used in the codebase when adopting this project.
Merged
nfx
added a commit
that referenced
this pull request
Mar 25, 2024
* Added instance pool id to WorkspaceConfig ([#1087](#1087)). In this release, the `create` method of the `_policy_installer` object has been updated to return an additional value, `instance_pool_id`, which is then assigned and passed as an argument to the `WorkspaceConfig` object in the `_configure_new_installation` method. The `ClusterPolicyInstaller` class in the `v0.15.0_added_cluster_policy.py` file has also been updated to return a fourth value, `instance_pool_id`, from the `create` method, allowing for more flexibility in future enhancements. Additionally, the test function `test_table_migration_job` in the `test_installation.py` file has been updated to skip when the script is not being run as part of a nightly test job or in debug mode, and the test functions in the `test_policy.py` file have been updated to reflect the new return value in the `create` method. These changes enable better management and scaling of resources through instance pools, provide more granular control in the WorkspaceConfig, and improve testing efficiency. * Added more cross-linking between CLI commands ([#1091](#1091)). In this release, we have introduced several enhancements to our open-source library's Command Line Interface (CLI) and documentation. Specifically, we have added more cross-linking between CLI commands to improve navigation and usability. The documentation has been updated to include a new step in the UCX installation process, where users are required to run the assessment workflow after installing UCX. This workflow is the first step in the migration process and checks the compatibility of the user's workspace with Unity Catalog. Additionally, we have added new commands for principal-prefix-access, migrate-credentials, and migrate-locations, which are part of the table migration process. These new commands require the assessment workflow and group migration workflow to be completed before they can be executed. Overall, these changes aim to provide a more streamlined and detailed installation and migration process, improving the user experience for software engineers. * Fixed command references in README.md ([#1093](#1093)). In this release, we have made improvements to the command references in the README.md file to enhance the overall readability and usability of the documentation for software engineers. Specifically, we have updated the links for the `migrate-locations` and `validate_external_locations` commands to use the correct syntax, enclosing them in backticks to denote code. This change ensures that the links are correctly interpreted as commands and addresses any issues that may have arisen with their previous formatting. It is important to note that no new methods have been added in this release, and the existing functionality of the commands has not been changed in scope or functionality. * Fixing the issue in workspace id flag in create-account-group command ([#1094](#1094)). In this update, we have improved the `create_account_group` command related to the `workspace_ids` flag in our open-source library. The `workspace_ids` flag's type has been changed from `list[int] | None` to `str | None`, allowing for easier input of multiple workspace IDs as a string of comma-separated integers. The `create_account_level_groups` function in the `AccountWorkspaces` class has been updated to accept this string and convert it to a list of integers before proceeding. To ensure proper functioning, we added a new test case `test_create_account_groups_with_id()` to check if the command handles the case when no workspace IDs are provided in the configuration. The `create_account_groups()` method now checks for this condition and raises a `ValueError`. Furthermore, the `manual_workspace_info()` method has been updated to handle workspace name input by the user, receiving the `ws` object, along with prompts that contain the user input for the workspace name and the next workspace ID. * Rely UCX on the latest 14.3 LTS DBR instead of 15.x ([#1097](#1097)). In this release, we have implemented a quick fix to rely on the Long Term Support (LTS) version 14.3 of the Databricks Runtime (DBR) instead of 15.x for UCX, addressing issue [#1096](#1096). This change affects the `_definition` function, which has been modified to use the latest LTS DBR instead of the latest Spark version. The `latest_lts_dbr` variable is now assigned the value returned by the `select_spark_version` method with the `latest=True` and `long_term_support=True` parameters. The `spark_version` key in the `policy_definition` dictionary is set to the value returned by the `_policy_config` method with `latest_lts_dbr` as the argument. Additionally, in the `tests/unit/installer/test_policy.py` file, the `select_spark_version` method of the `clusters` object has been updated to accept any number of arguments and consistently return the string "14.2.x-scala2.12", allowing for greater flexibility. This is a temporary solution, with a more comprehensive fix being tracked in issue [#1098](#1098). Developers should be aware of how the `clusters` object is used in the codebase when adopting this project.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Workspace Id was taj=king as string instead of list of int. Fixed the same by converting it int list of int using list comprehension.
Linked issues
#1090
Resolves #..
Functionality
databricks labs ucx .........Tests