Fix issue with missing users and temp groups after workspace-local groups migration and skip table when crawling table size if it does not exist anymore #770

mwojtyczka · 2024-01-10T15:57:25Z

Fix for issue: 767
Added new unit and integration tests, improved existing assertions
Improve get_table_size crawling to skip table if it does not exist anymore, instead of failing the task. This increases the stability of assessment job and integration tests.

codecov · 2024-01-10T15:59:22Z

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (54c07dc) 81.97% compared to head (4ba2676) 82.62%.

Files	Patch %	Lines
src/databricks/labs/ucx/workspace_access/redash.py	97.43%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #770      +/-   ##
==========================================
+ Coverage   81.97%   82.62%   +0.64%     
==========================================
  Files          39       39              
  Lines        4516     4564      +48     
  Branches      835      848      +13     
==========================================
+ Hits         3702     3771      +69     
+ Misses        605      590      -15     
+ Partials      209      203       -6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nfx

these integration tests are very brittle in the current shape and I'm not comfortable merging it in current state.

tests/integration/workspace_access/test_redash.py

tests/integration/test_installation.py

tests/integration/workspace_access/test_redash.py

src/databricks/labs/ucx/mixins/fixtures.py

tests/integration/workspace_access/test_redash.py

* Added assessment step to estimate the size of DBFS root tables ([#741](#741)). * Added `TableMapping` functionality to table migrate ([#752](#752)). * Added `databricks labs ucx move` command to move tables and schemas between catalogs ([#756](#756)). * Added functionality to determine migration method based on DBFS Root ([#759](#759)). * Added `get_tables_to_migrate` functionality in the mapping module ([#755](#755)). * Added retry and rate limit to rename workspace group operation and corrected rate limit for reflecting account groups to workspace ([#751](#751)). * Adopted `databricks-labs-blueprint` library for common utilities to be reused in the other projects ([#758](#758)). * Converted `RuntimeBackend` query executions exceptions to SDK exceptions ([#769](#769)). * Fixed issue with missing users and temp groups after workspace-local groups migration and skip table when crawling table size if it does not exist anymore ([#770](#770)). * Improved error handling by not failing group rename step if a group was removed from account before reflecting it to workspace ([#762](#762)). * Improved error message inference from failed workflow runs ([#753](#753)). * Moved `TablesMigrate` to a separate module ([#747](#747)). * Reorganized assessment dashboard to increase readability ([#738](#738)). * Updated databricks-sdk requirement from ~=0.16.0 to ~=0.17.0 ([#773](#773)). * Verify metastore exists in current workspace ([#735](#735)).

…oups migration and skip table when crawling table size if it does not exist anymore (#770)

* Added assessment step to estimate the size of DBFS root tables ([#741](#741)). * Added `TableMapping` functionality to table migrate ([#752](#752)). * Added `databricks labs ucx move` command to move tables and schemas between catalogs ([#756](#756)). * Added functionality to determine migration method based on DBFS Root ([#759](#759)). * Added `get_tables_to_migrate` functionality in the mapping module ([#755](#755)). * Added retry and rate limit to rename workspace group operation and corrected rate limit for reflecting account groups to workspace ([#751](#751)). * Adopted `databricks-labs-blueprint` library for common utilities to be reused in the other projects ([#758](#758)). * Converted `RuntimeBackend` query executions exceptions to SDK exceptions ([#769](#769)). * Fixed issue with missing users and temp groups after workspace-local groups migration and skip table when crawling table size if it does not exist anymore ([#770](#770)). * Improved error handling by not failing group rename step if a group was removed from account before reflecting it to workspace ([#762](#762)). * Improved error message inference from failed workflow runs ([#753](#753)). * Moved `TablesMigrate` to a separate module ([#747](#747)). * Reorganized assessment dashboard to increase readability ([#738](#738)). * Updated databricks-sdk requirement from ~=0.16.0 to ~=0.17.0 ([#773](#773)). * Verify metastore exists in current workspace ([#735](#735)).

Preserving users and temp groups permissions during migration.

63555e2

mwojtyczka self-assigned this Jan 10, 2024

mwojtyczka requested review from a team and larsgeorge-db January 10, 2024 15:57

mwojtyczka had a problem deploying to account-admin January 10, 2024 15:57 — with GitHub Actions Failure

mwojtyczka removed the request for review from larsgeorge-db January 10, 2024 15:57

mwojtyczka linked an issue Jan 10, 2024 that may be closed by this pull request

User and backup group permissions lost for redash objects after applying group permissions #767

Closed

mwojtyczka changed the title ~~Fixing issue with missing users and temp groups after groups migration~~ Fixing issue with missing users and temp groups after workspace-local groups migration Jan 10, 2024

added unit tests

ce21ef2

mwojtyczka had a problem deploying to account-admin January 10, 2024 16:56 — with GitHub Actions Failure

updated int tests

3ad1d9c

mwojtyczka had a problem deploying to account-admin January 10, 2024 17:13 — with GitHub Actions Failure

fmt

3719484

mwojtyczka had a problem deploying to account-admin January 10, 2024 17:17 — with GitHub Actions Failure

mwojtyczka changed the title ~~Fixing issue with missing users and temp groups after workspace-local groups migration~~ Fix issue with missing users and temp groups after workspace-local groups migration Jan 10, 2024

update timeout

199b1b6

mwojtyczka had a problem deploying to account-admin January 10, 2024 18:19 — with GitHub Actions Failure

update timeout

87d9d90

mwojtyczka had a problem deploying to account-admin January 10, 2024 18:43 — with GitHub Actions Failure

mwojtyczka had a problem deploying to account-admin January 10, 2024 19:14 — with GitHub Actions Failure

check table or view not found exception

82d22ca

mwojtyczka temporarily deployed to account-admin January 10, 2024 19:34 — with GitHub Actions Inactive

added unit tests

b8b21c8

mwojtyczka had a problem deploying to account-admin January 11, 2024 08:24 — with GitHub Actions Failure

refactored table size

5b7ca44

mwojtyczka had a problem deploying to account-admin January 11, 2024 08:30 — with GitHub Actions Failure

use warning for table not found to avoid failing the task

3af1f11

mwojtyczka had a problem deploying to account-admin January 11, 2024 09:04 — with GitHub Actions Failure

refactored table size

41a2d42

updated timeout

84c22d5

mwojtyczka had a problem deploying to account-admin January 11, 2024 14:43 — with GitHub Actions Failure

updated timeout

ee64ce4

mwojtyczka had a problem deploying to account-admin January 11, 2024 15:18 — with GitHub Actions Failure

mwojtyczka temporarily deployed to account-admin January 11, 2024 15:43 — with GitHub Actions Inactive

nfx requested changes Jan 11, 2024

View reviewed changes

Merge branch 'main' into fix/redash_permissions

1049cf2

mwojtyczka temporarily deployed to account-admin January 11, 2024 17:26 — with GitHub Actions Inactive

implemented code review feedback

1e42ae1

mwojtyczka had a problem deploying to account-admin January 11, 2024 18:07 — with GitHub Actions Failure

fmt

f338241

mwojtyczka temporarily deployed to account-admin January 11, 2024 18:47 — with GitHub Actions Inactive

implemented code review feedback

13e6fb1

mwojtyczka had a problem deploying to account-admin January 11, 2024 21:27 — with GitHub Actions Failure

mwojtyczka requested a review from nfx January 11, 2024 21:29

mwojtyczka temporarily deployed to account-admin January 11, 2024 22:15 — with GitHub Actions Inactive

nfx requested changes Jan 11, 2024

View reviewed changes

implemented code review feedback

edeade0

mwojtyczka requested a review from nfx January 12, 2024 08:23

mwojtyczka temporarily deployed to account-admin January 12, 2024 08:23 — with GitHub Actions Inactive

removed unused directive

4ba2676

mwojtyczka temporarily deployed to account-admin January 12, 2024 08:25 — with GitHub Actions Inactive

nfx approved these changes Jan 12, 2024

View reviewed changes

nfx merged commit 1e9c991 into main Jan 12, 2024
7 checks passed

nfx deleted the fix/redash_permissions branch January 12, 2024 09:48

nfx mentioned this pull request Jan 12, 2024

Release v0.9.0 #776

Merged

FastLee pushed a commit that referenced this pull request Jan 19, 2024

Fix issue with missing users and temp groups after workspace-local gr…

dd70e42

…oups migration and skip table when crawling table size if it does not exist anymore (#770)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix issue with missing users and temp groups after workspace-local groups migration and skip table when crawling table size if it does not exist anymore #770

Fix issue with missing users and temp groups after workspace-local groups migration and skip table when crawling table size if it does not exist anymore #770

mwojtyczka commented Jan 10, 2024 •

edited

Loading

codecov bot commented Jan 10, 2024 •

edited

Loading

nfx left a comment

Fix issue with missing users and temp groups after workspace-local groups migration and skip table when crawling table size if it does not exist anymore #770

Fix issue with missing users and temp groups after workspace-local groups migration and skip table when crawling table size if it does not exist anymore #770

Conversation

mwojtyczka commented Jan 10, 2024 • edited Loading

codecov bot commented Jan 10, 2024 • edited Loading

Codecov Report

nfx left a comment

Choose a reason for hiding this comment

mwojtyczka commented Jan 10, 2024 •

edited

Loading

codecov bot commented Jan 10, 2024 •

edited

Loading