Change the fields of the table "Tables" to lower case #684

HariGS-DB · 2023-12-07T22:26:02Z

Change:

Includes changes to tables.scala and tables.py to convert fields to lower case while writing
updated sql objects to remove upper and changing comparision to lower case
updated test cases to compare on lower cases
fixed issue in integration.conftest where data_source_format.value will return exception or views. updated to check for view then None

codecov · 2023-12-07T22:28:06Z

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (cbc7115) 81.37% compared to head (df89328) 81.38%.
Report is 1 commits behind head on main.

Files	Patch %	Lines
...databricks/labs/ucx/hive_metastore/data_objects.py	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #684      +/-   ##
==========================================
+ Coverage   81.37%   81.38%   +0.01%     
==========================================
  Files          37       37              
  Lines        3828     3831       +3     
  Branches      712      712              
==========================================
+ Hits         3115     3118       +3     
  Misses        555      555              
  Partials      158      158

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/databricks/labs/ucx/hive_metastore/tables.py

nfx · 2023-12-09T19:53:27Z

src/databricks/labs/ucx/hive_metastore/tables.scala

-val df = metadataForAllTables(spark.sharedState.externalCatalog.listDatabases(), failures)
+var df = metadataForAllTables(spark.sharedState.externalCatalog.listDatabases(), failures)
+var columnsToMap = df.columns
+columnsToMap.map(column => {


Why do we need this change?

this is required because in /queries/views/objects.sql there is join between tables and failures on catalog, database, name. I have updated the script to only lower case these 3 fields for the table_failures table

src/databricks/labs/ucx/hive_metastore/tables.py

src/databricks/labs/ucx/hive_metastore/tables.scala

src/databricks/labs/ucx/queries/assessment/main/01_3_all_tables.sql

src/databricks/labs/ucx/queries/assessment/main/02_0_database_summary.sql

Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>

src/databricks/labs/ucx/hive_metastore/tables.py

HariGS-DB · 2023-12-12T10:59:38Z

addressed the comments. Please check @nfx

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `workspace_group_name` and `account_group_name` to `make_ucx_group` fixture ([#664](#664)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added filters to run only cloud specific task ([#681](#681)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Cleanup README.md ([#695](#695)). * Cleanup installer framework and speed up test execution ([#711](#711)). * Decouple group configuration from `install.py` ([#714](#714)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed for secret scope apply task always raises ValueError ([#683](#683)). * Fixed some flaky integration tests ([#700](#700)). * More `mypy` chores ([#697](#697)). * Moved `ExternalLocations` and `Mounts` to `locations` module ([#692](#692)). * New CLI command for workspace mapping ([#678](#678)). * No more `mypy` chores after this ([#699](#699)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Some `mypy` chores ([#696](#696)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)). * Updated installation documentation to use Databricks CLI ([#686](#686)).

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `workspace_group_name` and `account_group_name` to `make_ucx_group` fixture ([#664](#664)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added filters to run only cloud specific task ([#681](#681)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Cleanup README.md ([#695](#695)). * Cleanup installer framework and speed up test execution ([#711](#711)). * Decouple group configuration from `install.py` ([#714](#714)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed for secret scope apply task always raises ValueError ([#683](#683)). * Fixed some flaky integration tests ([#700](#700)). * More `mypy` chores ([#697](#697)). * Moved `ExternalLocations` and `Mounts` to `locations` module ([#692](#692)). * New CLI command for workspace mapping ([#678](#678)). * No more `mypy` chores after this ([#699](#699)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Some `mypy` chores ([#696](#696)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated installation documentation to use Databricks CLI ([#686](#686)). Dependency updates: * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)).

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `databricks labs ucx validate-external-locations` command for cli ([#715](#715)). * Added `workspace_group_name` and `account_group_name` to `make_ucx_group` fixture ([#664](#664)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added filters to run only cloud specific task ([#681](#681)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Cleanup README.md ([#695](#695)). * Cleanup installer framework and speed up test execution ([#711](#711)). * Decouple group configuration from `install.py` ([#714](#714)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed for secret scope apply task always raises ValueError ([#683](#683)). * Fixed some flaky integration tests ([#700](#700)). * More `mypy` chores ([#697](#697)). * Moved `ExternalLocations` and `Mounts` to `locations` module ([#692](#692)). * New CLI command for workspace mapping ([#678](#678)). * No more `mypy` chores after this ([#699](#699)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Some `mypy` chores ([#696](#696)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated installation documentation to use Databricks CLI ([#686](#686)). Dependency updates: * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)).

* Added `databricks labs ucx create-table-mapping` and `databricks labs ucx manual-workspace-info` commands for CLI ([#682](#682)). * Added `databricks labs ucx ensure-assessment-run` to CLI commands ([#708](#708)). * Added `databricks labs ucx installations` command ([#679](#679)). * Added `databricks labs ucx skip --schema ... --table ...` command to mark table/schema for skipping in the table migration process ([#680](#680)). * Added `databricks labs ucx validate-external-locations` command for cli ([#715](#715)). * Added capturing `ANY FILE` and `ANONYMOUS FUNCTION` grants ([#653](#653)). * Added cluster override and handle case of write protected DBFS ([#610](#610)). * Added cluster policy selector in the installer ([#655](#655)). * Added detailed UCX pre-requisite information to README.md ([#689](#689)). * Added interactive wizard for `databricks labs uninstall ucx` command ([#657](#657)). * Added more granular error retry logic ([#704](#704)). * Added parallel fetching of registered model identifiers to speed-up assessment workflow ([#691](#691)). * Added retry on workspace listing ([#659](#659)). * Added support for mapping workspace group to account group by prefix/suffix/regex/external id ([#650](#650)). * Changed cluster security mode from NONE to LEGACY_SINGLE_USER, as `crawl_tables` was failing when run on non-UC Workspace in No Isolation mode with unable to access the config file ([#661](#661)). * Changed the fields of the table "Tables" to lower case ([#684](#684)). * Enabled integration tests for `EXTERNAL` table migrations ([#677](#677)). * Enforced `mypy` validation ([#713](#713)). * Filtering out inventory database from loading into tables and filtering out the same from grant detail view ([#705](#705)). * Fixed documentation for `reflect_account_groups_on_workspace` task and updated `CONTRIBUTING.md` guide ([#654](#654)). * Fixed secret scope apply task to raise ValueError ([#683](#683)). * Fixed legacy table ACL ownership migration and other integration testing issues ([#722](#722)). * Fixed some flaky integration tests ([#700](#700)). * New CLI command for workspace mapping ([#678](#678)). * Reduce server load for getting workspace groups and their members ([#666](#666)). * Throwing ManyError on migrate-groups tasks ([#710](#710)). * Updated installation documentation to use Databricks CLI ([#686](#686)). Dependency updates: * Updated databricks-sdk requirement from ~=0.13.0 to ~=0.14.0 ([#651](#651)). * Updated databricks-sdk requirement from ~=0.14.0 to ~=0.15.0 ([#687](#687)). * Updated databricks-sdk requirement from ~=0.15.0 to ~=0.16.0 ([#712](#712)).

HariGS-DB added 3 commits December 7, 2023 13:35

lower casing tables columns

fe03198

lower casing queries

5b4d386

fixes to casing

bd9c9f9

HariGS-DB requested review from a team and pritishpai December 7, 2023 22:26

HariGS-DB had a problem deploying to account-admin December 7, 2023 22:26 — with GitHub Actions Failure

HariGS-DB requested review from nfx and removed request for pritishpai December 7, 2023 22:26

nfx requested changes Dec 9, 2023

View reviewed changes

Update src/databricks/labs/ucx/hive_metastore/tables.py

9b06c49

Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>

HariGS-DB had a problem deploying to account-admin December 11, 2023 14:56 — with GitHub Actions Failure

Update src/databricks/labs/ucx/hive_metastore/tables.py

441864f

Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>

HariGS-DB had a problem deploying to account-admin December 11, 2023 14:56 — with GitHub Actions Failure

Update src/databricks/labs/ucx/hive_metastore/tables.py

548f4d3

Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>

HariGS-DB had a problem deploying to account-admin December 11, 2023 14:57 — with GitHub Actions Failure

fixes to casing

f651a56

HariGS-DB had a problem deploying to account-admin December 11, 2023 16:44 — with GitHub Actions Failure

Merge branch 'main' into feature/updatecase

398df2b

HariGS-DB had a problem deploying to account-admin December 11, 2023 21:13 — with GitHub Actions Failure

nfx requested changes Dec 11, 2023

View reviewed changes

src/databricks/labs/ucx/hive_metastore/tables.py Outdated Show resolved Hide resolved

src/databricks/labs/ucx/hive_metastore/tables.py Outdated Show resolved Hide resolved

fixes to review comments

df89328

HariGS-DB had a problem deploying to account-admin December 12, 2023 07:58 — with GitHub Actions Failure

nfx approved these changes Dec 12, 2023

View reviewed changes

nfx merged commit f3d1d99 into main Dec 12, 2023
5 of 7 checks passed

nfx deleted the feature/updatecase branch December 12, 2023 17:16

pritishpai pushed a commit that referenced this pull request Dec 12, 2023

Change the fields of the table "Tables" to lower case (#684)

c663876

HariGS-DB linked an issue Dec 14, 2023 that may be closed by this pull request

Ensure fixed casing for object names in inventory tables. #288

Closed

nfx mentioned this pull request Dec 21, 2023

Release v0.7.0 #719

Closed

nfx mentioned this pull request Dec 21, 2023

Release v0.7.0 #720

Closed

nfx mentioned this pull request Dec 21, 2023

Release v0.7.0 #723

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change the fields of the table "Tables" to lower case #684

Change the fields of the table "Tables" to lower case #684

HariGS-DB commented Dec 7, 2023

codecov bot commented Dec 7, 2023 •

edited

Loading

nfx Dec 9, 2023

HariGS-DB Dec 11, 2023

HariGS-DB commented Dec 12, 2023

Change the fields of the table "Tables" to lower case #684

Change the fields of the table "Tables" to lower case #684

Conversation

HariGS-DB commented Dec 7, 2023

codecov bot commented Dec 7, 2023 • edited Loading

Codecov Report

nfx Dec 9, 2023

Choose a reason for hiding this comment

HariGS-DB Dec 11, 2023

Choose a reason for hiding this comment

HariGS-DB commented Dec 12, 2023

codecov bot commented Dec 7, 2023 •

edited

Loading