Added documentation to describe interaction with external Hive Metastores #473

FastLee · 2023-10-18T13:15:02Z

Per our discussion.
Analysis of External HMS integration.

codecov · 2023-10-18T13:18:44Z

Codecov Report

Merging #473 (5ed453f) into main (ac7d30e) will increase coverage by 0.01%.
Report is 17 commits behind head on main.
The diff coverage is n/a.

❗ Current head 5ed453f differs from pull request most recent head 7b332c4. Consider uploading reports for the commit 7b332c4 to get more accurate results

@@            Coverage Diff             @@
##             main     #473      +/-   ##
==========================================
+ Coverage   80.01%   80.02%   +0.01%     
==========================================
  Files          31       31              
  Lines        3172     3174       +2     
  Branches      609      609              
==========================================
+ Hits         2538     2540       +2     
  Misses        486      486              
  Partials      148      148

see 1 file with indirect coverage changes

nfx · 2023-10-18T19:34:40Z

docs/external_hms.md

+
+### Current Considerations
+- Integration with external HMS is set up on individual clusters.
+- Theoretically we can integrate separate clusters in a workspace with different HMS repositories.


what do we need for it practically? please describe

nfx · 2023-10-18T19:37:04Z

docs/external_hms.md

+- Theoretically we can integrate separate clusters in a workspace with different HMS repositories.
+- In reality most customers use a single (internal or external) HMS within a workspace.
+- When migrating from an external HMS we have to consider that it is used by more than one workspace.
+- Integration with external HMS has to be set on all DB Warehouses together.


can Ext-HMS WH access workspace-local HMS?

docs/external_hms.md

nfx · 2023-10-18T19:38:58Z

docs/external_hms.md

+- We should allow overriding cluster settings and instance profile setting to accommodate novel settings.
+
+### Challenges
+- We cannot support multiple HMS


but we can store raw delta tables on DBFS

nfx · 2023-10-18T19:39:42Z

docs/external_hms.md

+
+### Challenges
+- We cannot support multiple HMS
+- Using an external HMS to persist UCX tables will break functionality for a second workspace using UCX


please suggest couple of options on addressing this

For the inventory tables, if we use the Workspace ID in either the schema name "ucx-################" or as a column for uniqueness, then we don't have to worry about conflicts.

Likely easier to introduce in the schema name since our teardown scripts drop tables.

nfx · 2023-10-19T10:30:51Z

docs/external_hms.md

+### Design Decisions
+- We should set up a single HMS for UCX
+- We should suggest copying the setup from an existing Cluster/Cluster policy
+- We shouldn't override the set up for the DB Warehouses (that may break functionality)


This requirement, not a design decision

nfx · 2023-10-19T10:31:20Z

docs/external_hms.md

+- We should set up a single HMS for UCX
+- We should suggest copying the setup from an existing Cluster/Cluster policy
+- We shouldn't override the set up for the DB Warehouses (that may break functionality)
+- We should allow overriding cluster settings and instance profile setting to accommodate novel settings.


This is requirement, not a design decision

nfx · 2023-10-19T10:32:46Z

docs/external_hms.md

+1. Start the installer.
+2. The installer looks for use of external HMS by the workspace. We review cluster policies or DBSQL warehouses settings.
+3. We alert the user that an external HMS is set and request ask a YES/NO to set external HMS.
+4. We alert the user if they opted for external HMS and the DB Warehouses are not set for external HMS


Should be step3, not 4 :)

Is the concern that a workspace may be connected to an external metatore on standard clusters, but the SQL Warehouse is not? If so, is this common?

HariGS-DB · 2023-10-19T10:55:11Z

docs/external_hms.md

+- We should allow overriding cluster settings and instance profile setting to accommodate novel settings.
+
+### Challenges
+- We cannot support multiple HMS


Could we not set up one job task for each unique hms. Usually workspace dont use different HMS, but they likely to have one Ext HMS and one local HMS.

HariGS-DB · 2023-10-19T11:13:39Z

docs/external_hms.md

+### Challenges
+- We cannot support multiple HMS
+- Using an external HMS to persist UCX tables will break functionality for a second workspace using UCX
+- We should consider using a pattern similar to our integration testing to rename the target database to allow persisting from multiple workspaces. For example WS1 --> UCX_ABC, WS2 --> UCX_DEF.


If multiple workspaces are sharing the same Ext HMS, we should do table crawl just once but should do multiple grants crawl per each workspace

changing the code to only crawl the external HMS once is a premature optimization. This operation only takes a few minutes right now.

**Breaking changes** (existing installations need to reinstall UCX and re-run assessment jobs) * Switched local group migration component to rename groups instead of creating backup groups ([#450](#450)). * Mitigate permissions loss in Table ACLs by folding grants belonging to the same principal, object id and object type together ([#512](#512)). **New features** * Added support for the experimental Databricks CLI launcher ([#517](#517)). * Added support for external Hive Metastores including AWS Glue ([#400](#400)). * Added more views to assessment dashboard ([#474](#474)). * Added rate limit for creating backup group to increase stability ([#500](#500)). * Added deduplication for mount point list ([#569](#569)). * Added documentation to describe interaction with external Hive Metastores ([#473](#473)). * Added failure injection for job failure message propagation ([#591](#591)). * Added uniqueness in the new warehouse name to avoid conflicts on installation ([#542](#542)). * Added a global init script to collect Hive Metastore lineage ([#513](#513)). * Added retry set/update permissions when possible and assess the changes in the workspace ([#519](#519)). * Use `~/.ucx/state.json` to store the state of both dashboards and jobs ([#561](#561)). **Bug fixes** * Fixed handling for `OWN` table permissions ([#571](#571)). * Fixed handling of keys with and without values. ([#514](#514)). * Fixed integration test failures related to concurrent group delete ([#584](#584)). * Fixed issue with workspace listing process on None type `object_type` ([#481](#481)). * Fixed missing group entitlement migration bug ([#583](#583)). * Fixed entitlement application for account-level groups ([#529](#529)). * Fixed assessment throwing an error when the owner of an object is empty ([#485](#485)). * Fixed installer to migrate between different configuration file versions ([#596](#596)). * Fixed cluster policy crawler to be aware of deleted policies ([#486](#486)). * Improved error message for not null constraints violated ([#532](#532)). * Improved integration test resiliency ([#597](#597), [#594](#594), [#586](#586)). * Introduced Safer access to workspace objects' properties. ([#530](#530)). * Mitigated permissions loss in Table ACLs by running appliers with single thread ([#518](#518)). * Running apply permission task before assessment should display message ([#487](#487)). * Split integration tests from blocking the merge queue ([#496](#496)). * Support more than one dashboard per step ([#472](#472)). * Update databricks-sdk requirement from ~=0.11.0 to ~=0.12.0 ([#505](#505)). * Update databricks-sdk requirement from ~=0.12.0 to ~=0.13.0 ([#575](#575)).

Created External HMS Doc.

18bf58e

FastLee had a problem deploying to account-admin October 18, 2023 13:15 — with GitHub Actions Failure

Created External Location Design

6bd457f

FastLee had a problem deploying to account-admin October 18, 2023 13:17 — with GitHub Actions Failure

FastLee marked this pull request as ready for review October 18, 2023 13:17

FastLee requested a review from a team October 18, 2023 13:17

nfx requested changes Oct 18, 2023

View reviewed changes

Added some context

5ed453f

FastLee had a problem deploying to account-admin October 19, 2023 09:41 — with GitHub Actions Failure

nfx requested changes Oct 19, 2023

View reviewed changes

HariGS-DB reviewed Oct 19, 2023

View reviewed changes

Update external_hms.md

2368fc8

nfx had a problem deploying to account-admin October 28, 2023 14:27 — with GitHub Actions Failure

Update external_hms.md

ee91dbe

nfx had a problem deploying to account-admin October 28, 2023 14:27 — with GitHub Actions Failure

Update external_hms.md

7b332c4

nfx had a problem deploying to account-admin October 28, 2023 14:28 — with GitHub Actions Failure

nfx changed the title ~~Created External HMS Doc~~ Added documentation to describe interaction with external Hive Metastores Oct 28, 2023

nfx merged commit f0c61a4 into main Oct 28, 2023
3 of 4 checks passed

nfx deleted the feature/ext_hms_integration_doc branch October 28, 2023 14:29

nfx mentioned this pull request Nov 17, 2023

Release v0.6.0 #598

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added documentation to describe interaction with external Hive Metastores #473

Added documentation to describe interaction with external Hive Metastores #473

FastLee commented Oct 18, 2023

codecov bot commented Oct 18, 2023 •

edited

Loading

nfx Oct 18, 2023

nfx Oct 18, 2023

nfx Oct 18, 2023

nfx Oct 18, 2023

pohlposition Oct 28, 2023

nfx Oct 19, 2023

nfx Oct 19, 2023

nfx Oct 19, 2023

pohlposition Oct 28, 2023

HariGS-DB Oct 19, 2023

HariGS-DB Oct 19, 2023

pohlposition Oct 28, 2023

Added documentation to describe interaction with external Hive Metastores #473

Added documentation to describe interaction with external Hive Metastores #473

Conversation

FastLee commented Oct 18, 2023

codecov bot commented Oct 18, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 18, 2023 •

edited

Loading