Added column `is_partitioned` to ucx.tables and dashboard #959

dleiva04 · 2024-02-19T21:00:26Z

Changes

Linked issues

Resolves #871

Related PR:

Automatically upgrade existing installations to avoid breaking changes #985

Functionality

added relevant user documentation
added new CLI command
modified existing command: databricks labs ucx ...
added a new workflow
modified existing workflow: ...
added a new table
modified existing table: ucx.tables

Tests

manually tested
added unit tests
added integration tests
verified on staging environment (screenshot attached)

CLAassistant · 2024-02-19T21:00:31Z

All committers have signed the CLA.

nfx

Please add integration test and more comments.

How this partitioning information should affect the migration process?

src/databricks/labs/ucx/hive_metastore/tables.scala

src/databricks/labs/ucx/queries/assessment/main/05_0_all_tables.sql

src/databricks/labs/ucx/hive_metastore/tables.scala

codecov · 2024-02-20T08:09:30Z

Codecov Report

Attention: Patch coverage is 75.00000% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 88.56%. Comparing base (41309a2) to head (732c119).

❗ Current head 732c119 differs from pull request most recent head fd529a1. Consider uploading reports for the commit fd529a1 to get more accurate results

Files	Patch %	Lines
src/databricks/labs/ucx/install.py	73.68%	1 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #959      +/-   ##
==========================================
+ Coverage   88.45%   88.56%   +0.10%     
==========================================
  Files          47       47              
  Lines        6211     6103     -108     
  Branches     1116     1094      -22     
==========================================
- Hits         5494     5405      -89     
+ Misses        477      463      -14     
+ Partials      240      235       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

gitguardian · 2024-03-04T22:55:31Z

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
Once a secret has been leaked into a git repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}}

nfx

This is a breaking change, add the upgrade script

tests/integration/test_installation.py

…itioned

src/databricks/labs/ucx/install.py

src/databricks/labs/ucx/mixins/sql.py

src/databricks/labs/ucx/upgrades/v0.15.0_add_is_partitioned_column.py

src/databricks/labs/ucx/mixins/sql.py

nfx

Remove security holes

src/databricks/labs/ucx/mixins/sql.py

nfx

Remove irrelevant changes

nfx · 2024-03-07T08:45:20Z

src/databricks/labs/ucx/install.py

@@ -476,20 +476,28 @@ def run_workflow(self, step: str):
        except OperationFailed as err:
            # currently we don't have any good message from API, so we have to work around it.
            job_run = self._ws.jobs.get_run(job_run_waiter.run_id)
-            raise self._infer_error_from_job_run(job_run) from err
+            raise self._infer_nested_error(job_run) from err


Properly fetch main branch and rebase. This file is irrelevant to this PR.

nfx · 2024-03-07T08:45:45Z

src/databricks/labs/ucx/mixins/sql.py

@@ -264,10 +265,17 @@ def _row_converters(self, execute_response):
            type_name = col.type_name
            if not type_name:
                type_name = ColumnInfoTypeName.NULL
-            conv = self.type_converters.get(type_name, None)
+            if type_name == ColumnInfoTypeName.BOOLEAN:


This is irrelevant

nfx · 2024-03-07T08:45:54Z

src/databricks/labs/ucx/mixins/sql.py

            if conv is None:
                msg = f"{col.name} has no {type_name.value} converter"
                raise ValueError(msg)
            col_conv.append(conv)
        row_factory = type("Row", (Row,), {"__columns__": col_names})
        return col_conv, row_factory
+
+    @staticmethod
+    def _convert_boolean_type(value):


nfx · 2024-03-07T10:05:49Z

src/databricks/labs/ucx/upgrades/v0.15.0_add_is_partitioned_column.py

@@ -0,0 +1,12 @@
+from databricks.labs.blueprint.installation import Installation


change name to v0.16.0

dleiva04 added 2 commits February 19, 2024 14:48

Added colum is_partitioned inside ucx.tables

6c44c29

Added comment inside tables.scala for explanation

1ce46b2

dleiva04 requested review from a team and dipankarkush-db February 19, 2024 21:00

nfx requested changes Feb 20, 2024

View reviewed changes

dleiva04 had a problem deploying to account-admin February 20, 2024 08:08 — with GitHub Actions Failure

nfx removed the request for review from dipankarkush-db February 20, 2024 08:14

nfx added pr/breaking-change this change does require data or configuration migration from previous versions step/assessment go/uc/upgrade - Assessment Step labels Mar 4, 2024

dleiva04 added 3 commits March 4, 2024 16:26

Integration test WIP

e00434f

remove comments

7504924

Merge branch 'databrickslabs:main' into is_partitioned

f6b9ad9

nfx requested changes Mar 5, 2024

View reviewed changes

tests/integration/test_installation.py Outdated Show resolved Hide resolved

dleiva04 added 8 commits March 5, 2024 11:05

updates to comply with fmt and lint

a1b20b8

Added colum is_partitioned inside ucx.tables

26f7e4f

Added comment inside tables.scala for explanation

38d4194

Integration test WIP

34a9544

remove comments

9b6e0c7

updates to comply with fmt and lint

4df3a83

adding upgrade script

c197d98

Merge branch 'is_partitioned' of github.com:dleiva04/ucx into is_part…

83b515c

…itioned

dleiva04 requested a review from nfx March 6, 2024 00:55

dleiva04 added 2 commits March 5, 2024 18:55

Merge branch 'main' into is_partitioned

50d44c7

Merge branch 'main' into is_partitioned

7c9b4dc

nfx requested changes Mar 6, 2024

View reviewed changes

dleiva04 added 2 commits March 6, 2024 12:57

Added colum is_partitioned inside ucx.tables

0cf8263

Added comment inside tables.scala for explanation

94c71da

dleiva04 added 10 commits March 6, 2024 12:57

adding upgrade script

48d082e

Added comment inside tables.scala for explanation

4f2b0d5

Integration test WIP

f44d17f

updates to comply with fmt and lint

0120412

Update

323ac98

change lambda variable name

2e987d0

Added comment inside tables.scala for explanation

e7fc88c

Integration test WIP

b22311a

updates to comply with fmt and lint

809d3d5

update

e2093ff

dleiva04 commented Mar 6, 2024

View reviewed changes

src/databricks/labs/ucx/mixins/sql.py Outdated Show resolved Hide resolved

nfx requested changes Mar 6, 2024

View reviewed changes

src/databricks/labs/ucx/mixins/sql.py Show resolved Hide resolved

dleiva04 had a problem deploying to account-admin March 6, 2024 20:18 — with GitHub Actions Failure

update

732c119

dleiva04 requested a review from nfx March 6, 2024 20:38

nfx approved these changes Mar 6, 2024

View reviewed changes

dleiva04 had a problem deploying to account-admin March 6, 2024 20:46 — with GitHub Actions Failure

dleiva04 added 9 commits March 7, 2024 01:15

update

c15114c

update

3bfb7f6

update

7f4e5d8

Merge branch 'main' into is_partitioned

0832d51

updates to comply with fmt and lint

c922a87

adding upgrade script

33bc922

update

d74df76

update

f850692

updates to comply with fmt and lint

8955c55

nfx requested changes Mar 7, 2024

View reviewed changes

Merge branch 'main' into is_partitioned

fd529a1

nfx reviewed Mar 7, 2024

View reviewed changes

dleiva04 closed this by deleting the head repository Mar 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added column `is_partitioned` to ucx.tables and dashboard #959

Added column `is_partitioned` to ucx.tables and dashboard #959

dleiva04 commented Feb 19, 2024 •

edited by nfx

Loading

CLAassistant commented Feb 19, 2024 •

edited

Loading

nfx left a comment

codecov bot commented Feb 20, 2024 •

edited

Loading

gitguardian bot commented Mar 4, 2024 •

edited

Loading

nfx left a comment

nfx left a comment

nfx left a comment

nfx Mar 7, 2024

nfx Mar 7, 2024

nfx Mar 7, 2024

nfx Mar 7, 2024

		@@ -0,0 +1,12 @@
		from databricks.labs.blueprint.installation import Installation

Added column is_partitioned to ucx.tables and dashboard #959

Added column is_partitioned to ucx.tables and dashboard #959

Conversation

dleiva04 commented Feb 19, 2024 • edited by nfx Loading

Changes

Linked issues

Functionality

Tests

CLAassistant commented Feb 19, 2024 • edited Loading

nfx left a comment

Choose a reason for hiding this comment

codecov bot commented Feb 20, 2024 • edited Loading

Codecov Report

gitguardian bot commented Mar 4, 2024 • edited Loading

️✅ There are no secrets present in this pull request anymore.

nfx left a comment

Choose a reason for hiding this comment

nfx left a comment

Choose a reason for hiding this comment

nfx left a comment

Choose a reason for hiding this comment

nfx Mar 7, 2024

Choose a reason for hiding this comment

nfx Mar 7, 2024

Choose a reason for hiding this comment

nfx Mar 7, 2024

Choose a reason for hiding this comment

nfx Mar 7, 2024

Choose a reason for hiding this comment

Added column `is_partitioned` to ucx.tables and dashboard #959

Added column `is_partitioned` to ucx.tables and dashboard #959

dleiva04 commented Feb 19, 2024 •

edited by nfx

Loading

CLAassistant commented Feb 19, 2024 •

edited

Loading

codecov bot commented Feb 20, 2024 •

edited

Loading

gitguardian bot commented Mar 4, 2024 •

edited

Loading