Databricks: support column of type Text() as key

[JCZuurmond]

Is your feature request related to a problem? Please describe.

We have tables that contain key columns with the type Text(). We can not change these columns; we have to use the tables as is.

Describe the solution you'd like

I want to use columns with the type Text() as the key.

Describe alternatives you've considered

None. If this feature will/can not be supported, then I will see if can create views that convert the key columns to numeric columns.

Additional context

Running on Databricks

[github-actions]

This issue has been marked as stale because it has been open for 60 days with no activity. If you would like the issue to remain open, please comment on the issue and it will be added to the triage queue. Otherwise, it will be closed in 7 days.

[JCZuurmond]

👆

[dlawin]

Is your feature request related to a problem? Please describe.

We have tables that contain key columns with the type Text(). We can not change these columns; we have to use the tables as is.

Describe the solution you'd like

I want to use columns with the type Text() as the key.

Describe alternatives you've considered

None. If this feature will/can not be supported, then I will see if can create views that convert the key columns to numeric columns.

Additional context

Running on Databricks

Thanks for reporting this @JCZuurmond, we're not currently working on this but may in the future, in the meantime we'd happily accept a community contribution

[leoebfolsom]

A similar issue has cropped up related to Databricks Date PKs:

ERROR - Cannot use a column of type UnknownColType(text='DATE') as a key

[github-actions]

This issue has been marked as stale because it has been open for 60 days with no activity. If you would like the issue to remain open, please comment on the issue and it will be added to the triage queue. Otherwise, it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment and it will be reopened for triage.

[JCZuurmond]

👆

[ism21]

Running into the same issue.

[ferdyh]

Yes, same issue here... All columns we use have surrogate keys which are strings.

[dlawin]

Context on why arbitrary strings are not supported (without some work) in #539

#539 (comment)

[glebmezh]

Hi all -

I'm sorry for the delay in following up on this. @JCZuurmond, thank you for taking the time to raise this issue!

We made a hard decision to sunset the data-diff package and won't provide further development or support.

If that's of interest, over the past few months, we have rewritten the diffing engine in Datafold Cloud and solved many issues that existed in this package.

-Gleb