java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long

[roberto-rosero]

Hi, everybody.

At this moment I have a problem when I try execute a dbt model with a partitioned table source. I ran the first time the model very well, but from second time it fails with the error in the title of this discussion whe it try to do the merge into, I tried whatever do you imagine for solve it but I didnt. Do you any idea of this?

Conf:

materialized: incremental
incremental_strategy: merge
unique_key: id
full_refresh: false

Table is in format Iceberg and db Spark.

Thank you all.

[JCZuurmond]

@roberto-rosero : Do you know if it is failing on the id column or another one?

[roberto-rosero]

I think that the column with the problem is the partition, it is like '2021-03-12T09:00:00' but varchar data type.

Additional, the table has a column with data type "row", I navigate into the column with "." like this: column.field1, etc. I test removing all column with this pattern and the problem dissapear, I mean, I ran first:

select
event._id as id,
to_timestamp(event.datetime / 1000) as event_datetime,
event.segment,
event.sub_segment,
event.interaction_id,
event.interaction_type,
event.social_id,
to_json(event.variables) as variables,
event.version,
cast(datetime as date) as event_date,
current_timestamp as audit_datetime
from user_segmentation_import

it ran ok, the second time fails.

Then, I ran:

select
row_number() over(order by datetime) as id,
current_timestamp as audit_datetime
from user_segmentation_import

i ran ok all times, but if I add any field like event.social_id, it fails. Too weird.

[JCZuurmond]

@Fokko : What do recommend? Does Iceberg support schema evolution to solve this?

Otherwise, would @roberto-rosero explicit casting of the columns work for you? Maybe do that in your staging layer.

[roberto-rosero]

Do you mean cast all columns to varchar? I tried cast the column partition but occurs the same.

[JCZuurmond]

Not necessarily varchar, explicit cast to a type that makes sense for each column. But if you tried and it did not solve the issue then I dunno