-
Notifications
You must be signed in to change notification settings - Fork 591
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor(postgres-cdc): refactor postgres_row_to_owned_row #16714
Conversation
Before this PR,
The correct way to handle NULL::TYPE[] is to check if the value is None before creating the array builder.
|
An open question: Do you think we should also add |
After refactoring, multi-dismension list should trivial to support. Will add it in next PR. During refactoring, I find an interesting thing: while testing, I setup a postgres as data source and I run the test multiple times. The second round of test for the postgres will usually fail but the first and following tests will succeed🤣. Will continue to investigate. |
https://docs.rs/postgres-types/latest/postgres_types/trait.FromSql.html#arrays
Related: #3811 (comment) #15614 |
During this refactoring, I found three another bugs of Debezium CDC. #16895 BTW, besides these three, I suspect Debezium may not support "inf/-inf/nan" for some types. Will check later. |
(ScalarRefImpl::List(list), &Type::NUMERIC_ARRAY, _) => { | ||
let mut vec = vec![]; | ||
for scalar in list.iter() { | ||
vec.push(match scalar { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The scalar
should rename to datum
, because Datum
is Option<ScalarImpl>
. And I am curious why not vec.push( datum.map(|scalar| ScalarAdapter::from_scalar(scalar, &Type::NUMERIC) ) )
to call the from_scalar
recursively? Just add one more branch to handle (ScalarRefImpl::Decimal, &Type::NUMERIC)
as base case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The scalar should rename to datum, because Datum is Option
updated.
And I am curious why not vec.push( datum.map(|scalar| ScalarAdapter::from_scalar(scalar, &Type::NUMERIC) ) ) to call the from_scalar recursively? Just add one more branch to handle (ScalarRefImpl::Decimal, &Type::NUMERIC) as base case.
For two reason:
from_scalar
returnScalarAdapter
directly, butScalarAdapter::NumericList
requiresVec<Option<PgNumeric>>
actually.- If we don't handle
(ScalarRefImpl::Decimal, &Type::NUMERIC)
,ScalarRefImpl::Decimal
will bebuiltin
. We have implementToSql
for rw'sDecimal
, so it acceptale by PG. If we handle(ScalarRefImpl::Decimal, &Type::NUMERIC)
, we have to first convert it to string and then parse asPgNumeric
, seems unnecessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we handle (ScalarRefImpl::Decimal, &Type::NUMERIC), we have to first convert it to string and then parse as PgNumeric, seems unnecessary.
Your are right, I got it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Date, Interval, JsonbVal, ScalarImpl, Time, Timestamp, Timestamptz, | ||
}; | ||
|
||
impl<'a> FromSql<'a> for ScalarImpl { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These FromSql
implementations LGTM, @xiangjinwu please also take a look
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the efforts to test all these tricky cases and help to get them organized!
Just a reminder to all of us: rather than going too far away to be bug-compatible with debezium, we may:
- Disallow ingesting such data type at all, for example
interval[]
- Spend more time on fixing debezium. IIRC there is dark magic in java class loader to replace a certain library class without forking the project
|
||
impl ToSql for ScalarRefImpl<'_> { | ||
impl ToSql for ScalarImpl { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we change from ScalarRefImpl
to ScalarImpl
? The former is always more general and there is as_scalar_ref
. (That is, to
for a ref type and from
for an owned type.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Most of our types that implement FromSql
are scalar
types instead of scalar_ref
. So it's easier to implement FromSql
for ScalarImpl
then ScalarRefImpl
. Besides, as the postgres_cell_to_scalar_impl
's return type is ScalarImpl
, ScalarImpl
would be more natural.
ScalarAdapter::List(vec) | ||
} | ||
}, | ||
_ => ScalarAdapter::Builtin(scalar.into_scalar_impl()), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.into_scalar_impl()
contains a clone
and is the original reason ScalarAdapter
contains ScalarRefImpl
. But given it is only used as primary key during backfill at this time, I guess cloning is okay here.
efabf6b
to
360a6af
Compare
Agree. I've noted down the bugs found as issues. But they may not be urgent, will fix when requested.
Should be |
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Try refactoring to resolve #16416 (comment)
Checklist
./risedev check
(or alias,./risedev c
)Documentation
Release note
If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.