Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(source): resolve avro Ref during access without hack #19746

Merged
merged 4 commits into from
Dec 12, 2024

Conversation

xiangjinwu
Copy link
Contributor

@xiangjinwu xiangjinwu commented Dec 10, 2024

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Sequel to #19601 and resolves #17020.

The recursive call is changed from Self {schema: inner, relax_numeric: self.relax_numeric}.convert_to_datum(v, field_type) into self.convert_to_datum(inner, v, field_type) so that

  • The changing part schema: inner is passed as argument rather than keep reconstructing new AvroParseOptions.
  • self: AvroParseOptionsInner captures the immutable "global" context: relax_numeric: bool and refs: NamesRef<'a>

On the use site, self.schema is changed to self.lookup_ref(unresolved_schema). Note that introducing a new type to distinguish resolved from unresolved would not make it less error-prone for a careless developer - nothing prevents them from matching against unresolved apache_avro::Schema directly and ignoring Schema::Ref.

Full removal of ResolvedAvroSchema will be done in an immediate follow-up, to minimize the signature changes in this PR. ResolvedAvroSchema will either be fully removed or repurposed to hold HashMap<Name, Schema> (an owned NamesRef) in a follow-up.

Same as #19601, this is intended to be part of v2.2 and NOT cherry-picked into earlier versions.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added test labels as necessary. See details.
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

@github-actions github-actions bot added the type/fix Bug fix label Dec 10, 2024
@graphite-app graphite-app bot requested a review from a team December 10, 2024 11:00
Copy link
Member

@xxchan xxchan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:GTM

@xiangjinwu xiangjinwu added this pull request to the merge queue Dec 12, 2024
Comment on lines -38 to -41
/// Currently, this schema is only used for decimal.
///
/// FIXME: In theory we should use resolved schema.
/// e.g., it's possible that a field is a reference to a decimal or a record containing a decimal field.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we can support reference to a decimal or a record containing a decimal field.

Merged via the queue into main with commit 1678ab1 Dec 12, 2024
32 of 33 checks passed
@xiangjinwu xiangjinwu deleted the fix-source-avro-ref-resolve-access branch December 12, 2024 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/fix Bug fix
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Avro ref type in source
2 participants