Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HUDI-4468] Simplify TimeTravel logic for Spark #12270

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

KnightChess
Copy link
Contributor

@KnightChess KnightChess commented Nov 17, 2024

Change Logs

After spark3.3 support TimeTravel SQL and Logical Plan, remove the implementation in hudi, remain index DML SQL

code detail:
before spark3.3, spark not support timetravel sql, so hudi supports the full DQL query syntax for timetravel.
now, we remove it, therefore, a large portion of the DQL syntax parsing will not be used; we only need to support Hudi's custom implementation of the DDL syntax related to indexing.

g4 file

  • delete queryStatement dmlStatement createTable in HoodieSqlBase.g4, it's for timetravel
  • delete SqlBase.g4 file, copy which index need.

astBuilder

  • delete a large amount of redundant implementations and retain only those related to indexing

Impact

in spark, TIMESTAMP dataType is TimestampType, VERSION is String. And the hoodie instance time format is not legal for spark to parse.
And from the perspective of usage, the concept of HUDI's timestamp concept is more suitable for using VERSION, so in this pr:

  • VERSION AS OF
    • support digit string, contain hudi base timeline format - [yyyyMMddhhmmssSSS], will return less or equal snapshots
  • TIMSTAMP AS OF
    • spark provides a rich date expression, will return less or equal snapshots

Risk level (write none, low medium or high below)

low

Documentation Update

spark Time Travel Query tutorial, update

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@github-actions github-actions bot added the size:XL PR with lines of changes > 1000 label Nov 17, 2024
@vinothchandar
Copy link
Member

@jonvex can you please review this

@jonvex jonvex self-requested a review November 18, 2024 15:50
@jonvex jonvex self-assigned this Nov 18, 2024
@jonvex
Copy link
Contributor

jonvex commented Nov 18, 2024

@KnightChess can you please explain what changes you made to the files with 1000+ added or removed lines of code. Thanks!

Copy link
Contributor Author

@KnightChess KnightChess left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonvex hi, update the describre in pr.

@KnightChess KnightChess marked this pull request as ready for review November 19, 2024 04:56
@hudi-bot
Copy link

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:XL PR with lines of changes > 1000
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants