Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Spark UT framework: SPARK-31159, SPARK-37705: compatibility with Spark 2.4/3.2 in reading dates/timestamps #11599

Open
Tracked by #11404
Feng-Jiang28 opened this issue Oct 12, 2024 · 0 comments
Labels
? - Needs Triage Need team to review and classify bug Something isn't working

Comments

@Feng-Jiang28
Copy link
Collaborator

Feng-Jiang28 commented Oct 12, 2024

This UT cases have 2 problems:

  1. test-data/before_1582_date_v2_4_5.snappy.parquet cannot be located.
    for fix this UT case, we need to modify the code
    def failInRead(path: String): Unit = {
      println(path)
      val e = intercept[SparkException](spark.read.parquet(testFile(path)).collect())
      assert(e.getCause.isInstanceOf[SparkUpgradeException])
    }
  1. Please refer to [BUG] Spark UT framework: [SPARK-31159] rebasing dates in write #11480
    The issue is caused by unsupported kernel, feature has been requested here [FEA] Implement kernel to support non-UTC time zone for LEGACY mode. #11562
    java.lang.UnsupportedOperationException: LEGACY datetime rebase mode is only supported for files written in UTC timezone. Actual file timezone: America/Los_Angeles
@Feng-Jiang28 Feng-Jiang28 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Oct 12, 2024
@Feng-Jiang28 Feng-Jiang28 changed the title SPARK-31159, SPARK-37705: compatibility with Spark 2.4/3.2 in reading dates/timestamps [BUG] Spark UT framework: SPARK-31159, SPARK-37705: compatibility with Spark 2.4/3.2 in reading dates/timestamps Oct 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
? - Needs Triage Need team to review and classify bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant