[BUG] Jackson compatibility issues with Spark and Azure SDK

[purijatin]

Describe the bug

Jackson.

Issues with jackson-module-scala compatibility.

Attached is my dependency-tree. Very briefly put, the project uses Spark, PlayJson, and Azure SDK. Spark brings in Jackson 2.10 dependencies, PlayJson also relies on Jackson 2.10 series. Azure SDK is on 2.12 series.

Exception or Stack Trace

[info] Cause: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0
[info] at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
[info] at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)
[info] at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
[info] at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:835)
[info] at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala:82)
[info] at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala)
[info] at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
[info] at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)

To Reproduce
Simple spark project that sends messages to event grid

Code Snippet
Simple spark project that uses Azure SDK. My list of sbt dependencies

import sbt._

object Dependencies {

  val SparkVersion = "3.1.2"
  val hadoop = Seq("org.apache.hadoop" % "hadoop-client" % "3.2.1")

  val spark = Seq(
    "org.apache.spark" %% "spark-core" % SparkVersion,
    "org.apache.spark" %% "spark-sql" % SparkVersion,
    "org.apache.spark" %% "spark-mllib" % SparkVersion
  )
  val others = Seq(
    "io.delta" %% "delta-core" % "1.0.0",
    "com.typesafe.play" %% "play-json" % "2.10.0-RC5",
    "org.postgresql" % "postgresql" % "42.3.1",
    "com.zaxxer" % "HikariCP" % "4.0.3"
  )

  val log = Seq(
    "org.slf4j" % "slf4j-api" % "1.7.32",
    "org.slf4j" % "slf4j-log4j12" % "1.7.32"
  )

  val azure = Seq(
    "com.azure" % "azure-messaging-eventgrid" % "4.7.1",
    "com.azure" % "azure-identity" % "1.4.1",
    "com.azure" % "azure-storage-file-datalake" % "12.7.2"
  )

  val test = Seq(
    "org.scalacheck" %% "scalacheck" % "1.14.3" % "test,it",
    "org.scalatest" %% "scalatest" % "3.2.2" % "test,it",
    "org.scalaj" %% "scalaj-http" % "2.4.2" % "test,it",
    "com.holdenkarau" %% "spark-testing-base" % "3.1.2_1.1.0" % "test,it"
  )

  val all = hadoop ++ spark ++ others ++ azure ++ log ++ test
}

Expected behavior

Screenshots
Dependency tree attached: tree-1.txt

If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

• OS: Windows
• IDE: SBT terminal
• Library/Libraries: Dependency tree attached
• Java version: 8
• App Server/Environment: Apache Spark
• Frameworks: PlayJson

Additional context

[joshfree]

@pallavit could you please follow up with @purijatin?

/cc @alzimmermsft

[purijatin]

The issue temporarily gets fixed by adding below. But we know that it is fragile. Adding this breaks at another point. Logging separate request for it (#25734)

"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.12.5"

[pallavit]

@purijatin let me know if there is anything pending on this particular issue. We have another issue created for the overall experience which we should continue to keep open however I would like to close this if you agree.

[purijatin]

Hi @pallavit. We can go ahead and close it. Thanks again for the help.