Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Jackson issues with Azure SDK when run on Azure Synapse Analytics #25734

Closed
purijatin opened this issue Nov 29, 2021 · 18 comments
Closed
Assignees
Labels
Azure.Core azure-core Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. dependency-issue Issue that is caused by dependency conflicts dependency-issue-jackson Issue caused by dependency version mismatch with one of the Jackson libraries question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@purijatin
Copy link

Describe the bug

We have a spark application that runs on Azure Synapse Spark Pools. Synapse brings a list of dependencies with it, which are by default available on class-path at runtime: https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/Apache-Spark-3-runtime

In the list is Jackson 2.10.

Our project uses PlayJson 2.9.2 that relies on Jackson 2.10. And latest version of Azure SDK (that brings in Jackson 2.12). When run against the Synapse Spark 3 runtime, we get the following exception.

If we remove the azure sdk dependency, then all the code works fine.

Exception or Stack Trace

When run on Synapse, we get the following exception:


21/11/29 17:38:00 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.LinkageError: Package versions: jackson-annotations=2.10.0, jackson-core=2.10.0, jackson-databind=2.10.0, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.12.5, azure-core=1.19.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
	at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:73)
	at com.azure.core.util.serializer.JacksonAdapter.<init>(JacksonAdapter.java:81)
	at com.azure.core.util.serializer.JacksonAdapter.<init>(JacksonAdapter.java:58)
	at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.<clinit>(JacksonAdapter.java:113)
	at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:122)
	at com.azure.identity.implementation.IdentityClient.<clinit>(IdentityClient.java:100)
	at com.azure.identity.implementation.IdentityClientBuilder.build(IdentityClientBuilder.java:139)
	at com.azure.identity.ManagedIdentityCredential.<init>(ManagedIdentityCredential.java:70)
	at com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:129)
	at com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:123)com.microsoft.aiops.anomaly.events.AnomalyEventGridSender.$anonfun$sendEvents$1$adapted(AnomalyEventGridSender.scala:25)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.sendEvents(xxxxxx.scala:25)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.runner.xxxxxx.xxxxxx(xxxxxx.scala:82)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx(xxxxxx.scala:61)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.$anonfun$start$2(xxxxxx.scala:39)
	at scala.collection.TraversableLike$WithFilter.$anonfun$map$2(TraversableLike.scala:827)
	at scala.collection.Iterator.foreach(Iterator.scala:941)
	at scala.collection.Iterator.foreach$(Iterator.scala:941)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:826)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.x.xxxxx.xxxxxx.start(xxxxxx.scala:36)
	at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx$.main(xxxxxx.scala:29)
	at xxxxxx.xxxxxx.aiops.xxxxxx.xxxxxx.main(xxxxxx.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:732)
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.xml.XmlMapper.coercionConfigDefaults()Lcom/fasterxml/jackson/databind/cfg/MutableCoercionConfig;
	at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:176)
	at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:145)
	at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:127)
	at com.fasterxml.jackson.dataformat.xml.XmlMapper.builder(XmlMapper.java:218)
	at com.azure.core.implementation.jackson.ObjectMapperFactory.createXmlMapper(ObjectMapperFactory.java:84)
	at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:70)
	... 45 more

To Reproduce
Run a synapse spark job with azure-messaging-eventgrid and azure-storage-file-datalake dependency

Code Snippet

Expected behavior

Screenshots
tree-2.txt
Attached is the dependency tree.

Note: As mentioned above, synapse introduces its own set of dependencies at runtime. And we have no control over it.

Setup (please complete the following information):

  • OS: WIndows
  • IDE: Intellij
  • Library/Libraries: Dependency tree attached
  • Java version: 8
  • App Server/Environment: Synapse
  • Frameworks: PlayJson, Spark
@ghost ghost added needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Nov 29, 2021
@purijatin
Copy link
Author

Here is another such exception with Azure SDK and Synapse Spark Pool.


[info] Cause: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0
[info] at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
[info] at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)
[info] at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
[info] at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:835)
[info] at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala:82)
[info] at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala)
[info] at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
[info] at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)

Suggestion

Azure SDK should not have Jackson dependencies. But instead use something else for its Json needs. Jackson doesn't have a good reputation of being compatible across versions. And these kinds of problems will keep popping up. Ultimately affecting the usage of Azure SDK.

@joshfree joshfree added Azure.Core azure-core Client This issue points to a problem in the data-plane of the library. dependency-issue Issue that is caused by dependency conflicts dependency-issue-jackson Issue caused by dependency version mismatch with one of the Jackson libraries labels Nov 30, 2021
@ghost ghost removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Nov 30, 2021
@joshfree
Copy link
Member

Azure SDK should not have Jackson dependencies. But instead use something else for its Json needs. Jackson doesn't have a good reputation of being compatible across versions. And these kinds of problems will keep popping up. Ultimately affecting the usage of Azure SDK.

Thanks for filing this, @purijatin. @lmolkova is working to reduce our dependency on Jackson.

@lmolkova
Copy link
Member

lmolkova commented Nov 30, 2021

@purijatin thanks for reporting it! We're working on a better story for JAckson, in the meantime, can you please try to specify 2.10 version for all of the Jackson packages?

Based on the error message you posted, there is a mismatch
Package versions: jackson-annotations=2.10.0, jackson-core=2.10.0, jackson-databind=2.10.0, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.12.5

So, you'd need to explicitly pin 2.10 version for jackson-dataformat-xml and jackson-datatype-jsr310.

@purijatin
Copy link
Author

Hi @lmolkova

As such, I don't have any direct dependency on Jackson. The project has a dependency on spark (which gets in 2.10) series. And azure sdk which gets in 2.12 series.

Do you mean, I should add exclusion for jackson while adding a dependency for azure-sdk? So that only 2.10 remains on the classpath. Will azure-sdk magically work on 2.10?

@hasan505
Copy link

hasan505 commented Dec 1, 2021

Hi @lmolkova

I am also facing similar issue. As you mentioned in earlier comment I tried with exact versions which are suggested in exception but still issue not resolved. I have excluded the jackson azure-core and explicitly added them. Below is the exception

Exception in thread "main" java.lang.LinkageError: Package versions: jackson-annotations=2.9.0, jackson-core=2.9.8, jackson-databind=2.9.8, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.9.9, azure-core=1.21.0
at com.azure.core.implementation.jackson.ObjectMapperShim.createHeaderMapper(ObjectMapperShim.java:129)
at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:75)
at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:53)
at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:111)
at com.azure.storage.blob.implementation.util.ModelHelper.(ModelHelper.java:53)
at com.azure.storage.blob.BlobUrlParts.parse(BlobUrlParts.java:371)
at com.azure.storage.blob.BlobServiceClientBuilder.endpoint(BlobServiceClientBuilder.java:147)
at com.zwayam.storage.AzureStorageProvider.getBlobContainerClient(AzureStorageProvider.java:183)

@lmolkova
Copy link
Member

lmolkova commented Dec 1, 2021

@purijatin yes, please add an explicit dependency on 2.10 for all Jackson packages
jackson-annotations, jackson-core, jackson-databind, jackson-dataformat-xml, jackson-datatype-jsr310. And please consider using the latest patch - 2.10.5.

Azure SDK is compatible with jackson 2.10+

@lmolkova
Copy link
Member

lmolkova commented Dec 1, 2021

@hasan505
Based on the log you provided, you're using mixture of versions: Package versions: jackson-annotations=2.9.0, jackson-core=2.9.8, jackson-databind=2.9.8, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.9.9,.
Make sure you're pinning versions for all packages and note that Azure SDK is compatible with Jackson 2.10+.

@hasan505
Copy link

hasan505 commented Dec 1, 2021

@lmolkova
Yes, It's an issue with different versions of jackson dependencies. I have removed them and added 2.12.5 which is working fine now. Thank you for the assistance.

@purijatin
Copy link
Author

purijatin commented Dec 2, 2021

@purijatin yes, please add an explicit dependency on 2.10 for all Jackson packages jackson-annotations, jackson-core, jackson-databind, jackson-dataformat-xml, jackson-datatype-jsr310. And please consider using the latest patch - 2.10.5.

Azure SDK is compatible with jackson 2.10+

Thanks. Let me try it out.

Suggestion: If azure-sdk supports 2.10+, then it shouldn't have a dependency for 2.12 series. By doing for 2.12, it unintentionally evicts the older Jackson versions that other libraries may introduce. Giving rise to conflicts like above.

If azure-sdk had relied on Jackson 2.10, then even if other libraries had relied on 2.11/2.12, maven/gradle/sbt would have evicted the 2.10 version by default and used the latest version. And things would have worked (because azure-sdk works with 2.11/2.12 on classpath). It is not perfect but may work for simple cases by default.

I again feel that libraries like azure-sdk should not rely on Jackson. Maybe use something else for its Json needs. These kinds of issues are never-ending with Jackson and it has remained like this for years. It is not worth the pain IMHO. Though I don't think it would be easy now as it would break the user's code for example in BinaryData.fromObject.

Thanks again for all the great work and for helping proactively with these issues.

@lmolkova
Copy link
Member

lmolkova commented Dec 2, 2021

thanks for the ideas! If we pin Jackson to 2.10 we'd expose users to some known bugs and vulnerabilities. Doing so now would also be breaking.
While we're considering reducing our dependency on Jackson libraries, this is a big investment, we track this work here #24480.

@limotley
Copy link

limotley commented Dec 8, 2021

I'm having a similar issue when trying to use the Azure SDK (azure-data-tables package) on Synapse with Spark 3.1, although it seems to be the inverse where an Azure SDK package is not compatible with Jackson 2.10. The code runs fine in local tests, but when I try to submit it to the Synapse cluster, I get this:

java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.xml.XmlMapper.coercionConfigDefaults()Lcom/fasterxml/jackson/databind/cfg/MutableCoercionConfig;
at com.fasterxml.jackson.dataformat.xml.XmlMapper.(XmlMapper.java:176)
at com.fasterxml.jackson.dataformat.xml.XmlMapper.(XmlMapper.java:145)
at com.fasterxml.jackson.dataformat.xml.XmlMapper.(XmlMapper.java:127)
at com.fasterxml.jackson.dataformat.xml.XmlMapper.builder(XmlMapper.java:218)
at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:140)
at com.azure.data.tables.implementation.TablesJacksonSerializer.(TablesJacksonSerializer.java:27)
at com.azure.data.tables.TableClientBuilder.(TableClientBuilder.java:54)
at .....

this is with azure-data-tables 12.1.2. Based on the comments above, I thought all Azure SDK packages were compatible with 2.10?

@lmolkova
Copy link
Member

lmolkova commented Dec 8, 2021

@limotley please go through the Troubleshooting guide. Most likely you have a mismatch between versions of different Jackson packages. The guide mentions all Jackson packages that Azure SDKs use and most likely you'd need to pin each of them to a specific version supported by Synapse.

If this does not help, please create a new issue and provide the information requested in the issue template.
Thanks!

@limotley
Copy link

limotley commented Dec 9, 2021

@lmolkova thanks for the tip! I had the Jackson packages pinned, but their version number wasn't set to the same one as Spark's. Working now

@nishanthvasu
Copy link

I also face the same issue. Can anyone help me out in this?

2022-03-23 15:46:02.077 19310-19337/com.downloadfile E/AndroidRuntime: FATAL EXCEPTION: pool-2-thread-1
Process: com.downloadfile, PID: 19310
java.lang.LinkageError: Package versions: jackson-annotations=0.0, jackson-core=0.0, jackson-databind=0.0, jackson-dataformat-xml=0.0, jackson-datatype-jsr310=0.0, azure-core=1.26.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:73)
at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:81)
at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:58)
at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.(JacksonAdapter.java:113)
at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.access$000(JacksonAdapter.java:109)
at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:122)
at com.azure.storage.blob.implementation.util.ModelHelper.(ModelHelper.java:63)
at com.azure.storage.blob.implementation.util.ModelHelper.determineAuthorityIsIpStyle(ModelHelper.java:82)
at com.azure.storage.blob.BlobUrlParts.parse(BlobUrlParts.java:371)
at com.azure.storage.blob.BlobServiceClientBuilder.endpoint(BlobServiceClientBuilder.java:171)
at com.azure.storage.blob.BlobServiceClientBuilder.connectionString(BlobServiceClientBuilder.java:328)
at com.downloadfile.FileDownloadTask.download(FileDownloadTask.java:85)
at com.downloadfile.FileDownloadTask.run(FileDownloadTask.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:923)
Caused by: java.lang.NoSuchMethodError: No static method newFactory(Ljava/lang/String;Ljava/lang/ClassLoader;)Ljavax/xml/stream/XMLInputFactory; in class Ljavax/xml/stream/XMLInputFactory; or its super classes (declaration of 'javax.xml.stream.XMLInputFactory' appears in /data/data/com.downloadfile/code_cache/.overlay/base.apk/classes3.dex)
at com.fasterxml.jackson.dataformat.xml.XmlFactory.(XmlFactory.java:115)
at com.fasterxml.jackson.dataformat.xml.XmlFactory.(XmlFactory.java:101)
at com.fasterxml.jackson.dataformat.xml.XmlFactory.(XmlFactory.java:85)
at com.fasterxml.jackson.dataformat.xml.XmlMapper.(XmlMapper.java:127)
at com.fasterxml.jackson.dataformat.xml.XmlMapper.builder(XmlMapper.java:218)
at com.azure.core.implementation.jackson.ObjectMapperFactory.createXmlMapper(ObjectMapperFactory.java:85)
at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:70)
at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:81) 
at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:58) 
at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.(JacksonAdapter.java:113) 
at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.access$000(JacksonAdapter.java:109) 
at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:122) 
at com.azure.storage.blob.implementation.util.ModelHelper.(ModelHelper.java:63) 
at com.azure.storage.blob.implementation.util.ModelHelper.determineAuthorityIsIpStyle(ModelHelper.java:82) 
at com.azure.storage.blob.BlobUrlParts.parse(BlobUrlParts.java:371) 
at com.azure.storage.blob.BlobServiceClientBuilder.endpoint(BlobServiceClientBuilder.java:171) 
at com.azure.storage.blob.BlobServiceClientBuilder.connectionString(BlobServiceClientBuilder.java:328) 
at com.downloadfile.FileDownloadTask.download(FileDownloadTask.java:85) 
at com.downloadfile.FileDownloadTask.run(FileDownloadTask.java:49) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) 
at java.lang.Thread.run(Thread.java:923) 

@jmakuc
Copy link

jmakuc commented Mar 24, 2022

It also happens when you use Kafka and Azure Datalake SDK together. My move was to use a previous version of the azure dependency to avoid conflicts, accepting the vulnerabilities @lmolkova says.

@jbauerrfid
Copy link

jbauerrfid commented Mar 24, 2022

Had similar issues with Spring Boot and Azure Cognitive Search using the wrong Jackson version. Considering that we currently have 10+ Spring applications using Spring Boot 2.5.2 with carefully selected dependencies I don't want to update all of them just because of Microsoft...

The way out was to identity the Search version using the wanted Jackson version like this:

com.azure/azure-search-documents: 11.4.0
    |
    +-> com.azure/azure-core: 1.17.0
        |
	+-> com.fasterxml.jackson-*/*: 2.12.3
        |
org.springframework.boot/spring-boot-starter-webflux: 2.5.2

Maybe not the best solution, because I cannot update Azure Search easily, but at least I got rid of the version conflicts,

@shrutisarda23
Copy link

Add all the required dependencies explicitly in your pom.xml and it will resolve. for example:

com.fasterxml.jackson.dataformat
jackson-dataformat-xml
2.8.9


com.fasterxml.jackson.datatype
jackson-datatype-jsr310
2.8.9


com.fasterxml.jackson.core
jackson-core
2.8.9


com.fasterxml.jackson.core
jackson-databind
2.8.9

azure-sdk pushed a commit to azure-sdk/azure-sdk-for-java that referenced this issue Sep 20, 2023
Azure OpenAI: minimal, partial specification for Whisper transcription/translation (Azure#25734)

* minimalistic whisper .tsp definitions

* merge, format, remove client.tsp changes for mvp simplicity

* speculative example JSON update for string response types

* restore header traits for swagger hints

* review pass, prioritize object response for OpenAPI v2

* PR feedback: fully distinguish transcription/translation models
Copy link

Hi @purijatin, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 13, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Mar 13, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Azure.Core azure-core Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. dependency-issue Issue that is caused by dependency conflicts dependency-issue-jackson Issue caused by dependency version mismatch with one of the Jackson libraries question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
Status: Done
Development

No branches or pull requests

9 participants