-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Jackson issues with Azure SDK when run on Azure Synapse Analytics #25734
Comments
Here is another such exception with Azure SDK and Synapse Spark Pool.
Suggestion Azure SDK should not have Jackson dependencies. But instead use something else for its Json needs. Jackson doesn't have a good reputation of being compatible across versions. And these kinds of problems will keep popping up. Ultimately affecting the usage of Azure SDK. |
Thanks for filing this, @purijatin. @lmolkova is working to reduce our dependency on Jackson. |
@purijatin thanks for reporting it! We're working on a better story for JAckson, in the meantime, can you please try to specify 2.10 version for all of the Jackson packages? Based on the error message you posted, there is a mismatch So, you'd need to explicitly pin 2.10 version for |
Hi @lmolkova As such, I don't have any direct dependency on Jackson. The project has a dependency on spark (which gets in 2.10) series. And azure sdk which gets in 2.12 series. Do you mean, I should add exclusion for jackson while adding a dependency for azure-sdk? So that only 2.10 remains on the classpath. Will azure-sdk magically work on 2.10? |
Hi @lmolkova I am also facing similar issue. As you mentioned in earlier comment I tried with exact versions which are suggested in exception but still issue not resolved. I have excluded the jackson azure-core and explicitly added them. Below is the exception Exception in thread "main" java.lang.LinkageError: Package versions: jackson-annotations=2.9.0, jackson-core=2.9.8, jackson-databind=2.9.8, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.9.9, azure-core=1.21.0 |
@purijatin yes, please add an explicit dependency on 2.10 for all Jackson packages Azure SDK is compatible with jackson 2.10+ |
@hasan505 |
@lmolkova |
Thanks. Let me try it out. Suggestion: If azure-sdk supports 2.10+, then it shouldn't have a dependency for 2.12 series. By doing for 2.12, it unintentionally evicts the older Jackson versions that other libraries may introduce. Giving rise to conflicts like above. If azure-sdk had relied on Jackson 2.10, then even if other libraries had relied on 2.11/2.12, maven/gradle/sbt would have evicted the 2.10 version by default and used the latest version. And things would have worked (because azure-sdk works with 2.11/2.12 on classpath). It is not perfect but may work for simple cases by default. I again feel that libraries like azure-sdk should not rely on Jackson. Maybe use something else for its Json needs. These kinds of issues are never-ending with Jackson and it has remained like this for years. It is not worth the pain IMHO. Though I don't think it would be easy now as it would break the user's code for example in Thanks again for all the great work and for helping proactively with these issues. |
thanks for the ideas! If we pin Jackson to 2.10 we'd expose users to some known bugs and vulnerabilities. Doing so now would also be breaking. |
I'm having a similar issue when trying to use the Azure SDK (azure-data-tables package) on Synapse with Spark 3.1, although it seems to be the inverse where an Azure SDK package is not compatible with Jackson 2.10. The code runs fine in local tests, but when I try to submit it to the Synapse cluster, I get this: java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.xml.XmlMapper.coercionConfigDefaults()Lcom/fasterxml/jackson/databind/cfg/MutableCoercionConfig; this is with azure-data-tables 12.1.2. Based on the comments above, I thought all Azure SDK packages were compatible with 2.10? |
@limotley please go through the Troubleshooting guide. Most likely you have a mismatch between versions of different Jackson packages. The guide mentions all Jackson packages that Azure SDKs use and most likely you'd need to pin each of them to a specific version supported by Synapse. If this does not help, please create a new issue and provide the information requested in the issue template. |
@lmolkova thanks for the tip! I had the Jackson packages pinned, but their version number wasn't set to the same one as Spark's. Working now |
I also face the same issue. Can anyone help me out in this? 2022-03-23 15:46:02.077 19310-19337/com.downloadfile E/AndroidRuntime: FATAL EXCEPTION: pool-2-thread-1 |
It also happens when you use Kafka and Azure Datalake SDK together. My move was to use a previous version of the azure dependency to avoid conflicts, accepting the vulnerabilities @lmolkova says. |
Had similar issues with Spring Boot and Azure Cognitive Search using the wrong Jackson version. Considering that we currently have 10+ Spring applications using Spring Boot 2.5.2 with carefully selected dependencies I don't want to update all of them just because of Microsoft... The way out was to identity the Search version using the wanted Jackson version like this:
Maybe not the best solution, because I cannot update Azure Search easily, but at least I got rid of the version conflicts, |
Add all the required dependencies explicitly in your pom.xml and it will resolve. for example: |
Azure OpenAI: minimal, partial specification for Whisper transcription/translation (Azure#25734) * minimalistic whisper .tsp definitions * merge, format, remove client.tsp changes for mvp simplicity * speculative example JSON update for string response types * restore header traits for swagger hints * review pass, prioritize object response for OpenAPI v2 * PR feedback: fully distinguish transcription/translation models
Hi @purijatin, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support. |
Describe the bug
We have a spark application that runs on Azure Synapse Spark Pools. Synapse brings a list of dependencies with it, which are by default available on class-path at runtime: https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/Apache-Spark-3-runtime
In the list is Jackson 2.10.
Our project uses PlayJson 2.9.2 that relies on Jackson 2.10. And latest version of Azure SDK (that brings in Jackson 2.12). When run against the Synapse Spark 3 runtime, we get the following exception.
If we remove the azure sdk dependency, then all the code works fine.
Exception or Stack Trace
When run on Synapse, we get the following exception:
To Reproduce
Run a synapse spark job with
azure-messaging-eventgrid
andazure-storage-file-datalake
dependencyCode Snippet
Expected behavior
Screenshots
tree-2.txt
Attached is the dependency tree.
Note: As mentioned above, synapse introduces its own set of dependencies at runtime. And we have no control over it.
Setup (please complete the following information):
The text was updated successfully, but these errors were encountered: