Compatibility issue with different Guava versions between gRPC and Apache Spark in a Milvus project

[liaomu123]

am working on a project that integrates Milvus (2.3.15) which depends on gRPC version 1.46.0. gRPC, in turn, has a direct dependency on Guava 31.0.1. However, I'm encountering compatibility issues because my project also utilizes Apache Spark(version 2.3.2-TBDS-5.1.3.0 Can be considered to be 2.3.2), which bundles its own version of Guava (14.0).
The conflict arises due to the significant difference in Guava versions, leading to potential classpath conflicts and method signature mismatches. I'm looking for guidance on how best to resolve this issue without modifying the core dependencies of either Spark or gRPC.
Here are a few questions I have in mind:
Is there an officially recommended approach to handle such Guava version discrepancies when working with both gRPC and Spark?
Are there any known compatibility layers or shading techniques that can be applied to isolate the Guava usage within the context of gRPC, ensuring it doesn't interfere with Spark's bundled Guava?
Has anyone successfully managed a similar setup and could share their configuration or approach?
Given the complexity of managing transitive dependencies and the critical nature of these libraries, any insights or previous experiences shared would be immensely helpful.
Thank you in advance for your assistance!

[ejona86]

Yeah, gRPC won't work with Guava 14 from 2013. #5592 (comment) talks about enabling userClassPathFirst so that you can use your dependencies instead of Spark overriding them. That was still broken class loader logic, but it was at least better than the default. I don't know the current state of Spark, though.

The gRPC version you are using is well out of support. The Spark version appears out of support. It seems like you need to upgrade multiple different things.

[ejona86]

Seems like this should be resolved if you use the Spark configuration. If not, comment, and it can be reopened.