[FEATURE REQ] Allow Passing Custom Thread Factory to Blob Store HTTP Client

[original-brownbear]

Is your feature request related to a problem? Please describe.

We are trying to integrate Azure blob storage SDK v11 into the Elasticsearch Azure repository plugin in elastic/elasticsearch#43309. One challenge we ran into is the fact that the SDK v11 does require certain permissions that our security manager setup does not grant by default.
One example would be Jackson's "java.lang.RuntimePermission" "accessDeclaredMembers":

Caused by: com.microsoft.rest.v2.RestException: HTTP response has malformed headers
   at com.microsoft.rest.v2.protocol.HttpResponseDecoder.decode(HttpResponseDecoder.java:68)
   at com.microsoft.rest.v2.policy.DecodingPolicyFactory$DecodingPolicy.lambda$sendAsync$0(DecodingPolicyFactory.java:34)
   at io.reactivex.internal.operators.single.SingleFlatMap$SingleFlatMapCallback.onSuccess(SingleFlatMap.java:76)
   at io.reactivex.internal.operators.single.SingleDoOnSuccess$DoOnSuccess.onSuccess(SingleDoOnSuccess.java:60)
   at io.reactivex.internal.operators.single.SingleDoOnError$DoOnError.onSuccess(SingleDoOnError.java:52)
   at io.reactivex.internal.operators.single.SingleCreate$Emitter.onSuccess(SingleCreate.java:67)
   at com.microsoft.rest.v2.http.NettyClient$HttpClientInboundHandler.channelRead(NettyClient.java:925)
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
   at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
   at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
   at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
   at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
   at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
   at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
   at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
   at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
   at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
   at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
   at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)
   at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)
   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
   at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
   at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
   at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   ... 1 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")
   at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:303)
   at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2509)
   at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:378)
   at com.microsoft.rest.v2.http.HttpHeaders.serialize(HttpHeaders.java:125)
   at com.fasterxml.jackson.databind.ser.std.SerializableSerializer.serialize(SerializableSerializer.java:49)
   at com.fasterxml.jackson.databind.ser.std.SerializableSerializer.serialize(SerializableSerializer.java:27)
   at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292)
   at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3697)
   at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3055)
   at com.microsoft.rest.v2.serializer.JacksonAdapter.serialize(JacksonAdapter.java:86)
   at com.microsoft.rest.v2.protocol.HttpResponseDecoder.deserializeHeaders(HttpResponseDecoder.java:291)
   at com.microsoft.rest.v2.protocol.HttpResponseDecoder.decode(HttpResponseDecoder.java:66)
   ... 29 more
Caused by: java.security.AccessControlException: access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")
   at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
   at java.base/java.security.AccessController.checkPermission(AccessController.java:1042)
   at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
   at java.base/java.lang.Class.checkMemberAccess(Class.java:2853)
   at java.base/java.lang.Class.getDeclaredFields(Class.java:2252)
   at com.microsoft.rest.v2.serializer.AdditionalPropertiesSerializer$1.modifySerializer(AdditionalPropertiesSerializer.java:78)
   at com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:252)
   at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:168)
   at com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1308)
   at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1258)
   at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:500)
   at com.fasterxml.jackson.databind.ser.impl.PropertySerializerMap.findAndAddSecondarySerializer(PropertySerializerMap.java:90)
   at com.fasterxml.jackson.databind.ser.std.MapSerializer._findAndAddDynamic(MapSerializer.java:928)
   at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:627)
   at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536)
   at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30)
   at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292)
   ... 40 more

While in the legacy SDK v8 we could cleanly resolve this by granting the necessary permissions to the SDKs codebase and executing calls to it in a privileged context, we cannot do this in v11. This is due to the fact that calls like the above example run on Netty threads that are set up by the SDK itself without any possibility to grant them the necessary privileges (happy to be corrected here obviously).

Describe the solution you'd like

It seems like the easiest solution here would be to allow for more customization of the Netty setup. In Elasticsearch itself we resolve the above problem by using a custom ThreadFactory when setting up a NioEventLoopGroup. This way we can make the custom factory create threads that are of the same group as the calling code and thus resolve any permissions/security manager issues by that route.

=> Having the ability to pass a java.util.concurrent.ThreadFactory when setting up the com.microsoft.azure.storage.blob.PipelineOptions (or the HttpClient, so we could set up a custom one with our thread factory) seems to be the most direct solution to enabling the use of the SDK with a security manager.

Describe alternatives you've considered

The only alternative I was able to come up with so far was to set up our own implementation of HttpClient which is less than desirable due to the complexity of maintaining that solution. I'd be happy to hear alternative suggestions for integration with the security manager though.

[praries880]

Thank you for opening this issue! We are routing it to the appropriate team for follow up.

[Petermarcu]

@jianghaolu @JonathanGiles is this something that will be possible in v12?

[joshfree]

@jianghaolu could you please take a look?

[rickle-msft]

@original-brownbear Thank you for this suggestion. We have included this in our work items for v12. We will notify you when there is a release that has this feature.

[original-brownbear]

@rickle-msft Thanks, sounds great!

[JonathanGiles]

We could probably expose API specific to Netty through the following draft PR: #5104.

[ghost]

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.