Self Hosted Gateway bug with Load-balanced pool (preview)

[ahalamir]

Report

I have a self hosted gateway hosted inside ARO and another self hosted gateway hosted on a docker within a VM, I'm i have created 2 backends using azure portal.

Then I have used APIM Management REST APIs to create another backend that is referencing the above backends, as I want to try round robin load balancing preview feature on the self hosted gateway

I have tested APIM managed instance and everything is working as expected where the RoundRobinNameResolver is using the 2 backends as expected in a round robin fashion, however when i try the same approach using self hosted gateway i'm getting the following exception: "Name does not resolve"

Expected Behavior

The RoundRobinNameResolver should be working within self hosted gateway compared to managed APIM instance.

Actual Behavior

RoundRobinNameResolver is failing with the following exception:

�[33m[Warn] 2024-01-19T07:39:55.040 [FailedToProcessRequest], apiId: roundrobin, operationId: rb, subscriptionId: master, tags: 20, httpMethod: GET, source: request-forwarder, exception: System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name does not resolve
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, ValueStopwatch stopwatch)
at System.Net.Dns.<>c.b__33_0(Object s, ValueStopwatch stopwatch)
at System.Net.Dns.<>c__DisplayClass39_01.<RunAsync>b__0(Task <p0>, Object <p1>) at System.Threading.Tasks.ContinuationResultTaskFromTask1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.WindowsAzure.ApiManagement.Proxy.Runtime.DefaultDnsNameResolver.ResolveAsync(String name, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline.IO\Dns\DefaultDnsNameResolver.cs:line 28
at Gateway.Http.Client.DotNetty.RoundRobinNameResolver.DoResolveAsync(String name, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\RoundRobinNameResolver.cs:line 178
at Gateway.Http.Client.DotNetty.RoundRobinNameResolver.ResolveAsync(IPipelineContext context, String name, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\RoundRobinNameResolver.cs:line 149
at Gateway.Http.Client.DotNetty.SingleThreadedBackendChannelPool.AcquireAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\SingleThreadedBackendChannelPool.cs:line 110
at Gateway.Http.Client.DotNetty.RoundRobinBackendChannelPool.Acquire0(Object state) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\RoundRobinBackendChannelPool.cs:line 84
at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.TrackLatency[T](ValueTask1 task, IPipelineContext ctx) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\DotNettyHttpBackend.cs:line 682 at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\DotNettyHttpBackend.cs:line 127 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable1 steps, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineWalker.cs:line 67
at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.ChildPipeline.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Pipeline\ChildPipeline.cs:line 38
at Gateway.Pipeline.Extensions.ValueTaskExtensions.Await[T](ValueTask1 input) in D:\a\1\s\Proxy\Gateway.Pipeline\Extensions\ValueTaskExtensions.cs:line 29 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable1 steps, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineWalker.cs:line 67
at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.ChildPipeline.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Pipeline\ChildPipeline.cs:line 38
at Gateway.Pipeline.Extensions.ValueTaskExtensions.Await[T](ValueTask1 input) in D:\a\1\s\Proxy\Gateway.Pipeline\Extensions\ValueTaskExtensions.cs:line 29 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Models.ConfigurationBasedBackend.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Model\ConfigurationBasedBackend.cs:line 91 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.IO.CallServiceHandler.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Policies.General\IO\CallServiceHandler.cs:line 103 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable1 steps, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineWalker.cs:line 67
at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.ChildPipeline.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Pipeline\ChildPipeline.cs:line 38
at Gateway.Pipeline.Extensions.ValueTaskExtensions.Await[T](ValueTask1 input) in D:\a\1\s\Proxy\Gateway.Pipeline\Extensions\ValueTaskExtensions.cs:line 29 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable1 steps, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineWalker.cs:line 67
at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.PipelineExecutor.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineExecutor.cs:line 204, transportError: -131073, httpError: 0, correlationId: 24dd042d-1741-40e3-9fcb-ca5a5acd27ea�[0m
�[37m[Info] 2024-01-19T07:39:54.666 [GatewayLogs], isRequestSuccess: False, totalTime: 374, category: GatewayLogs, callerIpAddress: 10.129.2.26, timeGenerated: 2024-01-19T07:39:54.666, region: ocp-dev-east, correlationId: 24dd042d-1741-40e3-9fcb-ca5a5acd27ea, method: GET, url: https://apim-dxf-azure-apim.apps.ocp.dxf-services.com/roundrobin/, responseCode: 500, responseSize: 191, cache: none, backendTime: 371, apiId: roundrobin, operationId: rb, apimSubscriptionId: master, clientProtocol: HTTP/1.1, backendId: roundrobin, apiRevision: 1, clientTlsVersion: 1.2, backendMethod: GET, backendUrl: http://unused/, lastError: {
"elapsed": 373,
"source": "request-forwarder",
"path": "forward-request\forward-request\backend",
"reason": "BackendConnectionFailure",
"message": "Name does not resolve",
"section": "backend",
"transportErrorCode": -131073
}, errors: [
{
"elapsed": 373,
"source": "request-forwarder",
"path": "forward-request\forward-request\backend",
"reason": "BackendConnectionFailure",
"message": "Name does not resolve",
"section": "backend",
"transportErrorCode": -131073
}
], correlationId: 24dd042d-1741-40e3-9fcb-ca5a5acd27ea�[0m

Steps to Reproduce the Problem

1. Create 2 different backends
2. Create a pooled backed that is referencing backends of step 1.
   {
   "properties": {
   "description": "Load balancer for multiple backends",
   "type": "Pool",
   "protocol": "http",
   "url": "http://unused",
   "pool": {
   "services": [
   {
   "id": "/backends/backend-1"
   },
   {
   "id": "/backends/backend-2"
   }
   ]
   }
   }
   }
3. Create a policy that is using backend in step 2:

Logs from self-hosted gateway

�[37m[Info] 2024-01-19T07:38:12.752 [GatewayLogs], isRequestSuccess: False, totalTime: 285, category: GatewayLogs, callerIpAddress: 10.129.2.26, timeGenerated: 2024-01-19T07:38:12.752, region: ocp-dev-east, correlationId: 6d3b4c55-dbed-42f2-baa1-c5d08bc05173, method: GET, url: https://apim-dxf-azure-apim.apps.ocp.dxf-services.com/roundrobin/, responseCode: 500, responseSize: 191, cache: none, backendTime: 282, apiId: roundrobin, operationId: rb, apimSubscriptionId: master, clientProtocol: HTTP/1.1, backendId: roundrobin, apiRevision: 1, clientTlsVersion: 1.2, backendMethod: GET, backendUrl: http://unused/, lastError: {
"elapsed": 283,
"source": "request-forwarder",
"path": "forward-request\forward-request\backend",
"reason": "BackendConnectionFailure",
"message": "Name does not resolve",
"section": "backend",
"transportErrorCode": -131073
}, errors: [
{
"elapsed": 283,
"source": "request-forwarder",
"path": "forward-request\forward-request\backend",
"reason": "BackendConnectionFailure",
"message": "Name does not resolve",
"section": "backend",
"transportErrorCode": -131073
}
], correlationId: 6d3b4c55-dbed-42f2-baa1-c5d08bc05173�[0m
�[37m[Info] 2024-01-19T07:39:54.667 [DnsResolutionStarted], message: unused, source: RoundRobinNameResolver�[0m
�[33m[Warn] 2024-01-19T07:39:55.040 [FailedToProcessRequest], apiId: roundrobin, operationId: rb, subscriptionId: master, tags: 20, httpMethod: GET, source: request-forwarder, exception: System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name does not resolve
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, ValueStopwatch stopwatch)
at System.Net.Dns.<>c.b__33_0(Object s, ValueStopwatch stopwatch)
at System.Net.Dns.<>c__DisplayClass39_01.<RunAsync>b__0(Task <p0>, Object <p1>) at System.Threading.Tasks.ContinuationResultTaskFromTask1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Microsoft.WindowsAzure.ApiManagement.Proxy.Runtime.DefaultDnsNameResolver.ResolveAsync(String name, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline.IO\Dns\DefaultDnsNameResolver.cs:line 28
at Gateway.Http.Client.DotNetty.RoundRobinNameResolver.DoResolveAsync(String name, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\RoundRobinNameResolver.cs:line 178
at Gateway.Http.Client.DotNetty.RoundRobinNameResolver.ResolveAsync(IPipelineContext context, String name, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\RoundRobinNameResolver.cs:line 149
at Gateway.Http.Client.DotNetty.SingleThreadedBackendChannelPool.AcquireAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\SingleThreadedBackendChannelPool.cs:line 110
at Gateway.Http.Client.DotNetty.RoundRobinBackendChannelPool.Acquire0(Object state) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\RoundRobinBackendChannelPool.cs:line 84
at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.TrackLatency[T](ValueTask1 task, IPipelineContext ctx) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\DotNettyHttpBackend.cs:line 682 at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\DotNettyHttpBackend.cs:line 127 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable1 steps, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineWalker.cs:line 67
at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.ChildPipeline.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Pipeline\ChildPipeline.cs:line 38
at Gateway.Pipeline.Extensions.ValueTaskExtensions.Await[T](ValueTask1 input) in D:\a\1\s\Proxy\Gateway.Pipeline\Extensions\ValueTaskExtensions.cs:line 29 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable1 steps, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineWalker.cs:line 67
at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.ChildPipeline.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Pipeline\ChildPipeline.cs:line 38
at Gateway.Pipeline.Extensions.ValueTaskExtensions.Await[T](ValueTask1 input) in D:\a\1\s\Proxy\Gateway.Pipeline\Extensions\ValueTaskExtensions.cs:line 29 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Models.ConfigurationBasedBackend.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Model\ConfigurationBasedBackend.cs:line 91 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.IO.CallServiceHandler.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Policies.General\IO\CallServiceHandler.cs:line 103 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable1 steps, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineWalker.cs:line 67
at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.ChildPipeline.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Pipeline\ChildPipeline.cs:line 38
at Gateway.Pipeline.Extensions.ValueTaskExtensions.Await[T](ValueTask1 input) in D:\a\1\s\Proxy\Gateway.Pipeline\Extensions\ValueTaskExtensions.cs:line 29 at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable1 steps, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineWalker.cs:line 67
at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.PipelineExecutor.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineExecutor.cs:line 204, transportError: -131073, httpError: 0, correlationId: 24dd042d-1741-40e3-9fcb-ca5a5acd27ea�[0m
�[37m[Info] 2024-01-19T07:39:54.666 [GatewayLogs], isRequestSuccess: False, totalTime: 374, category: GatewayLogs, callerIpAddress: 10.129.2.26, timeGenerated: 2024-01-19T07:39:54.666, region: ocp-dev-east, correlationId: 24dd042d-1741-40e3-9fcb-ca5a5acd27ea, method: GET, url: https://apim-dxf-azure-apim.apps.ocp.dxf-services.com/roundrobin/, responseCode: 500, responseSize: 191, cache: none, backendTime: 371, apiId: roundrobin, operationId: rb, apimSubscriptionId: master, clientProtocol: HTTP/1.1, backendId: roundrobin, apiRevision: 1, clientTlsVersion: 1.2, backendMethod: GET, backendUrl: http://unused/, lastError: {
"elapsed": 373,
"source": "request-forwarder",
"path": "forward-request\forward-request\backend",
"reason": "BackendConnectionFailure",
"message": "Name does not resolve",
"section": "backend",
"transportErrorCode": -131073
}, errors: [
{
"elapsed": 373,
"source": "request-forwarder",
"path": "forward-request\forward-request\backend",
"reason": "BackendConnectionFailure",
"message": "Name does not resolve",
"section": "backend",
"transportErrorCode": -131073
}
], correlationId: 24dd042d-1741-40e3-9fcb-ca5a5acd27ea�[0m
�[37m[Info] 2024-01-19T08:37:55.576 [StorageCredentialsRequestStarted], source: ConfigurationApiStorageConnectionProvider�[0m
�[37m[Info] 2024-01-19T08:37:55.579 [StorageCredentialsRequestCompleted], message: Next refresh in 01:00:00., source: ConfigurationApiStorageConnectionProvider�[0m
�[37m[Info] 2024-01-19T09:37:55.579 [StorageCredentialsRequestStarted], source: ConfigurationApiStorageConnectionProvider�[0m
�[37m[Info] 2024-01-19T09:37:55.582 [StorageCredentialsRequestCompleted], message: Next refresh in 01:00:00., source: ConfigurationApiStorageConnectionProvider�[0m
�[37m[Info] 2024-01-19T10:37:55.582 [StorageCredentialsRequestStarted], source: ConfigurationApiStorageConnectionProvider�[0m
�[37m[Info] 2024-01-19T10:37:55.586 [StorageCredentialsRequestCompleted], message: Next refresh in 01:00:00., source: ConfigurationApiStorageConnectionProvider�[0m

Self-hosted Gateway Version

mcr.microsoft.com/azure-api-management/gateway:2.3.6

Deployment

Azure Arc

Platform

None

Kubernetes Version

None

Anything else?

No response

[ahalamir]

[tomkerkhove]

Change unused to google.com and it will work

[tomkerkhove]

Updating docs to avoid this, but basically it should not use that DNS.

PS: Load Balancer is not supported for self-hosted gateway yet

[daviian]

@tomkerkhove Do you have an idea when this feature will be available for the self-hosted gateway?

[tomkerkhove]

In the next release. You should already be able to use it, but it's not officially supported yet