Azure App Service HTTP requests to Azure VNet IP Addresses fail after upgrading to .NET 5.0

[jonsagara]

I have an ASP.NET Core 3.1 application running in Azure App Service. It makes calls to Elasticsearch running on an Ubuntu Server 16.04 LTS Azure Virtual Machine, connected to the App Service via an Azure VNet. When I upgraded the application to target net5.0, all of my calls to Elasticsearch started failing with the following exception:

System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (10.1.0.5:9200)
 ---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.

Through trial and error, I eliminated Elasticsearch from the equation and reproduced the issue with simple HTTP requests via HttpClient:

public async Task<IActionResult> VNetRequest()
{
    string? body;
    try
    {
        using var requestMsg = new HttpRequestMessage(HttpMethod.Get, _testRequest.VNetEndpoint);
        using var responseMsg = await _httpClient.SendAsync(requestMsg);
        body = await responseMsg.Content.ReadAsStringAsync();
    }
    catch (Exception ex)
    {
        body = ex.ToString();
    }

    return Content(body);
}

When targeting netcoreapp3.1, this call will succeed and display some nginx boilerplate text.

When targeting net5.0, this call will fail with the following exception:

System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (10.1.0.4:80)
 ---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
   at AppServiceVNetCalls.Controllers.HomeController.VNetRequest() in D:\home\site\repository\src\AppServiceVNetCalls\Controllers\HomeController.cs:line 34

When I run these tests locally, I don't get any failures; everything works fine.

At this point, I'm stuck, and I don't know how where to go next.

Reproduction repository and Azure applications

I have a simple ASP.NET Core application that demonstrates the issue. This repository has two branches, net31 and net5:

• net31: This demonstrates that HttpClient GET requests to an Azure VNet IP address succeed. It also demonstrates that requests to a non-Azure VNet IP address succeed. See: https://appservicevnettest31.azurewebsites.net/
• net5: This demonstrates that HttpClient GET requests to an Azure VNet IP address fail. It also demonstrates that requests to a non-Azure VNet IP address succeed. See: https://appservicevnettest50.azurewebsites.net/

The architecture of my Azure reproduction is as follows:

• An Azure App Service hosting the ASP.NET applications. It's an S1 App Service Plan hosted in US West. They are Windows plans.
• An Azure Virtual Machine running Ubuntu Server 20.04 LTS on a Gen2 Standard B1s VM. I installed nginx solely to respond on port 80.
• The VM is connected to the App Service via an Azure VNet. As I am not a systems engineer, setting this up was laborious, and I'm not sure I could properly document it. However, it did require setting up an Azure Gateway VPN to connect the App Service to the VM.

Thank you.

ETA: Please let me know when you no longer need these VMs/App Services so that I can delete them. Thanks.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Description:	I have an ASP.NET Core 3.1 application running in Azure App Service. It makes calls to Elasticsearch running on an Ubuntu Server 16.04 LTS Azure Virtual Machine, connected to the App Service via an Azure VNet. When I upgraded the application to target net5.0, all of my calls to Elasticsearch started failing with the following exception: System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (10.1.0.5:9200) ---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions. Through trial and error, I eliminated Elasticsearch from the equation and reproduced the issue with simple HTTP requests via HttpClient: public async Task<IActionResult> VNetRequest() { string? body; try { using var requestMsg = new HttpRequestMessage(HttpMethod.Get, _testRequest.VNetEndpoint); using var responseMsg = await _httpClient.SendAsync(requestMsg); body = await responseMsg.Content.ReadAsStringAsync(); } catch (Exception ex) { body = ex.ToString(); } return Content(body); } When targeting netcoreapp3.1, this call will succeed and display some nginx boilerplate text. When targeting net5.0, this call will fail with the following exception: System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (10.1.0.4:80) ---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions. at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken) at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken) at AppServiceVNetCalls.Controllers.HomeController.VNetRequest() in D:\home\site\repository\src\AppServiceVNetCalls\Controllers\HomeController.cs:line 34 When I run these tests locally, I don't get any failures; everything works fine. At this point, I'm stuck, and I don't know how where to go next. Reproduction repository and Azure applications I have a simple ASP.NET Core application that demonstrates the issue. This repository has two branches, net31 and net5: net31: This demonstrates that HttpClient GET requests to an Azure VNet IP address succeed. It also demonstrates that requests to a non-Azure VNet IP address succeed. See: https://appservicevnettest31.azurewebsites.net/ net5: This demonstrates that HttpClient GET requests to an Azure VNet IP address fail. It also demonstrates that requests to a non-Azure VNet IP address succeed. See: https://appservicevnettest50.azurewebsites.net/ The architecture of my Azure reproduction is as follows: An Azure App Service hosting the ASP.NET applications. It's an S1 App Service Plan hosted in US West. They are Windows plans. An Azure Virtual Machine running Ubuntu Server 20.04 LTS on a Gen2 Standard B1s VM. I installed nginx solely to respond on port 80. The VM is connected to the App Service via an Azure VNet. As I am not a systems engineer, setting this up was laborious, and I'm not sure I could properly document it. However, it did require setting up an Azure Gateway VPN to connect the App Service to the VM. Thank you.
Author:	jonsagara
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[wfurt]

Do you have SELinux enabled?

[jonsagara]

I don't believe so. I'm using Windows App Service, not Linux.

[geroeinimo]

I have the same issue calling a service over vnet, this blocks the upgrade to net5.0 for us.

[kwesseling]

We have the same issue, it appears. Discovered a bizarre thing:

• deploy to the appservice where the 3.1 code used to be deployed: failure
• delete that appservice, recreate it with the same name and deploy to that new app service: failure
• deploy to a new appservice, with a different name: success
• deploy to a deployment slot (thus a different name): success

Azure App Service on a Windows service plan, running .net50

System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (sommeurl:443)
 ---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)

[kwesseling]

We found a solution: our app service was connected to our vnet through the gateway subnet. Disconnecting it and reconnecting the app to our vnet through a new subnet solved the issue.
For a short while.
Disconnecting (but not reconnecting) solves the issue, apparently vnet integration is the problem with the HttpClient in .Net 5.0.

Could someone please explain why .Net Core 3.1 does work and .Net 5.0 doesn't?

[antonfirsov]

@jonsagara and others facing this issue, any chance you can also raise a bug report on the Azure Portal feedback page (if doesn't exist yet):
https://feedback.azure.com/forums/223579-azure-portal/category/129061-bugs

At this moment it seems to us that the issue is more likely on their side, and they may be able to root cause it much faster since the investigation needs Azure infrastructural knowledge much more than anything else.

After opening it, can you share the link of the report so we can use it when contacting the Azure team?

[jonsagara]

Will do.

[jonsagara]

@antonfirsov Here you go: https://feedback.azure.com/forums/223579-azure-portal/suggestions/41965039-azure-app-service-http-requests-to-azure-vnet-ip-a

[kwesseling]

Did you see this:
https://github.com/MicrosoftDocs/azure-docs/issues/45991

I got a ticket with MS Support, and indeed the issue is that some client libraries don't force IPv4 (as TcpClient) as the virtual NIC installed on the App Services workers (for regional VNet integration) don't support IPv6.

[wfurt]

If IPv6 is not supported, I would expect it would not be advertised via DNS. If both protocols families are available, TcpClient should keep trying all available entries.

[kwesseling]

Found a workaround: set the url to the ipv4 address, add a header with "Host: domain.name" to the request and .net 5.0 httpclient can connect to a host running on the vnet.
Just noted that this only works when using the public ip-address of the VM that I need to connect to, not the private one.

[davebronson]

This issue is affecting me as well, and has halted our .NET 5 upgrade effort at work. Has anyone opened a support incident with Microsoft for this? Or is the bug report on the Azure Portal feedback page sufficient for triggering timely action on their end?

[antonfirsov]

We are looking into this with the help of Azure App Services team, but it's too early to provide a plan or ETA. Will update as soon as we can.

[antonfirsov]

According to our current (probably incomplete?) understanding, the root cause of the issue is that App Service currently does not support dual-stack sockets with VNET integration.

A workaround idea (specific to .NET 5) is to utilize ConnectCallback and enforce creating IPV4 sockets:

public HttpClient CreateWorkaroundClient()
{
    SocketsHttpHandler handler = new SocketsHttpHandler
    {
        ConnectCallback = IPv4ConnectAsync
    };
    return new HttpClient(handler);

    static async ValueTask<Stream> IPv4ConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
    {
        // By default, we create dual-mode sockets:
        // Socket socket = new Socket(SocketType.Stream, ProtocolType.Tcp);

        Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
        socket.NoDelay = true;

        try
        {
            await socket.ConnectAsync(context.DnsEndPoint, cancellationToken).ConfigureAwait(false);
            return new NetworkStream(socket, ownsSocket: true);
        }
        catch
        {
            socket.Dispose();
            throw;
        }
    }
}

@jonsagara @davebronson @kwesseling and others watching this issue: can you try if this workaround helps?

[jonsagara]

The connection is done within a third party library, NEST (https://github.com/elastic/elasticsearch-net). I'll see if they have extension points, though.

ETA: I asked about customizing HttpClient creation in NEST:

https://discuss.elastic.co/t/nest-customize-creation-of-httpclient/256687

[antonfirsov]

This is definitely a behavioral breaking change between 3.1 and 5.0, since we did not use Dual-Mode sockets before:

• https://github.com/dotnet/corefx/blob/release/3.1/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectHelper.cs#L49
• https://github.com/dotnet/corefx/blob/release/3.1/src/System.Net.Sockets/src/System/Net/Sockets/Socket.cs#L3960

As far as I can understand, the problem is that it can it happen in a given environment that despite the fact that IPV6 is supported by the OS (Socket.SupportsIPv6 returns true), dual-stack sockets are disallowed. We can probably try to detect such cases in the default Connect logic of SocketsHttpHandler and fall back to IPV4 sockets if dual mode is not available. @scalablecory @geoffkizer thoughts?

[kwesseling]

Hello and thank you for the example code. I can confirm it solves the problem for us. ```c# public class DOClient { public HttpClient HC { get; } public DOClient(IConfiguration config, HttpClient client) { var PersonalAccessToken = config["Tokens:AzureDevOps"]; var base64Token = Convert.ToBase64String(Encoding.ASCII.GetBytes($":{PersonalAccessToken}")); client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic", base64Token); client.DefaultRequestHeaders.Add("Accept", "application/json"); HC = client; } } public static void ConfigureBindings(IServiceCollection services) { services.AddHttpClient<DOClient>() .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler { ConnectCallback = IPv4ConnectAsync }); } static async ValueTask<Stream> IPv4ConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken) { // By default, we create dual-mode sockets: // Socket socket = new Socket(SocketType.Stream, ProtocolType.Tcp); Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp); socket.NoDelay = true; try { await socket.ConnectAsync(context.DnsEndPoint, cancellationToken).ConfigureAwait(false); return new NetworkStream(socket, ownsSocket: true); } catch { socket.Dispose(); throw; } } ``` Best, Karel Wesseling

[karelz]

How common do we think such environments are?
Should we view that AppService did something exceptional and rather unusual (in which case we should ask them to fix it there), or do we believe there will be more environments where it happens (in which case we should address it in .NET)?

[scalablecory]

Ugh, that's unfortunate. This is a trivial fix that we should do.

The behavior change you're seeing is because we stopped using the static Socket.ConnectAsync, and started using the instance variant. The static variant will create a non-dualmode Socket for each address family, while the instance variant operates just on the your specific Socket.

runtime/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs

Lines 1285 to 1300 in f066972

	private static async ValueTask<Stream> DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
	{
	Socket socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
	socket.NoDelay = true;
	try
	{
	await socket.ConnectAsync(context.DnsEndPoint, cancellationToken).ConfigureAwait(false);
	return new NetworkStream(socket, ownsSocket: true);
	}
	catch
	{
	socket.Dispose();
	throw;
	}
	}

The change will be pretty low risk. How common we think it is should dictate if it goes to servicing. I'm assuming Azure App Service users would be thankful, at least :).

[lahma]

I personally would love to see this in servicing release as we have so far identified three external libraries in use that hide the socket behavior and would all require changes in order for them to work in our app service to vNet scenario. We address our internal vNet servers/services via IP without using internal DNS.

[davebronson]

@jonsagara @davebronson @kwesseling and others watching this issue: can you try if this workaround helps?

Your workaround did indeed help in my scenario. Code is executing without error in our Azure app service now.

[karelz]

@antonfirsov given that this is an accidental regression in .NET 5 as @scalablecory points out above, can you please put up a fix in master for .NET 6?
We can then discuss if we need to service .NET 5 or not.

@lahma note that hot fix in .NET 5 is not necessarily the only / the fastest option. Key thing is to find out if App Service is the only environment this will ever happen. If they push fix on their side, servicing .NET 5 may not be needed.

[ericsampson]

This is the best short overview of its architecture that I know of. Hybrid Connections use the Azure Relay service. Someone from the Azure Messaging team could help with anything on the Azure server side.
https://azure.microsoft.com/en-us/resources/videos/azure-service-bus-relay-overview/

[superjulius]

@karelz Hybrid Connections are very similar to a VPN. The main difference is that there is no public IP exposed. Instead some services installed on-premises connects to an Azure Relay, which is "routing" the requests.

https://docs.microsoft.com/en-us/azure/app-service/app-service-hybrid-connections

[karelz]

Thanks for explanation. Azure App Service team realized they will need to do similar fix for Hybrid Connections as well (similar to the one for VPN).
I don't have answer from them yet on tracking status of "Is it fixed and deployed yet?" ... will let you know when/if I know.

[bhop2112]

@antonfirsov I'm responding to your request on twitter to provide additional information about the VPN configuration that is causing the problem. I'll provide as much detail as I am able and if you have follow up questions I can try to get answers from our IT team.

We are using a paloalto/GlobalProtect split tunnel VPN. When I initially encountered this issue, I inquired with our IT team whether or not our VPN could be configured to support IPv6 split tunneling. The response was that we don't have any IPv6 setup on that infrastructure, so it wasn't something they could easily do. I ran across this article, with details on the required configuration, so my assumption was they were either unable or unwilling to implement the suggested changes.

https://live.paloaltonetworks.com/t5/blogs/configure-globalprotect-and-ipv6/ba-p/329390

We have a separate Cisco AnyConnect full tunnel VPN that is working fine. I added a TelemetryListener and verified that with both VPNs the socket is attempting to connect via IPv6, which indicated to me the issue was not with the service, but with the VPN.

GlobalProtect:

Event = System.Net.Sockets - 1:ConnectStart
 - address: InterNetworkV6:28:{....}
Event Counter = HTTP Requests Started: 1

Event Counter = HTTP Requests Failed: 0
...

Cisco AnyConnect:

Event = System.Net.Sockets - 1:ConnectStart
 - address: InterNetworkV6:28:{....}
Event = System.Net.Sockets - 2:ConnectStop
Event = System.Net.Security - 1:HandshakeStart
...

After applying your workaround, I am able to connect via the GlobalProtect split tunnel VPN, and the telemetry data confirms that the socket is connecting via IPv4:

Event = System.Net.Sockets - 1:ConnectStart
 - address: InterNetwork:16:{....}
Event = System.Net.Sockets - 2:ConnectStop
Event = System.Net.Security - 1:HandshakeStart
...

Let me know if there is additional information that would be helpful.

[wfurt]

Note that the DualModde socket will still send IPv4 e.g. there is really no need to deploy IPv6. At least on Azure, the issue is how the VPN integrates with OS. Also disabling IPv6 on the system should have same impact @bhop2112 . There is probing and if IPv6 fails, the code will not try to use the dual mode logic.

[bhop2112]

@wfurt I currently have IPv6 disabled on all of my network adapters

[wfurt]

can you please try https://docs.microsoft.com/en-us/troubleshoot/windows-server/networking/configure-ipv6-in-windows @bhop2112 ?

and something like

var s = new Socket( AddressFamily.InterNetworkV6,  SocketType.Stream, ProtocolType.Tcp );
s.DualMode = true;

if either one fail, we would fall-back automatically. Perhaps we can improve detection of IPv6 availability.

[JohnYoungers]

We're experiencing a similar issue in an aspnetcore 5 app when using:

services.AddMicrosoftIdentityWebApiAuthentication(Configuration, "AzureAd"); from Microsoft.Identity.Web

It appears one of the first things it'll do is attempt to pull down the openid details from https://login.microsoftonline.com/, which is resulting in this socket error.

Since it sounds like as of now I can not specify the app service to report ipv6 is disabled, is there a way to convince libraries such as this one to use ipv4?

[karelz]

The Azure App Service incident number is 219225320 if anyone works with support to get update / more information.

[EvanMachusakNCQA]

Further, if you have a single App Service Plan which has both an ASP.NET application and a NodeJS Azure Functions application assigned to it, upgrading the ASP.NET application to .NET 5 irreversibly breaks your NodeJS Azure Functions app if it has a VNet configuration set.

You receive this error in the host logs:
gRPC client connection info is missing or incorrect (--port is 0)

Disabling the VNet makes this issue go away, but obviously, we use VNet for a reason and can't turn it off.

When I say irreversibly, I mean that downgrading the .NET app back to .NET Core 3.1 does not solve the Azure Functions issue. Deleting the entire App Service Plan and recreating it also does not solve the issue. You have to create a brand new App Service Plan with a different name, and be careful not to put .NET 5 on it.

[karelz]

@EvanMachusakNCQA did you raise the problem with Azure App Service? Based on your description it seems to be different problem.
BTW: Did you mean reverting to .NET Core 3.1? Or what did you mean by ".NET 4"?

[EvanMachusakNCQA]

@karelz Yes, sorry, I meant .NET Core 3.1.

The specific situation we encountered is that we are using a single App Service Plan in our dev environment, so we have a .NET Core 3.1 App Service for LOB application "A" and a NodeJS Azure Functions app for LOB application "B" in the same App Service Plan.

When a developer for LOB app "A" tried to upgrade his .NET Core 3.1 app to .NET 5, he encountered the issue described in the ticket. But this also inadvertently broke application "B", because once .NET 5 is activated a single time on an App Service Plan, from that point forward, any service that uses .NET's new networking stack in any capacity and does not give you an explicit .NET tooling choice (e.g., NodeJS Azure Functions) will be broken.

It could be considered a separate issue, but I wanted to bring this information here to underscore the criticality of this problem. It doesn't just break your .NET Core app. It breaks your entire App Service Plan.

[JohnYoungers]

It looks like Microsoft has a fix per this comment: https://github.com/MicrosoftDocs/azure-docs/issues/45991#issuecomment-755582909

Adding the setting WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS=1 resolved this problem for our application

[jonsagara]

Confirmed on my test site. I added the setting to the App Service configuration, restarted the site, and the following call no longer throws:

https://appservicevnettest50.azurewebsites.net/Home/VNetRequest

[ericsampson]

Now I'm curious, is this a temporary workaround that's planned to be set to true permanently without users having to do anything?

[SFinistrosa]

Hi, we are facing the same issue using HC on our web app. Adding the flag did not solve the issue. Any idea?
Thanks

[karelz]

@SFinistrosa I would ask Azure App Service why it does not work for you.
@ericsampson also no idea, I would wait for Azure App Service official resolution ...

[jraadt]

We continue to have this issue with our App Services. Incredibly frustrating. The flag does not solve the issue. For testing we created a simple ASP.NET 5 web API in App Service and a NodeJS server running on a VM that simply returns "Hello World!". The App Service and VM are in our virtual network.

The ASP.NET web API has two endpoints. One uses HttpClient to call the NodeJS server. The other uses HttpClient as well but forced to be IPv4.

When I deploy the ASP.NET web API to a new App Service (on an App Service Plan V3) the simple HttpClient fails with An attempt was made to access a socket in a way forbidden by its access permissions. and the HttpClient with forced IPv4 returns "Hello World!" successfully. This behavior is the same regardless if I have set WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS = 1.

HOWEVER, I just tried to "downgrade" my test App Service Plan to V2 and BOTH HttpClient calls worked, again regardless of the flag. I flipped back and forth between an V2 and V3 app service plans and it always works for V2 and never for V3. Any ideas why?

[jraadt]

@karelz - Appreciate your help as we navigate this issue. How would we ask Azure App Service for help? Is that through the portal support features or do they have a GitHub issues type thing? I would like to voice my support that this is important for us to have fixed. And is there any way of tracking Azure App Service resolution on this? Thanks again.

[karelz]

@jraadt they don't have GitHub AFAIK. Support portal sounds reasonable (I don't know how that works though).

[antonfirsov]

We've got a confirmation from the Azure team regarding the fix:

The patch has been out since December 23rd – it contained two changes. One was the flag WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS (= 1 to enable, 0 to disable), which enables this new behavior, and the second was logic to detect if the app is running .NET5 in which case we’d enable this automatically (and this flag would be used only for disabling the behavior in case of regression). If customers are setting this flag manually to resolve themselves, it could be one of two things – either our .net5 detection logic is not working right for them, or it only started to work for them because by adding the setting they triggered a restart, which is necessary for the patch to take effect. Easy test is if they remove the app setting entirely and check if they are still mitigated.

According to our information, in their next release the switch will be enabled by default, and it will also contain a fix for Hybrid Connections, but we can not provide any ETA or promise on their behalf.

I'm closing this now, since the original issue is fixed, and I believe related problems are better tracked in separate issues.

Thanks again for the detailed reports, and all the amazing contribution we've seen here from the community!

[ericsampson]

Thank you very much for closing the circle here @antonfirsov , and to everyone involved in addressing the issue. Thanks to @karelz too for managing this ticket.
Cheers

[jonsagara]

I removed the WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS configuration setting, restarted the app, and confirmed that VNet requests still work.

I'm going to delete both of the test app services now.

Thank you @antonfirsov and everyone else who helped!

[Danthar]

I'd like to note, that we started deploying a netcore5 application to azure, January this month. And the auto detect logic from the Azure team does not work for us.
So to anyone who finds this issue. You might have to toggle the magic setting (WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS) yourself. Or utilize the workaround httpclient mentioned by @antonfirsov earlier in this thread.

Be aware though that if you do use the SocketHttpClient you will run into this problem: #31261 (TLDR, no AI Dependency logging on your http calls)