NTLM authentication sometimes broken by multiple WWW-Authenticate headers

[rmkerr]

This issue has been split off from #17545, which turned out to be a problem in the tool being used to observe network traffic. Other users saw similar results, but under different conditions and not caused by the testing tool. That issue will be tracked here to clearly separate the two issues.

The issue tracked here occurs with the following code, targeting .NET Core 2.0:

var creds = new CredentialCache();
creds.Add(new Uri(addy),"NTLM",new NetworkCredential(Username,Password));
var handler = new HttpClientHandler
{
	Credentials = creds,
};

HttpClient client = new HttpClient(handler);
client.BaseAddress = new Uri(addy);
            
var response = await client.GetAsync("api/myresource");

In Windows, the server receives the following headers, but does not initiate the NTLM handshake:

HTTP/1.1 401 Unauthorized
Content-Type: text/html
Server: Microsoft-IIS/8.5
WWW-Authenticate: NTLM
WWW-Authenticate: Negotiate
X-Powered-By: ASP.NET
Date: Thu, 01 Mar 2018 22:17:34 GMT
Content-Length: 1293

@dbrownxc, can you provide more information on the situation in which you were able to reproduce this issue? It would be good to have full logs for the unsuccessful authentication attempt.

cc: @seriouz @karelz @davidsh

[dbrownxc]

I am developing on Win7 using Visual Studio. My ultimate target is a ASP.NET Core application running in Docker hosted on CentOS. I am using .NET Core 2.0.

I am reproducing the issue with a unit test running as .NET Core 2.0 application. Here is my csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.0</TargetFramework>
  </PropertyGroup>

  <PropertyGroup>
    <!-- Assembly Info (for build replacement-->
    <Version>1.0.0.0</Version>
  </PropertyGroup>


  <ItemGroup>
    <!--Required for NUnit and TeamCity integration -->
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.5.0" />
    <PackageReference Include="NSubstitute" Version="3.1.0" />
    <PackageReference Include="NUnit" Version="3.9.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.9.0" />
    <PackageReference Include="FluentAssertions" version="5.0.0-beta0004" />
    <PackageReference Include="TeamCity.VSTest.TestAdapter" Version="1.0.6" />
  </ItemGroup>

  <ItemGroup>
  </ItemGroup>
 
</Project>

I'm running with Resharper in Visual Studio and can consistently get the issue when using NTLM as the type.

On the same machine, I have a CentOS VM running Docker and I'm using the following dockerfile and just issuing a docker image build. I've mapped the directory directly to the VM so I'm sure it is the same code.

FROM microsoft/aspnetcore-build:2.0 as build
ARG TEAMCITY_VERSION=2017.2
WORKDIR /solution
COPY . ./
RUN dotnet build
ENV TEAMCITY_VERSION $TEAMCITY_VERSION
RUN ls *Test*/*.csproj | xargs -L1 dotnet test 

The from the teamcity output, I'm seeing a success using the same code that fails in Windows. If I switch the type to Negotiate, I get the opposite results. Linux fails and Windows succeeds.

These calls are to the exact same IIS server and service.

[dbrownxc]

I am working on getting cleared to send the network trace, but I do notice a difference in the failures.

After the test receives the Unauthorized HTTP packet:
When Windows fails, the test sends an ACK and receives a RST, ACK almost immediately and closes.
When Linux fails, the test sends an ACK and then sends a FIN, ACK and receives a FIN,ACK, then ACKs to close.

These results are consistently reproduceable.

[dbrownxc]

Is it ok to send trace traces directly to you?

[rmkerr]

I'll have to check if that's okay and get back to you. In the mean time though, I have some questions you might be able to answer without having to worry about disclosure issues. The key question for me is about the client response to the 401.

1. Does the client make any attempt to authenticate? If so, which auth protocol is it using?
2. Is the behavior the same when you disable either NTLM or Negotiate on the server side?
3. When Negotiate works on Windows, is it using NTLM or Kerberos as the underlying auth protocol?

Unfortunately we're probably seeing two separate issues here, since authentication handling is completely different between Windows and Linux

[dbrownxc]

1. No. After the 401, I see an ACK, and then the server sends a RST, ACK.
2. I'll try that.
3. I see Negotiate, but it says NTLMSSP
   Server > 401 Unauthorized
   Client > NTLMSSP_NEGOTIATE.
   Server > NTLMSSP_CHALLENGE
   Client > NTLMSSP_AUTH

[karelz]

@dbrownxc is it possible to create a smaller repro on your side from which you can publish the traces (and source)?

[dbrownxc]

Yes. I'll get one out later today.

[karelz]

@dbrownxc any update on the repro? Our chances to fix it in 2.1 are slowly going down. Thanks!

[dbrownxc]

HttpClientErrorRepro.zip

Sorry! It got busy around here.
To test on Windows I just run the test against an IIS hosted web site in visual studio
To test on Docker, I run

docker build .

I make it always fail (hack) so I see the returned output even if the call succeeds.

[karelz]

@rmkerr can you please take a look?

[rmkerr]

Yep! Thanks for getting around to sending out the repro @dbrownxc -- it makes things way more manageable on our end.

[rmkerr]

Something definitely looks wrong here. With NTLM credentials set on the client side, and both NTLM and Negotiate enabled on the server side, we don't even attempt to authenticate.

GET / HTTP/1.1
Connection: Keep-Alive
Host: localhost

HTTP/1.1 401 Unauthorized
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/10.0
WWW-Authenticate: Basic realm="localhost"
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM

After the exchange above, the client closes the connection to the server without even attempting to authenticate. If I leave NTLM enabled but disable Negotiate, it works. If I change the credential type to Negotiate (as provided in the repro), it works.

I don't have time to investigate more today, but this looks pretty serious so I'll be back on it tomorrow.

[rmkerr]

I was concerned that this issue might lie lower down the stack, or that it was an issue of machine policy blocking NTLM. I've gone ahead and tested it directly with the WinHttp authentication API and I don't see the same result, so that isn't the case. Since the issue does not occur on Linux I expect that the problem is somewhere in our WinHttpHandler authentication code.

Besides that I haven't made a lot of progress here -- I'm still mostly collecting information to help me track down the issue.

[seriouz]

@rmkerr Do i understand you correctly: There is no issue with dotnet core (wether on linux nor on windows). The problem lies within iis based services like OWA which use WinHttpHandler?

[dbrownxc]

@rmkerr I'm seeing the mirror error on Linux where NTLM correctly works but Negotiate and Negotiate\NTLM combo never attempts to authenticate. It is completely possible that it is further down the stack, too, but just some extra information.

[rmkerr]

@seriouz -- I should have been more clear there. WinHttpHandler is part of .NET. It's the underlying implementation of HttpClientHandler on Windows. I think it is very likely that this is a .NET problem.

@dbrownxc Thanks for the info -- I'm not as familiar with our authentication setup on Linux, but I'll take a look.

[rmkerr]

I think I've tracked this issue down. Here are the conditions under which it will repro:

• There are at least two authentication schemes enabled on the server.
• The user only provides credentials that have an authentication scheme less secure⁽¹⁾ than the most secure option offered by the server.

In the conditions we see here, the user provides NTLM credentials, but the server supports both NTLM and Negotiate (which is considered more secure). WinHttpHandler erroneously chooses to attempt authentication with Negotiate. When we later detect that there are no credentials in the cache that support Negotiate, we close the connection.

This happens because the code we use to choose the authentication scheme only considers the schemes supported by the server, and not those supported by the client. If the we don't have credentials for the most secure protocol supported by the server, we will fail the authentication attempt.

You can see the code that chooses the authentication scheme below. The parameter supportedSchemes indicates the schemes supported by the server.
https://github.com/dotnet/corefx/blob/1de2b37722e0987eaea07bd8e23a3d78d4ea36b2/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpAuthHelper.cs#L374-L385
The fix for this is simple, and I have a tentative version working. I'm adding some additional tests now, and if all goes well I'll try to get the PR out this afternoon.

(1) Our implementation ranks schemes in the following order: Negotiate, NTLM, Digest, Basic

[dbrownxc]

Great news!

[rmadisonhaynie]

@rmkerr Do you have a recommended work around?

[dbrownxc]

@rmadisonhaynie
For us, it is a matter of switching between NTLM and Negotiate depending on environment.

For now, what we are doing is setting this in configuration. Using the new configuration classes, we just set up a development configuration (appSettings) that works for Windows and a release configuration that works for Linux (environment variable override works here).

[karelz]

Or you can use 2.1 (currently Preview2) where it is fixed.