SPNEGO logon fails from linux client against windows server

[Aarthiumayaval]

Attaching client and server application compiled in netcoreapp2.0 . Authentication is successful from windows client and failed from linux client. The server is windows . In the client code I used impersonation level as System.Security.Principal.TokenImpersonationLevel.Identification for linux as this is only supported. From windows I used System.Security.Principal.TokenImpersonationLevel.Impersponation

From linux client , I see the below error
[root@sdl17146 publish]# dotnet krbclient.dll

Unhandled Exception: System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> System.ComponentModel.Win32Exception: GSSAPI operation failed with error - An invalid status code was supplied (Message stream modified).
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Net.Security.NegoState.StartSendAuthResetSignal(LazyAsyncResult lazyResult, Byte[] message, Exception exception)
   at System.Net.Security.NegoState.StartSendBlob(Byte[] message, LazyAsyncResult lazyResult)
   at System.Net.Security.NegoState.CheckCompletionBeforeNextSend(Byte[] message, LazyAsyncResult lazyResult)
   at System.Net.Security.NegoState.ProcessReceivedBlob(Byte[] message, LazyAsyncResult lazyResult)
   at System.Net.Security.NegoState.StartReceiveBlob(LazyAsyncResult lazyResult)
   at System.Net.Security.NegoState.CheckCompletionBeforeNextReceive(LazyAsyncResult lazyResult)
   at System.Net.Security.NegoState.StartSendBlob(Byte[] message, LazyAsyncResult lazyResult)
   at System.Net.Security.NegoState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.NegotiateStream.AuthenticateAsClient(NetworkCredential credential, ChannelBinding binding, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel)
   at System.Net.Security.NegotiateStream.AuthenticateAsClient(NetworkCredential credential, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel)
   at krbclient.ASynchronousAuthenticatingTcpClient.Main(String[] args)
Aborted (core dumped)

From windows client:

C:\Users\rl151007.ESROOTDOM\aarthi>krbclient.exe
Client waiting for authentication...
IsAuthenticated: True
IsMutuallyAuthenticated: False
IsEncrypted: True
IsSigned: True
IsServer: False
Sent 22 bytes.
Client closed.
krbclient.cs

using System;
using System.IO;
using System.Net;
using System.Net.Security;
using System.Net.Sockets;
using System.Text;

namespace krbclient
{
    public class ASynchronousAuthenticatingTcpClient
    {
        static TcpClient client = null;

        public static void Main(String[] args)
        {
            // Establish the remote endpoint for the socket.
            // For this example, use the local machine.
            //IPHostEntry ipHostInfo = Dns.GetHostEntry("localhost");
            //IPAddress ipAddress = ipHostInfo.AddressList[0];
            // Client and server use port 11000. 
            //IPEndPoint remoteEP = new IPEndPoint(ipAddress, 11000);
            // Create a TCP/IP socket.
            client = new TcpClient();
            // Connect the socket to the remote endpoint.
            client.Connect("esroot", 9999);
            //Console.WriteLine("Client connected to {0}.", remoteEP.ToString());
            // Ensure the client does not close when there is 
            // still data to be sent to the server.
            client.LingerState = (new LingerOption(true, 0));

            System.Security.Principal.TokenImpersonationLevel impLevel = System.Security.Principal.TokenImpersonationLevel.Identification;
                
             // Request authentication.
             NetworkStream clientStream = client.GetStream();
            NegotiateStream authStream = new NegotiateStream(clientStream, false);
            // Pass the NegotiateStream as the AsyncState object 
            // so that it is available to the callback delegate.
            /*  IAsyncResult ar = authStream.BeginAuthenticateAsClient(
                  new AsyncCallback(EndAuthenticateCallback),
                  authStream
                  );*/
            //NetworkCredential cred = new NetworkCredential("au185009", "@Thirukami9876");
            authStream.AuthenticateAsClient(CredentialCache.DefaultNetworkCredentials, "HOST/ESROOT",
       ProtectionLevel.EncryptAndSign,
       impLevel);

            StreamWriter writer = new StreamWriter(authStream);
            Console.WriteLine("Client waiting for authentication...");
            // Wait until the result is available.
            //ar.AsyncWaitHandle.WaitOne();
            // Display the properties of the authenticated stream.
            AuthenticatedStreamReporter.DisplayProperties(authStream);
            // Send a message to the server.
            // Encode the test data into a byte array.
            byte[] message = Encoding.UTF8.GetBytes("Hello from the client.");
            //ar = authStream.BeginWrite(message, 0, message.Length,
              //  new AsyncCallback(EndWriteCallback),
              //  authStream);
            //ar.AsyncWaitHandle.WaitOne();
            authStream.Write(message, 0, message.Length);


            Console.WriteLine("Sent {0} bytes.", message.Length);
            // Close the client connection.
            authStream.Close();
            Console.WriteLine("Client closed.");
        }
        // The following method is called when the authentication completes.
        public static void EndAuthenticateCallback(IAsyncResult ar)
        {
            Console.WriteLine("Client ending authentication...");
            NegotiateStream authStream = (NegotiateStream)ar.AsyncState;
            Console.WriteLine("ImpersonationLevel: {0}", authStream.ImpersonationLevel);

            // End the asynchronous operation.
            authStream.EndAuthenticateAsClient(ar);
        }
        // The following method is called when the write operation completes.
        public static void EndWriteCallback(IAsyncResult ar)
        {
            Console.WriteLine("Client ending write operation...");
            NegotiateStream authStream = (NegotiateStream)ar.AsyncState;

            // End the asynchronous operation.
            authStream.EndWrite(ar);
        }
    }

    // The following class displays the properties of an authenticatedStream.
    public class AuthenticatedStreamReporter
    {
        public static void DisplayProperties(AuthenticatedStream stream)
        {
            Console.WriteLine("IsAuthenticated: {0}", stream.IsAuthenticated);
            Console.WriteLine("IsMutuallyAuthenticated: {0}", stream.IsMutuallyAuthenticated);
            Console.WriteLine("IsEncrypted: {0}", stream.IsEncrypted);
            Console.WriteLine("IsSigned: {0}", stream.IsSigned);
            Console.WriteLine("IsServer: {0}", stream.IsServer);
        }
    }
}

krbserver.cs

using System;
using System.Net;
using System.Net.Security;
using System.Net.Sockets;
using System.Security.Authentication;
using System.Security.Principal;
using System.Text;
using System.Threading;

namespace krbserver
{

        public class Program
        {
            public static void Main()
            {
                // Create an IPv4 TCP/IP socket. 
                TcpListener listener = new TcpListener(IPAddress.Any, 9999);
                // Listen for incoming connections.
                listener.Start();
                while (true)
                {
                    TcpClient clientRequest = null;
                    // Application blocks while waiting for an incoming connection.
                    // Type CNTL-C to terminate the server.
                    clientRequest = listener.AcceptTcpClient();
                    Console.WriteLine("Client connected.");
                    // A client has connected. 
                    try
                    {
                        AuthenticateClient(clientRequest);
                    }
                    catch (Exception e)
                    {
                        Console.WriteLine(e);
                        continue;
                    }

                }

            }
            public static void AuthenticateClient(TcpClient clientRequest)
            {
                NetworkStream stream = clientRequest.GetStream();
                // Create the NegotiateStream.
                NegotiateStream authStream = new NegotiateStream(stream, false);
                // Save the current client and NegotiateStream instance 
                // in a ClientState object.
                ClientState cState = new ClientState(authStream, clientRequest);
                /* Listen for the client authentication request.
                authStream.BeginAuthenticateAsServer(
                    new AsyncCallback(EndAuthenticateCallback),
                    cState
                    );*/

            authStream.AuthenticateAsServer(CredentialCache.DefaultNetworkCredentials,
      ProtectionLevel.EncryptAndSign,
      TokenImpersonationLevel.Impersonation);

            Console.WriteLine("Client Identity", authStream.ToString());
            // Wait until the authentication completes.
            cState.Waiter.WaitOne();
                cState.Waiter.Reset();
                authStream.BeginRead(cState.Buffer, 0, cState.Buffer.Length,
                       new AsyncCallback(EndReadCallback),
                       cState);
                cState.Waiter.WaitOne();
                // Finished with the current client.
                authStream.Close();
                clientRequest.Close();
            }
            // The following method is invoked by the
            // BeginAuthenticateAsServer callback delegate.

            public static void EndAuthenticateCallback(IAsyncResult ar)
            {
                // Get the saved data.
                ClientState cState = (ClientState)ar.AsyncState;
                TcpClient clientRequest = cState.Client;
                NegotiateStream authStream = (NegotiateStream)cState.AuthenticatedStream;
                Console.WriteLine("Ending authentication.");
                // Any exceptions that occurred during authentication are
                // thrown by the EndAuthenticateAsServer method.
                try
                {
                    // This call blocks until the authentication is complete.
                    authStream.EndAuthenticateAsServer(ar);
                }
                catch (AuthenticationException e)
                {
                    Console.WriteLine(e);
                    Console.WriteLine("Authentication failed - closing connection.");
                    cState.Waiter.Set();
                    return;
                }
                catch (Exception e)
                {
                    Console.WriteLine(e);
                    Console.WriteLine("Closing connection.");
                    cState.Waiter.Set();
                    return;
                }
                // Display properties of the authenticated client.
                IIdentity id = authStream.RemoteIdentity;
                Console.WriteLine("{0} was authenticated using {1}.",
                    id.Name,
                    id.AuthenticationType
                    );
                cState.Waiter.Set();

            }
            public static void EndReadCallback(IAsyncResult ar)
            {
                // Get the saved data.
                ClientState cState = (ClientState)ar.AsyncState;
                TcpClient clientRequest = cState.Client;
                NegotiateStream authStream = (NegotiateStream)cState.AuthenticatedStream;
                // Get the buffer that stores the message sent by the client.
                int bytes = -1;
                // Read the client message.
                try
                {
                    bytes = authStream.EndRead(ar);
                    cState.Message.Append(Encoding.UTF8.GetChars(cState.Buffer, 0, bytes));
                    if (bytes != 0)
                    {
                        authStream.BeginRead(cState.Buffer, 0, cState.Buffer.Length,
                              new AsyncCallback(EndReadCallback),
                              cState);
                        return;
                    }
                }
                catch (Exception e)
                {
                    // A real application should do something
                    // useful here, such as logging the failure.
                    Console.WriteLine("Client message exception:");
                    Console.WriteLine(e);
                    cState.Waiter.Set();
                    return;
                }
                IIdentity id = authStream.RemoteIdentity;
                Console.WriteLine("{0} says {1}", id.Name, cState.Message.ToString());
                cState.Waiter.Set();
            }
        }
        // ClientState is the AsyncState object.
        internal class ClientState
        {
            private AuthenticatedStream authStream = null;
            private TcpClient client = null;
            byte[] buffer = new byte[2048];
            StringBuilder message = null;
            ManualResetEvent waiter = new ManualResetEvent(false);
            internal ClientState(AuthenticatedStream a, TcpClient theClient)
            {
                authStream = a;
                client = theClient;
            }
            internal TcpClient Client
            {
                get { return client; }
            }
            internal AuthenticatedStream AuthenticatedStream
            {
                get { return authStream; }
            }
            internal byte[] Buffer
            {
                get { return buffer; }
            }
            internal StringBuilder Message
            {
                get
                {
                    if (message == null)
                        message = new StringBuilder();
                    return message;
                }
            }
            internal ManualResetEvent Waiter
            {
                get
                {
                    return waiter;
                }
            }
        }
    
}

[EDIT] Fixing code and exception formatting by @karelz

[CalArabshahi]

@karelz and @davidsh Would you please describe/state whether .NET Core 2.1 supports Kerberos Authentication from a Linux Client against a service? IMO, there are lots of confusing statements/issues around Kerberos Authentication on Linux and macOS and no clear statement as to what works and what does not work. Please clarify. For example how does ".NET Data Provider for SQL Server" support Kerberos authentication on Linux. Does it revert back to P-Invoke calls; if yes why? We are trying to create a roadmap for our product and hope to make it work on all platforms. I think Kerberos Authentication is fundamental and it has been tested on Linux (e.g. Red-Hat) already. I hope we are missing something fundamental (e.g. how to populate the cache with a Kerberos TGT for). Thanks in advance for clarification and help with Kerberos authentication on Linux and eventually macOS.

[karelz]

@CalArabshahi it is a good question, but does not seem to be directly related to this issue. Let's not hijack it. Please file a new issue for discussion. @wfurt is the right expert.
In nutshell: Networking APIs in 2.1 support Kerberos. For SqlClient question we will likely need separate set of experts from the area to answer.

[CalArabshahi]

@karelz I will open a new issue. I work with @Aarthiumayaval. I hoped that you can provide guidance about this issue and whether it is really a bug without any workaround (i.e. Kerberos Authentication on Linux does not work). I think you have affirmed that .NET Core 2.1 supports Kerberos authentication on Linux but now we have to figure out how; and try to understand what is the bug in this specific case.

[karelz]

@CalArabshahi @Aarthiumayaval that is valuable information you are on the same team. The original post is about NegotiateStream, while your questions are more general + poking at SqlClient, so it was not clear to me.
Given the new information, there is no reason to have 2 issues. We just need to be specific what the questions are.

Kerberos has likely rough edges. We tested it only on HttpClient in the sense "it is possible to make it work with proper machine setup".
We didn't have many asks beyond that - this is likely the first one.

Anyway, I think it would be best to close this issue and continue the discussion in dotnet/corefx#30203. Is that ok?

[CalArabshahi]

@karelz dotnet/corefx#30203 is a question but this issue is a bug. I rather keep this issue open and close dotnet/corefx#30203. As you know there are tests for NegotiateStream ( #22855 ) but maybe they are not comprehensive; Anyway hence our confusion whether the NegotiateStream failures are specific one Linux distro or whether NegotiateStream does not work on Linux in general.

[CalArabshahi]

If you can help us debug through your scenario and tell us where things don't work as expected, we can likely help.

yes we are willing to help; but we have no idea where to start to debug "A call to SSPI failed, see inner exception. ---> System.ComponentModel.Win32Exception: GSSAPI operation failed with error - An invalid status code was supplied (Message stream modified)". I still think the issue is related to Kerberos ticket (i.e. CredentialCache.DefaultNetworkCredentials) that I eluded to in my earlier comment ("how to populate the cache with a Kerberos TGT"). Currently I am looking at SafeGssCredHandle.Create .

[karelz]

yes we are willing to help; but we have no idea where to start to debug "A call to SSPI failed, see inner exception. ---> System.ComponentModel.Win32Exception: GSSAPI operation failed with error - An invalid status code was supplied (Message stream modified)"

That's great. I would start by just debugging the failed call and step into CoreFX sources. First, try it out on 2.1 of course.

Closing to avoid duplicate discussions - let's use dotnet/corefx#30203 as main issue (we can flip it to bug once it is clear it is a bug and not just misconfiguration).