https requestCert unusable with Firefox and Chrome

[plexel]

With the below code you can see a lot of what is going on when a node https server asks for a certificate from the client browser. I ran my tests on localhost, XP pro sp3 with node.exe 0.5.4 serving FF5, Chrome12, IE8, and Opera 11.5.

All the possible server and socket events cause printout to the console allowing a view of the client server negotiations.

What the tests suggest is that something quite bad is going on with node if requestCert is set true in the options object. With Firefox and Chrome a first client request is handled ok but subsequent requests fail to reach node because of socket problem(s) caused by node I believe.

The test results are below the code. Free server key and cert, and client cert were used from startSSL.com (startCom Ltd) which is an approved root CA in all the browsers tested.

serverEvents.js

var https = require('https'),
fs = require('fs')

var options = {
requestCert: true,
key: fs.readFileSync('serverKey.pem'),
cert: fs.readFileSync('serverCert.pem')
};

var server = https.createServer(options, function (req, res) {
var reply = 'OK'
res.writeHead( 200, {
'Content-Type': 'text/html',
'Content-length': reply.length,
'Connection': 'close'
})
res.write(reply)
res.end()
})

server.on('request', function(req, res) {
console.log("REQUEST")
})

server.on('connection', function(socket) {
console.log("\nSOCKET")
socket.on("connect", function(){console.log("connect")})
socket.on("data", function(data){console.log("data")})
socket.on("end", function(){console.log("end")})
socket.on("timeout", function(){console.log("timeout")})
socket.on("drain", function(){console.log("drain")})
socket.on("error", function(ex){console.log("error: ", ex)})
socket.on("close", function(had_error){console.log("close: ", had_error)})
})

server.on('close', function(errno) {
console.log("CLOSE: ", errno)
})

server.on('checkContinue', function(req, res) {
console.log("CHECK_CONTINUE")
})

server.on('upgrade', function(request, socket, head) {
console.log("UPGRADE");
})

server.on('clientError', function(ex) {
console.log("EXCEPTION: ", ex);
})

server.listen(443, function() {
console.log('Secure server running on port 443')
})

What I did for each browser

Console: node serverEvents.js (start https server)
Client: request https://localhost, accept all security and certificate prompts and finally see 'OK' in the browser window. Restart console serverEvents.js, take results from here.
Client: first request to https://localhost. Shows 'OK' in all browsers
Client: second request to http://localhost (reload browser). 'OK' in IE8 and Opera, but errors in Chrome and Firefox

Results:

Chrome 12 console output (my comments in brackets):

C:\Nodejs>node serverEvents.js
Secure server running on port 443

SOCKET
connect

SOCKET
connect
data
data
drain
drain
end
close: false
end
close: false

SOCKET
connect
data
drain
end
close: false

SOCKET
connect
data
drain
data
end

SOCKET
connect
drain
close: false
data
drain
data
drain
REQUEST (First request comes through allright)
data
end
drain
close: false

SOCKET
connect
data
close: false

(end of first request)

SOCKET
connect

SOCKET
connect
data
close: false
data
close: false

SOCKET
connect
data
close: false

(No second request ever gets to the server)

On the second request Chrome gives the following error message which is the same as noted by gasteve in the related issue at #1494.

"SSL connection error
Unable to make a secure connection to the server. This may be a problem with the server, or it may be requiring a client authentication certificate that you don't have.
Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error."

Why are there so many socket events? Further analysis with the Wireshark network protocol analyser suggests that node ignores the (Wireshark defined) SSL type client hello's that Chrome issues and only gives server hello's to SSLv2 and TLSv1.0 type client hello's. IE8 behaves much better because it adapts by issuing SSLv2 type if SSL type gets no response.

IE8 console output (my comments in brackets)

C:\Nodejs>node serverEvents.js
Secure server running on port 443

SOCKET
connect
data
drain
data
drain
REQUEST (First https request received ok)
data
close: false

SOCKET (IE8 SSL type client hello ignored by node for this socket event)
connect
data
close: false

SOCKET (IE8 issues SSLv2 type client hello accepted by node, so this request works)
connect
data
drain
data
drain
REQUEST (second https request fine and 'OK' appears in browser)
data
close: false

IE8 works nicely because it adapts. This doesn't let node off the hook though. Maybe node should accept SSL type client hello's.

Firefox 5 console output:

C:\Nodejs>node serverEvents.js
Secure server running on port 443

SOCKET (FF issues TLSv1.0 client hello for this socket and node replies with server hello)
connect
data
drain
data
drain
REQUEST
data
end
drain
close: false

(end of first https request, all ok so far)

SOCKET (FF issues SSL client hello, node does not reply with server hello)
connect
data
close: false

SOCKET (FF repeats SSL client hello)
connect
data
close: false

SOCKET (etc, etc)
connect
data
close: false

SOCKET
connect
data
close: false

SOCKET
connect
data
close: false

SOCKET
connect
data
close: false

SOCKET
connect
data
close: false

SOCKET
connect
data
close: false

SOCKET
connect
data
close: false

SOCKET
connect
data
close: false

(second https request never reaches node)

(FF gives up and issues the following error)

The connection was reset
The connection to the server was reset while the page was loading.
The site could be temporarily unavailable or too busy. Try again in a few moments.
If you are unable to load any pages, check your computer's network connection.
If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web.

Opera 11.5 console output:

C:\Nodejs>node serverEvents.js
Secure server running on port 443

SOCKET
connect
data
drain
data
data
drain
REQUEST (First https request OK)
data
close: false

SOCKET
connect
data
close: false

SOCKET
connect
data
drain
data
data
drain
REQUEST (Opera always asks for favicon.ico as well with every request)
data
close: false

(end of first https request)

SOCKET
connect
data
close: false

SOCKET
connect
data
drain
data
data
drain
REQUEST
data
close: false

SOCKET
connect
data
close: false

SOCKET
connect
data
drain
data
drain
REQUEST
data
close: false

(second https request completes OK)

Conclusions

Please note that the above Chrome and Firefox errors do not occur if requestCert is not set true in the options object.

Node is not therefore usable with Firefox or Chrome when requestCert is true. This appears to be a major flaw. Effectively the use of client certificate authentication with node is impractical if two major browsers fail.

Wireshark protocol analysis suggests the root problem is that node does not give server hello's for all types of client hello, in particular the 'SSL' type defined and reported by wireshark. SSLv2 and TLSv1.0 types are accepted. The above event analysis shows that something is very wrong indeed.

This issue is related to:
SSL Client Certificates Mandatory/Flakey #1494
Node doesn't respond to SSL client hello #1492

[plexel]

If you add the following below res.end() in the above, then no browser errors occur.

server.requestCert = false

In this case, after starting the server, the first request will fetch any client certificate and authorize it ok. Subsequent requests although not causing browser errors, will fetch the client certificate (using getPeerCertificate()), but not authorize it.

So not really a proper workaround, but at least something that doesn't generate errors.

The real problem is probably to be found in the low level node client/server handshaking routines.

[koichik]

With Chrome and Firefox, OpenSSL reports this error (thanks @nahi):

error:140D9115:SSL routines:SSL_GET_PREV_SESSION:session id context uninitialized

The problem is same as the link.

http://marc.info/?l=openssl-users&m=114803322529676&w=2

Workaround (disabled session cache):

diff --git a/src/node_crypto.cc b/src/node_crypto.cc
index 77550f5..58de972 100644
--- a/src/node_crypto.cc
+++ b/src/node_crypto.cc
@@ -147,8 +147,8 @@ Handle<Value> SecureContext::Init(const Arguments& args) {

   sc->ctx_ = SSL_CTX_new(method);
   // Enable session caching?
-  SSL_CTX_set_session_cache_mode(sc->ctx_, SSL_SESS_CACHE_SERVER);
-  // SSL_CTX_set_session_cache_mode(sc->ctx_,SSL_SESS_CACHE_OFF);
+  // SSL_CTX_set_session_cache_mode(sc->ctx_, SSL_SESS_CACHE_SERVER);
+  SSL_CTX_set_session_cache_mode(sc->ctx_,SSL_SESS_CACHE_OFF);

   sc->ca_store_ = NULL;
   return True();
@@ -520,6 +520,9 @@ int Connection::HandleSSLError(const char* func, int rv) {
     return 0;

   } else {
+    if (err == SSL_ERROR_SSL) {
+      err = ERR_peek_last_error();
+    }
     static char ssl_error_buf[512];
     ERR_error_string_n(err, ssl_error_buf, sizeof(ssl_error_buf));

[nahi]

When you pass err to ERR_error_string_n() to get error string, you need to pass the resulting value of ERR_peek_last_error() always (not the value of SSL_get_error()). And you'd better check SSL_ERROR_NONE to return 0 as well.

Note: older version of OpenSSL does not have ERR_peek_last_error(). For such environment, you should call ERR_peek_error() instead. I don't know what version of OpenSSL are supported by Node.js though.

[plexel]

Anyone have an idea when this issue could be resolved? In the meantime should a note not be put in the manual at http://nodejs.org/docs/v0.5.4/api/tls.html#tls.createServer that requestCert is unusable so that people do not lose time on it.

[koichik]

@pquerna @nahi - Can you review 58916ca (for master) and 7326444 (for v0.4)?

@bnoordhuis @piscisaureus - with --use-uv, test/simple/test-tls-session-cache.js fails.

verify error:num=18:self signed certificate
verify return:1
depth=0 /C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
verify return:1
node: src/uv-unix.c:850: uv__read: Assertion `!(0 + ((ev_watcher *)(void *)(&stream->read_watcher))->active)' failed.

The result is the same even if I do not modify lib/tls.js and src/node_crypto.cc.
Can you check it?

[bnoordhuis]

@koichik: Confirmed. I'll look into it this week.

[plexel]

Firefox problem with requestCert confirmed for node v0.5.5-pre on Ubuntu 11.04. 1st https request works but second one gives "The connection was reset" error as reported above.

[koichik]

@plexel - Can you test with 58916ca (for master) or 7326444 (for v0.4)?
The problem was fixed both Firefox and Chrome in my environment.

[plexel]

@koichik

Your 58916ca (master) mods tested on node v0.5.5-pre using Ubuntu 11.04 with FF6 and Chrome 12. All ok, thanks, great work.

7326444 (v0.4) mods tested on node v0.4.12-pre with FF6. Ok. Not tested with Chrome (I forgot).

Here are the outputs I got for the modified master with two https requests per browser:

Firefox 6

SOCKET
connect
data
data
Client certificate authorised Gecko: true
REQUEST (1st request for https://localhost:8443)
data
close: false

SOCKET
connect
data
Client certificate authorised Gecko: true
REQUEST (2nd request - browser reload)
data
close: false

Chrome 12

SOCKET
connect

SOCKET
connect
data
data
end
close: false

SOCKET
connect
end
close: false

SOCKET
connect
data
data
data

SOCKET
connect
end
close: false
data
data
Client certificate authorised Chrome: true
REQUEST (1st request for https://localhost:8443)
data
end
close: false
close: false

SOCKET
connect
data
data
end
close: false

SOCKET
connect
data
data
Client certificate authorised Chrome: true
REQUEST (favicon.ico request)
data
close: false

SOCKET
connect

SOCKET
connect
data
data
data
end
close: false
data
end
close: false

SOCKET
connect
data
data
Client certificate authorised Chrome: true
REQUEST (2nd request - browser reload)
data
close: false

SOCKET
connect
data
data
end
close: false

SOCKET
connect
data
data
Client certificate authorised Chrome: true
REQUEST (favicon.ico request)
data
close: false

[Note: can't use https port 443 directly with node on Ubuntu unless running as root which is not recommended]

So node with Chrome still uses a lot of sockets (why?) and they don't seem to close properly, but no fatal errors as before and the certificate getting verified every time. Firefox works fine, with no errors.

Could I ask how long till your mods might be available on the node.exe Windows download?

Thanks again very much, I had little hope of fixing it myself.

[koichik]

@plexel - Thanks for your test.

Could I ask how long till your mods might be available on the node.exe Windows download?

Ah... I need LGTM from a core member before merging it.
And I am no expert on the OpenSSL, I want @pquerna to review it.

[bnoordhuis]

@koichik: the test in 58916ca works with today's master (b5643cb) with and without --use-uv.

[koichik]

@bnoordhuis - Great! Can you review this? @pquerna might be busy.

[koichik]

@ry - Can you review 1d37ad0?

[koichik]

@bnoordhuis - Can you review 29b8680? (I will write another patch for NPN/SNI).

[koichik]

Rebased and added docs. Can some one review 3393026?

[bnoordhuis]

@koichik - I'll review it. I was half-way through 29b8680 when I got side-tracked by other stuff, sorry about that.

[koichik]

@bnoordhuis - Don't mind it. I know that you work hard for libuv stuff.

[koichik]

@bnoordhuis - Thanks for your review!! I updated: 8987be5

[koichik]

@bnoordhuis - Thanks again, updated: 19a8553

[bnoordhuis]

@koichik - LGTM. Good work.

[koichik]

@bnoordhuis - Thank you for spending a looooooong time!! ＼(＾o＾)／