Expose and write headers on 1xx intermediate status codes

[awwright]

Background: I'm currently working on a document for standardizing how servers may communicate intermediate status of a long-running operation over HTTP (operations running minutes to days). Part of this involves use of 1xx status codes, available in HTTP/1.1 and HTTP/2.

HTTP specifies that a request may have multiple 1xx responses before a final 2xx-5xx response. These responses, like any other, may have headers:101 Switching Protocols, 102 Processing, and 103 Early Hints are all known to use headers to convey additional information about the intermediate status. (For example, 101 uses Upgrade, 102 uses Status-URI, and 103 uses Link).

There is currently seemingly no way to write or read these headers. Given the definition of 1xx (which is mandatory in HTTP/1.1 and HTTP/2), I would expect to be able to call ServerResponse#writeHead multiple times with a 1xx status code, however, this does not flush the headers to the response, and appears to cause an error to be thrown once the final headers are written! Node.js added ServerResponse#writeProcessing in v10.0.0, however, this is specific to a single status code, does not support headers, and is not forward compatible with future status codes.

I would also expect the ClientRequest#on("information") event to include a headers object, so I can read this data.

The only known workarounds are to use ServerResponse#_writeRaw which is not a public API; and the only available option for clients is to parse responses manually.

---

Here is a script demonstrating the expected behavior:

const http = require('http');
const server = http.createServer(handleRequest);
server.listen(0);
console.error('Listening on port '+server.address().port);

function handleRequest(req, res){
    var tasks = [
        'Herding cats',
        'Digging holes',
        'Filling in holes',
        'Making tea',
    ];
    function writeProgress(i){
        console.error('writeProgress('+i+')');
        if(false){
            res.writeHead(102, 'Processing', {
                'Progress': `${i}/${tasks.length} (${tasks[i]})`,
            });
        }else{
            res._writeRaw('HTTP/1.1 102 Processing\r\n');
            res._writeRaw('Progress: '+i+'/'+tasks.length+' ('+tasks[i]+')\r\n');
            res._writeRaw('\r\n');
        }
    }
    function next(){
        if(++i===tasks.length){
            res.setHeader('Content-Type', 'text/plain');
            res.end('All done!\r\n');
        }else{
            writeProgress(i);
            setTimeout(next, 100);
        }
    }
    var i = 0;
    writeProgress(i);
    setTimeout(next, 100);
}

var req = http.request({
    host: server.address().address,
    port: server.address().port,
    path: '/',
});
req.end();
req.on('information', function(res){
    console.log(res);
});
req.on('response', function(res){
    res.pipe(process.stdout);
    res.on('end', console.error);
});

When _writeRaw is used, the Node.js client sees only the statusCode:

$ node -v
v12.1.0
$ node demo.js 
Listening on port 49213
writeProgress(0)
{ statusCode: 102 }
writeProgress(1)
{ statusCode: 102 }
writeProgress(2)
{ statusCode: 102 }
writeProgress(3)
{ statusCode: 102 }
All done!

When _writeRaw is used, curl is able to parse the response as expected:

$ curl -v http://localhost:49213/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 49213 (#0)
> GET / HTTP/1.1
> Host: localhost:49213
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 102 Processing
< Progress: 0/4 (Herding cats)
< HTTP/1.1 102 Processing
< Progress: 1/4 (Digging holes)
< HTTP/1.1 102 Processing
< Progress: 2/4 (Filling in holes)
< HTTP/1.1 102 Processing
< Progress: 3/4 (Making tea)
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Date: Sun, 26 May 2019 21:02:58 GMT
< Connection: keep-alive
< Content-Length: 11
< 
All done!
* Connection #0 to host localhost left intact

However, when writeHead is used, the data never makes it out of the socket, and an error is emitted towards the end:

$ node -v
v12.1.0
$ node demo.js 
Listening on port 49224
writeProgress(0)
writeProgress(1)
writeProgress(2)
writeProgress(3)
_http_outgoing.js:467
    throw new ERR_HTTP_HEADERS_SENT('set');
    ^

Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ServerResponse.setHeader (_http_outgoing.js:467:11)
    at Timeout.next [as _onTimeout] (.../demo.js:28:17)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7)

$ curl -v http://localhost:49226/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 49226 (#0)
> GET / HTTP/1.1
> Host: localhost:49226
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

---

In summary:

• ClientRequest#on("information") should expose a res-like object with headers, and
• ServerResponse#writeHead should allow multiple calls with a 1xx status code that is immediately flushed to the socket.

[zero1five]

This is a good example for me, although http 1xx code itself represents a hehavior on the server side, it is not enough at a more granular level(as in this progress 'example).

ClientRequest#on("information") should expose a res-like object with headers

I'm willing to work for it.

ServerResponse#writeHead should allow multiple calls with a 1xx status code that is immediately flushed to the socket.

= 。 = I probably prefer what's discussed in "Make http.OutgoingMessage._writeRaw public" #22624 to this.

@node/http(This @ doesn't work very well) What do you think？

[awwright]

@zero1five #22624 is not a good solution for this issue. _writeRaw can implement some of the behavior I'm looking for, but it is not compatible with HTTP/2, it does not verify that the contents are well-formed, and can inject data at any point in the stream, whereas I'm looking to write a well-formed 1xx response only where it's allowed.

[markb-trustifi]

Maybe it is better just to add custom message to writeProcessing public function?

ServerResponse.prototype.writeProcessing = function writeProcessing(cb) {
  this._writeRaw(`HTTP/1.1 102 Processing${CRLF}${CRLF}`, 'ascii', cb);
};

[github-actions]

There has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.

For more information on how the project manages feature requests, please consult the feature request management document.

[awwright]

This is still desired.

[mcollina]

This is still desired.

There does not seem to be collaborators interested in picking it up. Given this has been opened for 3 years without much activity, I'd close it. Feel free to let us know if you would like to implement this yourself.

[Acconut]

I second that this method would a handy addition to Node;s API. Is there still interest in accepting a PR for this?

[awwright]

@Acconut Since this feature request is partially accepted (I wrote #28459) it may be best to file a new issue that deals exclusively with enabling multiple calls to response.writeHead. As for a PR, last time I took a look at it, Node.js had many optimizations in the header sending logic that are difficult to untangle.