Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http: optimize by corking outgoing requests #7946

Closed
wants to merge 2 commits into from

Conversation

brendanashworth
Copy link
Contributor

@brendanashworth brendanashworth commented Aug 2, 2016

Checklist
  • make -j4 test (UNIX), or vcbuild test nosign (Windows) passes
  • commit message follows commit guidelines
Affected core subsystem(s)

http

Description of change

This commit optimizes outgoing HTTP responses by corking the socket upon each write and then uncorking on the next tick. By doing this we get to remove a "shameful hack" that is years old while at the same time fixing #7914 and getting some performance improvements.

The second commit is just a cleanup commit that makes nothing do even less nothing.

benchmarks!
                                                           improvement significant      p.value
 http/simple.js c=50 chunks=0 length=1024 type="buffer"         1.79 %             6.312062e-01
 http/simple.js c=50 chunks=0 length=1024 type="bytes"         -0.05 %             9.835961e-01
 http/simple.js c=50 chunks=0 length=102400 type="buffer"       5.62 %             1.250185e-01
 http/simple.js c=50 chunks=0 length=102400 type="bytes"      504.10 %         *** 3.837119e-12
 http/simple.js c=50 chunks=0 length=4 type="buffer"            6.68 %          ** 9.582778e-03
 http/simple.js c=50 chunks=0 length=4 type="bytes"            -5.70 %           * 3.114749e-02
 http/simple.js c=50 chunks=1 length=1024 type="buffer"         2.64 %             4.060173e-01
 http/simple.js c=50 chunks=1 length=1024 type="bytes"          7.69 %           * 2.072578e-02
 http/simple.js c=50 chunks=1 length=102400 type="buffer"       1.05 %             6.860244e-01
 http/simple.js c=50 chunks=1 length=102400 type="bytes"       43.97 %         *** 1.209619e-11
 http/simple.js c=50 chunks=1 length=4 type="buffer"            2.05 %             4.243028e-01
 http/simple.js c=50 chunks=1 length=4 type="bytes"             2.94 %             2.278701e-01
 http/simple.js c=50 chunks=4 length=1024 type="buffer"         3.99 %             1.028562e-01
 http/simple.js c=50 chunks=4 length=1024 type="bytes"         93.05 %         *** 6.741354e-16
 http/simple.js c=50 chunks=4 length=102400 type="buffer"      -0.67 %             8.158624e-01
 http/simple.js c=50 chunks=4 length=102400 type="bytes"       21.27 %         *** 2.395868e-05
 http/simple.js c=50 chunks=4 length=4 type="buffer"            1.20 %             7.638681e-01
 http/simple.js c=50 chunks=4 length=4 type="bytes"            83.91 %         *** 9.042857e-12
 http/simple.js c=500 chunks=0 length=1024 type="buffer"        3.91 %             2.187250e-01
 http/simple.js c=500 chunks=0 length=1024 type="bytes"        -3.62 %             2.884252e-01
 http/simple.js c=500 chunks=0 length=102400 type="buffer"      4.06 %             1.118611e-01
 http/simple.js c=500 chunks=0 length=102400 type="bytes"     462.59 %         *** 1.329366e-14
 http/simple.js c=500 chunks=0 length=4 type="buffer"           3.36 %             1.562332e-01
 http/simple.js c=500 chunks=0 length=4 type="bytes"           -6.84 %           * 2.868192e-02
 http/simple.js c=500 chunks=1 length=1024 type="buffer"        1.44 %             6.530913e-01
 http/simple.js c=500 chunks=1 length=1024 type="bytes"         7.24 %          ** 1.642590e-03
 http/simple.js c=500 chunks=1 length=102400 type="buffer"     -1.07 %             7.202359e-01
 http/simple.js c=500 chunks=1 length=102400 type="bytes"      38.69 %         *** 1.560892e-07
 http/simple.js c=500 chunks=1 length=4 type="buffer"           0.22 %             9.431102e-01
 http/simple.js c=500 chunks=1 length=4 type="bytes"            2.24 %             2.752881e-01
 http/simple.js c=500 chunks=4 length=1024 type="buffer"        4.81 %             1.307765e-01
 http/simple.js c=500 chunks=4 length=1024 type="bytes"        86.55 %         *** 9.028564e-15
 http/simple.js c=500 chunks=4 length=102400 type="buffer"      0.91 %             7.931741e-01
 http/simple.js c=500 chunks=4 length=102400 type="bytes"      11.95 %          ** 1.007196e-03
 http/simple.js c=500 chunks=4 length=4 type="buffer"           3.75 %             1.063908e-01
 http/simple.js c=500 chunks=4 length=4 type="bytes"           77.90 %         *** 2.854701e-10

CI: https://ci.nodejs.org/job/node-test-pull-request/3498/

@nodejs-github-bot nodejs-github-bot added the http Issues or PRs related to the http subsystem. label Aug 2, 2016
@mscdex mscdex added the performance Issues and PRs related to the performance of Node.js. label Aug 2, 2016
@addaleax
Copy link
Member

addaleax commented Aug 2, 2016

CI failed with some possibly related failures:

@ronkorving
Copy link
Contributor

Makes you wonder if this block could benefit from being changed to multiple sends like we do with Buffers:

chunk = len.toString(16) + CRLF + chunk + CRLF;
ret = this._send(chunk, encoding, callback);

@brendanashworth
Copy link
Contributor Author

I do think those failures are related, so I'm going to do some looking into those. @ronkorving good idea — the string concatenation could be a problem. I might try that, thanks!

I'm having slight second thoughts about my approach to this. I think the idea of corking and uncorking on the next tick (like #2020) should be in streams rather than HTTP so I'm going to put together an alternative PR with that in mind. This sort of stuff should also be available to TLS and such.

@jasnell
Copy link
Member

jasnell commented Aug 3, 2016

@nodejs/http @nodejs/streams

@mcollina
Copy link
Member

mcollina commented Aug 4, 2016

Maybe this might benefit from some string concatenation optimizations, e.g. https://github.com/davidmarkclements/flatstr. cc @davidmarkclements.

@brendanashworth #2020 is unrelated to this case. I agree that uncork is not the perfect API, but the cost of changing it would be very high, and we might discuss that in another issue (feel free to open!)

if (this.socket) {
// If the socket is corked from a write, uncork it before destroying it.
if (this.socket._writableState.corked)
this.socket.uncork();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not correct. Multiple cork() call increase the corked counter: https://github.com/nodejs/node/blob/master/lib/_stream_writable.js#L226-L230.

This needs to be a for loop that calls uncork() until corked is 0.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK that would be a backwards-incompatible change. It would change behavior in something like this:

(req, res) => {
  res.cork();
  res.write('should not write');
  res.destroy();
}

Because Socket#destroy() stops all IO on the socket, it isn't supposed to flush the last message. With this commit, writing to the stream will only cork it once, and thus we only uncork once, to leave the dev's previous cork-intent there. Looping until we uncork fully would write the message which I don't believe happens right now (edge case, yes 😛 )

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm lost. Why are you uncorking on destroy then?
Probably we shouldn't.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm uncorking on destroy because, without it, it broke a test — leaving it corked would be backwards-incompatible as well. Unfortunately this is part of a gray area. There isn't any guarantee that the message would be flushed regardless, as net.js may not be able to write it synchronously. If the message is buffered to be written asynchronously, and res.destroy() is called before that happens, the message will not be written at all. It's an odd gray area, but this is the best way to keep code breakage down.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which test was failing?

IMHO not transmitting the data when destroy() is hit is the correct behavior of destroy(). destroy is a dangerous operation anyway. @mafintosh what do you think?

Beware that, if the socket was corked twice, we need to call enough uncork() for this to take into effect.

Copy link
Contributor Author

@brendanashworth brendanashworth Oct 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO not transmitting the data when destroy() is hit is the correct behavior of destroy().

Yes, that's how it behaves right now. .write() can however flush the data synchronously if the socket is ready, so this has undefined behavior:

res.write('this may or may not send');
res.destroy();

The data may or may not be flushed. This has been around in node forever.

Beware that, if the socket was corked twice, we need to call enough uncork() for this to take into effect.

That's not what I'm doing here — we add only a single cork, so we uncork only once. If the user wants to cork the socket, write to it and destroy it, we shouldn't flush the message. That would be backwards-incompatible.

(edit): which test was failing?

test/parallel/test-http-abort-client.js fails without this change.

Copy link
Member

@mcollina mcollina Oct 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO not transmitting the data when destroy() is hit is the correct behavior of destroy().
Yes, that's how it behaves right now. .write() can however flush the data synchronously if the socket is ready, so this has undefined behavior:

res.write('this may or may not send');
res.destroy();

The behavior of destroy() is something we should define and clarify throughout core.
I'm ok with this particular change. Can you add a reference to the failing unit test in the comment? Otherwise it will be extremely hard to make the connection when looking at the code.

Beware that, if the socket was corked twice, we need to call enough uncork() for this to take into effect.
That's not what I'm doing here — we add only a single cork, so we uncork only once. If the user wants to cork the socket, write to it and destroy it, we shouldn't flush the message. That would be backwards-incompatible.

In this PR, every time write() is called, this.connection.cork() is called. It is entirely possible that _writableState.corked is greater that one.

res.write('a')
res.write('b')
res.destroy() // will not uncork

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The behavior of destroy() is something we should define and clarify throughout core.

Not so much destroy(), it's just the socket.write() API is weird. It returns a boolean about whether or not it works like a sync or async function. It doesn't necessarily have to be fixed, it just has to be worked around because test/parallel/test-http-abort-client.js and userland doesn't always use it perfectly. I'll add a better comment.

It is entirely possible that _writableState.corked is greater that one.

Ah, I think you're right. I'll fix that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's just the socket.write() API is weird. It returns a boolean about whether or not it works like a sync or async function.

This is not true. socket.write() is consistent from the point of view of the stream. It returns true if the user can keep writing, and false otherwise. From an implementation perspective, it will return true for a while after "operations become asynchronous" (that's a simplification) because of the buffering.

Add a unit test for the "write-write-destroy" use case, similar to test/parallel/test-http-abort-client.js.

@mcollina
Copy link
Member

mcollina commented Aug 4, 2016

Good work! I think it is a good contribution!

@brendanashworth
Copy link
Contributor Author

@mcollina thanks for your review! I like your judgement on the streams api, I might open an issue once this PR is 👍

@mcollina
Copy link
Member

mcollina commented Aug 4, 2016

I think this should be semver-major.
It's probably safe to be semver-minor, but I prefer to be safe than sorry, and we are cutting v7 soon anyway.

Any other opinion?

@jasnell
Copy link
Member

jasnell commented Aug 4, 2016

I happily defer to your judgement on the semveriness of it. Marking semver-major to be safe.

@jasnell jasnell added the semver-major PRs that contain breaking changes and should be released in the next major version. label Aug 4, 2016
@addaleax
Copy link
Member

@mcollina
Copy link
Member

@addaleax we are still waiting for some nits to be fixed. However, I would love to see this lands on v7.

@brendanashworth how are you with this? Do you need any help?

@brendanashworth
Copy link
Contributor Author

@mcollina doing good! It's not so much the nits that I need to spend time working on, but I need to investigate the test failures. I just haven't had the time to ask rvagg for access to one of the nodes yet — I'll try to get it working within the week.

@ronkorving
Copy link
Contributor

@brendanashworth Any updates on the suggestion I made?

@jbergstroem
Copy link
Member

Just checking in! Keen on progress updates.

@mcollina
Copy link
Member

mcollina commented Oct 7, 2016

@brendanashworth I would love to see this landed. Would you like somebody else to continue your work? Have you have more time to work on this?

@jbergstroem
Copy link
Member

I can facilitate access to any test host if need be.

Since CI results are long gone, I rebased (no conflicts) and started a new run: https://ci.nodejs.org/job/node-test-commit/5483/

@brendanashworth
Copy link
Contributor Author

Sorry about the lack of updates — I'll be doing the rest of the work on this over the long weekend 😄 happy to see that there's interest in landing this!

Copy link
Member

@indutny indutny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if CI is green and benchmarks are good!

Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work!

LGTM with some nits to be fixed regarding comments.

// This is a shameful hack to get the headers and first body chunk onto
// the same packet. Future versions of Node are going to take care of
// this at a lower level and in a more general way.
// Send the headers before the body. OutgoingMessage#write should cork
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you remove "should" here? They are corked.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it 👍

} else if (data.length === 0) {
this._flushOutput(connection);

// Don't bother with an empty message.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you rephrase this? Something like "avoid writing an empty buffer"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure!

Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the confusion, not LGTM yet. There is the discussion around _writableState.corked to be finalized.

@brendanashworth
Copy link
Contributor Author

@mcollina thank you for your very thorough review 😅 please take another look. @ronkorving thanks for your suggestion earlier — I've incorporated it into the latest changes.

Regarding the CI and other platforms — there were problems with test-http-client-upgrade2 on freebsd and folks which seem to have gone away on the latest run. I'm not sure whether that's because its flaky or because its fixed...

Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to be picky, but connectionCorkNT is needed

process.nextTick(() => {
this.connection.uncork();
this._corkedForWrite = false;
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do not allocate a closure here, it will slow things down considerably. Use connectionCorkNT as it was done before.

@brendanashworth brendanashworth force-pushed the http-fix-cork branch 2 times, most recently from c109972 to ec42085 Compare October 17, 2016 03:04
@rvagg rvagg force-pushed the master branch 2 times, most recently from c133999 to 83c7a88 Compare October 18, 2016 17:02
This commit opts for a simpler way to batch writes to HTTP clients into
fewer packets. Instead of the complicated snafu which was before, now
OutgoingMessage#write automatically corks the socket and uncorks on the
next tick, allowing streams to batch them efficiently. It also makes the
code cleaner and removes an ugly-ish hack.
The first change in `_writeRaw`:
This reduces drops an unnecessary if check (`outputLength`),
because it is redone in `_flushOutput`. It also changes an if/else
statement to an if statement, because the blocks were unrelated.

The second change in `write`:
This consolidates code in #write() that handled different
string encodings and Buffers. There was no reason to handle the
encodings differently, so after splitting them based on Buffer vs
encoding, the code is consolidated. This might see a speedup. Shoutout
to Ron Korving <ron@ronkorving.nl> for spotting this.
@brendanashworth
Copy link
Contributor Author

brendanashworth commented Dec 14, 2016

Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if CI is green and the perf results are still confirmed.

@mscdex
Copy link
Contributor

mscdex commented Jan 7, 2017

I'm seeing regressions with strong significance when both chunks <= 1 and length <= 1024 for strings (mostly the 'bytes' type in the benchmark):

                                                                             improvement significant      p.value
 http/simple.js c=50 chunks=0 length=1024 type="buffer" benchmarker="wrk"         2.88 %         *** 1.972424e-07
 http/simple.js c=50 chunks=0 length=1024 type="bytes" benchmarker="wrk"        -11.54 %         *** 6.466690e-32
 http/simple.js c=50 chunks=0 length=102400 type="buffer" benchmarker="wrk"       2.57 %         *** 3.612979e-08
 http/simple.js c=50 chunks=0 length=102400 type="bytes" benchmarker="wrk"      213.15 %         *** 2.030384e-32
 http/simple.js c=50 chunks=0 length=4 type="buffer" benchmarker="wrk"            3.66 %         *** 7.068239e-10
 http/simple.js c=50 chunks=0 length=4 type="bytes" benchmarker="wrk"           -12.58 %         *** 5.493742e-35
 http/simple.js c=50 chunks=1 length=1024 type="buffer" benchmarker="wrk"        -1.17 %          ** 2.759235e-03
 http/simple.js c=50 chunks=1 length=1024 type="bytes" benchmarker="wrk"          0.50 %             2.348991e-01
 http/simple.js c=50 chunks=1 length=102400 type="buffer" benchmarker="wrk"      -0.66 %             7.578784e-02
 http/simple.js c=50 chunks=1 length=102400 type="bytes" benchmarker="wrk"      354.77 %         *** 3.996312e-36
 http/simple.js c=50 chunks=1 length=4 type="buffer" benchmarker="wrk"           -1.36 %          ** 2.483386e-03
 http/simple.js c=50 chunks=1 length=4 type="bytes" benchmarker="wrk"            -5.23 %         *** 3.465286e-14
 http/simple.js c=50 chunks=4 length=1024 type="buffer" benchmarker="wrk"         0.60 %             3.069889e-01
 http/simple.js c=50 chunks=4 length=1024 type="bytes" benchmarker="wrk"        775.19 %         *** 7.762663e-52
 http/simple.js c=50 chunks=4 length=102400 type="buffer" benchmarker="wrk"       0.36 %             3.421042e-01
 http/simple.js c=50 chunks=4 length=102400 type="bytes" benchmarker="wrk"       18.99 %         *** 1.534301e-32
 http/simple.js c=50 chunks=4 length=4 type="buffer" benchmarker="wrk"           -0.11 %             7.803265e-01
 http/simple.js c=50 chunks=4 length=4 type="bytes" benchmarker="wrk"           835.54 %         *** 7.120948e-51
 http/simple.js c=500 chunks=0 length=1024 type="buffer" benchmarker="wrk"        3.87 %         *** 4.717166e-10
 http/simple.js c=500 chunks=0 length=1024 type="bytes" benchmarker="wrk"       -11.26 %         *** 2.732567e-33
 http/simple.js c=500 chunks=0 length=102400 type="buffer" benchmarker="wrk"      2.71 %         *** 6.950788e-09
 http/simple.js c=500 chunks=0 length=102400 type="bytes" benchmarker="wrk"     230.10 %         *** 2.497100e-43
 http/simple.js c=500 chunks=0 length=4 type="buffer" benchmarker="wrk"           2.87 %         *** 1.664165e-07
 http/simple.js c=500 chunks=0 length=4 type="bytes" benchmarker="wrk"          -11.78 %         *** 1.783927e-30
 http/simple.js c=500 chunks=1 length=1024 type="buffer" benchmarker="wrk"       -1.56 %         *** 3.648309e-04
 http/simple.js c=500 chunks=1 length=1024 type="bytes" benchmarker="wrk"         0.59 %             9.217040e-02
 http/simple.js c=500 chunks=1 length=102400 type="buffer" benchmarker="wrk"     -0.50 %             2.265089e-01
 http/simple.js c=500 chunks=1 length=102400 type="bytes" benchmarker="wrk"     406.52 %         *** 2.627477e-51
 http/simple.js c=500 chunks=1 length=4 type="buffer" benchmarker="wrk"          -1.64 %         *** 1.109017e-04
 http/simple.js c=500 chunks=1 length=4 type="bytes" benchmarker="wrk"           -4.80 %         *** 1.372847e-14
 http/simple.js c=500 chunks=4 length=1024 type="buffer" benchmarker="wrk"        0.30 %             4.962431e-01
 http/simple.js c=500 chunks=4 length=1024 type="bytes" benchmarker="wrk"        24.65 %         *** 2.527061e-47
 http/simple.js c=500 chunks=4 length=102400 type="buffer" benchmarker="wrk"      0.31 %             5.864431e-01
 http/simple.js c=500 chunks=4 length=102400 type="bytes" benchmarker="wrk"      25.25 %         *** 4.540902e-49
 http/simple.js c=500 chunks=4 length=4 type="buffer" benchmarker="wrk"           0.17 %             6.059476e-01
 http/simple.js c=500 chunks=4 length=4 type="bytes" benchmarker="wrk"           23.39 %         *** 2.387182e-40

@jasnell jasnell added the stalled Issues and PRs that are stalled. label Mar 1, 2017
@ronkorving
Copy link
Contributor

@brendanashworth Any chance you could look into those performance regressions?

@mscdex
Copy link
Contributor

mscdex commented May 11, 2017

A rebase is needed.

@mcollina mcollina mentioned this pull request Jun 9, 2017
3 tasks
@jasnell
Copy link
Member

jasnell commented Aug 24, 2017

Ping. What do we want to do with this one?

@jasnell jasnell closed this Aug 24, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
http Issues or PRs related to the http subsystem. performance Issues and PRs related to the performance of Node.js. semver-major PRs that contain breaking changes and should be released in the next major version. stalled Issues and PRs that are stalled.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants