stream: 'readable' have precedence over flowing #18994

mcollina · 2018-02-26T08:55:27Z

In Streams3 the 'readable' event/.read() method had a lower precedence
than the 'data' event that made them impossible to use them together.
This make .resume() a no-op if there is a listener for the
'readable' event, making the stream non-flowing if there is a
'data' listener.

Fixes: #18058

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

stream, http

mcollina · 2018-02-26T08:56:55Z

CI: https://ci.nodejs.org/job/node-test-pull-request/13369/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1308/

mscdex · 2018-02-26T11:07:55Z

Wouldn't removeAllListeners() to be handled as well, since there could be no 'removeListener' event handlers?

mcollina · 2018-02-26T11:11:41Z

@mscdex you are right. I didn't account for https://github.com/nodejs/node/blob/master/lib/events.js#L324-L336. Do you think I can just listen for 'removeListener' instead? I thought this approach could theoretically be more performing, but it's an edge case anyway.

mafintosh · 2018-02-26T11:45:12Z

@mcollina so basically just suspends flowing when readable is set?

mcollina · 2018-02-26T11:51:05Z

@mafintosh that's the goal yes. It also restores the behavior after 'readable' is removed.
I'm not sure this is something we want to land in this form (or we might want to refactor).
This is the fix for #18058.

mafintosh · 2018-02-27T12:34:49Z

@mcollina wondering if we should even fix this. the same behaivor described in the issue would happen if there are two consumers of the readable event anyway (which is what happens in flowing mode). not too much of a fan of implicit event listener side effects

mcollina · 2018-02-27T12:47:31Z

I am good with not fixing it. But it’s a discussion to have.

This fixed a major usability problem if you want to have both a on(‘data’) and on(‘readable’)/read() at the same time (with pipe) read() will always return null.

I’m ok if we want to remove the resume side effect when removing ‘readable’, as that might be confusing. However we should really be updating readableListening: this is also a bug without the fix for read() itself.

mafintosh · 2018-02-27T15:07:22Z

Removing the removeListener stuff but keeping the other part sounds like a good middleway to me 👍

BridgeAR · 2018-03-06T19:25:16Z

@nodejs/streams @mafintosh PTAL

BridgeAR · 2018-03-06T19:29:49Z

CI https://ci.nodejs.org/job/node-test-pull-request/13542/

mcollina · 2018-03-07T10:48:46Z

@nodejs/tsc what do you think?

An alternative approach might be to remove the readableListening variable and just use listenerCount, hopefully it would be fast enough.

vsemozhetbyt · 2018-03-14T14:32:54Z

doc/api/stream.md

@@ -801,6 +811,10 @@ In general, the `readable.pipe()` and `'data'` event mechanisms are easier to
 understand than the `'readable'` event. However, handling `'readable'` might
 result in increased throughput.

+If both `'readable'` and [`'data'`][]  are used at the same time, `'readable'`
+takes precedence in controlling the flow, i.e. `'data'` will be emitted
+only when [`stream.read()`][stream-read] is called


A nit: missing period.

mafintosh · 2018-03-14T15:39:41Z

test/parallel/test-stream3-pause-then-read.js

@@ -35,6 +35,7 @@ let expectEndingData = expectTotalData;
 const r = new Readable({ highWaterMark: 1000 });
 let chunks = totalChunks;
 r._read = function(n) {
+  console.log('_read called', chunks);


did you mean to add these logs?

Yes, they follow the prevailing pattern in this test file. Without the console.log statements, this is undebuggable.

mafintosh · 2018-03-14T15:41:39Z

Lots of good tests! LGTM from me 👍

mcollina · 2018-03-14T21:28:04Z

CI: https://ci.nodejs.org/job/node-test-pull-request/13685/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1331/

mcollina · 2018-03-14T21:28:46Z

Tagging @nodejs/tsc because it is semver-major.

lpinca · 2018-04-04T13:09:21Z

doc/api/stream.md

@@ -770,10 +774,16 @@ cause some amount of data to be read into an internal buffer.

 ```javascript
 const readable = getReadableStreamSomehow();
-readable.on('readable', () => {
+readable.on('readable', function() {


Nit: not sure why this was changed and it doesn't really matter but for consistency I would keep the arrow function.

@lpinca it is using this.read() below

Oops, ignore me.

mcollina · 2018-04-04T17:06:28Z

CI: https://ci.nodejs.org/job/node-test-pull-request/14051/

addaleax · 2018-04-04T17:17:16Z

doc/api/stream.md

+  - version: REPLACEME
+    pr-url: FILLMEIN
+    description: >
+      using 'readable' requires calling .read().


Can you avoid the line break and capitalize the sentence?

addaleax · 2018-04-04T17:18:02Z

doc/api/stream.md

+  - version: REPLACEME
+    pr-url: FILLMEIN
+    description: >
+      resume has no effect if there is a 'readable' event listening


addaleax · 2018-04-04T17:24:13Z

lib/_stream_readable.js

@@ -223,6 +223,7 @@ Readable.prototype.unshift = function(chunk) {
 };

 function readableAddChunk(stream, chunk, encoding, addToFront, skipChunkCheck) {
+  debug('readableAddChunk');


Might be good to have chunk printed out here as well

addaleax · 2018-04-04T17:26:16Z

lib/_stream_readable.js


  if (ev === 'data') {
+    // update readableListening
+    state.readableListening = this.listenerCount('readable') > 0;


Can you change to comment above this line so that it says why we do this, rather than what we do? I assume it’s about the this.resume() call below?

In Streams3 the 'readable' event/.read() method had a lower precedence than the `'data'` event that made them impossible to use them together. This make `.resume()` a no-op if there is a listener for the `'readable'` event, making the stream non-flowing if there is a `'data'` listener. Fixes: nodejs#18058

mcollina · 2018-04-04T21:11:58Z

@addaleax PTAL

mcollina · 2018-04-05T17:11:53Z

CI: https://ci.nodejs.org/job/node-test-pull-request/14077/
CITGM (master): https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1353/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1354/

mcollina · 2018-04-06T12:17:36Z

Landed as cf5f986

In Streams3 the 'readable' event/.read() method had a lower precedence than the `'data'` event that made them impossible to use them together. This make `.resume()` a no-op if there is a listener for the `'readable'` event, making the stream non-flowing if there is a `'data'` listener. Fixes: #18058 PR-URL: #18994 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net>

MylesBorins · 2018-04-11T03:59:09Z

Seems like failures in cf5f986#diff-573b8412079b987e160a3fde511d5b9b are showing up in a couple places

In a couple CI runs in #19201 and also in http://github.com/nodejs/node/pull/19924

@mcollina thoughts?

not ok 750 parallel/test-http-readable-data-event
  ---
  duration_ms: 0.725
  severity: fail
  stack: |-
    assert.js:79
      throw new AssertionError(obj);
      ^
    
    AssertionError [ERR_ASSERTION]: 'Hello World!Hello again later!' strictEqual 'Hello World!'
        at IncomingMessage.res.on.common.mustCall (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/parallel/test-http-readable-data-event.js:43:14)
        at IncomingMessage.<anonymous> (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/common/index.js:467:15)
        at IncomingMessage.emit (events.js:182:13)
        at IncomingMessage.Readable.read (_stream_readable.js:489:10)
        at IncomingMessage.res.on.common.mustCall (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/parallel/test-http-readable-data-event.js:36:20)
        at IncomingMessage.<anonymous> (/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/common/index.js:467:15)
        at IncomingMessage.emit (events.js:182:13)
        at emitReadable_ (_stream_readable.js:537:12)
        at process._tickCallback (internal/process/next_tick.js:174:19)
  ...

mcollina · 2018-04-11T06:02:42Z

@MylesBorins We are already tracking it in #19905.

Caused by nodejs/node#18994

Caused by nodejs/node#18994 Closes #26

nodejs-github-bot added the stream Issues and PRs related to the stream subsystem. label Feb 26, 2018

mcollina force-pushed the readable-and-data branch from c2fb53b to 53ef8f2 Compare February 26, 2018 08:55

mcollina added semver-major PRs that contain breaking changes and should be released in the next major version. dont-land-on-v4.x labels Feb 26, 2018

mcollina mentioned this pull request Feb 26, 2018

"readable" event behaving inconsistently #18058

Closed

jasnell approved these changes Feb 26, 2018

View reviewed changes

mcollina mentioned this pull request Mar 14, 2018

readline: add support for async iteration #18904

Closed

4 tasks

vsemozhetbyt reviewed Mar 14, 2018

View reviewed changes

mafintosh reviewed Mar 14, 2018

View reviewed changes

mcollina force-pushed the readable-and-data branch from 03e66ed to 44fad80 Compare March 14, 2018 21:17

mcollina requested review from lpinca and a team March 14, 2018 21:28

mcollina added the tsc-agenda Issues and PRs to discuss during the meetings of the TSC. label Apr 4, 2018

lpinca reviewed Apr 4, 2018

View reviewed changes

mcollina force-pushed the readable-and-data branch from 44fad80 to 8085e2e Compare April 4, 2018 15:58

mcollina added this to the 10.0.0 milestone Apr 4, 2018

Trott removed the tsc-agenda Issues and PRs to discuss during the meetings of the TSC. label Apr 4, 2018

addaleax reviewed Apr 4, 2018

View reviewed changes

mcollina force-pushed the readable-and-data branch from 8085e2e to 578a7b9 Compare April 4, 2018 21:11

addaleax approved these changes Apr 5, 2018

View reviewed changes

mcollina closed this Apr 6, 2018

mcollina deleted the readable-and-data branch April 6, 2018 12:17

This was referenced Apr 11, 2018

assert: fix actual & expected input #19925

Closed

deps: update V8 to 6.6 #19201

Closed

lpinca mentioned this pull request May 6, 2018

Behavior of readablestreams changed in v10 #20520

Closed

schnittstabil added a commit to schnittstabil/cp-file that referenced this pull request May 11, 2018

fix node 10 compatibility

2841594

Caused by nodejs/node#18994

sindresorhus pushed a commit to sindresorhus/copy-file that referenced this pull request May 12, 2018

Fix Node.js 10 compatibility

4957feb

Caused by nodejs/node#18994 Closes #26

mcollina mentioned this pull request Jun 2, 2018

Node read stream hangs on specific file, base64 encoding #19862

Closed

mcollina mentioned this pull request Jul 5, 2018

http request 'end' event not fired #21398

Closed

lundibundi mentioned this pull request Jul 30, 2018

ReadStream paused and can't resume, when listened readable event #22032

Closed

mcollina mentioned this pull request Aug 9, 2018

stream: restore flow if there are 'data' handlers after once('readable') #22209

Closed

4 tasks

mcollina mentioned this pull request Aug 21, 2018

fs.createWriteStream does not fire close event #21122

Closed

snakamura mentioned this pull request Nov 10, 2018

stream.resume() doesn't make the stream flowing after removing readable listener #24281

Closed

arantes555 mentioned this pull request Dec 7, 2018

Flow problem with node >= 10.10 juliangruber/multipipe#48

Closed

mcollina mentioned this pull request Nov 11, 2021

stream: adjust src hwm when pipelining #40751

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stream: 'readable' have precedence over flowing #18994

stream: 'readable' have precedence over flowing #18994

mcollina commented Feb 26, 2018

mcollina commented Feb 26, 2018 •

edited

Loading

mscdex commented Feb 26, 2018

mcollina commented Feb 26, 2018

mafintosh commented Feb 26, 2018

mcollina commented Feb 26, 2018

mafintosh commented Feb 27, 2018

mcollina commented Feb 27, 2018

mafintosh commented Feb 27, 2018

BridgeAR commented Mar 6, 2018

BridgeAR commented Mar 6, 2018

mcollina commented Mar 7, 2018

vsemozhetbyt Mar 14, 2018

mafintosh Mar 14, 2018

mcollina Mar 14, 2018

mafintosh commented Mar 14, 2018

mcollina commented Mar 14, 2018

mcollina commented Mar 14, 2018

lpinca Apr 4, 2018

mafintosh Apr 4, 2018

lpinca Apr 4, 2018

mcollina commented Apr 4, 2018

addaleax Apr 4, 2018

addaleax Apr 4, 2018

addaleax Apr 4, 2018

addaleax Apr 4, 2018

mcollina commented Apr 4, 2018

mcollina commented Apr 5, 2018

mcollina commented Apr 6, 2018

MylesBorins commented Apr 11, 2018

mcollina commented Apr 11, 2018

stream: 'readable' have precedence over flowing #18994

stream: 'readable' have precedence over flowing #18994

Conversation

mcollina commented Feb 26, 2018

Checklist

Affected core subsystem(s)

mcollina commented Feb 26, 2018 • edited Loading

mscdex commented Feb 26, 2018

mcollina commented Feb 26, 2018

mafintosh commented Feb 26, 2018

mcollina commented Feb 26, 2018

mafintosh commented Feb 27, 2018

mcollina commented Feb 27, 2018

mafintosh commented Feb 27, 2018

BridgeAR commented Mar 6, 2018

BridgeAR commented Mar 6, 2018

mcollina commented Mar 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mafintosh commented Mar 14, 2018

mcollina commented Mar 14, 2018

mcollina commented Mar 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcollina commented Apr 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcollina commented Apr 4, 2018

mcollina commented Apr 5, 2018

mcollina commented Apr 6, 2018

MylesBorins commented Apr 11, 2018

mcollina commented Apr 11, 2018

mcollina commented Feb 26, 2018 •

edited

Loading