Skip to content
This repository has been archived by the owner on Jul 21, 2023. It is now read-only.

refactor: async iterators #94

Merged
merged 18 commits into from
Sep 18, 2019
Merged

refactor: async iterators #94

merged 18 commits into from
Sep 18, 2019

Conversation

alanshaw
Copy link
Member

@alanshaw alanshaw commented Aug 15, 2019

@jacobheun this is https://github.com/alanshaw/it-mplex + the example from this repo updated to use the new API.

Note this includes an adapter class so it can be used immediately in JS IPFS when merged and released. I've run the JS IPFS tests and interop tests against it and they pass!

TODO:

Depends on:

BREAKING CHANGE: This module now has a completely new API based on async iterators. See the README for documentation.

Alan Shaw added 2 commits August 14, 2019 20:55
License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
Thanks @vasco-santos!

License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
Copy link
Contributor

@jacobheun jacobheun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some small comments. I am running the adapter through the benchmarks at https://github.com/libp2p/pull-mplex/tree/master/benchmarks to see what the difference is. I'm hitting some issues with the adapter benchmarks, I'm looking into it. Right now the bench test is really slow and the large file test hangs.

src/coder/decode.js Outdated Show resolved Hide resolved
src/mplex.js Outdated Show resolved Hide resolved
Alan Shaw and others added 5 commits August 15, 2019 13:18
Co-Authored-By: Jacob Heun <jacobheun@gmail.com>
License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
@jacobheun
Copy link
Contributor

@alanshaw the restrictSize fix took care of the issue with benchmarks. When I was running that the restrict size error ends up getting swallowed and never logged, only abort errors show in the logs. Can we have the source errors propagate up to the logs, they seem to be getting swallowed?

@alanshaw
Copy link
Member Author

I'll look into it - this still needs work clearly. The slow pingpong benchmark is also worrying.

Alan Shaw added 2 commits August 16, 2019 14:17
License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
@jacobheun
Copy link
Contributor

I used npm alias support to install the interface-stream-muxer@0.6 code along side the async test suite to test the adapter, as I was curious about the close tests. It's failing the close tests there as well (both close tests fail for the adapter), so I think there is something wrong with error/close propagation.

@alanshaw
Copy link
Member Author

@jacobheun I'm working on a fork of the libp2p-tcp async/await lib that's used in that test. I built https://www.npmjs.com/package/stream-to-it to ensure all cases were covered when a writable stream closes or errors. When libp2p-tcp is ready the test should pass and we should be gtg. My plan is then to circle back to the interface-connection PR.

License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
try {
this._headerInfo = this._decodeHeader(this._bufferProxy)
} catch (_) {
break // not enough data yet...probably
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate a bit more? The "probably" is not really comforting :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the comment for this. We should only hit the break in 2 instances.

  1. We haven't gotten enough data to parse a header (most common).
  2. The data we have can't be parsed, likely due to an invalid varint from bad data.

@jacobheun
Copy link
Contributor

Copy link
Contributor

@jacobheun jacobheun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added some comments/jsdocs to the code, removed the adapter (we dont need it), and fixed an issue with message logging. This should be all set.

Copy link
Member

@vasco-santos vasco-santos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying out the examples and I am having issues with the dialer, not sure why

@jacobheun
Copy link
Contributor

Fixed, the BufferList chunks need to be converted before going to node streams .write

Copy link
Member

@vasco-santos vasco-santos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@vasco-santos vasco-santos merged commit c9bede5 into master Sep 18, 2019
@vasco-santos vasco-santos deleted the refactor/async-iterators branch September 18, 2019 07:40
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants