New Streaming and Buffered Interfaces #162

daviddias · 2017-10-22T20:25:14Z

Files
- Review and declare new function signatures
- add
  - review spec
  - update tests
  - impl js-ipfs-api
  - impl js-ipfs
- addReadableStream
  - review spec
  - update tests
  - impl js-ipfs-api
  - impl js-ipfs
- addPullStream
  - review spec
  - create tests
  - impl js-ipfs-api
  - impl js-ipfs
- cat
  - review spec
  - update tests
  - impl js-ipfs-api
  - impl js-ipfs
- catReadableStream
  - review spec
  - create tests
  - impl js-ipfs-api
  - impl js-ipfs
- catPullStream
  - review spec
  - create tests
  - impl js-ipfs-api
  - impl js-ipfs
- get
  - review spec
  - update tests
  - impl js-ipfs-api
  - impl js-ipfs
- getReadableStream
  - review spec
  - create tests
  - impl js-ipfs-api
  - impl js-ipfs
- getPullStream
  - review spec
  - create tests
  - impl js-ipfs-api
  - impl js-ipfs
- ls
  - review spec
  - update tests
  - impl js-ipfs-api
  - impl js-ipfs
- lsReadableStream
  - review spec
  - create tests
  - impl js-ipfs-api
  - impl js-ipfs
- lsPullStream
  - review spec
  - create tests
  - impl js-ipfs-api
  - impl js-ipfs
- ~~merge addFromFS to add~~ will do in another PR
- ~~merge addFromUrl to add~~ will do in another PR

dignifiedquire · 2017-10-23T10:24:23Z

SPEC/FILES.md


 > Add files and data to IPFS using a transform stream.

 ##### `Go` **WIP**

-##### `JavaScript` - ipfs.files.createAddStream([options], [callback])
+##### `JavaScript` - ipfs.files.AddReadableStream([options], [callback])


please don't capitalize method names in JavaScript, that makes for some very surprising apis.

Beanow

Haven't had a moment to look at the tests yet, but some thoughts on the spec document.

Beanow · 2017-10-31T08:37:43Z

SPEC/FILES.md

+- a [`Buffer instance`][b]
+- a [`Readable Stream`][rs]
+- a [`Pull Stream`][ps]
+- a Path (caveat: will only work in Node.js)


If we're adding node only path strings, shouldn't we add browser only File instances? https://developer.mozilla.org/en-US/docs/Web/API/File

I think it would allow for great simplification and ample opportunity to do the right thing under the hood.

Sounds good!

Beanow · 2017-10-31T08:59:48Z

SPEC/FILES.md

+    throw err
+  }
+
+  console.log(file.toString())


This toString call is tricky as it assumes utf8 encoding of the binary data. Perhaps a comment about encoding or passing utf8 explicitly would be a good hint.

Beanow · 2017-10-31T09:04:51Z

SPEC/FILES.md

    console.log(file.path)
-    file.content.pipe(process.stdout)
+    console.log(file.path.toString())


Logging the path twice?
What about the path and filesize so we touched the content buffer as well.

Thank you for catching that!

daviddias · 2017-10-31T09:45:03Z

SPEC/FILES.md

@@ -52,142 +57,342 @@ const files = [
 ]


Really tempted to change the name of hash to cid.

daviddias · 2017-11-03T12:37:54Z

Spec and js-ipfs-api have been updated!

I've finished updating the spec and the implementation in js-ipfs-api. Next step is to implement the same interface in js-ipfs.

Please review as soon as possible to make sure that nothing gets missed. Overall I'm really happy with the simplicity of the new calls and I bet a lot more people will be able to use both modules without having to go through the learning of Node.js Streams and/or Pull Streams

dignifiedquire

Only reviewed the interface, not the code

pgte · 2017-11-13T10:06:53Z

SPEC/FILES.md

+```JavaScript
+const validCID = 'QmQ2r6iMNpky5f1m4cnm3Yqw8VSvjuKpTcK1X7dBR1LkJF'
+
+const stream = ipfs.files.getReadableStream(validCID)


Shouldn't this be ... ipfs.lsReadableStream(.. instead?

pgte · 2017-11-13T10:07:59Z

SPEC/FILES.md

+
+#### `lsPullStream`
+
+> Fetch a file or an entire directory tree from IPFS that is addressed by a valid IPFS Path. The files will be yielded as Readable Streams.


s/Readable Streams/pull stream ?

pgte · 2017-11-13T10:09:18Z

SPEC/FILES.md

+```JavaScript
+const validCID = 'QmQ2r6iMNpky5f1m4cnm3Yqw8VSvjuKpTcK1X7dBR1LkJF'
+
+const stream = ipfs.files.getReadableStream(validCID)


shouldn't this be ipfs.lsPullStream instead?

thanks for the catch! :)

pgte · 2017-11-13T10:11:20Z

SPEC/FILES.md

+```JavaScript
+const validCID = 'QmQ2r6iMNpky5f1m4cnm3Yqw8VSvjuKpTcK1X7dBR1LkJF'
+
+ipfs.files.ls(validCID, function (err, files) {


shouldn't this be ipfs.ls instead?

mitra42 · 2017-11-16T21:07:20Z

Does this fix the bug "c" from #164 which you commented was being worked on here?

Beanow · 2017-11-16T22:19:14Z

Would this be what you're looking for?

https://github.com/ipfs/interface-ipfs-core/blob/dd922523e32ae1d99ed63d1bd18d4e5cb7a20323/SPEC/FILES.md#get

const validCID = 'QmQ2r6iMNpky5f1m4cnm3Yqw8VSvjuKpTcK1X7dBR1LkJF'

ipfs.files.get(validCID, function (err, files) {
  files.forEach((file) => {
    console.log(file.path)
    console.log(file.content.toString('utf8')) // <-- file.content is a Buffer
  })
})

daviddias · 2017-11-17T09:49:58Z

@mitra42 yes :)

daviddias · 2017-11-17T17:41:34Z

mitra42 · 2017-11-17T22:01:40Z

@Beanow That case you've given is another case, i.e. where the hash refers to multiple files and having .content available would be a much nicer interface than having to use streams. Does that work if it was a single file. My two tests are with QmTds3bVoiM9pzfNJX6vT2ohxnezKPdaGHLd4Ptc4ACMLa (a file that should be 184324 bytes and therefore not sharded - and failed previous files.cat.
Qmbzs7jhkBZuVixhnM3J3QhMrL6bcAoSYiRPZrdoX3DhzB single sharded file which works in files.cat
And I also test with a simple json file uploaded with dag.put, which i think would break your files case as it will throw an "invalid node type" error.

ghost assigned daviddias Oct 22, 2017

ghost added the in progress label Oct 22, 2017

This was referenced Oct 22, 2017

Exposed Interfaces (Callbacks & Promises, node.js streams and pull-streams) ipfs/js-ipfs#557

Closed

Proposal for Standardising API that can return buffered results vs Node.js Streams vs Pull Streams #126

Closed

dignifiedquire reviewed Oct 23, 2017

View reviewed changes

daviddias mentioned this pull request Oct 27, 2017

How to get/cat a hash when don't know the type of the hash? ipfs/js-ipfs#1049

Closed

daviddias changed the title ~~WIP~~ New Streaming Interfaces Oct 30, 2017

daviddias mentioned this pull request Oct 30, 2017

docs(example): show folks how to use pull-streams instead ipfs/js-ipfs#988

Closed

daviddias changed the title ~~New Streaming Interfaces~~ New Streaming and Buffered Interfaces Oct 30, 2017

This was referenced Oct 30, 2017

Lack of hash -> Buffer interface #164

Closed

Implementing the new interfaces ipfs-inactive/js-ipfs-http-client#619

Merged

Beanow reviewed Oct 31, 2017

View reviewed changes

daviddias commented Oct 31, 2017

View reviewed changes

dignifiedquire approved these changes Nov 3, 2017

View reviewed changes

pgte reviewed Nov 13, 2017

View reviewed changes

pgte mentioned this pull request Nov 13, 2017

fixed .ls spec #168

Merged

daviddias mentioned this pull request Nov 14, 2017

Impl the http endpoints and cli cmds for PubSub ipfs/js-ipfs#1068

Closed

daviddias added 6 commits November 17, 2017 10:46

add function signatures

daf1914

fix typo

8bd8c77

complete update the spec

d43f80e

chore: update deps

946d612

structure tests, add placeholders

ef5f02f

apply cr

c622832

daviddias added 16 commits November 17, 2017 10:47

files.add done

b4354d3

.addReadableStream

f04cd2e

.addPullStream

53df9dc

files.cat

fcd7f74

.catReadableStream

135c30d

catPulLStream

4192ecf

.get

79373d0

getReadableStream

d231a49

chore: fix travis

394ff27

chore: add ipfs.ls spec

36d7f31

chore: update CI

90cc281

chore: fix CI and linting

abb7c01

fix directory test

9b2fb0f

add remaining tests for .ls .lsReadableStream and .lsPullStream

0b459fa

apply spec CR

5a8caea

enable all tests to fix ipfs-inactive/js-ipfs-http-client#339

f6aba4a

daviddias force-pushed the revamp-stream-apis branch from dd92252 to f6aba4a Compare November 17, 2017 11:11

daviddias mentioned this pull request Nov 17, 2017

Implementing the new interfaces ipfs/js-ipfs#1086

Merged

daviddias added 4 commits November 17, 2017 11:34

remove dup ls test

f24d83d

.add tested in js-ipfs

04cf417

get tests also passing on js-ipfs

9d4eef8

ls is implemented in js-ipfs too

8c2ab7b

daviddias merged commit 8f34b0e into master Nov 17, 2017

ghost removed the in progress label Nov 17, 2017

daviddias deleted the revamp-stream-apis branch November 17, 2017 17:41

daviddias mentioned this pull request Nov 20, 2017

⚡️ v0.27.0 RELEASE 🚀 ipfs/js-ipfs#1095

Closed

24 tasks

mitra42 mentioned this pull request Dec 17, 2017

List of API changes for app developers (e.g. files.cat)? #184

Closed

daviddias mentioned this pull request Mar 23, 2018

Error passing a filepath string to files.add ipfs-inactive/js-ipfs-http-client#722

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Streaming and Buffered Interfaces #162

New Streaming and Buffered Interfaces #162

daviddias commented Oct 22, 2017 •

edited

Loading

dignifiedquire Oct 23, 2017

daviddias Oct 23, 2017

Beanow left a comment

Beanow Oct 31, 2017

daviddias Oct 31, 2017

Beanow Oct 31, 2017

Beanow Oct 31, 2017

daviddias Oct 31, 2017

daviddias Oct 31, 2017

daviddias commented Nov 3, 2017

dignifiedquire left a comment

pgte Nov 13, 2017

pgte Nov 13, 2017

pgte Nov 13, 2017 •

edited

Loading

daviddias Nov 13, 2017

pgte Nov 13, 2017

mitra42 commented Nov 16, 2017

Beanow commented Nov 16, 2017

daviddias commented Nov 17, 2017

daviddias commented Nov 17, 2017

mitra42 commented Nov 17, 2017


		#### `lsPullStream`

		> Fetch a file or an entire directory tree from IPFS that is addressed by a valid IPFS Path. The files will be yielded as Readable Streams.

New Streaming and Buffered Interfaces #162

New Streaming and Buffered Interfaces #162

Conversation

daviddias commented Oct 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Beanow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias commented Nov 3, 2017

Spec and js-ipfs-api have been updated!

dignifiedquire left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgte Nov 13, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mitra42 commented Nov 16, 2017

Beanow commented Nov 16, 2017

daviddias commented Nov 17, 2017

daviddias commented Nov 17, 2017

mitra42 commented Nov 17, 2017

daviddias commented Oct 22, 2017 •

edited

Loading

pgte Nov 13, 2017 •

edited

Loading