reading data via object get vs. cat vs. object data

[thisconnect]

Version information:

go-ipfs version: 0.4.10-4679f806
Repo version: 5
System version: amd64/darwin
Golang version: go1.8.3

using Nodejs v6.11.0
ipfs-api@14.1.1

Type:

bug

Severity:

High?

Description:

trying to read the data of a PNG (38,994 bytes) in nodejs via js-ipfs-api and go-ipfs.

observations:

• api.object.get returns a JSON object with a 'data' property containing a buffer with 72014 elements.
• api.object.data returns a buffer with 39004
• api.cat concating all chunks of the stream into a single buffer gives 38994 (correct!)

questions:

• is this a bug / should they all return the same data?
• what kind of buffer/array is the data field of the result of api.object.get and why is it so big?
• api.object.data seems to have a few bytes in front
  • result.slice(6) seems to give the correct start, but I am unsure what to cut off the end

This is probably related to

• JSON in ipfs object get transform bytes > 127 into U+FFFD (replacement character �) #2454
• JSON encoding corrupts non-UTF-8 binary data #1582

Slicing off data (off the start/end) seems wrong and might break in the future if this is fixed. Any suggestion is appreciated.

Currently I try to

• receive the first bytes to guess what file type
• depending on the guess trying to stream the data (audio, video, picture)

[Stebalien]

TL;DR, you probably don't want to use the object APIs.

IPFS files are stored in DagProtobuf IPLD nodes with the following format:

{
    "data": [bytes],
    "links": [ link1, link2, link3 ] // Links to other IPLD nodes.
}

In IPFS, the "data" field is usually another protocol buffer (encoded into raw bytes).

So, what's happening here?

• api.object.get is returning a JSON representation of the DagProtobuf IPLD node with the data field encoded (raw binary and JSON don't play well together). The encoding here is a known and the discussion is currently happening at Handle IPLD changes proposed by @davidar and @mildred ipld/specs#39 (follow the trail of links if you want more context...).
• api.object.data is returning the raw bytes of the data field (no encoding). This is slightly larger than the file itself because it's actually a protocol buffer that stores some metadata along with the file.
• api.cat returns the actual file (as bytes).

[thisconnect]

Thanks so much for your explanation. Sorry I wanted to reply earlier..

The discussion is kind of hard to follow

• Handle IPLD changes proposed by @davidar and @mildred ipld/specs#39
  • moved from Proposed IPLD changes specs#111 (closed)
    • Blocks don't have a nice size #2053 - allow raw data in leaf nodes (closed)
    • JSON encoding corrupts non-UTF-8 binary data #1582 - use EJSON for encoding (non-UTF8) binary data (open)
  • Add EJSON spec specs#116 Add EJSON spec (closed)

Is it correct that in the future we could use dag abi, with either CBOR or EJSON that would have better support for binary?

https://github.com/ipfs/ipfs/blob/master/IMPLEMENTATION_STATUS.md#dag

Should I close this issue, duplicate of #1582 ?

[Stebalien]

The discussion is kind of hard to follow

I agree, sorry about that.

Is it correct that in the future we could use dag abi, with either CBOR or EJSON that would have better support for binary?

Yes. The object API will eventually be replaced with the dag API anyways. Here is the latest state of things: https://github.com/ipfs/interface-ipfs-core/issues/81#issuecomment-322756354

However, for what you're doing, you probably want to just use api.cat to cat the file as-is.

Should I close this issue, duplicate of #1582 ?

Yes. All of these issues are known issues, it's just a simple matter of fixing them 😄.

[thisconnect]

thanks a lot!