Can you explain what thsi does (so we can help)

[mitra42]

Hi Juan pointed this out to me on ipfs-shipyard/py-ipfs#29 (comment) but its unclear to me (absence of Readme or comments or tests) which exact problem you are working on, if I could get clarity on that we can figure out whether we should be interfacing with this, building something else, or helping you finish it :-)

Problem I'm trying to solve is turning a file (on a disk, or more likely read via a HTTP call) into one or other version of the IPFS IPLD structures, i.e. something we can return a IPFS multihash to.

This is as part of a project to build a gateway between some of the Internet Archive collections and IPFS.

[dhruvbaldawa]

Hi @mitra42, sorry I just started working on this yesterday.
ipld/ipld#24 should give you some context.

As far as the current status goes, there is no usable implementation for now and I will be working on implementing a DAG and add protobuf support next. I have implemented the basic building blocks
required with:

• https://github.com/multiformats/py-multibase
• https://github.com/multiformats/py-multicodec
• https://github.com/ipld/py-cid

I am assuming that when you say "IPFS IPLD structures" you are referring to the MerkleDAG, which is under development in this repo. Does this answer your question or give you additional context?

[mitra42]

To be honest @dhruvbaldawa I'm not sure if that is what I mean because the terminology on IPFS is seriously confusing, I'm refering to the format in https://github.com/ipld/specs/tree/master/ipld and specifically, the part about taking a large file and sharding it into pieces and returning the JSON that represents that, its called a "chunked file" in that spec.

The goal of this project is to serve up a file, from our (Internet Archive) disk, (which we've found by other methods). Since IPFS cannot consume the entire file in one block we'll compute an IPLD (and return a multihash (e.g. /ipfs/Q.....). We'll cache that JSON structure, and enough info to retrieve the shards from the middle of the files.

I'd hoped to find some code that does this conversion. Ideally in Python or Javascript, but we'll convert from Go if necessary, though if we have to build that module as well, it will take time from the Hackathon that could be used to build the gateway.

[dhruvbaldawa]

@mitra42 I think this is what you are looking for - https://github.com/ipfs/js-ipfs-unixfs-engine

[mitra42]

Thanks - that is useful, presuming there is no Python implementation, do you know if there is a Go one, I'm guessing that Go may be easier to port to Python given JS's asynchronicities.

[dhruvbaldawa]

Yes, you can check the Go implementation over here -- https://github.com/ipfs/go-ipfs/tree/master/unixfs

[rvagg]

closing for archival