Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Seperate packages for each system #4

Open
NoraCodes opened this issue Oct 6, 2015 · 12 comments
Open

Proposal: Seperate packages for each system #4

NoraCodes opened this issue Oct 6, 2015 · 12 comments

Comments

@NoraCodes
Copy link
Contributor

I suggest that we separate the ipfs package into packages for each IPFS subsystem:
* ipfs-api - Used to connect to Py-IPFS by applications. This is where most users should touch. From https://github.com/ipfs/python-ipfs-api
* ipfs-naming or ipfs-ipns - Provides access to the IPNS pulbic key infrastructure and naming system. Handles HRNs and mutable pointers
* ipfs-merkledag - Provides access to Merkledag for resolving paths
* ipfs-block - Provides access to the block exchange system for transferring data.
* ipfs-routing - Provides access to the routing system, probably mostly DHT and mdns
* ipfs-network - Provides access to the lowest level IPFS network transport functions, like NAT traversal/hole punching, encryption/signing, and multi/broadcasting

Client applications would initialize the stack from the bottom up: choose a transport and encryption with ipfs-network, connect to the DHT with ipfs-routing, then use ipfs-naming (ipfs-ipns) to find the location of desired content and ipfs-merkledag and ipfs-block to recall that data.
On the other hand, if an instance of IPFS is already running on the system (which is preferred), they could simply use ipfs-api to interact with it.

@BrendanBenshoof
Copy link
Contributor

I am a fan of using multiple packages, but I suggest doing so internally and not polluting the global namespace or a package manager with them. This is a standalone application, not a package.

ipfs.api and naming already exists in the form of https://github.com/ipfs/python-ipfs-api and since they are universal (can be used with any ipfs implementation) it does not make sense to put it in the same package/namespace as this project.

@jbenet
Copy link
Contributor

jbenet commented Oct 6, 2015

I think the more standalone packages that can be used by other things, the better. (Yes published to package managers). A convention could be used like in node ipfs: ipfs-<module>

In particular, all the network stuff, definitely make it separate packages. Name the main one libp2p -- and check with @diasdavid and I on it -- we're actively modularizing it and making the pieces even more useful


Sent from Mailbox

On Tue, Oct 6, 2015 at 9:07 AM, BrendanBenshoof notifications@github.com
wrote:

I am a fan of using multiple packages, but I suggest doing so internally and not polluting the global namespace or a package manager with them. This is a standalone application, not a package.

ipfs.api and naming already exists in the form of https://github.com/ipfs/python-ipfs-api and since they are universal (can be used with any ipfs implementation) it does not make sense to put it in the same package/namespace as this project.

Reply to this email directly or view it on GitHub:
#4 (comment)

@daviddias
Copy link
Member

I believe it would be also good to align organisation conventions across implementations (e.g. node-ipfs-bitswap, go-ipfs-bitswap, py-ipfs-bitswap and others all do the same thing and offer the same interface). So on this, I happy to bring more libp2p context, the IPFS network stack.

@NoraCodes
Copy link
Contributor Author

@diasdavid Would we then use submodules for those packages into this repository? My intent was to allow user-side or project-side swapping of each subpackage for either alternate Python implementations or for C, Go, or other implementations, for speed, while not breaking the API.

@daviddias
Copy link
Member

@SilverWingedSeraph the Node.js implementation is breaking everything into modules per repo https://github.com/ipfs/node-ipfs#roadmap. modularity and composability ftw :)

@NoraCodes
Copy link
Contributor Author

OK, I updated my comment to reflect the naming convention, and the proper use of ipfs-api. I like the idea of using modules per repo.

@BrendanBenshoof
Copy link
Contributor

We have solved a lot of the dht related problems over at UrDHT. The latest Kademlia DHT logic has not left the simulator and been implemented into UrDHT yet, but we can use it soon. I should test if my kademlia peer selection criteria plays well with the other nodes using the k-buckets approach

@NoraCodes
Copy link
Contributor Author

@diasdavid To clarify, do you mean we should create additional repos and use git submodules, or that we should make Python modules for each section in this repository? Sorry, the word "module" is a bit unclear here.

@daviddias
Copy link
Member

@SilverWingedSeraph I was pointing out that on the same train of thought that you proposed with I suggest that we separate the ipfs package into packages for each IPFS subsystem, we started to modularize the IPFS for the Node.js implementation and we intend to go through a similar process in the go implementation.

It would be good for the ecosystem to have the same level of modularity across implementations, so that a user of 'libp2p' in Node.js can have the same expectations when using its 'go', 'python', 'rust', etc implementations.

Now if the best option is to create several repos vs git submodules vs python modules, I believe that depends on what is defined as best practice for the python ecosystem (I honestly don't know what is the convention for Python).

@NoraCodes
Copy link
Contributor Author

I think it basically depends on whether or not we want to allow people to use our functionality piecemeal. If we want, say, ipfs-naming to work standalone, we should create separate repos and use submodules to bring them here. Otherwise, we can just use a structure like

ipfs/
    __init__.py
    ipfs-naming/
        __init__.py
    ipfs-routing/
        __init__.py
    ...

@sirMackk
Copy link
Contributor

Correct me if I'm wrong @diasdavid , but I believe the closest translation of what is happening in the nodejs implemention to what we could do here would be:

  • Each component in its own repository ie. python-ipfs-ipns.
  • Each repository contains a python package ie. python-ipfs-ipns/ipns/__init__.py
  • Each repository contains a setup.py so that each repo can easily be packaged into an egg/wheel and/or uploaded to pypi.

We can use this repository as the mothership and have it contain git submodules/subtrees.

I'm still reading through the go-ipfs code so my knowledge of the project is somewhat shoddy, but to me it looks like it'd be cool to split the functionality into multiple repositories and have each piece work in a stand-alone fashion for that ease of use in other projects.

@mvanveen
Copy link
Contributor

I'm worried about how this would affect imports, e.g.:

>>> from test.ipf-test import foo
  File "<stdin>", line 1
    from test.ipf-test import foo
SyntaxError: invalid syntax

The issue is - is not a valid identifier. If we want to preserve uniformity with the aesthetic of other repos I think this is a deal breaker.

My read on the general in the greater Python community is that it seems like modularizing sub-packages into separate repositories is fairly rare, possibly due to the perceived overhead with git submodules. Python also favors a lot of imports between sub-packages (as long as they aren't circular references 😏), so there aren't as many advantages to this structure out in the wild generally. I believe in JS and golang that sort of encapsulation offers more benefits.

The python-ipfs-api project seems to favor this pattern as well. Everything is currently namespaced into a master ipfsApi package.

I'm not completely against a separate repo, but I'm personally with @BrendanBenshoof and think we should keep everything in one repo.

Edit: re-reading, I'm not sure if @BrendanBenshoof is advocating for or against git submodules. I favor sub-package (i.e. Python package) encapsulation, but believe git submodules are probably unnecessary/more trouble than they're worth in this case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants