Skip to content
This repository has been archived by the owner on Jun 27, 2023. It is now read-only.

feat(Directory): Add EnumLinksAsync method #39

Merged
merged 6 commits into from
Oct 30, 2018

Conversation

hannahhoward
Copy link
Contributor

Goal

child of ipfs/kubo#5600
Implement an EnumLinksAsync method on Directory interface to support streaming version of ls command

Implementation

  • Add LinkResult type to unix-fs - is an ipld Link or an error
  • Add EnumLinksAsync method to Directory interface, returns channel of directory links or error
  • Add EnumLinksAsync method to Shard interface in HAMT, returns channel
    of directory links or error
  • EnumLinks method in Shard interface in HAMT uses EnumLinksAsync now
  • modify makeAsyncTrieGetLinks to use channel

For Discussion

  • Arguably an async link enumeration is sufficiently generic it could live in go-ipld-format, and LinkResult is a type that is not super specific to go-unixfs
  • However I would prefer to hold on putting LinkResult in go-ipld-format to a later commit where we a generalized out-of-order parallel walker to IPLD format (essentially moving over enumerateChildrenAsync from go-merkledag)
  • Once this is merged, will move on to adding streaming ls to go-ipfs

Copy link
Contributor

@schomatis schomatis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll take a closer look at the end of the week (left some comments for now) but this makes plenty of sense. 👍

hamt/hamt.go Outdated Show resolved Hide resolved
unixfs.go Show resolved Hide resolved
io/directory.go Outdated Show resolved Hide resolved
hamt/hamt.go Outdated Show resolved Hide resolved
io/directory.go Outdated
go func() {
defer close(linkResults)
for _, l := range d.node.Links() {
linkResults <- format.LinkResult{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should also select on ctx to avoid leaking goroutines (same in other places)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it. will do.

Copy link
Member

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few more nits, then LGTM

hamt/hamt.go Outdated
"github.com/spaolacci/murmur3"

format "github.com/ipfs/go-unixfs"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be above this group

unixfs.go Show resolved Hide resolved
io/directory.go Outdated
@@ -141,6 +146,26 @@ func (d *BasicDirectory) AddChild(ctx context.Context, name string, node ipld.No
return d.node.AddNodeLink(name, node)
}

// EnumLinksAsync returns a channel which will receive Links in the directory
// as they are enumerated, where order is not gauranteed
func (d *BasicDirectory) EnumLinksAsync(ctx context.Context) (<-chan format.LinkResult, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd drop the error from the signature since it's not used here (or other impls of this function)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fair

- Add LinkResult type to unix-fs - is an ipld Link or an error
- Add EnumLinksAsync method to Directory interface, returns channel of directory links or error
- Add EnumLinksAsync method to Shard interface in HAMT, returns channel
of directory links or error
- EnumLinks method in Shard interface in HAMT uses EnumLinksAsync now
- modify makeAsyncTrieGetLinks to use channel
@hannahhoward
Copy link
Contributor Author

@schomatis are you good with my responses? Can we go ahead and merge this PR?

Copy link
Contributor

@schomatis schomatis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I have one follow-up question but if this is blocking something else go ahead and merge.

Copy link
Member

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM module the @schomatis comment (not blocking for me)

@hannahhoward
Copy link
Contributor Author

I think I'm going to go ahead and merge but feel free to continue this discussion.

@Stebalien Stebalien deleted the features/streaming-ls-5600 branch November 8, 2018 22:41
Jorropo pushed a commit to Jorropo/go-libipfs that referenced this pull request Jan 25, 2023
…5600

feat(Directory): Add EnumLinksAsync method

This commit was moved from ipfs/go-unixfs@a3eae7f
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants