Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor ipfs get to use CoreAPI #5943

Merged
merged 5 commits into from
Jan 30, 2019
Merged

Refactor ipfs get to use CoreAPI #5943

merged 5 commits into from
Jan 30, 2019

Conversation

magik6k
Copy link
Member

@magik6k magik6k commented Jan 24, 2019 •
edited by Stebalien
Loading

@magik6k magik6k added the topic/core-api Topic core-api label Jan 24, 2019
@magik6k magik6k requested a review from Kubuxu as a code owner January 24, 2019 19:40
@ghost ghost assigned magik6k Jan 24, 2019
@ghost ghost added the status/in-progress In progress label Jan 24, 2019
This was referenced Jan 24, 2019
Open
Closed
hannahhoward
hannahhoward approved these changes Jan 29, 2019
View reviewed changes
Copy link
Contributor

@hannahhoward hannahhoward left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and on the the two connected PRs.. but I don't consider myself a core API expert by any means.

I have two quick questions:

  1. Why move unixfile to go-unixfs & why put tar-writer in go-ipfs-files instead of go-unixfs?
  2. Am I correct in understanding that neither of these two changes are strickly essential to delivering the refactor to make get use CoreAPI?

Non-blocking, just want to understand.

@Stebalien
Copy link
Member

why put tar-writer in go-ipfs-files

Because coreapi.Unixfs().Get() now returns a go-ipfs-files.Node object. Therefore, the TarWriter needs to convert this (instead of a dag) into a tarfile.

However: ipfs/go-unixfs#59 (comment)

Am I correct in understanding that neither of these two changes are strickly essential to delivering the refactor to make get use CoreAPI?

We need some way to go from a go-ipfs-files.Node to a tar because coreapi.Unixfs().Get() returns a go-ipfs-files.Node instead of a raw IPLD DAG. We could just keep DagArchive as it is as a way to go directly from a DAG to a tar file.

@magik6k
Copy link
Member Author

magik6k commented Jan 30, 2019

We could just keep DagArchive as it is as a way to go directly from a DAG to a tar file.

As a note - I decided to drop it as the combo of NewUnixfsFile+files.TarWriter do the same thing with the only difference being the files interface in between (there shouldn't be any meaningful performance impact). Plus after switching ipfs get to the new thing, it would basically become dead code.

This was referenced Jan 30, 2019
Merged
Merged
@magik6k
coreapi: support symlinks in unixfile
e4020f9 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
@magik6k
Use coreapi in ipfs get
aa3e087 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
@magik6k
coreapi: extract unixfile to go-unixfs
a151041 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
@magik6k
commands/get: move FileArchive here
e71e700 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
Stebalien
Stebalien approved these changes Jan 30, 2019
View reviewed changes
core/commands/get.go
return err
}

res.SetLength(uint64(size))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually incorrect but it's an existing bug... #5690

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The alternative is not setting this at all, but it will break progress bar... (We could pass this in a separate header)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have to. Unfortunately, that'll require a change to the commands library.

Stebalien
Stebalien requested changes Jan 30, 2019
View reviewed changes
Copy link
Member

@Stebalien Stebalien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this doesn't work with sharding. t0260-sharding.sh is failing.

@Stebalien
Copy link
Member

Error is: "Error: cannot write to readonly DAGService"

@Stebalien
Copy link
Member

The issue is: https://github.com/ipfs/go-unixfs/blob/0eb1ef8f33371b1b9f74f74655b584144919ce91/file/unixfile.go#L121. This is causing the hamt to serialize and save itself.

@magik6k
Copy link
Member Author

magik6k commented Jan 30, 2019

It should probably just do return 0, files.ErrNotSupported, I'll try to see how it affects things

@magik6k
Copy link
Member Author

magik6k commented Jan 30, 2019

Ah, right, it breaks the progress bar ._.

@Stebalien
Copy link
Member

Maybe we should have a separate hamt reader?

@magik6k magik6k mentioned this pull request Jan 30, 2019
Merged
@magik6k
Copy link
Member Author

magik6k commented Jan 30, 2019

See ipfs/go-unixfs#61

@magik6k
gx: update go-unixfs to propagate archive changes
7c2aa0e 
License: MIT
Signed-off-by: Łukasz Magiera <magik6k@gmail.com>
@magik6k magik6k mentioned this pull request Jan 30, 2019
Merged
@Stebalien Stebalien merged commit 35aaa98 into master Jan 30, 2019
@Stebalien Stebalien deleted the misc/coreapi-get branch January 30, 2019 20:52
@ghost ghost removed the status/in-progress In progress label Jan 30, 2019
hacdias pushed a commit to ipfs/boxo that referenced this pull request Jan 27, 2023
@Stebalien
Merge pull request ipfs/kubo#5943 from ipfs/misc/coreapi-get
573067d 
Refactor ipfs get to use CoreAPI

This commit was moved from ipfs/kubo@35aaa98
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hannahhoward hannahhoward hannahhoward approved these changes

@Stebalien Stebalien Stebalien approved these changes

@Kubuxu Kubuxu Awaiting requested review from Kubuxu

Assignees

@magik6k magik6k

Labels
topic/core-api Topic core-api
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@magik6k @Stebalien @hannahhoward