Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove intermediate InsertNodeAtPath in ipfs add #1964

Closed
wants to merge 78 commits into from
Closed

Remove intermediate InsertNodeAtPath in ipfs add #1964

wants to merge 78 commits into from

Conversation

rht
Copy link
Contributor

@rht rht commented Nov 13, 2015

This is an experiment.
The time it takes for an ipfs add is halved when the intermediate root(s) are not created.

daviddias and others added 30 commits November 11, 2015 10:04
@daviddias @whyrusleeping
update version
f72b675 
License: MIT
Signed-off-by: David Dias <daviddias.p@gmail.com>
@tv42 @whyrusleeping
pin: Guard against callers causing refcount underflow
80a8af7 
This used to lead to large refcount numbers, causing Flush to create a
lot of IPFS objects, and merkledag to consume tens of gigabytes of
RAM.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
sharness: Use sed in a cross-platform safe way
11185e3 
OS X sed is documented as "-i SUFFIX", GNU sed as "-iSUFFIX". The one
consistent case seems to be "-iSUFFIX", where suffix cannot empty (or
OS X will parse the next argument as the suffix).

This used to leave around files named `refsout=` on Linux, and was
just confusing.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
sharness: fusermount -u is the documented way to unmount FUSE on Linux
86d781a 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
pin: unexport NewIndirectPin, it's not useful and not used elsewhere
cb46a52 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
pin: Remove code shadowing pins as datastore keys
cad9e47 
These secondary copies were never actually queried, and didn't contain
the indirect refcounts so they couldn't become the authoritative
source anyway as is. New goal is to move pinning into IPFS objects.

A migration will be needed to remove the old data from the datastore.
This can happen at any time after this commit.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
blocks/set: Remove now-unused NewDBWrapperSet
34f8231 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
Simplify Pinner interface by folding ManualPinner into Pinner
3a05a94 
Pinner had method GetManual that returned a ManualPinner, so every
Pinner had to implement ManualPinner anyway.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
pin: Remove dead code
832d639 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
pin: Remove double bookkeeping of refcount keys
c65bf3f 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
Use uint64 for indirect pin refcounts
f85862b 
Platform-dependent behavior is not nice, and negative refcounts are
not very useful.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
Typo
99448d4 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
sharness: Don't assume we know all things that can create garbage
21e7b05 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

sharness: Don't assume we know all things that can create garbage

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
pin: Rewrite to store pins in IPFS objects
60a505f 
WARNING: No migration performed! That needs to come in a separate
commit, perhaps amended into this one.

This is the minimal rewrite, only changing the storage from
JSON(+extra keys) in Datastore to IPFS objects. All of the pinning
state is still loaded in memory, and written from scratch on Flush. To
do more would require API changes, e.g. adding error returns.

Set/Multiset is not cleanly separated into a library, yet, as it's API
is expected to change radically.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
pin: Future-proof against refcount marshaled size changes
283432f 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@whyrusleeping
bump fsrepo version to 3
81b16e2 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
pin: Do not accidentally delete indirect pins on Flush
5db177e 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
dagmodifier: Don't lose pin if old and new key happen to be equal
fd28a39 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@tv42 @whyrusleeping
dagmodifier test: Add TODO note about how bad luck can cause test fai…
1e0d72f 
…lure

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@jbenet @whyrusleeping
remove msgio double wrap
b45d248 
There was doublewrapping with an unneeded msgio. given that we
use a stream muxer now, msgio is only needed by secureConn -- to
signal the boundaries of an encrypted / mac-ed ciphertext.

Side note: i think including the varint length in the clear is
actually a bad idea that can be exploited by an attacker. it should
be encrypted, too. (TODO)

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@whyrusleeping
buffer msgio
8ee7129 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@whyrusleeping
using multistream muxer
b4754c8 
* ID service stream
* make the relay service use msmux
* fix nc tests

Note from jbenet: Maybe we should remove the old protocol/muxer
and see what breaks. It shouldn't be used by anything now.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
Signed-off-by: Juan Batiz-Benet <juan@benet.ai>
@whyrusleeping
Add locking interface to blockstore
41b58e1 
The addition of a locking interface to the blockstore allows us to
perform atomic operations on the underlying datastore without having to
worry about different operations happening in the background, such as
garbage collection.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@whyrusleeping
merkledag FetchGraph and EnumerateChildren
ae3a453 
This commit improves (fixes) the FetchGraph call for recursively
fetching every descendant node of a given merkledag node. This operation
should be the simplest way of ensuring that you have replicated a dag
locally.

This commit also implements a method in the merkledag package called
EnumerateChildren, this method is used to get a set of the keys of every
descendant node of the given node. All keys found are noted in the
passed in KeySet, which may in the future be implemented on disk to
avoid excessive memory consumption.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@whyrusleeping
address concerns from PR
506c46f 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@whyrusleeping
move locking out of GC branch
06184a6 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@whyrusleeping
lock blockstore for pin add
588f53e 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@jbenet @whyrusleeping
renamed {R,}Lock -> {Pin,GC}Lock
d71bb78 
License: MIT
Signed-off-by: Juan Batiz-Benet <juan@benet.ai>
@whyrusleeping
implement mark and sweep GC
d0aa03c 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

dont GC blocks used by pinner

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

comment GC algo

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

add lock to blockstore to prevent GC from eating wanted blocks

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

improve FetchGraph

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

separate interfaces for blockstore and GCBlockstore

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

reintroduce indirect pinning, add enumerateChildren dag method

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@whyrusleeping
break up GC logic
8650e44 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@rht
Copy link
Contributor Author

rht commented Nov 19, 2015

Found it: in flatfs.New, the sync argument doesn't get passed into the Datastore object creation.

@rht rht mentioned this pull request Nov 19, 2015
Closed
@whyrusleeping
Copy link
Member

@rht wow, good catch. thanks!

whyrusleeping and others added 6 commits November 19, 2015 11:15
@whyrusleeping
if bucket doesnt have enough peers, grab more elsewhere
06bc124 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@whyrusleeping
add closenotify and large timeout to gateway
cb56ec1 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@jbenet
Merge pull request ipfs#1967 from ipfs/fix/kbucket-fill
d46c0d4 
if bucket doesnt have enough peers, grab more elsewhere
@whyrusleeping
Merge pull request ipfs#1980 from ipfs/fix/gateway-close-notif
856e250 
add closenotify and large timeout to gateway
@rht @whyrusleeping
Add config option for flatfs no-sync
cab5868 
License: MIT
Signed-off-by: rht <rhtbot@gmail.com>
@whyrusleeping
Merge pull request ipfs#1985 from ipfs/rht-add-no-sync
b4ab684 
Add config option for flatfs no-sync
@rht rht added the topic/perf Performance label Nov 22, 2015
@rht
Replace strings.Join(elms, "/") with path.Join(elms)
8ffc9ea 
License: MIT
Signed-off-by: rht <rhtbot@gmail.com>
@rht rht mentioned this pull request Nov 24, 2015
Closed
rht and others added 4 commits November 24, 2015 14:48
@rht
strings.Split -> path.SplitList
d6891f4 
License: MIT
Signed-off-by: rht <rhtbot@gmail.com>
@rht
s/\[0:/\[:/g
2ffef14 
License: MIT
Signed-off-by: rht <rhtbot@gmail.com>
@whyrusleeping
ipfs files ls without -l is faster
c419049 
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@jbenet
Merge pull request ipfs#2007 from ipfs/feat/fast-files-ls
e04a31d 
ipfs files ls without -l is faster
@rht rht mentioned this pull request Nov 28, 2015
Open
@rht rht mentioned this pull request Dec 1, 2015
Closed
42 tasks
@jbenet
Copy link
Member

jbenet commented Dec 1, 2015

LGTM. ok so do we still want to merge this @whyrusleeping ?

(what if the "edited dag" is too big to hold in memory? not sure what happens now? fails? but in future should store on disk and gc memory re-reading as necessary? (should be possible to edit massive dags with very little memory, but enough disk))

@rht
Remove intermediate InsertNodeAtPath in ipfs add
aec8e82 
License: MIT
Signed-off-by: rht <rhtbot@gmail.com>

Small cleanup coreunix.Add

License: MIT
Signed-off-by: rht <rhtbot@gmail.com>
@rht rht force-pushed the no-insertnodeatpath branch from 6484ea2 to aec8e82 Compare December 1, 2015 16:17
@rht
Copy link
Contributor Author

rht commented Dec 1, 2015

(rebased)

The last / only InsertNodeAtPath in this PR doesn't consume much memory, since what it does is just patching the final hash to the "global" root of an IpfsNode (as in, the root dir one would see of /ipfs if the repo is fuse-mounted).

@rht rht mentioned this pull request Dec 16, 2015
Merged
@whyrusleeping whyrusleeping force-pushed the dev0.4.0 branch 2 times, most recently from 68b9745 to b0a8591 Compare December 28, 2015 14:36
@whyrusleeping
Copy link
Member

closing, we use the mfs code for adds now (if changes from here are still relevant to that process, please file a new PR on top of master with them)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
status/in-progress In progress topic/perf Performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@rht @whyrusleeping @jbenet @daviddias @tv42 @eminence