Remove chunker channel

[rht]

PR to keep track and measure the time and memory perf of ipfs add with channel iterator replaced with chunker object. Perhaps have to wait until go2.x when channel iterator becomes about as fast as counter object.

• fix tests
• fixes Remove chunk channels as input to DagBuilderHelper #1969

Ran perf test on lots of 1MB files, ipfs add is about O(rsync) (faster than git add?)

Edit: this plot below is misleading (the add operation is run under a goroutine, and without outputDagnode checking, the client exits long before the (ephemeral) daemon has finished the process), updated plot is in my last comment.

[whyrusleeping]

@rht this is wonderful.

[whyrusleeping]

@rht have you tried things with the changes in #1965?

[rht]

I did, but #1965's memorydagservice optimizes mainly InsertNodeAtPath.
I already saw similar memory profile (i.e. plateau at ~55MB) from that PR when intermediate roots are removed (no-insertnodeatpath).

[jbenet]

@rht this is great!! ping me here when i should CR

[rht]

Here is the most recent plot (no-sync + no-insertnodeatpath + no-chunk-channel, each file 1MB).
(to make it simple, removing outputDagnode or not doesn't actually affect the perf, so it is actually slower than rsync)

[rht]

@whyrusleeping if you can confirm the perf of this branch:

• no-sync (~55(2) s)
• no-sync + no-insertnodeatpath (~49(2) s)
• no-sync + no-insertnodeatpath + no-chunk-channel (?)

[jbenet]

@rht this LGTM -- @whyrusleeping LGTU?

[whyrusleeping]

yeah, it does. Lets get it rebased first though

[rht]

Rebased.

[rht]

It is likely the commit that causes random os sharness fails is not in dev0.4.0.
And the bisect in master can be investigated more effectively by someone with an osx.

[jbenet]

Thanks @rht !