Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use MmapStore for MerkleTree generation #529

Merged
merged 1 commit into from
Mar 22, 2019

Conversation

schomatis
Copy link
Contributor

@schomatis schomatis commented Mar 7, 2019

Blocked on filecoin-project/merkletree#13, after that is merged the dependency here should point to merkle_light/master.


This is now dropping the allocated memory an order of magnitude:

# master:

Total: 309.4 MB
   309.0  99.9%  99.9%    309.0  99.9% ::allocate_in (inline) /rustc/14997d56a550f4aa99fe737593cd2758227afc56/src/liballoc/raw_vec.rs:109
     0.3   0.1% 100.0%      0.3   0.1% ::reserve_internal (inline) /rustc/14997d56a550f4aa99fe737593cd2758227afc56/src/liballoc/raw_vec.rs:683
     0.1   0.0% 100.0%      0.1   0.0% sapling_crypto::jubjub::JubjubBls12::new (inline) ??:0

# This PR with mmap'ed MT:

Total: 19.4 MB
    13.0  67.1%  67.1%     13.0  67.1% ::allocate_in (inline) /rustc/14997d56a550f4aa99fe737593cd2758227afc56/src/liballoc/raw_vec.rs:109
     4.3  22.0%  89.1%      4.3  22.0% ::reserve_internal (inline) /rustc/14997d56a550f4aa99fe737593cd2758227afc56/src/liballoc/raw_vec.rs:683
     2.0  10.3%  99.4%      2.0  10.3% rayon::iter::collect::special_extend@1594e0 (inline) :0

@schomatis

This comment has been minimized.

@schomatis

This comment has been minimized.

@schomatis

This comment has been minimized.

@schomatis schomatis closed this Mar 12, 2019
@schomatis schomatis reopened this Mar 12, 2019
@schomatis schomatis changed the title [WIP] dgraph::Graph: use MmapStore for MerkleTree generation use MmapStore for MerkleTree generation Mar 14, 2019
@schomatis

This comment has been minimized.

@schomatis schomatis force-pushed the feat/dgraph/use-mmap-merkle branch 2 times, most recently from f19a16f to 26833f0 Compare March 15, 2019 15:09
@schomatis schomatis force-pushed the feat/dgraph/use-mmap-merkle branch 2 times, most recently from da14888 to 21124c9 Compare March 21, 2019 15:15
porcuquine
porcuquine previously approved these changes Mar 21, 2019
Copy link
Collaborator

@porcuquine porcuquine left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems fine when the rest is sorted out (might be already).

@schomatis schomatis merged commit e4fdb86 into master Mar 22, 2019
@schomatis schomatis deleted the feat/dgraph/use-mmap-merkle branch March 22, 2019 01:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants