OutOfMemory when trying to traverse the state trie of the latest Ethereum mainnet block

[ivica7]

I have the JavaScript code below, which is traversing the state trie from geth's leveldb for Ethereum mainnet Block #5200035.

After some time with high cpu usage and having reserved about 6Gb RAM, it crashes with FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory.

It looks like the library is having trouble with the size of the mainnet database (~70Gb after a fast sync). A simple key/value lookup with db.get works fine.

Any idea how to approach this problem?

    var Trie = require('merkle-patricia-tree/secure');
    var levelup = require('levelup');
    var leveldown = require('leveldown');
    
    //Connecting to the leveldb database
    var db = levelup(leveldown('../datadir/geth/chaindata'));
    
    //Adding the "stateRoot" value from the block so that we can inspect the state root at that block height.
    var root = "0x4d675087a16f13fa5f61af74d79e08c82de7cf200e63d5b225b4a7937705a3e2"; // Block #5200035
    
    //Creating a trie object of the merkle-patricia-tree library
    var trie = new Trie(db, root);
    
    //Creating a nodejs stream object so that we can access the data
    var stream = trie.createReadStream()
    
    //Turning on the stream (because the node js stream is set to pause by default)
    stream.on('data', function (data){
      console.log(data)
    });

[holgerd77]

What versions of levelup and leveldown are you actually using? Having trouble to get this to work. Have you manually installed a newer levelup version?

[ivica7]

This is the depndencies block from package.json

  "dependencies": {                                                                                                                         
    "ethereumjs-account": "^2.0.4",                                                                                                         
    "ethereumjs-util": "^5.1.5",                                                                                                            
    "leveldown": "^3.0.0",                                                                                                                  
    "levelup": "^2.0.2",                                                                                                                    
    "merkle-patricia-tree": "^2.3.0",                                                                                                       
    "rlp": "^2.0.0"                                                                                                                         
  }                                                                                                                                         

Installed with npm 5.6.0 / node v8.9.4

[holgerd77]

Ah, thanks, yes already had the suspicion that the levelup version required by the repo is too old, we actually have to update this, this is working now when I manually update dependencies.

[ivica7]

Did you update to the latest? I can try if it works with the 70gb trie from the mainnet.