const performance loss ? Node 6 vs Node 7

[antoine-duchenet]

I just tried a simple var vs const benchmark https://gist.github.com/srikumarks/1431640

Results :

• Version: v6.9.2
• Platform: Linux 60c7f1bcdf48 4.4.0-57-generic Mani #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 GNU/Linux

ctl = {"min":87,"max":131,"mean":88.78,"spread":6.162110028228955}
con = {"min":87,"max":90,"mean":87.74,"spread":0.7432361670428108}
var = {"min":87,"max":90,"mean":87.72,"spread":0.6013318551350664}

• Version: v7.3.0
• Platform:Linux 310ceb709e32 4.4.0-57-generic Mani #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 GNU/Linux

ctl = {"min":87,"max":134,"mean":88.84,"spread":6.60411992622783}
con = {"min":87,"max":464,"mean":321.32,"spread":123.30684328130376}
var = {"min":87,"max":89,"mean":87.68,"spread":0.5810335618524364}

What could explain thiscon.spread evolution ?
This benchmark is maybe not reliable, but if it is, there's probably something to fix.

[vsemozhetbyt]

@antoine-duchenet

1. Is the author of the test aware that all the sum += return NaN due to not initialized sum?

Here is my result on Node.js 7.3.0 (Windows 7 x64) with the original test:

ctl = {"min":140,"max":172,"mean":154.96,"spread":5.956374736364391}
con = {"min":141,"max":876,"mean":709.52,"spread":289.626879968003}
var = {"min":140,"max":851,"mean":686.86,"spread":282.06729764366514}

Here is my result on Node.js 7.3.0 with the var i, sum; replaced by var i, sum = 0; in all the three functions:

ctl = {"min":46,"max":63,"mean":53.58,"spread":7.897062745097074}
con = {"min":46,"max":78,"mean":53.74,"spread":8.279637673231818}
var = {"min":46,"max":63,"mean":53.14,"spread":7.512682610093411}

2. I've tested the case with the benchmark.js and this code:

/******************************************************************************/
'use strict';
/******************************************************************************/
const functions = [
  function testNum()   {                           return 1 + 2; },
  function testVar()   { var a = 1;   var b = 2;   return a + b; },
  function testLet()   { let a = 1;   let b = 2;   return a + b; },
  function testConst() { const a = 1; const b = 2; return a + b; },
];
/******************************************************************************/
const suite = require('benchmark').Suite();

functions.forEach((func) => { suite.add(func.name, func); });
suite.on('cycle', (evt) => { console.log(String(evt.target)); }).run({ async: false });
/******************************************************************************/

The result is:

testNum x 42,522,354 ops/sec ±1.27% (85 runs sampled)
testVar x 49,026,518 ops/sec ±1.52% (83 runs sampled)
testLet x 49,404,921 ops/sec ±1.31% (86 runs sampled)
testConst x 49,022,550 ops/sec ±1.38% (83 runs sampled)

3. Recently I've run into many cases of microbenchmark flakyness. Modern JS engines vary performance according to many different reasons. Here are an issue and some articles on this topic:

#9729
https://mathiasbynens.be/notes/javascript-benchmarking
http://benediktmeurer.de/2016/12/16/the-truth-about-traditional-javascript-benchmarks/
http://benediktmeurer.de/2016/12/19/following-up-on-javascript-benchmarks/

[antoine-duchenet]

@vsemozhetbyt Thanks for your reply !

1. No, I suppose that the author is not aware that all the sum += return NaN.

Here are my results with a sum initialized version on the test :

• Version: v6.9.2
• Platform: Linux 60c7f1bcdf48 4.4.0-57-generic Mani #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 GNU/Linux

ctl = {"min":31,"max":47,"mean":32.18,"spread":2.558827856656222}
con = {"min":31,"max":32,"mean":31.62,"spread":0.48538644398032277}
var = {"min":31,"max":32,"mean":31.6,"spread":0.48989794855652885}

• Version: v7.3.0
• Platform:Linux 310ceb709e32 4.4.0-57-generic Mani #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 GNU/Linux

ctl = {"min":31,"max":45,"mean":32.24,"spread":2.2410711724529935}
con = {"min":31,"max":32,"mean":31.68,"spread":0.46647615158768974}
var = {"min":31,"max":33,"mean":31.78,"spread":0.45999999999985014}

Everything comes back in order.

2 & 3. Thanks for this code/results example and the references !
I quickly browsed through some parts and it looks really interesting, I'll find some time to read them all.

I'll close this topic by the end of the day if nobody wants to say something more.

P.S : I'm still curious to know why the NaN handling seems much slower in Node 7.3.0.