feat(bench-trial): new package to benchmark performance #192

acatl · 2018-01-29T21:49:16Z

What: new performance benchmark tool

Why: to test changes we make to data-point

How: created a new module that runs multiple iterations of benchmarkjs tests

Checklist:

acatl · 2018-01-29T21:56:35Z

packages/bench-trail/runner.js

@@ -0,0 +1,333 @@
+#!/usr/bin/env node --expose-gc


so many many many tests for this file they didn't fit in this PR, I will try to send it via USB to each maintainer of DataPoint

coveralls · 2018-01-29T21:57:36Z

Coverage remained the same at 100.0% when pulling 6f45729 on acatl:bench-trail into bb6c7bc on ViacomInc:master.

coveralls · 2018-01-29T21:57:36Z

Coverage remained the same at 100.0% when pulling 5a89cd0 on acatl:bench-trail into 485f4ab on ViacomInc:master.

acatl · 2018-01-29T21:57:37Z

packages/bench-trail/runner.js

+  return Math.round(bytes / 1024 * 100) / 100
+}
+
+function runGC (val) {


all memory related code is not being used ATM, I left it here because it took me a while to figure some things out. will look back into it with more time to reintegrate it

test performance

raingerber · 2018-01-30T13:25:51Z

packages/bench-trial/runner.js

+}
+
+function runTest (suite) {
+  const isASync = !!suite.async


isAsync? (also in runBenchmark function)

…o bench-trail

paulmolluzzo

This is awesome!

I provided some suggestions for rewording the README, so I'm requesting changes mostly for that. There are a couple of comments in the code itself too, but nothing is big.

Stoked to start using this!

paulmolluzzo · 2018-01-30T14:49:05Z

packages/bench-trial/README.md

+
+While running [benchmarkjs](https://benchmarkjs.com) to compare different versions of code I found out a couple of things:
+
+- **Ambiguous results**: tests would throw different results every time I ran the test, specially if the difference was minimal. If I ran the same test multiple times the numbers changed, as time went by I would get more and more operations per second on each benchmark. This would only happen if I ran the tests consecutively, the reason of this might be related to the v8 engine warming up and optimizing the code the more I ran it. If I let some time to cool off, the tests would go back down on operations per second.


This is a rewrite of the line, so copy and pasting might be easier (if you want to keep them):

Ambiguous results: I noticed that the same benchmark tests were returning different results every time they were run. If they were re-run consecutively I would get more operations per second on each benchmark. I believe the reason may be related to the v8 engine warming up and optimizing the code the more it ran, since if I let some time to "cool off" the operations per second for each test would decrease. These ambiguous results meant having to manually repeat tests to try to ensure some consistency.

paulmolluzzo · 2018-01-30T15:43:30Z

packages/bench-trial/README.md

+While running [benchmarkjs](https://benchmarkjs.com) to compare different versions of code I found out a couple of things:
+
+- **Ambiguous results**: tests would throw different results every time I ran the test, specially if the difference was minimal. If I ran the same test multiple times the numbers changed, as time went by I would get more and more operations per second on each benchmark. This would only happen if I ran the tests consecutively, the reason of this might be related to the v8 engine warming up and optimizing the code the more I ran it. If I let some time to cool off, the tests would go back down on operations per second.
+- **Reliable Execution**: more than once I made changes on the code being tested and never did I notice the change I had made was not even executing correctly. So the results I was getting were really unreliable.


This is a rewrite of the line, so copy and pasting might be easier (if you want to keep them):

Unreliable Test Execution: Occasionally I made changes to the code being benchmarked and would overlook that it was not executing correctly, further compounding the issue of making the results unreliable.

paulmolluzzo · 2018-01-30T15:46:26Z

packages/bench-trial/README.md

+
+## Solution
+
+- **Ambiguous results**: Run benchmark more than once to get median and average results, because the test will run multiple times the code will get optimized, using the median we can get more reliable results. 


This is a rewrite of the line, so copy and pasting might be easier (if you want to keep them):

Clear, consistent results: By running benchmark tests more than once we can get median and average results and get a bigger picture with less fluctuation. Because the tests will run multiple times in succession, the code will get optimized by the engine, and we can use the median time as a more consistent and stable metric.

paulmolluzzo · 2018-01-30T19:41:57Z

packages/bench-trial/README.md

+## Solution
+
+- **Ambiguous results**: Run benchmark more than once to get median and average results, because the test will run multiple times the code will get optimized, using the median we can get more reliable results. 
+- **Reliable Execution**: Run a simple assertion tests on each suite before the actual benchmark runs, this helps us make sure our test are executing correctly. 


This is a rewrite of the line, so copy and pasting might be easier (if you want to keep them):

Reliable Tests to Execute: By running simple assertion tests on each suite before the actual benchmark runs we can be sure our tests are executing correctly.

paulmolluzzo · 2018-01-30T19:43:02Z

packages/bench-trial/README.md

+
+### Writing your benchmark suites
+
+The file you provide to bench-trial should export an `array` of suites, each suite is an object in the form of: 


The file provided to bench-trial should export an array of test suites, each test suite is an object in the form of:

paulmolluzzo · 2018-01-30T19:44:04Z

packages/bench-trial/README.md

+test(test:function, value:*)
+```
+
+to write your manual test see the manual test example below


To write your own tests see the "manual test" example below

paulmolluzzo · 2018-01-30T19:51:50Z

packages/bench-trial/package.json

+    "email": "acatl.pacheco@viacom.com"
+  },
+  "devDependencies": {
+    "benchmark": "2.1.4",


Is there a reason this is pinned to this specific version?

paulmolluzzo · 2018-01-30T20:11:08Z

packages/bench-trial/runner.js

+
+function middle (values) {
+  var len = values.length
+  var half = Math.floor(len / 2)


acatl · 2018-01-31T14:25:36Z

@paulmolluzzo applied your feedback, thanks!!!

paulmolluzzo · 2018-01-31T14:41:37Z

packages/bench-trial/README.md

+
+While running [benchmarkjs](https://benchmarkjs.com) to compare different versions of code I found out a couple of things:
+
+- **Consistency**: I noticed that the same benchmark tests were returning different results every time they executed. If they were re-run consecutively, I would get more operations per second on each benchmark. I believe the reason may be related to the v8 engine warming up and optimizing the code the more it ran, since if I let some time to "cool off" the operations per second for each test would decrease. These ambiguous results meant having to repeat tests to ensure some consistency.


The title should have stayed as **Ambiguous Results**. This is describing the problem.

paulmolluzzo

Two tiny things.

paulmolluzzo · 2018-01-31T14:42:14Z

packages/bench-trial/README.md

+While running [benchmarkjs](https://benchmarkjs.com) to compare different versions of code I found out a couple of things:
+
+- **Consistency**: I noticed that the same benchmark tests were returning different results every time they executed. If they were re-run consecutively, I would get more operations per second on each benchmark. I believe the reason may be related to the v8 engine warming up and optimizing the code the more it ran, since if I let some time to "cool off" the operations per second for each test would decrease. These ambiguous results meant having to repeat tests to ensure some consistency.
+- **Reliable Execution**: Occasionally I made changes to the benchmarked code and would overlook that it was not executing correctly, further compounding the issue of making the results unreliable.


This should be Unreliable Exection. "Reliable Execution" isn't a problem 😉

paulmolluzzo · 2018-01-31T14:43:00Z

packages/bench-trial/README.md

+
+Test synchronous code [example](examples/array-iteration.js)
+Test asynchronous code [example](examples/async-example.js)
+Write manual test sync/asynchronous code [example](examples/manual-tests.js)


Can we put a bullet on these please? They currently get squished to a single line.

paulmolluzzo

This is awesome! 🎉

Thanks for putting up with my comments on the README. 🙊

feat(bench-trail): new package to benchmark performance

6f45729

acatl added this to the v2 - new features milestone Jan 29, 2018

acatl self-assigned this Jan 29, 2018

acatl requested review from paulmolluzzo and raingerber January 29, 2018 21:49

acatl commented Jan 29, 2018

View reviewed changes

Acatl Pacheco added 2 commits January 29, 2018 17:17

feat(bench-trial): new package

430b5f2

test performance

fix(bench-trial): fix typos

fa3d0d7

acatl changed the title ~~feat(bench-trail): new package to benchmark performance~~ feat(bench-trial): new package to benchmark performance Jan 29, 2018

Merge branch 'master' into bench-trail

900cd0c

raingerber reviewed Jan 30, 2018

View reviewed changes

acatl and others added 4 commits January 30, 2018 08:52

Merge branch 'master' into bench-trail

b8d4516

docs(bench-trial): thanks + feedback

bc77f11

Merge remote-tracking branch 'upstream/master' into bench-trail

b54d97e

Merge branch 'bench-trail' of https://github.com/acatl/data-point int…

fc2656f

…o bench-trail

paulmolluzzo requested changes Jan 30, 2018

View reviewed changes

Acatl Pacheco added 2 commits January 31, 2018 09:20

docs(bench-trial): updates wording

9ae18ab

build(bench-trial): changes to use dependencies

18be79e

acatl modified the milestone: v2 - new features Jan 31, 2018

paulmolluzzo reviewed Jan 31, 2018

View reviewed changes

paulmolluzzo requested changes Jan 31, 2018

View reviewed changes

Acatl Pacheco added 2 commits January 31, 2018 09:54

build(bench-trial): update to explicit versions

7b23cec

docs(bench-trial): wording

5a89cd0

paulmolluzzo approved these changes Jan 31, 2018

View reviewed changes

acatl merged commit a704568 into ViacomInc:master Jan 31, 2018

acatl deleted the bench-trail branch January 31, 2018 15:07

victorsingh pushed a commit to victorsingh/data-point that referenced this pull request Feb 13, 2018

feat(bench-trial): new package to benchmark performance (ViacomInc#192)

baabb29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(bench-trial): new package to benchmark performance #192

feat(bench-trial): new package to benchmark performance #192

acatl commented Jan 29, 2018

acatl Jan 29, 2018

coveralls commented Jan 29, 2018

coveralls commented Jan 29, 2018 •

edited

Loading

acatl Jan 29, 2018

raingerber Jan 30, 2018 •

edited

Loading

paulmolluzzo left a comment

paulmolluzzo Jan 30, 2018

paulmolluzzo Jan 30, 2018

paulmolluzzo Jan 30, 2018

paulmolluzzo Jan 30, 2018

paulmolluzzo Jan 30, 2018

paulmolluzzo Jan 30, 2018

paulmolluzzo Jan 30, 2018

paulmolluzzo Jan 30, 2018

acatl commented Jan 31, 2018

paulmolluzzo Jan 31, 2018

paulmolluzzo left a comment

paulmolluzzo Jan 31, 2018

paulmolluzzo Jan 31, 2018

paulmolluzzo left a comment


		While running [benchmarkjs](https://benchmarkjs.com) to compare different versions of code I found out a couple of things:

		- Ambiguous results: tests would throw different results every time I ran the test, specially if the difference was minimal. If I ran the same test multiple times the numbers changed, as time went by I would get more and more operations per second on each benchmark. This would only happen if I ran the tests consecutively, the reason of this might be related to the v8 engine warming up and optimizing the code the more I ran it. If I let some time to cool off, the tests would go back down on operations per second.


		## Solution

		- Ambiguous results: Run benchmark more than once to get median and average results, because the test will run multiple times the code will get optimized, using the median we can get more reliable results.


		### Writing your benchmark suites

		The file you provide to bench-trial should export an `array` of suites, each suite is an object in the form of:


		While running [benchmarkjs](https://benchmarkjs.com) to compare different versions of code I found out a couple of things:

		- Consistency: I noticed that the same benchmark tests were returning different results every time they executed. If they were re-run consecutively, I would get more operations per second on each benchmark. I believe the reason may be related to the v8 engine warming up and optimizing the code the more it ran, since if I let some time to "cool off" the operations per second for each test would decrease. These ambiguous results meant having to repeat tests to ensure some consistency.

feat(bench-trial): new package to benchmark performance #192

feat(bench-trial): new package to benchmark performance #192

Conversation

acatl commented Jan 29, 2018

Choose a reason for hiding this comment

coveralls commented Jan 29, 2018

coveralls commented Jan 29, 2018 • edited Loading

Choose a reason for hiding this comment

raingerber Jan 30, 2018 • edited Loading

Choose a reason for hiding this comment

paulmolluzzo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acatl commented Jan 31, 2018

Choose a reason for hiding this comment

paulmolluzzo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulmolluzzo left a comment

Choose a reason for hiding this comment

coveralls commented Jan 29, 2018 •

edited

Loading

raingerber Jan 30, 2018 •

edited

Loading