Add node worker-thread support to jest-worker #7408
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Clean up types, create missing interfaes and make Flow and Jest happy
rickhanlonii
commented
Nov 25, 2018
rickhanlonii
commented
Nov 25, 2018
SimenB
approved these changes
Nov 25, 2018
…reads # Conflicts: # CHANGELOG.md
mjesun
approved these changes
Dec 4, 2018
|
Woooooo! 🎉 |
thymikee
added a commit
to spion/jest
that referenced
this pull request
Dec 18, 2018
* master: (24 commits) Add `jest.isolateModules` for scoped module initialization (jestjs#6701) Migrate to Babel 7 (jestjs#7016) docs: changed "Great Scott!" link (jestjs#7524) Use reduce instead of filter+map in dependency_resolver (jestjs#7522) Update Configuration.md (jestjs#7455) Support dashed args (jestjs#7497) Allow % based configuration of max workers (jestjs#7494) chore: Standardize filenames: jest-runner pkg (jestjs#7464) allow `bail` setting to control when to bail out of a failing test run (jestjs#7335) Add issue template labels (jestjs#7470) chore: standardize filenames in e2e/babel-plugin-jest-hoist (jestjs#7467) Add node worker-thread support to jest-worker (jestjs#7408) Add `testPathIgnorePatterns` to CLI documentation (jestjs#7440) pretty-format: Omit non-enumerable symbol properties (jestjs#7448) Add Jest Architecture overview to docs. (jestjs#7449) chore: run appveyor tests on node 10 chore: fix failures e2e test for node 8 (jestjs#7446) chore: update docusaurus to v1.6.0 (jestjs#7445) Enhancement/expect-to-be-close-to-with-infinity (jestjs#7444) Update CHANGELOG formatting (jestjs#7429) ...
willsmythe
pushed a commit
to willsmythe/jest
that referenced
this pull request
Dec 20, 2018
* Restructure workers (create a base class and inherit from it) Clean up types, create missing interfaes and make Flow and Jest happy * Move child.js to workers folder * Restructure base classes and make a working POC * Remove BaseWorker * Remove MessageChannel and cleanup super() calls * Use worker threads implementation if possible * Restructure queues * Support experimental modules in jest-resolver * Rename child.js to processChild.js * Remove private properties from WorkerPoolInterface * Move common line outside of if-else * Unify interface (use workerId) and remove recursion * Remove opt-out option for worker_threads in node 10.5+ * Alphabetical import sorting * Unlock worker after onEnd * Cache queue head in the getNextJob loop * Elegant while loop * Remove redundand .binds * Clean up interfaces and responsibilites * Update jest-worker * Add changelog and update jest-worker readme * Fix lint lol * Fixes from review * Update Changelog * rm function * rm [] * Make imports alphabetical 🤮 * Go back to any * Fix lint * \n * Fix formatting * Add docs * Revert canUseWorkerThreads * Fix lint * Fix pathing on windows
This was referenced Jan 11, 2019
captain-yossarian
pushed a commit
to captain-yossarian/jest
that referenced
this pull request
Jul 18, 2019
* Restructure workers (create a base class and inherit from it) Clean up types, create missing interfaes and make Flow and Jest happy * Move child.js to workers folder * Restructure base classes and make a working POC * Remove BaseWorker * Remove MessageChannel and cleanup super() calls * Use worker threads implementation if possible * Restructure queues * Support experimental modules in jest-resolver * Rename child.js to processChild.js * Remove private properties from WorkerPoolInterface * Move common line outside of if-else * Unify interface (use workerId) and remove recursion * Remove opt-out option for worker_threads in node 10.5+ * Alphabetical import sorting * Unlock worker after onEnd * Cache queue head in the getNextJob loop * Elegant while loop * Remove redundand .binds * Clean up interfaces and responsibilites * Update jest-worker * Add changelog and update jest-worker readme * Fix lint lol * Fixes from review * Update Changelog * rm function * rm [] * Make imports alphabetical 🤮 * Go back to any * Fix lint * \n * Fix formatting * Add docs * Revert canUseWorkerThreads * Fix lint * Fix pathing on windows
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR is a continuation of #6676
Node 10 shipped with a "threading API" that uses SharedBuffers to communicate between the main process and its child threads. Being jest-worker a parallelization library, we can take advantage of this API when available. At the same time, we'll decouple our scheduling logic from our communication logic for better re-usability.
Here are some key things:
Test plan
Add unit and integration tests, near 100% coverage
I also tried as much as possible to move existing tests rather than writing new ones. In a few places where the refactoring broke the test irrecoverably I wrote new tests
Performance
Without experimental worker
total worker-farm: { wFGT: 3110, wFPT: 2947 } total jest-worker: { jWGT: 2958, jWPT: 2577 } --------------------------------------------------------------------------- % improvement over 10000 calls (global time): 4.887459807073955 % improvement over 10000 calls (processing time): 12.555140821174076With experimental worker
❯ node --experimental-worker --expose-gc test.js empty 10000 total worker-farm: { wFGT: 3084, wFPT: 2913 } total jest-worker: { jWGT: 2070, jWPT: 1504 } --------------------------------------------------------------------------- % improvement over 10000 calls (global time): 32.87937743190661 % improvement over 10000 calls (processing time): 48.3693786474425