Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial version of jest-worker #4497

Merged
merged 8 commits into from
Oct 4, 2017
Merged

Initial version of jest-worker #4497

merged 8 commits into from
Oct 4, 2017

Conversation

mjesun
Copy link
Contributor

@mjesun mjesun commented Sep 17, 2017

This PR introduces a new module, jest-worker, intended to allow heavy task parallelization over multiple workers.

The module has a few advantages over the currently one used both in jest and metro-bundler:

  • 100% flow-ified.

  • 100% test coverage on it, all statements, methods and branches.

  • Slightly faster than the currenly used one.

  • Natively provides a Promise based interface, which allow us to avoid the extra wrapping layer in order to be used with async/`await.

  • It only has one single dependency (merge-stream), which we could also remove.

  • Lazily instantiated code in worker, meaning no code is loaded on child processes until the first call is done (lazy require). This allows to spawn a farm with minimal RAM consumption, and only load them when needed.

  • Sticky workers: tasks will be processed by the first available worker if no stickyness is needed, or by a particular one if the task is forced to do so. Specially useful for workers implementing caches.

Performance test

It can be run by doing node --expose-gc test.js under __performance_tests__. Note that the percentage improvement shown (~ 10%) applies to 10,000 calls, meaning the performance improvement per single call is negligible. The test implements a Promise wrapper over the current implementation, so we can equivalently test both implementations as we use them in real scenarios.

---------------------------------------------------------------------------
jest-worker: { globalTime: 738, processingTime: 707 }
worker-farm: { globalTime: 885, processingTime: 866 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 738, processingTime: 718 }
worker-farm: { globalTime: 865, processingTime: 849 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 708, processingTime: 685 }
worker-farm: { globalTime: 769, processingTime: 753 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 682, processingTime: 656 }
worker-farm: { globalTime: 780, processingTime: 764 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 704, processingTime: 684 }
worker-farm: { globalTime: 775, processingTime: 757 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 705, processingTime: 677 }
worker-farm: { globalTime: 767, processingTime: 748 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 700, processingTime: 675 }
worker-farm: { globalTime: 766, processingTime: 751 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 728, processingTime: 702 }
worker-farm: { globalTime: 770, processingTime: 755 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 721, processingTime: 695 }
worker-farm: { globalTime: 769, processingTime: 756 }
---------------------------------------------------------------------------
jest-worker: { globalTime: 702, processingTime: 675 }
worker-farm: { globalTime: 801, processingTime: 784 }
---------------------------------------------------------------------------
total worker-farm: { wFGT: 7947, wFPT: 7783 }
total jest-worker: { jWGT: 7126, jWPT: 6874 }
---------------------------------------------------------------------------
% improvement over 10000 calls (global time): 10.330942494022901
% improvement over 10000 calls (processing time): 11.679301040729795

Coverage

Coverage

@mjesun
Copy link
Contributor Author

mjesun commented Sep 17, 2017

@aaronabramov Tests failed on Node 4 because ... is not parseable; any idea why?

@thymikee
Copy link
Collaborator

@mjesun We transform rest/spread only in function declaration params (transform-es2015-parameters plugin). You'll need to use call/apply directly instead.

@mjesun mjesun requested a review from cpojer September 17, 2017 19:08
import type {Readable} from 'stream';

/**
* The Metro Bundler farm is a class that allows you to queue methods across
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/Metro Bundler farm/JestFarm

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

});

this._child = child;
this._busy = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this needed to be here? AFAICT this could go into the constructor directly (if creating a process was asynchronous then it would make sense, but then you would also have to set busy = true at the beginning of the initialization and then call _process() at the end).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The initialize method can be called multiple times; not only when instantiating JestWorker, but also if the child process unexpectedly dies. Once the process is restarted, it cannot be busy (whatever was the state of the previous process), so that's why it's set here.

};

for (let i = 0; i < options.workers; i++) {
const metroWorker = new JestWorker(workerOptions);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

metro... 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@rafeca
Copy link
Contributor

rafeca commented Sep 17, 2017

Awesome job! I love how easy is to understand the internal logic!

One question: could a user of this farm have sticky workers without having to decide explicitly in which worker gets initially assigned to each call?

Let me explain: As a user, I would prefer not to force JestFarm to process each task on a specific worker the first time the task is executed. Instead, I would like JestFarm to assign a random worker based on load on the first execution (JestFarm will be more optimal by assigning the task to the worker that its free) and stick to it on the next calls (so the cache is reused).

AFAICT this would not be possible with the current API: JestFarm would need to expose the worker that has processed each task inside the response, this way the user could use this information in his implementation of getWorker().

This use case would work really well for Metro Bundler, since in general hundreds/thousands of different tasks are processed concurrently on the first load (when building the initial bundle), but only a small amount of concurrent tasks are processed afterwards (incremental builds), which we want to be sticky.

@mjesun
Copy link
Contributor Author

mjesun commented Sep 18, 2017

@rafeca What you described is exactly how jest-parallel works: you can return null or any string from the getWorker method. If you return null, or if you return any string that was never seen before, the task will be executed ASAP by the first available worker, no matter which one it is.

Once executed, the string will be tied to the worker that executed it, so the next time you call the task and you return that same string, the task will be executed by the worker that processed your first call.

This allows for optimal parallelization of tasks when there is no worker preference, but it also allows to get a sticked worker when needed. I changed the README.md to add some information about how sticky workers work, in order to clarify it.

@codecov-io
Copy link

codecov-io commented Sep 18, 2017

Codecov Report

Merging #4497 into master will increase coverage by 1.62%.
The diff coverage is 72.91%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4497      +/-   ##
==========================================
+ Coverage   55.68%   57.31%   +1.62%     
==========================================
  Files         186      194       +8     
  Lines        6348     6494     +146     
  Branches        3        3              
==========================================
+ Hits         3535     3722     +187     
+ Misses       2812     2771      -41     
  Partials        1        1
Impacted Files Coverage Δ
...r/src/__performance_tests__/workers/worker_farm.js 0% <0%> (ø)
...ages/jest-worker/src/__performance_tests__/test.js 0% <0%> (ø)
...est-worker/src/__performance_tests__/workers/pi.js 0% <0%> (ø)
...r/src/__performance_tests__/workers/jest_worker.js 0% <0%> (ø)
packages/jest-worker/src/worker.js 100% <100%> (ø)
packages/jest-worker/src/index.js 100% <100%> (ø)
packages/jest-worker/src/types.js 100% <100%> (ø)
packages/jest-worker/src/child.js 100% <100%> (ø)
...ackages/jest-editor-support/src/test_reconciler.js 73.58% <0%> (-4.68%) ⬇️
.../eslint-plugin-jest/src/rules/no_disabled_tests.js 94.44% <0%> (-2.78%) ⬇️
... and 185 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1cda2e2...007b3d2. Read the comment docs.

The only exposed method is a constructor (`JestFarm`) that is initialized by passing an options object. This object can have the following properties:


### `exposedMethods: $ReadOnlyArray<string>` (required)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if you want to call a module.exports = function() {}? This is done internally in jest now

I came here to ask about using the babel interop to make transpiled export default work, but if a default export is not supported, I guess we can just do exposedMethods: ['default']

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, the way default exports are exported by Babel is by adding a default key to the object, so it is technically possible to call them with the current implementation.

That said, standard exports can't be called, so this is a fair point. I will change the default exported object of getApi from an Object.create(null) to a bound _makeCall that calls with default.

Copy link
Member

@SimenB SimenB Sep 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I modified the code, so api is now also a wrapped call :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome!

Any chance of using the babel interop instead of plain require, or do you think that'd be confusing behavior?

Copy link
Member

@SimenB SimenB Sep 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And to clarify, is the way you invoke the main function by passing exposedMethods: [''] or exposedMethods: []? And call it by getApi()()?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'm fine implementing the Babel logic to be able to treat Babel-ified modules as if they were a standard one. Regarding the [''], it is not necessary to provide it, the default export is always exposed (i.e. the result of getApi() is always a method). This makes me think that exposedMethods should then become optional (there is no sense in making [] a required option).

I will also adjust README.md to add more information about default exports and potentially provide an example on how to use it.


### `getWorker: (method: string, ...args: Array<any>) => ?string` (optional)

Every time a method exposed via the API is called, `getWorker` is also called in order to stick the call into a worker. This is useful for workers that are able to cache the result or part of it. You stick calls to worker by making `getWorker` return the same identifier for all of them. If you do not want to stick the call to any worker, return `null`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!!! could we have a simple example of usage with sticky workers in the usage section?

@rafeca
Copy link
Contributor

rafeca commented Sep 18, 2017

@rafeca What you described is exactly how jest-parallel works: you can return null or any string from the getWorker method. If you return null, or if you return any string that was never seen before, the task will be executed ASAP by the first available worker, no matter which one it is.

Ohhh nice!! 😃 I misread the Readme, thanks for making it more clear. Apart from that, I would suggest changing the getWorker() method name: I find it misleading since it's not actually returning a worker, but an identifier for each call that will be used to stick calls to specific workers. A suggestion for the new name could be getStickyId(), but if you have a better name feel free to use it 😀

Allow customizing all options passed to `childProcess.fork`. By default, some values are set (`cwd` and `env`), but you can override them and customize the rest. For a list of valid values, check [the Node documentation](https://nodejs.org/api/child_process.html#child_process_child_process_fork_modulepath_args_options).


### `getWorker: (method: string, ...args: Array<any>) => ?string` (optional)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getWorker sounds as if this would return a worker, which it does not.

What about getWorkerKey, or computeWorkerKey, or computeKey?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, from all given names the one I like the most is getWorkerKey. It is a getter, and tells stuff about the workers. I will adjust all references to have that new name.

* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
*
* @format
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think these are needed in this code base

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😅 true! These ones come from a copy of an internal codebase. I will remove them 🙂

@mjesun
Copy link
Contributor Author

mjesun commented Sep 22, 2017

@SimenB Updated child.js so that it is able to use Babel __esModule's now.
@davidaurelio Updated getWorker to getWorkerKey, as well as the README.md file.
@rafeca Provided example on how to use the getWorkerKey functionality.

@mjesun
Copy link
Contributor Author

mjesun commented Sep 22, 2017

The test for Node 4 broke with a Segmentation fault :| Is there any way to re-trigger them?

@aaronabramov
Copy link
Contributor

there's a "rebuild" button in the top right corner.
i just restarted the build

@@ -0,0 +1,173 @@
# jest-parallel

Module for executing heavy tasks under forked processes in parallel, by providing a modern interface (`Promise` based), minimum overhead, and sticky workers.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just say "a Promise based interface" rather than modern? The word modern only works at one point in time, and will sound dated in a year. Make it timeless.


The module works by providing an absolute path of the module to be loaded in all forked processes, as well as the list of methods that can be remotely called. All methods are exposed on the parent process as promises, so they can be `await`'ed. Child (worker) methods can either be synchronous or asynchronous.

The way sticky workers work is by using the returned string of the `getWorkerKey` method. If the string was used before, the call will be queued to the related worker; if not, it will be executed by the first available worker, then sticked to the worker that executed it; so the next time it will be processed by the same worker. If you have no preference on the worker executing the task, but you have defined a `getWorkerKey` method because you want _some_ of the tasks to be sticked, you can return `null` from it.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain at first what a sticky worker is meant to be before you explain how to use it? Instead of saying "sticked", can you say "bound"? "it will be executed by the first available worker, then bound to the that same worker for subsequent calls"

## Install

```sh
$ npm install --save jest-parallel
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just... no!

Allow customizing all options passed to `childProcess.fork`. By default, some values are set (`cwd` and `env`), but you can override them and customize the rest. For a list of valid values, check [the Node documentation](https://nodejs.org/api/child_process.html#child_process_child_process_fork_modulepath_args_options).


### `getWorkerKey: (method: string, ...args: Array<any>) => ?string` (optional)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would go with @davidaurelio's suggestion and use computeWorkerKey because this function is potentially expensive, and it would be good to denote that in the name.

// Wait a bit.
await sleep(10000);

// Transform again the same file. Will immediately return because the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Transform the same file again"

@@ -0,0 +1 @@
node_modules
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add this to the top level gitignore file instead? Thanks!

@@ -0,0 +1,13 @@
{
"name": "jest-parallel",
"version": "20.0.4",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update to latest jest version pls.

const pi = require('./pi');

module.exports.loadTest = function() {
pi();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if you want to test perf accurately, you need to return the value here back to the parent process.


/* eslint-disable no-unclear-flowtypes */

export type JestApi = {[string]: (...any) => Promise<any>};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you drop "Jest" from the types everywhere? I don't really think that adds much.

const aggregatedStderr = mergeStream();
const workers = [];

if (!path.isAbsolute(options.workerPath)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we instead just run "require.resolve" here ourselves? It will throw if the module isn't found, and it allows people to pass my-worker-package which may be a form of a "plugin API" for workers some day.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't. require.resolve is relative to the file calling it. If we call it ourselves, all paths will always be relative to jest-parallel itself, which is undesirable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's call require.resolve anyway as it'll work with packages (like 'my-worker-package') and throw otherwise?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nah, I think that's going too far. It should either be a pre-resolved module or a node_module.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolve-from would work, but you need to go one level up the stack to check who called into jest-parallel. I really dislike that approach.

return this._api;
}

getAggregatedStdout(): Readable {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think just getStdout is sufficient as a name for this, no? It's implied that it is the aggregated data.

_aggregagedStderr: Readable;
_api: JestApi;
_ending: boolean;
_hashes: {[string]: JestWorker};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: rename to "_keys" or "_cacheKeys"

}
}

module.exports = Worker;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

export default

"type": "git",
"url": "https://github.com/facebook/jest.git"
},
"license": "BSD-3-Clause",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MIT. All headers as well

@cpojer
Copy link
Member

cpojer commented Oct 3, 2017

Oh yeah, we switched from BSD+Patents to MIT license since your PTO.

@mjesun
Copy link
Contributor Author

mjesun commented Oct 3, 2017

One thing we leverage in Jest is transient failures/retries. Is jest-worker going to implement this in a follow-up or are we going to push the responsibility up the stack?

When a worker fails (OOM, the process exits, throws...), a new worker is spawned to cover the empty gap left by that worker that died. Calls are not removed from the queue (and not marked as processed) up until we get a result back, so as soon as the new worker is up, the last call will be passed into it again.

@cpojer
Copy link
Member

cpojer commented Oct 3, 2017

What happens if the same call to the worker throws every time?

@mjesun
Copy link
Contributor Author

mjesun commented Oct 3, 2017

A call to the worker throwing does not mean the process exits; so nothing happens. If the process is consistently dying, it is re-spawned all the time; but that's a failure on the jest-worker level and not on the "forked module" level.

@SimenB
Copy link
Member

SimenB commented Oct 3, 2017

Sounds like it should only retry a set amount of times before throwing, so we avoid it retrying forever and making the process seemingly hang

@cpojer
Copy link
Member

cpojer commented Oct 3, 2017

Exactly, worker-farm throws after like 2 retries and exits, I think we should retain that.

@mjesun
Copy link
Contributor Author

mjesun commented Oct 3, 2017

I'm not very sure about that. If we have workers OOMing and we don't care about that, after three OOMs we would kill the worker without further notice, and you're trapped. Adding an option as well to allow the amount of times this can happen would complicate the API.

@cpojer
Copy link
Member

cpojer commented Oct 3, 2017

@mjesun I'm not sure I follow. With the code as you explained it, it'll cause an infinite loop of the child crashing and the parent retrying the same thing. Keep in mind we don't have control over the code that is run in the worker (for example with Jest we run user code in tests in child processes). Right now, if one test crashes the process, we retry twice (could be an intermittent failure) and then worker-farm bails and says it cannot retry. What will the behavior be with jest-worker?

@mjesun
Copy link
Contributor Author

mjesun commented Oct 3, 2017

What I mean is that when a child crashes is not because a method called threw; it is because something, either on the jest-worker side, or on the Node internals, made it crash. In that case, I think it's ok to re-spawn the thread. If the thread keeps dying, it's almost certainly because of a bug on jest-worker and not on the child module itself.

@cpojer
Copy link
Member

cpojer commented Oct 3, 2017 via email

process.send([
PARENT_MESSAGE_ERROR,
error.constructor.name,
error.constructor && error.constructor.name,
error.message,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens when a string is thrown? throw 'Foo'?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just added a commit to allow this.

@cpojer
Copy link
Member

cpojer commented Oct 4, 2017

Let's ship it.

@cpojer cpojer merged commit ab6d017 into jestjs:master Oct 4, 2017
@pedrottimark
Copy link
Contributor

Awesome work! For your info, after yarn clean-all && yarn local yarn jest fails:

FAIL  packages/jest-worker/src/__tests__/child.test.js
 ● lazily requires the file

   TypeError: Cannot read property 'concat' of undefined

     at handle (node_modules/worker-farm/lib/child/index.js:44:24)
     at emitOne (events.js:120:20)
     at process.emit (events.js:210:7)
         at Promise (<anonymous>)

MacOS 10.12.6
yarn 1.1.0
node 8.2.0 and 6.11.3 and 4.8.4

However yarn jest child succeeds.

@mjesun
Copy link
Contributor Author

mjesun commented Oct 4, 2017

Interesting. It might be related to the fact that child.js messes with process.send, which is used also by the internal Jest worker (right now, worker-farm, soon jest-worker) to perform IPC communication.

I also wonder why CI environments, which are always starting from a clean environment, did not catch it. I will try to reproduce locally and find a fix if it reproduces. Thanks for noticing!

@pedrottimark
Copy link
Contributor

pedrottimark commented Oct 4, 2017

There is vague memory of earlier discussion about number of threads testing locally versus CI?

EDIT: yarn jest --maxWorkers=1 passes on node 8.2.0

@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.