Using Cucumber CLI programatically #786
Comments
|
Hey Jan, I'm glad you're doing this! Serenity is great! I think you shouldn't use the CLI at all in your tests but rather invoke the Runtime directly. Could you try it and report back if anything goes wrong again or am I missing some context that prevents your from doing that? |
|
Hey @jbpros, thanks, the Runtime looks like a better alternative. It seems like I'd need to invoke the I'll look into using the Runtime over the next couple of evenings. My gut feeling is though that this Would you or @charlierudolph be able to see if my suggestion around defensive coding could work? |
|
I'm good with more defensive programming and/or trying to use the async version of stacktrace. Thoughts on first trying out the async version of stacktrace and then add the defensive programming only if needed after that? |
|
I think this approach makes sense, according to the manual:
Which is what might be causing the problem... In the meantime, I'll play around with using the Runtime directly as per @jbpros's suggestion Thanks! |
|
@jan-molak any update here? |
|
Hey @charlierudolph, I ended up using the CLI. Serenity/JS should be ready to use with CucumberJS 2 shortly. |
|
Okay. Can this issue be closed or is there more to be discussed? |
|
Hi guys! I've exactly the same problem as @jan-molak . I was running my Node project programmatically with the Cucumber1 CLI and it always worked pretty well. Now, i'm trying to migrate to Cucumber2 and using the same code (adapting it to the new implementation) and it brokes at unit testing. I'm doing something like this: const
cucumberInfo = {
argv: this.pluginArgs,
cwd: process.cwd(),
stdout: process.stdout
},
cucumberCli = new cucumber.Cli(cucumberInfo);
return cucumberCli.run()
.then((succeeded) => {
...
}).catch((error) => {
...
});@jan-molak have you solved the problem running with the CLI? |
|
@DavidGDD - yeah, CLI seems to be working fine for me - example here |
|
But @jan-molak, what have you change from your first version to your last version to solve the problem? I read your solution before writing but i don't see the difference... or is it in the unit test? |
|
I haven't changed anything between those two versions; My question was related to using Runtime class instead of the CLI; Runtime seemed cleaner but it had too many dependencies to get it to run so I settled on using the CLI class instead. Also, in my tests I'm starting a new cucumber process per test, so that might be what makes the difference. |
|
Cucumber V1 API differs from V2. V1 //CLI would just receive an array of argument
let cli = require('cucumber').Cli(runArgs);
//CLI.run received a callback function
return new Promise(function (resolve) {
return cli.run(function(result){
let exitCode = (result === true) ? 0 : 1;
return resolve(exitCode);
});V2 //CLI *Constructor* requires an options object with argv, cwd and stdout
let cli = new (require('cucumber').Cli)({argv: runArgs, cwd: process.cwd(), stdout: process.stdout});
//CLI.run returns a promise
return new Promise(function (resolve, reject) {
try {
return cli.run()
.then(success => resolve((success === true) ? 0 : 1));
} catch (e) {
return reject(e);
} |
|
@jan-molak one process per test it's the cleanest workaround to the issue reported by you. With the new version it's hard to run multiple Cucumber executions in the same process due to the Node require cache. |
|
@yaronassa i'm running cucumber as you propose without the try/catch block and the new Promise, because chaining a catch to the promises, a throw error will be managed. |
|
@DavidGDD - Agreed, I'd much rather run several Cucumber executions per Node process as that would greatly speed up my integration tests. To be fair though, this use case might be slightly incompatible with Cucumber's design goal - 1 execution per process. |
|
Reading the features testing implementation, in the World i see the test are running using the CLI programmatically as @jan-molak and me are trying. Executing the Cucumber features tests and logging the Stacktrace content, it always has traces, so never appears the problem with the stacktrace empty: In tests, the cucumber execution is running always with the const
cucumberInfo = {
argv: this.pluginArgs.concat(['--backtrace']),
cwd: process.cwd(),
stdout: process.stdout
},
cucumberCli = new cucumber.Cli(cucumberInfo);
return cucumberCli.run()
.then((succeeded) => {
...
}).catch((error) => {
...
}); |
|
How are we clearing the cucumber state. First test run passes: Second always fails with: I'm using version 4. The reason I'm asking is that I'm using cucumber as part of a production test service that stays running for potentially multiple test runs and it seems that there is state built up on the first test run that still exists after that, even though I'm creating a new cucumber.Cli on subsequent runs. This works, but there must be a better way: By the way, my code is all in a hapi route handler, so I'm not sure why state would be hanging around unless I start a new process on each request. |
|
Just ran into this problem today. If you try and create more than one Cucumber.Cli in the same node process it fails because cucumber relies on side effects when requiring your support library files to define steps, and subsequent requires do not execute the files so end up with no steps defined. Its a bit of a hack but we can force cucumber to reuse the valid support library by doing something like this: But it does assume you want the same library each time (e.g. for reruns). |
|
What exactly is the support code library @RalphSleighC4 ? I'm about to move to the next stage which at this stage is spinning up a container for each cucumber instance, as I need them running in parallel. Is what you're suggesting able to be run in parallel? Also how do you use what you're suggesting, is your code snippet a module that can be consumed and run many times via |
|
It's the code for the step definitions and hooks cucumber uses to run your scenarios in the files you specify using the --require argument. If we write them in the typical style: Then the first time cucumber requires them, the Given function is executed, but subsequent calls to require will return a cached version of the files module.exports, so the steps are not recreated and the run fails as undefined. Hence the need to cache the support library from the first run if you want to use the ClI more than once. This only applies to sequential runs inside the same process. If you are using containers to run tests in parallel you won't have this issue. |
|
Thanks @RalphSleighC4, I'm comming back to this soon, I'll have a play when I get there and provide my feedback. |
|
This doesn't work for my case. I use the same step definitions and hooks, but with different data applied. |
|
For anyone arriving here in the future looking for help: runCuc.js wraps and modifies slightly the cli run.js. I need to also modify I created a bin file similar to the cucumber-js bin file. The whole commit is here. All of the above is run in parallel with my app.parallel.js For testing (as working out what's going wrong with your steps can be tricky in parallel sometimes), I conditionally run in parallel or sequence (sequence only runce one test session for me as noted in my previous comment) by checking a config value Sequence looks like this Hope this helps someone. |
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Hey guys!
As part of my effort to make Serenity/JS play nice with Cucumber 2.0 (serenity-js/serenity-js#28) I'd like to invoke the Cucumber CLI programmatically from a Mocha test to avoid having to spawn a new process (which is over 20 times slower).
I appreciate that this use case is not fully compatible with the design goals of the CLI, but I think it shouldn't be difficult to make it work, not to mention that it would make my life much easier ;-).
The (simplified) structure of the code looks more or less like this:
As the CLI loads listeners, event handlers and step definitions, I'm invoking the
clearSupportCodeFns()after each scenario to get rid of the support code loaded for that scenario. I'm also invoking theclearRequireCache()to remove the step definitions from the node require cache, so that thedefineSupportCodefunction gets called again when the step defs file is loaded.I think (?) that this should be enough to clean up any side effects, but I'm observing a weird behaviour where:
stacktrace-jsnot being able to figure out the full stack trace ingetDefinitionLineAndUri. This is because the stack trace attached to the artificially generated error only contains frames related tostacktrace-jsitself, which then get filtered out, leaving the list empty.Because Cucumber receives an empty list, the
getDefinitionLineAndUrifunction's optimistic assumption about the list containing at least one element fails.This is manifested by the following error:
I think the issue could be resolved by either:
getDefinitionLineAndUrislightly more defensive, so that it checks if the stack trace contains any framesstacktrace.getSync()withstacktrace.get(), which provides a more sophisticated implementation which I think should tackle the problem and also provide support for source maps. This however would have to be done at the expense of making the function callasync.I suppose the first option should be good enough (?) and easy to implement, but I'm not sure what should be the sensible defaults provided in the absence of the stack trace?
I see that it's fine for the
urito be set tounknownfor example.If my thinking makes sense and I didn't miss any official way of cleaning up the cucumber state, I'd be happy to submit a PR implementing the first of the suggested solutions.
Thoughts?
All the best,
Jan
The text was updated successfully, but these errors were encountered: