Scenario outlines and parallel execution

[rdmark]

Is there a summary somewhere that explains which cucumber implementations support true parallel execution of scenario outlines?

The documentation only covers Java (cucumber-jvm) as far as I can tell. Does this mean that this capability is only present in the Java implementation?

https://cucumber.io/docs/guides/parallel-execution/?lang=java

In my current team we have run experiments with cucumber.js (Node.js) and cucumber.cpp but with no success in achieving parallel test execution of scenario outlines.

Our use case is functional system integration testing of a complex cyclical state machine where we want to use test parallelization to simultaneously control multiple moving units on a linear route.

[mpkorstanje]

There is no feature matrix.

Cucumber JS supports parallel execution. See https://github.com/cucumber/cucumber-js/blob/main/docs/parallel.md

Cucumber CPP is ran through Cucumber Ruby, which has some but limited support for parallelism. See cucumber/cucumber-ruby#1760. Though I can't say how much of that will work in combination with Cucumber CPP.

which cucumber implementations support true parallel execution of scenario outlines?

What do you mean by true parallel execution?

Our use case is functional system integration testing of a complex cyclical state machine where we want to use test parallelization to simultaneously control multiple moving units on a linear route.

It sounds like you want to set the conditions of your test as a second order effect of how Cucumber runs your tests.

This is not advisable. Cucumber treats each scenario and example as independent. The correctness of your overall test should not depend on the exact manner in which Cucumber executes individual scenarios and examples.

Instead of running Cucumber scenarios in parallel you may want to consider merging your scenario outlines into a single scenario. Then within that scenario use a data table to set up the actors in your system and run these actors in parallel.

I.e:

Scenario: Example 
  Given these actors
    | actor | speed |
    | A1.    | fast | 
    | B2.    | slow |
  When ran in parall
  Then it worked 

[rdmark]

Thanks for your response, this is very helpful.

To expand a bit on our use case: For each actor, we have a common sequence of actions that are executed in order. All actors traverse a shared linear path, and there is queueing involved where one actor has to complete their task and move on for the next actor in line to proceed. Hence, we concluded that test parallelism is the most feasible option, while relying on the SUT's queueing logic to manage the interdependence between actors (i.e. the test scenario will asynchronously wait for a particular flag to change.)

A very simplified example of the cyclical scenario:

1. Send command to actor
2. Validate that actor state immediately changes to "X"
3. Validate that actor state changes to "Y" after n seconds
4. Send command to actor
5. Validate that actor state immediately changes to "X"
6. Validate that actor state changes to "Y" after n seconds

The complexity when trying to define this as a single scenario occurs at step 5, because actor#1's state has to change to "X" in order for actor#2 to be allowed to change to "Y" (step 3). The complexity increases exponentially with more actors. Whereas with parallel execution, the test scenario can be kept very simple, and easy to scale by adding more actors.

Do you have a suggestion how to translate this into the structure that you suggest, with actors defined in a data table of a single scenario?

BTW please disregard the "true" statment. Wrong choice of word.

[mpkorstanje]

Without knowing the specifics it is hard to go beyond the advice that when your system involves actors acting in parallel, setting up the exact conditions in which those actors act should be done inside of a scenario, not outside of it.

I'm also having a hard time imagining what exactly you might be testing. From your example it seems that the other actors are not actually relevant for your tests. You are for example, not checking if the "one actor at time" invariant holds. Only that an actor proceeds, in a timely manner, regardless of what other actors are doing.

It does come to mind that you might be mis-using Cucumber for a load test. In which case you might want to look into dedicated load testing tools instead.

Hence, we concluded that test parallelism is the most feasible option, while relying on the SUT's queueing logic to manage the interdependence between actors (i.e. the test scenario will asynchronously wait for a particular flag to change.)

The complexity when trying to define this as a single scenario occurs at step 5, because actor#1's state has to change to "X" in order for actor#2 to be allowed to change to "Y" (step 3) The complexity increases exponentially with more actors. Whereas with parallel execution, the test scenario can be kept very simple, and easy to scale by adding more actors.

It seems that between these two statements a new requirement materialized. When running your scenarios in parallel you could rely on the SUTs queueing logic to manage the interdependence between actors, but when running the actors in parallel inside a scenario there now an explosion of complexity that can not be handled by the SUTs queueing logic?