Scenario Outlines / Examples

[hairyhum]

Hello. I made some work on scenario outline and examples. Maybe it can be useful

[jbpros]

Thanks @hairyhum for the pull request. This is on my TODO list.

[jbpros]

This is a potential resolution for #10.

[darrhiggs]

May you let me know an estimate time to inclusion? @jbpros

[sheebz]

+1

[darrhiggs]

@jbpros what is the hold up with this?

[mattwynne]

@darrhiggs what are you, his project manager?

[jbpros]

@darrhiggs I've been waiting for some traction from users on this particular feature before considering its implementation. It should land in 0.3.1. ETA unknown. As with all core features, it has to be carefully tested through cucumber-tck. .

[darrhiggs]

@jbpros
Ok, thanks for the response.
As I don't follow you on twitter so I wasn't able to raise my hand a few months back when you did a poll to gauge interest

@mattwynne
No, why?

[kevinpeno]

+1

[sivad77]

+1 here.

[mummybot]

+1 :)

[jbpros]

Lots of plus ones!

Would you guys mind describing some use cases for scenario outlines/examples (some example features would be the best). I'm picky on this but I rarely see good uses of them. Please, convince me it's not as bad as I think 😁

BTW, be reassured it will eventually land in cukejs. It's quite on top of the priorities, in fact.

[jbpros]

From @dkowis, via Twitter:

http://pastebin.com/Ww4U1Yen
http://pastebin.com/mKKTYWPc

[aslakhellesoy]

Dudes use gists :-)

https://gist.github.com/3948205

[darrhiggs]

https://gist.github.com/3948260

[darrhiggs]

@jbpros let me know if I can do anything to help this along, or far that matter any part of this project. Currently I am using tags to associate the expanded versions of these with the Scenario Outline versions that are currently sat idle in a folder.

[jbpros]

@aslakhellesoy totally.

@darrhiggs thank you! I know I'm a complete foreigner to the domain of that scenario but could you tell me what is the rationale behind those numbers? They seem to come out of the blue and I'd have to do some reverse maths to understand the business rules.

[darrhiggs]

@jbpros I have sent you an email.

[sivad77]

hey mate, i find scenarios outlines useful to eliminate repetition in my scenarios. we can easily refactor the repetition into a table where the information is communicated concisely. i understand some may say scenario outlines are a code smell however i use cucumber for integration testing (full system stack) where i may have many happy paths which can be tested easily using a table! thanks for the awesome work!!!!

[neilD2]

+1

[jbpros]

Yep, I made some progress. Still working on it :)

[JRedgrave]

Julien , any idea when we are likely to see this?

[aslakhellesoy]

@jbpros: You could consider holding off on this until gherkin3.js is done - it precompiles scenario outlines (and background) into regular scenarios, making the cucumber code a lot simpler.

gherkin3 is still in early stages, but the js implementation is the one that has made the most progress (over java/c/ruby).

If you want to play with it, check out the gherkin branch in the bool repo where it currently lives.

[jbpros]

Thanks for the heads up Aslak. I'm in the middle of this work but I'll have a look at gherkin 3 then.

This is the most requested feature atm. Holding it off for some more months could annoy (to say the least) some people ;)

Due to cucumber.js's decoupled architecture, moving to gherkin3 shouldn't be of a big issue, hopefully. It means we can implement outlines/examples the old way for now (the same way I implemented backgrounds: as simple macros to prepend/append steps/scenarios) and move to gherkin3 when it's ready.

[JRedgrave]

Yes please don't hold off on this any longer, I have been waiting patiently for several months now and i am sure i am not alone :)

[aslakhellesoy]

Gherkin3 is several months away, so don't let it hold up this PR. As you say we can swap it out later.

[toolness]

Hey, I just bought The Cucumber Book and am attempting to go through its second chapter with cucumber-js instead of the Ruby version. Unfortunately, section 2.8 (page 22) involves converting their scenario into a scenario outline, which fails on cucumber-js and points me here. While I really appreciate it pointing me here (instead of exploding in confusion), it'd be really great if the feature worked.

TL;DR: I guess this is just another +1. :)

[mseankelly]

+1

[mandalae]

+1

[maciejblinkbox]

@jbpros I think we can still use scenario outlines here:

So instead of having the following:

Scenario: premier discount on shipping fees
  Given I am a member of the premier discount group
  When I buy less than 4 items
  Then I get the premier discount on the shipping fees

Scenario: staff discount on shipping fees
  Given I am a member of the staff discount group
  When I buy less than 4 items
  Then I get the staff discount on the shipping fees

We could have one scenario outline:

Scenario Outline: <GROUP> discount on shipping fees
  Given I am a member of the <GROUP> discount group
  When I buy less than 4 items
  Then I get the <GROUP> discount on the shipping fees

Examples:
  | GROUP   |
  | premier |
  | staff   |

[JRedgrave]

Julien, are we any closer to getting scenario outlines in place?

[niclashoyer]

@jbpros @aslakhellesoy @hairyhum
I also bought the cucumber book as a guidance for writing features. Now that I have my first scenarios (and scenario outlines) in place I wondered about failing builds. Also giving 👍 for not holding this any longer, because more and more people might read the book and fail to implement the features with JavaScript which leads to frustration.

[christian-bromann]

@jbpros I read this thread and see advantages and disadvantages on both sides. In my opinion the user should decide if he wants to use scenario outlines or not. I'ld appreciate this feature!

Anyway, thanks for your great work!

[mattwynne]

Folks, all of you sending +1s, are you thinking this is going to help Julien to code faster?

Why not send him a Gittip instead?

[the-simian]

I would really love to see Scenario Outlines implemented as a feature in cucumber.js Unfortunately this is a non-starter for my shop until that happens. I hope things are going well, and this will come soon.

[manuels]

a.k.a. I wanna make money, so spend your weekend writing code for me!

[AdrienLemaire]

Same here, was really excited when I saw that RTD was integrating cucumber, but seeing that Scenario outlines and examples are not yet implemented (after 2 years), I'll skip it.

[vantreeseba]

So, I've got this working again on my local fork.

The question I have about this implementation is...

It builds steps individually and runs them outside of the scenario context (i.e. instead of building a scenario per example row, it builds the steps themselves to run).

I know that @jbpros is working on an implementation of this, or was, so what I was wondering was if the implementation is changed to produce entire scenarios, renaming them based on the scenario outline, would this make a more acceptable implementation of this feature?

For example:

Feature: testing scenarios
        Background:
          Given a background step

        Scenario Outline: outline
          When a <some> step
          Then i get <result>
        Examples:
          | some    | result  |
          | passing | passed  |
          | failing | skipped |

is not producing multiple scenarios, but instead only generating many steps like ..

when a passing step, then i get passed,
when a failing step, then i get skipped.

Instead, I am proposing that internally it would look more like

Feature: testing scenarios
        Background:
          Given a background step

        Scenario: outline #1
          When a passing step
          Then i get passed

        Scenario: outline #2
          When a failing step
          Then i get skipped

This would allow all of the formatters and other tools to continue using the same data structures without needing to account for outlines in specific.

I would appreciate any feedback on this.

Thanks!

As a side note, I don't know how the ruby implementation deals with outlines, so I'm going to do some digging into that as well.

[jbpros]

@vantreeseba thanks for sharing your thoughts. Your proposed solution is almost exactly what I'd like to end up with. Scenario outline examples yielding scenarios is the cleanest solution, imo. Actually, the gherkin syntax for scenario outlines and examples itself has always looked as a kind of macro set to generate scenarios on the fly, to me.

The only thing I wouldn't do is to change the generated scenario names as I don't want it to be some arbitrary string end users have no control over.

Based on your example, here is the resulting feature (and this would be the output of the pretty formatter):

Feature: testing scenarios
  Background:
    Given a background step

  Scenario: outline
    When a passing step
    Then i get passed

  Scenario: outline
    When a failing step
    Then i get skipped

The visited AST objects would still carry the original outline/example data, meaning we can still use those information within XML/JSON/whatever formatters/listeners.

[mattwynne]

Bear in mind that you could have multiple examples tables, and that examples tables can have names.

In Cucumber Ruby 2.0, we're generating names for the test cases based on the names of the outline, examples table and the row.

See https://github.com/cucumber/cucumber-ruby-core/blob/master/spec/cucumber/core/test/case_spec.rb#L77-L110 for details.

I suggest you follow the same principle.

[the-simian]

@vantreeseba and I are working on this,
@jbpros Thank you for that clarification, and we are in complete agreement, though I think that we will follow the spec proposed in cucumber ruby 2.0 by @mattwynne. We are looking at that code now and will work on that today.

[jbpros]

@JesseHarlin @vantreeseba thank you chaps. I'm looking forward to seeing your work!

[the-simian]

@jbpros @mattwynne just a quick update:

@vantreeseba and I have had some success with this yesterday and today. In addition to getting Scenario Outlines working, we also worked in the snippet_builder to make sure the suggested step definitions look correct. We're doing some ex post facto cleanup before we submit the pull request. Thanks for your help, thusfar.

[jfstephe]

Hi all,
It's great that so much effort is being put into this. Scenario outlines are a big deal for us and I'm wondering roughly when (a day/a week/a month/a year) it is expected that this feature will be merged in? Knowing this would give me something to placate the management.

I'm also confused by #155. Do this and #155 do the same thing exactly? Is there some coordination so that multiple people aren't doing the same work? Honestly, I don't much care - I just would like the functionality! :-).

Please don't take this the wrong way and please keep up your great efforts!
Thanks,
John

[the-simian]

@jfstephe they achieve the same goal of implementing scenario outlines, but do not achieve that goal the exact same way. #155 is the more recent fix, and is linked because it also would resolve this issue when it is pulled in. I wouldn't worry about duplicate work. The project leads have been commenting on the work thusfar, and this thread - the one we are posting in is many years old anyways.

Go to #155 and start there to further discuss or review this feature, as it is the most up to date.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.