There are plenty of unit/regression testing tools for Emacs, and even some for functional testing. What Emacs is missing though is a really good testing framework for integration testing. This is where Ecukes comes in.
Cucumber is a great integration testing tool, used mostly for testing web applications. Ecukes is Cucumber for Emacs. No, it's not a major mode to edit feature files. It is a package that makes it possible to write Cucumber like tests for your Emacs packages.
Ecukes is not a complete clone of Cucumber and is not intended to be. If however Ecukes is missing some feature that you feel really should be included, please make a bug report.
If you don't know anything about Cucumber I suggest you read up a bit about it before continuing with Ecukes.
Debugging
Example
Contribution
Add ecukes
to your Cask file:
(source melpa)
(package "super-duper" "0.0.1" "Super Duper.")
(development
(depends-on "ecukes"))
Then run cask
to install.
Create an Elisp file in the root directory called super-duper.el
and
(provide 'super-duper)
. To setup Ecukes for the project, run:
$ cask exec ecukes new
create features
create step-definition
create super-duper-steps.el
create support
create env.el
create super-duper.feature
The project will now look like this:
$ ls super-duper
super-duper/
|-- README
|-- features
| |-- step-definitions
| | |-- super-duper-steps.el
| |-- super-duper.feature
| |-- support
| |-- env.el
|-- super-duper.el
First off, install cucumber.el to get proper syntax highlighting and indenting.
To run Ecukes, use:
$ cask exec ecukes [COMMAND] [OPTIONS]
If you add a file called .ecukes
in the project root, options are
automatically used for each run. For example:
--tags ~@exclude
--no-win
--reporter gangsta
Ecukes can run in a few different modes: Script (--script
), no win
(--no-win
) and win (--win
).
If no mode is specified, the features will run in script mode. This
will run the features as a batch job (emacs --script
or emacs --batch
).
This mode has a few quirks and it's recommended as a way to run
features in local development, because it's fast and the output is
printed directly (there's no way in Emacs to print to stdout
or
stderr
when not running as a batch job).
It is for example not possible to test font faces in script mode. There's also no concept of lines so it's not possible to make assertions on lines.
If you run features on travis, use --no-win
mode.
Example:
$ cask exec ecukes
$ cask exec ecukes --script # same as above
In this mode, Ecukes will run Emacs with the -nw
option.
Example:
$ cask exec ecukes --no-win
This will start up an Emacs window and run the features. To use this
mode, use the --win
option.
Example:
$ cask exec ecukes --win
Features and scenarios can be tagged using syntax @tag
, for example:
@core
Feature: Foo
@wip @io
Scenario: Bar
@io
Scenario: Baz
All tags on the feature will be copied down to the scenarios.
You can choose to run scenarios tagged with a tag using:
$ cask exec ecukes --tags @io
or run scenarios not tagged with a tag using:
$ cask exec ecukes --tags ~@io
or combine the two:
$ cask exec ecukes --tags @io,~@wip
Ecukes has support for reporters so that you can get the output in a few different ways depending on your philosophical bent. To list all available reporters, run:
$ cask exec ecukes list-reporters
To use a specific reporter, run:
$ cask exec ecukes --reporter gangsta
Default reporter is dot
.
You can run scenarios matching or not matching a pattern using
--pattern
(-p
) and --anti-pattern
(-a
) respectively. For example:
$ cask exec ecukes --pattern awesome
$ cask exec ecukes -p awesome almost-awesome
$ cask exec ecukes --anti-pattern awesome
$ cask exec ecukes -a awesome almost-awesome
If you have a failing scenario and you only want to run that, you can
use a tag, but you can also use the --only-failing
(-f
)
option. For example:
$ cask exec ecukes --only-failing
There are three different kind of steps: Regular, table and py string. Check out Espuds for a comprehensive collection of steps.
A regular step always only consumes one line. The most basic step looks like this:
Given a known state
The corresponding step definition is:
(Given "a known state"
(lambda ()
;; ...
))
The second argument could also have been the symbol of a function name:
(Given "a known state" 'do-something)
Steps accept arbitrary arguments. The arguments sent to the step definition function are all regular expression match groupings from the matching of the step name and the match string.
Single argument:
Given I am in buffer "buffer-name"
The corresponding step definition is:
(Given "I am in buffer \"\\(.+\\)\""
(lambda (buffer-name)
;; ...
))
Multiple arguments:
Given I am in buffer "buffer-name" with text "Foo"
The corresponding step definition is:
(Given "I am in buffer \"\\(.+\\)\" with text \"\\(.+\\)\""
(lambda (buffer text)
;; ...
))
A table step looks like this:
Given these meals:
| meal | price |
| Hamburger | $4.50 |
| Pizza | $5.30 |
The corresponding step definition is:
(Given "these meals:"
(lambda (meals)
;; ...
))
The argument meals
is a simple list where the car
of the list is
the header and the cdr
or the list are the rows.
The header would in the above case be:
("meal" "price")
And the rows would be:
(("Hamburger" "$4.50")
("Pizza" "$5.30"))
To pick out the header and rows attribute from a table you do:
(let* ((table ...)
(header (car table))
(rows (cdr table)))
;; ...
)
It is also possible to send arguments to a table step. The table list will be the last argument.
Given these meals at "fast food":
| meal | price |
| Hamburger | $4.50 |
| Pizza | $5.30 |
The corresponding step definition is:
(Given "these meals at \"\\(.+\\)\":"
(lambda (restaurant meals)
;; ...
))
A Py string step looks like this:
Given the following text:
"""
some text
"""
The corresponding step definition is:
(Given "the following text:"
(lambda (text)
;; ...
))
It is also possible to send arguments to a Py string step. The Py string will be the last argument.
Given the following text in buffer "buffer-name":
"""
some text
"""
The corresponding step definition is:
(Given "the following text in buffer \"\\(.+\\)\":"
(lambda (buffer text)
;; ...
))
In order to keep your steps DRY, you can call steps from other steps like this:
(Given "I go to a"
(lambda ()
;; Go to a
))
(Given "I go to b"
(lambda ()
;; Go to b
))
(Given "I go from a to b"
(lambda ()
(Given "I go to a")
(Given "I go to b")))
You can also pass arguments when calling other steps, like this:
(Given "I am in buffer \"%s\"" buffer-name-variable)
Or like this:
(Given "I am in buffer \"buffer-name\"")
If you pass one extra parameter to a step definition, that is used as a callback. Once called, the step is considered done.
(When "^run command \\(.+\\)$"
(lambda (command callback)
(run-command command callback)))
If callback
is not called within 10
seconds, it will fail
automatically. The timeout can be changed using the --timeout
(-t
), for example:
$ cask exec ecukes --timeout 30
Checkout the features for Async for examples how to use: https://github.com/rejeep/ecukes.el/blob/master/features/async.feature
You can list all defined steps with list-steps
, for example:
$ cask exec ecukes list-steps
If you want to get the step description, use the --with-doc
option:
$ cask exec ecukes list-steps --with-doc
And if you want to know where it is defined, use the --with-file
option:
$ cask exec ecukes list-steps --with-file
Ecukes provides a few different hooks. They are useful if you test your program and change the state in some feature. Since all scenarios execute in the same environment, the state change will affect all scenarios after. You can solve that by resetting the state in a before or after hook.
Hooks should be placed in any file in the project features/support
directory.
Runs once before anything runs.
(Setup
;; Run code
)
Runs once before each scenario runs.
(Before
;; Run code
)
Runs once after each scenario runs.
(After
;; Run code
)
Runs once when everything is done.
(Teardown
;; Run code
)
Runs when a scenario fails.
(Fail
;; Run code
)
If you have a failing scenario and you cant figure out how why, first
off, try adding the --debug
flag. For example:
$ cask exec ecukes --debug
Next step if that doesn't help, use the --error-log
(-l
)
argument. If an error occurs, the backtrace will be logged to that
file.
$ cask exec ecukes --error-log ecukes.err
To get an idea of how Ecukes can be used, here is an example from a feature in drag-stuff.
~/Code/drag-stuff $ cask exec ecukes features/line.feature --reporter spec
Feature: Drag line
In order to move a line up and down
As an Emacs user
I want to drag it
Background:
Given I am in the buffer "*drag-stuff*"
And the buffer is empty
And I insert
"""
line 1
line 2
"""
Scenario: Drag line up
When I go to line "2"
And I press "<M-up>"
Then I should see:
"""
line 2
line 1
"""
Scenario: Drag line down
When I go to line "1"
And I press "<M-down>"
Then I should see:
"""
line 2
line 1
"""
2 scenarios (0 failed, 2 passed)
11 steps (0 failed, 11 passed)
All contributions are much welcome and appreciated!
Before submitting a patch, make sure to write a test for it (if possible). Ecukes is unit tested with a testing framework called Ert (Emacs Lisp Regression Testing). But most notably, it's tested using Ecukes! :)
To run the tests, you have to install
Cask if you haven't already. Once
installed, run the cask
command to install all dependencies.
To run all tests, simply run the make
command.
You can always report issues on Github if you're not up to fixing it yourself.