-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Cucumber Test Suite
The correct functionality of OSRM is supported by a test suite based on Cucumber (https://cucumber.io/). The tests are run automatically on Travis CI (https://travis-ci.org/Project-OSRM/osrm-backend).
Overview:
Cucumber is the framework we use for writing and running tests. We use the JavaScript port of Cucumber, distributed as an NPM module. All support code and step definitions for the tests are written in JavaScript, and tests are written using the Gherkin syntax. They run against a compiled version of osrm-backend by using its commandline tools.
The test suite currently does not have complete coverage for all osrm-backend functionality, but we are striving for full test coverage.
Whenever you are making a change to osrm-backend, you should run the test suite locally to verify that the change does not introduce regressions. If your change introduces new or changed behaviour, you need to adjust the test suite accordingly.
- Have
osrm-backend
built locally as explained here- You need to build the Node bindings as well; when running
cmake
include-DBUILD_NODE_BINDINGS=ON
- You need to build the Node bindings as well; when running
- Install Node.js
- Tests require Node 4.0+. The
nvm
project is very helpful for installing and switching between Node versions.
- Tests require Node 4.0+. The
- Install Node dependencies
- Once Node.js 4.0+ is installed you can install modules via
npm
.npm install
will install all dependencies listed inpackage.json
, then runnpm link
to linkcucumber
as an executable. - Alternately, you can replace
cucumber
withnode_modules/cucumber/bin/cucumber.js
in the commands below.
- Once Node.js 4.0+ is installed you can install modules via
The following profiles each run different subsets of cucumber tests.
cucumber # default profile, runs all tests that should pass (exclude @todo and @bug)
cucumber -p verify # same as above, but with minimal output format
cucumber -p bugs # run all tests covering known bugs (@bug)
cucumber -p todo # run all tests covering unimplemented features (@todo)
cucumber -p all # run all tests, including @todo and @bug
You can also run specific tests or groups of tests:
- To run tests with a specific tag, use:
cucumber --tags @restrictions
- To exclude tests withs a specific tag, use:
cucumber --tags ~@restrictions
- To run a specific scenario:
cucumber features/bicycle/restrictions.feature:6
- To run a named scenario:
cucumber --name "A single way with two nodes"
- To run all scenario in a specific folder:
cucumber features/foot
Note that the default profile excludes tests tagged with @todo and @bug, so if you want to run a test with one of those tags, you need to use the 'all' profile:
cucumber -p all -t @weird_bug_7
Finally, run your tests by calling test
:
npm test
Tests are evaluated by parsing the output returned by osrm-routed
. Results are shown in the terminal. A passed route will be shown using a green row. A failed row will be shown with two rows: The expected result in a yellow row, followed by the actual result in a grey row.
(Note that there's currently an open cucumber-js issue about implementing formatted diffs)
During the tests, several log files are used:
- Output from
osrm-extract
andosrm-contract
are redirected to one file located attest/logs/path/to/test.feature/hash/name_of_test.log
- Output from
osrm-routed
is redirected totest/osrm-routed.log
The general structure of executing one cucumber test scenario is this:
- Create an OSM fixture file
- Run the OSM fixture file through
osrm-extract
andosrm-contract
- Start
osrm-routed
with the generated dataset - Make a request against
osrm-routed
- Compare request results with expectations
Let's dive into details:
Tests are located in a sub-directory called features/
and organized into feature files, each containing many scenarios.
Actual test runs are performed inside the folder test/
. The folder contain a .stxxl config file using a small 10MB disk.
For each test scenario, an .osm
data file is constructed based on the description in the test scenario.
It's then preprocessed with osrm-extract
and osrm-contract
. The relevant profile script is copied to test/profile.ini
before each preprocessing, so the binaries pick it up.
The resulting .osrm
files are cached, so that preprocessing is only repeated if either the .osm
data, the profile or the binaries have changed. This helps to speed up re-running of tests.
test/server.ini
is now rewritten to ensure that the right configuration is used. osrm-routed
is then launched, and routes are requested via http calls to localhost:5000
, and the response is compared to the expected result. All request are timed out after 10 seconds. Both timeouts and crashes are caught and reported.
To ensure a clean run of every test, all processes named osrm-extract
, osrm-contract
and osrm-routed
are killed before and after each test, using a signal 9, no matter who launched them.
For Cucumber documentation, see https://github.com/cucumber/cucumber/wiki/A-Table-Of-Content.
The OSRM tests currently come in three main flavors:
With node maps you geometrically place nodes on a mini map. Each node is identified by a letter. You then string together nodes to create ways, and optionally add relations. Finally you specify a list of routes to be tested, described by start and end points, and the expected route:
Scenario: Zig Zag
Given the node map
"""
a c
\ /
b
"""
And the ways
| nodes |
| ab |
| bc |
When I route I should get
| from | to | route |
| a | c | ab,bc,bc |
| c | a | bc,ab,ab |
The final segment of the route appears twice in this response. This is actually desired behaviour. The first part is for a turn onto the segment, the second one is for the arrival operation. The full list of instructions would be depart
, new name
, arrive
. Both the new name
and arrive
indicate the same segment, hence the replication of the segment in route
.
Under 'Given the node map', the node map is constructed using nodes. Each node must be specified by a letter a-z
. Only one node can exists at each location, and a node can only be present once on the map. Other characters like \ / | - can be used to visually indicate connections,but note that these characters are in fact ignored by the test. To construct ways you need to use an "And the ways" table.
When we generate the .osm
file for the test case from the node map, node IDs are assigned, starting at 1, in left-to-right, top-to-bottom order. In the example above, a=1, c=2, b=3.
You can add OSM tags to nodes by adding using a And the nodes
table:
And the nodes
| node | barrier |
| a | bollard |
| b | gate |
The node
column identifies the node, while additional columns are added as tags to the corresponding node.
In addition to the nodes specified with a-z
, you can add locations specified with numbers 0-9
. Locations can be used as start/end/via points when requesting routes, but are not included in the OSM file, meaning you add/move them without affecting the OSM input data.
For example, this map results in the same OSM input as the example above, but you can test a route from 1 to 2.
Given the node map
| a | | c |
| 1 | b | 2 |
Under And the ways
, you create ways by stringing together nodes in the node map. For example, ab
is a way starting at a and ending at b, while abc
is a way starting at a, passing b, and ending at c. A ways can have two or more nodes. All nodes of the way must exist in the node map.
You can add OSM tags to ways by adding columns to the And the ways
table:
And the ways
| nodes | oneway |
| ab | yes |
| bc | no |
Note that Given ...
and And ...
is interchangeable in cucumber, and you should just use what creates a natural language flow.
An OSM file is constructed based on the given nodes, ways and relations, and then used for testing. Using When I route I should get
you specify routes to be tested, each with a start and an end node, and the ways you expect the route to follow:
When I route I should get
| from | to | route |
| a | c | ab,bc,bc |
| c | a | bc,ab,ab |
An alternative syntax uses a single waypoints
column to specify the nodes you want to follow:
When I route I should get
| waypoints | route |
| a,c | ab,bc,bc | # from a, to b
| a,b,c | ab,bc,bc | # from a, via b, to c
If you specify more than two waypoints, via points are used.
Note that you cannot specify the nodes you expect the computed route to follow, only the ways. This is because OSRM does not return nodes in query results, only ways.
In addition to the route itself, you can test for total length or time (or both) by adding the corresponding columns to the test table:
When I route I should get
| from | to | route | time | distance |
| a | b | ab,ab | 24s | 100m |
| c | d | cd,cd | 72s | 200m |
Time is specified in seconds, while length is specified in meters. (Note that OSRM currently quantized length to the nearest 10m, and time to the nearest second)
OSRM uses spherical coordinates. Distances in node maps deviate from the naive flat geometry when you move away from the equator.
You can test the turn instructions by adding a "turns" columns to the test table:
When I route I should get
| from | to | route | turns |
| a | c | ab,bc,bc | depart,left,arrive |
| c | a | bc,ab,ab | depart,right,arrive |
The expected turns are specified using words that correspond to the v5 turns and modiffiers that are returned by OSRM:
depart
arrive
new name [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
continue [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
turn [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
merge [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
ramp [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
fork [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
end of road [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
roundabout-exit-[#|undefined]
rotary-exit-[#|undefined]
roundabout turn [uturn|sharp right|right|slight right|straight|slight left|left|sharp left]
notification
Fuzzy ranges for expected numerical values can be specified using either a percentage range, or an absolute +- range:
When I route I should get
| from | to | route | distance |
| a | d | abcde,abcde | 300m +-1 |
| a | e | abcde,abcde | 700m ~1% |
You can add a comment column by using the column name #
. The column will be ignored, but can be useful if you need to explain individual rows:
When I route I should get
| from | to | route | time | # |
| a | d | abcde,abcde | 30s | we're going downhill |
| a | e | abcde,abcde | 60s | we're going uphill |
Lat/lon tables are quite similar to node maps, except the nodes are placed using a table with absolute latitude and longitude values:
Scenario: Longitudinal distances at latitude 45
Given the node locations
| node | lat | lon |
| c | 45 | 80 |
| d | 45 | 0 |
And the ways
| nodes |
| cd |
When I route I should get
| from | to | route | distance |
| c | d | cd,cd | 6028844 ~0.5% |
This is especially useful for testing things related to the Mercator projection and huge distances. In this case, node IDs are assigned in the .osm
file in top-to-bottom order. In the example above, c=1, d=2.
Response codes of OSRM queries can be checked using a code
column as
When I route I should get
| from | to | code |
| a | b | NoRoute |
If annotations are included in the request,
Given the query options
| annotations | true |
then they can also be checked for in the response output with the following syntax: a:{annotation_type}
When I route I should get
| from | to | route | a:nodes | a:duration |
| a | c | ab,bc,bc | 1:2,2:3 | 10,2.5 |
The fields that can be checked for are the fields that can be requested with the annotations parameter: duration, distance, datasources, nodes, weight
.
A :
delineates annotation values within an annotation object within a step, a ,
delineates grouped data from the same leg object.
This flavor of tests is used for checking individual isolated ways, and is often useful for testing access and barrier tags. You can test each direction (forward/backward) separately:
Scenario: Simplest possible oneway
Given the speedprofile "car"
Then routability should be
| highway | oneway | forw | backw |
| primary | | x | x |
| primary | yes | x | |
| primary | -1 | | x |
Column headers are used to specify tag keys. In the example above, the highway
and oneway
tags are used.
You can also set tags on a central node of the way, by using columns with titles in the form of node/...
Scenario: Car - Barriers
Then routability should be
| highway | node/barrier | bothw |
| primary | | x |
| primary | bollard | |
Columns forw
, backw
and bothw
have special meaning:
- If a
forw_
column is present, the way is tested for forward routability - If a
backw_
column is present, the way is tested for backwards routability - If a
_bothw_
column is present, the way is tested for both forwards and backwards routability
A single OSM file is constructed, containing an isolated way for each row. The ways are labelled w1
, w2
, w3
, etc and placed in a line, but with spacing in between:
--w1-- --w2-- --w3--
For all types of tests, nodes are by default spaced on a grid with origin at 1,1
and a grid size of 100m
. The grid size can be changed using:
Given a grid size of 500 meters
The origin can be changed by:
Given the origin 55.5,12.2
Note that grid spacing uses fixed lat/lon increments (ignoring the latitude), and will therefore be distorted when you move away from the equator. For this reason, nodes far ways from the equator are often better placed using absolute node locations.
By default, tests use the "bicycle" profile in profiles/bicycle.lua
, but this can be changed for each scenario by adding:
Given the speedprofile "car"
Note that you can use a Background:
section if you need to specify things for all test scenarios in a .feature
file:
Background:
Given the speedprofile "car"
Given a grid size of 500 meters
I think this page is obsolete. I add few commet since I tried to compile OSMR recently using Codeblocks 20.03 on Windows 10. What I found was that
Several extra libraries ave to be installed like BZip2, lua. I was using MSYS2 so I did, pacman -S mingw-w64-x86_64-bzip2 pacman -S mingw-w64-x86_64-lua pacman -S mingw-w64-x86_64-zlib
instal Intell tbb https://github.com/oneapi-src/oneTBB/releases Extract it to a folder (e.g., C:\tbb).
Ensure to add TBB Path to CMake Command and that the bin folder of MinGW or MSYS2 (e.g., C:\msys64\mingw64\bin) is added to your system PATH environment variable.
Then in osrm-backend\build run
cmake -G "CodeBlocks - MinGW Makefiles" .. -DCMAKE_BUILD_TYPE=Release -DTBB_INCLUDE_DIR="C:/tbb/include" -DTBB_LIBRARY="C:/tbb/lib/intel64/gcc4.8/libtbb.so"
cmake -G "CodeBlocks - MinGW Makefiles" .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-fno-lto -mconsole" -DCMAKE_EXE_LINKER_FLAGS="-Wl,-e,mainCRTStartup"
It may be enough to compile.
However for some compilation problems I add to do
- remove "-Werror # Treat all warnings like error" in a CMakeLists.txt file
- add in shared_memory.hcp in line 208: (void)lock_file; // This explicitly marks lock_file as used to avoid warning of unused variable
- to avoid an Link Time Optimization (LTO) error run cmake -G "CodeBlocks - MinGW Makefiles" .. -DCMAKE_BUILD_TYPE=Release -DIPO=OFF
- put OFF in option(ENABLE_LTO "Use Link Time Optimisation" OFF) and I add set(CMAKE_INTERPROCEDURAL_OPTIMIZATION OFF) in Cmake
I finally gave up because of Windows console incompatibility (Winmain not found) without knowing the reason even after having, In codebleocks Project properties, Built target, type put Console application