Introduction
Cucumber is a specification for running tests in a BDD (behavior-driven development) style workflow.
It assumes involvement of non-technical members on a project and as such provides a human-readable syntax for the definition of features, via the language Gherkin. A typical feature could look something like this:
Feature: Eating too much cucumbers may not be good for you
Scenario: Eating a few isn't a problem
Given Alice is hungry
When she eats 3 cucumbers
Then she is full
These features are agnostic to the implementation, the only requirement is that they follow the expected format of phrases followed by the keywords (Given
, When
, Then
). Gherkin offers support for languages other than English, as well.
Cucumber implementations then simply hook into these keywords and execute the logic corresponding to the keywords. cucumber
crate is one of such implementations and is the subject of this book.
extern crate cucumber; extern crate tokio; use std::time::Duration; use cucumber::{given, then, when, World as _}; use tokio::time::sleep; #[derive(cucumber::World, Debug, Default)] struct World { user: Option<String>, capacity: usize, } #[given(expr = "{word} is hungry")] // Cucumber Expression async fn someone_is_hungry(w: &mut World, user: String) { sleep(Duration::from_secs(2)).await; w.user = Some(user); } #[when(regex = r"^(?:he|she|they) eats? (\d+) cucumbers?$")] async fn eat_cucumbers(w: &mut World, count: usize) { sleep(Duration::from_secs(2)).await; w.capacity += count; assert!(w.capacity < 4, "{} exploded!", w.user.as_ref().unwrap()); } #[then("she is full")] async fn is_full(w: &mut World) { sleep(Duration::from_secs(2)).await; assert_eq!(w.capacity, 3, "{} isn't full!", w.user.as_ref().unwrap()); } #[tokio::main] async fn main() { World::run("tests/features/readme").await; }
Since the goal is the testing of externally identifiable behavior of some feature, it would be a misnomer to use Cucumber to test specific private aspects or isolated modules. Cucumber tests are more likely to take the form of integration, functional or E2E testing.
Quickstart
Adding Cucumber to a project requires some groundwork. Cucumber tests are run along with other tests via cargo test
, but rely on .feature
files corresponding to the given test, as well as a set of step matchers (described in code) corresponding to the steps in those .feature
files.
To start, let's create a directory called tests/
in the root of the project and add a file to represent the test target (in this walkthrough it's example.rs
).
Add this to Cargo.toml
:
[dev-dependencies]
cucumber = "0.20"
futures = "0.3"
[[test]]
name = "example" # this should be the same as the filename of your test target
harness = false # allows Cucumber to print output instead of libtest
At this point, while it won't do anything, it should successfully run cargo test --test example
without errors, as long as the example.rs
file has at least a main()
function defined.
Now, let's create a directory to store .feature
files somewhere in the project (in this walkthrough it's tests/features/book/quickstart/
directory), and put a .feature
file there (such as simple.feature
). It should contain a Gherkin spec for the scenario we want to test. Here's a very simple example:
Feature: Animal feature
Scenario: If we feed a hungry cat it will no longer be hungry
Given a hungry cat
When I feed the cat
Then the cat is not hungry
To relate the text of the .feature
file with the actual tests we would need a World
object, holding a state that is newly created for each scenario and is changing as Cucumber goes through each step of that scenario.
To enable testing of our simple.feature
, let's add this code to example.rs
:
extern crate cucumber; extern crate futures; use cucumber::{given, World}; // These `Cat` definitions would normally be inside your project's code, // not test code, but we create them here for the show case. #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } // `World` is your shared, likely mutable state. // Cucumber constructs it via `Default::default()` for each scenario. #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } // Steps are defined with `given`, `when` and `then` attributes. #[given("a hungry cat")] fn hungry_cat(world: &mut AnimalWorld) { world.cat.hungry = true; } // This runs before everything else, so you can set up things here. fn main() { // You may choose any executor you like (`tokio`, `async-std`, etc.). // You may even have an `async` main, it doesn't matter. The point is that // Cucumber is composable. :) futures::executor::block_on(AnimalWorld::run( "tests/features/book/quickstart/simple.feature", )); }
TIP: Using
Default::default()
for constructing aWorld
object may be not enough. In such case a custom constructor may be specified via#[world(init = my_constructor)]
attribute.extern crate cucumber; use cucumber::World; #[derive(Debug)] struct Cat { pub hungry: bool, } #[derive(Debug, World)] // Accepts both sync/async and fallible/infallible functions. #[world(init = Self::new)] pub struct AnimalWorld { cat: Cat, } impl AnimalWorld { fn new() -> Self { Self { cat: Cat { hungry: true } } } } fn main() {}
If we run this, we should see an output like this:
A checkmark ✔
next to the Given a hungry cat
step means that it has been matched, executed and passed.
But then, for the next When I feed the cat
step there is a question mark ?
, meaning that we have nothing in our tests matching this sentence. The remaining steps in the scenario are not looked and run at all, since they depend on the skipped one.
There are 3 types of steps:
given
: for defining scenario starting conditions and often initializing the data in theWorld
;when
: for events or actions triggering the tested changes in theWorld
representing the scenario;then
: to validate that theWorld
has changed in the way expected by the scenario.
These various step matching functions are executed to transform the World
. As such, mutable reference to the world must always be passed in. The Step
itself is also made available.
NOTE: Unlike official Cucumber implementation the
cucumber
crate makes explicit separation betweengiven
,when
andthen
steps. This allows to prevent ambiguity problems when running tests (i.e. to avoid accidental uses of athen
step as agiven
one). To remain compliant with existing scenarios abusing this, it will be enough to place multiple attributes on the same step matching function.
We can add a when
step matcher:
extern crate cucumber; extern crate futures; use cucumber::{given, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given("a hungry cat")] fn hungry_cat(world: &mut AnimalWorld) { world.cat.hungry = true; } // Don't forget to additionally `use cucumber::when;`. #[when("I feed the cat")] fn feed_cat(world: &mut AnimalWorld) { world.cat.feed(); } fn main() { futures::executor::block_on(AnimalWorld::run( "tests/features/book/quickstart/simple.feature", )); }
Once we run the tests again, we see that two lines are green now and the next one is marked as not yet implemented:
Finally, how do we check our result? We expect that this will cause some change in the cat and that the cat will no longer be hungry since it has been fed. The then
step matcher follows to assert this, as our feature says:
extern crate cucumber; extern crate futures; use cucumber::{given, then, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given("a hungry cat")] fn hungry_cat(world: &mut AnimalWorld) { world.cat.hungry = true; } #[when("I feed the cat")] fn feed_cat(world: &mut AnimalWorld) { world.cat.feed(); } // Don't forget to additionally `use cucumber::then;`. #[then("the cat is not hungry")] fn cat_is_fed(world: &mut AnimalWorld) { assert!(!world.cat.hungry); } fn main() { futures::executor::block_on(AnimalWorld::run( "tests/features/book/quickstart/simple.feature", )); }
Once we run the tests, now we see all steps being accounted for and the whole scenario passing:
TIP: In addition to assertions, we may also return a
Result<()>
from a step matching function. ReturningErr
will cause the step to fail. This lets using the?
operator for more concise step implementations just like in unit tests.
To assure that assertion is indeed happening, let's reverse it temporarily:
extern crate cucumber; extern crate futures; use cucumber::{given, then, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given("a hungry cat")] fn hungry_cat(world: &mut AnimalWorld) { world.cat.hungry = true; } #[when("I feed the cat")] fn feed_cat(world: &mut AnimalWorld) { world.cat.feed(); } #[then("the cat is not hungry")] fn cat_is_fed(world: &mut AnimalWorld) { assert!(world.cat.hungry); } fn main() { futures::executor::block_on(AnimalWorld::run( "tests/features/book/quickstart/simple.feature", )); }
And see the test failing:
TIP: By default, unlike unit tests, failed steps don't terminate the execution instantly, and the whole test suite is executed regardless of them. Use
--fail-fast
CLI option to stop execution on first failure.
What if we also want to validate that even if the cat was never hungry to begin with, it won't end up hungry after it was fed? So, we may add another scenario, that looks quite similar (let's put it into a separate concurrent.feature
):
Feature: Animal feature
Scenario: If we feed a hungry cat it will no longer be hungry
Given a hungry cat
When I feed the cat
Then the cat is not hungry
Scenario: If we feed a satiated cat it will not become hungry
Given a satiated cat
When I feed the cat
Then the cat is not hungry
The only thing that is different is the Given
step. But we don't have to write a new matcher here! We can leverage regex
support:
extern crate cucumber; extern crate futures; use cucumber::{given, then, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(regex = r"^a (hungry|satiated) cat$")] fn hungry_cat(world: &mut AnimalWorld, state: String) { match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] fn feed_cat(world: &mut AnimalWorld) { world.cat.feed(); } #[then("the cat is not hungry")] fn cat_is_fed(world: &mut AnimalWorld) { assert!(!world.cat.hungry); } fn main() { futures::executor::block_on(AnimalWorld::run( "tests/features/book/quickstart/concurrent.feature", )); }
NOTE: We surround the regex with
^..$
to ensure an exact match. This is much more useful when adding more and more steps, so they won't accidentally interfere with each other.
Cucumber will reuse these step matchers:
NOTE: Captured values are bold to indicate which part of a step is actually captured.
Alternatively, we also may use Cucumber Expressions for the same purpose (less powerful, but much more readable):
extern crate cucumber; extern crate futures; use cucumber::{given, then, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(expr = "a {word} cat")] fn hungry_cat(world: &mut AnimalWorld, state: String) { match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, s => panic!("expected 'hungry' or 'satiated', found: {s}"), } } #[when("I feed the cat")] fn feed_cat(world: &mut AnimalWorld) { world.cat.feed(); } #[then("the cat is not hungry")] fn cat_is_fed(world: &mut AnimalWorld) { assert!(!world.cat.hungry); } fn main() { futures::executor::block_on(AnimalWorld::run( "tests/features/book/quickstart/concurrent.feature", )); }
A contrived example, but it demonstrates that steps can be reused as long as they are sufficiently precise in both their description and implementation. If, for example, the wording for our Then
step was The cat is no longer hungry
, it would imply something about the expected initial state, when that is not the purpose of a Then
step, but rather of the Given
step.
Asyncness
async
execution is supported naturally.
Let's switch our runtime to tokio
:
[dev-dependencies]
cucumber = "0.20"
tokio = { version = "1.10", features = ["macros", "rt-multi-thread", "time"] }
[[test]]
name = "example" # this should be the same as the filename of your test target
harness = false # allows Cucumber to print output instead of libtest
And, simply sleep
on each step to test the async
support (in the real world, of course, there will be web/database requests, etc.):
extern crate cucumber; extern crate tokio; use std::time::Duration; use cucumber::{given, then, when, World}; use tokio::time::sleep; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } // Don't forget to additionally `use tokio::time::{sleep, Duration};`. #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::run("tests/features/book/quickstart/concurrent.feature").await; }
Hm, it looks like the runtime waited only for the first Feature
, while the second was printed instantly. What's going on? 🤔
By default, Cucumber executes scenarios concurrently! That means that runtime actually did wait for all the steps, but overlapped! This allows us to execute tests much faster!
If for some reason we don't want to run scenarios concurrently, we may use @serial
tag on them:
Feature: Animal feature
@serial
Scenario: If we feed a hungry cat it will no longer be hungry
Given a hungry cat
When I feed the cat
Then the cat is not hungry
@serial
Scenario: If we feed a satiated cat it will not become hungry
Given a satiated cat
When I feed the cat
Then the cat is not hungry
NOTE: Any scenario marked with
@serial
tag will be executed in isolation, ensuring that there are no other scenarios running concurrently at the moment.
TIP: To run the whole test suite serially, consider using
--concurrency=1
CLI option, rather than marking evey single feature with a@serial
tag.
Writing tests
This chapter contains overview and examples of some Cucumber and Gherkin features allowing to write tests in a more idiomatic and maintainable way.
Also, it's worth to become familiar with Gherkin language.
- Capturing and variation
- Asserting
- Data tables
- Doc strings
Rule
keywordBackground
keywordScenario Outline
keyword- Scenario hooks
- Spoken languages
- Tags
- Retrying failed scenarios
- Modules organization
Capturing and variation
Using regular expressions or Cucumber Expressions for our step matching functions allows us:
- to capture values from a step and use them inside a test function;
- to match variations of a step with a single test function.
Regular expressions
Using a regular expression for a step matching function is possible with regex =
attribute modifier:
extern crate cucumber; extern crate tokio; use cucumber::{given, then, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(regex = r"^a (hungry|satiated) cat$")] fn hungry_cat(world: &mut AnimalWorld, state: String) { match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when(regex = r"^I feed the cat \d+ times?$")] fn feed_cat(world: &mut AnimalWorld) { world.cat.feed(); } #[then("the cat is not hungry")] fn cat_is_fed(world: &mut AnimalWorld) { assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::run("tests/features/book/writing/capturing.feature").await; }
NOTE: We surround the regex with
^..$
to ensure an exact match. This is much more useful when adding more and more steps, so they won't accidentally interfere with each other.
NOTE: Captured values are bold to indicate which part of a step is actually captured.
FromStr
arguments
For matching a captured value we are not restricted to use only String
. In fact, any type implementing a FromStr
trait can be used as a step function argument (including primitive types).
extern crate cucumber; extern crate tokio; use std::str::FromStr; use cucumber::{given, then, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: State, } impl Cat { fn feed(&mut self) { self.hungry = State::Satiated; } } #[derive(Debug, Default)] enum State { Hungry, #[default] Satiated, } impl FromStr for State { type Err = String; fn from_str(s: &str) -> Result<Self, Self::Err> { Ok(match s { "hungry" => Self::Hungry, "satiated" => Self::Satiated, invalid => return Err(format!("Invalid `State`: {invalid}")), }) } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(regex = r"^a (hungry|satiated) cat$")] fn hungry_cat(world: &mut AnimalWorld, state: State) { world.cat.hungry = state; } #[when(regex = r"^I feed the cat (\d+) times?$")] fn feed_cat(world: &mut AnimalWorld, times: u8) { for _ in 0..times { world.cat.feed(); } } #[then("the cat is not hungry")] fn cat_is_fed(world: &mut AnimalWorld) { assert!(matches!(world.cat.hungry, State::Satiated)); } #[tokio::main] async fn main() { AnimalWorld::run("tests/features/book/writing/capturing.feature").await; }
Cucumber Expressions
Alternatively, a Cucumber Expression may be used to capture values. This is possible with expr =
attribute modifier and parameters usage:
extern crate cucumber; extern crate tokio; use std::str::FromStr; use cucumber::{given, then, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: State, } impl Cat { fn feed(&mut self) { self.hungry = State::Satiated; } } #[derive(Debug, Default)] enum State { Hungry, #[default] Satiated, } impl FromStr for State { type Err = String; fn from_str(s: &str) -> Result<Self, Self::Err> { Ok(match s { "hungry" => Self::Hungry, "satiated" => Self::Satiated, invalid => return Err(format!("Invalid `State`: {invalid}")), }) } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(expr = "a {word} cat")] fn hungry_cat(world: &mut AnimalWorld, state: State) { world.cat.hungry = state; } #[when(expr = "I feed the cat {int} time(s)")] fn feed_cat(world: &mut AnimalWorld, times: u8) { for _ in 0..times { world.cat.feed(); } } #[then("the cat is not hungry")] fn cat_is_fed(world: &mut AnimalWorld) { assert!(matches!(world.cat.hungry, State::Satiated)); } #[tokio::main] async fn main() { AnimalWorld::run("tests/features/book/writing/capturing.feature").await; }
Cucumber Expressions are less powerful in terms of parsing and capturing values, but are much more readable than regular expressions, so it's worth to prefer using them for simple matching.
NOTE: Captured parameters are bold to indicate which part of a step is actually captured.
Custom parameters
Another useful advantage of using Cucumber Expressions is an ability to declare and reuse custom parameters in addition to default ones.
extern crate cucumber; extern crate tokio; use std::str::FromStr; use cucumber::{given, then, when, World}; use cucumber::Parameter; #[derive(Debug, Default)] struct Cat { pub hungry: State, } impl Cat { fn feed(&mut self) { self.hungry = State::Satiated; } } #[derive(Debug, Default, Parameter)] // NOTE: `name` is optional, by default the lowercased type name is implied. #[param(name = "hungriness", regex = "hungry|satiated")] enum State { Hungry, #[default] Satiated, } // NOTE: `Parameter` requires `FromStr` being implemented. impl FromStr for State { type Err = String; fn from_str(s: &str) -> Result<Self, Self::Err> { Ok(match s { "hungry" => Self::Hungry, "satiated" => Self::Satiated, invalid => return Err(format!("Invalid `State`: {invalid}")), }) } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(expr = "a {hungriness} cat")] fn hungry_cat(world: &mut AnimalWorld, state: State) { world.cat.hungry = state; } #[when(expr = "I feed the cat {int} time(s)")] fn feed_cat(world: &mut AnimalWorld, times: u8) { for _ in 0..times { world.cat.feed(); } } #[then("the cat is not hungry")] fn cat_is_fed(world: &mut AnimalWorld) { assert!(matches!(world.cat.hungry, State::Satiated)); } #[tokio::main] async fn main() { AnimalWorld::run("tests/features/book/writing/capturing.feature").await; }
NOTE: Using custom parameters allows declaring and reusing complicated and precise matches without a need to repeat them in different step matching functions.
TIP: In case regex of a custom parameter consists of several capturing groups, only the first non-empty match will be returned.
extern crate cucumber; extern crate tokio; use std::str::FromStr; use cucumber::{given, then, when, World}; use cucumber::Parameter; #[derive(Debug, Default)] struct Cat { pub hungry: Hungriness, } impl Cat { fn feed(&mut self) { self.hungry = Hungriness::Satiated; } } #[derive(Debug, Default, Eq, Parameter, PartialEq)] #[param(regex = "(hungry)|(satiated)|'([^']*)'")] // We want to capture without quotes ^^^^^^^ enum Hungriness { Hungry, #[default] Satiated, Other(String), } // NOTE: `Parameter` requires `FromStr` being implemented. impl FromStr for Hungriness { type Err = String; fn from_str(s: &str) -> Result<Self, Self::Err> { Ok(match s { "hungry" => Self::Hungry, "satiated" => Self::Satiated, other => Self::Other(other.to_owned()), }) } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(expr = "a {hungriness} cat")] fn hungry_cat(world: &mut AnimalWorld, hungry: Hungriness) { world.cat.hungry = hungry; } #[then(expr = "the cat is {string}")] fn cat_is(world: &mut AnimalWorld, other: String) { assert_eq!(world.cat.hungry, Hungriness::Other(other)); } #[when(expr = "I feed the cat {int} time(s)")] fn feed_cat(world: &mut AnimalWorld, times: u8) { for _ in 0..times { world.cat.feed(); } } #[then("the cat is not hungry")] fn cat_is_fed(world: &mut AnimalWorld) { assert_eq!(world.cat.hungry, Hungriness::Satiated); } #[tokio::main] async fn main() { AnimalWorld::run("tests/features/book/writing/capturing_multiple_groups.feature").await; }
Asserting
There are two ways of doing assertions in a step matching function:
Panic
Throwing a panic in a step matching function makes the appropriate step failed:
extern crate cucumber; extern crate tokio; use cucumber::{given, then, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(regex = r"^a (hungry|satiated) cat$")] fn hungry_cat(world: &mut AnimalWorld, state: String) { match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] fn feed_cat(world: &mut AnimalWorld) { world.cat.feed(); } #[then("the cat is not hungry")] fn cat_is_fed(_: &mut AnimalWorld) { panic!("Cats are always hungry!") } #[tokio::main] async fn main() { AnimalWorld::cucumber() .run_and_exit("tests/features/book/writing/asserting.feature") .await; }
NOTE: Failed step prints its location in a
.feature
file and the captured assertion message.
TIP: To additionally print the state of the
World
at the moment of failure, increase output verbosity via-vv
CLI option.
TIP: By default, unlike unit tests, failed steps don't terminate the execution instantly, and the whole test suite is executed regardless of them. Use
--fail-fast
CLI option to stop execution on first failure.
Result
and ?
Similarly to using the ?
operator in Rust tests, we may also return a Result<()>
from a step matching function, so returning an Err
will cause the step to fail (anything implementing Display
is sufficient).
extern crate cucumber; extern crate tokio; use cucumber::{given, then, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(regex = r"^a (hungry|satiated) cat$")] fn hungry_cat(world: &mut AnimalWorld, state: String) { match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] fn feed_cat(_: &mut AnimalWorld) {} #[then("the cat is not hungry")] fn cat_is_fed(world: &mut AnimalWorld) -> Result<(), &'static str> { (!world.cat.hungry).then_some(()).ok_or("Cat is still hungry!") } #[tokio::main] async fn main() { AnimalWorld::cucumber() .run_and_exit("tests/features/book/writing/asserting.feature") .await; }
Data tables
Data tables represent a handy way for passing a list of values to a step definition (and so, to a step matching function). This is a vital ability for writing table driven tests.
Feature: Animal feature
Scenario: If we feed a hungry animal it will no longer be hungry
Given a hungry animal
| animal |
| cat |
| dog |
| 🦀 |
When I feed the animal multiple times
| animal | times |
| cat | 2 |
| dog | 3 |
| 🦀 | 4 |
Then the animal is not hungry
Data, declared in the table, may be accessed via Step
argument:
extern crate cucumber; extern crate tokio; use std::collections::HashMap; use cucumber::{gherkin::Step, given, then, when, World}; #[given(regex = r"^a (hungry|satiated) animal$")] async fn hungry_animal(world: &mut AnimalWorld, step: &Step, state: String) { let state = match state.as_str() { "hungry" => true, "satiated" => false, _ => unreachable!(), }; if let Some(table) = step.table.as_ref() { for row in table.rows.iter().skip(1) { // NOTE: skip header let animal = &row[0]; world .animals .entry(animal.clone()) .or_insert(Animal::default()) .hungry = state; } } } #[when("I feed the animal multiple times")] async fn feed_animal(world: &mut AnimalWorld, step: &Step) { if let Some(table) = step.table.as_ref() { for row in table.rows.iter().skip(1) { // NOTE: skip header let animal = &row[0]; let times = row[1].parse::<usize>().unwrap(); for _ in 0..times { world.animals.get_mut(animal).map(Animal::feed); } } } } #[then("the animal is not hungry")] async fn animal_is_fed(world: &mut AnimalWorld) { for animal in world.animals.values() { assert!(!animal.hungry); } } #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { animals: HashMap<String, Animal>, } #[tokio::main] async fn main() { AnimalWorld::run("tests/features/book/writing/data_tables.feature").await; }
NOTE: The whole table data is processed during a single step run.
Escaping
- To use a newline character in a table cell, write it as
\n
. - To use a
|
as a part in a table cell, escape it as\|
. - And finally, to use a
\
, escape it as\\
.
Doc strings
Doc strings provide an ability to pass a large piece of text to a step definition (and so, to a step matching function).
The text should be offset by delimiters consisting of three double-quote marks """
on lines of their own:
Feature: Animal feature
Scenario: If we feed a hungry cat it will no longer be hungry
Given a hungry cat
"""
A hungry cat called Felix is rescued from a Whiskas tin in a calamitous
mash-up of cat food brands.
"""
When I feed the cat
Then the cat is not hungry
NOTE: Indentation of the opening
"""
is unimportant, although the common practice is to indent them. The indentation inside the triple quotes, however, is significant. Each line of the doc string will be dedented according to the opening"""
. Indentation beyond the column of the opening"""
will therefore be preserved.
Doc strings also support using three backticks ```
as the delimiter, which might be familiar for those used to writing with Markdown:
Feature: Animal feature
Scenario: If we feed a hungry Leo it will no longer be hungry
Given a hungry cat
```
A hungry cat called Leo is rescued from a Whiskas tin in a calamitous
mash-up of cat food brands.
```
When I feed the cat
Then the cat is not hungry
It’s also possible to annotate the doc string with the type of content it contains, as follows:
Feature: Animal feature
Scenario: If we feed a hungry Simba it will no longer be hungry
Given a hungry cat
"""markdown
About Simba
===========
A hungry cat called Simba is rescued from a Whiskas tin in a calamitous
mash-up of cat food brands.
"""
When I feed the cat
Then the cat is not hungry
NOTE: Whilst
cucumber
andgherkin
crates support content types and backticks as the delimiter, many tools like text editors don’t (yet).
In a step matching function, there’s no need to find this text and match it with a pattern. Instead, it may be accessed via Step
argument:
extern crate cucumber; extern crate tokio; use cucumber::{gherkin::Step, given, then, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, step: &Step, state: String) { // Feed only Leo and Felix. if !step .docstring .as_ref() .map_or(false, |text| text.contains("Felix") || text.contains("Leo")) { panic!("Only Felix and Leo can be fed"); } match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::cucumber() .run_and_exit("tests/features/book/writing/doc_strings.feature") .await; }
Rule
keyword
The purpose of the Rule
keyword is to represent a business rule that should be implemented. It provides additional information for a feature. A Rule
is used to group together several scenarios belonging to the business rule. A Rule
should contain one or more scenarios illustrating the particular rule.
No additional work is required on the implementation side to support Rule
s.
Feature: Animal feature
Rule: Hungry cat becomes satiated
Scenario: If we feed a hungry cat it will no longer be hungry
Given a hungry cat
When I feed the cat
Then the cat is not hungry
Rule: Satiated cat remains the same
Scenario: If we feed a satiated cat it will not become hungry
Given a satiated cat
When I feed the cat
Then the cat is not hungry
extern crate cucumber; extern crate tokio; use std::time::Duration; use cucumber::{given, then, when, World}; use tokio::time::sleep; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::run("tests/features/book/writing/rule.feature").await; }
Background
keyword
Occasionally, we may find ourselves repeating the same Given
steps in all the scenarios of a feature.
Since it's repeated in each scenario, this is an indication that those steps are not quite essential to describe the scenarios, but rather are incidental details. So, we can move such Given
steps to background, by grouping them under a Background
section.
Background
allows you to add some context to the scenarios following it. It can contain one or more steps, which are run before each scenario (but after any Before
hooks).
Feature: Animal feature
Background:
Given a hungry cat
Rule: Hungry cat becomes satiated
Scenario: If we feed a hungry cat it will no longer be hungry
When I feed the cat
Then the cat is not hungry
Rule: Satiated cat remains the same
Background:
When I feed the cat
Scenario: If we feed a satiated cat it will not become hungry
When I feed the cat
Then the cat is not hungry
extern crate cucumber; extern crate tokio; use std::time::Duration; use cucumber::{given, then, when, World}; use tokio::time::sleep; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::run("tests/features/book/writing/background.feature").await; }
NOTE:
Background
steps indicated by>
mark in the output.
NOTE: In case
Background
is declared outside anyRule
, it will be run on any scenario. Otherwise, ifBackground
is declared inside aRule
, it will be run only for scenarios belonging to it, and only after top-levelBackground
steps (if any).
Best practices
- Don’t use
Background
to set up complicated states, unless that state is actually something the client needs to know. - Keep your
Background
section short. - Make your
Background
section vivid, use colorful names, and try to tell a story. - Keep your
Scenario
s short, and don’t have too many.
Clearly, example provided above doesn't need Background
and was made for demonstration purposes only.
Scenario Outline
keyword
The Scenario Outline
keyword can be used to run the same scenario multiple times, with different combinations of values.
Feature: Animal feature
Scenario Outline: If we feed a hungry animal it will no longer be hungry
Given a hungry <animal>
When I feed the <animal> <n> times
Then the <animal> is not hungry
Examples:
| animal | n |
| cat | 2 |
| dog | 3 |
| 🦀 | 4 |
At parsing stage <template>
s are replaced by value from cells, so we may get that value in step matching functions (if we need though).
NOTE:
<template>
s are replaced even inside doc strings and data tables.
extern crate cucumber; extern crate tokio; use std::{collections::HashMap, time::Duration}; use cucumber::{given, then, when, World}; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { animals: HashMap<String, Animal>, } #[given(regex = r"^a (hungry|satiated) (\S+)$")] async fn hungry_animal(world: &mut AnimalWorld, state: String, which: String) { sleep(Duration::from_secs(2)).await; world.animals.entry(which).or_insert(Animal::default()).hungry = match state.as_str() { "hungry" => true, "satiated" => false, _ => unreachable!(), }; } #[when(expr = "I feed the {word} {int} time(s)")] async fn feed_animal(world: &mut AnimalWorld, which: String, times: usize) { sleep(Duration::from_secs(2)).await; for _ in 0..times { world.animals.get_mut(&which).map(Animal::feed); } } #[then(expr = "the {word} is not hungry")] async fn animal_is_fed(world: &mut AnimalWorld, which: String) { sleep(Duration::from_secs(2)).await; assert!(!world.animals.get(&which).map_or(true, |a| a.hungry)); } #[tokio::main] async fn main() { AnimalWorld::run("tests/features/book/writing/scenario_outline.feature") .await; }
NOTE:
Scenario Outline
runs the whole scenario for each table row separately, unlike data tables, which run the whole table inside a single step.
Scenario hooks
Scenario hooks represent a code running for each scenario and not visible in .feature
files.
Before
hook
Before
hook runs before the first step of each scenario, even before Background
ones.
extern crate cucumber; extern crate futures; extern crate tokio; use std::time::Duration; use cucumber::World as _; use futures::FutureExt as _; use tokio::time; #[derive(cucumber::World, Debug, Default)] struct World; fn main() { World::cucumber() .before(|_feature, _rule, _scenario, _world| { time::sleep(Duration::from_millis(300)).boxed_local() }) .run_and_exit("tests/features/book"); }
NOTE:
Before
hook is enabled globally for all the executed scenarios. No exception is possible.
WARNING: Think twice before using
Before
hook!
Whatever happens in aBefore
hook is invisible to people reading.feature
s. You should consider using aBackground
keyword as a more explicit alternative, especially if the setup should be readable by non-technical people. Only use aBefore
hook for low-level logic such as starting a browser or deleting data from a database.
After
hook
After
hook runs after the last step of each scenario, even when that step fails or is skipped.
extern crate cucumber; extern crate futures; extern crate tokio; use std::time::Duration; use cucumber::World as _; use futures::FutureExt as _; use tokio::time; #[derive(cucumber::World, Debug, Default)] struct World; fn main() { World::cucumber() .after(|_feature, _rule, _scenario, _ev, _world| { time::sleep(Duration::from_millis(300)).boxed_local() }) .run_and_exit("tests/features/book"); }
NOTE:
After
hook is enabled globally for all the executed scenarios. No exception is possible.
TIP:
After
hook receives anevent::ScenarioFinished
as one of its arguments, which indicates why the scenario has finished (passed, failed or skipped). This information, for example, may be used to decide whether some external resources (like files) should be cleaned up if the scenario passes, or leaved "as is" if it fails, so helping to "freeze" the failure conditions for better investigation.
Spoken languages
The language chosen for Gherkin should be the same language users and domain experts use when they talk about the domain. Translating between two languages should be avoided.
This is why Gherkin has been translated to over 70 languages.
A # language:
header on the first line of a .feature
file tells Cucumber which spoken language to use (for example, # language: fr
for French). If you omit this header, Cucumber will default to English (en
).
# language: no
Egenskap: Dyr egenskap
Scenario: Hvis vi mater en sulten katt, vil den ikke lenger være sulten
Gitt en sulten katt
Når jeg mater katten
Så katten er ikke sulten
extern crate cucumber; extern crate tokio; use cucumber::{given, then, when, World}; #[derive(Debug, Default)] struct Cat { pub hungry: bool, } impl Cat { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Cat, } #[given(regex = r"^en (sulten|mett) katt$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { match state.as_str() { "sulten" => world.cat.hungry = true, "mett" => world.cat.hungry = false, _ => unreachable!(), } } #[when("jeg mater katten")] async fn feed_cat(world: &mut AnimalWorld) { world.cat.feed(); } #[then("katten er ikke sulten")] async fn cat_is_fed(world: &mut AnimalWorld) { assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::run("tests/features/book/writing/languages.feature").await; }
TIP: In case most of your
.feature
files aren't written in English and you want to avoid endless# language:
comments, useCucumber::language()
method to override the default language globally.
Tags
Tags represent meta information of scenarios and features.
They can be used for different purposes, but in the majority of cases it's just:
- either running a subset of scenarios filtering by tag;
- or making scenario run in isolation via
@serial
tag; - or allowing scenarios to be skipped with
@allow.skipped
tag.
Filtering
A scenario may have as many tags as it requires (they should be separated with spaces):
Feature: Animal feature
@hungry
Scenario: If we feed a hungry cat it will no longer be hungry
Given a hungry cat
When I feed the cat
Then the cat is not hungry
@satiated @second
Scenario: If we feed a satiated cat it will not become hungry
Given a satiated cat
When I feed the cat
Then the cat is not hungry
To filter out running scenarios we may use:
- either
--tags
CLI option providing tag expressions (also consider escaping); - or
filter_run()
-like method.
Inheritance
Tags may be placed above the following Gherkin elements:
It's not possible to place tags above Background
or steps (Given
, When
, Then
, And
and But
).
Tags are inherited by child elements:
Feature
andRule
tags will be inherited byScenario
,Scenario Outline
, orExamples
.Scenario Outline
tags will be inherited byExamples
.
@feature
Feature: Animal feature
@scenario
Scenario Outline: If we feed a hungry animal it will no longer be hungry
Given a hungry <animal>
When I feed the <animal> <n> times
Then the <animal> is not hungry
@home
Examples:
| animal | n |
| cat | 2 |
| dog | 3 |
@dire
Examples:
| animal | n |
| lion | 1 |
| wolf | 1 |
NOTE: In
Scenario Outline
it's possible to use tags on differentExamples
.
Isolated execution
cucumber
crate provides out-of-the-box support for @serial
tag. Any scenario marked with @serial
tag will be executed in isolation, ensuring that there are no other scenarios running concurrently at the moment.
Feature: Animal feature
Scenario: If we feed a hungry cat it will no longer be hungry
Given a hungry cat
When I feed the cat
Then the cat is not hungry
@serial
Scenario: If we feed a satiated cat it will not become hungry
Given a satiated cat
When I feed the cat
Then the cat is not hungry
NOTE:
@serial
tag may also be used for filtering as a regular one.
TIP: To run the whole test suite serially, consider using
--concurrency=1
CLI option, rather than marking evey single feature with a@serial
tag.
Failing on skipped steps
As a test suit grows, it may become harder to notice how minimal changes to regular expressions can lead to mismatched steps.
Using Cucumber::fail_on_skipped()
method fails the whole test suite if some steps miss the implementation, so ensures that the whole test suite is covered.
Feature: Animal feature
Scenario: If we feed a hungry cat it will no longer be hungry
Given a hungry cat
When I feed the cat
Then the cat is not hungry
Scenario: If we feed a satiated cat it will not become hungry
Given a wild cat
When I feed the cat
Then the cat is not hungry
extern crate cucumber; extern crate tokio; use std::time::Duration; use cucumber::{given, then, when, World}; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::cucumber() .fail_on_skipped() .run_and_exit("tests/features/book/writing/tags_skip_failed.feature") .await; }
TIP: Using
@allow.skipped
tag allows scenarios being skipped even inCucumber::fail_on_skipped()
mode. Use the one to intentionally skip the implementation.
Feature: Animal feature
Scenario: If we feed a hungry cat it will no longer be hungry
Given a hungry cat
When I feed the cat
Then the cat is not hungry
@allow.skipped
Scenario: If we feed a satiated cat it will not become hungry
Given a wild cat
When I feed the cat
Then the cat is not hungry
NOTE:
@allow.skipped
tag may also be used for filtering as a regular one.
Retrying failed scenarios
Often, it's nearly impossible to create fully-deterministic test case, especially when you are relying on environments like external services, browsers, file system, networking etc. That's why there is an ability to retry failed scenarios.
WARNING: Although this feature is supported, we highly recommend to use it as the last resort only. First, consider implementing in-step retries with your own needs (like exponential backoff). Other ways of dealing with flaky tests include, but not limited to: reducing number of concurrently executed scenarios (maybe even using
@serial
tag), mocking external environment, controlling time in tests or even simulation testing. It's always better to move towards tests determinism, rather than trying to tame their flakiness.
Tags
Recommended way to specify retried scenarios is using tags (inheritance is supported too):
Feature: Heads and tails
# Attempts a single retry immediately.
@retry
Scenario: Tails
Given a coin
When I flip the coin
Then I see tails
# Attempts a single retry in 1 second.
@retry.after(1s)
Scenario: Heads
Given a coin
When I flip the coin
Then I see heads
# Attempts to retry 5 times with no delay between them.
@retry(5)
Scenario: Edge
Given a coin
When I flip the coin
Then I see edge
# Attempts to retry 10 times with 100 milliseconds delay between them.
@retry(10).after(100ms)
Scenario: Levitating
Given a coin
When I flip the coin
Then the coin never lands
extern crate cucumber; extern crate rand; extern crate tokio; use std::time::Duration; use cucumber::{given, then, when, World}; use rand::Rng as _; use tokio::time::sleep; #[derive(Debug, Default, World)] pub struct FlipWorld { flipped: &'static str, } #[given("a coin")] async fn coin(_: &mut FlipWorld) { sleep(Duration::from_secs(2)).await; } #[when("I flip the coin")] async fn flip(world: &mut FlipWorld) { sleep(Duration::from_secs(2)).await; world.flipped = match rand::thread_rng().gen_range(0.0..1.0) { p if p < 0.2 => "edge", p if p < 0.5 => "heads", _ => "tails", } } #[then(regex = r#"^I see (heads|tails|edge)$"#)] async fn see(world: &mut FlipWorld, what: String) { sleep(Duration::from_secs(2)).await; assert_eq!(what, world.flipped); } #[then("the coin never lands")] async fn never_lands(_: &mut FlipWorld) { sleep(Duration::from_secs(2)).await; unreachable!("coin always lands") } #[tokio::main] async fn main() { FlipWorld::cucumber() .fail_on_skipped() .run_and_exit("tests/features/book/writing/retries.feature") .await; }
NOTE: On failure, the whole scenario is re-executed with a new fresh
World
instance.
CLI
The following CLI options are related to the scenario retries:
--retry <int>
Number of times a scenario will be retried in case of a failure
--retry-after <duration>
Delay between each scenario retry attempt.
Duration is represented in a human-readable format like `12min5s`.
Supported suffixes:
- `nsec`, `ns` — nanoseconds.
- `usec`, `us` — microseconds.
- `msec`, `ms` — milliseconds.
- `seconds`, `second`, `sec`, `s` - seconds.
- `minutes`, `minute`, `min`, `m` - minutes.
--retry-tag-filter <tagexpr>
Tag expression to filter retried scenarios
--retry
CLI option is similar to@retry(<number-of-retries>)
tag, but is applied to all scenarios matching the--retry-tag-filter
(if not provided, all possible scenarios are matched).--retry-after
CLI option is similar to@retry.after(<delay-after-each-retry>)
tag in the same manner.
Precedence of tags and CLI options
- Just
@retry
tag takes the number of retries and the delay from--retry
and--retry-after
CLI options respectively, if they're specified, otherwise defaults to a single retry attempt with no delay. @retry(3)
tag always retries failed scenarios at most 3 times, even if--retry
CLI option provides a greater value. Delay is taken from--retry-after
CLI option, if it's specified, otherwise defaults to no delay.@retry.after(1s)
tag always delays 1 second before next retry attempt, even if--retry-after
CLI option provides another value. Number of retries is taken from--retry-after
CLI option, if it's specified, otherwise defaults a single retry attempt.@retry(3).after(1s)
always retries failed scenarios at most 3 times with 1 second delay before each attempt, ignoring--retry
and--retry-after
CLI options.
NOTE: When using with
--fail-fast
CLI option (or.fail_fast()
builder config), scenarios are considered as failed only in case they exhaust all retry attempts and then still do fail.
TIP: It could be handy to specify
@retry
tags only, without any explicit values, and use--retry=n --retry-after=d --retry-tag-filter=@retry
CLI options to overwrite retrying parameters without affecting any other scenarios.
Modules organization
When the project is started it's okay to have all the steps defined in a single .feature
file. However, as the project grows, it will be more efficient to split all the steps into meaningful groups in different .feature
files. This will make the project tests more logically organized and easier to maintain.
Grouping
Technically, it doesn't matter how .feature
files are named, or which scenarios are put in there. However, as the project grows, big .feature
files becomes messy and hard to maintain. Instead, we recommend creating a separate .rs
file for each domain concept (in a way that is meaningful to your project).
Following this pattern allows us also to avoid the feature-coupled step definitions anti-pattern.
Avoiding duplication
It's better to avoid writing similar step matching functions, as they can lead to clutter. While documenting steps helps, making use of regular and Cucumber expressions can do wonders.
Managing growth
As the test suit grows, it may become harder to notice how minimal changes to regular expressions can lead to mismatched steps.
TIP: We recommend using
Cucumber::fail_on_skipped()
method in combination with@allow.skipped
tag. The latter allows marking the scenarios which steps are explicitly allowed to be skipped.
CLI (command-line interface)
cucumber
crate provides several options that can be passed to the command-line.
Use --help
flag to print out all the available options:
cargo test --test <test-name> -- --help
Default output is:
Run the tests, pet a dog!
Usage: cucumber [OPTIONS]
Options:
-n, --name <regex>
Regex to filter scenarios by their name
[aliases: scenario-name]
-t, --tags <tagexpr>
Tag expression to filter scenarios by.
Note: Tags from Feature, Rule and Scenario are merged together on filtering, so be careful about conflicting tags on different levels.
-i, --input <glob>
Glob pattern to look for feature files with. By default, looks for `*.feature`s in the path configured tests runner
-c, --concurrency <int>
Number of scenarios to run concurrently. If not specified, uses the value configured in tests runner, or 64 by default
--fail-fast
Run tests until the first failure
[aliases: ff]
--retry <int>
Number of times a scenario will be retried in case of a failure
--retry-after <duration>
Delay between each scenario retry attempt.
Duration is represented in a human-readable format like `12min5s`.
Supported suffixes:
- `nsec`, `ns` — nanoseconds.
- `usec`, `us` — microseconds.
- `msec`, `ms` — milliseconds.
- `seconds`, `second`, `sec`, `s` - seconds.
- `minutes`, `minute`, `min`, `m` - minutes.
--retry-tag-filter <tagexpr>
Tag expression to filter retried scenarios
-v...
Verbosity of an output.
`-v` is default verbosity, `-vv` additionally outputs world on failed steps, `-vvv` additionally outputs step's doc string (if present).
--color <auto|always|never>
Coloring policy for a console output
[default: auto]
-h, --help
Print help information (use `-h` for a summary)
NOTE: CLI options override any configurations set in the code.
Customizing
By default, the whole CLI is composed of Parser::Cli
, Runner::Cli
and Writer::Cli
, provided by the used components. Once a custom Parser
, Runner
or Writer
is used, its CLI is automatically emerged into the final CLI.
CLI may be extended even more with arbitrary options, if required. In such case we should combine the final CLI by ourselves and apply it via Cucumber::with_cli()
method.
extern crate clap; extern crate cucumber; extern crate futures; extern crate humantime; extern crate tokio; use std::time::Duration; use cucumber::{cli, given, then, when, World}; use futures::FutureExt as _; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { assert!(!world.cat.hungry); } #[derive(cli::Args)] // re-export of `clap::Args` struct CustomOpts { /// Additional time to wait in before hook. #[arg( long, value_parser = humantime::parse_duration, )] pre_pause: Option<Duration>, } #[tokio::main] async fn main() { let opts = cli::Opts::<_, _, _, CustomOpts>::parsed(); let pre_pause = opts.custom.pre_pause.unwrap_or_default(); AnimalWorld::cucumber() .before(move |_, _, _, _| sleep(pre_pause).boxed_local()) .with_cli(opts) .run_and_exit("tests/features/book/cli.feature") .await; }
NOTE: For extending CLI options of exising
Parser
,Runner
orWriter
when wrapping it, consider usingcli::Compose
.
NOTE: If a custom
Parser
,Runner
orWriter
implementation doesn't expose any CLI options, thencli::Empty
should be used.
Aliasing
Cargo alias is a neat way to define shortcuts for regularly used customized tests running commands.
extern crate clap; extern crate cucumber; extern crate futures; extern crate humantime; extern crate tokio; use std::time::Duration; use cucumber::{cli, given, then, when, World}; use futures::FutureExt as _; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { assert!(!world.cat.hungry); } #[derive(clap::Args)] struct CustomOpts { #[command(subcommand)] command: Option<SubCommand>, } #[derive(clap::Subcommand)] enum SubCommand { Smoke(Smoke), } #[derive(clap::Args)] struct Smoke { /// Additional time to wait in before hook. #[arg( long, value_parser = humantime::parse_duration, )] pre_pause: Option<Duration>, } #[tokio::main] async fn main() { let opts = cli::Opts::<_, _, _, CustomOpts>::parsed(); let pre_pause = if let Some(SubCommand::Smoke(Smoke { pre_pause })) = opts.custom.command { pre_pause } else { None } .unwrap_or_default(); AnimalWorld::cucumber() .before(move |_, _, _, _| sleep(pre_pause).boxed_local()) .with_cli(opts) .run_and_exit("tests/features/book/cli.feature") .await; }
The alias should be specified in .cargo/config.toml
file of the project:
[alias]
smoke = "test -p cucumber --test cli -- smoke --pre-pause=5s -vv --fail-fast"
Now it can be used as:
cargo smoke
cargo smoke --tags=@hungry
NOTE: The default CLI options may be specified after a custom subcommand, because they are defined as global ones. This may be applied to custom CLI options too, if necessary.
Output
This chapter describes possible way and tweaks of outputting test suite results.
- Terminal
- JUnit XML report
- Cucumber JSON format
- Multiple outputs
tracing
integration- IntelliJ Rust (
libtest
) integration
Terminal output
By default, cucumber
crate outputs tests result to STDOUT. It provides some CLI options for configuring the output.
Verbosity
By default, cucumber
crate tries to keep the output quite minimal, but its verbosity may be increased with -v
CLI option.
Just specifying -v
makes no difference, as it refers to the default verbosity level (no additional info).
Output World
on failures (-vv
)
Increasing verbosity level with -vv
CLI option, makes the state of the World
being printed at the moment of failure.
extern crate cucumber; extern crate tokio; use std::time::Duration; use cucumber::{given, then, when, World}; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(_: &mut AnimalWorld) {} #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::cucumber() .run_and_exit("tests/features/book/output/terminal_verbose.feature") .await; }
This is intended to help debugging failed tests.
Output doc strings (-vvv
)
By default, outputting doc strings of steps is omitted. To include them into the output use -vvv
CLI option:
Feature: Animal feature
Scenario: If we feed a hungry cat it will no longer be hungry
Given a hungry cat
"""
A hungry cat called Felix is rescued from a Whiskas tin in a calamitous
mash-up of cat food brands.
"""
When I feed the cat
Then the cat is not hungry
extern crate cucumber; extern crate tokio; use std::time::Duration; use cucumber::{given, then, when, World}; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::cucumber() .run_and_exit("tests/features/book/output/terminal_verbose.feature") .await; }
Coloring
Coloring may be disabled by specifying --color
CLI option:
extern crate cucumber; extern crate tokio; use std::time::Duration; use cucumber::{given, then, when, World}; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::cucumber() .run_and_exit("tests/features/book/output/terminal.feature") .await; }
NOTE: By default,
cucumber
crate automatically disables coloring for non-interactive terminals, so there is no need to specify--color
CLI option explicitly on CI.
Debug printing and/or logging
Though cucumber
crate doesn't capture any manual debug printing produced in a step matching function (such as dbg!
or println!
macros), it may be quite misleading to produce and use it for debugging purposes. The reason is simply because cucumber
crate executes scenarios concurrently and normalizes their results before outputting, while any manual print is produced instantly at the moment of its step execution.
WARNING: Moreover, manual printing will very likely interfere with default interactive pretty-printing.
extern crate cucumber; extern crate tokio; use std::time::Duration; use cucumber::{given, then, when, World}; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; dbg!("here!"); world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::cucumber() .run_and_exit("tests/features/book/output/terminal.feature") .await; }
To achieve natural output for debugging, the following preparations are required:
- Setting
.max_concurrent_scenarios()
to1
for executing all the scenarios sequentially. - Creating
writer::Basic::raw
withColoring::Never
to avoid interactive pretty-printed output. - Wrapping it into
writer::AssertNormalized
to assurecucumber
about the output being normalized already (due to sequential execution).
extern crate cucumber; extern crate tokio; use std::{io, time::Duration}; use cucumber::{given, then, when, writer, World, WriterExt as _}; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; dbg!("here!"); world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::cucumber() .max_concurrent_scenarios(1) .with_writer( writer::Basic::raw(io::stdout(), writer::Coloring::Never, 0) .summarized() .assert_normalized(), ) .run_and_exit("tests/features/book/output/terminal.feature") .await; }
NOTE: The custom print is still output before its step, because is printed during the step execution.
Much better option for debugging would be using tracing
crate integration instead of dbg!
/println!
for doing logs.
extern crate cucumber; extern crate tokio; extern crate tracing; use std::{ sync::atomic::{AtomicUsize, Ordering}, time::Duration, }; use cucumber::{given, then, when, World as _}; use tokio::time; #[derive(cucumber::World, Debug, Default)] struct World; #[given(regex = r"(\d+) secs?")] #[when(regex = r"(\d+) secs?")] #[then(regex = r"(\d+) secs?")] async fn sleep(_: &mut World, secs: u64) { static ID: AtomicUsize = AtomicUsize::new(0); let id = ID.fetch_add(1, Ordering::Relaxed); tracing::info!("before {secs}s sleep: {id}"); time::sleep(Duration::from_secs(secs)).await; tracing::info!("after {secs}s sleep: {id}"); } #[tokio::main] async fn main() { World::cucumber() .init_tracing() .run("tests/features/wait") .await; }
Repeating failed and/or skipped steps
As a number of scenarios grows, it may become quite difficult to find failed/skipped ones in a large output. This issue may be mitigated by duplicating failed and/or skipped steps at the and of output via Cucumber::repeat_failed()
and Cucumber::repeat_skipped()
methods respectively.
extern crate cucumber; extern crate tokio; use std::{time::Duration}; use cucumber::{given, then, when, World}; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::cucumber() .repeat_failed() .run_and_exit("tests/features/book/output/terminal_repeat_failed.feature") .await; }
extern crate cucumber; extern crate tokio; use std::{time::Duration}; use cucumber::{given, then, when, World}; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } #[tokio::main] async fn main() { AnimalWorld::cucumber() .repeat_skipped() .run_and_exit("tests/features/book/output/terminal_repeat_skipped.feature") .await; }
JUnit XML report
cucumber
crate provides an ability to output tests result as a JUnit XML report.
This requires output-junit
feature to be enabled in Cargo.toml
:
cucumber = { version = "0.20", features = ["output-junit"] }
And configuring output to writer::JUnit
:
extern crate cucumber; extern crate tokio; use std::{fs, io}; use cucumber::{writer, World as _}; #[derive(cucumber::World, Debug, Default)] struct World; #[tokio::main] async fn main() -> io::Result<()> { let file = fs::File::create(format!("{}/junit.xml", env!("OUT_DIR")))?; World::cucumber() .with_writer(writer::JUnit::new(file, 0)) .run("tests/features/book") .await; Ok(()) }
Cucumber JSON format
cucumber
crate provides an ability to output tests result in a Cucumber JSON format.
This requires output-json
feature to be enabled in Cargo.toml
:
cucumber = { version = "0.20", features = ["output-json"] }
And configuring output to writer::Json
:
extern crate cucumber; extern crate tokio; use std::{fs, io}; use cucumber::{writer, World as _}; #[derive(cucumber::World, Debug, Default)] struct World; #[tokio::main] async fn main() -> io::Result<()> { let file = fs::File::create(format!("{}/report.json", env!("OUT_DIR")))?; World::cucumber() .with_writer(writer::Json::new(file)) .run("tests/features/book") .await; Ok(()) }
Multiple outputs
Reporting tests result to multiple outputs simultaneously may be achieved by using writer::Tee
.
extern crate cucumber; extern crate tokio; use std::{fs, io}; use cucumber::{writer, World as _, WriterExt as _}; #[derive(cucumber::World, Debug, Default)] struct World; #[tokio::main] async fn main() -> io::Result<()> { let file = fs::File::create(format!("{}/report.xml", env!("OUT_DIR")))?; World::cucumber() .with_writer( // NOTE: `Writer`s pipeline is constructed in a reversed order. writer::Basic::stdout() // And output to STDOUT. .summarized() // Simultaneously, add execution summary. .tee::<World, _>(writer::JUnit::for_tee(file, 0)) // Then, output to XML file. .normalized() // First, normalize events order. ) .run_and_exit("tests/features/book") .await; Ok(()) }
Using the same Writer
multiple times
While using writer::Tee
for different Writer
s is OK and straightforward most of the time, reusing the same Writer
multiple times isn't so obvious, because of the clap
complaining about identical CLI options (unfortunately, in a form of runtime panic only).
extern crate cucumber; extern crate tokio; use std::{fs, io}; use cucumber::{writer, World as _, WriterExt as _}; #[derive(cucumber::World, Debug, Default)] struct World; #[tokio::main] async fn main() -> io::Result<()> { let file = fs::File::create(format!("{}/report.txt", env!("OUT_DIR")))?; World::cucumber() .with_writer( writer::Basic::raw( io::stdout(), writer::Coloring::Auto, writer::Verbosity::Default, ) .tee::<World, _>(writer::Basic::raw( file, writer::Coloring::Never, 2, )) .summarized() .normalized(), ) .run_and_exit("tests/features/book") .await; Ok(()) }
thread 'main' panicked at 'Command cucumber: Argument names must be unique, but 'verbose' is in use by more than one argument or group'
To avoid this, you should manually construct the desired cli::Opts
and supply them via Cucumber::with_cli()
method. Example below uses two different writer::Basic
s, where one outputs to STDOUT and another one outputs to a file:
extern crate cucumber; extern crate tokio; use std::{fs, io}; use cucumber::{cli, writer, World as _, WriterExt as _}; #[derive(cucumber::World, Debug, Default)] struct World; #[tokio::main] async fn main() -> io::Result<()> { // Parse CLI arguments for a single `writer::Basic`. let cli = cli::Opts::<_, _, writer::basic::Cli>::parsed(); let cli = cli::Opts { re_filter: cli.re_filter, tags_filter: cli.tags_filter, parser: cli.parser, runner: cli.runner, // Replicate CLI arguments for every `writer::Basic`. writer: cli::Compose { left: cli.writer.clone(), right: cli.writer, }, custom: cli.custom, }; let file = fs::File::create(format!("{}/report.txt", env!("OUT_DIR")))?; World::cucumber() .with_writer( writer::Basic::raw( io::stdout(), writer::Coloring::Auto, writer::Verbosity::Default, ) .tee::<World, _>(writer::Basic::raw( file, writer::Coloring::Never, 2, )) .summarized() .normalized(), ) .with_cli(cli) // Supply the parsed `cli::Opts`. .run_and_exit("tests/features/book") .await; Ok(()) }
tracing
integration
Cucumber::init_tracing()
(enabled by tracing
feature in Cargo.toml
) initializes global tracing::Subscriber
that intercepts all the tracing
events and transforms them into event::Scenario::Log
s. Each Writer
can handle those event::Scenario::Log
s in its own way. writer::Basic
, for example, emits all the event::Scenario::Log
s only whenever scenario itself is outputted.
extern crate cucumber; extern crate tokio; extern crate tracing; use std::{ sync::atomic::{AtomicUsize, Ordering}, time::Duration, }; use cucumber::{given, then, when, World as _}; use tokio::time; #[derive(cucumber::World, Debug, Default)] struct World; #[given(regex = r"(\d+) secs?")] #[when(regex = r"(\d+) secs?")] #[then(regex = r"(\d+) secs?")] async fn sleep(_: &mut World, secs: u64) { static ID: AtomicUsize = AtomicUsize::new(0); let id = ID.fetch_add(1, Ordering::Relaxed); tracing::info!("before {secs}s sleep: {id}"); time::sleep(Duration::from_secs(secs)).await; tracing::info!("after {secs}s sleep: {id}"); } #[tokio::main] async fn main() { World::cucumber() .init_tracing() .run("tests/features/wait") .await; }
Loosing tracing::Span
tracing::Span
is used to wire emitted tracing
events (logs) to concrete scenarios: each scenario is executed in its own tracing::Span
. In case a tracing
event is emitted outside the tracing::Span
of a scenario, it will be propagated to every running scenario at the moment.
extern crate cucumber; extern crate tokio; extern crate tracing; use std::{ sync::atomic::{AtomicUsize, Ordering}, time::Duration, }; use cucumber::{given, then, when, World as _}; use tokio::time; #[derive(cucumber::World, Debug, Default)] struct World; #[given(regex = r"(\d+) secs?")] #[when(regex = r"(\d+) secs?")] #[then(regex = r"(\d+) secs?")] async fn sleep(_: &mut World, secs: u64) { static ID: AtomicUsize = AtomicUsize::new(0); let id = ID.fetch_add(1, Ordering::Relaxed); tracing::info!("before {secs}s sleep: {id}"); time::sleep(Duration::from_secs(secs)).await; tracing::info!("after {secs}s sleep: {id}"); } #[tokio::main] async fn main() { // Background task outside of any scenario. tokio::spawn(async { let mut id = 0; loop { time::sleep(Duration::from_secs(3)).await; tracing::info!("Background: {id}"); id += 1; } }); World::cucumber() .init_tracing() .run("tests/features/wait") .await; }
As we see, Background: 2
/3
/4
is shown in multiple scenarios, while being emitted only once each.
TIP: If you're
spawn
ing aFuture
inside your step matching function, consider to propagate itstracing::Span
into thespawn
edFuture
for outputting its logs properly.
IntelliJ Rust (libtest
) integration
writer::Libtest
(enabled by libtest
feature in Cargo.toml
) allows IntelliJ Rust plugin to interpret output of cucumber
tests similar to unit tests. To use it, just add Cargo configuration (current example uses cargo test --test wait --features libtest
command) or run it via Cargo command. This automatically adds --format=json
CLI option, which makes the cucumber
's output IDE-compatible.
Example below is set up to output with the default writer::Basic
if there is no --format=json
option, or with writer::Libtest
otherwise.
cucumber = { version = "0.20", features = ["libtest"] }
extern crate cucumber; extern crate tokio; use cucumber::{writer, World as _}; #[derive(cucumber::World, Debug, Default)] struct World; #[tokio::main] async fn main() { World::cucumber() .with_writer(writer::Libtest::or_basic()) .run("tests/features/book") .await; }
NOTE: There are currently 2 caveats with IntelliJ Rust integration:
- Because of output interpretation issue, current timing reports for individual tests are accurate only for serial tests (or for all in case
--concurrency=1
CLI option is used);- Although debugger works, test window may select
Step
that didn't trigger the breakpoint. To fix this, use--concurrency=1
CLI option.
TIP: In the multi-crate Cargo workspace, to support jump-to-definition in the reported paths (step or its matcher definition) correctly, consider to define
CARGO_WORKSPACE_DIR
environment variable in the.cargo/config.toml
file:[env] CARGO_WORKSPACE_DIR = { value = "", relative = true }
libtest
support
Only a small subset of libtest
harness is supported to integrate with other tools:
- Only
--format=json
output (JUnit
support is done separately); --report-time
option;--show-output
option.
Architecture
On high level, the whole Cucumber
is composed of three components:
Parser
, representing a source of features (default one parses.feature
files).Runner
, executing scenarios of features received from aParser
, and emittingevent
s (default one executes concurrently).Writer
, outputtingevent
s (default one outputs to STDOUT).
Any of these components is replaceable. This makes Cucumber
fully extensible, without a need to rewrite the whole library if it doesn't meet some exotic requirements. One could always write its own component, satisfying the needs, and use it. Imagine the situation, where features are sourced from distributed queue (like Kafka), then executed by a cluster of external workers (like Kubernetes Job
s), and, finally, results are emitted to different reporting systems by network. All this possible by introducing custom components, capable of doing that, without a need to change the framework.
To feel a little bit of its taste, we will write some trivial implementations of each component in subchapters below.
Custom Parser
Let's start by implementing a custom Parser
which statically emits a single feature for execution.
Parser
represents anything that emits a Stream
of features.
extern crate cucumber; extern crate futures; extern crate tokio; use std::{path::PathBuf, time::Duration}; use cucumber::{cli, gherkin, given, parser, then, when, World}; use futures::{future, stream}; use tokio::time::sleep; #[derive(Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } struct CustomParser; impl<I> cucumber::Parser<I> for CustomParser { type Cli = cli::Empty; // we provide no CLI options type Output = stream::Once<future::Ready<parser::Result<gherkin::Feature>>>; fn parse(self, _: I, _: Self::Cli) -> Self::Output { let keyword = "Feature"; let name = "Animal feature"; stream::once(future::ok(gherkin::Feature { keyword: keyword.into(), name: name.into(), description: None, background: None, scenarios: vec![gherkin::Scenario { keyword: "Scenario".into(), name: "If we feed a hungry cat it won't be hungry".into(), description: None, steps: vec![ gherkin::Step { keyword: "Given ".into(), ty: gherkin::StepType::Given, value: "a hungry cat".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 18 }, position: gherkin::LineCol { line: 3, col: 5 }, }, gherkin::Step { keyword: "When ".into(), ty: gherkin::StepType::When, value: "I feed the cat".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 19 }, position: gherkin::LineCol { line: 4, col: 5 }, }, gherkin::Step { keyword: "Then ".into(), ty: gherkin::StepType::Then, value: "the cat is not hungry".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 26 }, position: gherkin::LineCol { line: 5, col: 5 }, }, ], examples: vec![], tags: vec![], span: gherkin::Span { start: 3, end: 52 }, position: gherkin::LineCol { line: 2, col: 3 }, }], rules: vec![], tags: vec![], span: gherkin::Span { start: 1, end: 23 }, position: gherkin::LineCol { line: 1, col: 1 }, path: Some(PathBuf::from(file!())), })) } } #[tokio::main] async fn main() { AnimalWorld::cucumber::<&str>() // aiding type inference .with_parser(CustomParser) .run_and_exit("tests/features/book") // path doesn't actually matter .await; // here due to our implementation }
Custom Runner
Now, let's implement a custom Runner
which simply executes scenarios in features sequentially, without considering tags, rules, Background
s or other extras, and specifically suitable for our AnimalWorld
(for implementation simplicity).
Runner
represents anything that transforms a Stream
of features into a Stream
of cucumber events.
extern crate cucumber; extern crate futures; extern crate tokio; use std::{ panic::{self, AssertUnwindSafe}, path::PathBuf, sync::{Arc, LazyLock}, time::Duration, }; use cucumber::{ cli, event, gherkin, given, parser, step, then, when, Event, World, }; use futures::{ future::{self, FutureExt as _}, stream::{self, LocalBoxStream, Stream, StreamExt as _, TryStreamExt as _}, }; use tokio::time::sleep; #[derive(Clone, Copy, Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Clone, Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } struct CustomParser; impl<I> cucumber::Parser<I> for CustomParser { type Cli = cli::Empty; type Output = stream::Once<future::Ready<parser::Result<gherkin::Feature>>>; fn parse(self, _: I, _: Self::Cli) -> Self::Output { let keyword = "Feature"; let name = "Animal feature"; stream::once(future::ok(gherkin::Feature { keyword: keyword.into(), name: name.into(), description: None, background: None, scenarios: vec![gherkin::Scenario { keyword: "Scenario".into(), name: "If we feed a hungry cat it won't be hungry".into(), description: None, steps: vec![ gherkin::Step { keyword: "Given ".into(), ty: gherkin::StepType::Given, value: "a hungry cat".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 18 }, position: gherkin::LineCol { line: 3, col: 5 }, }, gherkin::Step { keyword: "When ".into(), ty: gherkin::StepType::When, value: "I feed the cat".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 19 }, position: gherkin::LineCol { line: 4, col: 5 }, }, gherkin::Step { keyword: "Then ".into(), ty: gherkin::StepType::Then, value: "the cat is not hungry".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 26 }, position: gherkin::LineCol { line: 5, col: 5 }, }, ], examples: vec![], tags: vec![], span: gherkin::Span { start: 3, end: 52 }, position: gherkin::LineCol { line: 2, col: 3 }, }], rules: vec![], tags: vec![], span: gherkin::Span { start: 1, end: 23 }, position: gherkin::LineCol { line: 1, col: 1 }, path: Some(PathBuf::from(file!())), })) } } struct CustomRunner; impl CustomRunner { fn steps_fns() -> &'static step::Collection<AnimalWorld> { // Wire the static collection of step matching functions. static STEPS: LazyLock<step::Collection<AnimalWorld>> = LazyLock::new(AnimalWorld::collection); &STEPS } async fn execute_step( mut world: AnimalWorld, step: gherkin::Step, ) -> (AnimalWorld, event::Step<AnimalWorld>) { let ev = if let Some((step_fn, captures, loc, ctx)) = Self::steps_fns().find(&step).expect("Ambiguous match") { // Panic represents a failed assertion in a step matching // function. match AssertUnwindSafe(step_fn(&mut world, ctx)) .catch_unwind() .await { Ok(()) => event::Step::Passed(captures, loc), Err(e) => event::Step::Failed( Some(captures), loc, Some(Arc::new(world.clone())), event::StepError::Panic(e.into()), ), } } else { event::Step::Skipped }; (world, ev) } async fn execute_scenario( scenario: gherkin::Scenario, ) -> impl Stream<Item = event::Feature<AnimalWorld>> { // Those panic hook shenanigans are done to avoid console messages like // "thread 'main' panicked at ..." // // 1. We obtain the current panic hook and replace it with an empty one. // 2. We run tests, which can panic. In that case we pass all panic info // down the line to the `Writer`, which will print it at right time. // 3. We restore original panic hook, because suppressing all panics // doesn't sound like a very good idea. let hook = panic::take_hook(); panic::set_hook(Box::new(|_| {})); let mut world = AnimalWorld::new().await.unwrap(); let mut steps = Vec::with_capacity(scenario.steps.len()); for step in scenario.steps.clone() { let (w, ev) = Self::execute_step(world, step.clone()).await; world = w; let should_stop = matches!(ev, event::Step::Failed(..)); steps.push((step, ev)); if should_stop { break; } } panic::set_hook(hook); let scenario = Arc::new(scenario); stream::once(future::ready(event::Scenario::Started)) .chain(stream::iter(steps.into_iter().flat_map(|(step, ev)| { let step = Arc::new(step); [ event::Scenario::Step(step.clone(), event::Step::Started), event::Scenario::Step(step, ev), ] }))) .chain(stream::once(future::ready(event::Scenario::Finished))) .map(move |event| event::Feature::Scenario( scenario.clone(), event::RetryableScenario { event, retries: None }, )) } fn execute_feature( feature: gherkin::Feature, ) -> impl Stream<Item = event::Cucumber<AnimalWorld>> { let feature = Arc::new(feature); stream::once(future::ready(event::Feature::Started)) .chain( stream::iter(feature.scenarios.clone()) .then(Self::execute_scenario) .flatten(), ) .chain(stream::once(future::ready(event::Feature::Finished))) .map(move |ev| event::Cucumber::Feature(feature.clone(), ev)) } } impl cucumber::Runner<AnimalWorld> for CustomRunner { type Cli = cli::Empty; // we provide no CLI options type EventStream = LocalBoxStream< 'static, parser::Result<Event<event::Cucumber<AnimalWorld>>>, >; fn run<S>(self, features: S, _: Self::Cli) -> Self::EventStream where S: Stream<Item = parser::Result<gherkin::Feature>> + 'static, { stream::once(future::ok(event::Cucumber::Started)) .chain( features .map_ok(|f| Self::execute_feature(f).map(Ok)) .try_flatten(), ) .chain(stream::once(future::ok(event::Cucumber::Finished))) .map_ok(Event::new) .boxed_local() } } #[tokio::main] async fn main() { AnimalWorld::cucumber::<&str>() // aiding type inference for `CustomParser` .with_parser(CustomParser) .with_runner(CustomRunner) .run_and_exit("tests/features/book") .await; }
NOTE: Output is printed only after all the steps were executed, because we have implemented the
CustomRunner
in the way to emitevent::Cucumber
s only after executing all the scenario steps (seeexecute_scenario()
function).
Custom Writer
Finally, let's implement a custom Writer
which simply outputs cucumber events to STDOUT in the order of receiving.
Writer
represents anything that consumes a Stream
of cucumber events.
extern crate cucumber; extern crate futures; extern crate tokio; use std::{ panic::{self, AssertUnwindSafe}, path::PathBuf, sync::{Arc, LazyLock}, time::Duration, }; use cucumber::{ cli, event, gherkin, given, parser, step, then, when, Event, World, WriterExt as _, }; use futures::{ future::{self, FutureExt as _}, stream::{self, LocalBoxStream, Stream, StreamExt as _, TryStreamExt as _}, }; use tokio::time::sleep; #[derive(Clone, Copy, Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Clone, Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } struct CustomParser; impl<I> cucumber::Parser<I> for CustomParser { type Cli = cli::Empty; type Output = stream::Once<future::Ready<parser::Result<gherkin::Feature>>>; fn parse(self, _: I, _: Self::Cli) -> Self::Output { let keyword = "Feature"; let name = "Animal feature"; stream::once(future::ok(gherkin::Feature { keyword: keyword.into(), name: name.into(), description: None, background: None, scenarios: vec![gherkin::Scenario { keyword: "Scenario".into(), name: "If we feed a hungry cat it won't be hungry".into(), description: None, steps: vec![ gherkin::Step { keyword: "Given".into(), ty: gherkin::StepType::Given, value: "a hungry cat".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 18 }, position: gherkin::LineCol { line: 3, col: 5 }, }, gherkin::Step { keyword: "When".into(), ty: gherkin::StepType::When, value: "I feed the cat".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 19 }, position: gherkin::LineCol { line: 4, col: 5 }, }, gherkin::Step { keyword: "Then".into(), ty: gherkin::StepType::Then, value: "the cat is not hungry".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 26 }, position: gherkin::LineCol { line: 5, col: 5 }, }, ], examples: vec![], tags: vec![], span: gherkin::Span { start: 3, end: 52 }, position: gherkin::LineCol { line: 2, col: 3 }, }], rules: vec![], tags: vec![], span: gherkin::Span { start: 1, end: 23 }, position: gherkin::LineCol { line: 1, col: 1 }, path: Some(PathBuf::from(file!())), })) } } struct CustomRunner; impl CustomRunner { fn steps_fns() -> &'static step::Collection<AnimalWorld> { static STEPS: LazyLock<step::Collection<AnimalWorld>> = LazyLock::new(AnimalWorld::collection); &STEPS } async fn execute_step( mut world: AnimalWorld, step: gherkin::Step, ) -> (AnimalWorld, event::Step<AnimalWorld>) { let ev = if let Some((step_fn, captures, loc, ctx)) = Self::steps_fns().find(&step).expect("Ambiguous match") { match AssertUnwindSafe(step_fn(&mut world, ctx)) .catch_unwind() .await { Ok(()) => event::Step::Passed(captures, loc), Err(e) => event::Step::Failed( Some(captures), loc, Some(Arc::new(world.clone())), event::StepError::Panic(e.into()), ), } } else { event::Step::Skipped }; (world, ev) } async fn execute_scenario( scenario: gherkin::Scenario, ) -> impl Stream<Item = event::Feature<AnimalWorld>> { let hook = panic::take_hook(); panic::set_hook(Box::new(|_| {})); let mut world = AnimalWorld::new().await.unwrap(); let mut steps = Vec::with_capacity(scenario.steps.len()); for step in scenario.steps.clone() { let (w, ev) = Self::execute_step(world, step.clone()).await; world = w; let should_stop = matches!(ev, event::Step::Failed(..)); steps.push((step, ev)); if should_stop { break; } } panic::set_hook(hook); let scenario = Arc::new(scenario); stream::once(future::ready(event::Scenario::Started)) .chain(stream::iter(steps.into_iter().flat_map(|(step, ev)| { let step = Arc::new(step); [ event::Scenario::Step(step.clone(), event::Step::Started), event::Scenario::Step(step, ev), ] }))) .chain(stream::once(future::ready(event::Scenario::Finished))) .map(move |event| event::Feature::Scenario( scenario.clone(), event::RetryableScenario { event, retries: None }, )) } fn execute_feature( feature: gherkin::Feature, ) -> impl Stream<Item = event::Cucumber<AnimalWorld>> { let feature = Arc::new(feature); stream::once(future::ready(event::Feature::Started)) .chain( stream::iter(feature.scenarios.clone()) .then(Self::execute_scenario) .flatten(), ) .chain(stream::once(future::ready(event::Feature::Finished))) .map(move |ev| event::Cucumber::Feature(feature.clone(), ev)) } } impl cucumber::Runner<AnimalWorld> for CustomRunner { type Cli = cli::Empty; type EventStream = LocalBoxStream< 'static, parser::Result<Event<event::Cucumber<AnimalWorld>>>, >; fn run<S>(self, features: S, _: Self::Cli) -> Self::EventStream where S: Stream<Item = parser::Result<gherkin::Feature>> + 'static, { stream::once(future::ok(event::Cucumber::Started)) .chain( features .map_ok(|f| Self::execute_feature(f).map(Ok)) .try_flatten(), ) .chain(stream::once(future::ok(event::Cucumber::Finished))) .map_ok(Event::new) .boxed_local() } } struct CustomWriter; impl<W: 'static> cucumber::Writer<W> for CustomWriter { type Cli = cli::Empty; // we provide no CLI options async fn handle_event( &mut self, ev: parser::Result<Event<event::Cucumber<W>>>, _: &Self::Cli, ) { match ev { Ok(Event { value, .. }) => match value { event::Cucumber::Feature(feature, ev) => match ev { event::Feature::Started => { println!("{}: {}", feature.keyword, feature.name) } event::Feature::Scenario(scenario, ev) => match ev.event { event::Scenario::Started => { println!("{}: {}", scenario.keyword, scenario.name) } event::Scenario::Step(step, ev) => match ev { event::Step::Started => { print!("{} {}...", step.keyword, step.value) } event::Step::Passed(..) => println!("ok"), event::Step::Skipped => println!("skip"), event::Step::Failed(_, _, _, err) => { println!("failed: {err}") } }, _ => {} }, _ => {} }, _ => {} }, Err(e) => println!("Error: {e}"), } } } #[tokio::main] async fn main() { AnimalWorld::cucumber::<&str>() // aiding type inference for `CustomParser` .with_parser(CustomParser) .with_runner(CustomRunner) .with_writer(CustomWriter.assert_normalized()) // OK because of `CustomRunner` .run("tests/features/book") .await; }
TIP:
CustomWriter
will print trash if we feed unorderedevent::Cucumber
s into it. Though, we shouldn't care about order normalization in our implementations. Instead, we may just wrapCustomWriter
intowriter::Normalize
, which will do that for us.
extern crate cucumber; extern crate futures; extern crate tokio; use std::{path::PathBuf, time::Duration}; use cucumber::{ cli, event, gherkin, given, parser, then, when, Event, World, WriterExt as _, }; use futures::{future, stream}; use tokio::time::sleep; #[derive(Clone, Copy, Debug, Default)] struct Animal { pub hungry: bool, } impl Animal { fn feed(&mut self) { self.hungry = false; } } #[derive(Clone, Debug, Default, World)] pub struct AnimalWorld { cat: Animal, } #[given(regex = r"^a (hungry|satiated) cat$")] async fn hungry_cat(world: &mut AnimalWorld, state: String) { sleep(Duration::from_secs(2)).await; match state.as_str() { "hungry" => world.cat.hungry = true, "satiated" => world.cat.hungry = false, _ => unreachable!(), } } #[when("I feed the cat")] async fn feed_cat(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; world.cat.feed(); } #[then("the cat is not hungry")] async fn cat_is_fed(world: &mut AnimalWorld) { sleep(Duration::from_secs(2)).await; assert!(!world.cat.hungry); } struct CustomParser; impl<I> cucumber::Parser<I> for CustomParser { type Cli = cli::Empty; type Output = stream::Once<future::Ready<parser::Result<gherkin::Feature>>>; fn parse(self, _: I, _: Self::Cli) -> Self::Output { let keyword = "Feature"; let name = "Animal feature"; stream::once(future::ok(gherkin::Feature { keyword: keyword.into(), name: name.into(), description: None, background: None, scenarios: vec![gherkin::Scenario { keyword: "Scenario".into(), name: "If we feed a hungry cat it won't be hungry".into(), description: None, steps: vec![ gherkin::Step { keyword: "Given".into(), ty: gherkin::StepType::Given, value: "a hungry cat".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 18 }, position: gherkin::LineCol { line: 3, col: 5 }, }, gherkin::Step { keyword: "When".into(), ty: gherkin::StepType::When, value: "I feed the cat".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 19 }, position: gherkin::LineCol { line: 4, col: 5 }, }, gherkin::Step { keyword: "Then".into(), ty: gherkin::StepType::Then, value: "the cat is not hungry".into(), docstring: None, table: None, span: gherkin::Span { start: 5, end: 26 }, position: gherkin::LineCol { line: 5, col: 5 }, }, ], examples: vec![], tags: vec![], span: gherkin::Span { start: 3, end: 52 }, position: gherkin::LineCol { line: 2, col: 3 }, }], rules: vec![], tags: vec![], span: gherkin::Span { start: 1, end: 23 }, position: gherkin::LineCol { line: 1, col: 1 }, path: Some(PathBuf::from(file!())), })) } } struct CustomWriter; impl<W: 'static> cucumber::Writer<W> for CustomWriter { type Cli = cli::Empty; // we provide no CLI options async fn handle_event( &mut self, ev: parser::Result<Event<event::Cucumber<W>>>, _: &Self::Cli, ) { match ev { Ok(Event { value, .. }) => match value { event::Cucumber::Feature(feature, ev) => match ev { event::Feature::Started => { println!("{}: {}", feature.keyword, feature.name) } event::Feature::Scenario(scenario, ev) => match ev.event { event::Scenario::Started => { println!("{}: {}", scenario.keyword, scenario.name) } event::Scenario::Step(step, ev) => match ev { event::Step::Started => { print!("{} {}...", step.keyword, step.value) } event::Step::Passed(..) => println!("ok"), event::Step::Skipped => println!("skip"), event::Step::Failed(_, _, _, err) => { println!("failed: {err}", ) } }, _ => {} }, _ => {} }, _ => {} }, Err(e) => println!("Error: {e}"), } } } #[tokio::main] async fn main() { AnimalWorld::cucumber::<&str>() // aiding type inference for `CustomParser` .with_parser(CustomParser) .with_writer(CustomWriter.normalized()) // wrapping into `writer::Normalize`, .run("tests/features/book") // so it works OK with the default .await; // concurrent `Runner` }
NOTE:
Writer
s are easily pipelined. SeeWriterExt
trait andwriter
module for moreWriter
machinery "included batteries".