Retrying failed scenarios

Often, it's nearly impossible to create fully-deterministic test case, especially when you are relying on environments like external services, browsers, file system, networking etc. That's why there is an ability to retry failed scenarios.

WARNING: Although this feature is supported, we highly recommend to use it as the last resort only. First, consider implementing in-step retries with your own needs (like exponential backoff). Other ways of dealing with flaky tests include, but not limited to: reducing number of concurrently executed scenarios (maybe even using @serial tag), mocking external environment, controlling time in tests or even simulation testing. It's always better to move towards tests determinism, rather than trying to tame their flakiness.

Tags

Recommended way to specify retried scenarios is using tags (inheritance is supported too):

Feature: Heads and tails

  # Attempts a single retry immediately.
  @retry
  Scenario: Tails
    Given a coin
    When I flip the coin
    Then I see tails
      
  # Attempts a single retry in 1 second.
  @retry.after(1s)
  Scenario: Heads
    Given a coin
    When I flip the coin
    Then I see heads

  # Attempts to retry 5 times with no delay between them.
  @retry(5)
  Scenario: Edge
    Given a coin
    When I flip the coin
    Then I see edge

  # Attempts to retry 10 times with 100 milliseconds delay between them.
  @retry(10).after(100ms)
  Scenario: Levitating
    Given a coin
    When I flip the coin
    Then the coin never lands

extern crate cucumber;
extern crate rand;
extern crate tokio;

use std::time::Duration;

use cucumber::{given, then, when, World};
use rand::Rng as _;
use tokio::time::sleep;

#[derive(Debug, Default, World)]
pub struct FlipWorld {
    flipped: &'static str,
}

#[given("a coin")]
async fn coin(_: &mut FlipWorld) {
    sleep(Duration::from_secs(2)).await;
}

#[when("I flip the coin")]
async fn flip(world: &mut FlipWorld) {
    sleep(Duration::from_secs(2)).await;

    world.flipped = match rand::thread_rng().gen_range(0.0..1.0) {
        p if p < 0.2 => "edge",
        p if p < 0.5 => "heads",
        _ => "tails",
    }
}

#[then(regex = r#"^I see (heads|tails|edge)$"#)]
async fn see(world: &mut FlipWorld, what: String) {
    sleep(Duration::from_secs(2)).await;

    assert_eq!(what, world.flipped);
}

#[then("the coin never lands")]
async fn never_lands(_: &mut FlipWorld) {
    sleep(Duration::from_secs(2)).await;

    unreachable!("coin always lands")
}

#[tokio::main]
async fn main() {
    FlipWorld::cucumber()
        .fail_on_skipped()
        .run_and_exit("tests/features/book/writing/retries.feature")
        .await;
}

NOTE: On failure, the whole scenario is re-executed with a new fresh World instance.

CLI

The following CLI options are related to the scenario retries:

--retry <int>
    Number of times a scenario will be retried in case of a failure

--retry-after <duration>
    Delay between each scenario retry attempt.
    
    Duration is represented in a human-readable format like `12min5s`.
    Supported suffixes:
    - `nsec`, `ns` — nanoseconds.
    - `usec`, `us` — microseconds.
    - `msec`, `ms` — milliseconds.
    - `seconds`, `second`, `sec`, `s` - seconds.
    - `minutes`, `minute`, `min`, `m` - minutes.

--retry-tag-filter <tagexpr>
    Tag expression to filter retried scenarios

• --retry CLI option is similar to @retry(<number-of-retries>) tag, but is applied to all scenarios matching the --retry-tag-filter (if not provided, all possible scenarios are matched).
• --retry-after CLI option is similar to @retry.after(<delay-after-each-retry>) tag in the same manner.

Precedence of tags and CLI options

• Just @retry tag takes the number of retries and the delay from --retry and --retry-after CLI options respectively, if they're specified, otherwise defaults to a single retry attempt with no delay.
• @retry(3) tag always retries failed scenarios at most 3 times, even if --retry CLI option provides a greater value. Delay is taken from --retry-after CLI option, if it's specified, otherwise defaults to no delay.
• @retry.after(1s) tag always delays 1 second before next retry attempt, even if --retry-after CLI option provides another value. Number of retries is taken from --retry-after CLI option, if it's specified, otherwise defaults a single retry attempt.
• @retry(3).after(1s) always retries failed scenarios at most 3 times with 1 second delay before each attempt, ignoring --retry and --retry-after CLI options.

NOTE: When using with --fail-fast CLI option (or .fail_fast() builder config), scenarios are considered as failed only in case they exhaust all retry attempts and then still do fail.

TIP: It could be handy to specify @retry tags only, without any explicit values, and use --retry=n --retry-after=d --retry-tag-filter=@retry CLI options to overwrite retrying parameters without affecting any other scenarios.