Retry & Runs

LLMs are non-deterministic. assertllm has built-in retry at two levels.

Decorator Retry

Decorator-level retry

Retry the entire test up to N times:

test_retry.py


@llm_test(
    expect.contains("Paris"),
    model="gpt-4o-mini",
    retries=3,
    retry_delay=0.5,
)
def test_capital_with_retry(llm):
    llm("What is the capital of France?")

Output


test_retry.py::test_capital_with_retry
  Attempt 1/3: FAILED — contains("Paris") not found
  Attempt 2/3: PASSED
  ✓ contains("Paris")
  PASSED    [1.4s]

Use decorator retries when the whole test is flaky and you want it to pass if any single attempt succeeds.

Statistical Runs

Statistical testing (runs)

Run the test N times and pass if enough runs succeed:

test_stats.py


@llm_test(
    expect.contains("Paris"),
    model="gpt-4o-mini",
    runs=10,
    min_pass_rate=0.8,
)
def test_capital_reliable(llm):
    llm("What is the capital of France?")

Output


test_stats.py::test_capital_reliable
  Run 1/10: PASSED
  Run 2/10: PASSED
  ...
  Run 10/10: PASSED
  Pass rate: 9/10 (90%) >= 80% required
  PASSED    [8.2s]

This runs the test 10 times and passes if at least 8/10 (80%) succeed. Great for measuring reliability.

Fixture Retry

Fixture-level retry

Retry individual LLM calls with a condition:

test_fixture_retry.py


def test_with_retry_condition(llm):
    output = llm(
        "Name a European capital",
        model="gpt-4o-mini",
        retries=3,
        retry_if=lambda out: "Paris" not in out.content,
    )
    assert output.content

retry_if returns True to retry, False to accept. Use when a specific LLM call needs a particular output.

When to Use Which

Level	Use when
Decorator `retries`	The whole test is flaky
Decorator `runs`	You want statistical pass rate (e.g. 8/10)
Fixture `retry_if`	A specific LLM call needs a particular output