Pytest integration

Ensuring your LLM applications is working as expected is a crucial step before deploying to production. Opik provides a Pytest integration so that you can easily track the overall pass / fail rates of your tests as well as the individual pass / fail rates of each test.

Using the Pytest Integration

We recommend using the llm_unit decorator to wrap your tests. This will ensure that Opik can track the results of your tests and provide you with a detailed report. It also works well when used in conjunction with the track decorator used to trace your LLM application.

1
import pytest
2
from opik import track, llm_unit
3
4
@track
5
def llm_application(user_question: str) -> str:
6
    # LLM application code here
7
    return "Paris"
8
9
@llm_unit()
10
def test_simple_passing_test():
11
    user_question = "What is the capital of France?"
12
    response = llm_application(user_question)
13
    assert response == "Paris"

When you run the tests, Opik will create a new experiment for each run and log each test result. By navigating to the tests dataset, you will see a new experiment for each test run.

Advanced Usage

The llm_unit decorator also works well when used in conjunctions with the parametrize Pytest decorator that allows you to run the same test with different inputs:

1
import pytest
2
from opik import track, llm_unit
3
4
@track
5
def llm_application(user_question: str) -> str:
6
    # LLM application code here
7
    return "Paris"
8
9
@llm_unit(expected_output_key="expected_output")
10
@pytest.mark.parametrize("user_question, expected_output", [
11
    ("What is the capital of France?", "Paris"),
12
    ("What is the capital of Germany?", "Berlin")
13
])
14
def test_simple_passing_test(user_question, expected_output):
15
    response = llm_application(user_question)
16
    assert response == expected_output