Custom model

Opik provides a set of LLM as a Judge metrics that are designed to be model-agnostic and can be used with any LLM. In order to achieve this, we use the LiteLLM library to abstract the LLM calls.

By default, Opik will use the gpt-4o model. However, you can change this by setting the model parameter when initializing your metric to any model supported by LiteLLM:

1from opik.evaluation.metrics import Hallucination
2
3hallucination_metric = Hallucination(
4    model="gpt-4-turbo"
5)

Using a model supported by LiteLLM

In order to use many models supported by LiteLLM, you also need to pass additional parameters. For this, you can use the LiteLLMChatModel class and passing it to the metric:

1from opik.evaluation.metrics import Hallucination
2from opik.evaluation import models
3
4model = models.LiteLLMChatModel(
5    name="<model_name>",
6    base_url="<base_url>"
7)
8
9hallucination_metric = Hallucination(
10    model=model
11)

Creating your own custom model class

You can create your own custom model class by subclassing the OpikBaseModel class and implementing a few methods:

1from opik.evaluation.models import OpikBaseModel
2from typing import Any
3
4class CustomModel(OpikBaseModel):
5    def __init__(self, model_name: str):
6        super().__init__(model_name)
7
8    def generate_provider_response(self, **kwargs: Any) -> str:
9        """
10        Generate a provider-specific response. Can be used to interface with
11        the underlying model provider (e.g., OpenAI, Anthropic) and get raw output.
12        """
13        pass
14
15    def agenerate_provider_response_stream(self, **kwargs: Any) -> str:
16        """
17        Generate a provider-specific response. Can be used to interface with
18        the underlying model provider (e.g., OpenAI, Anthropic) and get raw output.
19        Async version.
20        """
21        pass
22
23    def agenerate_provider_response(self, **kwargs: Any) -> str:
24        """
25        Generate a provider-specific response. Can be used to interface with
26        the underlying model provider (e.g., OpenAI, Anthropic) and get raw output.
27        Async version.
28        """
29        pass
30
31    def agenerate_string(self, input: str, **kwargs: Any) -> str:
32        """Simplified interface to generate a string output from the model. Async version."""
33        pass
34
35    def generate_string(self, input: str, **kwargs: Any) -> str:
36        """Simplified interface to generate a string output from the model."""
37        return input

This model class can then be used in the same way as the built-in models:

1from opik.evaluation.metrics import Hallucination
2
3hallucination_metric = Hallucination(
4    model=CustomModel(model_name="demo_model")
5)
Built with