Using Opik with LiteLLM

Lite allows you to call all LLM APIs using the OpenAI format [Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq etc.]. You can learn more about LiteLLM here.

There are two main approaches to using LiteLLM, either using the litellm python library that will query the LLM API for you or by using the LiteLLM proxy server. In this cookbook we will focus on the first approach but you can learn more about using Opik with the LiteLLM proxy server in our documentation.

Creating an account on Comet.com

Comet provides a hosted version of the Opik platform, simply create an account and grab you API Key.

You can also run the Opik platform locally, see the installation guide for more information.

1 %pip install --upgrade opik litellm

1 import opik
2 
3 opik.configure(use_local=False)

Preparing our environment

In order to use LiteLLM, we will configure the OpenAI API Key, if you are using any other providers you can replace this with the required API key:

1 import os
2 import getpass
3 
4 if "OPENAI_API_KEY" not in os.environ:
5     os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")

Logging traces

In order to log traces to Opik, you will need to set the opik callback:

1 from litellm.integrations.opik.opik import OpikLogger
2 from opik.opik_context import get_current_span_data
3 from opik import track
4 import litellm
5 
6 os.environ["OPIK_PROJECT_NAME"] = "litellm-integration-demo"
7 opik_logger = OpikLogger()
8 litellm.callbacks = [opik_logger]

Every LiteLLM call will now be logged to Opik:

1 response = litellm.completion(
2     model="gpt-3.5-turbo",
3     messages=[
4         {"role": "user", "content": "Why is tracking and evaluation of LLMs important?"}
5     ],
6 )
7 
8 print(response.choices[0].message.content)

The trace will now be viewable in the Opik platform:

OpenAI Integration

Logging LLM calls within a tracked function

If you are using LiteLLM within a function tracked with the @track decorator, you will need to pass the current_span_data as metadata to the litellm.completion call:

1 @track
2 def streaming_function(input):
3     messages = [{"role": "user", "content": input}]
4     response = litellm.completion(
5         model="gpt-3.5-turbo",
6         messages=messages,
7         metadata={
8             "opik": {
9                 "current_span_data": get_current_span_data(),
10                 "tags": ["streaming-test"],
11             },
12         },
13     )
14     return response
15 
16 
17 response = streaming_function("Why is tracking and evaluation of LLMs important?")
18 chunks = list(response)

1	import os
2	import getpass
3
4	if "OPENAI_API_KEY" not in os.environ:
5	os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")

1	from litellm.integrations.opik.opik import OpikLogger
2	from opik.opik_context import get_current_span_data
3	from opik import track
4	import litellm
5
6	os.environ["OPIK_PROJECT_NAME"] = "litellm-integration-demo"
7	opik_logger = OpikLogger()
8	litellm.callbacks = [opik_logger]

1	response = litellm.completion(
2	model="gpt-3.5-turbo",
3	messages=[
4	{"role": "user", "content": "Why is tracking and evaluation of LLMs important?"}
5	],
6	)
7
8	print(response.choices[0].message.content)

1	@track
2	def streaming_function(input):
3	messages = [{"role": "user", "content": input}]
4	response = litellm.completion(
5	model="gpt-3.5-turbo",
6	messages=messages,
7	metadata={
8	"opik": {
9	"current_span_data": get_current_span_data(),
10	"tags": ["streaming-test"],
11	},
12	},
13	)
14	return response
15
16
17	response = streaming_function("Why is tracking and evaluation of LLMs important?")
18	chunks = list(response)