Ollama

Ollama allows users to run, interact with, and deploy AI models locally on their machines without the need for complex infrastructure or cloud dependencies.

There are multiple ways to interact with Ollama from Python including but not limited to the ollama python package, LangChain or by using the OpenAI library. We will cover how to trace your LLM calls for each of these methods.

You can check out the Colab Notebook if you’d like to jump straight to the code:

Getting started

Configure Ollama

Before starting, you will need to have an Ollama instance running. You can install Ollama by following the quickstart guide which will automatically start the Ollama API server. If the Ollama server is not running, you can start it using ollama serve.

Once Ollama is running, you can download the llama3.1 model by running ollama pull llama3.1. For a full list of models available on Ollama, please refer to the Ollama library.

Configure Opik

You will also need to have Opik installed. You can install and configure it by running the following command:

$ pip install --upgrade --quiet opik
> 
> opik configure

Opik is fully open-source and can be run locally or through the Opik Cloud platform. You can learn more about hosting Opik on your own infrastructure in the self-hosting guide.

Tracking Ollama calls made with Ollama Python Package

To get started you will need to install the Ollama Python package:

$ pip install --quiet --upgrade ollama

We will then utilize the track decorator to log all the traces to Opik:

1 import ollama
2 from opik import track, opik_context
3 
4 @track(tags=['ollama', 'python-library'])
5 def ollama_llm_call(user_message: str):
6     # Create the Ollama model
7     response = ollama.chat(model='llama3.1', messages=[
8         {
9             'role': 'user',
10             'content': user_message,
11         },
12     ])
13 
14     opik_context.update_current_span(
15         metadata={
16             'model': response['model'],
17             'eval_duration': response['eval_duration'],
18             'load_duration': response['load_duration'],
19             'prompt_eval_duration': response['prompt_eval_duration'],
20             'prompt_eval_count': response['prompt_eval_count'],
21             'done': response['done'],
22             'done_reason': response['done_reason'],
23         },
24         usage={
25             'completion_tokens': response['eval_count'],
26             'prompt_tokens': response['prompt_eval_count'],
27             'total_tokens': response['eval_count'] + response['prompt_eval_count']
28         }
29     )
30     return response['message']
31 
32 ollama_llm_call("Say this is a test")

The trace will now be displayed in the Opik platform.

Tracking Ollama calls made with OpenAI

Ollama is compatible with the OpenAI format and can be used with the OpenAI Python library. You can therefore leverage the Opik integration for OpenAI to trace your Ollama calls:

1 from openai import OpenAI
2 from opik.integrations.openai import track_openai
3 
4 # Create an OpenAI client
5 client = OpenAI(
6     base_url='http://localhost:11434/v1/',
7 
8     # required but ignored
9     api_key='ollama',
10 )
11 
12 # Log all traces made to with the OpenAI client to Opik
13 client = track_openai(client)
14 
15 # call the local ollama model using the OpenAI client
16 chat_completion = client.chat.completions.create(
17     messages=[
18         {
19             'role': 'user',
20             'content': 'Say this is a test',
21         }
22     ],
23     model='llama3.1',
24 )

The local LLM call is now traced and logged to Opik.

Tracking Ollama calls made with LangChain

In order to trace Ollama calls made with LangChain, you will need to first install the langchain-ollama package:

$ pip install --quiet --upgrade langchain-ollama langchain

You will now be able to use the OpikTracer class to log all your Ollama calls made with LangChain to Opik:

1 from langchain_ollama import ChatOllama
2 from opik.integrations.langchain import OpikTracer
3 
4 # Create the Opik tracer
5 opik_tracer = OpikTracer(tags=["langchain", "ollama"])
6 
7 # Create the Ollama model and configure it to use the Opik tracer
8 llm = ChatOllama(
9     model="llama3.1",
10     temperature=0,
11 ).with_config({"callbacks": [opik_tracer]})
12 
13 # Call the Ollama model
14 messages = [
15     (
16         "system",
17         "You are a helpful assistant that translates English to French. Translate the user sentence.",
18     ),
19     (
20         "human",
21         "I love programming.",
22     ),
23 ]
24 ai_msg = llm.invoke(messages)
25 ai_msg

You can now go to the Opik app to see the trace:

1	import ollama
2	from opik import track, opik_context
3
4	@track(tags=['ollama', 'python-library'])
5	def ollama_llm_call(user_message: str):
6	# Create the Ollama model
7	response = ollama.chat(model='llama3.1', messages=[
8	{
9	'role': 'user',
10	'content': user_message,
11	},
12	])
13
14	opik_context.update_current_span(
15	metadata={
16	'model': response['model'],
17	'eval_duration': response['eval_duration'],
18	'load_duration': response['load_duration'],
19	'prompt_eval_duration': response['prompt_eval_duration'],
20	'prompt_eval_count': response['prompt_eval_count'],
21	'done': response['done'],
22	'done_reason': response['done_reason'],
23	},
24	usage={
25	'completion_tokens': response['eval_count'],
26	'prompt_tokens': response['prompt_eval_count'],
27	'total_tokens': response['eval_count'] + response['prompt_eval_count']
28	}
29	)
30	return response['message']
31
32	ollama_llm_call("Say this is a test")

1	from openai import OpenAI
2	from opik.integrations.openai import track_openai
3
4	# Create an OpenAI client
5	client = OpenAI(
6	base_url='http://localhost:11434/v1/',
7
8	# required but ignored
9	api_key='ollama',
10	)
11
12	# Log all traces made to with the OpenAI client to Opik
13	client = track_openai(client)
14
15	# call the local ollama model using the OpenAI client
16	chat_completion = client.chat.completions.create(
17	messages=[
18	{
19	'role': 'user',
20	'content': 'Say this is a test',
21	}
22	],
23	model='llama3.1',
24	)

1	from langchain_ollama import ChatOllama
2	from opik.integrations.langchain import OpikTracer
3
4	# Create the Opik tracer
5	opik_tracer = OpikTracer(tags=["langchain", "ollama"])
6
7	# Create the Ollama model and configure it to use the Opik tracer
8	llm = ChatOllama(
9	model="llama3.1",
10	temperature=0,
11	).with_config({"callbacks": [opik_tracer]})
12
13	# Call the Ollama model
14	messages = [
15	(
16	"system",
17	"You are a helpful assistant that translates English to French. Translate the user sentence.",
18	),
19	(
20	"human",
21	"I love programming.",
22	),
23	]
24	ai_msg = llm.invoke(messages)
25	ai_msg