Opik is Open Source! You can find the full source code on [GitHub](https://github.com/comet-ml/opik) and the complete
self-hosting guide can be found [here](/self-host/local_deployment).
## How to use Opik
**Using Claude Code, Cursor, or VS Code Copilot?** Install the [Opik MCP server](/mcp-server) and drive your entire workspace from chat — read traces, score outputs, save prompts, and run experiments without opening the UI.
### See what your application is doing
Opik records every LLM call, tool invocation, and agent step so you can inspect the full chain of events that led to any output. Add a few lines of code and you'll have a complete log of every request and response.
[Start logging traces →](/tracing/getting-started)
### Build test suites from your traces
When you spot a trace that looks wrong, turn it into a test case. Use [Ollie](/ollie) to do this automatically (just describe what went wrong), or add test cases through the UI or SDK. Then run your test suite with Ollie or from the SDK to verify your fixes.
Over time, your test suite grows from real production failures, not hypothetical examples.
[Build your first test suite →](/evaluation/advanced/building-test-suites)
### Track quality in production
Set up online evaluation rules that automatically score incoming traces, and monitor feedback scores, latency, cost, and error rates from the project dashboard.
[Set up production monitoring →](/production/online-evaluation/rules)
### Automatically improve your prompts
Opik's optimization algorithms test variations of your prompts against your metrics and datasets to find what works best, without manual trial and error.
[Run your first optimization →](/development/optimization-runs/quickstart)
## Explore by feature
Get Opik running with your existing AI stack in minutes. Works with OpenAI, Anthropic, LangChain, and 50+ other providers and frameworks.
Connect Claude Code, Cursor, or VS Code Copilot directly to your Opik workspace. Read traces, score outputs, and run experiments from chat — no UI required.
Record every LLM call, tool invocation, and agent step. Debug failures, track token costs, and understand what your application is doing.
Score your application on hallucination, context recall, relevance, and more using automated LLM-as-a-judge and heuristic metrics.
Automatically generate and test better prompts for every step in your agent using six optimization algorithms.
Store and version your prompts, compare results in the [Prompt Playground](/development/prompt-playground), and experiment with different models.
Deploy on your own infrastructure with Docker locally or Kubernetes at scale. Full control over your data.
## See it in action
## Open-source access meets enterprise performance
All Opik versions ([cloud](https://www.comet.com/signup?from=llm),
[open source](https://github.com/comet-ml/opik), and
[enterprise](https://www.comet.com/site/pricing/)) include the full AI engineering feature set
and run on the Comet platform, with proven performance at scale supporting many of the world's
largest organizations.
# Quickstart
This guide helps you integrate the Opik platform with your existing Agent. The goal of
this guide is to help you log your first traces and start tracking your prompts and agent
configuration in Opik.
## Meet Ollie
Ollie is a conversational AI assistant built into Opik. It lives next to every trace, dataset, experiment, and prompt you've logged — and when you pair your local project with `opik connect`, Ollie can also read your source files, run your agent, and propose code changes. One assistant, full context: your data on one side, your code on the other.
## What Ollie can do
Ollie reads full span trees, identifies root causes, and compares failing
runs to successful ones. Ask *"why did the final answer ignore the context?"*
and get a grounded answer, not a guess.
Pair your local project with `opik connect` and Ollie gains the ability to
read your source files, propose edits (with your approval), and rerun your
agent with the changes in place.
Every fix becomes a test. Ollie can add a failing trace to a test suite,
then trigger the suite against your updated agent to confirm the regression
is gone.
Traces, threads, datasets, experiments, prompts — Ollie queries across all
of them in a single conversation. No tab-switching, no manual stitching.
## From debugging to improvement
The real workflow — and the reason Ollie exists — is the full loop from *"something looks wrong"* to *"it's fixed and it won't regress."* Here's what that looks like in Opik:
### Spot a failing trace
Start where every debugging session starts: a trace that looks wrong. Filter by error status, low feedback score, or latency spike.
### Ask Ollie what went wrong
Open Ollie from the trace view. Describe what looks off — *"the model ignored the retrieved context"*, *"the tool call returned empty"* — and Ollie walks the span tree to find the cause.
### Let Ollie read your code
Once Ollie identifies *where* the bug lives, it asks to read the relevant source file. With `opik connect` running on your machine, it has secure access to the files in your project — and shows you exactly what it's looking at.
### Approve the fix
Ollie proposes a change. You see the diff, approve it, and the file is updated on your machine. Nothing happens without your click.
### Rerun the agent from Opik
With the fix applied locally, Ollie reruns your agent through `opik connect` using the same inputs from the original failing trace. The new trace streams back into Opik in real time.
### Verify with a test suite
Ask Ollie to add the original trace to a test suite as a regression case, then run the suite against the updated agent. You get a pass/fail summary — and a test that will catch the bug if it ever comes back.
## Get started with Ollie
### Run `opik connect` in your agent folder
From the root of your agent project, run:
```bash
opik connect --project 
If you are using the Open-Source Opik platform, you will not have Opik API keys. You can configure Opik by running
`opik configure` in your terminal which will prompt you for your Opik deployment and create all the required
configurations.
### How do I find my workspace and project name?
Your **workspace name** and **project name** are displayed in the Opik UI:
**Workspace Name:**
* Look at the top of the page in the breadcrumb navigation
* It appears as the first item after "opik" in the breadcrumb path
* Example: `opik > your-workspace-name > Projects > your-project-name`
**Project Name:**
* When you're inside a project, it appears as the main header in the content area
* It's also the last item in the breadcrumb navigation
* Example: `opik > your-workspace-name > Projects > your-project-name`
You can also see your workspace information in the left sidebar under "Projects" which shows the
count of projects in your workspace.
### Are there are rate limits on Opik Cloud?
Yes, in order to ensure all users have a good experience we have implemented rate limits. When you
encounter a rate limit, endpoints will return the status code `429`.
There's a global rate limit of `2,000` request/minute per user across all REST API endpoints, with
an extra burst of `100` requests.
Afterward, there's a data ingestion limit of `10,000` events/minute per user. An event is any trace,
span, feedback score, dataset item, experiment item, etc. which is ingested, stored and persisted by
Opik.
Additionally, there's another data ingestion limit of `5,000` events/minute per workspace and per
user.
Finally, there's a rate limit of `250` requests/minute per user for the `Get span by id` endpoint:
`GET /api/v1/private/spans/:id`.
The Python SDK has implemented some logic to slow down the logging to avoid data loss when
encountering rate limits. You will see the message: `OPIK: Ingestion rate limited, retrying in 55 seconds, remaining queue size: 1, ...`.
If you are using other logging methods, you will need to implement your own "backoff and retry"
strategy
For questions about rate limits, reach out to us on [Slack](https://chat.comet.com).
## Integrations
### What integrations does Opik support?
Opik supports a comprehensive range of popular LLM frameworks, providers, and tools. You can find
detailed integration guides in our [Integrations Overview](/integrations/overview).
**Model Providers:**
Anthropic, AWS Bedrock, BytePlus, Cloudflare Workers AI, Cohere, DeepSeek, Fireworks AI, Google
Gemini, Groq, Mistral AI, Novita AI, Ollama, OpenAI (Python & JS/TS), Predibase, Together AI, IBM
WatsonX, xAI Grok
**Frameworks:**
AG2, Agent Spec, Agno, Autogen, CrewAI, DSPy, Haystack, Instructor, LangChain (Python & JS/TS), LangGraph,
LlamaIndex, Mastra, Pydantic AI, Semantic Kernel, Smolagents, Spring AI, Strands Agents, VoltAgent,
OpenAI Agents, Google Agent Development Kit, LiveKit Agents, BeeAI
**Evaluation & Testing:**
Ragas
**Gateways & Proxies:**
LiteLLM, OpenRouter, AISuite
**No-Code Tools:**
Dify, Flowise
**OpenTelemetry:**
OpenTelemetry (Python & Ruby SDKs)
**Other Tools:**
Guardrails AI
### What if Opik doesn't support my preferred framework or tool?
If you don't see your preferred framework or tool listed in our integrations, we encourage you to:
1. Open an [issue on GitHub](https://github.com/comet-ml/opik/issues) to request the integration
2. In the meantime, you can manually log your LLM interactions using our SDK's core logging
functions - see our [tracing documentation](/tracing/advanced/log_traces) for examples
We actively maintain and expand our integration support based on community feedback.
## Troubleshooting
### Why am I getting 403 errors?
If you're encountering 403 (Forbidden) errors, this typically indicates an authentication or
authorization issue. If you haven't configured your credentials yet, the easiest way to get started
is to run:
```bash
opik configure
```
This interactive command will guide you through setting up the required configuration.
Otherwise, please double-check your existing configuration:
For Opik Cloud by Comet:
* `api_key` (required): Verify your API key is correct and active
* `workspace` (optional): If specified, confirm you have access to the specified workspace
* `project_name` (optional): If specified, ensure the project name is valid
* `url_override`: Should be set to `https://www.comet.com/opik/api` (this is the default)
For Self-hosted Opik:
* `url_override` (required): Verify your base URL points to your Opik instance (e.g.,
`http://your-instance:5173/api`)
You can find your current configuration in the Opik configuration file (`~/.opik.config`) or by
checking your environment variables (`OPIK_API_KEY`, `OPIK_WORKSPACE`, `OPIK_URL_OVERRIDE`,
`OPIK_PROJECT_NAME`). For more details on configuration, see our
[SDK Configuration guide](/tracing/advanced/sdk_configuration).
## How can I diagnose issues with Opik?
If you are experiencing any problems using Opik, such as receiving 400 or 500 errors from the
backend, or being unable to connect at all, we recommend running the following command in your
terminal:
```bash
opik healthcheck
```
This command will analyze your configuration and backend connectivity, providing useful insights
into potential issues.
Reviewing these sections can help pinpoint the source of the problem and suggest possible
resolutions.
### ⌨️ Using Comet Debugger Mode (UI/Browser)
**Comet Debugger Mode** is a hidden diagnostic feature in the **Opik web application** that displays real-time technical information to help you troubleshoot issues. This mode is particularly useful when investigating connectivity problems, reporting bugs, or verifying your deployment version.
**To toggle Comet Debugger Mode:**
Press `Command + Shift + .` on macOS or `Ctrl + Shift + .` on Windows/Linux
**What it displays:**
* **Network Status**: Real-time connectivity indicator with RTT (Round Trip Time) showing latency to the Opik backend server in seconds
* **Opik Version**: The current version of Opik you're running (click to copy to clipboard)
This information is helpful when:
* Reporting issues to the Opik team (include the version number and RTT)
* Verifying your Opik version matches expected deployment
* Diagnosing connectivity problems between UI and backend (check RTT for latency issues)
* Troubleshooting UI-related issues or unexpected behavior
* Confirming successful updates or deployments
* Monitoring network performance and latency to the backend server
**How it works:**
The keyboard shortcut toggles the debug information overlay on and off. When enabled, a small
status bar appears in the UI showing the network connectivity status and version information.
The mode persists across browser sessions (stored in local storage), so you only need to enable
it once until you toggle it off again.
The debugger mode setting persists across sessions and is stored in your browser's local storage.
Press the keyboard shortcut again to hide the debug information.
# Upgrading to Opik 2.0
Opik 2.0 reorganized the product around **projects**. A project now maps to a single agent or app and is the home for everything related to it. This page explains what changed, how to work with the SDK from now on, where your existing data was moved, and how to relocate it if it didn't land where you want.
If you started on Opik 2.0, there's nothing to do here — projects are the default. This
guide is for users upgrading from Opik 1.x. For the full list of new features, see the
[2.0 release notes](/changelog).
## What changed: projects are the home for everything
In Opik 1.x, traces and threads lived in projects, but **datasets, prompts, experiments, optimizations, automation rules, alerts, and dashboards were workspace-wide**. In 2.0, a project maps to one agent or app and scopes all of its work.
These entities are now **scoped to a project**:
* Datasets and test suites
* Experiments and optimizations
* Prompts
* Automation rules — in 1.x a rule could target multiple projects; in 2.0 a rule is scoped to a single project
* Alerts
* Dashboards — workspace-level dashboards remain supported through a dedicated view
You also get a workspace-level project selector, project-scoped navigation across the app, a unified Logs page (threads, traces, and spans in one place), and a redesigned trace view. The result is a focused view of everything tied to a single agent. See the [2.0 release notes](/changelog) for the full picture.
## How to work now
Upgrade to the Opik SDK **2.0 or later**:
```bash
pip install --upgrade "opik>=2.0.0"
```
```bash
npm install opik@latest
```
The main change is to **pass the project to the methods you call**. The dataset, test suite, prompt, and experiment APIs all take a `project_name` (`projectName` in TypeScript). Omit it and Opik uses the project from the `OPIK_PROJECT_NAME` environment variable or your config, falling back to `Default Project`.
Running `opik configure` now also prompts you to pick a project — it suggests your most
recent one and stores your choice as `project_name` in `~/.opik.config`, so you only pass
`project_name` on a call when you want to override that default.
All snippets below assume a client:
```python
import opik
client = opik.Opik()
```
```typescript
import { Opik } from "opik";
const client = new Opik();
```
### Datasets
```python
client.create_dataset(name="qa-pairs", project_name="my-agent")
client.get_dataset(name="qa-pairs", project_name="my-agent")
client.get_or_create_dataset(name="qa-pairs", project_name="my-agent")
client.get_datasets(project_name="my-agent")
client.delete_dataset(name="qa-pairs", project_name="my-agent")
client.get_dataset_experiments(dataset_name="qa-pairs", project_name="my-agent")
```
```typescript
await client.createDataset("qa-pairs", "QA pairs", "my-agent");
await client.getDataset("qa-pairs", "my-agent");
await client.getOrCreateDataset("qa-pairs", "QA pairs", "my-agent");
await client.getDatasets(100, "my-agent");
await client.deleteDataset("qa-pairs", "my-agent");
await client.getDatasetExperiments("qa-pairs", 100, "my-agent");
```
### Test suites
```python
client.create_test_suite(name="qa-suite", project_name="my-agent")
client.get_test_suite(name="qa-suite", project_name="my-agent")
client.get_or_create_test_suite(name="qa-suite", project_name="my-agent")
client.get_test_suites(project_name="my-agent")
client.delete_test_suite(name="qa-suite", project_name="my-agent")
client.get_test_suite_experiments(name="qa-suite", project_name="my-agent")
```
```typescript
await client.createTestSuite({ name: "qa-suite", projectName: "my-agent" });
await client.getTestSuite("qa-suite", "my-agent");
await client.getOrCreateTestSuite({ name: "qa-suite", projectName: "my-agent" });
await client.getTestSuites(1000, "my-agent");
await client.deleteTestSuite("qa-suite", "my-agent");
await client.getTestSuiteExperiments("qa-suite", 100, "my-agent");
```
### Prompts
```python
client.create_prompt(name="assistant", prompt="Answer: {{question}}", project_name="my-agent")
client.create_chat_prompt(
name="assistant-chat",
messages=[{"role": "user", "content": "{{question}}"}],
project_name="my-agent",
)
client.get_prompt(name="assistant", project_name="my-agent")
client.get_chat_prompt(name="assistant-chat", project_name="my-agent")
client.get_prompt_history(name="assistant", project_name="my-agent")
client.get_chat_prompt_history(name="assistant-chat", project_name="my-agent")
client.get_all_prompts(name="assistant", project_name="my-agent")
client.search_prompts(project_name="my-agent")
```
```typescript
await client.createPrompt({
name: "assistant",
prompt: "Answer: {{question}}",
projectName: "my-agent",
});
await client.createChatPrompt({
name: "assistant-chat",
messages: [{ role: "user", content: "{{question}}" }],
projectName: "my-agent",
});
await client.getPrompt({ name: "assistant", projectName: "my-agent" });
await client.getChatPrompt({ name: "assistant-chat", projectName: "my-agent" });
```
### Experiments
An experiment — and **every trace, span, and feedback score it produces** — always lands in the **same project as the dataset it runs against**. You don't scope an experiment separately: put the dataset in the project you want, and each run of it follows. This holds whether you call `opik.evaluate(...)` or create the experiment directly.
```python
# project_name must be the dataset's project — the experiment and its data land there
client.create_experiment(dataset_name="qa-pairs", name="run-1", project_name="my-agent")
client.get_experiment_by_name(name="run-1", project_name="my-agent")
client.get_experiments_by_name(name="run-1", project_name="my-agent")
```
```typescript
await client.createExperiment({
datasetName: "qa-pairs",
name: "run-1",
projectName: "my-agent", // the dataset's project
});
await client.getExperiment("run-1", "my-agent");
await client.getExperimentsByName("run-1", "my-agent");
```
## Where your existing data lands
When your workspace is upgraded to 2.0, your workspace-scoped 1.x data — datasets, prompts, experiments, and optimizations — is **moved into projects**.
Each item is placed in the project Opik can infer from the runs that used it. When there's nothing to infer from (no associated runs, or the original project had been deleted), the item is placed in your **`Default Project`**.
So if you don't see a dataset, prompt, or experiment where you expected it, check the project its traces and experiments belong to — and check **`Default Project`**.
**Opik Cloud (Comet-hosted):** the move runs automatically — there is nothing to start by hand.
A workspace is promoted to 2.0 as soon as its blocking workspace-level (orphan) data has been
migrated into projects.
On **self-hosted installations**, migration requires enabling six background jobs on your
`opik-backend` deployment. See the [self-hosted migration guide](/self-host/v1-to-v2-migration)
for step-by-step instructions.
## Moving data to a different project
If something landed in a project you don't want, you can move it with the [`opik migrate`](/tracing/advanced/migrate-data) CLI. It relocates a dataset (with its experiments, traces, and spans) or a prompt (with its full version history) into another project.
```bash
# Move a dataset that landed in Default Project into your agent's project
opik migrate dataset "qa-pairs" --to-project="my-agent"
```
Preview any move with `--dry-run` first. See [Migrate data](/tracing/advanced/migrate-data) for the full guide — what's copied, the options, and troubleshooting.
# Observability Overview
If you want to jump straight to code, head to the [Getting started](/tracing/getting-started) guide to add tracing in under five minutes.
LLM applications are more than a single API call. A typical agent involves retrieval steps, tool calls, prompt assembly, multiple LLM invocations, and post-processing — all wired together in ways that are invisible at runtime. When something goes wrong, you need to see exactly what happened at every step.
Opik gives you full visibility into every request your agent handles. Every LLM call, every tool invocation, every retrieval step is captured as a trace you can inspect, search, and analyze.
## Why use Opik for observability
Debugging LLM applications without observability means guessing. You see the final output but not why the model hallucinated, which retrieval step returned irrelevant context, or where latency spiked.
With Opik, you can:
* **See the full execution path** of every request — from user input through tool calls and LLM completions to the final response
* **Root-cause production issues fast** — filter and search traces by status, latency, cost, or custom tags to find the problem in seconds
* **Track costs and latency over time** — monitor token usage and spending across models and providers
* **Capture multi-turn conversations** — group related traces into threads to understand how interactions evolve across turns
* **Close the feedback loop** — attach human or automated scores to traces and use them to drive evaluations
## What you can capture
Full execution trees with inputs, outputs, timing, and metadata for every step
Multi-turn threads that group related traces into coherent sessions
Token usage and spending broken down by model, provider, and trace
Images, audio, video, and files logged alongside your traces
Qualitative and quantitative scores attached to individual traces
Visual execution graphs showing how your agent's steps connect
## How it works
Run `opik connect` from your agent's directory to pair it with Opik:
```bash
opik connect --project 
Use traces to debug failures, identify slow steps, and track quality over time. Attach feedback scores, run evaluations against datasets, and use [Ollie](/tracing/debug-agents) — Opik's AI assistant — to help root-cause issues automatically.
## Integrations
Opik has first-class support for 30+ frameworks in Python, TypeScript, and OpenTelemetry — so you can start capturing traces without changing how your application is built.
**[View all integrations →](/integrations/overview)**
## Next steps
* [Getting started](/tracing/getting-started) — Add observability to your agent in minutes
* [Concepts](/tracing/concepts) — Understand traces, spans, threads, and feedback scores
* [Debugging agents with Ollie](/tracing/debug-agents) — Use AI-assisted root-cause analysis
* [Cost tracking](/tracing/advanced/cost_tracking) — Monitor token usage and spending
# Getting started with Observability
Opik makes it easy to add observability to your existing LLM application. The fastest way is to let
your coding agent do it — install the Opik skill in Claude Code, Cursor, Codex, or any other
coding agent and it will instrument your code for you. If you'd rather stay inside Opik, use Opik
Connect to have [Ollie](/ollie) set up tracing from the dashboard. You can also add
tracing manually with the SDK.
## Adding observability to your code
The fastest way to add observability is to install the Opik skill in your coding agent and let
it instrument your code for you. The skill is compatible with Claude Code, Codex, Cursor,
OpenCode and any other agent that supports skills.
```bash
npx skills add comet-ml/opik-skills
```
Ask your coding agent to instrument your code:
```
Instrument my agent with Opik using the /instrument command.
```
The agent will read your code, pick the right Opik integration, and add tracing.
Opik Connect links your local repository to Opik so that [Ollie](/ollie), Opik's
built-in AI coding assistant, can inspect your code and add tracing from the dashboard — no
local agent setup required.
```bash
pip install opik
```
```bash
export OPIK_API_KEY="
You can use [Ollie](/ollie) to analyze your traces, identify issues in your agent's
behavior, and get actionable suggestions for improvement.
## Next steps
* [Concepts](/tracing/concepts) — Learn about traces, spans, threads, and feedback scores
* [Log traces](/tracing/advanced/log_traces) — In-depth guide on customizing what gets logged
* [Cost tracking](/tracing/advanced/cost_tracking) — Monitor token usage and spending
# Tracing Core Concepts
> Learn about the core concepts of Opik's tracing system, including traces, spans, and threads, and how they work together to provide comprehensive observability for your LLM applications.
Ready to start logging? Head to [Log traces](/tracing/advanced/log_traces) or [Log agents](/tracing/advanced/log_agent_graphs).
Opik's tracing system gives you full visibility into what your LLM application or agent is doing — every call, every step, every intermediate result. There are three building blocks you need to understand:
1. **Trace**: A complete execution path for a single interaction with an LLM or agent
2. **Span**: An individual operation or step within a trace
3. **Thread**: A collection of related traces that form a conversation or multi-turn workflow
## Traces
A **trace** represents a complete execution path for a single interaction with an LLM or agent. Think of it as a detailed record of everything that happened during one request-response cycle — inputs, outputs, timing, token usage, and any intermediate steps.
### Key characteristics
* **Unique identity**: Each trace has a unique identifier for tracking and referencing
* **Complete context**: All information needed to understand what happened during the interaction
* **Timing information**: When the interaction started, ended, and how long each part took
* **Input/output data**: The exact prompts sent to the LLM and the responses received
* **Metadata**: Model used, temperature settings, custom tags, and more
### Common uses
* **Debugging**: When an LLM produces unexpected output, examine the trace to understand what went wrong
* **Performance analysis**: Identify slow operations by analyzing trace timing
* **Cost tracking**: Monitor token usage and costs for each interaction
* **Quality assurance**: Review traces to ensure expected behavior
## Spans
A **span** represents an individual operation or step within a trace. While a trace shows the complete picture, spans break it down into measurable components. Spans are hierarchical — they can contain other spans, creating a tree structure within the trace.
### Key characteristics
* **Hierarchical structure**: Spans can contain child spans, forming a tree within a trace
* **Specific operations**: Each span represents a distinct action — an API call, a function, a data transformation
* **Precise timing**: Start and end times for each operation
* **Custom attributes**: Additional metadata specific to the operation
### Common span types
* **LLM Calls**: Individual requests to language models
* **Function Calls**: Tool or function invocations within an agent
* **Data Processing**: Transformations or manipulations of data
* **External API Calls**: Requests to third-party services
### Example span hierarchy
```
Trace: "Customer Support Chat"
├── Span: "Parse User Intent"
├── Span: "Query Knowledge Base"
│ ├── Span: "Search Vector Database"
│ └── Span: "Rank Results"
├── Span: "Generate Response"
│ ├── Span: "LLM Call: GPT-4"
│ └── Span: "Post-process Response"
└── Span: "Log Interaction"
```
## Threads
A **thread** is a collection of related traces that form a coherent conversation or multi-turn workflow. Threads are essential for chat applications and agents where context evolves across multiple interactions.
### Key characteristics
* **Conversation context**: Maintains the flow of multi-turn interactions
* **Trace grouping**: Organizes related traces under a single thread identifier
* **Chronological ordering**: Traces within a thread are ordered by time
* **Cross-trace analysis**: Enables analysis of patterns across related interactions
### When to use threads
* **Chat applications**: Group all messages in a conversation
* **Multi-step workflows**: Track complex processes that span multiple LLM calls
* **User sessions**: Organize all interactions from a single user session
* **Agent conversations**: Follow the complete interaction between an agent and a user
Threads are created by setting a `thread_id` on your traces:
```python
import opik
client = opik.Opik()
client.trace(
name="chat-turn-1",
thread_id="user-session-abc123",
input={"message": "Hello"},
output={"response": "Hi! How can I help?"},
)
```
## Best practices
A trace should represent a complete user interaction or business operation — not a single function call and not an entire session.
Descriptive span names make debugging much easier. Name spans after what they do: `search_vector_db`, `call_gpt4`, `rank_results`.
Assign a consistent `thread_id` to all traces from the same conversation or session. This is especially important for chat applications.
Include custom attributes that will be useful for analysis — user IDs, session context, model versions, experiment names.
Avoid logging personally identifiable information (PII) or sensitive business data in your traces. Use Opik's data filtering capabilities to protect sensitive information.
## Next steps
* [Log traces](/tracing/advanced/log_traces) — Capture traces in your application
* [Log agent graphs](/tracing/advanced/log_agent_graphs) — Trace agent-based applications with full span trees
* [Annotate traces](/tracing/advanced/annotate_traces) — Add scores and feedback to traces
* [Cost tracking](/tracing/advanced/cost_tracking) — Monitor token usage and costs
# Debugging agents with Ollie and Opik Connect
Your agent returned the wrong answer, ignored context it was given, or took twice as long as it
should. The trace is right there in Opik — but tracing alone doesn't tell you *why* it happened or
how to fix it. That's where [Ollie](/ollie) comes in.
## What Ollie has access to
Ollie is more than a chatbot — it has tools that let it act on your workspace and your code.
* **Read and analyze traces** — Ollie reads full span trees including inputs, outputs, latencies, token counts, and feedback scores. It can drill into individual spans, compare traces side by side, and search across your project for patterns.
* **Search your workspace** — Traces, threads, datasets, experiments, and prompts are all queryable. Ollie can aggregate data, find outliers, and surface trends you'd otherwise need to query manually.
* **Read and edit your code** — When you connect your repository with [`opik connect`](/development/agent-playground), Ollie gains secure, read-only access to your source files. It can propose edits that you review and approve before anything changes on disk.
* **Run your agent** — With `opik connect` active, Ollie can rerun your agent using inputs from a failing trace to verify a fix in real time. New traces stream back into Opik automatically.
* **Manage test suites** — Ollie can add traces as test cases to test suites, define assertions, trigger evaluation runs, and summarize pass/fail results.
* **Navigate the Opik UI** — Ollie can link you directly to traces, experiments, datasets, and prompts it references during a conversation.
Code access and agent execution require [`opik connect`](/development/agent-playground) to be running
in your project directory. Without it, Ollie can still analyze traces and search your workspace
but cannot read your source files or rerun your agent.
## The debug-fix-verify loop
The fastest way to improve agent quality is a tight loop: find a bad trace, understand it, fix it,
and make sure it stays fixed. Ollie handles this end-to-end.
Start in the Opik dashboard. Filter traces by error status, low feedback score, or latency
spike to find a run that didn't behave as expected.
Open Ollie from the trace view and describe what looks off. Ollie reads the full span tree —
every LLM call, tool invocation, and retrieval step — and identifies the root cause.
Once Ollie knows where the bug lives, it reads the relevant source file via `opik connect`
and proposes a change. You see the diff and approve it — nothing happens without your click.
Ask Ollie to add the original trace as a regression test case, then run the suite against
your updated agent. You get a pass/fail summary and a test that catches the bug if it ever
comes back.
Each cycle makes your agent more robust. Over time, your test suite becomes a comprehensive
regression guard built directly from real failures.
## Example prompts
Ollie works best when you describe the problem in plain language. Here are prompts for common
debugging scenarios:
### Investigating failures
* *"Why did the final answer ignore the retrieved context?"*
* *"Which span caused the latency spike in this trace?"*
* *"The tool call returned empty — what went wrong?"*
### Comparing traces
* *"Compare this failed trace to a recent successful one for the same query"*
* *"Find all traces where the tool call timed out this week"*
* *"What changed between the last successful run and this failure?"*
### Building test coverage
* *"Add this trace to my customer-support-qa suite with the assertion: the response must cite a specific step from the provided context"*
* *"Run the customer-support-qa suite against the updated prompt"*
* *"Why did 3 of the 5 items in this run fail?"*
### Understanding your workspace
* *"Show me the dataset for the last experiment"*
* *"What's the average latency for traces in this project over the past week?"*
* *"Which prompts are used by the most experiments?"*
## Next steps
* [Ollie overview](/ollie) — Full introduction to Ollie's capabilities and setup
* [Agent playground](/development/agent-playground) — How `opik connect` discovers and runs your agent
* [Evaluation overview](/evaluation/overview) — Build the regression net Ollie populates for you
# Log traces
If you are just getting started with Opik, we recommend first checking out the [Quickstart](/quickstart) guide that
will walk you through the process of logging your first LLM call.
LLM applications are complex systems that do more than just call an LLM API, they will often involve retrieval, pre-processing and post-processing steps.
Tracing is a tool that helps you understand the flow of your application and identify specific points in your application that may be causing issues.
Opik's tracing functionality allows you to track not just all the LLM calls made by your application but also any of the other steps involved.
Opik supports agent observability using our [Typescript SDK](/reference/typescript-sdk/overview),
[Python SDK](https://www.comet.com/docs/opik/python-sdk-reference/), [first class OpenTelemetry support](/integrations/opentelemetry)
and our [REST API](/reference/rest-api/overview).
We recommend starting with one of our integrations to get started quickly, you can find a full list of our
integrations in the [integrations overview](/integrations/overview) page.
We won't be covering how to track chat conversations in this guide, you can learn more about this in the
[Logging conversations](/tracing/advanced/log_chat_conversations) guide.
## Enable agent observability
### 1. Installing the SDK
Before adding observability to your application, you will first need to install and configure the
Opik SDK.
```bash
npm install opik
```
You can then set the Opik environment variables in your `.env` file:
```bash
# Set OPIK_API_KEY and OPIK_WORKSPACE in your .env file
OPIK_API_KEY=your_api_key_here
OPIK_WORKSPACE=your_workspace_name
# Optional if you are using Opik Cloud:
OPIK_URL_OVERRIDE=https://www.comet.com/opik/api
```
```bash
# Install the SDK
pip install opik
```
You can then configure the SDK using the `opik configure` CLI command or by calling
[`opik.configure`](https://www.comet.com/docs/opik/python-sdk-reference/configure.html) from
your Jupyter Notebook.
You will need to set the following environment variables for your OpenTelemetry setup:
```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=https://www.comet.com/opik/api/v1/private/otel
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=Integrate with Opik faster using this pre-built prompt
## Advanced usage
### Using function decorators
Function decorators are a great way to add Opik logging to your existing application. When you add
the `@track` decorator to a function, Opik will create a span for that function call and log the
input parameters and function output for that function. If we detect that a decorated function
is being called within another decorated function, we will create a nested span for the inner
function.
While decorators are most popular in Python, we also support them in our Typescript SDK:
TypeScript started supporting decorators from version 5 but it's use is still not widespread.
The Opik typescript SDK also supports decorators but it's currently considered experimental.
```typescript maxLines=100
import { track } from "opik";
class TranslationService {
@track({ type: "llm" })
async generateText() {
// Your LLM call here
return "Generated text";
}
@track({ name: "translate" })
async translate(text: string) {
// Your translation logic here
return `Translated: ${text}`;
}
@track({ name: "process", projectName: "translation-service" })
async process() {
const text = await this.generateText();
return this.translate(text);
}
}
```
You can also specify custom `tags`, `metadata`, and/or a `thread_id` for each trace and/or
span logged for the decorated function. For more information, see
[Logging additional data using the opik\_args parameter](#logging-additional-data)
You can add the `@track` decorator to any function in your application and track not just
LLM calls but also any other steps in your application:
```python maxLines=100
import opik
import openai
client = openai.OpenAI()
@opik.track
def retrieve_context(input_text):
# Your retrieval logic here, here we are just returning a
# hardcoded list of strings
context =[
"What specific information are you looking for?",
"How can I assist you with your interests today?",
"Are there any topics you'd like to explore?",
]
return context
@opik.track
def generate_response(input_text, context):
full_prompt = (
f" If the user asks a non-specific question, use the context to provide a relevant response.\n"
f"Context: {', '.join(context)}\n"
f"User: {input_text}\n"
f"AI:"
)
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": full_prompt}]
)
return response.choices[0].message.content
@opik.track(name="my_llm_application")
def llm_chain(input_text):
context = retrieve_context(input_text)
response = generate_response(input_text, context)
return response
# Use the LLM chain
result = llm_chain("Hello, how are you?")
print(result)
```
When using the track decorator, you can customize the data associated with both the trace
and the span using either the `opik_args` parameter or the
[`opik_context`](https://www.comet.com/docs/opik/python-sdk-reference/opik_context/index.html)
module. This is particularly useful if you want to specify the conversation thread id, tags
and metadata for example.
```python title="opik_context module"
import opik
@opik.track
def llm_chain(text: str) -> str:
opik_context.update_current_trace(
tags=["llm_chatbot"],
metadata={"version": "1.0", "method": "simple"},
thread_id="conversation-123",
feedback_scores=[
{
"name": "user_feedback",
"value": 1
}
],
)
opik_context.update_current_span(
metadata={"model": "gpt-4o"},
)
return f"Processed: {text}"
```
```python title="opik_args parameter"
import opik
@opik.track
def llm_chain(text: str) -> str:
# LLM chain code
# ...
return f"Processed: {text}"
# Call with opik_args - it won't be passed to the function
result = llm_chain(
"hello world",
opik_args={
"span": {
"tags": ["llm", "agent"],
"metadata": {"version": "1.0", "method": "simple"}
},
"trace": {
"thread_id": "conversation-123",
"tags": ["user-session"],
"metadata": {"user_id": "user-456"}
}
}
)
print(result)
```
If you specify the opik\_args parameter as part of your function call, you can propagate
the configuration to the nested functions.
### Using the low-level SDKs
If you need full control over the logging process, you can use the low-level SDKs to log your traces and spans:
You can use the [`Opik`](/reference/typescript-sdk/overview) client to log your traces and spans:
```typescript
import { Opik } from "opik";
const client = new Opik({
apiUrl: "https://www.comet.com/opik/api",
apiKey: "your-api-key", // Only required if you are using Opik Cloud
projectName: "your-project-name",
workspaceName: "your-workspace-name", // Optional
});
// Log a trace with an LLM span
const trace = client.trace({
name: `Trace`,
input: {
prompt: `Hello!`,
},
output: {
response: `Hello, world!`,
},
});
const span = trace.span({
name: `Span`,
type: "llm",
input: {
prompt: `Hello, world!`,
},
output: {
response: `Hello, world!`,
},
});
// Flush the client to send all traces and spans
await client.flush();
```
Make sure you define the environment variables for the Opik client in your `.env` file,
you can find more information about the configuration [here](/tracing/advanced/sdk_configuration).
If you want full control over the data logged to Opik, you can use the
[`Opik`](https://www.comet.com/docs/opik/python-sdk-reference/Opik.html) client.
Logging traces and spans can be achieved by first creating a trace using
[`Opik.trace`](https://www.comet.com/docs/opik/python-sdk-reference/Opik.html#opik.Opik.trace)
and then adding spans to the trace using the
[`Trace.span`](https://www.comet.com/docs/opik/python-sdk-reference/Objects/Trace.html#opik.api_objects.trace.Trace.span)
method:
```python
from opik import Opik
client = Opik(project_name="Opik client demo")
# Create a trace
trace = client.trace(
name="my_trace",
input={"user_question": "Hello, how are you?"},
output={"response": "Comment ça va?"}
)
# Add a span
trace.span(
name="Add prompt template",
input={"text": "Hello, how are you?", "prompt_template": "Translate the following text to French: {text}"},
output={"text": "Translate the following text to French: hello, how are you?"}
)
# Add an LLM call
trace.span(
name="llm_call",
type="llm",
input={"prompt": "Translate the following text to French: hello, how are you?"},
output={"response": "Comment ça va?"}
)
# End the trace
trace.end()
```
It is recommended to call `trace.end()` and `span.end()` when you are finished with the trace and span to ensure that
the end time is logged correctly.
Opik's logging functionality is designed with production environments in mind. To optimize
performance, all logging operations are executed in a background thread.
If you want to ensure all traces are logged to Opik before exiting your program, you can use the `opik.Opik.flush` method:
```python
from opik import Opik
client = Opik()
# Log some traces
client.flush()
```
### Logging traces/spans using context managers
If you are using the low-level SDKs, you can use the context managers to log traces and spans. Context managers provide a clean and Pythonic way to manage the lifecycle of traces and spans, ensuring proper cleanup and error handling.
Opik provides two main context managers for logging:
#### `opik.start_as_current_trace()`
Use this context manager to create and manage a trace. A trace represents the overall execution flow of your application.
For detailed API reference, see [`opik.start_as_current_trace`](https://www.comet.com/docs/opik/python-sdk-reference/context_manager/start_as_current_trace.html).
```python
import opik
# Basic trace creation
with opik.start_as_current_trace("my-trace", project_name="my-project") as trace:
# Your application logic here
trace.input = {"user_query": "What is the weather?"}
trace.output = {"response": "It's sunny today!"}
trace.tags = ["weather", "api-call"]
trace.metadata = {"model": "gpt-4", "temperature": 0.7}
```
**Parameters:**
* `name` (str): The name of the trace
* `input` (Dict\[str, Any], optional): Input data for the trace
* `output` (Dict\[str, Any], optional): Output data for the trace
* `tags` (List\[str], optional): Tags to categorize the trace
* `metadata` (Dict\[str, Any], optional): Additional metadata
* `project_name` (str, optional): Project name (falls back to active project context, then client configuration)
* `thread_id` (str, optional): Thread identifier for multi-threaded applications
* `flush` (bool, optional): Whether to flush data immediately (default: False)
#### `opik.start_as_current_span()`
Use this context manager to create and manage a span within a trace. Spans represent individual operations or function calls.
For detailed API reference, see [`opik.start_as_current_span`](https://www.comet.com/docs/opik/python-sdk-reference/context_manager/start_as_current_span.html).
```python
import opik
# Basic span creation
with opik.start_as_current_span("llm-call", type="llm", project_name="my-project") as span:
# Your LLM call here
span.input = {"prompt": "Explain quantum computing"}
span.output = {"response": "Quantum computing is..."}
span.model = "gpt-4"
span.provider = "openai"
span.usage = {
"prompt_tokens": 10,
"completion_tokens": 50,
"total_tokens": 60
}
```
**Parameters:**
* `name` (str): The name of the span
* `type` (SpanType, optional): Type of span ("general", "tool", "llm", "guardrail", etc.)
* `input` (Dict\[str, Any], optional): Input data for the span
* `output` (Dict\[str, Any], optional): Output data for the span
* `tags` (List\[str], optional): Tags to categorize the span
* `metadata` (Dict\[str, Any], optional): Additional metadata
* `project_name` (str, optional): Project name
* `model` (str, optional): Model name for LLM spans
* `provider` (str, optional): Provider name for LLM spans
* `flush` (bool, optional): Whether to flush data immediately
#### Nested Context Managers
You can nest spans within traces to create hierarchical structures:
```python
import opik
with opik.start_as_current_trace("chatbot-conversation", project_name="chatbot") as trace:
trace.input = {"user_message": "Help me with Python"}
# First span: Process user input
with opik.start_as_current_span("process-input", type="general") as span:
span.input = {"raw_input": "Help me with Python"}
span.output = {"processed_input": "Python programming help request"}
# Second span: Generate response
with opik.start_as_current_span("generate-response", type="llm") as span:
span.input = {"prompt": "Python programming help request"}
span.output = {"response": "I'd be happy to help with Python!"}
span.model = "gpt-4"
span.provider = "openai"
trace.output = {"final_response": "I'd be happy to help with Python!"}
```
#### Error Handling
Context managers automatically handle errors and ensure proper cleanup:
```python
import opik
try:
with opik.start_as_current_trace("risky-operation", project_name="my-project") as trace:
trace.input = {"data": "important data"}
# This will raise an exception
result = 1 / 0
trace.output = {"result": result}
except ZeroDivisionError:
# The trace is still properly closed and logged
print("Error occurred, but trace was logged")
```
#### Dynamic Parameter Updates
You can modify trace and span parameters both inside and outside the context manager:
```python
import opik
# Parameters set outside the context manager
with opik.start_as_current_trace(
"dynamic-trace",
input={"initial": "data"},
tags=["initial-tag"],
project_name="my-project"
) as trace:
# Override parameters inside the context manager
trace.input = {"updated": "data"}
trace.tags = ["updated-tag", "new-tag"]
trace.metadata = {"custom": "metadata"}
# The final trace will use the updated values
```
#### Flush Control
Control when data is sent to Opik:
```python
import opik
# Immediate flush
with opik.start_as_current_trace("immediate-trace", flush=True) as trace:
trace.input = {"data": "important"}
# Data is sent immediately when exiting the context
# Deferred flush (default)
with opik.start_as_current_trace("deferred-trace", flush=False) as trace:
trace.input = {"data": "less urgent"}
# Data will be sent asynchronously later or when the program exits
```
#### Best Practices
1. **Use descriptive names**: Choose clear, descriptive names for your traces and spans that explain what they represent.
2. **Set appropriate types**: Use the correct span types ("llm", "retrieval", "general", etc.) to help with filtering and analysis.
3. **Include relevant metadata**: Add metadata that will be useful for debugging and analysis, such as model names, parameters, and custom metrics.
4. **Handle errors gracefully**: Let the context manager handle cleanup, but ensure your application logic handles errors appropriately.
5. **Use project organization**: Organize your traces by project to keep your Opik dashboard clean and organized.
6. **Consider performance**: Use `flush=True` only when immediate data availability is required, as it can slow down your application by triggering a synchronous, immediate data upload.
### Logging to a specific project
By default, traces are logged to the `Default Project` project. You can change the project you want
the trace to be logged to in a couple of ways:
You can use the `OPIK_PROJECT_NAME` environment variable to set the project you want the trace
to be logged or pass a parameter to the `Opik` client.
```typescript
import { Opik } from "opik";
const client = new Opik({
projectName: "my_project",
// apiKey: "my_api_key",
// apiUrl: "https://www.comet.com/opik/api",
// workspaceName: "my_workspace",
});
```
You can use the `OPIK_PROJECT_NAME` environment variable to set the project you want traces
to be logged to.
If you are using function decorators, you can set the project as part of the decorator parameters:
```python
@track(project_name="my_project")
def my_function():
pass
```
If you are using the low level SDK, you can set the project as part of the `Opik` client constructor:
```python
from opik import Opik
client = Opik(project_name="my_project")
```
### Project name resolution (Python SDK)
The project name is determined differently depending on whether an active project context already exists.
#### When no project context is active
This applies to the **top-level** `@track`-decorated function call, the `Opik()` client, or a native integration (e.g., `track_openai`, `OpikTracer`) used outside any traced context. The project name is resolved in this order:
1. **Explicit `project_name` argument** — passed directly to `@track(project_name="...")`, `Opik(project_name="...")`, `OpikTracer(project_name="...")`, or a client method like `client.trace(project_name="...")`
2. **Client configuration** — from the `OPIK_PROJECT_NAME` environment variable or `~/.opik.config` file
3. **Default** — falls back to `"Default Project"` (a warning is logged once to remind you to configure a project name)
The first `@track(project_name="...")` or `opik.project_context("...")` call that runs establishes the **active project context** for all nested operations.
#### When a project context is active
Once a project context is established (by a parent `@track(project_name="...")` or `opik.project_context("...")`), **all nested operations use the context project name**. This includes:
* Nested `@track`-decorated functions — even if they pass a different `project_name`, the outer context wins (a warning is logged)
* Native integrations (e.g., `OpikTracer`, `track_openai`) — if initialized inside an active context, the context project overrides the integration's `project_name` argument (a warning is logged)
* `Opik()` client methods — if a method like `client.trace(project_name="...")` is called with an explicit `project_name`, the explicit argument wins; if `project_name` is omitted, the context project is used
This ensures that all traces and spans within a single execution flow are logged to the same project.
#### `@track` context propagation
When `@track(project_name="...")` is used on the top-level function, it sets the project context for the entire call tree:
```python
from opik import track
@track(project_name="my-agent")
def agent(query):
context = retrieve(query)
return generate(context)
@track
def retrieve(query):
# Inherits "my-agent" from the parent context
...
@track
def generate(context):
# Also inherits "my-agent" from the parent context
...
```
If a nested function specifies a different `project_name`, it is ignored and the outer project is preserved:
```python
@track(project_name="my-agent")
def agent(query):
helper(query) # Still logs to "my-agent", NOT "other-project"
@track(project_name="other-project")
def helper(query):
# Warning is logged: outer project "my-agent" will be used
...
```
#### `opik.project_context()`
The `opik.project_context()` context manager sets the project name for all Opik operations within a block — `@track`-decorated functions, native integrations, and `Opik()` client calls (when `project_name` is not passed explicitly):
```python
import opik
with opik.project_context("customer-support"):
# @track-decorated functions and native integrations
# all use "customer-support" as the project name
my_agent(query)
```
Nesting rules are the same: the first `project_context` or `@track(project_name=...)` to run owns the context. Inner calls with a different project name are ignored (a warning is logged).
When a script combines `@track` tracing with other Opik API calls — such as `evaluate()`, `get_or_create_dataset()`, or `Prompt()` — traces and API objects can land in different projects if the project name is not set consistently. Make sure the value passed to `opik.configure(project_name=...)` (which controls where `@track` traces go) matches the `project_name` argument passed explicitly to each API call:
```python
import opik
opik.configure(project_name="my-project")
dataset = client.get_or_create_dataset(name="my-dataset", project_name="my-project")
evaluation = evaluate(
dataset=dataset,
task=evaluation_task,
project_name="my-project", # must match opik.configure value above
...
)
```
### Logging to a specific environment
Environments let you tag traces with a lifecycle stage — for example `development`, `staging`, or `production` — so you can segment and filter your observability data in the Opik UI.
#### Setting the environment
The environment is resolved in this order:
1. **Explicit argument** — passed directly to `client.trace(environment: ...)`
2. **`OPIK_ENVIRONMENT` environment variable**
Using the low-level SDK:
```typescript
import { Opik } from "opik";
const client = new Opik({ projectName: "my-project" });
const trace = client.trace({
name: "my_trace",
input: { question: "Hello" },
environment: "production",
});
trace.end();
await client.flush();
```
#### Setting the environment
The environment is resolved in this order:
1. **Explicit argument** — passed directly to `@track(environment=...)` or `client.trace(environment=...)`
2. **`OPIK_ENVIRONMENT` environment variable**
Using the `@track` decorator:
```python
import opik
@opik.track(environment="production")
def my_pipeline(input_text: str) -> str:
return input_text
my_pipeline("Hello, world!")
```
Using the low-level SDK:
```python
from opik import Opik
client = Opik(project_name="my_project")
trace = client.trace(
name="my_trace",
input={"question": "Hello"},
environment="production",
)
trace.end()
```
You can also set the environment via the `OPIK_ENVIRONMENT` environment variable instead of passing it explicitly to each call.
#### Managing environments
You can manage the set of named environments in your workspace programmatically:
```typescript
import { Opik } from "opik";
const client = new Opik();
// Create a new environment
const env = await client.createEnvironment("production", {
description: "Live production traffic",
color: "#FF0000",
});
// List all environments
const envs = await client.getEnvironments();
// Update an environment
await client.updateEnvironment("production", { description: "Updated description" });
// Delete an environment
await client.deleteEnvironment("production");
```
```python
from opik import Opik
client = Opik()
# Create a new environment
env = client.create_environment(
name="production",
description="Live production traffic",
color="#FF0000",
)
# List all environments
envs = client.get_environments()
# Update an environment
client.update_environment("production", description="Updated description")
# Delete an environment
client.delete_environment("production")
```
#### Filtering by environment
Once traces are tagged, you can filter them programmatically using the `environment` field in `filter_string`. It supports `=`, `!=`, `in`, and `not_in`:
```python
from opik import Opik
client = Opik()
# Only production traces
traces = client.search_traces(
project_name="my_project",
filter_string='environment = "production"'
)
# Multiple environments
traces = client.search_traces(
project_name="my_project",
filter_string='environment in ("production", "staging")'
)
# Same filtering applies to spans
spans = client.search_spans(
project_name="my_project",
filter_string='environment = "production"'
)
# And to conversation threads
threads = client.search_threads(
project_name="my_project",
filter_string='environment = "production"'
)
# Combine with other thread filters
active_prod_threads = client.search_threads(
project_name="my_project",
filter_string='environment = "production" AND status = "active"'
)
```
### Flushing traces and spans
This process is optional and is only needed if you are running a short-lived script or if you are
debugging why traces and spans are not being logged to Opik.
As the Typescript SDK has been designed to be used in production environments, we batch traces
and spans and send them to Opik in the background.
If you are running a short-lived script, you can flush the traces to Opik by using the
`flush` method of the `Opik` client.
```typescript
import { Opik } from "opik";
const client = new Opik();
client.flush();
```
As the Python SDK has been designed to be used in production environments, we batch traces
and spans and send them to Opik in the background.
If you are running a short-lived script, you can flush the traces to Opik by using the
`flush` method of the `Opik` client.
```python maxLines=100
from opik import Opik
client = Opik()
client.flush()
```
You can also set the `flush` parameter to `True` when you are using the `@track` decorator to make sure
the traces are flushed to Opik before the program exits.
```python
from opik import track
@track(flush=True)
def llm_chain(input_text):
# LLM chain code
# ...
return f"Processed: {input_text}"
```
### Disabling the logging process
This is currently not supported in the Typescript SDK. To disable the logging process,
You can disable the logging process globally using the `OPIK_TRACK_DISABLE` environment variable.
If you are looking for more control, you can also use the `set_tracing_active` function to
dynamically disable the logging process.
```python
import opik
# Check the current state of the tracing flag
print(opik.is_tracing_active())
# Disable the logging process
opik.set_tracing_active(False)
# re-enable the logging process
print(opik.set_tracing_active(True))
```
## Next steps
Once you have the observability set up for your agent, you can go one step further and:
* [Logging chat conversations](/tracing/advanced/log_chat_conversations)
* [Logging user feedback](/tracing/advanced/annotate_traces)
* [Setup online evaluation metrics](/production/online-evaluation/rules)
# Log conversations
You can log chat conversations to the Opik platform and track the full conversations
your users are having with your chatbot. Threads allow you to group related traces together, creating a conversational flow that makes it easy to review multi-turn interactions and track user sessions.
## Understanding Threads
Threads in Opik are collections of traces that are grouped together using a unique `thread_id`. This is particularly useful for:
* **Multi-turn conversations**: Track complete chat sessions between users and AI assistants
* **User sessions**: Group all interactions from a single user session
* **Conversational agents**: Follow the flow of agent interactions and tool usage
* **Workflow tracking**: Monitor complex workflows that span multiple function calls
The `thread_id` is a user-defined identifier that must be unique per project. All traces with the same `thread_id` will be grouped together and displayed as a single conversation thread in the Opik UI.
## Logging conversations
You can log chat conversations by specifying the `thread_id` parameter when using either the low level SDK, Python decorators, or integration libraries:
```typescript title="Typescript SDK" language="typescript"
import { Opik } from "opik";
const client = new Opik({
apiUrl: "https://www.comet.com/opik/api", // Only required if you are using Opik Cloud
apiKey: "your-api-key",
projectName: "your-project-name",
workspaceName: "your-workspace-name", // Optional
});
const threadId = "your-thread-id"; // any unique string per conversation
// Option A: set on trace creation
const trace = client.trace({
name: "chat turn",
input: { user: "Hi there" },
output: { assistant: "Hello!" },
threadId
});
```
```python title="Python decorators" language="python"
import opik
from opik import opik_context
@opik.track
def chat_message(input):
return "Opik is an Open Source GenAI platform"
chat_message("What is Opik ?", opik_args={"trace": {"thread_id": "f174a"}})
# Alternatively, using the opik_context module
@opik.track
def chat_message(input):
thread_id = "f174a"
opik_context.update_current_trace(
thread_id=thread_id
)
return "Opik is an Open Source GenAI platform"
chat_message("What is Opik ?", thread_id)
```
```python title="Python SDK"
import opik
opik_client = opik.Opik()
thread_id = "55d84"
# Log a first message
trace = opik_client.trace(
name="chat_conversation",
input="What is Opik?",
output="Opik is an Open Source GenAI platform",
thread_id=thread_id
)
```
```python title="LangGraph integration"
# LangGraph automatically uses its thread_id as Opik thread_id
from langgraph.graph import StateGraph
from opik.integrations.langchain import OpikTracer
# Create your LangGraph workflow
graph = StateGraph(...)
compiled_graph = graph.compile()
# Configure with thread_id for conversation tracking
thread_id = "user-conversation-789"
config = {
"callbacks": [OpikTracer(project_name="langgraph-conversations")],
"configurable": {"thread_id": thread_id}
}
# First turn in conversation
result1 = compiled_graph.invoke(
{"messages": [{"role": "user", "content": "What is machine learning?"}]},
config=config
)
# Follow-up turn in same conversation thread
result2 = compiled_graph.invoke(
{"messages": [{"role": "user", "content": "Can you give me an example?"}]},
config=config
)
```
```python title="ADK integration"
# ADK automatically maps session_id to thread_id
from opik.integrations.adk import OpikTracer
from google.adk import sessions as adk_sessions, runners as adk_runners
# Create ADK session for conversation tracking
session_service = adk_sessions.InMemorySessionService()
session = session_service.create_session_sync(
app_name="my_chatbot",
user_id="user_123",
session_id="conversation_456" # This becomes the thread_id in Opik
)
opik_tracer = OpikTracer(project_name="adk-conversations")
runner = adk_runners.Runner(
agent=your_agent,
app_name="my_chatbot",
session_service=session_service
)
# First message - automatically grouped by session_id
result1 = runner.run(
user_id="user_123",
session_id="conversation_456",
new_message="What is machine learning?"
)
# Follow-up message - same conversation thread
result2 = runner.run(
user_id="user_123",
session_id="conversation_456",
new_message="Can you give me an example?"
)
```
```python title="OpenAI Agents"
# Using trace context manager with group_id for threading
import uuid
from opik import trace
thread_id = str(uuid.uuid4())
# First conversation turn
with trace(workflow_name="Agent Conversation", group_id=thread_id):
result1 = await Runner.run(agent, "What is machine learning?")
print(result1.final_output)
# Follow-up turn in same conversation
with trace(workflow_name="Agent Conversation", group_id=thread_id):
# Continue conversation with context
new_input = result1.to_input_list() + [
{"role": "user", "content": "Can you give me an example?"}
]
result2 = await Runner.run(agent, new_input)
print(result2.final_output)
```
The input to each trace will be displayed as the user message while the output will be displayed as the AI assistant
response.
## Thread ID Best Practices
### Generating Thread IDs
Choose a thread ID strategy that fits your application:
```python title="User session based"
import uuid
import opik
# Generate unique thread ID per user session
user_id = "user_12345"
session_start_time = "2024-01-15T10:30:00Z"
thread_id = f"{user_id}-{session_start_time}"
@opik.track
def process_user_message(message, user_id):
return "Response to: " + message
process_user_message("What is Opik ?", opik_args={"trace": {"thread_id": thread_id}})
```
```python title="UUID based"
import uuid
import opik
# Generate random UUID for each conversation
thread_id = str(uuid.uuid4()) # e.g., "f47ac10b-58cc-4372-a567-0e02b2c3d479"
@opik.track
def start_conversation(initial_message):
return f"Processing: {initial_message}"
start_conversation("What is Opik ?", opik_args={"trace": {"thread_id": thread_id}})
```
```python title="Timestamp based"
import time
import opik
# Use timestamp for time-based grouping
thread_id = f"conversation-{int(time.time())}"
@opik.track
def handle_conversation_turn(message):
return f"Response to: {message}"
handle_conversation_turn("What is Opik ?", opik_args={"trace": {"thread_id": thread_id}})
```
### Integration-Specific Threading
Different integrations handle thread IDs in various ways:
```python title="LangChain"
from opik.integrations.langchain import OpikTracer
# Set thread_id at tracer level - applies to all traces
opik_tracer = OpikTracer(
project_name="my-chatbot",
thread_id="conversation-123"
)
# Or pass dynamically via metadata
chain.invoke(
{"input": "Hello"},
config={
"callbacks": [opik_tracer],
"metadata": {"thread_id": "dynamic-conversation-456"}
}
)
```
```python title="LangGraph"
from opik.integrations.langchain import OpikTracer
# LangGraph automatically uses its thread_id as Opik thread_id
thread_id = "langgraph-conversation-789"
config = {
"callbacks": [OpikTracer()],
"configurable": {"thread_id": thread_id}
}
result = compiled_graph.invoke(input_data, config=config)
```
```python title="OpenAI Agents"
import uuid
from opik import trace
# Use trace context manager with group_id for threading
thread_id = str(uuid.uuid4())
with trace(workflow_name="Agent Conversation", group_id=thread_id):
# All agent interactions within this context share the thread_id
result1 = await Runner.run(agent, "First question")
result2 = await Runner.run(agent, "Follow-up question")
```
```python title="GenAI"
from google import genai
from opik.integrations.genai import track_genai
client = genai.Client()
gemini_client = track_genai(client, project_name="opik_args demo")
response = gemini_client.models.generate_content(
model="gemini-2.0-flash",
contents="What is Opik?",
opik_args={"trace": {"thread_id": "f174a"}}
)
```
```python title="OpenAI"
import openai
from opik.integrations.openai import track_openai
client = openai.OpenAI()
wrapped_client = track_openai(
openai_client=client,
project_name="opik_args demo",
)
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is Opik?"},
]
_ = wrapped_client.responses.create(
model="gpt-4o-mini",
input=messages,
opik_args={"trace": {"thread_id": "f174a"}}
)
```
## Reviewing conversations
Conversations can be viewed at a project level in the `threads` tab. All conversations are tracked and by clicking on the thread ID you will be able to
view the full conversation.
The thread view supports markdown making it easier for you to review the content that was returned to the user. If you would like to dig in deeper, you
can click on the `View trace` button to deepdive into how the AI assistant response was generated.
By clicking on the thumbs up or thumbs down icons, you can quickly rate the AI assistant response. This feedback score will be logged and associated to
the relevant trace. By switching to the trace view, you can review the full trace as well as add additional feedback scores through the annotation
functionality.
## Scoring conversations
You can assign conversation-level feedback scores to threads at any time. Threads are aggregated traces
that are created when tracking agents or simply traces interconnected by a `thread_id`.
In the conversation list, you can see the feedback scores associated to each thread.
You can also tag a thread and add comments to it. This is useful to add additional context during the review process or investigate a specific conversation.
### Thread Online Scoring Rule Cooldown Period
For thread-level online evaluation rules (automatic scoring), Opik waits for a "cooldown period" after the last activity
in a thread before running the rules. This gives conversations time to settle before automatic evaluation.
By default, the cooldown period is 15 minutes. You can change this value by setting the
`OPIK_TRACE_THREAD_TIMEOUT_TO_MARK_AS_INACTIVE` environment variable (if you are using the Opik self-hosted version).
On cloud, you can change this setting at workspace level under "Thread online scoring rule cooldown period".
#### Behavior When Adding Traces to Existing Threads
When a new trace is added to an existing thread, the following happens:
* **Existing feedback scores are preserved**: Any manual feedback scores or online evaluation scores you have added remain intact.
* **The cooldown timer restarts**: The timer resets from the moment the new trace is added, ensuring online evaluation waits for the full cooldown period before scoring the updated thread.
* **Online evaluation re-runs**: Once the cooldown period expires, thread-level online scoring rules will automatically evaluate the complete conversation again. If a new score is logged with the same name as an existing score, the existing score is updated.
## Advanced Thread Features
### Filtering and Searching Threads
You can filter threads using the `thread_id` field in various Opik features:
#### In Data Export
When exporting data, you can filter by `thread_id` using these operators:
* `=` (equals), `!=` (not equals)
* `contains`, `not_contains`
* `starts_with`, `ends_with`
* `>`, `<` (lexicographic comparison)
#### In Thread Evaluation
You can evaluate entire conversation threads using the thread evaluation features. This is particularly useful for:
* Conversation quality assessment
* Multi-turn coherence evaluation
* User satisfaction scoring across complete interactions
### Thread Management
Threads can have traces added to them at any time, and you can add feedback scores, comments, and tags
to threads regardless of whether new traces are still being added.
### Programmatic Thread Management
You can also manage threads programmatically using the Opik SDK:
```python title="Python" language="python"
import opik
# Initialize client
client = opik.Opik()
# Search for threads by various criteria
threads = client.search_traces(
project_name="my-chatbot",
filter_string='thread_id contains "user-session"'
)
# Get specific thread content
for trace in threads:
if trace.thread_id:
thread_content = client.get_trace_content(trace.id)
print(f"Thread: {trace.thread_id}")
print(f"Input: {thread_content.input}")
print(f"Output: {thread_content.output}")
# Add feedback scores to thread traces
for trace in threads:
trace.log_feedback_score(
name="conversation_quality",
value=0.8,
reason="Good multi-turn conversation flow"
)
```
## Next steps
Once you have added observability to your multi-turn agent, why not:
1. [Run offline multi-turn conversation evaluation](/evaluation/evaluate_threads)
2. [Create online evaluation rules](/production/online-evaluation/rules) to score your multi-turn conversations in
production
# Log media & attachments
Opik supports multimodal traces allowing you to track not just the text input
and output of your LLM, but also images, videos and audio and any other media.
## Logging Attachments
In the Python SDK, you can use the `Attachment` type to add files to your traces.
Attachements can be images, videos, audio files or any other file that you might
want to log to Opik.
Each attachment is made up of the following fields:
* `data`: The path to the file, raw bytes, or a base64 encoded string of the file
* `file_name`: Optional name for the attachment (required when using raw bytes without a file path)
* `content_type`: The content type of the file formatted as a MIME type
These attachements can then be logged to your traces and spans using The
`opik_context.update_current_span` and `opik_context.update_current_trace`
methods:
### Using file paths
The most common way to log attachments is by providing a file path:
```python wordWrap
from opik import opik_context, track, Attachment
@track
def my_llm_agent(input):
# LLM chain code
# ...
# Update the trace with a file path
opik_context.update_current_trace(
attachments=[
Attachment(
data="
In order to preview the attachements in the UI, you will need to supply a
supported content type for the attachment. We support the following content types:
* Image: `image/jpeg`, `image/png`, `image/gif` and `image/svg+xml`
* Video: `video/mp4` and `video/webm`
* Audio: `audio/wav`, `audio/vorbis` and `audio/x-wav`
* Text: `text/plain` and `text/markdown`
* PDF: `application/pdf`
* Other: `application/json` and `application/octet-stream`
## Managing Attachments Programmatically
You can also manage attachments programmatically using the [`AttachmentClient`](https://www.comet.com/docs/opik/python-sdk-reference/Objects/AttachmentClient.html):
```python wordWrap
import opik
opik_client = opik.Opik()
attachment_client = opik_client.get_attachment_client()
# Get list of attachments
attachments_details = attachment_client.get_attachment_list(
project_name="my-project",
entity_id="some-trace-uuid-7",
entity_type="trace"
)
# Download an attachment
attachment_data = attachment_client.download_attachment(
project_name="my-project",
entity_type="trace",
entity_id="some-trace-uuid-7",
file_name="report.pdf",
mime_type="application/pdf"
)
# Upload a new attachment
attachment_client.upload_attachment(
project_name="my-project",
entity_type="trace",
entity_id="some-trace-uuid-7",
file_path="/path/to/document.pdf"
)
```
## Previewing base64 encoded images and image URLs
Opik automatically detects base64 encoded images and URLs logged to the platform,
once an image is detected we will hide the string to make the content more readable
and display the image in the UI. This is supported in the tracing view, datasets
view and experiment view.
For example if you are using the OpenAI SDK, if you pass an image to the model
as a URL, Opik will automatically detect it and display
the image in the UI:
```python wordWrap
from opik.integrations.openai import track_openai
from openai import OpenAI
# Make sure to wrap the OpenAI client to enable Opik tracing
client = track_openai(OpenAI())
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{
"type": "image_url",
"image_url": {
"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
},
},
],
}
],
max_tokens=300,
)
print(response.choices[0])
```
## Embedded Attachments
When you embed base64-encoded media directly in your trace/span `input`, `output`, or `metadata` fields, Opik automatically optimizes storage and retrieval for performance.
### How It Works
For base64-encoded content larger than 250KB, Opik automatically extracts and stores it separately. This happens transparently - you don't need to change your code.
When you retrieve your traces or spans later, the attachments are automatically included by default. For faster queries when you don't need the attachment data, use the `strip_attachments=true` parameter.
### Size Limits
Opik Cloud supports embedded attachments up to **100MB per field**. This limit applies to individual string values in your `input`, `output`, or `metadata` fields.
Base64 encoding increases file size by about 33%. For example, a 75MB video becomes \~100MB when base64-encoded.
If you need to work with larger files:
1. **Use the Attachment API** - Upload files separately using `AttachmentClient` (recommended for files >50MB). See [Managing Attachments Programmatically](#managing-attachments-programmatically)
2. **Contact us** - [Get in touch](https://www.comet.com/site/about-us/contact-us/) if you need higher limits
3. **Self-host Opik** - Configure your own limits. See the [Self-hosting Guide](/self-host/overview)
### Best Practices
* Embed smaller files directly - Opik handles them efficiently
* For files >50MB, use the Attachment API for better performance
* Use `strip_attachments=true` when querying if you don't need the attachment data
## Downloading attachments
You can download attachments in two ways:
1. **From the UI**: Hover over the attachments and click on the download icon
2. **Programmatically**: Use the `AttachmentClient` as shown in the examples above
Let's us know on [Github](https://github.com/comet-ml/opik/issues/new/choose) if you would like to us to support
additional image formats.
# Log Agent Graphs
Agent Graphs are a great way to visualize the flow of an agent and simplifies it's debugging.
Opik supports logging agent graphs for the following frameworks:
1. LangGraph
2. Google Agent Development Kit (ADK)
3. Manual Tracking
## LangGraph
You can log the agent execution graph by specifying the `graph` parameter in the
[OpikTracer](https://www.comet.com/docs/opik/python-sdk-reference/integrations/langchain/OpikTracer.html) callback:
```python
from opik.integrations.langchain import OpikTracer
opik_tracer = OpikTracer(graph=app.get_graph(xray=True))
```
Opik will log the agent graph definition in the Opik dashboard which you can access by clicking on
`Show Agent Graph` in the trace sidebar.
## Google Agent Development Kit (ADK)
Opik automatically generates visual representations of your agent workflows for Google ADK without requiring any additional configuration. Simply integrate Opik's OpikTracer callback as shown in the [ADK integration configuration guide](https://www.comet.com/docs/opik/integrations/adk#configuring-google-adk), and your agent graphs will be automatically captured and visualized.
The graph automatically shows:
* Agent hierarchy and relationships
* Sequential execution flows
* Parallel processing branches
* Tool connections and dependencies
* Loop structures and iterations
For example, a basic weather and time agent will display its execution flow with all agent steps, LLM calls, and tool invocations:
For more complex multi-agent architectures, the automatic graph visualization becomes even more valuable, providing clear visibility into nested agent hierarchies and complex execution patterns.
## Manual Tracking
You can also log the agent graph definition manually by logging the agent graph definition as a
mermaid graph definition in the metadata of the trace:
```python
import opik
from opik import opik_context
@opik.track
def chat_agent(input: str):
# Update the current trace with the agent graph definition
opik_context.update_current_trace(
metadata={
"_opik_graph_definition": {
"format": "mermaid",
"data": "graph TD; U[User]-->A[Agent]; A-->L[LLM]; L-->A; A-->R[Answer];"
}
}
)
return "Hello, how can I help you today?"
chat_agent("Hi there!")
```
Opik will log the agent graph definition in the Opik dashboard which you can access by clicking on
`Show Agent Graph` in the trace sidebar.
## Next steps
Why not check out:
* [Opik's 50+ integrations](/integrations/overview)
* [Logging traces](/tracing/advanced/log_traces)
* [Evaluating agents](/evaluation/evaluate_agents)
# Log distributed traces
When working with complex LLM applications, it is common to need to track a traces across multiple services. Opik supports distributed tracing out of the box when integrating using function decorators using a mechanism that is similar to how OpenTelemetry implements distributed tracing.
For the purposes of this guide, we will assume that you have a simple LLM application that is made up of two services: a client and a server. We will assume that the client will create the trace and span, while the server will add a nested span. In order to do this, the `trace_id` and `span_id` will be passed in the headers of the request from the client to the server.
The Python SDK includes some helper functions to make it easier to fetch headers in the client and ingest them in the server:
```python title="client.py"
from opik import track, opik_context
@track()
def my_client_function(prompt: str) -> str:
headers = {}
# Update the headers to include Opik Trace ID and Span ID
headers.update(opik_context.get_distributed_trace_headers())
# Make call to backend service
response = requests.post("http://.../generate_response", headers=headers, json={"prompt": prompt})
return response.json()
```
On the server side, you can pass the headers to your decorated function:
```python title="server.py"
from opik import track
from fastapi import FastAPI, Request
@track()
def my_llm_application():
pass
app = FastAPI() # Or Flask, Django, or any other framework
@app.post("/generate_response")
def generate_llm_response(request: Request) -> str:
return my_llm_application(opik_distributed_trace_headers=request.headers)
```
The `opik_distributed_trace_headers` parameter is added by the `track` decorator to each function that is decorated
and is a dictionary with the keys `opik_trace_id` and `opik_parent_span_id`.
## Using the distributed\_headers Context Manager
As an alternative to passing `opik_distributed_trace_headers` as a parameter, you can use the `distributed_headers()` context manager for more explicit control over distributed header handling. This approach provides automatic cleanup, error handling, and optional data flushing.
```python title="server.py"
from opik import track
from opik.decorator.context_manager import distributed_headers
from fastapi import FastAPI, Request
@track()
def my_llm_application():
pass
app = FastAPI() # Or Flask, Django, or any other framework
@app.post("/generate_response")
def generate_llm_response(request: Request) -> str:
# Extract distributed headers from the request
headers = {
"opik_trace_id": request.headers.get("opik_trace_id"),
"opik_parent_span_id": request.headers.get("opik_parent_span_id"),
}
# Use the context manager to handle distributed headers
with distributed_headers(headers, flush=False):
result = my_llm_application()
return result
```
The `distributed_headers()` context manager accepts two parameters:
* `headers`: A dictionary containing the distributed trace headers (`opik_trace_id` and `opik_parent_span_id`)
* `flush` (optional): Whether to flush the Opik client data after the root span is processed. Defaults to `False`. Set to `True` if you want to ensure immediate data transmission.
The context manager automatically creates a root span with the provided headers, handles any errors that occur during execution, and cleans up the context when complete.
For more details and additional examples, see the [distributed\_headers context manager API reference](https://www.comet.com/docs/opik/python-sdk-reference/context_manager/distributed_headers.html).
## Distributed Traces with a Remote Service Using OpenTelemetry
When the downstream service is instrumented with the standard OpenTelemetry SDK (rather than the Opik SDK), Opik provides helpers to bridge the two systems so the OTel span produced by the remote service still appears under the correct Opik trace and parent span.
The bridge works through two HTTP headers carried from the client to the remote service:
* `opik_trace_id` — the Opik trace the OTel span should be attached to.
* `opik_parent_span_id` — the Opik span to use as the parent (optional).
On the receiving side, the helper translates these headers into two OpenTelemetry span attributes (`opik.trace_id`, `opik.parent_span_id`) recognized by the Opik OTLP ingest endpoint. Both values must be valid UUIDs; blank or malformed values are dropped with a warning so a misconfigured caller never silently corrupts the parent linkage.
### Client: emitting distributed-trace headers
```python title="client.py"
import requests
from opik import opik_context, track
@track()
def my_client_function(prompt: str) -> str:
headers = {
# Adds 'opik_trace_id' and 'opik_parent_span_id'
**opik_context.get_distributed_trace_headers(),
}
response = requests.post(
"http://.../generate_response",
headers=headers,
json={"prompt": prompt},
)
return response.json()
```
```ts title="client.ts"
import { getDistributedTraceHeaders, track } from "opik";
const myClientFunction = track(
{ name: "client" },
async (prompt: string) => {
const response = await fetch("http://.../generate_response", {
method: "POST",
headers: {
"Content-Type": "application/json",
// Adds 'opik_trace_id' and 'opik_parent_span_id'.
// Returns null outside of a track() context.
...(getDistributedTraceHeaders() ?? {}),
},
body: JSON.stringify({ prompt }),
});
return response.json();
}
);
```
### Remote service: attaching the headers to an OpenTelemetry span
The remote service creates a span with the OpenTelemetry SDK as usual and then calls the Opik bridging helper with the incoming HTTP headers. The helper sets the `opik.trace_id` / `opik.parent_span_id` / `opik.span_id` attributes on the *boundary* span only.
To make sure descendant OpenTelemetry spans (children created inside the boundary span via `start_as_current_span` / `tracer.startSpan`) also land under the original Opik trace and parent, register the `OpikSpanProcessor` on the same `TracerProvider` as your OTLP exporter. Without it, only the boundary span is linked and its descendants are orphaned in a synthetic Opik trace.
In Python, `OpikSpanProcessor` ships with the main `opik` package under `opik.integrations.otel`. In TypeScript it lives in a separate `opik-otel` package — install it alongside `opik` (`npm install opik-otel @opentelemetry/api @opentelemetry/sdk-trace-base`).
```python title="server.py"
from fastapi import FastAPI, Request
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opik.integrations.otel import OpikSpanProcessor, distributed_trace
# Configure the tracer provider with the OTLP exporter that ships spans to Opik
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
# Register OpikSpanProcessor so descendants of the boundary span inherit
# opik.trace_id / opik.parent_span_id automatically.
provider.add_span_processor(OpikSpanProcessor())
trace.set_tracer_provider(provider)
app = FastAPI()
tracer = trace.get_tracer("my-service")
@app.post("/generate_response")
def generate_response(request: Request) -> str:
with tracer.start_as_current_span("server-span") as span:
# Reads opik_trace_id / opik_parent_span_id from the request headers
# and sets the corresponding OTel span attributes on the boundary span.
distributed_trace.attach_to_parent(span, dict(request.headers))
# Any descendants are picked up automatically by OpikSpanProcessor.
with tracer.start_as_current_span("child-span"):
# ... handle the request, set additional span attributes ...
pass
return "ok"
```
```ts title="server.ts"
import http from "node:http";
import { context, trace } from "@opentelemetry/api";
import {
BasicTracerProvider,
BatchSpanProcessor,
} from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { attachToParent, OpikSpanProcessor } from "opik-otel";
// Configure the tracer provider with the OTLP exporter that ships spans to Opik.
// OpikSpanProcessor must be registered before the exporter so the opik.* attributes
// it sets at span start are visible at export time.
const provider = new BasicTracerProvider({
spanProcessors: [
new OpikSpanProcessor(),
new BatchSpanProcessor(new OTLPTraceExporter()),
],
});
provider.register();
const tracer = trace.getTracer("my-service");
const server = http.createServer((req, res) => {
const boundary = tracer.startSpan("server-span");
// Reads opik_trace_id / opik_parent_span_id from req.headers
// and sets the corresponding OTel span attributes on the boundary span.
attachToParent(boundary, req.headers);
// Set the boundary as the active parent so child spans inherit its OTel
// context — this is what lets OpikSpanProcessor see the boundary's
// opik.trace_id / opik.span_id when the child starts. Plain
// `tracer.startSpan("child")` would otherwise create an orphan root span.
const parentCtx = trace.setSpan(context.active(), boundary);
const child = tracer.startSpan("child-span", {}, parentCtx);
// ... handle the request, set additional span attributes ...
child.end();
boundary.end();
res.end("ok");
});
```
`OpikSpanProcessor` only mutates spans whose parent already carries the Opik attributes (set by `attach_to_parent` / `attachToParent` on the boundary, or inherited from upstream W3C `baggage`). Spans outside an attached subtree are left untouched, so today's behaviour for unrelated OTel traces is unchanged.
The remote service must be configured with an OTLP exporter pointing at the Opik backend (`/v1/private/otel/v1/traces`). See the [OpenTelemetry Python SDK](/integrations/opentelemetry-python-sdk) integration guide for a full exporter configuration example; the same endpoint is used by the OpenTelemetry JS/Node SDK.
# Log user feedback
Logging user feedback and scoring traces is a crucial aspect of evaluating and improving your agent.
By systematically recording qualitative or quantitative feedback on specific interactions or entire
conversation flows, you can:
1. Track performance over time
2. Identify areas for improvement
3. Compare different model versions or prompts
4. Gather data for fine-tuning or retraining
5. Provide stakeholders with concrete metrics on system effectiveness
## Logging user feedback using the SDK
You can use the SDKs to log user feedback and score traces:
```typescript title="Typescript SDK" language="typescript"
import { Opik } from "opik";
const client = new Opik();
// Create a new trace with a span
const trace = client.trace({
name: "my_trace",
input: { input: "Hi!" },
output: { output: "Hello!" },
});
const span = trace.span({
name: "processing",
input: { step: 1 },
});
span.update({ output: { result: "processed" } });
span.end();
trace.end();
// Log feedback scores to existing traces
client.logTracesFeedbackScores([
{ id: trace.data.id, name: "overall_quality", value: 0.9, reason: "Good answer" },
{ id: trace.data.id, name: "coherence", value: 0.8 }
]);
// Log feedback scores to existing spans
client.logSpansFeedbackScores([
{ id: span.data.id, name: "accuracy", value: 0.95 }
]);
// Flush to ensure all data is sent
await client.flush();
```
```python title="Python Function Decorator" language="python"
import opik
from opik import opik_context
@opik.track
def my_function():
opik_context.update_current_trace(
feedback_scores=[
{
"name": "user_feedback",
"value": 1,
"reason": "Good answer" # Optional
}
]
)
return "Hello, world!"
```
```python title="Python SDK" language="python"
import opik
client = opik.Opik()
# Log feedback scores to an existing trace
client.log_traces_feedback_scores(
scores=[
{"id": "trace_id", "name": "user_feedback", "value": 1, "project_name": "my-project"},
{"id": "trace_id", "name": "accuracy", "value": 1, "reason": "Good answer", "project_name": "my-project"} # Optional reason score
]
)
# Log feedback score to a new trace
client.trace(
name="my_trace",
input={"input": "Hi!"},
output={"output": "Hello!"},
feedback_scores=[
{"name": "user_feedback", "value": 1, "reason": "Good answer"}
]
)
```
## Annotating Traces through the UI
To annotate traces through the UI, you can navigate to the trace you want to annotate in the traces
page and click on the `Annotate` button. This will open a sidebar where you can add annotations to
the trace.
You can annotate both traces and spans through the UI, make sure you have selected the correct span
in the sidebar.
Once a feedback scores has been provided, you can also add a reason to explain why this particular
score was provided. This is useful to add additional context to the score.
If multiple team members are annotating the same trace, you can see the annotations of each team
member in the UI in the `Feedback scores` section. The average score will be displayed at a trace
and trace level.
If you want a more dedicated annotation interface, you can use the [Annotation Queues](/evaluation/advanced/annotation_queues)
feature.
## Online evaluation
You don't need to manually annotate each trace to measure the performance of your agents! By using
Opik's [online evaluation feature](/production/online-evaluation/rules), you can define LLM as a Judge metrics that
will automatically score all, or a subset, of your production traces.
## Manual evaluation
While online evaluation automatically scores traces based on sampling rates and enabled rules, manual evaluation gives you complete control over which traces or threads get evaluated and when. This is particularly useful when you want to:
* Evaluate specific traces or threads that failed or require closer inspection
* Apply evaluation rules to historical data that wasn't captured by sampling
* Test new evaluation rules on selected examples before enabling them for automatic scoring
* Re-evaluate traces with updated or modified rules
### How manual evaluation works
Manual evaluation allows you to apply any existing evaluation rule to selected traces or threads directly from the UI, bypassing sampling rates and rule enablement status. You can trigger manual evaluation from:
1. **Traces page**: Select one or more traces and click "Evaluate" to apply trace-level rules
2. **Threads page**: Select one or more threads and click "Evaluate" to apply thread-level rules
**Important**: Trace-level rules can only be applied to traces, and thread-level rules can only be applied to threads. Make sure you're using the appropriate rule type for your selected entities.
When you trigger manual evaluation:
* All selected traces/threads will be queued for evaluation, regardless of sampling rate
* You can apply multiple rules at once
* Rules will execute even if they are currently disabled
* Evaluation results will appear as feedback scores on the evaluated traces/threads
* The evaluation is processed asynchronously, so you may need to wait a few seconds or refresh the page to see the results
This gives you the flexibility to evaluate exactly what you need, when you need it, without waiting for the next sampled trace or modifying your online evaluation configuration.
## Next steps
You can go one step further and:
1. [Score your agent in production](/production/online-evaluation/rules) to track and catch specific issues with your
agent
2. [Use annotation queues](/evaluation/advanced/annotation_queues) to organize your traces for review and
labeling by your team of experts
3. [Checkout our LLM as a Judge metrics](/evaluation/metrics/overview)
# Cost tracking
Opik has been designed to track and monitor costs for your LLM applications by measuring token usage across all traces. Using the Opik dashboard, you can analyze spending patterns and quickly identify cost anomalies. All costs across Opik are estimated and displayed in USD.
## Monitoring Costs in the Dashboard
You can use the Opik dashboard to review costs at three levels: spans, traces, and projects. Each level provides different insights into your application's cost structure.
### Span-Level Costs
Individual spans show the computed costs (in USD) for each LLM spans of your traces:
### Trace-Level Costs
If you are using one of Opik's integrations, we automatically aggregates costs from all spans within a trace to compute total trace costs:
### Project-Level Analytics
Track your overall project costs in:
1. The main project view, through the Estimated Cost column:
2. The project Metrics tab, which shows cost trends over time:
## Retrieving Costs Programmatically
You can retrieve the estimated cost programmatically for both spans and traces. Note that the cost will be `None` if the span or trace used an unsupported model.
### Retrieving Span Costs
```python
import opik
client = opik.Opik()
span = client.get_span_content("
The UI exports up to 100 traces or spans at a time. For larger exports use the SDK or CLI.
## Command-line tools
The `opik export` and `opik import` commands let you export traces, spans, datasets, prompts, and experiments to local JSON or CSV files, and import them back — useful for migrations, backups, and cross-environment syncs.
### Export
```bash
opik export WORKSPACE TYPE NAME [OPTIONS]
```
`TYPE` is one of: `all`, `dataset`, `project`, `experiment`, `prompt`
```bash
# Export everything in a workspace
opik export my-workspace all
# Export a specific project
opik export my-workspace project "my-project"
# Export a specific dataset
opik export my-workspace dataset "my-test-dataset"
# Export with a date filter
opik export my-workspace project "my-project" \
--filter 'created_at >= "2024-01-01T00:00:00Z"'
# Export as CSV for analysis
opik export my-workspace project "my-project" --format csv --path ./csv_data
```
### Import
```bash
opik import WORKSPACE TYPE NAME [OPTIONS]
```
```bash
# Import a dataset
opik import my-workspace dataset "my-dataset"
# Import a project
opik import my-workspace project "my-project"
# Preview what would be imported
opik import my-workspace project "my-project" --dry-run
```
Imports are automatically resumable — if interrupted, re-run the same command and it picks up where it left off using a local `migration_manifest.db`.
### Migrating between environments
```bash
# Step 1: Export from source (use source credentials)
OPIK_API_KEY=
Reviewing the health check output can help pinpoint the source of the problem and suggest possible resolutions.
### TypeScript SDK Troubleshooting
#### Configuration Validation Errors
The TypeScript SDK validates configuration at startup. Common errors:
* **"OPIK\_URL\_OVERRIDE is not set"**: Set the `OPIK_URL_OVERRIDE` environment variable
* **"OPIK\_API\_KEY is not set"**: Required for Opik Cloud deployments
* **"OPIK\_WORKSPACE is not set"**: Optional, but can be set for Opik Cloud deployments
#### Debug Logging
Enable debug logging to troubleshoot issues:
```bash
export OPIK_LOG_LEVEL="DEBUG"
```
If you are using the Opik Optimizer SDK, you can also enable optimizer-side debug logs:
```bash
export OPIK_OPTIMIZER_LOG_LEVEL="DEBUG"
```
Or programmatically:
```typescript
import { setLoggerLevel } from "opik";
setLoggerLevel("DEBUG");
```
#### Batch Queue Issues
If data isn't appearing in Opik:
1. **Check if data is batched**: Call `await client.flush()` to force sending
2. **Verify configuration**: Ensure correct API URL and credentials
3. **Check network connectivity**: Verify firewall and proxy settings
### General Troubleshooting
#### Environment Variables Not Loading
1. **Python**: Ensure `load_dotenv()` is called before importing `opik`
2. **TypeScript**: The SDK automatically loads `.env` files
3. **Verify file location**: `.env` file should be in project root
4. **Check file format**: No spaces around `=` in `.env` files
#### Configuration File Issues
1. **File location**: Default is `~/.opik.config`
2. **Custom location**: Use `OPIK_CONFIG_PATH` environment variable
3. **File format**: Python uses TOML, TypeScript uses INI format
4. **Permissions**: Ensure file is readable by your application
# Offline fallback and message replay
# Offline fallback and message replay
The Opik Python SDK includes a built-in offline fallback mechanism that protects your tracing data during
network outages. When the SDK cannot reach the Opik server, messages are automatically persisted to a local
SQLite database. Once connectivity is restored, all stored messages are replayed to the server transparently,
with no changes required in your application code.
The offline fallback feature is available in the Python SDK only and is enabled by default with no
configuration required.
## How it works
The feature operates entirely in the background across three phases:
**1. Detection** — A lightweight background thread uses (`OpikConnectionMonitor`) to periodically ping the `/is-alive/ping`
endpoint on the Opik server. When a ping fails, or when a message sending encounters a connection error, the
SDK marks the connection as unavailable.
**2. Storage** — While the connection is unavailable, every new message is immediately written to a local
SQLite database (stored in a system temporary directory) instead of being sent over the network. If a message
was in flight when the connection dropped, it is re-marked as failed and added to the same store.
**3. Replay** — When the `OpikConnectionMonitor` detects that the server is reachable again, a `ReplayManager`
thread reads all stored messages in configurable batches and reinjects them into the SDK's normal processing
pipeline. After that, they are delivered to the server just like any other message.
```
Application code
│
▼
Opik SDK client
│
├─ Connection OK? ──Yes──▶ Send to REST API ──Success──▶ Done
│ └─Failure──▶ Write to SQLite
│
└─ Connection down? ───▶ Write to SQLite as "failed"
│
ConnectionMonitor
detects recovery
│
▼
ReplayManager reads
failed messages in
batches and resubmits
```
The SQLite database is cleaned up automatically when the SDK shuts down. Delivered messages are deleted
from the database as soon as the server confirms receipt.
## Supported message types
All SDK operations that produce the following message types are protected by the offline fallback:
| Operation | Message type stored |
| -------------------------------------- | ------------------------------------------------ |
| `client.trace()` | `CreateTraceMessage` / `CreateTraceBatchMessage` |
| `trace.update()` | `UpdateTraceMessage` |
| `trace.span()` / `client.span()` | `CreateSpanMessage` / `CreateSpansBatchMessage` |
| `span.update()` | `UpdateSpanMessage` |
| `client.log_traces_feedback_scores()` | `AddTraceFeedbackScoresBatchMessage` |
| `client.log_spans_feedback_scores()` | `AddSpanFeedbackScoresBatchMessage` |
| `client.log_threads_feedback_scores()` | `AddThreadsFeedbackScoresBatchMessage` |
| Guardrail evaluations | `GuardrailBatchMessage` |
| `experiment.insert()` | `CreateExperimentItemsBatchMessage` |
| File attachments | `CreateAttachmentMessage` |
## Configuration
The offline fallback works out of the box with sensible defaults. You can tune its behaviour using
environment variables or the `~/.opik.config` file.
### Environment variables
Set these before starting your application:
```bash
# How often (seconds) to ping the server to check connectivity (default: 10)
export OPIK_CONNECTION_MONITOR_PING_INTERVAL=10
# Timeout (seconds) for each connectivity ping (default: 5)
export OPIK_CONNECTION_MONITOR_CHECK_TIMEOUT=5
# Number of failed messages to replay in one batch after recovery (default: 50)
export OPIK_REPLAY_BATCH_SIZE=50
# Delay (seconds) between replay batches to control throughput (default: 0.5)
export OPIK_REPLAY_BATCH_REPLAY_DELAY=0.5
# How often (seconds) the replay manager thread checks connection state (default: 0.3)
export OPIK_REPLAY_TICK_INTERVAL=0.3
```
### Configuration file
Add the parameters to your `~/.opik.config` file under the `[opik]` section:
```toml
[opik]
url_override = https://www.comet.com/opik/api
api_key = 
If you have any feedback or feature requests for dashboards, please [open an issue on GitHub](https://github.com/comet-ml/opik/issues).
## Dashboard types
Every dashboard has a **type** that determines what kind of data it works with and which widgets are available:
| Type | Purpose | Available widgets |
| ----------------- | -------------------------------------------------------------------------- | ------------------------------------ |
| **Multi-project** | Track metrics across one or more projects (traces, threads, cost, latency) | Time series, Single metric, Markdown |
| **Experiments** | Compare feedback scores and results across experiment runs | Metrics, Leaderboard, Markdown |
## Accessing dashboards
### Dashboards page
Access the standalone Dashboards page from the sidebar navigation to create and manage workspace-level dashboards. The dashboards list includes a **Type** column showing whether each dashboard is Multi-project or Experiments.
### Project page — Insights tab
Within any project, the **Insights** tab provides built-in and custom views for monitoring that project's traces, threads, and quality metrics.
### Compare Experiments — Insights tab
When comparing experiments, the **Insights** tab shows a built-in read-only view with experiment comparison charts.
## Insights tab
The Insights tab provides curated, in-context monitoring views directly within project and experiment pages.
### Project Insights
When you open a project's Insights tab, you land on the built-in **Project Overview** view — a read-only dashboard covering key health metrics: trace volume, errors, latency, cost, feedback scores, and thread activity.
#### Custom views
Beyond the built-in view, you can create custom Insight views for your project:
1. Open the **views selector** dropdown in the Insights tab
2. Click **Add new** at the bottom
3. Enter a name for your view
Custom views are fully editable — you can add sections, configure widgets, and rearrange the layout. The current project is automatically set as the data source for all widgets.
**Views selector dropdown:**
* Search box at the top for filtering views
* Built-in "Project Overview" is always listed first with a "Built-in" tag
* Custom views appear below with their widget count and last modified date
* **Add new** button at the bottom
**View actions** (available on hover for custom views):
* **Edit name** — rename the view
* **Duplicate** — create a copy of the view
* **Delete** — remove the view (this action cannot be undone)
You can also **duplicate the built-in view** to create an editable copy as a custom view.
The Insights tab has its own time range selector, separate from the Logs tab. Each tab remembers its own time range across sessions.
### Experiment Insights
When comparing experiments, the Insights tab shows a single built-in read-only view displaying experiment comparison charts for the currently selected experiments. There is no view selector — only the built-in view is available.
## Widget types
Dashboards support several widget types. The available types depend on the dashboard type (Multi-project or Experiments).
### Time series widget (Multi-project)
Displays time-series charts for project metrics over time. Supports both line and bar chart visualizations.
**Available metrics:**
* **Trace feedback scores** - Quality metrics for traces over time
* **Number of traces** - Trace volume trends
* **Trace duration** - Trace performance trends
* **Token usage** - Token consumption over time
* **Estimated cost** - Spending trends
* **Failed guardrails** - Guardrail violations over time
* **Number of threads** - Thread volume trends
* **Thread duration** - Thread performance trends
* **Thread feedback scores** - Quality metrics for threads over time
**Configuration options:**
* **Project**: Select the project to pull data from
* **Metric type**: Choose from any of the metrics listed above
* **Chart type**: Line chart (best for trends) or Bar chart (good for volume/period comparisons)
* **Breakdown**: Optionally group data by a field to see per-group patterns. Available fields depend on the data source:
* Trace metrics: Tags, Name, Has error, Error type, Metadata key
* Span metrics: Tags, Name, Has error, Error type, Metadata key, Model, Provider, Span type
* Thread metrics: Tags
When a breakdown is active, use the **aggregation toggle** to control how data is bucketed: **Total** shows one value per group for the entire date range, while **Time-based** shows values in time buckets (hourly, daily, or weekly). Click a label in the chart legend to navigate directly to the traces list filtered to that group.
* **Filters**: Apply trace or thread filters to focus on specific data based on tags, metadata, or other attributes
* **Feedback scores**: When using feedback score metrics, optionally select specific scores to display (leave empty to show all)
### Single metric widget (Multi-project)
Shows a single metric value with a compact card display. Ideal for summary dashboards and key performance indicators.
**Data sources:** Traces or Spans
**Trace-specific metrics:**
* Total trace count
* Total thread count
* Average LLM span count
* Average span count
* Average estimated cost per trace
* Total guardrails failed count
**Span-specific metrics:**
* Total span count
* Average estimated cost per span
**Shared metrics (available for both traces and spans):**
* P50 duration - Median duration
* P90 duration - 90th percentile duration
* P99 duration - 99th percentile duration
* Total input count
* Total output count
* Total metadata count
* Average number of tags
* Total estimated cost sum
* Output tokens (avg.)
* Input tokens (avg.)
* Total tokens (avg.)
* Total error count
* Average feedback scores - Any feedback score defined in your project
### Metrics widget (Experiments)
Compares feedback scores across multiple experiments. Ideal for visualizing A/B test results and prompt iteration outcomes.
**Chart types:**
* **Line chart** - Show trends across experiments (default)
* **Bar chart** - View detailed score distributions side by side
* **Radar chart** - Compare multiple feedback scores across experiments in a radial view
**Configuration options:**
* **Filters**: Filter experiments by:
* Dataset — show only experiments from a specific dataset
* Configuration — filter by metadata keys and values (e.g., model="gpt-4")
* Experiment IDs — include specific experiments by ID
* **Groups** (collapsible, collapsed by default): Group aggregated results by:
* Dataset — compare results across different datasets
* Configuration — group by metadata keys to aggregate feedback scores (e.g., group by model type)
* Supports up to 5 grouping levels for hierarchical comparisons
* **Max experiments**: Limit the number of experiments displayed
* **Chart type**: Choose line, bar, or radar chart visualization
* **Metrics**: Optionally display only specific feedback scores (leave empty to show all)
### Leaderboard widget (Experiments)
Displays a table comparing experiments with configurable columns. Useful for ranking experiments by specific metrics and comparing results at a glance.
**Configuration options:**
* **Filters**: Same filtering options as the Metrics widget (dataset, configuration, experiment IDs)
* **Groups**: Same grouping options as the Metrics widget
* **Max experiments**: Limit the number of experiments displayed
* **Columns**: Select and reorder which columns to display. The columns menu shows all available columns with a "N of N selected" indicator and drag handles for reordering
* **Ranking**: Rank experiments by a specific metric. Options are "No ranking" (default) and any available feedback score metric. When "No ranking" is selected, the ranking order option is disabled
### Markdown text widget
Available for both Multi-project and Experiments dashboards. Add custom notes, descriptions, or documentation using markdown formatting. Use this widget to:
* Add section headers and explanations
* Document dashboard purpose and context
* Include links to related resources
* Add team notes or guidelines
## Creating a workspace dashboard
1. Navigate to the **Dashboards** page from the sidebar
2. Click **Create new dashboard**
3. Select the dashboard type: **Multi-project** or **Experiments**
4. Enter a name (description is optional)
5. Click **Create**
## Adding and configuring widgets
When you click the **+** button within a section, a unified widget configuration modal opens:
1. **Select a widget type** from the clickable cards at the top. The available types depend on the dashboard type:
* **Multi-project**: Time series, Single metric, Markdown
* **Experiments**: Metrics, Leaderboard, Markdown
2. Configure the widget settings below. The configuration area updates based on the selected widget type.
3. Each widget has its own **project or experiment selector** — there are no global dashboard defaults. For Insight views, the current project is automatically set.
4. For chart widgets, select the **visualization type** (line, bar, or radar) using clickable cards.
5. Click **Save** to add the widget.
## Customizing dashboards
### Adding sections
Dashboards are organized into sections, each containing one or more widgets:
1. Click **Add section** at the bottom of the dashboard
2. Give the section a title
3. Add widgets to the section
### Editing widgets
1. Click the menu icon on any widget
2. Select **Edit** to modify the widget configuration
3. Make your changes and save
### Rearranging widgets
* **Drag and drop**: Use the drag handle on widgets to reorder them within a section
* **Resize**: Drag the edges of widgets to adjust their size
### Collapsing sections
Click on a section title to collapse or expand it. The collapsed state is preserved across sessions.
## Date range filtering
Use the date picker in the toolbar to filter data by time range. Select a preset range (Last 24 hours, Last 7 days, etc.) or choose custom dates.
**Widgets that use date range filtering:**
* Time series widget - filters time-series data to the selected range
* Single metric widget - calculates statistics within the selected range
**Widgets not affected by date range:**
* Experiments metrics widget - displays experiment results regardless of date
* Leaderboard widget - displays experiment results regardless of date
* Markdown text widget - static content
## Saving changes
All dashboard changes are **saved automatically**. Built-in Insight views are read-only — duplicate them to create an editable copy.
## Sharing dashboards
To share your current dashboard view:
1. Click the **Share** button in the toolbar
2. The URL is copied to your clipboard
3. Share this URL with team members who have access to the workspace
The shared URL includes the dashboard ID, active date range, and any active filters, so recipients see the same view.
## Next steps
* Set up [Online Evaluation Rules](/production/online-evaluation/rules) to automatically generate feedback scores for your dashboards
* Configure [Alerts](/production/alerts/alerts) to get notified when metrics exceed thresholds
* Learn about [Production Monitoring](/tracing/dashboards/production_monitoring) best practices
# Production monitoring
Opik has been designed from the ground up to support high volumes of traces making it the ideal tool for monitoring your production LLM applications.
You can use the **Insights** tab within any project to review your feedback scores, trace count, latency, and cost over time. The built-in Project Overview provides an at-a-glance health check with stats cards and time-series charts. For more details, see [Dashboards](/tracing/dashboards/dashboards).
In addition to viewing scores over time, you can also view the average feedback scores for all the traces in your project from the traces table.
## Logging feedback scores
To monitor the performance of your LLM application, you can log feedback scores using the [Python SDK and through the UI](/tracing/advanced/annotate_traces).
### Defining online evaluation metrics
You can define LLM as a Judge metrics in the Opik platform that will automatically score all, or a subset, of your production traces. You can find more information about how to define LLM as a Judge metrics in the [Online evaluation](/production/online-evaluation/rules) section.
Once a rule is defined, Opik will score all the traces in the project and allow you to track these feedback scores over time.
In addition to allowing you to define LLM as a Judge metrics, Opik will soon allow you to define Python metrics to
give you even more control over the feedback scores.
### Manually logging feedback scores alongside traces
Feedback scores can be logged while you are logging traces:
```python
from opik import track, opik_context
@track
def llm_chain(input_text):
# LLM chain code
# ...
# Update the trace
opik_context.update_current_trace(
feedback_scores=[
{"name": "user_feedback", "value": 1.0, "reason": "The response was helpful and accurate."}
]
)
```
### Updating traces with feedback scores
You can also update traces with feedback scores after they have been logged. For this we are first going to fetch all the traces using the search API and then update the feedback scores for the traces we want to annotate.
#### Fetching traces using the search API
You can use the [`Opik.search_traces`](https://www.comet.com/docs/opik/python-sdk-reference/Opik.html#opik.Opik.search_traces) method to fetch all the traces you want to annotate.
```python
import opik
opik_client = opik.Opik()
traces = opik_client.search_traces(
project_name="Default Project"
)
```
The `search_traces` method allows you to fetch traces based on any trace attribute.
#### Updating feedback scores
Once you have fetched the traces you want to annotate, you can update the feedback scores using the [`Opik.log_traces_feedback_scores`](https://www.comet.com/docs/opik/python-sdk-reference/Opik.html#opik.Opik.log_traces_feedback_scores) method.
```python
for trace in traces:
opik_client.log_traces_feedback_scores(
scores=[
{
"id": trace.id,
"name": "user_feedback",
"value": 1.0,
"reason": "The response was helpful and accurate.",
"project_name": "Default Project"
}
],
)
```
You will now be able to see the feedback scores in the Opik dashboard and track the changes over time.
### Updating trace content
#### Get trace content
You can view the content of your traces using [`Opik.get_trace_content(id: str)`](https://www.comet.com/docs/opik/python-sdk-reference/Opik.html#opik.Opik.get_trace_content), to look up your trace by id. Trace ids can be found using the [`Opik.search_traces()`](https://www.comet.com/docs/opik/python-sdk-reference/Opik.html#opik.Opik.search_traces) method or by looking at the ID column within the Projects > 'My-project' view.
```python
from opik import Opik
TRACE_ID = 'EXAMPLE-ID' # UUIDv7 Identifier
opik_client = Opik()
trace_content = opik_client.get_trace_content(id = TRACE_ID)
```
This will return a [`TracePublic`](https://www.comet.com/docs/opik/python-sdk-reference/Objects/TracePublic.html#opik.rest_api.types.trace_public.TracePublic) object, a pydantic model object with all the data associated with the trace found.
#### Update trace by ID
You can update a given trace by first re-instantiating the trace using [`opik.Opik.trace()`](https://www.comet.com/docs/opik/python-sdk-reference/Opik.html#opik.Opik.trace) and then updating any one of the trace attributes using [`Trace.update()`](https://www.comet.com/docs/opik/python-sdk-reference/Objects/Trace.html#opik.api_objects.trace.Trace.update). See above section for guidance on how to retrieve trace ids.
```python
from opik import Opik
TRACE_ID = 'EXAMPLE-ID' # UUIDv7 Identifier
opik_client = Opik()
trace = opik_client.trace(id = TRACE_ID)
trace.update(output = updated_output)
```
The trace attributes that can be used as parameters are as follows:
* end\_time: The end time of the trace.
* metadata: Additional metadata to be associated with the trace.
* input: The input data for the trace.
* output: The output data for the trace.
* tags: A list of tags to be associated with the trace.
* error\_info: The dictionary with error information (typically used when the trace function has failed).
* thread\_id: Used to group multiple traces into a thread. The identifier is user-defined and has to be unique per project.
# Development Overview
Opik provides a complete workflow for developing your agent: manage your prompt and model configuration with version control, test changes in the Agent Playground with full tracing, and deploy new versions directly from the UI.
## Manage your prompts and agent configuration
Define your agent's system prompt, model, and parameters in the [Prompt Library](/development/prompt-library/overview). Every change is versioned automatically (`v1`, `v2`, `v3`, …), so you can compare configurations side-by-side and roll back if needed.
Your agent pulls the requested prompt version at runtime, so you can update prompts without redeploying code.
## Test in the Agent Playground
Connect your local agent to Opik with a single command:
```bash
opik endpoint --project 
Switch to the **Configuration** tab to tweak prompts and parameters without changing code. The playground runs your agent against the unsaved configuration so you can test before committing changes.
## Roll out new versions
When you're happy with a configuration, save it as a new version in the Prompt Library and
update your code to pin to it (or keep it on `"latest"`). Every change is versioned, so you can
always roll back.
## More tools
Test and compare prompt variants side-by-side across models. Run against datasets for systematic evaluation.
Automatically optimize prompts and agent configurations using built-in optimization algorithms.
# Agent playground
The Agent Playground lets you run agents on your local machine while connected to Opik. Every agent
execution is fully traced — you get LLM calls, latencies, token usage, and the complete execution
graph, all visible in the Opik UI.
Beyond running your agent, the playground lets you tweak prompts and parameters from the
[Prompt Library](/development/prompt-library/overview) without changing your code. Once
you're happy with the results, save the new configuration and your agent picks it up automatically.
## Getting started
Mark your agent's main function as an entrypoint. Opik automatically detects the function
signature and registers it as a runnable agent.
```python title="Python"
import opik
@opik.track(entrypoint=True, project_name="my-agent")
def my_agent(query: str, max_results: int = 5) -> str:
# Your agent logic here
return result
```
```ts title="TypeScript"
import { track } from "opik";
const myAgent = track(
{
entrypoint: true,
name: "my-agent",
params: [
{ name: "query", type: "string" },
{ name: "maxResults", type: "number" },
],
},
async (query: string, maxResults: number = 5): Promise
Every run in the playground produces a full trace — every LLM call, tool invocation, and sub-step is
captured as spans with inputs, outputs, latencies, and token costs. Logs from your running agent
stream to the UI in real time, so you can watch execution as it happens.
The playground monitors your runner with a heartbeat and updates its status automatically if it
disconnects. When `--watch` is enabled, file changes are detected and your agents are
re-registered without restarting the process. For CI or programmatic setups, use `--headless` to
skip the browser pairing flow entirely.
## Troubleshooting
**"No entrypoint found" error**
Make sure at least one function is decorated with `@opik.track(entrypoint=True)` in Python or
`track({ entrypoint: true }, fn)` in TypeScript. The entrypoint must be discoverable from the
current working directory.
**Pairing times out**
The browser pairing session expires after 5 minutes. Re-run the command to generate a new session.
Make sure your Opik environment variables are set correctly — see
[Getting started with Observability](/tracing/getting-started) for configuration details.
**Runner disconnects**
Opik uses heartbeat monitoring to detect disconnects. If your runner shows as disconnected in the
UI, check that the process is still running locally and that your network connection is stable.
## FAQ
They serve different purposes:
* **`opik connect`** starts a lightweight bridge daemon that gives [Ollie](/ollie)
remote access to your repository. Ollie can then read your source files, propose code
changes, and rerun your agent — all from the Opik UI. It does not run your agent process
itself.
* **`opik endpoint`** runs your agent process and connects it to the Agent Playground. You can
submit inputs from the UI, see results with full traces, and iterate on your
[Prompt Library](/development/prompt-library/overview) without changing code.
You can use both at the same time: `opik endpoint` to run and test your agent in the playground,
and `opik connect` to let Ollie inspect and edit your code.
## Next steps
* [Ollie](/ollie) — Use Ollie with `opik connect` to debug and improve your agent's code
* [Debugging agents with Ollie](/tracing/debug-agents) — The debug-fix-verify workflow
* [Getting started with Observability](/tracing/getting-started) — Configure your Opik environment
# Prompt Playground
The Playground lets you test prompt changes and compare models without writing code. Create
multiple prompt variants, run them side by side, and validate the results against a test suite —
all from the Opik UI.
## Compare prompt variants side by side
Each variant in the Playground is independent — it has its own model, messages, and configuration.
This means you can test a prompt change against the current version, try different models on the
same prompt, or experiment with temperature and sampling parameters, all in a single view.
Supported providers include OpenAI, Anthropic, Gemini, OpenRouter, Vertex AI, and custom
endpoints. Reasoning models like Claude and o1/o3 expose additional controls such as thinking
effort.
Click **Run** (or press **Shift+Enter**) to execute all variants at once. Results stream in real
time with the model's response, token usage, latency, and a link to the full trace.
## Validate against test suites
The real power of the Playground is running your prompt variants against a dataset or test suite.
Instead of manually checking a handful of inputs, you can validate across your full set of test
cases and see which variant performs better.
Click **Test on Dataset** in the header and select a dataset or test suite. If you're using
template variables (`{{variable_name}}`), they are automatically mapped to dataset columns.
Click **Run experiment** to execute all prompt variants against every item in the dataset.
Results appear in a table below the prompts, with each variant's output shown side by side.
When using a test suite, each output is scored against the suite's evaluation rules and
displayed as pass/fail. You can click into any result to inspect the full trace.
Experiments are saved automatically — compare them over time in the **Experiments** tab.
## Template variables
Use `{{variable_name}}` syntax in your prompt messages to create dynamic templates. When running
in standard mode, the Playground asks you to fill in the values. In dataset mode, variables are
mapped to dataset columns automatically.
## Next steps
* [Prompt Library](/development/prompt-library/overview) — Manage prompts and the rest of your agent configuration in one place
* [Test suites](/evaluation/advanced/building-test-suites) — Build the test cases your playground experiments run against
* [Experiments](/evaluation/advanced/evaluate_your_llm) — Review and compare experiment results over time
# Prompt Library Overview
The Prompt Library is the central place to manage every prompt your agent depends on. Version
them, iterate on new versions from the Opik UI or the SDK, and link the exact version that
produced a trace.
## Why use the Prompt Library
When prompts are hardcoded, every change requires a code change to update. There is no history
of what changed, no way to roll back quickly, and no clean way to compare alternative versions.
The Prompt Library gives you version control for the prompts that drive your agent. Push them to
Opik once, and from there you can compare versions, fetch a specific one, or always follow the
latest.
## Key features
* **Project-scoped prompts** — Every prompt lives inside a project, so different agents can have prompts with the same name without colliding.
* **Versioning by `v1`, `v2`, `v3`, …** — Each change creates a new immutable, sequentially numbered version. Fetch a specific one with the `version` selector (e.g. `"v3"`), or omit it to fetch the most recent.
* **Text and chat prompts** — Version simple string templates as well as multi-message chat templates with `{{variable}}` substitution.
* **Agent configuration in one place** — Prompts live alongside the rest of an agent configuration (model, parameters, tools) so a trace links back to the exact bundle that produced it.
* **Python and TypeScript SDKs** — Create, fetch, and render prompts from either language.
## How it works
1. **Define your prompts** — Use `create_prompt` / `create_chat_prompt` to send your prompts to Opik. This becomes `v1`.
2. **Fetch at runtime** — Call `get_prompt` / `get_chat_prompt` from inside a tracked function. The SDK returns the requested version and links it to the trace automatically.
3. **Iterate from the UI** — Edit the prompt and save a new version. If your code doesn't pin a `version`, the agent picks up the new version on its next fetch — otherwise update your code to point at the new version name.
## Next steps
* [Getting started](/development/prompt-library/getting-started) — Add the Prompt Library to your code in minutes
* [Text and chat prompts](/development/prompt-library/prompt-types) — When to use each, multimodal content, template engines, and searching
* [Version control](/development/prompt-library/version-control) — Create and compare versions
* [Prompt playground](/development/prompt-playground) — Test prompt variants side-by-side across models, without writing code
# Getting started with the Prompt Library
Your agent depends on prompts that change frequently. The Prompt Library lets you manage them
outside your codebase, version each change automatically, and link the exact version that ran
to its trace.
## Adding the Prompt Library to your code
You can use the Opik skills to wire your existing agent up to the Prompt Library:
```bash
npx skills add comet-ml/opik-skills
```
This skill is compatible with all coding agents including Claude Code, Codex, Cursor, OpenCode and more.
Once the skill is installed, you can integrate with Opik using the following prompt:
```
Version my prompts in Opik using the /instrument command.
```
There are two parts: you **create** a prompt, and then you **fetch** it at runtime from
inside your agent.
### Step 1 — Define and push your first prompt
Push the prompt to Opik. You only do this once (or whenever you want to create a new version
from code).
```python title="Python"
import opik
client = opik.Opik()
client.create_prompt(
name="system_prompt",
prompt="You are a helpful assistant specializing in {{domain}}.",
project_name="my-agent",
)
```
```ts title="TypeScript"
import { Opik } from "opik";
const client = new Opik();
await client.createPrompt({
name: "system_prompt",
prompt: "You are a helpful assistant specializing in {{domain}}.",
projectName: "my-agent",
});
```
Prompts are **project-scoped**. Always pass a `project_name` / `projectName` so two agents can use the same prompt names independently.
### Step 2 — Fetch your prompt at runtime
Use `get_prompt` / `getPrompt` inside your agent to pull the version you want and render it
with your runtime values.
```python title="Python"
import opik
client = opik.Opik()
@opik.track(project_name="my-agent")
def run_agent(user_input: str):
prompt = client.get_prompt(
name="system_prompt",
project_name="my-agent",
)
system_prompt = prompt.format(domain="customer support")
response = call_llm(
model="gpt-4o-mini",
system_prompt=system_prompt,
user_input=user_input,
)
return response
```
```ts title="TypeScript"
import { Opik, track } from "opik";
const client = new Opik();
const runAgent = track(
{ name: "run_agent", projectName: "my-agent" },
async (userInput: string) => {
const prompt = await client.getPrompt({
name: "system_prompt",
projectName: "my-agent",
});
const systemPrompt = prompt?.format({ domain: "customer support" });
const response = await callLlm({
model: "gpt-4o-mini",
systemPrompt,
userInput,
});
return response;
},
);
```
Calling `get_prompt` / `getPrompt` inside a tracked function (`@opik.track` in Python,
`track()` in TypeScript) lets Opik link the prompt version to the trace automatically.
## Choosing a version
Pass the `version` parameter to control which version is returned:
* **Omit `version`** (default) — The most recently created version. Useful when you want the agent to pick up new prompt edits automatically.
* **`"v3"`** (or any `v
You can edit a prompt later by clicking on its name in the library and choosing **Edit prompt**.
Each call with new content creates a new version, which is visible in the library:
### Fetching a text prompt
```python
import opik
client = opik.Opik()
prompt = client.get_prompt(name="prompt-summary", project_name="my-agent")
# Render the template
print(prompt.format(text="Hello, world!"))
```
```typescript
import { Opik } from "opik";
const client = new Opik();
const prompt = await client.getPrompt({
name: "prompt-summary",
projectName: "my-agent",
});
if (prompt) {
console.log(prompt.format({ text: "Hello, world!" }));
}
```
If you are not using the SDK, you can also fetch a prompt through the
[REST API](/reference/rest-api/overview).
## Chat prompts
A chat prompt is a structured list of messages with roles (`system`, `user`, `assistant`), in
the same shape that OpenAI-compatible chat completion APIs accept. They are the right choice
when your agent has a multi-turn message structure.
### Key features
* **Structured messages** — Organize prompts as a list of messages with roles (system, user, assistant)
* **Multimodal support** — Include images, videos, and text in the same prompt
* **Variable substitution** — Use Mustache (`{{variable}}`) or Jinja2 syntax
* **Version control** — Automatic versioning when messages change
### Creating a chat prompt
```python
import opik
client = opik.Opik()
messages = [
{"role": "system", "content": "You are a helpful assistant specializing in {{domain}}."},
{"role": "user", "content": "Explain {{topic}} in simple terms."},
]
chat_prompt = client.create_chat_prompt(
name="educational-assistant",
messages=messages,
metadata={"category": "education"},
project_name="my-agent",
)
# Render the messages with variables
formatted_messages = chat_prompt.format(
variables={
"domain": "physics",
"topic": "quantum entanglement",
}
)
print(formatted_messages)
# [
# {"role": "system", "content": "You are a helpful assistant specializing in physics."},
# {"role": "user", "content": "Explain quantum entanglement in simple terms."},
# ]
```
```typescript
import { Opik } from "opik";
const client = new Opik();
const messages = [
{
role: "system",
content: "You are a helpful assistant specializing in {{domain}}.",
},
{ role: "user", content: "Explain {{topic}} in simple terms." },
];
const chatPrompt = await client.createChatPrompt({
name: "educational-assistant",
messages,
metadata: { category: "education" },
projectName: "my-agent",
});
// Render the messages with variables
const formattedMessages = chatPrompt?.format({
domain: "physics",
topic: "quantum entanglement",
});
console.log(formattedMessages);
// [
// { role: "system", content: "You are a helpful assistant specializing in physics." },
// { role: "user", content: "Explain quantum entanglement in simple terms." },
// ]
```
To create a chat prompt in the UI, navigate to the Prompt Library and click **Create new
prompt**. Select **Chat prompt** as the prompt type, then add your messages with their
roles (system, user, assistant):
Once saved, a chat prompt has its own view in the Prompt Library:
### Using chat prompts with the OpenAI API
The output of `chat_prompt.format()` is already in the shape the OpenAI chat completion API
expects:
```python
import opik
from openai import OpenAI
client = opik.Opik()
openai_client = OpenAI()
chat_prompt = client.get_chat_prompt(
name="educational-assistant",
project_name="my-agent",
)
formatted_messages = chat_prompt.format(
variables={"domain": "physics", "topic": "quantum entanglement"},
)
response = openai_client.chat.completions.create(
model="gpt-4o-mini",
messages=formatted_messages,
)
print(response.choices[0].message.content)
```
```typescript
import { Opik } from "opik";
import OpenAI from "openai";
const client = new Opik();
const openai = new OpenAI();
const chatPrompt = await client.getChatPrompt({
name: "educational-assistant",
projectName: "my-agent",
});
const formattedMessages = chatPrompt?.format({
domain: "physics",
topic: "quantum entanglement",
});
const response = await openai.chat.completions.create({
model: "gpt-4o-mini",
messages: formattedMessages,
});
console.log(response.choices[0].message.content);
```
### Multi-turn templates
Chat prompts can capture a multi-turn flow, with assistant turns inline and variables anywhere
in the conversation:
```python
import opik
client = opik.Opik()
messages = [
{"role": "system", "content": "You are a customer support agent for {{company}}."},
{"role": "user", "content": "I have an issue with {{product}}."},
{"role": "assistant", "content": "I'd be happy to help with your {{product}}. Can you describe the issue?"},
{"role": "user", "content": "{{issue_description}}"},
]
chat_prompt = client.create_chat_prompt(
name="customer-support-flow",
messages=messages,
project_name="my-agent",
)
formatted = chat_prompt.format(
variables={
"company": "Acme Corp",
"product": "Widget Pro",
"issue_description": "It won't turn on",
}
)
```
```typescript
import { Opik } from "opik";
const client = new Opik();
const messages = [
{
role: "system",
content: "You are a customer support agent for {{company}}.",
},
{ role: "user", content: "I have an issue with {{product}}." },
{
role: "assistant",
content:
"I'd be happy to help with your {{product}}. Can you describe the issue?",
},
{ role: "user", content: "{{issue_description}}" },
];
const chatPrompt = await client.createChatPrompt({
name: "customer-support-flow",
messages,
projectName: "my-agent",
});
const formatted = chatPrompt?.format({
company: "Acme Corp",
product: "Widget Pro",
issue_description: "It won't turn on",
});
```
### Multimodal content
Chat prompts can include images and videos alongside text — useful for vision-enabled models.
#### Image analysis
```python
import opik
client = opik.Opik()
messages = [
{"role": "system", "content": "You analyze images and provide detailed descriptions."},
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image of {{subject}}?"},
{
"type": "image_url",
"image_url": {
"url": "{{image_url}}",
"detail": "high",
},
},
],
},
]
chat_prompt = client.create_chat_prompt(
name="image-analyzer",
messages=messages,
project_name="my-agent",
)
formatted = chat_prompt.format(
variables={
"subject": "a sunset",
"image_url": "https://example.com/sunset.jpg",
},
supported_modalities={"vision": True},
)
```
```typescript
import { Opik } from "opik";
const client = new Opik();
const messages = [
{
role: "system",
content: "You analyze images and provide detailed descriptions.",
},
{
role: "user",
content: [
{ type: "text", text: "What's in this image of {{subject}}?" },
{
type: "image_url",
image_url: {
url: "{{image_url}}",
detail: "high",
},
},
],
},
];
const chatPrompt = await client.createChatPrompt({
name: "image-analyzer",
messages,
projectName: "my-agent",
});
const formatted = chatPrompt?.format(
{
subject: "a sunset",
image_url: "https://example.com/sunset.jpg",
},
{ vision: true },
);
```
#### Video analysis
```python
import opik
client = opik.Opik()
messages = [
{"role": "system", "content": "You analyze videos and provide insights."},
{
"role": "user",
"content": [
{"type": "text", "text": "Analyze this video: {{description}}"},
{
"type": "video_url",
"video_url": {
"url": "{{video_url}}",
"mime_type": "video/mp4",
},
},
],
},
]
chat_prompt = client.create_chat_prompt(
name="video-analyzer",
messages=messages,
project_name="my-agent",
)
formatted = chat_prompt.format(
variables={
"description": "traffic analysis",
"video_url": "https://example.com/traffic.mp4",
},
supported_modalities={"vision": True},
)
```
```typescript
import { Opik } from "opik";
const client = new Opik();
const messages = [
{
role: "system",
content: "You analyze videos and provide insights.",
},
{
role: "user",
content: [
{ type: "text", text: "Analyze this video: {{description}}" },
{
type: "video_url",
video_url: {
url: "{{video_url}}",
mime_type: "video/mp4",
},
},
],
},
];
const chatPrompt = await client.createChatPrompt({
name: "video-analyzer",
messages,
projectName: "my-agent",
});
const formatted = chatPrompt?.format(
{
description: "traffic analysis",
video_url: "https://example.com/traffic.mp4",
},
{ vision: true, video: true },
);
```
#### Mixed content
You can combine multiple content blocks in a single message — e.g. two images with text around
them:
```python
import opik
client = opik.Opik()
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": "Compare these two images:"},
{"type": "image_url", "image_url": {"url": "{{image1_url}}"}},
{"type": "text", "text": "and"},
{"type": "image_url", "image_url": {"url": "{{image2_url}}"}},
{"type": "text", "text": "What are the main differences?"},
],
},
]
chat_prompt = client.create_chat_prompt(
name="image-comparison",
messages=messages,
project_name="my-agent",
)
formatted = chat_prompt.format(
variables={
"image1_url": "https://example.com/before.jpg",
"image2_url": "https://example.com/after.jpg",
},
supported_modalities={"vision": True},
)
```
```typescript
import { Opik } from "opik";
const client = new Opik();
const messages = [
{
role: "user",
content: [
{ type: "text", text: "Compare these two images:" },
{ type: "image_url", image_url: { url: "{{image1_url}}" } },
{ type: "text", text: "and" },
{ type: "image_url", image_url: { url: "{{image2_url}}" } },
{ type: "text", text: "What are the main differences?" },
],
},
];
const chatPrompt = await client.createChatPrompt({
name: "image-comparison",
messages,
projectName: "my-agent",
});
const formatted = chatPrompt?.format(
{
image1_url: "https://example.com/before.jpg",
image2_url: "https://example.com/after.jpg",
},
{ vision: true },
);
```
When formatting multimodal prompts, you can specify `supported_modalities` to control how
content is rendered:
* If a modality is supported (e.g. `{"vision": True}`), the structured content is preserved.
* If a modality is not supported, it's replaced with text placeholders (e.g. `<<
# Agent Optimization
> Learn about Opik's automated LLM prompt and agent optimization SDK. Discover MetaPrompt, Few-shot Bayesian, and Evolutionary optimization algorithms for enhanced performance.
**Opik Agent Optimizer** is a turnkey, open-source agent and prompt optimization SDK. It automatically tunes prompts, tools, and agent workflows using the datasets, metrics, and traces you already log to Opik. Instead of hand-editing instructions and re-running evaluations, pick an optimizer (MetaPrompt, HRPO, Evolutionary, GEPA, etc.) and let it iterate for you online or fully offline inside Docker and Kubernetes.
## Why teams choose Opik Agent Optimizer
* **Automatic prompt optimization** – end-to-end workflow that installs in minutes and runs locally or in your stack.
* **Open-source and framework agnostic** – no lock-in, use Opik’s first-party optimizers or community favorites like GEPA in the same SDK.
* **Agent-aware** – optimize beyond system prompts, including MCP tool signatures, function-calling schemas, and multi-step agent workflows.
* **Deep observability** – every trial logs prompts, tool calls, traces, and metric reasons to Opik so you can explain and ship changes confidently.
## Key capabilities
MetaPrompt, HRPO, Few-Shot Bayesian, Evolutionary, GEPA, Parameter tuning. Swap optimizers without changing your workflow.
Optimize full agent systems with multiple prompts, tools, and orchestration logic, not just a single system message.
Optimize tool schemas and function calling alongside prompt text with the same metrics and datasets.
Track trials, candidates, datasets, and trace-level evidence to explain and ship improvements confidently.
Run optimizer workflows directly from the UI with no-code configuration and result review.
Run the SDK locally or inside Opik Docker to keep data inside your network.
## How it works
Use Opik datasets (CSV upload, API, or trace exports) plus deterministic metrics/`ScoreResult` functions. See [Define datasets](/development/optimization-runs/optimization/define_datasets) and [Define metrics](/development/optimization-runs/optimization/define_metrics).
Choose the best algorithm for your task (see [Optimization algorithms](/development/optimization-runs/algorithms/overview)). All optimizers expose the same API, so you can swap them easily or chain runs.
Results land in the Opik dashboard under **Evaluation → Optimization runs**, where you can compare prompts, failure modes, and dataset coverage before promoting the change.
## Start fast
* Want a no-code workflow? Use [Optimization Studio](/development/optimization-runs/optimization_studio) to run optimizations from the Opik UI.
* Follow the [Quickstart](/development/optimization-runs/quickstart) to run your first optimization locally.
* Prefer notebooks? Launch the [Quickstart notebook](/development/optimization-runs/cookbooks/optimizer_introduction_cookbook).
* Need scenario-specific guidance? Explore the [Cookbooks](/development/optimization-runs/cookbooks/optimizer_introduction_cookbook).
## Optimization Algorithms
The optimizer implements both proprietary and open-source optimization algorithms. Each one has its
strengths and weaknesses, we recommend first trying out either GEPA or HRPO (Hierarchical Reflective Prompt Optimizer)
as a first step:
| **Algorithm** | **Description** |
| ---------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [MetaPrompt Optimization](/development/optimization-runs/algorithms/metaprompt_optimizer) | Uses an LLM ("reasoning model") to critique and iteratively refine an initial instruction prompt. Good for general prompt wording, clarity, and structural improvements. **Supports [MCP tool calling optimization](/development/optimization-runs/algorithms/tool_optimization).** |
| [HRPO (Hierarchical Reflective Prompt Optimizer)](/development/optimization-runs/algorithms/hierarchical_adaptive_optimizer) | Uses hierarchical root cause analysis to systematically improve prompts by analyzing failures in batches, synthesizing findings, and addressing identified failure modes. Best for complex prompts requiring systematic refinement based on understanding why they fail. |
| [Few-shot Bayesian Optimization](/development/optimization-runs/algorithms/fewshot_bayesian_optimizer) | Specifically for chat models, this optimizer uses Bayesian optimization (Optuna) to find the optimal number and combination of few-shot examples (demonstrations) to accompany a system prompt. |
| [Evolutionary Optimization](/development/optimization-runs/algorithms/evolutionary_optimizer) | Employs genetic algorithms to evolve a population of prompts. Can discover novel prompt structures and supports multi-objective optimization (e.g., score vs. length). Can use LLMs for advanced mutation/crossover. |
| [GEPA Optimization](/development/optimization-runs/algorithms/gepa_optimizer) | Wraps the external GEPA package to optimize a single system prompt for single-turn tasks using a reflection model. Requires `pip install gepa`. |
| [Parameter Optimization](/development/optimization-runs/algorithms/parameter_optimizer) | Optimizes LLM call parameters (temperature, top\_p, etc.) using Bayesian optimization. Uses Optuna for efficient parameter search with global and local search phases. Best for tuning model behavior without changing the prompt. |
Want to see numbers? Check the new [optimizer benchmarks](/development/optimization-runs/algorithms/benchmarks) page for the latest performance table and instructions for running the benchmark suite yourself.
## Next Steps
1. Explore specific [Optimizers](/development/optimization-runs/algorithms/overview) for algorithm details.
2. Refer to the [FAQ](/development/optimization-runs/faq) for common questions and troubleshooting.
3. Refer to the [API Reference](/development/optimization-runs/advanced/api_reference) for detailed configuration options.
🚀 Want to see Opik Agent Optimizer in action? Check out our [Example Projects & Cookbooks](/development/optimization-runs/cookbooks/optimizer_introduction_cookbook) for runnable Colab notebooks covering real-world optimization workflows, including HotPotQA and synthetic data generation.
# Optimization Studio
> Run prompt optimizations from the Opik UI with datasets, metrics, and visual progress tracking.
Optimization Studio helps you improve prompts without writing code. You bring a prompt, define what “good” looks like, and Opik tests variations to find a better version you can ship with confidence. Teams like it because it shortens the loop from idea to evidence: you see scores and examples, not just a hunch. If you prefer a programmatic workflow, use the [Optimize prompts](/development/optimization-runs/optimization/optimize_prompts) guide.
## Start an optimization
An optimization run is a structured way to improve a prompt. Opik takes your current prompt, tries small variations, and scores each one so you can pick the best-performing version with evidence instead of guesswork.
## Configure the run
### Name the run
Give the run a descriptive name so you can find it later. A good pattern is `goal + dataset + date`, for example “Support intent v1 - Jan 2026”.
### Configure the prompt
Choose the model that will generate responses, then set the message roles (System, User, and so on). If your dataset has fields like `question` or `answer`, insert them with `{{variable}}` placeholders so each example flows into the prompt correctly. Start with the prompt you already use in production so improvements are easy to compare.
### Pick an algorithm
Choose how Opik should search for better prompts. GEPA works well for single-turn prompts and quick improvements, while HRPO is better when you need deeper analysis of why a prompt fails. If you are new, start with GEPA to get a quick baseline, then switch to HRPO if you need deeper insight. For technical details, see [Optimization algorithms](/development/optimization-runs/algorithms/overview).
### Choose a dataset
Pick an existing dataset to supply examples. Aim for diverse, real-world cases rather than edge cases only, and keep the first run small so you can iterate quickly. If you need to create or upload data first, see [Manage datasets](/evaluation/advanced/manage_datasets).
### Define a metric
Pick how Opik should score each prompt. Use Equals if the output should match exactly, or G-Eval if you want a model to grade quality. When using G-Eval, make sure the grading prompt reflects what “good” means for your task.
* **Equals**: Use when you have a single correct answer and want a strict match.
* **G-Eval**: Use when answers can vary and you want a model to score quality.
## Monitor progress
Once the run starts, Optimization Studio shows the best score so far and a progress chart for each trial.
## Analyze results
The Trials tab is where you compare prompt variations and scores, by clicking on a specific trial you can view the individual trial items that were evaluated.
## Actions
You can rerun the same setup, cancel a run to change inputs, or select multiple runs to compare outcomes.
## Reuse results outside the UI
If you want to automate optimizations in code later, follow [Optimize prompts](/development/optimization-runs/optimization/optimize_prompts) and use the same dataset and metric from this run.
## Next steps
For a deeper breakdown of trials and traces, visit [Dashboard results](/development/optimization-runs/optimization/dashboard_results). If you want to automate this workflow, use [Optimize prompts](/development/optimization-runs/optimization/optimize_prompts). To fine-tune your strategy, explore [Optimization algorithms](/development/optimization-runs/algorithms/overview).
# Quickstart
> Install the Agent Optimizer SDK, run your first optimization, and inspect the results in under 10 minutes.
**Opik Agent Optimizer Quickstart** gives you the fastest path from “hello world” to a successful optimization run. If you already walked through the main [Opik Quickstart](/quickstart) (tracing + evaluation), this is the next stop—it layers on the `opik-optimizer` SDK so you can automatically improve prompts and agents. Prefer a UI workflow? Use [Optimization Studio](/development/optimization-runs/optimization_studio) instead.
## Why Opik Agent Optimizer?
* **Production-grade workflows** – reuse the same datasets, metrics, and tracing you already have in Opik.
* **Multiple strategies** – swap between MetaPrompt, Hierarchical Reflective Prompt Optimizer (HRPO), Evolutionary, GEPA, and more with one API.
* **Deep analysis** – every trial is logged to Opik so you can inspect prompts, tool calls, and failure modes.
Estimated time: **≤10 minutes** if you already have Python and an Opik API key configured.
## Prerequisites
* Python 3.10+
* Opik account
* Access to an OpenAI-compatible LLM via LiteLLM (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.)
## 1. Install and authenticate
```bash
pip install --upgrade opik opik-optimizer
opik configure # paste your API key
export OPIK_PROJECT_NAME="optimization-quickstart"
```
Setting `OPIK_PROJECT_NAME` ensures all traces, experiments, and optimization runs are logged to the same project without having to pass `project_name` to every SDK call.
## 2. Create a dataset and metric
```python
import opik
from opik.evaluation.metrics import LevenshteinRatio
client = opik.Opik()
dataset = client.get_or_create_dataset(name="agent-opt-quickstart")
dataset.insert([
{"question": "What is Opik?", "answer": "Opik is an LLM observability and optimization platform."},
{"question": "How do I reduce hallucinations?", "answer": "Use evaluations and prompt optimization to enforce grounding."},
])
def answer_quality(item, output):
metric = LevenshteinRatio()
return metric.score(reference=item["answer"], output=output)
```
## 3. Run the optimizer
```python
from opik_optimizer import MetaPromptOptimizer, ChatPrompt
prompt = ChatPrompt(
messages=[
{"role": "system", "content": "You are a precise assistant."},
{"role": "user", "content": "{question}"},
],
model="openai/gpt-5-nano" # The model your prompt runs on
)
optimizer = MetaPromptOptimizer(model="openai/gpt-5-nano") # The model that improves your prompt
result = optimizer.optimize_prompt(
prompt=prompt,
dataset=dataset,
metric=answer_quality,
max_trials=3,
n_samples=2,
)
result.display()
```
**Using a different LLM provider?** The optimizer supports OpenAI, Anthropic, Gemini, Azure, Ollama, and 100+ other providers via LiteLLM. See the [Configure LLM Providers](/development/optimization-runs/optimization/configure_models) guide for setup instructions.
## 4. Inspect results
* Run `opik dashboard` or open [https://www.comet.com/opik](https://www.comet.com/opik).
* In the left nav, go to **Evaluation → Optimization runs**, then select your latest run.
* Review the optimization-progress chart, trial table, and per-trial traces to decide whether to ship the new prompt.
## Common first issues
Import `ChatPrompt` from `opik_optimizer` and wrap your `messages` list before passing it to any optimizer.
Re-run `opik configure` and confirm the account has Agent Optimizer access. If you changed machines, copy the `~/.opik/config` file or re-enter the key.
Ensure provider keys (e.g., `OPENAI_API_KEY`) are exported in the same shell running the script, and verify the model you selected is enabled for that key.
## Next steps
* Prefer notebooks? Launch the [Quickstart notebook](/development/optimization-runs/cookbooks/optimizer_introduction_cookbook).
* Dive deeper into [Define datasets](/development/optimization-runs/optimization/define_datasets) and [Define metrics](/development/optimization-runs/optimization/define_metrics).
* Explore the [Optimization Algorithms overview](/development/optimization-runs/algorithms/overview) to pick the best strategy for your workload.
# Optimizer Introduction Cookbook
> Learn how to use Opik Agent Optimizer with the HotPotQA dataset through an interactive notebook, covering setup, configuration, and optimization techniques.
This example demonstrates end-to-end prompt optimization on the HotPotQA dataset using Opik Agent Optimizer. All
steps, code, and explanations are provided in the interactive Colab notebook below.
This notebook powers the **Quickstart notebook** entry in the Agent Optimization navigation.
## Load Example Notebook
To follow this example, simply open the Colab notebook below. You can run, modify, and experiment with the workflow
directly in your browser—no local setup required.
| Platform | Launch Link |
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Google Colab (Preferred)** | [
](https://colab.research.google.com/github/comet-ml/opik/blob/main/sdks/opik_optimizer/notebooks/OpikOptimizerIntro.ipynb) |
| **GitHub** | [View the notebook on GitHub](https://github.com/comet-ml/opik/blob/main/sdks/opik_optimizer/notebooks/OpikOptimizerIntro.ipynb) |
## What you'll learn
* How to set up Opik Agent Optimizer SDK
* How to setup Opik Cloud (Comet Account) for prompt optimization
* How to use the HotPotQA dataset for multi-hop question answering
* How to define metrics and task configs
* How to run the `FewShotBayesianOptimizer` and interpret results
* How to visualize optimization runs in the Opik UI
## Quick Start
1. Click the Colab badge above to launch the notebook.
2. Follow the step-by-step instructions in the notebook.
3. For more details, see the [Opik Agent Optimizer documentation](/development/optimization-runs/overview).
# Optimizer Frequently Asked Questions
> Find answers to common questions about Opik Agent Optimizer, including optimizer selection, configuration, usage, and best practices.
## Getting started
The Agent Optimizer provides a unified interface for optimizing **your existing prompts and
agents** with cutting edge optimization algorithms. In addition to giving you access to cutting
edge academic optimizers like GEPA, we also provide a set of algorithms that have been developed
in-house based on production applications.
The optimizer will allow you to improve the performance of your agents without the need for
manual prompt engineering. You can also use the optimizer to reduce the size of the prompts in
your agents, reducing cost and latency while maintaining performance.
To get started, you will need:
1. The prompt you want to optimize
2. A dataset of examples to optimizer on, you can start with as little as 10
3. A metric to evaluate the performance of the prompt
Once you have these, check out the [Quickstart Guide](/development/optimization-runs/quickstart)
to run your first optimization.
Yes, we would be more than happy to help you setup the Opik Optimizer for your use case ! You can
join our [Slack community](https://chat.comet.com) and ask for help there.
## Optimization Algorithms
Opik Agent Optimizer supports a wide range of optimization algorithms including:
* [HRPO (Hierarchical Reflective Prompt Optimizer)](/development/optimization-runs/algorithms/hierarchical_adaptive_optimizer)
* [GEPA Optimization](/development/optimization-runs/algorithms/gepa_optimizer)
* [Evolutionary Optimization](/development/optimization-runs/algorithms/evolutionary_optimizer)
* [Few-shot Bayesian Optimization](/development/optimization-runs/algorithms/fewshot_bayesian_optimizer)
* [MetaPrompt Optimization](/development/optimization-runs/algorithms/metaprompt_optimizer)
If you would like us to add a new optimization algorithm, simply create an issue on our
[GitHub repository](https://github.com/comet-ml/opik/issues) and we will be happy to add it !
Knowing which optimizer to use depends on your specific needs. As a rule of thumb, we recommend
starting with [HRPO (Hierarchical Reflective Prompt Optimizer)](/development/optimization-runs/algorithms/hierarchical_adaptive_optimizer)
as this has been shown to be a strong baseline for most tasks.
You can also try to use:
1. GEPA: This is one of the top performing academic optimizers and is a good option if you have a
complex task.
2. FewShotBayesianOptimizer: If you have a task that is quite repetitive in the formatting of the prompt
and responses then this is a good option.
While some optimizations run for shorts period of time, it is common for optimizations to take a
couple of hours to complete. As you are starting out, we recommend setting the `max_trials`
parameter to a reasonable number and increasing / decreasing it as you go.
The number of samples you need depends on your task. As a rule of thumb, we recommend starting with
at least 10 samples. The more samples you have, the more accurate the optimization will be.
The Opik Agent Optimizer uses [LiteLLM](https://docs.litellm.ai/) to support 100+ LLM providers including:
* **OpenAI** (GPT-4o, GPT-4o-mini, o1, o3-mini)
* **Anthropic** (Claude 3.5 Sonnet, Claude 3 Opus)
* **Google Gemini** (Gemini 2.0 Flash, Gemini 1.5 Pro)
* **Azure OpenAI**
* **Ollama** (local models like Llama 3, Mistral)
* **OpenRouter** (access to multiple providers)
Models use the LiteLLM format: `provider/model-name` (e.g., `gemini/gemini-2.0-flash`).
See the [Configure LLM Providers](/development/optimization-runs/optimization/configure_models) guide for detailed setup instructions and examples for each provider.
There are a few things you can try to improve the performance of your optimization:
1. Review your optimization metric, ideally it should provide the model with an insightful
`reason` for the score it gives. By improving the quality of the metric, the optimizer will be
able to make better optimizations.
2. Review your dataset, ideally it should be a diverse set of examples that cover the different
scenarios you want to optimize for.
3. Use more powerful models for both the chatPrompt model and the optimizer, as models get more
powerful, they will be able to generate better optimizations.
## Common Errors
For frequent setup and usage errors, head to [Known Issues](/development/optimization-runs/known_issues). We keep the error catalog there so fixes and version notes stay in one place.
## Open challenges & advanced topics
Research has shown "evil twin" prompts and unusual delimiters can perform well despite being hard to interpret. Optimizers explore the search space indiscriminately, so high-performing instructions aren't always human-readable. When interpretability matters, prefer algorithms like HRPO that include reasoning traces or enforce structure via custom metrics.
Cost varies widely. Reflection-heavy optimizers (e.g., HRPO, GEPA) may call LLMs multiple times per trial, while MetaPrompt/Few-Shot Bayesian are lighter weight. Start with small `n_samples` and `max_trials`, monitor API usage, and review the [Benchmarks](/development/optimization-runs/algorithms/benchmarks) page for sample-efficiency notes.
Optimizers tune prompts for the dataset you provide. Prompts may overfit if the dataset lacks coverage. Use diverse datasets, consider chaining optimizers (e.g., Evolutionary → Few-Shot Bayesian) to encourage generalization, and re-evaluate on unseen samples before shipping.
To prevent overfitting, use the `validation_dataset` parameter when calling `optimize_prompt()`. This separates your data into training (for generating prompt improvements) and validation (for evaluating candidate prompts). The optimizer will use the training dataset to understand failure modes and generate improvements, then rank candidates using the validation dataset. This mirrors standard ML practices and helps ensure your optimized prompt generalizes to unseen data.
```python
result = optimizer.optimize_prompt(
prompt=my_prompt,
dataset=training_dataset, # Used for analysis and improvements
validation_dataset=validation_dataset, # Used for ranking candidates
metric=my_metric,
max_trials=5,
)
```
If you don't provide a validation dataset, the optimizer will use the same dataset for both training and validation, which may lead to overfitting on that specific dataset. For best results, split your data 70/30 or 80/20 between training and validation sets.
Yes—compose metrics using `MultiMetricObjective` or custom heuristics so optimizers weigh multiple goals. For complex trade-offs, capture human preferences in the metric reasons, or explore Pareto-aware optimizers like GEPA that surface trade-offs between accuracy and cost.
Opik supports optimizing prompts that handle images/videos, as well as agents built with LangGraph, Google ADK, or MCP toolchains. Use the [Optimize agents](/development/optimization-runs/optimization/optimize_agents) and [Optimize multimodal](/development/optimization-runs/optimization/optimize_multimodal) guides for modality-specific advice.
LLM-based metrics can be noisy. Always include deterministic checks when possible, and ensure your `ScoreResult.reason` is informative so reflective optimizers can identify true failure modes. See [Define metrics](/development/optimization-runs/optimization/define_metrics) and [Custom metrics](/development/optimization-runs/advanced/custom_metrics) for best practices.
Yes. When optimizing prompts that handle sensitive content, bake alignment constraints into your dataset/metrics (e.g., moderation scores) and review outputs manually before deployment. Multi-objective setups help enforce safety alongside accuracy.
## Next Steps
* Explore [API Reference](/development/optimization-runs/advanced/api_reference) for detailed technical documentation.
* Review the individual Optimizer pages under [Optimization Algorithms](/development/optimization-runs/algorithms/overview).
* Check out the [Quickstart Guide](/development/optimization-runs/quickstart).
# Changelog
> Release notes focused on Agent Optimizer features.
Stay current on Agent Optimizer updates. Each entry below reflects a version bump in `sdks/opik_optimizer/pyproject.toml` on `main`. The compare links jump straight to the commits that landed in that release. As Opik is a monorepo, you will see other non-optimizer related changes in the commit links below.
## Version 3.x
Highlights: Full public, production-ready release designed to work alongside Optimization Studio. MCP/tool optimization is temporarily removed in `3.0.0`–`3.0.1` due to compatibility reasons.
### Version 3 highlights
* First fully native multi-agent and multi-prompt optimization support end-to-end across optimizers.
* Specific reasoning models separate from training supported across all optimizers.
* Multimodal optimization supported across all optimizers.
* Pass\@k evaluation across all optimizers via the `n` parameter and selection policies.
* Full refactor and normalization of all optimizers for simpler code and better performance.
| Date | Version | Highlights |
| -------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **2026-02-24** | `3.1.0` | Reintroduced MCP/tool optimization across optimizers, added optimizer benchmark runtime/engine refactor, improved multi-metric span wiring and fail-fast scoring behavior, and added Python 3.14 support. [Commits →](https://github.com/comet-ml/opik/compare/0bdabe06505db3cbe50fb12f8d0ee9d055448fef...7a7c15ff2593faad503ef18324ae730433fe9cd6) |
| **2026-01-28** | `3.0.1` | Suppress noisy Hugging Face notebook warnings, fix HotpotQA multihop agent with new LiteLLM calls + JSON mode, map missing error enum so UI shows “Error”, HRPO rich output fixes and performance improvements, sanitize attachment spans to prevent output errors, fix HRPO async in notebooks. |
| **2026-01-26** | `3.0.0` | Major rewrite: native multi-agent/multi-prompt optimization, reasoning model support, pass\@k evaluation, and optimizer refactor. Breaking change: MCP-based optimizers removed. [Commits →](https://github.com/comet-ml/opik/compare/105f3ef0340d3e2c7c2e8d342766576c3519f748...98434bb3374280054ef6d8902b0cdc887bab25aa) |
## Version 2.x
Highlights: Introduced new optimizers including HRPO and GEPA, tool optimization and expanding coverage beyond early prompt-only workflows. Breaking change includes dropping MIPRO in favour of GEPA.
| Date | Version | Highlights |
| -------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **2026-01-22** | `2.3.12` | Fixes for optimizer breaking issues. [Commits →](https://github.com/comet-ml/opik/compare/a327fa3c2fb4f7a9fac2d9bba05c02c8cc234a1e...105f3ef0340d3e2c7c2e8d342766576c3519f748) |
| **2026-01-06** | `2.3.11` | Fixed hierarchical optimizer reason handling and numeric equals. [Commits →](https://github.com/comet-ml/opik/compare/fe02947b0843f28d8f0e67eee55e6ba377a3cc42...a327fa3c2fb4f7a9fac2d9bba05c02c8cc234a1e) |
| **2025-12-24** | `2.3.10` | Added missing model parameter in HRPO. [Commits →](https://github.com/comet-ml/opik/compare/2735ea6fcaf06d1dd275fbe0ef8297b0d5e5f490...fe02947b0843f28d8f0e67eee55e6ba377a3cc42) |
| **2025-12-19** | `2.3.9` | Fixed hierarchical optimizer recreating optimizations when ID is provided. [Commits →](https://github.com/comet-ml/opik/compare/2f51ca37d072b91e0c6e1611884e254bfa95556a...2735ea6fcaf06d1dd275fbe0ef8297b0d5e5f490) |
| **2025-12-15** | `2.3.8` | Fixed Evolutionary Optimizer with OpenAI Responses API, added validation in GEPA, updated multimodal example. [Commits →](https://github.com/comet-ml/opik/compare/f1cd349e30a1ec8672733f0219ec5304cf6e4bc3...2f51ca37d072b91e0c6e1611884e254bfa95556a) |
| **2025-12-15** | `2.3.7` | Renamed Hierarchical optimizer to HAPO, fixed MetaPrompt inheritance, lint cleanup. [Commits →](https://github.com/comet-ml/opik/compare/f98b169510ae185b269990d01995fe1594dbc6a5...f1cd349e30a1ec8672733f0219ec5304cf6e4bc3) |
| **2025-12-02** | `2.3.6` | Bumped opik\_optimizer, added optimization\_id skip, cost/latency metrics, GEPA API/log fixes. [Commits →](https://github.com/comet-ml/opik/compare/68b46e0af5cd362b1ba80a83f8187e6596680d78...f98b169510ae185b269990d01995fe1594dbc6a5) |
| **2025-11-24** | `2.3.5` | Version mismatch fix and Few-Shot Optimizer improvements. [Commits →](https://github.com/comet-ml/opik/compare/04e8bab7b96d7cf94551bfc41bd44620813a8a6b...68b46e0af5cd362b1ba80a83f8187e6596680d78) |
| **2025-11-24** | `2.3.4` | Optimizer tests, benchmarks, and MetaPrompt improvements. [Commits →](https://github.com/comet-ml/opik/compare/6a1bdba25642f4e5a1d3d54c4572b356b1d4f6b8...04e8bab7b96d7cf94551bfc41bd44620813a8a6b) |
| **2025-11-22** | `2.3.3` | Benchmarks refactor, fixed validation dataset support in Few-Shot Optimizer. [Commits →](https://github.com/comet-ml/opik/compare/5aa839b303c260b682f84693bbb60bbd10c7f5f3...6a1bdba25642f4e5a1d3d54c4572b356b1d4f6b8) |
| **2025-11-21** | `2.3.2` | Agent examples fixes. [Commits →](https://github.com/comet-ml/opik/compare/57aca7223afccabad482a9a1fcc9223f7e1d887f...5aa839b303c260b682f84693bbb60bbd10c7f5f3) |
| **2025-11-21** | `2.3.1` | Fixed validation datasets, refactored optimizers to dedicated prompts modules. [Commits →](https://github.com/comet-ml/opik/compare/eaea2994b870c8e587e01bc6cdc23832bb70e462...57aca7223afccabad482a9a1fcc9223f7e1d887f) |
| **2025-11-21** | `2.3.0` | Validation dataset support, better logging and eval traces, optimizer version bump. [Commits →](https://github.com/comet-ml/opik/compare/8e27945686005acf5e7c0e60e7fc0ce9ba58e4e8...eaea2994b870c8e587e01bc6cdc23832bb70e462) |
| **2025-11-21** | `2.2.7` | Cache/speed improvements for e2e and workflows, GEPA adapter fixes. [Commits →](https://github.com/comet-ml/opik/compare/aa39d0bab2b03d85de99cebc96275c497efffe84...8e27945686005acf5e7c0e60e7fc0ce9ba58e4e8) |
| **2025-11-20** | `2.2.6` | Refactored Datasets API. [Commits →](https://github.com/comet-ml/opik/compare/670dc8e0318719a9a133d527544babedd77671f9...aa39d0bab2b03d85de99cebc96275c497efffe84) |
| **2025-11-20** | `2.2.5` | Ensured OptimizableAgent support, minor fixes. [Commits →](https://github.com/comet-ml/opik/compare/15c079f39785320d8e7ca7f0353452ffc94d2be7...670dc8e0318719a9a133d527544babedd77671f9) |
| **2025-11-19** | `2.2.4` | Added multimodal optimization e2e test. [Commits →](https://github.com/comet-ml/opik/compare/edc874c1905fe3b2bc6ad532ae8d3f992567accd...15c079f39785320d8e7ca7f0353452ffc94d2be7) |
| **2025-11-19** | `2.2.3` | Manifest-based task support, extra benchmark datasets. [Commits →](https://github.com/comet-ml/opik/compare/7d545959ad198c3ba3a9474c5e9e18b4fe3b938d...edc874c1905fe3b2bc6ad532ae8d3f992567accd) |
| **2025-11-18** | `2.2.2` | Multimodal input support for HRPO, algorithms package refactor, custom run names. [Commits →](https://github.com/comet-ml/opik/compare/582646e0c8da8d8ea74c3424e1aca94c18cc3666...7d545959ad198c3ba3a9474c5e9e18b4fe3b938d) |
| **2025-11-08** | `2.2.1` | Removed cache, added MCP agent prompt, benchmarks, fixed GEPA e2e tests. [Commits →](https://github.com/comet-ml/opik/compare/54eeef28c7eb0ddfd25bd471c3e33d8c95d3a57f...582646e0c8da8d8ea74c3424e1aca94c18cc3666) |
| **2025-10-23** | `2.2.0` | Public method refactor and optimizer cleanup. [Commits →](https://github.com/comet-ml/opik/compare/ae4eb603337cd298013dfc54e3f3c3eb3084e102...54eeef28c7eb0ddfd25bd471c3e33d8c95d3a57f) |
| **2025-10-23** | `2.1.3` | Multi-metric optimization objectives, HRPO logging improvements, public method refactor. [Commits →](https://github.com/comet-ml/opik/compare/a0d049f8a45a010f8ae1f6eecc73ff4c2c1ba29f...ae4eb603337cd298013dfc54e3f3c3eb3084e102) |
| **2025-10-11** | `2.1.2` | Fixed missing optimizer names across metadata. [Commits →](https://github.com/comet-ml/opik/compare/04e0937e26bead34a70421057025bc182bf957e3...a0d049f8a45a010f8ae1f6eecc73ff4c2c1ba29f) |
| **2025-10-10** | `2.1.1` | Introduced the Hierarchical Reflective Optimizer. [Commits →](https://github.com/comet-ml/opik/compare/47354f6d41123acd45330a0e3969892f86281ece...04e0937e26bead34a70421057025bc182bf957e3) |
| **2025-10-09** | `2.1.0` | Pinned LiteLLM, shipped optimizer API docs, added parameter-only tuning. [Commits →](https://github.com/comet-ml/opik/compare/516ba494ffeeaf7ba9d49337f336ac7cd360a4f2...47354f6d41123acd45330a0e3969892f86281ece) |
| **2025-10-02** | `2.0.1` | Added GEPA dependency and test package init files. [Commits →](https://github.com/comet-ml/opik/compare/c2230e13a2d956635f588bc0a52df16421507b6f...516ba494ffeeaf7ba9d49337f336ac7cd360a4f2) |
| **2025-10-01** | `2.0.0` | Updated pyproject.toml for v2.0 release. [Commits →](https://github.com/comet-ml/opik/compare/69dd9e6e61d3cd8aabe30a1ef814a5a57d74ef50...c2230e13a2d956635f588bc0a52df16421507b6f) |
## Version 1.x
Highlights: Initial research release for testing early internal optimizers MetaPrompter, Evolutionary and support for MIPRO.
| Date | Version | Highlights |
| -------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **2025-10-01** | `1.1.0` | Optimizer MCP with EO, signature refactor, docs and dotfiles for optimizer release. [Commits →](https://github.com/comet-ml/opik/compare/67d147a8be9ad8ae70dea37fe2f649cda3a34ad3...69dd9e6e61d3cd8aabe30a1ef814a5a57d74ef50) |
| **2025-09-24** | `1.0.6` | Initial optimizer packaging cleanup. [Commit →](https://github.com/comet-ml/opik/commit/67d147a8be9ad8ae70dea37fe2f649cda3a34ad3) |
Looking for older updates? Browse the [full Opik changelog](/changelog) and filter by the **Agent Optimization** tag.
# Known Issues
> Known runtime issues when running Opik Optimizer with current dependencies.
In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a `project_name` when creating datasets and running experiments so they are associated with the correct project.
## Known Issues
If `pyrate-limiter` 4.x is installed you may see `TypeError: Limiter.__init__() got an unexpected keyword argument 'raise_when_fail'`. That version dropped the legacy flag our optimizer still passes.
**Workaround**: pin `pyrate-limiter` to a 3.x release:
```bash
pip install "pyrate-limiter>=3.0.0,<4.0.0"
```
**Fixed in**: `3.0.0` (2026-01-26). Upgrade the SDK to remove the legacy flag entirely.
`convert_tqdm_to_rich.
The optimizer is open-source, you can check out the code in the
[Opik repository](https://github.com/comet-ml/opik/tree/main/sdks/opik_optimizer/src/opik_optimizer/algorithms/meta_prompt_optimizer).
## Quickstart
You can use the `MetaPromptOptimizer` to optimize a prompt by following these steps:
```python maxLines=1000
from opik_optimizer import MetaPromptOptimizer
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer import datasets, ChatPrompt
# Initialize optimizer
optimizer = MetaPromptOptimizer(
model="openai/gpt-4",
model_parameters={
"temperature": 0.1,
"max_tokens": 5000
},
n_threads=8,
seed=42
)
# Prepare dataset
dataset = datasets.hotpot(count=300)
# Define metric and task configuration (see docs for more options)
def levenshtein_ratio(dataset_item, llm_output):
return LevenshteinRatio().score(reference=dataset_item['answer'], output=llm_output)
prompt = ChatPrompt(
messages=[
{"role": "system", "content": "Provide an answer to the question."},
{"role": "user", "content": "{question}"}
]
)
# Run optimization
results = optimizer.optimize_prompt(
prompt=prompt,
dataset=dataset,
metric=levenshtein_ratio,
n_samples=100
)
# Access results
results.display()
```
## Configuration Options
### Optimizer parameters
The optimizer has the following parameters:
LiteLLM model name for optimizer's internal reasoning/generation calls
Optional dict of LiteLLM parameters for optimizer's internal LLM calls. Common params: temperature, max\_tokens, max\_completion\_tokens, top\_p.
Number of candidate prompts to generate per optimization round
Whether to include task-specific context when reasoning about improvements
Number of parallel threads for prompt evaluation
Controls internal logging/progress bars (0=off, 1=on)
Random seed for reproducibility
### `optimize_prompt` parameters
The `optimize_prompt` method has the following parameters:
The ChatPrompt to optimize. Can include system/user/assistant messages, tools, and model configuration.
Opik Dataset containing evaluation examples. Each item is passed to the prompt during evaluation.
Evaluation function that takes (dataset\_item, llm\_output) and returns a score (float). Higher scores indicate better performance.
Optional metadata dictionary to log with Opik experiments. Useful for tracking experiment parameters and context.
Number of dataset items to use per evaluation. Use counts (e.g., `50`), fractions (e.g., `0.1`), percentages (e.g., "10%"), or "all"/"full"/None for the full dataset.
Optional number of samples for inner-loop minibatches (defaults to n\_samples).
Sampling strategy for subsampling (default: "random\_sorted").
If True, optimizer may continue beyond max\_trials if improvements are still being found.
Custom agent class for prompt execution. If None, uses default LiteLLM-based agent. Must inherit from OptimizableAgent.
Opik project name for logging traces and experiments. Default: "Optimization"
Maximum total number of prompts to evaluate across all rounds. Optimizer stops when this limit is reached.
Optional MCP (Model Context Protocol) execution configuration for prompts that use external tools. Enables tool-calling workflows. Default: None
Optional custom function to generate candidate prompts. Overrides default meta-reasoning generator. Should return list\[ChatPrompt].
Optional kwargs to pass to candidate\_generator.
### Model Support
There are two models to consider when using the `MetaPromptOptimizer`:
* `MetaPromptOptimizer.model`: The model used for the reasoning and candidate generation.
* `ChatPrompt.model`: The model used to evaluate the prompt.
The `model` parameter accepts any LiteLLM-supported model string (e.g., `"gpt-4o"`, `"azure/gpt-4"`,
`"anthropic/claude-3-opus"`, `"gemini/gemini-1.5-pro"`). You can also pass in extra model parameters
using the `model_parameters` parameter:
```python
optimizer = MetaPromptOptimizer(
model="anthropic/claude-3-opus-20240229",
model_parameters={
"temperature": 0.7,
"max_tokens": 4096
}
)
```
## MCP Tool Calling Support
The MetaPrompt Optimizer is the only optimizer that currently supports **MCP (Model Context
Protocol) tool calling optimization**. This means you can optimize prompts that include MCP tools
and function calls.
MCP tool calling optimization is a specialized feature that allows the optimizer to understand and optimize prompts
that use external tools and functions through the Model Context Protocol. This is particularly useful for complex
agent workflows that require tool usage.
For comprehensive information about tool optimization, see the [Tool Optimization Guide](/development/optimization-runs/algorithms/tool_optimization).
## Research and References
* [Meta-Prompting for Language Models](https://arxiv.org/abs/2401.12954)
# HRPO (Hierarchical Reflective Prompt Optimizer)
> Learn how to use HRPO (Hierarchical Reflective Prompt Optimizer) to improve prompts through systematic root cause analysis of failure modes and targeted refinement.
`HRPO` (Hierarchical Reflective Prompt Optimizer) uses hierarchical root cause analysis to identify and address
specific failure modes in your prompts. It analyzes evaluation results, identifies patterns in
failures, and generates targeted improvements to address each failure mode systematically.
In code, use the `HRPO` class (for example `from opik_optimizer import HRPO`). The underlying module name still uses
`hierarchical_reflective_optimizer` for backwards compatibility.
`HRPO` is ideal when you have a complex prompt that you want to refine
based on understanding *why* it's failing. Unlike optimizers that generate many random variations,
this optimizer systematically analyzes failures, identifies root causes, and makes surgical
improvements to address each specific issue.
## How It Works
HRPO (Hierarchical Reflective Prompt Optimizer) has been developed by the Opik team to improve prompts that
might have already gone through a few rounds of manual prompt engineering. It focuses on identifying
why a prompt is failing and then updating the prompts to address the issues.
As datasets can be large, we split the analysis into batches and analyze them in parallel. We then
synthesize the findings across all batches to identify the core issues with the prompt.
The optimizer is open-source, you can check out the root cause analysis code and prompts in the
[Opik repository](https://github.com/comet-ml/opik/tree/main/sdks/opik_optimizer/src/opik_optimizer/algorithms/hierarchical_reflective_optimizer).
## Quickstart
You can use `HRPO` to optimize a prompt:
```python maxLines=1000
from opik_optimizer import HRPO, ChatPrompt, datasets
from opik.evaluation.metrics.score_result import ScoreResult
# 1. Define your evaluation dataset
dataset = datasets.hotpot(count=300) # or use your own dataset
# 2. Configure the evaluation metric (MUST return reasons!)
def answer_quality_metric(dataset_item, llm_output):
reference = dataset_item.get("answer", "")
# Your scoring logic
is_correct = reference.lower() in llm_output.lower()
score = 1.0 if is_correct else 0.0
# IMPORTANT: Provide detailed reasoning
if is_correct:
reason = f"Output contains the correct answer: '{reference}'"
else:
reason = f"Output does not contain expected answer '{reference}'. Output was too vague or incorrect."
return ScoreResult(
name="answer_quality",
value=score,
reason=reason # Critical for root cause analysis!
)
# 3. Define your initial prompt
initial_prompt = ChatPrompt(
project_name="reflective_optimization",
messages=[
{
"role": "system",
"content": "You are a helpful assistant that answers questions accurately."
},
{
"role": "user",
"content": "Question: {question}\n\nProvide a concise answer."
}
]
)
# 4. Initialize HRPO
optimizer = HRPO(
model="gpt-4o",
n_threads=8,
max_parallel_batches=5,
seed=42,
model_parameters={"temperature": 0.7}
)
# 5. Run the optimization
optimization_result = optimizer.optimize_prompt(
prompt=initial_prompt,
dataset=dataset,
metric=answer_quality_metric,
n_samples=100,
max_trials=5,
max_retries=2
)
# 6. View the results
optimization_result.display()
```
## Configuration Options
### Optimizer parameters
The optimizer has the following parameters:
LiteLLM model name for optimizer's internal reasoning/generation calls
Controls internal logging/progress bars (0=off, 1=on).
Random seed for reproducibility (default: 42)
Optional dict of LiteLLM parameters for optimizer's internal LLM calls.
### `optimize_prompt` parameters
The `optimize_prompt` method has the following parameters:
Opik dataset name, or Opik dataset
A metric function, this function should have two arguments:
Number of dataset items to use per evaluation. Use counts (e.g., `50`), fractions (e.g., `0.1`), percentages (e.g., "10%"), or "all"/"full"/None for the full dataset.
Optional number of samples for inner-loop minibatches (defaults to n\_samples).
Sampling strategy for subsampling (default: "random\_sorted").
### Model Support
There are two models to consider when using `HRPO`:
* `HRPO.model`: The model used for the root cause analysis and failure mode synthesis.
* `ChatPrompt.model`: The model used to evaluate the prompt.
The `model` parameter accepts any LiteLLM-supported model string (e.g., `"gpt-4o"`, `"azure/gpt-4"`,
`"anthropic/claude-3-opus"`, `"gemini/gemini-1.5-pro"`). You can also pass in extra model parameters
using the `model_parameters` parameter:
```python
optimizer = HRPO(
model="anthropic/claude-3-opus-20240229",
model_parameters={
"temperature": 0.7,
"max_tokens": 4096
}
)
```
## Next Steps
1. Explore specific [Optimizers](/development/optimization-runs/algorithms/overview) for algorithm details.
2. Refer to the [FAQ](/development/optimization-runs/faq) for common questions and troubleshooting.
3. Refer to the [API Reference](/development/optimization-runs/advanced/api_reference) for detailed configuration options.
# Few-Shot Bayesian Optimizer
> Learn how to use the Few-Shot Bayesian Optimizer to find optimal few-shot examples for your chat-based prompts using Bayesian optimization techniques.
The FewShotBayesianOptimizer is a sophisticated prompt optimization tool adds relevant examples from
your sample questions to the system prompt using Bayesian optimization techniques.
The `FewShotBayesianOptimizer` is a strong choice when your primary goal is to find the optimal number and
combination of few-shot examples (demonstrations) to accompany your main instruction prompt,
particularly for **chat models**. If your task performance heavily relies on the quality and relevance of in-context examples, this optimizer is ideal.
## How It Works
The `FewShotBayesianOptimizer` uses Bayesian optimization to find the optimal set and number of
few-shot examples to include with your base instruction prompt for chat models. It Uses
[Optuna](https://optuna.org/), a hyperparameter optimization framework, to guide the search for the
optimal set and number of few-shot examples.
## Quickstart
You can use the `FewShotBayesianOptimizer` to optimize a prompt by following these steps:
```python maxLines=1000
from opik_optimizer import FewShotBayesianOptimizer
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer import datasets, ChatPrompt
# Initialize optimizer
optimizer = FewShotBayesianOptimizer(
model="openai/gpt-4",
model_parameters={
"temperature": 0.1,
"max_tokens": 5000
},
)
# Prepare dataset
dataset = datasets.hotpot(count=300)
# Define metric and prompt (see docs for more options)
def levenshtein_ratio(dataset_item, llm_output):
return LevenshteinRatio().score(reference=dataset_item["answer"], output=llm_output)
prompt = ChatPrompt(
messages=[
{"role": "system", "content": "Provide an answer to the question."},
{"role": "user", "content": "{question}"}
]
)
# Run optimization
results = optimizer.optimize_prompt(
prompt=prompt,
dataset=dataset,
metric=levenshtein_ratio,
n_samples=100
)
# Access results
results.display()
```
## Configuration Options
### Optimizer parameters
The optimizer has the following parameters:
LiteLLM model name for optimizer's internal reasoning (generating few-shot templates)
Optional dict of LiteLLM parameters for optimizer's internal LLM calls. Common params: temperature, max\_tokens, max\_completion\_tokens, top\_p.
Minimum number of examples to include in the prompt
Maximum number of examples to include in the prompt
Number of threads for parallel evaluation
Controls internal logging/progress bars (0=off, 1=on)
Random seed for reproducibility
### `optimize_prompt` parameters
The `optimize_prompt` method has the following parameters:
The prompt to optimize
Opik Dataset to optimize on
Metric function to evaluate on
Optional configuration for the experiment, useful to log additional metadata
Number of dataset items to use per evaluation. Use counts (e.g., `50`), fractions (e.g., `0.1`), percentages (e.g., "10%"), or "all"/"full"/None for the full dataset.
Optional number of samples for inner-loop minibatches (defaults to n\_samples).
Sampling strategy for subsampling (default: "random\_sorted").
Whether to auto-continue optimization
Optional agent class to use
Opik project name for logging traces (default: "Optimization")
Number of trials for Bayesian Optimization (default: 10)
### Model Support
There are two models to consider when using the `FewShotBayesianOptimizer`:
* `FewShotBayesianOptimizer.model`: The model used to generate the few-shot template and placeholder.
* `ChatPrompt.model`: The model used to evaluate the prompt.
The `model` parameter accepts any LiteLLM-supported model string (e.g., `"gpt-4o"`, `"azure/gpt-4"`,
`"anthropic/claude-3-opus"`, `"gemini/gemini-1.5-pro"`). You can also pass in extra model parameters
using the `model_parameters` parameter:
```python
optimizer = FewShotBayesianOptimizer(
model="anthropic/claude-3-opus-20240229",
model_parameters={
"temperature": 0.7,
"max_tokens": 4096
}
)
```
## Next Steps
1. Explore specific [Optimizers](/development/optimization-runs/algorithms/overview) for algorithm details.
2. Refer to the [FAQ](/development/optimization-runs/faq) for common questions and troubleshooting.
3. Refer to the [API Reference](/development/optimization-runs/advanced/api_reference) for detailed configuration options.
# Evolutionary Optimizer: Genetic Algorithms
> Learn how to use the Evolutionary Optimizer to discover optimal prompts through genetic algorithms, with support for multi-objective optimization and LLM-driven genetic operations.
The `EvolutionaryOptimizer` uses genetic algorithms to refine and discover effective prompts. It
iteratively evolves a population of prompts, applying selection, crossover, and mutation operations
to find prompts that maximize a given evaluation metric. This optimizer can also perform
multi-objective optimization (e.g., maximizing score while minimizing prompt length) and leverage
LLMs for more sophisticated genetic operations.
`EvolutionaryOptimizer` is a great choice when you want to explore a very diverse range of prompt
structures or when you have multiple objectives to optimize for (e.g., performance score and
prompt length). Its strength lies in its ability to escape local optima and discover novel prompt
solutions through its evolutionary mechanisms, especially when enhanced with LLM-driven genetic
operators.
## How It Works
The `EvolutionaryOptimizer` is built upon the [DEAP](https://deap.readthedocs.io/) library for
evolutionary computation. The core concept behind the optimizer is that we evolve a population of
prompts over multiple generations to find the best one.
We utilize different techniques to evolve the population of prompts:
* **Selection**: We select the best prompts from the population to be the parents of the next generation.
* **Crossover**: We crossover the parents to create the children of the next generation.
* **Mutation**: We mutate the children to create the new population of prompts.
We repeat this process for a number of generations until we find the best prompt.
The optimizer is open-source, you can check out the code in the
[Opik repository](https://github.com/comet-ml/opik/tree/main/sdks/opik_optimizer/src/opik_optimizer/algorithms/evolutionary_optimizer).
## Quickstart
You can use the `EvolutionaryOptimizer` to optimize a prompt:
```python maxLines=1000
from opik_optimizer import EvolutionaryOptimizer
from opik.evaluation.metrics import LevenshteinRatio # or any other suitable metric
from opik_optimizer import datasets, ChatPrompt
# 1. Define your evaluation dataset
dataset = datasets.tiny_test() # Replace with your actual dataset
# 2. Configure the evaluation metric
def levenshtein_ratio(dataset_item, llm_output):
return LevenshteinRatio().score(reference=dataset_item["label"], output=llm_output)
# 3. Define your base prompt and task configuration
initial_prompt = ChatPrompt(
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "{text}"}
]
)
# 4. Initialize the EvolutionaryOptimizer
optimizer = EvolutionaryOptimizer(
model="openai/gpt-4o-mini",
model_parameters={"temperature": 0.4},
population_size=20,
num_generations=10,
)
# 5. Run the optimization
optimization_result = optimizer.optimize_prompt(
prompt=initial_prompt,
dataset=dataset,
metric=levenshtein_ratio,
n_samples=5
)
# 6. View the results
optimization_result.display()
```
## Configuration Options
### Optimizer parameters
The optimizer has the following parameters:
### `optimize_prompt` parameters
The `optimize_prompt` method has the following parameters:
The prompt to optimize
The dataset to use for evaluation
Metric function to optimize with, should have the arguments `dataset_item` and `llm_output`
Optional experiment configuration
Number of dataset items to use per evaluation. Use counts (e.g., `50`), fractions (e.g., `0.1`), percentages (e.g., "10%"), or "all"/"full"/None for the full dataset.
Optional number of samples for inner-loop minibatches (defaults to n\_samples).
Sampling strategy for subsampling (default: "random\_sorted").
Whether to automatically continue optimization
Optional agent class to use
Opik project name for logging traces (default: "Optimization")
MCP tool calling configuration (default: None)
## Model Support
There are two models to consider when using the `EvolutionaryOptimizer`:
* `EvolutionaryOptimizer.model`: The model used for the evolution of the population of prompts.
* `ChatPrompt.model`: The model used to evaluate the prompt.
The `model` parameter accepts any LiteLLM-supported model string (e.g., `"gpt-4o"`, `"azure/gpt-4"`,
`"anthropic/claude-3-opus"`, `"gemini/gemini-1.5-pro"`). You can also pass in extra model parameters
using the `model_parameters` parameter:
```python
optimizer = EvolutionaryOptimizer(
model="anthropic/claude-3-opus-20240229",
model_parameters={
"temperature": 0.7,
"max_tokens": 4096
}
)
```
## Next Steps
1. Explore specific [Optimizers](/development/optimization-runs/algorithms/overview) for algorithm details.
2. Refer to the [FAQ](/development/optimization-runs/faq) for common questions and troubleshooting.
3. Refer to the [API Reference](/development/optimization-runs/advanced/api_reference) for detailed configuration options.
# GEPA Optimizer
> Use the external GEPA package through Opik's `GepaOptimizer` to optimize a single system prompt for single-turn tasks with a reflection model.
`GepaOptimizer` wraps the external [GEPA](https://github.com/gepa-ai/gepa) package to optimize a
single system prompt for single-turn tasks. It maps Opik datasets and metrics into GEPA’s expected
format, runs GEPA’s optimization using a task model and a reflection model, and returns a standard
`OptimizationResult` compatible with the Opik SDK.
`GepaOptimizer` is ideal when you have a single-turn task (one user input → one model
response) and you want to optimize the system prompt using a reflection-driven search.
## How it works
The GEPA optimizer companies two key approaches to optimize agents:
1. **Reflection**: The optimizer uses the outcomes from evaluations to improve the prompts.
2. **Evolution**: The optimizer uses an evolutionary algorithm to explore the space of prompts.
You can learn more about the algorithm in the [GEPA paper](https://arxiv.org/abs/2507.19457) but in
short, the optimizer will:
## Quickstart
```python
"""
Optimize a simple system prompt on the tiny_test dataset.
Requires: pip install gepa, and a valid OPENAI_API_KEY for LiteLLM-backed models.
"""
from typing import Any, Dict
from opik.evaluation.metrics import LevenshteinRatio
from opik.evaluation.metrics.score_result import ScoreResult
from opik_optimizer import ChatPrompt, datasets
from opik_optimizer.gepa_optimizer import GepaOptimizer
def levenshtein_ratio(dataset_item: Dict[str, Any], llm_output: str) -> ScoreResult:
return LevenshteinRatio().score(reference=dataset_item["label"], output=llm_output)
dataset = datasets.tiny_test()
prompt = ChatPrompt(
system="You are a helpful assistant. Answer concisely with the exact answer.",
user="{text}",
)
optimizer = GepaOptimizer(
model="openai/gpt-4o-mini",
n_threads=6,
model_parameters={"temperature": 0.2, "max_tokens": 200},
)
result = optimizer.optimize_prompt(
prompt=prompt,
dataset=dataset,
metric=levenshtein_ratio,
max_trials=12,
reflection_minibatch_size=2,
n_samples=5,
)
result.display()
```
### Determinism and tool usage
* GEPA’s seed is forwarded directly to the underlying `gepa.optimize` call, but any non-determinism in your prompt (tool calls, non-zero temperature, external APIs) will still introduce variance. To test seeding in isolation, disable tools or substitute cached responses.
* GEPA emits its own baseline evaluation inside the optimization loop. You’ll see one baseline score from Opik’s wrapper and another from GEPA before the first trial; this is expected and does not double-charge the metric budget.
* Reflection only triggers after GEPA accepts at least `reflection_minibatch_size` unique prompts. If the minibatch is larger than the trial budget, the optimizer logs a warning and skips reflection.
* GEPA supports **tool use during evaluation** (`allow_tool_use=True`) but does **not** support `optimize_tools=True` yet. Tool-description optimization requests are currently degraded/blocked until the adapter supports it.
### GEPA scores vs. Opik scores
* The **GEPA Score** column reflects the aggregate score GEPA computes on its train/validation split when deciding which candidates stay on the Pareto front. It is useful for understanding how GEPA’s evolutionary search ranks prompts.
* The **Opik Score** column is a fresh evaluation performed through Opik’s metric pipeline on the same dataset (respecting `n_samples`). This is the score you should use when comparing against your baseline or other optimizers.
* Because the GEPA score is based on GEPA’s internal aggregation, it can diverge from the Opik score for the same prompt. This is expected—treat the GEPA score as a hint about why GEPA kept or discarded a candidate, and rely on the Opik score for final comparisons.
### `skip_perfect_score`
* When `skip_perfect_score=True`, GEPA immediately ignores any candidate whose GEPA score meets or exceeds the `perfect_score` threshold (default `1.0`). This keeps the search moving toward imperfect prompts instead of spending budget refining already perfect ones.
* Set `skip_perfect_score=False` if your metric tops out below `1.0`, or if you still want to see how GEPA mutates a perfect-scoring prompt—for example, when you care about ties being broken by Opik’s rescoring step rather than GEPA’s aggregate.
## Configuration Options
### Optimizer parameters
The optimizer has the following parameters:
LiteLLM model name for the optimization algorithm
Optional dict of LiteLLM parameters for optimizer's internal LLM calls. Common params: temperature, max\_tokens, max\_completion\_tokens, top\_p.
Number of parallel threads for evaluation
Controls internal logging/progress bars (0=off, 1=on)
Random seed for reproducibility
### `optimize_prompt` parameters
The `optimize_prompt` method has the following parameters:
The prompt to optimize
Opik Dataset to optimize on
Metric function to evaluate on
Optional configuration for the experiment
Number of dataset items to use per evaluation. Use counts (e.g., `50`), fractions (e.g., `0.1`), percentages (e.g., "10%"), or "all"/"full"/None for the full dataset.
Optional number of samples for inner-loop minibatches (defaults to n\_samples).
Sampling strategy for subsampling (default: "random\_sorted").
Whether to auto-continue optimization
Optional agent class to use
Maximum number of different prompts to test (default: 10)
Size of reflection minibatches (default: 3)
Strategy for candidate selection (choose from "pareto", "current\_best", or "epsilon\_greedy"; default: "pareto")
Skip candidates with perfect scores (default: True)
Score considered perfect (default: 1.0)
Enable merge operations (default: False)
Maximum merge invocations (default: 5)
Directory for run outputs (default: None)
Track best outputs during optimization (default: False)
Display progress bar (default: False)
Random seed for reproducibility (default: 42)
Raise exceptions instead of continuing (default: True)
### Model Support
GEPA coordinates two model contexts:
* `GepaOptimizer.model`: LiteLLM model string the optimizer uses for internal reasoning (reflection, mutation prompts, etc.).
* `ChatPrompt.model`: The model evaluated against your dataset—this should match what you run in production.
Set `model` to any LiteLLM-supported provider (e.g., `"gpt-4o"`, `"azure/gpt-4"`, `"anthropic/claude-3-opus"`, `"gemini/gemini-1.5-pro"`) and pass extra parameters via `model_parameters` when you need to tune temperature, max tokens, or other limits:
```python
optimizer = GepaOptimizer(
model="anthropic/claude-3-opus-20240229",
model_parameters={
"temperature": 0.7,
"max_tokens": 4096
}
)
```
Reflection is handled internally; there is no separate `reflection_model` argument to set.
## Limitations & tips
* **Instruction-focused**: The current wrapper optimizes the instruction/system portion of your prompt. If you rely heavily on few-shot exemplars, consider pairing GEPA with the Few-Shot Bayesian optimizer or an Evolutionary run.
* **Reflection can misfire**: GEPA’s reflective mutations are only as good as the metric reasons you supply. If `ScoreResult.reason` is vague, the optimizer may reinforce bad behaviors. Invest in descriptive metrics before running GEPA at scale.
* **Cost-aware**: Although GEPA is more sample-efficient than some RL-based methods, reflection and Pareto scoring still consume multiple LLM calls per trial. Start with small `max_trials` and monitor API usage.
## Next Steps
1. Explore specific [Optimizers](/development/optimization-runs/algorithms/overview) for algorithm details.
2. Refer to the [FAQ](/development/optimization-runs/faq) for common questions and troubleshooting.
3. Refer to the [API Reference](/development/optimization-runs/advanced/api_reference) for detailed configuration options.
# Parameter Optimizer: Bayesian Parameter Tuning
> Learn how to use the Parameter Optimizer to find optimal LLM call parameters (temperature, top_p, etc.) using Bayesian optimization without changing your prompt.
The `ParameterOptimizer` uses Bayesian optimization to tune LLM call parameters such as temperature, top\_p, frequency\_penalty, and other sampling parameters. Unlike other optimizers that modify the prompt itself, this optimizer keeps your prompt unchanged and focuses solely on finding the best parameter configuration for your specific task.
**When to Use:** Optimize LLM parameters (temperature, top\_p) without changing your prompt. Best when you have a good prompt but need to tune model behavior.
**Key Trade-offs:** Requires defining parameter search space; doesn't modify prompt text; uses two-phase Bayesian search.
Have questions about `ParameterOptimizer`? Our [Optimizer & SDK FAQ](/development/optimization-runs/faq)
answers common questions, including when to use this optimizer, how parameters like `default_n_trials` and `local_search_ratio`
work, and how to define custom parameter search spaces.
## How It Works
This optimizer uses [Optuna](https://optuna.org/), a hyperparameter optimization framework, to search for the best LLM parameters:
1. **Baseline Evaluation**: First evaluates your prompt with its current parameters (or default parameters) to establish a baseline score.
2. **Parameter Space Definition**: You define which parameters to optimize and their valid ranges using a `ParameterSearchSpace`. For example:
* `temperature`: float between 0.0 and 2.0
* `top_p`: float between 0.0 and 1.0
* `frequency_penalty`: float between -2.0 and 2.0
3. **Global Search Phase**:
* Optuna explores the full parameter space using Bayesian optimization (TPESampler by default).
* Tries various parameter combinations to find promising regions.
* Evaluates each combination against your dataset using the specified metric.
4. **Local Search Phase** (optional):
* After global search, focuses on the best parameter region found.
* Performs fine-grained optimization around the best parameters.
* Controlled by `local_search_ratio` and `local_search_scale`.
5. **Parameter Importance Analysis**:
* Calculates which parameters had the most impact on performance.
* Uses FANOVA importance (requires scikit-learn) or falls back to correlation-based sensitivity analysis.
6. **Result**: Returns the best parameter configuration found, along with detailed optimization history and parameter importance rankings.
The optimizer intelligently balances exploration (trying diverse parameters) with exploitation (refining promising configurations) to efficiently find optimal settings.
The core of this optimizer relies on robust evaluation where each parameter configuration is assessed using your `metric` against the `dataset`. Understanding Opik's evaluation platform is key to effective use:
* [Evaluation Overview](/evaluation/overview)
* [Metrics Overview](/evaluation/metrics/overview)
## Configuration Options
### Basic Configuration
```python
from opik_optimizer import ParameterOptimizer
from opik_optimizer.algorithms.parameter_optimizer.parameter_search_space import (
ParameterSearchSpace,
)
optimizer = ParameterOptimizer(
model="openai/gpt-4",
default_n_trials=20, # Number of optimization trials
n_threads=4, # Parallel evaluation threads
seed=42
)
```
### Advanced Configuration
```python
optimizer = ParameterOptimizer(
model="openai/gpt-4",
default_n_trials=50, # More trials for thorough optimization
n_threads=8, # More parallel threads
local_search_ratio=0.3, # 30% of trials for local refinement
local_search_scale=0.2, # Scale of local search range
seed=42,
verbose=1 # Verbosity level (0=off, 1=info, 2=debug)
)
```
The key parameters are:
* `model`: The LLM used for evaluation with different parameter configurations.
* `default_n_trials`: Default number of optimization trials (can be overridden in `optimize_parameter`).
* `n_threads`: Number of parallel threads for evaluation (balance with API rate limits).
* `local_search_ratio`: Ratio of trials dedicated to local search around best parameters (0.0-1.0).
* `local_search_scale`: Scale factor for local search range (0.0 = no local search, higher = wider range).
* `seed`: Random seed for reproducibility.
* `verbose`: Logging level (0=warnings only, 1=info, 2=debug).
## Example Usage
### Basic Example
```python
from opik_optimizer import ParameterOptimizer, ChatPrompt
from opik_optimizer.algorithms.parameter_optimizer.parameter_search_space import (
ParameterSearchSpace,
)
from opik.evaluation.metrics import LevenshteinRatio
from opik_optimizer import datasets
# Initialize optimizer
optimizer = ParameterOptimizer(
model="openai/gpt-4o-mini",
default_n_trials=30,
n_threads=8,
seed=42
)
# Prepare dataset
dataset = datasets.hotpot(count=300)
# Define metric
def levenshtein_ratio(dataset_item, llm_output):
return LevenshteinRatio().score(reference=dataset_item["answer"], output=llm_output)
# Define prompt (this stays unchanged)
prompt = ChatPrompt(
project_name="parameter-optimization",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "{question}"}
]
)
# Define parameter search space
parameter_space = ParameterSearchSpace(
parameters=[
{
"name": "temperature",
"distribution": "float",
"low": 0.0,
"high": 2.0
},
{
"name": "top_p",
"distribution": "float",
"low": 0.1,
"high": 1.0
}
]
)
# Run optimization
results = optimizer.optimize_parameter(
prompt=prompt,
dataset=dataset,
metric=levenshtein_ratio,
parameter_space=parameter_space,
n_samples=100
)
# Access results
results.display()
print(f"Best temperature: {results.details['optimized_parameters']['temperature']}")
print(f"Best top_p: {results.details['optimized_parameters']['top_p']}")
print(f"Parameter importance: {results.details['parameter_importance']}")
```
### Advanced Example with Custom Parameters
```python
# Optimize more parameters including model selection
parameter_space = ParameterSearchSpace(
parameters=[
{
"name": "temperature",
"distribution": "float",
"low": 0.0,
"high": 1.5,
"step": 0.05 # Optional: quantize values
},
{
"name": "top_p",
"distribution": "float",
"low": 0.5,
"high": 1.0
},
{
"name": "frequency_penalty",
"distribution": "float",
"low": -1.0,
"high": 1.0
},
{
"name": "presence_penalty",
"distribution": "float",
"low": -1.0,
"high": 1.0
},
{
"name": "model",
"distribution": "categorical",
"choices": ["openai/gpt-4o-mini", "openai/gpt-4o", "openai/gpt-4-turbo"]
}
]
)
# Run with more trials and custom Optuna sampler
import optuna
results = optimizer.optimize_parameter(
prompt=prompt,
dataset=dataset,
metric=levenshtein_ratio,
parameter_space=parameter_space,
n_trials=100, # Override default_n_trials
n_samples=150,
sampler=optuna.samplers.TPESampler(seed=42, n_startup_trials=20)
)
```
## Parameter Search Space
The `ParameterSearchSpace` defines which parameters to optimize and their valid ranges. It supports:
### Float Parameters
```python
{
"name": "temperature",
"distribution": "float",
"low": 0.0,
"high": 2.0,
"step": 0.1, # Optional: quantize to 0.1 increments
"log": False # Optional: use log scale for sampling
}
```
### Integer Parameters
```python
{
"name": "max_tokens",
"distribution": "int",
"low": 100,
"high": 4000,
"step": 100, # Optional: sample in steps of 100
"log": False # Optional: use log scale
}
```
### Categorical Parameters
```python
{
"name": "model",
"distribution": "categorical",
"choices": ["gpt-4o-mini", "gpt-4o", "claude-3-haiku"]
}
```
### Boolean Parameters
```python
{
"name": "stream",
"distribution": "bool"
}
```
### Targeting Nested Parameters
You can optimize nested parameters in `model_parameters`:
```python
{
"name": "model_parameters.response_format.type",
"distribution": "categorical",
"choices": ["text", "json_object"]
}
```
## Model Support
The ParameterOptimizer supports all models available through LiteLLM. This provides broad
compatibility with providers like OpenAI, Azure OpenAI, Anthropic, Google, and many others,
including locally hosted models.
### Configuration Example using LiteLLM model string
```python
optimizer = ParameterOptimizer(
model="openai/gpt-4o-mini", # Using OpenAI via LiteLLM
default_n_trials=30,
n_threads=8
)
```
## Best Practices
1. **Start Simple**
* Begin with 1-2 key parameters (e.g., temperature, top\_p)
* Add more parameters once you understand their impact
* Too many parameters increases search space and trial requirements
2. **Define Reasonable Ranges**
* Use tighter ranges based on domain knowledge
* For temperature: 0.0-1.0 for factual tasks, 0.5-1.5 for creative tasks
* For top\_p: 0.8-1.0 for most tasks
3. **Trial Budget**
* Start with 20-30 trials for 2-3 parameters
* Increase to 50-100 trials for 4+ parameters
* Monitor convergence - stop if improvements plateau
4. **Local Search**
* Use `local_search_ratio=0.3` (default) for refinement
* Increase to 0.4-0.5 if global search found good region quickly
* Decrease to 0.1-0.2 for more exploration
5. **Parallel Evaluation**
* Set `n_threads` based on API rate limits
* More threads = faster optimization but may hit limits
* Balance speed with cost and rate limit constraints
6. **Parameter Importance**
* Check `parameter_importance` in results
* Focus future optimization on high-impact parameters
* Consider fixing low-impact parameters to reduce search space
7. **Validation**
* **Note**: While ParameterOptimizer accepts a `validation_dataset` parameter, it doesn't improve performance due to the optimizer's internal implementation and we don't recommend using it
* Instead, use `evaluate_prompt()` on a held-out test dataset after optimization completes
* This confirms that optimized parameters generalize to unseen data
* Example:
```python
# After optimization
from opik.evaluation import evaluate_prompt
test_dataset = client.get_dataset(name="test-set")
test_results = evaluate_prompt(
prompt=result.prompt,
dataset=test_dataset,
scoring_metrics=[my_metric],
project_name="my-project",
)
print(f"Test score: {test_results.mean_scores}")
```
## Research and References
* [Optuna: A hyperparameter optimization framework](https://optuna.org/)
## Next Steps
1. Explore specific [Optimizers](/development/optimization-runs/algorithms/overview) for algorithm details.
2. Refer to the [FAQ](/development/optimization-runs/faq) for common questions and troubleshooting.
3. Refer to the [API Reference](/development/optimization-runs/advanced/api_reference) for detailed configuration options.
# Tool Optimization (MCP & Function Calling)
> Learn how to optimize prompts that leverage MCP (Model Context Protocol) tools and function calling capabilities across supported optimizers.
Tool optimization is a specialized feature that allows you to optimize prompts that use external tools and the Model Context Protocol (MCP). This capability is supported by all optimizers **except** `FewShotBayesianOptimizer`, `ParameterOptimizer`, and `GepaOptimizer`.
## What is Tool Optimization?
Tool optimization extends traditional prompt optimization to handle prompts that include:
* **MCP tools** - Model Context Protocol tools for external integrations
* **Tool schemas** - Structured tool definitions and parameters
* **Multi-step workflows** - Complex agent workflows involving multiple tools
Tool optimization **does not** change tool names or schemas. It updates:
* Tool **descriptions**
* Tool **parameter descriptions**
**Important Distinction:** Many optimizers (including GEPA, MetaPrompt, etc.) can optimize **agents that use tools** -
this means optimizing prompts for agents that have access to function calling or external tools. However, **true tool
optimization** (optimizing the tool descriptions and parameter descriptions) is a separate capability you can enable
with `optimize_tools=True` across supported optimizers.
**Why Tool Optimization Matters:**
Traditional prompt optimization focuses on text-based prompts, but modern AI applications often
require:
* Integration with external APIs and services
* Structured data processing through function calls
* Complex multi-step reasoning with tool usage
* Dynamic tool selection based on context Tool optimization ensures these sophisticated prompts
can be improved just like simple text prompts.
## Supported Tool Types
### 1. Agent Function Calling (Not True Tool Optimization)
Many optimizers can optimize **agents that use function calling**, but this is different from true tool optimization. Here's an example from the GEPA optimizer:
```python
from opik_optimizer import GepaOptimizer, ChatPrompt
# GEPA example: optimizing an agent with function calling
prompt = ChatPrompt(
system="You are a helpful assistant. Use the search_wikipedia tool when needed.",
user="{question}",
tools=[
{
"type": "function",
"function": {
"name": "search_wikipedia",
"description": "This function searches Wikipedia abstracts.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "The term or phrase to search for."
}
},
"required": ["query"]
}
}
}
],
function_map={
"search_wikipedia": lambda query: search_wikipedia(query, use_api=True)
}
)
# GEPA optimizes the agent's prompt, not the tools themselves
optimizer = GepaOptimizer(model="gpt-5-nano")
result = optimizer.optimize_prompt(prompt=prompt, dataset=dataset, metric=metric)
```
This is **agent optimization** (optimizing prompts for agents that use tools), not **tool optimization** (optimizing
the tools themselves).
### 2. MCP (Model Context Protocol) Tools
**True tool optimization** is available for MCP tools and function-calling tools across supported optimizers. MCP tools provide standardized interfaces for external integrations:
```python
# MCP tool optimization example
# See scripts/litellm_metaprompt_context7_remote_example.py for a working example
from opik_optimizer import MetaPromptOptimizer
# MCP tools are configured as OpenAI-style entries (local or remote)
optimizer = MetaPromptOptimizer(model="gpt-5-nano")
# Any supported optimizer works here (e.g., HRPO, Evolutionary, MetaPrompt).
```
MCP Tool Optimization:
* Supported by all optimizers **except** FewShotBayesianOptimizer, ParameterOptimizer, and GepaOptimizer
* Working examples available in `scripts/litellm_metaprompt_context7_remote_example.py`
## How Tool Optimization Works
Supported optimizers handle tool-enabled prompts through a specialized optimization process:
### 1. Tool-Aware Analysis
The optimizer analyzes:
* **Tool schemas** - Understanding available functions and their parameters
* **Tool usage patterns** - How tools are typically invoked in the prompt
* **Tool dependencies** - Relationships between different tools
* **Context requirements** - What information tools need to function effectively
### 2. Prompt-Tool Integration Optimization
The optimizer can improve:
* **Tool selection logic** - Better instructions for when to use which tools
* **Parameter formatting** - Clearer guidance on how to structure tool inputs
* **Error handling** - Instructions for handling tool failures or edge cases
* **Tool chaining** - Optimizing multi-step tool workflows
### 3. Context Enhancement
Tool optimization also improves:
* **Input validation** - Better prompts for validating tool inputs
* **Output processing** - Instructions for handling tool outputs
* **Fallback strategies** - Alternative approaches when tools are unavailable
## Example: Optimizing a Research Assistant
Let's see how tool optimization works with a research assistant that uses multiple tools:
```python
from opik_optimizer import MetaPromptOptimizer, ChatPrompt
from opik.evaluation.metrics import LevenshteinRatio
# Define a research assistant prompt with tools
research_prompt = ChatPrompt(
messages=[
{
"role": "system",
"content": """You are a research assistant. When given a research question:
1. Search for relevant information using the search tool
2. Analyze the results using the analysis tool
3. Provide a comprehensive answer based on your findings
Always cite your sources and be thorough in your research."""
},
{
"role": "user",
"content": "{research_question}"
}
],
tools=[
{
"type": "function",
"function": {
"name": "search_academic_database",
"description": "Search academic papers and research",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"},
"year_range": {"type": "string"},
"max_results": {"type": "integer"}
}
}
}
},
{
"type": "function",
"function": {
"name": "analyze_text",
"description": "Analyze and summarize text content",
"parameters": {
"type": "object",
"properties": {
"text": {"type": "string"},
"analysis_type": {"type": "string"}
}
}
}
}
]
)
# Initialize the optimizer
optimizer = MetaPromptOptimizer(
model="openai/gpt-5-nano"
)
# Define evaluation metric
def research_quality_metric(dataset_item, llm_output):
return LevenshteinRatio().score(
reference=dataset_item['expected_answer'],
output=llm_output
)
# Run optimization
result = optimizer.optimize_prompt(
prompt=research_prompt,
dataset=research_dataset,
metric=research_quality_metric,
n_samples=100,
max_trials=5
)
print("Optimized prompt with tools:")
print(result.prompt)
```
## Best Practices for Tool Optimization
### 1. Tool Schema Design
* **Clear descriptions** - Provide detailed descriptions for each tool
* **Comprehensive parameters** - Include all necessary parameters with types
* **Example usage** - Add examples in tool descriptions when helpful
* **Error handling** - Define expected error conditions and responses
### 2. Prompt Structure
* **Tool introduction** - Clearly explain available tools to the model
* **Usage guidelines** - Provide specific instructions on when and how to use tools
* **Output formatting** - Specify how tool outputs should be processed
* **Fallback instructions** - Define what to do when tools fail
### 3. Evaluation Considerations
* **Tool usage metrics** - Measure not just final output quality but tool usage effectiveness
* **Multi-step evaluation** - Evaluate each step in tool-based workflows
* **Error rate tracking** - Monitor tool failure rates and recovery strategies
* **Context preservation** - Ensure important context is maintained across tool calls
## Limitations and Considerations
### Current Limitations
* **Optimizer coverage** - Tool optimization is not available in `FewShotBayesianOptimizer` or `ParameterOptimizer`
* **Tool Complexity** - Very complex tool workflows may require manual optimization
* **Tool Availability** - Optimization assumes tools are available during evaluation
* **Schema Changes** - Tool schema modifications may require re-optimization
### Performance Considerations
* **Evaluation Cost** - Tool-enabled prompts require more LLM calls for evaluation
* **Tool Latency** - External tool calls can slow down optimization
* **Resource Usage** - Complex tool workflows may require significant computational resources
## Future Roadmap
Tool optimization is an active area of development. Planned improvements include:
* **Tool-specific metrics** - Specialized evaluation metrics for tool usage
* **Automated tool discovery** - Automatic detection and optimization of tool patterns
* **Tool performance optimization** - Optimizing not just prompts but tool usage efficiency
## Getting Started
To start optimizing tool-enabled prompts:
1. **Choose a supported optimizer** - Any optimizer except FewShotBayesianOptimizer and ParameterOptimizer
2. **Define your tools** - Create clear tool schemas with comprehensive descriptions
3. **Structure your prompt** - Include clear instructions for tool usage
4. **Prepare evaluation data** - Ensure your dataset includes tool usage scenarios
5. **Run optimization** - Use the standard optimization process with tool-enabled prompts
**Need Help?**
For questions about tool optimization, please reach out on
[GitHub](https://github.com/comet-ml/opik/issues) or check the
[MetaPrompt Optimizer documentation](/development/optimization-runs/algorithms/metaprompt_optimizer) for detailed configuration options.
## References
* [MetaPrompt Optimizer Documentation](/development/optimization-runs/algorithms/metaprompt_optimizer)
* [Optimize tools (MCP)](/development/optimization-runs/optimization/optimize_tools)
* [Model Context Protocol Specification](https://modelcontextprotocol.io/)
# Synthetic Data Optimizer Cookbook
> Learn how to generate synthetic Q&A data from Opik traces and optimize prompts using the MetaPromptOptimizer through an interactive notebook.
This page is a high-level entry point for the synthetic data workflow. Use the notebook or SDK script to run the full example end-to-end.
## Launch the example
The notebook is the fastest way to explore synthetic data optimization in your browser.
| Platform | Launch Link |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Google Colab (Preferred)** | [](https://colab.research.google.com/github/comet-ml/opik/blob/main/sdks/opik_optimizer/notebooks/OpikSyntheticDataOptimizer.ipynb) |
| **GitHub** | [View the notebook on GitHub](https://github.com/comet-ml/opik/blob/main/sdks/opik_optimizer/notebooks/OpikSyntheticDataOptimizer.ipynb) |
## What this example covers
* Generating synthetic Q\&A datasets from Opik traces
* Using TinyQA (via [tinyqabenchmarkpp](https://pypi.org/project/tinyqabenchmarkpp/)) and variants like TinyQA++
* Optimizing prompts with MetaPrompt on synthetic data
* Reviewing results in the Opik UI
## Where the full implementation lives
Notebook: [`sdks/opik_optimizer/notebooks/OpikSyntheticDataOptimizer.ipynb`](https://github.com/comet-ml/opik/blob/main/sdks/opik_optimizer/notebooks/OpikSyntheticDataOptimizer.ipynb)
SDK codebase: browse `sdks/opik_optimizer/` for dataset utilities, metrics, and optimizer implementations.
## Next steps
* Run the notebook and swap in your own traces or datasets.
* Explore [Define datasets](/development/optimization-runs/optimization/define_datasets) and [Define metrics](/development/optimization-runs/optimization/define_metrics) for deeper control.
# ARC-AGI Optimization Tutorial
> End-to-end ARC-AGI optimization walkthrough with Opik Agent Optimizer.
This guide introduces ARC-AGI, why it is a strong fit for optimizer-driven prompt iteration, and where to find the full, runnable implementation in the SDK.
Codebase entry point: [`sdks/opik_optimizer/scripts/arc_agi/tasks_optimizer.py`](https://github.com/comet-ml/opik/blob/main/sdks/opik_optimizer/scripts/arc_agi/tasks_optimizer.py) and the ARC-AGI utilities in [`sdks/opik_optimizer/scripts/arc_agi/`](https://github.com/comet-ml/opik/tree/main/sdks/opik_optimizer/scripts/arc_agi).
## What is ARC-AGI?
ARC-AGI tasks are grid-based reasoning puzzles that test an agent’s ability to infer transformation rules from a few examples. They are a natural fit for optimization because small prompt changes can dramatically improve generalization across tasks.
## Why use optimizers here?
ARC-AGI evaluation is deterministic and repeatable, which makes it ideal for iterative optimization. HRPO is especially useful because it captures failure modes and proposes targeted fixes.
## How the SDK implementation works
The SDK ships a full ARC-AGI workflow you can run locally:
1. **Dataset loader**: [`sdks/opik_optimizer/src/opik_optimizer/datasets/arc_agi2.py`](https://github.com/comet-ml/opik/blob/main/sdks/opik_optimizer/src/opik_optimizer/datasets/arc_agi2.py) loads ARC-AGI-2 tasks and embeds optional grid images.
2. **Prompt templates**: `sdks/opik_optimizer/scripts/arc_agi/prompts/` contains system and HRPO prompt templates.
3. **Evaluator + metrics**: `sdks/opik_optimizer/scripts/arc_agi/utils/code_evaluator.py` executes candidate solvers and scores ARC-AGI metrics via `utils/metrics.py`.
4. **Optimizer wiring**: `tasks_optimizer.py` connects dataset, HRPO, metrics, and logging into a repeatable run.
If you want to run the code as-is, start with the `tasks_optimizer.py` entry point and follow the CLI flags listed at the top of that file.
## Next steps
* Explore the ARC-AGI scripts in the repo and swap in your own datasets or prompt templates.
* Review the run summaries under `scripts/arc_agi/` to compare optimizer iterations.
# Multimodal Agent Optimization Tutorial
> Multimodal agent optimization tutorial inspired by a self-driving car example.
This tutorial outlines how to optimize a multimodal agent (vision + text) and links to the full walkthrough for a self-driving car scenario. The SDK already includes a working example script and dataset you can run locally.
Full guide: [Automatic prompt optimization for multimodal vision agents (self-driving car example)](https://towardsdatascience.com/automatic-prompt-optimization-for-multimodal-vision-agents-a-self-driving-car-example/).
Codebase entry point: [`sdks/opik_optimizer/scripts/multimodal_example.py`](https://github.com/comet-ml/opik/blob/main/sdks/opik_optimizer/scripts/multimodal_example.py) using the driving hazard dataset in [`sdks/opik_optimizer/src/opik_optimizer/datasets/driving_hazard.py`](https://github.com/comet-ml/opik/blob/main/sdks/opik_optimizer/src/opik_optimizer/datasets/driving_hazard.py).
## What is the multimodal optimizer example?
The SDK includes a complete example that optimizes a vision agent on a driving hazard dataset. It demonstrates how to pass image content parts through `ChatPrompt`, score outputs, and compare trials in the Optimization Studio.
## Why use optimizers here?
Multimodal prompts are sensitive to phrasing and output structure. Running HRPO or MetaPrompt helps you converge on safer, more consistent outputs without rewriting prompts manually.
## How the SDK example works
1. `multimodal_example.py` loads the driving hazard dataset (images + hazard labels).
2. A multimodal `ChatPrompt` inserts an image URL content part next to the textual instruction.
3. The metric (Levenshtein ratio) scores predicted hazard text against the expected label.
4. HRPO optimizes the prompt using the training split, with a small validation split for ranking.
5. Results display in the Opik UI (Optimization runs and trial details).
Screenshot placeholder: multimodal trial comparisons and failure analysis.
## Next steps
* Explore the full SDK script and adapt the dataset to your own vision tasks.
* Use pass\@k evaluation (`n` parameter) to reduce stochastic failures.
* Read the full external guide for the complete workflow and visuals.
# Extending Optimizers
> Learn how to extend Opik's optimization framework with custom algorithms and contribute to the project's development.
Opik Agent Optimizer is designed to be a flexible framework for prompt and agent optimization. While
it provides a suite of powerful built-in algorithms, you might have unique optimization strategies or
specialized needs. This guide shows how to build your own optimizer by extending the `BaseOptimizer`
class that all built-in optimizers use.
## Architecture Overview
All optimizers in the SDK extend `BaseOptimizer`, giving you access to the same infrastructure:
```mermaid
classDiagram
class BaseOptimizer {
<reasoning: ...
synthesis: ...] end subgraph PromptLibrary M[Merge with overrides] V[Validate keys] F[Format templates] end O[prompt_overrides
dict or callable] D --> M O --> M M --> V --> F F --> G[optimizer.get_prompt] ``` ## Override Methods Best when you know exactly which prompt to replace with a static string: ```python from opik_optimizer import EvolutionaryOptimizer optimizer = EvolutionaryOptimizer( model="gpt-4o", prompt_overrides={ "synonyms_system_prompt": "Return exactly ONE synonym. No explanation.", "infer_style_system_prompt": "Analyze the writing style briefly.", } ) ``` Best when you need to modify existing prompts, apply conditional logic, or update multiple prompts: ```python from opik_optimizer import MetaPromptOptimizer from opik_optimizer.utils.prompt_library import PromptLibrary def customize_prompts(prompts: PromptLibrary) -> None: # List available keys print("Available keys:", prompts.keys()) # Prepend a constraint to the reasoning prompt original = prompts.get("reasoning_system") prompts.set("reasoning_system", "Always respond in English.\n\n" + original) # Append format instructions to another prompt if "candidate_generation" in prompts.keys(): prompts.set( "candidate_generation", prompts.get("candidate_generation") + "\n\nUse markdown formatting." ) optimizer = MetaPromptOptimizer( model="gpt-4o", prompt_overrides=customize_prompts ) ``` ## Discovering Available Keys Each optimizer has different prompt keys. Use `list_prompts()` to discover them: ```python from opik_optimizer import MetaPromptOptimizer optimizer = MetaPromptOptimizer(model="gpt-4o") print("Available prompt keys:") for key in optimizer.list_prompts(): print(f" - {key}") ``` ### Common Keys by Optimizer | Optimizer | Key Examples | | ----------------------------------- | ------------------------------------------------------------------------------------------------- | | **MetaPromptOptimizer** | `reasoning_system`, `candidate_generation`, `synthesis`, `pattern_extraction_system` | | **EvolutionaryOptimizer** | `infer_style_system_prompt`, `synonyms_system_prompt`, `semantic_mutation_system_prompt_template` | | **FewShotBayesianOptimizer** | `example_placeholder`, `system_prompt_template` | | **HierarchicalReflectiveOptimizer** | `batch_analysis_prompt`, `synthesis_prompt`, `improve_prompt_template` | ## Reading Prompts at Runtime After creating an optimizer, you can inspect the current prompts: ```python optimizer = MetaPromptOptimizer( model="gpt-4o", prompt_overrides={"reasoning_system": "Custom prompt here..."} ) # Get the current (possibly overridden) prompt current = optimizer.get_prompt("reasoning_system") print(current) # Get the original default (before any overrides) default = optimizer.prompts.get_default("reasoning_system") print(default) ``` ## Template Variables Some prompts contain placeholders that get filled at runtime using Python's `{variable}` format. When overriding prompts with placeholders, **keep the same placeholders**: ```python # Original: "Generate {num_prompts} variations of the prompt." # Your override should keep {num_prompts}: prompt_overrides = { "candidate_generation": "Be creative. Generate {num_prompts} diverse variations." } ``` ## Use Cases ```python def add_legal_constraints(prompts: PromptLibrary) -> None: for key in prompts.keys(): original = prompts.get(key) prompts.set(key, "LEGAL CONTEXT: Do not reference specific case law.\n\n" + original ) optimizer = MetaPromptOptimizer( model="gpt-4o", prompt_overrides=add_legal_constraints ) ``` ```python optimizer = EvolutionaryOptimizer( model="gpt-4o", prompt_overrides={ "infer_style_system_prompt": """ Analyze writing style. Return a JSON object with: { "tone": "formal|casual|technical", "complexity": "simple|moderate|complex", "key_patterns": ["list", "of", "patterns"] } """ } ) ``` ```python def add_safety_layer(prompts: PromptLibrary) -> None: safety_prefix = """ SAFETY REQUIREMENTS: - Never generate harmful or offensive content - Avoid personal identifiable information - Flag uncertain responses """ for key in prompts.keys(): if "system" in key.lower(): prompts.set(key, safety_prefix + prompts.get(key)) optimizer = MetaPromptOptimizer( model="gpt-4o", prompt_overrides=add_safety_layer ) ``` ## Error Handling The PromptLibrary validates keys to catch typos early: ```python # This will raise KeyError - "reasoing_system" is misspelled optimizer = MetaPromptOptimizer( model="gpt-4o", prompt_overrides={"reasoing_system": "Oops, typo!"} # KeyError! ) # Error message shows available keys: # KeyError: "Unknown prompt keys: ['reasoing_system']. # Available: ['candidate_generation', 'reasoning_system', ...]" ``` ## Best Practices **Tips for effective prompt customization:** 1. **List keys first** – Always use `list_prompts()` to see available keys before overriding 2. **Keep placeholders** – If a prompt has `{variables}`, keep them in your override 3. **Test incrementally** – Override one prompt at a time to isolate effects 4. **Use callable for complex logic** – Dict is simpler, but callable is more powerful 5. **Don't break JSON** – Some prompts expect JSON output; maintain that structure ## Full Example ```python from opik_optimizer import MetaPromptOptimizer from opik_optimizer.utils.prompt_library import PromptLibrary def my_customizations(prompts: PromptLibrary) -> None: """Customize prompts for a code generation task.""" # 1. Add coding focus to reasoning prompts.set( "reasoning_system", "You are an expert code prompt engineer.\n\n" + prompts.get("reasoning_system") ) # 2. Enforce Python-specific patterns prompts.set( "candidate_generation", prompts.get("candidate_generation") + """ ADDITIONAL REQUIREMENTS: - Prompts should encourage well-documented code - Prefer type hints and docstrings - Emphasize error handling and edge cases """ ) # Create optimizer with customizations optimizer = MetaPromptOptimizer( model="gpt-4o", prompt_overrides=my_customizations ) # Verify customizations applied print("Customized reasoning prompt:") print(optimizer.get_prompt("reasoning_system")[:200] + "...") ``` ## Related * [API Reference](/development/optimization-runs/advanced/api_reference) – Full parameter documentation * [Custom metrics](/development/optimization-runs/advanced/custom_metrics) – Build specialized evaluation metrics * [Extending optimizers](/development/optimization-runs/advanced/extending_optimizers) – Create custom optimizer subclasses # Sampling controls > Use n_samples, n_samples_minibatch, and the n parameter to trade cost for stability and exploration. When optimizing prompts, there are two independent sampling layers you can control: * **Dataset subsampling**: choose which dataset rows are evaluated (`n_samples`, `n_samples_minibatch`, `n_samples_strategy`). * **Model sampling**: request multiple completions per row (`n` in `model_parameters`). Use both to balance cost, stability, and exploration. Available in Opik Optimizer `v3.0.0+`. ## Dataset subsampling (n\_samples) `n_samples` limits how many dataset rows are evaluated per trial. It applies to the **evaluation dataset** (the `validation_dataset` if provided, otherwise `dataset`). ```python result = optimizer.optimize_prompt( prompt=prompt, dataset=dataset, metric=metric, n_samples=50, ) ``` Notes: * `n_samples` accepts an integer, a fractional float, a percent string (e.g. `"10%"`), or the special values `"all"`, `"full"`, or `None`. * If `n_samples` is larger than the evaluation dataset size, the optimizer falls back to the full dataset and logs a warning. ## Deterministic subsampling (n\_samples\_strategy) `n_samples_strategy` controls *how* dataset rows are selected when `n_samples` is set. The default strategy is `"random_sorted"`, which: 1. Sorts dataset item IDs. 2. Shuffles them deterministically using the optimizer seed and evaluation phase. 3. Takes the first `n_samples` IDs. If your dataset items do not include IDs, the optimizer falls back to the dataset order. ```python result = optimizer.optimize_prompt( prompt=prompt, dataset=dataset, metric=metric, n_samples=50, n_samples_strategy="random_sorted", ) ``` Only `"random_sorted"` is supported today. Passing another strategy will raise a `ValueError`. ## Minibatch sampling (n\_samples\_minibatch) Some optimizers run *inner-loop* evaluations (for example, HRPO and GEPA). Use `n_samples_minibatch` to cap those inner evaluations without reducing the outer evaluation size. ```python result = optimizer.optimize_prompt( prompt=prompt, dataset=dataset, metric=metric, n_samples=200, n_samples_minibatch=25, ) ``` If `n_samples_minibatch` is not set, it defaults to `n_samples`. ## Explicit item selection (dataset\_item\_ids) For fully deterministic evaluations, you can pass an explicit list of dataset item IDs to `evaluate_prompt`. This bypasses the sampling strategy and is **mutually exclusive** with `n_samples`. ```python score = optimizer.evaluate_prompt( prompt=prompt, dataset=dataset, metric=metric, dataset_item_ids=["item-1", "item-2", "item-3"], ) ``` When debug logging is enabled, evaluation logs include the sampling mode and resolved dataset size. ## Multiple completions per example (n parameter) Single-sample evaluation can be noisy. The `n` parameter lets you generate **multiple candidate outputs per example** and select the best one, introducing variety and reducing evaluation variance. ### How It Works When you set `n > 1` in your prompt's `model_parameters`, the optimizer: 1. Requests N completions from the LLM in a single API call (pass\@N) 2. Scores each candidate output using your metric 3. Selects the best candidate (`best_by_metric` policy) 4. Logs all scores and selection info to the Opik trace In optimizers that already generate multiple prompt variants per round, `n` is applied per evaluation, so total candidate evaluations scale by `prompts_per_round * n`. For tasks that execute generated code (like ARC-AGI or tool-driven agents), this means each prompt produces multiple candidate programs that are executed and scored, and the best candidate is used for optimization feedback. ```mermaid sequenceDiagram participant P as ChatPrompt participant L as LLM participant E as Evaluator participant T as Opik Trace P->>L: Request with n=3 L-->>P: [choice_0, choice_1, choice_2] P->>E: Score each candidate Note over E: choice_0 → 0.7 Note over E: choice_1 → 0.9 (best) Note over E: choice_2 → 0.6 E-->>P: Select choice_1 E->>T: Log scores & chosen_index ``` ### Configuration Set the `n` parameter in your `ChatPrompt.model_parameters`: ```python from opik_optimizer import ChatPrompt # Generate 3 candidates per evaluation, select best prompt = ChatPrompt( model="gpt-4o-mini", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Answer: {question}"}, ], model_parameters={ "n": 3, # Generate 3 completions per call "temperature": 0.7, # Higher temp = more variety between candidates }, ) ``` Higher `temperature` values increase diversity between the N candidates. Consider using `temperature: 0.7-1.0` with `n > 1` to maximize variety. The low-level `call_model` and `call_model_async` helpers return a single response unless you pass `return_all=True`. Optimizers handle `n` internally, so you only need `return_all` when calling those helpers directly. ### Use Cases Single-sample evaluation is noisy. With `n=3`, the optimizer scores each candidate and uses the best result, which makes optimization more robust to stochastic failures. ```python # Before: Single sample - noisy evaluation prompt = ChatPrompt(model="gpt-4o-mini", messages=[...]) # Score might be 0.6 or 0.9 depending on luck # After: Best-of-3 - more stable evaluation prompt = ChatPrompt( model="gpt-4o-mini", messages=[...], model_parameters={"n": 3, "temperature": 0.8}, ) # Score reflects best achievable output ``` Inspired by code generation benchmarks (pass\@k), this approach measures whether a prompt *can* produce correct output, not just whether it *usually* does. ```python # Optimize for "can this prompt ever get it right?" prompt = ChatPrompt( model="gpt-4o-mini", messages=[...], model_parameters={"n": 5}, # pass@5 style ) ``` This is useful when: * Correctness matters more than consistency * You'll use majority voting or best-of-k at inference time * Tasks have high variance (creative writing, complex reasoning) Some tasks naturally have multiple valid answers. Using `n > 1` helps the optimizer find prompts that can generate *any* valid answer. ```python # Creative task: multiple valid outputs prompt = ChatPrompt( model="gpt-4o-mini", messages=[ {"role": "user", "content": "Write a haiku about {topic}"}, ], model_parameters={"n": 3, "temperature": 1.0}, ) ``` ## Selection Policy Currently, the optimizer supports these selection policies: * `best_by_metric` (default): score each candidate with the metric and pick the best. * `first`: pick the first candidate (fast, deterministic, but ignores scoring). * `concat`: join all candidates into one output string. * `random`: pick a random candidate (seeded if provided). * `max_logprob`: pick the candidate with the highest average token logprob (provider support required; logprobs must be enabled in model kwargs). Use the `selection_policy` key in `model_parameters` to override. The optimizer routes these policies through a shared candidate-selection utility so behavior is consistent across optimizers: ```python prompt = ChatPrompt( model="gpt-4o-mini", messages=[...], model_parameters={ "n": 3, "selection_policy": "first", }, ) ``` For `max_logprob`, enable logprobs in your model kwargs (provider support varies): ```python prompt = ChatPrompt( model="gpt-4o-mini", messages=[...], model_parameters={ "n": 3, "selection_policy": "max_logprob", "logprobs": True, "top_logprobs": 1, }, ) ``` When `selection_policy=best_by_metric`, the optimizer: 1. Each candidate is scored independently using your metric function 2. The candidate with the highest score is selected as the final output 3. All scores and the chosen index are logged to the trace metadata ```python # What happens internally: candidates = ["output_1", "output_2", "output_3"] scores = [metric(item, c) for c in candidates] # [0.7, 0.9, 0.6] best_idx = argmax(scores) # 1 final_output = candidates[best_idx] # "output_2" ``` The trace metadata includes: * `n_requested`: Number of completions requested * `candidates_scored`: Number of candidates evaluated * `candidate_scores`: List of all scores (best\_by\_metric only) * `candidate_logprobs`: List of logprob scores (max\_logprob only) * `chosen_index`: Index of the selected candidate ## Cost Considerations Using `n > 1` increases API costs proportionally. With `n=3`, you pay roughly 3x the completion tokens per evaluation call. | n value | Relative cost | Variance reduction | | ------- | ------------- | ------------------ | | 1 | 1x | Baseline | | 3 | \~3x | Significant | # Multiple Completions (n parameter) > Learn how to use the n parameter to generate multiple candidate outputs per evaluation, reducing variance and improving optimization results through best-of-k selection. When optimizing prompts, single-sample evaluation can be noisy - a good prompt might fail on a particular trial due to LLM stochasticity. The `n` parameter lets you generate **multiple candidate outputs per evaluation** and select the best one, introducing variety and reducing evaluation variance. Available in Opik Optimizer `v3.0.0+`. ## How It Works When you set `n > 1` in your prompt's `model_parameters`, the optimizer requests N completions per evaluation, scores each candidate, selects the best one, and logs all scores to the trace. The full explanation of how the `n` parameter works is maintained in [Sampling controls](/development/optimization-runs/advanced/n_samples#multiple-completions-per-example-n-parameter). ```mermaid sequenceDiagram participant P as ChatPrompt participant L as LLM participant E as Evaluator participant T as Opik Trace P->>L: Request with n=3 L-->>P: [choice_0, choice_1, choice_2] P->>E: Score each candidate Note over E: choice_0 → 0.7 Note over E: choice_1 → 0.9 (best) Note over E: choice_2 → 0.6 E-->>P: Select choice_1 E->>T: Log scores & chosen_index ``` ## Configuration Set the `n` parameter in your `ChatPrompt.model_parameters`: ```python from opik_optimizer import ChatPrompt # Generate 3 candidates per evaluation, select best prompt = ChatPrompt( model="gpt-4o-mini", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Answer: {question}"}, ], model_parameters={ "n": 3, # Generate 3 completions per call "temperature": 0.7, # Higher temp = more variety between candidates }, ) ``` Higher `temperature` values increase diversity between the N candidates. Consider using `temperature: 0.7-1.0` with `n > 1` to maximize variety. The low-level `call_model` and `call_model_async` helpers return a single response unless you pass `return_all=True`. Optimizers handle `n` internally, so you only need `return_all` when calling those helpers directly. ## Use Cases Single-sample evaluation is noisy. With `n=3`, the optimizer scores each candidate and uses the best result, which makes optimization more robust to stochastic failures. ```python # Before: Single sample - noisy evaluation prompt = ChatPrompt(model="gpt-4o-mini", messages=[...]) # Score might be 0.6 or 0.9 depending on luck # After: Best-of-3 - more stable evaluation prompt = ChatPrompt( model="gpt-4o-mini", messages=[...], model_parameters={"n": 3, "temperature": 0.8}, ) # Score reflects best achievable output ``` Inspired by code generation benchmarks (pass\@k), this approach measures whether a prompt *can* produce correct output, not just whether it *usually* does. ```python # Optimize for "can this prompt ever get it right?" prompt = ChatPrompt( model="gpt-4o-mini", messages=[...], model_parameters={"n": 5}, # pass@5 style ) ``` This is useful when: * Correctness matters more than consistency * You'll use majority voting or best-of-k at inference time * Tasks have high variance (creative writing, complex reasoning) Some tasks naturally have multiple valid answers. Using `n > 1` helps the optimizer find prompts that can generate *any* valid answer. ```python # Creative task: multiple valid outputs prompt = ChatPrompt( model="gpt-4o-mini", messages=[ {"role": "user", "content": "Write a haiku about {topic}"}, ], model_parameters={"n": 3, "temperature": 1.0}, ) ``` ## Selection Policy Currently, the optimizer supports these selection policies: * `best_by_metric` (default): score each candidate with the metric and pick the best. * `first`: pick the first candidate (fast, deterministic, but ignores scoring). * `concat`: join all candidates into one output string. * `random`: pick a random candidate (seeded if provided). * `max_logprob`: pick the candidate with the highest average token logprob (provider support required; logprobs must be enabled in model kwargs). Use the `selection_policy` key in `model_parameters` to override. The optimizer routes these policies through a shared candidate-selection utility so behavior is consistent across optimizers: ```python prompt = ChatPrompt( model="gpt-4o-mini", messages=[...], model_parameters={ "n": 3, "selection_policy": "first", }, ) ``` For `max_logprob`, enable logprobs in your model kwargs (provider support varies): ```python prompt = ChatPrompt( model="gpt-4o-mini", messages=[...], model_parameters={ "n": 3, "selection_policy": "max_logprob", "logprobs": True, "top_logprobs": 1, }, ) ``` When `selection_policy=best_by_metric`, the optimizer: 1. Each candidate is scored independently using your metric function 2. The candidate with the highest score is selected as the final output 3. All scores and the chosen index are logged to the trace metadata ```python # What happens internally: candidates = ["output_1", "output_2", "output_3"] scores = [metric(item, c) for c in candidates] # [0.7, 0.9, 0.6] best_idx = argmax(scores) # 1 final_output = candidates[best_idx] # "output_2" ``` The trace metadata includes: * `n_requested`: Number of completions requested * `candidates_scored`: Number of candidates evaluated * `candidate_scores`: List of all scores (best\_by\_metric only) * `candidate_logprobs`: List of logprob scores (max\_logprob only) * `chosen_index`: Index of the selected candidate ## Cost Considerations Using `n > 1` increases API costs proportionally. With `n=3`, you pay roughly 3x the completion tokens per evaluation call. | n value | Relative cost | Variance reduction | | ------- | ------------- | ------------------ | | 1 | 1x | Baseline | | 3 | \~3x | Significant | | 5 | \~5x | High | | 10 | \~10x | Very high | **Recommendations:** * Start with `n=3` for most use cases * Use `n=5-10` only for high-variance tasks * Consider the total optimization budget when choosing N ## Limitations When `allow_tool_use=True` and tools are defined, the optimizer forces `n=1`. This is because tool-calling requires maintaining a coherent message thread, which isn't compatible with multiple independent completions. ```python # Tool-calling prompt - n will be forced to 1 prompt = ChatPrompt( model="gpt-4o-mini", messages=[...], tools=[...], model_parameters={"n": 3}, # Ignored when tools are used ) ``` Prompt synthesis steps that expect a single structured response (such as few-shot and parameter optimizers) ignore `n` to avoid returning multiple conflicting templates. Some LLM providers don't support the `n` parameter. Check your provider's documentation. LiteLLM will drop unsupported parameters automatically. ## Full Example ```python from opik_optimizer import ChatPrompt, MetaPromptOptimizer from opik.evaluation.metrics import LevenshteinRatio # Create prompt with n=3 for variety prompt = ChatPrompt( model="gpt-4o-mini", messages=[ {"role": "system", "content": "Extract the key entities from the text."}, {"role": "user", "content": "{text}"}, ], model_parameters={ "n": 3, # Generate 3 candidates "temperature": 0.7, # Moderate variety }, ) # Define metric def extraction_accuracy(dataset_item, llm_output): expected = dataset_item["expected_entities"] return LevenshteinRatio().score(expected, llm_output) # Optimize - each trial evaluates 3 candidates, picks best optimizer = MetaPromptOptimizer(model="gpt-4o") result = optimizer.optimize_prompt( prompt=prompt, dataset=my_dataset, metric=extraction_accuracy, ) print(f"Best prompt score: {result.score}") ``` ## Related * [Optimize prompts](/development/optimization-runs/optimization/optimize_prompts) - Core optimization guide * [Define metrics](/development/optimization-runs/optimization/define_metrics) - Create custom metrics * [Custom metrics](/development/optimization-runs/advanced/custom_metrics) - Advanced metric patterns * [API Reference](/development/optimization-runs/advanced/api_reference) - Full parameter documentation # Chaining optimizers > Run multiple optimizers in sequence to balance exploration and fine-tuning. Some projects benefit from running two or more optimizers back-to-back. For example, use MetaPrompt to improve wording, then Parameter optimizer to fine-tune sampling settings. This guide explains why you might chain runs, the trade-offs, and the APIs you use to pass prompts and metadata between stages. ## Strategy patterns | Pipeline | Why run it | Pros | Cons | Complexity | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ---------- | | HRPO → Parameter | Existing long/complex prompt scenario: HRPO's reflective analysis finds failure modes; Parameter optimizer then tightens parameters on the improved prompt. | Excellent for legacy prompts with lots of existing complexity. Helps produce explainable changelog. | Requires metrics with rich `reason` strings; two stages increase cost. | Medium | | Evolutionary → Few-Shot Bayesian | Cold-start scenario: explore many prompt architectures first, then let Few-Shot Bayesian pick the best example combination for the winning structure. | High diversity followed by precise example selection. | Evolutionary runs are expensive; Bayesian stage relies on curated datasets. | High | | MetaPrompt → Parameter | Baseline prompts need polish plus sampling tweaks. | Quick wins with minimal configuration; can run in under an hour. | Less insight into failure modes than reflective pipelines. | Low. | | Evolutionary → Parameter | Hunt for novel prompts, then squeeze out cost/latency by tuning temperature/top\_p. | Balances creativity with production readiness. | Two heavy optimization loops; ensure budget headroom. | High. | ## Example pipeline ```python from opik_optimizer import MetaPromptOptimizer, ParameterOptimizer, ChatPrompt from opik_optimizer.parameter_optimizer import ParameterSearchSpace meta = MetaPromptOptimizer(model="openai/gpt-4o") parameter = ParameterOptimizer(model="openai/gpt-4o") meta_result = meta.optimize_prompt( prompt=prompt, dataset=dataset, metric=metric, max_trials=4, ) # Reuse the optimized prompt from the first stage optimized_prompt = prompt.with_messages(meta_result.prompt) search_space = ParameterSearchSpace(parameters=[ {"name": "temperature", "distribution": "float", "low": 0.1, "high": 0.9}, {"name": "top_p", "distribution": "float", "low": 0.7, "high": 1.0}, ]) final_result = parameter.optimize_parameter( prompt=optimized_prompt, dataset=dataset, metric=metric, parameter_space=search_space, max_trials=20, ) ``` ## Checklist * **Freeze datasets and metrics** between stages to keep comparisons fair. * **Use validation datasets consistently** – if you provide a `validation_dataset` in the first stage, use the same split in subsequent stages to ensure fair comparison and avoid overfitting. * **Log pipeline metadata** (e.g., `experiment_config={"pipeline": "hierarchical_then_param"}`) so dashboards show lineage. * **Budget tokens** – chained runs multiply costs; start with smaller `n_samples` and increase once results look promising. * **Reuse OptimizationResult** – every optimizer returns an `OptimizationResult`, so you can pass `result.prompt` (and `result.details`, `result.history`) directly into the next stage without rebuilding state. ## Automation tips * Use Makefiles or CI workflows to run stage 1 → stage 2 with clear checkpoints. * Store intermediate prompts in version control alongside metadata (optimizer, score, dataset). * Notify stakeholders with summary reports generated from `final_result.history`. ## Related docs * [Optimize prompts](/development/optimization-runs/optimization/optimize_prompts) * [Few-Shot Bayesian optimizer](/development/optimization-runs/algorithms/fewshot_bayesian_optimizer) * [HRPO (Hierarchical Reflective Prompt Optimizer)](/development/optimization-runs/algorithms/hierarchical_adaptive_optimizer) * [Parameter optimizer](/development/optimization-runs/algorithms/parameter_optimizer) # Opik Agent Optimizer API Reference In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a `project_name` when creating datasets and running experiments so they are associated with the correct project. The Opik Agent Optimizer SDK provides a comprehensive set of tools for optimizing LLM prompts and agents. This reference guide documents the standardized API that all optimizers follow, ensuring consistency and interoperability across different optimization algorithms. ## Key Features * **Standardized API**: All optimizers follow the same interface for `optimize_prompt()` methods * **Multiple Algorithms**: Support for various optimization strategies including evolutionary, few-shot, meta-prompt, and GEPA * **MCP Support**: Built-in support for Model Context Protocol tool calling and optimization * **Consistent Results**: All optimizers return standardized `OptimizationResult` objects * **Counter Tracking**: Built-in LLM and tool call counters for monitoring usage * **Backward Compatibility**: All original parameters preserved through kwargs extraction * **Deprecation Warnings**: Clear warnings for deprecated parameters with migration guidance ## Core Classes The SDK provides several optimizer classes that all inherit from `BaseOptimizer` and implement the same standardized interface: * **ParameterOptimizer**: Optimizes LLM call parameters (temperature, top\_p, etc.) using Bayesian optimization * **FewShotBayesianOptimizer**: Uses few-shot learning with Bayesian optimization * **MetaPromptOptimizer**: Employs meta-prompting techniques for optimization * **EvolutionaryOptimizer**: Uses genetic algorithms for prompt evolution * **GepaOptimizer**: Leverages GEPA (Genetic-Pareto) optimization approach * **HRPO (Hierarchical Reflective Prompt Optimizer)**: Uses hierarchical root cause analysis for targeted prompt refinement ## Standardized Method Signatures All optimizers implement these core methods with identical signatures: ### optimize\_prompt() ```python def optimize_prompt( self, prompt: ChatPrompt | dict[str, ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: OptimizableAgent | None = None, experiment_config: dict | None = None, n_samples: int | None = None, auto_continue: bool = False, project_name: str | None = None, optimization_id: str | None = None, validation_dataset: Dataset | None = None, max_trials: int = 10, allow_tool_use: bool = True, **kwargs: Any, ) -> OptimizationResult ``` ## Deprecation Warnings The following parameters are deprecated and will be removed in future versions: ### Constructor Parameters * **`num_threads`** in optimizer constructors: Use `n_threads` instead ### Example Migration ```python # ❌ Deprecated optimizer = FewShotBayesianOptimizer( model="gpt-4o-mini", num_threads=16, # Deprecated ) # ✅ Correct optimizer = FewShotBayesianOptimizer( model="gpt-4o-mini", n_threads=16, # Use n_threads instead ) ``` ## FewShotBayesianOptimizer ```python FewShotBayesianOptimizer( model: str = 'openai/gpt-5-nano', model_parameters: dict[str, typing.Any] | None = None, min_examples: int = 2, max_examples: int = 8, n_threads: int = 12, verbose: int = 1, seed: int = 42, name: str | None = None, enable_columnar_selection: bool = True, enable_diversity: bool = True, enable_multivariate_tpe: bool = True, enable_optuna_pruning: bool = True, prompt_overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None, skip_perfect_score: bool = True, perfect_score: float = 0.95 ) ``` **Parameters:** ### Methods #### begin\_round ```python begin_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### cleanup ```python cleanup() ``` #### evaluate ```python evaluate( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, sampling_tag: str | None = None ) ``` **Parameters:** Optimization context for this run. Dict of named prompts to evaluate (e.g., \{"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry. Optional experiment configuration. Optional sampling tag for deterministic subsampling per candidate. #### evaluate\_prompt ```python evaluate_prompt( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, n_threads: int | None = None, verbose: int = 1, dataset_item_ids: list[str] | None = None, experiment_config: dict | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, seed: int | None = None, return_evaluation_result: bool = False, allow_tool_use: bool = False, use_evaluate_on_dict_items: bool | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### evaluate\_with\_result ```python evaluate_with_result( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, empty_score: float | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### finish\_candidate ```python finish_candidate( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### finish\_round ```python finish_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### get\_config ```python get_config( context: OptimizationContext ) ``` **Parameters:** #### get\_default\_prompt ```python get_default_prompt( key: str ) ``` **Parameters:** The prompt key to retrieve #### get\_history\_entries ```python get_history_entries() ``` #### get\_history\_rounds ```python get_history_rounds() ``` #### get\_metadata ```python get_metadata( context: OptimizationContext ) ``` **Parameters:** #### get\_optimizer\_metadata ```python get_optimizer_metadata() ``` #### get\_prompt ```python get_prompt( key: str, fmt: Any ) ``` **Parameters:** The prompt key to retrieve #### list\_prompts ```python list_prompts() ``` #### on\_trial ```python on_trial( context: OptimizationContext, prompts: dict, score: float, prev_best_score: float | None = None ) ``` **Parameters:** #### optimize\_mcp ```python optimize_mcp( args: Any, kwargs: Any ) ``` **Parameters:** #### optimize\_prompt ```python optimize_prompt( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, experiment_config: dict | None = None, n_samples: int | float | str | None = None, n_samples_minibatch: int | None = None, n_samples_strategy: str | None = None, auto_continue: bool = False, project_name: str | None = None, optimization_id: str | None = None, validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None, max_trials: int = 10, allow_tool_use: bool = True, optimize_prompts: bool | str | list[str] | None = 'system', optimize_tools: bool | dict[str, bool] | None = None, args: Any, kwargs: Any ) ``` **Parameters:** The prompt to optimize (single ChatPrompt or dict of prompts) Opik dataset (training set - used for feedback/context) TODO/FIXME: This parameter will be deprecated in favor of dataset\_training. For now, it serves as the training dataset parameter. A metric function with signature (dataset\_item, llm\_output) -> float Optional agent for prompt execution (defaults to LiteLLMAgent) Optional configuration for the experiment Number of samples to use for evaluation Optional number of samples for inner-loop minibatches Sampling strategy name (default "random\_sorted") Whether to continue optimization automatically Opik project name for logging traces (defaults to OPIK\_PROJECT\_NAME env or "Optimization") Optional ID to use when creating the Opik optimization run Optional validation dataset for ranking candidates Maximum number of optimization trials Whether tools may be executed during evaluation (default True) Which prompt roles to allow for optimization Optional tool optimization selector. Only supported by optimizers that explicitly document tool optimization support. #### post\_baseline ```python post_baseline( context: OptimizationContext, score: float ) ``` **Parameters:** #### post\_optimize ```python post_optimize( context: OptimizationContext, result: OptimizationResult ) ``` **Parameters:** #### post\_round ```python post_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### post\_trial ```python post_trial( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### pre\_baseline ```python pre_baseline( context: OptimizationContext ) ``` **Parameters:** #### pre\_optimize ```python pre_optimize( context: OptimizationContext ) ``` **Parameters:** The optimization context #### pre\_round ```python pre_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### pre\_trial ```python pre_trial( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### record\_candidate\_entry ```python record_candidate_entry( prompt_or_payload: Any, score: float | None = None, id: str | None = None, metrics: dict[str, typing.Any] | None = None, notes: str | None = None, extra: dict[str, typing.Any] | None = None, context: opik_optimizer.core.state.OptimizationContext | None = None ) ``` **Parameters:** #### run\_optimization ```python run_optimization( context: OptimizationContext ) ``` **Parameters:** The optimization context with prompts, dataset, metric, etc. #### set\_default\_dataset\_split ```python set_default_dataset_split( dataset_split: str | None ) ``` **Parameters:** #### set\_pareto\_front ```python set_pareto_front( pareto_front: list[dict[str, typing.Any]] | None ) ``` **Parameters:** #### set\_selection\_meta ```python set_selection_meta( selection_meta: dict[str, typing.Any] | None ) ``` **Parameters:** #### start\_candidate ```python start_candidate( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### with\_dataset\_split ```python with_dataset_split( dataset_split: str | None ) ``` **Parameters:** ## GepaOptimizer ```python GepaOptimizer( model: str = 'openai/gpt-5-nano', model_parameters: dict[str, typing.Any] | None = None, n_threads: int = 12, verbose: int = 1, seed: int = 42, name: str | None = None, skip_perfect_score: bool = True, perfect_score: float = 0.95, prompt_overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None ) ``` **Parameters:** ### Methods #### begin\_round ```python begin_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### cleanup ```python cleanup() ``` #### evaluate ```python evaluate( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, sampling_tag: str | None = None ) ``` **Parameters:** Optimization context for this run. Dict of named prompts to evaluate (e.g., \{"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry. Optional experiment configuration. Optional sampling tag for deterministic subsampling per candidate. #### evaluate\_prompt ```python evaluate_prompt( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, n_threads: int | None = None, verbose: int = 1, dataset_item_ids: list[str] | None = None, experiment_config: dict | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, seed: int | None = None, return_evaluation_result: bool = False, allow_tool_use: bool | None = None, use_evaluate_on_dict_items: bool | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### evaluate\_with\_result ```python evaluate_with_result( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, empty_score: float | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### finish\_candidate ```python finish_candidate( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### finish\_round ```python finish_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### get\_config ```python get_config( context: OptimizationContext ) ``` **Parameters:** #### get\_default\_prompt ```python get_default_prompt( key: str ) ``` **Parameters:** The prompt key to retrieve #### get\_history\_entries ```python get_history_entries() ``` #### get\_history\_rounds ```python get_history_rounds() ``` #### get\_metadata ```python get_metadata( context: OptimizationContext ) ``` **Parameters:** #### get\_optimizer\_metadata ```python get_optimizer_metadata() ``` #### get\_prompt ```python get_prompt( key: str, fmt: Any ) ``` **Parameters:** The prompt key to retrieve #### list\_prompts ```python list_prompts() ``` #### on\_trial ```python on_trial( context: OptimizationContext, prompts: dict, score: float, prev_best_score: float | None = None ) ``` **Parameters:** #### optimize\_mcp ```python optimize_mcp( args: Any, kwargs: Any ) ``` **Parameters:** #### optimize\_prompt ```python optimize_prompt( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, experiment_config: dict | None = None, n_samples: int | float | str | None = None, n_samples_minibatch: int | None = None, n_samples_strategy: str | None = None, auto_continue: bool = False, project_name: str | None = None, optimization_id: str | None = None, validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None, max_trials: int = 10, allow_tool_use: bool = True, optimize_prompts: bool | str | list[str] | None = 'system', optimize_tools: bool | dict[str, bool] | None = None, args: Any, kwargs: Any ) ``` **Parameters:** The prompt to optimize (single ChatPrompt or dict of prompts) Opik dataset (training set - used for feedback/context) TODO/FIXME: This parameter will be deprecated in favor of dataset\_training. For now, it serves as the training dataset parameter. A metric function with signature (dataset\_item, llm\_output) -> float Optional agent for prompt execution (defaults to LiteLLMAgent) Optional configuration for the experiment Number of samples to use for evaluation Optional number of samples for inner-loop minibatches Sampling strategy name (default "random\_sorted") Whether to continue optimization automatically Opik project name for logging traces (defaults to OPIK\_PROJECT\_NAME env or "Optimization") Optional ID to use when creating the Opik optimization run Optional validation dataset for ranking candidates Maximum number of optimization trials Whether tools may be executed during evaluation (default True) Which prompt roles to allow for optimization Optional tool optimization selector. Only supported by optimizers that explicitly document tool optimization support. #### post\_baseline ```python post_baseline( context: OptimizationContext, score: float ) ``` **Parameters:** #### post\_optimize ```python post_optimize( context: OptimizationContext, result: OptimizationResult ) ``` **Parameters:** #### post\_round ```python post_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### post\_trial ```python post_trial( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### pre\_baseline ```python pre_baseline( context: OptimizationContext ) ``` **Parameters:** #### pre\_optimize ```python pre_optimize( context: OptimizationContext ) ``` **Parameters:** #### pre\_round ```python pre_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### pre\_trial ```python pre_trial( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### record\_candidate\_entry ```python record_candidate_entry( prompt_or_payload: Any, score: float | None = None, id: str | None = None, metrics: dict[str, typing.Any] | None = None, notes: str | None = None, extra: dict[str, typing.Any] | None = None, context: opik_optimizer.core.state.OptimizationContext | None = None ) ``` **Parameters:** #### run\_optimization ```python run_optimization( context: OptimizationContext ) ``` **Parameters:** The optimization context with prompts, dataset, metric, etc. #### set\_default\_dataset\_split ```python set_default_dataset_split( dataset_split: str | None ) ``` **Parameters:** #### set\_pareto\_front ```python set_pareto_front( pareto_front: list[dict[str, typing.Any]] | None ) ``` **Parameters:** #### set\_selection\_meta ```python set_selection_meta( selection_meta: dict[str, typing.Any] | None ) ``` **Parameters:** #### start\_candidate ```python start_candidate( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### with\_dataset\_split ```python with_dataset_split( dataset_split: str | None ) ``` **Parameters:** ## MetaPromptOptimizer ```python MetaPromptOptimizer( model: str = 'openai/gpt-5-nano', model_parameters: dict[str, typing.Any] | None = None, prompts_per_round: int = 4, enable_context: bool = True, num_task_examples: int = 5, task_context_columns: list[str] | None = None, n_threads: int = 12, verbose: int = 1, seed: int = 42, name: str | None = None, use_hall_of_fame: bool = True, prompt_overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None, skip_perfect_score: bool = True, perfect_score: float = 0.95 ) ``` **Parameters:** ### Methods #### begin\_round ```python begin_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### cleanup ```python cleanup() ``` #### evaluate ```python evaluate( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, sampling_tag: str | None = None ) ``` **Parameters:** Optimization context for this run. Dict of named prompts to evaluate (e.g., \{"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry. Optional experiment configuration. Optional sampling tag for deterministic subsampling per candidate. #### evaluate\_prompt ```python evaluate_prompt( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, n_threads: int | None = None, verbose: int = 1, dataset_item_ids: list[str] | None = None, experiment_config: dict | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, seed: int | None = None, return_evaluation_result: bool = False, allow_tool_use: bool | None = None, use_evaluate_on_dict_items: bool | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### evaluate\_with\_result ```python evaluate_with_result( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, empty_score: float | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### finish\_candidate ```python finish_candidate( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### finish\_round ```python finish_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### get\_config ```python get_config( context: OptimizationContext ) ``` **Parameters:** #### get\_default\_prompt ```python get_default_prompt( key: str ) ``` **Parameters:** The prompt key to retrieve #### get\_history\_entries ```python get_history_entries() ``` #### get\_history\_rounds ```python get_history_rounds() ``` #### get\_metadata ```python get_metadata( context: OptimizationContext ) ``` **Parameters:** #### get\_optimizer\_metadata ```python get_optimizer_metadata() ``` #### get\_prompt ```python get_prompt( key: str, fmt: Any ) ``` **Parameters:** The prompt key to retrieve #### list\_prompts ```python list_prompts() ``` #### on\_trial ```python on_trial( context: OptimizationContext, prompts: dict, score: float, prev_best_score: float | None = None ) ``` **Parameters:** #### optimize\_mcp ```python optimize_mcp( args: Any, kwargs: Any ) ``` **Parameters:** #### optimize\_prompt ```python optimize_prompt( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, experiment_config: dict | None = None, n_samples: int | float | str | None = None, n_samples_minibatch: int | None = None, n_samples_strategy: str | None = None, auto_continue: bool = False, project_name: str | None = None, optimization_id: str | None = None, validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None, max_trials: int = 10, allow_tool_use: bool = True, optimize_prompts: bool | str | list[str] | None = 'system', optimize_tools: bool | dict[str, bool] | None = None, args: Any, kwargs: Any ) ``` **Parameters:** The prompt to optimize (single ChatPrompt or dict of prompts) Opik dataset (training set - used for feedback/context) TODO/FIXME: This parameter will be deprecated in favor of dataset\_training. For now, it serves as the training dataset parameter. A metric function with signature (dataset\_item, llm\_output) -> float Optional agent for prompt execution (defaults to LiteLLMAgent) Optional configuration for the experiment Number of samples to use for evaluation Optional number of samples for inner-loop minibatches Sampling strategy name (default "random\_sorted") Whether to continue optimization automatically Opik project name for logging traces (defaults to OPIK\_PROJECT\_NAME env or "Optimization") Optional ID to use when creating the Opik optimization run Optional validation dataset for ranking candidates Maximum number of optimization trials Whether tools may be executed during evaluation (default True) Which prompt roles to allow for optimization Optional tool optimization selector. Only supported by optimizers that explicitly document tool optimization support. #### post\_baseline ```python post_baseline( context: OptimizationContext, score: float ) ``` **Parameters:** #### post\_optimize ```python post_optimize( context: OptimizationContext, result: OptimizationResult ) ``` **Parameters:** #### post\_round ```python post_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### post\_trial ```python post_trial( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### pre\_baseline ```python pre_baseline( context: OptimizationContext ) ``` **Parameters:** #### pre\_optimize ```python pre_optimize( context: OptimizationContext ) ``` **Parameters:** The optimization context #### pre\_round ```python pre_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### pre\_trial ```python pre_trial( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### record\_candidate\_entry ```python record_candidate_entry( prompt_or_payload: Any, score: float | None = None, id: str | None = None, metrics: dict[str, typing.Any] | None = None, notes: str | None = None, extra: dict[str, typing.Any] | None = None, context: opik_optimizer.core.state.OptimizationContext | None = None ) ``` **Parameters:** #### run\_optimization ```python run_optimization( context: OptimizationContext ) ``` **Parameters:** The optimization context with prompts, dataset, metric, etc. #### set\_default\_dataset\_split ```python set_default_dataset_split( dataset_split: str | None ) ``` **Parameters:** #### set\_pareto\_front ```python set_pareto_front( pareto_front: list[dict[str, typing.Any]] | None ) ``` **Parameters:** #### set\_selection\_meta ```python set_selection_meta( selection_meta: dict[str, typing.Any] | None ) ``` **Parameters:** #### start\_candidate ```python start_candidate( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### with\_dataset\_split ```python with_dataset_split( dataset_split: str | None ) ``` **Parameters:** ## EvolutionaryOptimizer ```python EvolutionaryOptimizer( model: str = 'openai/gpt-5-nano', model_parameters: dict[str, typing.Any] | None = None, population_size: int = 30, num_generations: int = 15, mutation_rate: float = 0.2, crossover_rate: float = 0.8, tournament_size: int = 4, elitism_size: int = 3, adaptive_mutation: bool = True, enable_moo: bool = True, enable_llm_crossover: bool = True, enable_semantic_crossover: bool = False, output_style_guidance: str | None = None, infer_output_style: bool = False, n_threads: int = 12, verbose: int = 1, seed: int = 42, name: str | None = None, prompt_overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None, skip_perfect_score: bool = True, perfect_score: float = 0.95 ) ``` **Parameters:** ### Methods #### begin\_round ```python begin_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### cleanup ```python cleanup() ``` #### evaluate ```python evaluate( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, sampling_tag: str | None = None ) ``` **Parameters:** Optimization context for this run. Dict of named prompts to evaluate (e.g., \{"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry. Optional experiment configuration. Optional sampling tag for deterministic subsampling per candidate. #### evaluate\_prompt ```python evaluate_prompt( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, n_threads: int | None = None, verbose: int = 1, dataset_item_ids: list[str] | None = None, experiment_config: dict | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, seed: int | None = None, return_evaluation_result: bool = False, allow_tool_use: bool | None = None, use_evaluate_on_dict_items: bool | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### evaluate\_with\_result ```python evaluate_with_result( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, empty_score: float | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### finish\_candidate ```python finish_candidate( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### finish\_round ```python finish_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### get\_config ```python get_config( context: OptimizationContext ) ``` **Parameters:** #### get\_default\_prompt ```python get_default_prompt( key: str ) ``` **Parameters:** The prompt key to retrieve #### get\_history\_entries ```python get_history_entries() ``` #### get\_history\_rounds ```python get_history_rounds() ``` #### get\_metadata ```python get_metadata( context: OptimizationContext ) ``` **Parameters:** #### get\_optimizer\_metadata ```python get_optimizer_metadata() ``` #### get\_prompt ```python get_prompt( key: str, fmt: Any ) ``` **Parameters:** The prompt key to retrieve #### list\_prompts ```python list_prompts() ``` #### on\_trial ```python on_trial( context: OptimizationContext, prompts: dict, score: float, prev_best_score: float | None = None ) ``` **Parameters:** #### optimize\_mcp ```python optimize_mcp( args: Any, kwargs: Any ) ``` **Parameters:** #### optimize\_prompt ```python optimize_prompt( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, experiment_config: dict | None = None, n_samples: int | float | str | None = None, n_samples_minibatch: int | None = None, n_samples_strategy: str | None = None, auto_continue: bool = False, project_name: str | None = None, optimization_id: str | None = None, validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None, max_trials: int = 10, allow_tool_use: bool = True, optimize_prompts: bool | str | list[str] | None = 'system', optimize_tools: bool | dict[str, bool] | None = None, args: Any, kwargs: Any ) ``` **Parameters:** The prompt to optimize (single ChatPrompt or dict of prompts) Opik dataset (training set - used for feedback/context) TODO/FIXME: This parameter will be deprecated in favor of dataset\_training. For now, it serves as the training dataset parameter. A metric function with signature (dataset\_item, llm\_output) -> float Optional agent for prompt execution (defaults to LiteLLMAgent) Optional configuration for the experiment Number of samples to use for evaluation Optional number of samples for inner-loop minibatches Sampling strategy name (default "random\_sorted") Whether to continue optimization automatically Opik project name for logging traces (defaults to OPIK\_PROJECT\_NAME env or "Optimization") Optional ID to use when creating the Opik optimization run Optional validation dataset for ranking candidates Maximum number of optimization trials Whether tools may be executed during evaluation (default True) Which prompt roles to allow for optimization Optional tool optimization selector. Only supported by optimizers that explicitly document tool optimization support. #### post\_baseline ```python post_baseline( context: OptimizationContext, score: float ) ``` **Parameters:** #### post\_optimize ```python post_optimize( context: OptimizationContext, result: OptimizationResult ) ``` **Parameters:** #### post\_round ```python post_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### post\_trial ```python post_trial( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### pre\_baseline ```python pre_baseline( context: OptimizationContext ) ``` **Parameters:** #### pre\_optimize ```python pre_optimize( context: OptimizationContext ) ``` **Parameters:** #### pre\_round ```python pre_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### pre\_trial ```python pre_trial( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### record\_candidate\_entry ```python record_candidate_entry( prompt_or_payload: Any, score: float | None = None, id: str | None = None, metrics: dict[str, typing.Any] | None = None, notes: str | None = None, extra: dict[str, typing.Any] | None = None, context: opik_optimizer.core.state.OptimizationContext | None = None ) ``` **Parameters:** #### run\_optimization ```python run_optimization( context: OptimizationContext ) ``` **Parameters:** The optimization context with prompts, dataset, metric, etc. #### set\_default\_dataset\_split ```python set_default_dataset_split( dataset_split: str | None ) ``` **Parameters:** #### set\_pareto\_front ```python set_pareto_front( pareto_front: list[dict[str, typing.Any]] | None ) ``` **Parameters:** #### set\_selection\_meta ```python set_selection_meta( selection_meta: dict[str, typing.Any] | None ) ``` **Parameters:** #### start\_candidate ```python start_candidate( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### with\_dataset\_split ```python with_dataset_split( dataset_split: str | None ) ``` **Parameters:** ## HierarchicalReflectiveOptimizer ```python HierarchicalReflectiveOptimizer( model: str = 'openai/gpt-5-nano', model_parameters: dict[str, typing.Any] | None = None, reasoning_model: str | None = None, reasoning_model_parameters: dict[str, typing.Any] | None = None, max_parallel_batches: int = 5, batch_size: int = 25, convergence_threshold: float = 0.01, n_threads: int = 12, verbose: int = 1, seed: int = 42, name: str | None = None, prompt_overrides: dict[str, str] | collections.abc.Callable[[opik_optimizer.utils.prompt_library.PromptLibrary], None] | None = None, skip_perfect_score: bool = True, perfect_score: float = 0.95 ) ``` **Parameters:** ### Methods #### begin\_round ```python begin_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### cleanup ```python cleanup() ``` #### evaluate ```python evaluate( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, sampling_tag: str | None = None ) ``` **Parameters:** Optimization context for this run. Dict of named prompts to evaluate (e.g., \{"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry. Optional experiment configuration. Optional sampling tag for deterministic subsampling per candidate. #### evaluate\_prompt ```python evaluate_prompt( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, n_threads: int | None = None, verbose: int = 1, dataset_item_ids: list[str] | None = None, experiment_config: dict | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, seed: int | None = None, return_evaluation_result: bool = False, allow_tool_use: bool | None = None, use_evaluate_on_dict_items: bool | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### evaluate\_with\_result ```python evaluate_with_result( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, empty_score: float | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### finish\_candidate ```python finish_candidate( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### finish\_round ```python finish_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### get\_config ```python get_config( context: OptimizationContext ) ``` **Parameters:** #### get\_default\_prompt ```python get_default_prompt( key: str ) ``` **Parameters:** The prompt key to retrieve #### get\_history\_entries ```python get_history_entries() ``` #### get\_history\_rounds ```python get_history_rounds() ``` #### get\_metadata ```python get_metadata( context: OptimizationContext ) ``` **Parameters:** #### get\_optimizer\_metadata ```python get_optimizer_metadata() ``` #### get\_prompt ```python get_prompt( key: str, fmt: Any ) ``` **Parameters:** The prompt key to retrieve #### list\_prompts ```python list_prompts() ``` #### on\_trial ```python on_trial( context: OptimizationContext, prompts: dict, score: float, prev_best_score: float | None = None ) ``` **Parameters:** #### optimize\_mcp ```python optimize_mcp( args: Any, kwargs: Any ) ``` **Parameters:** #### optimize\_prompt ```python optimize_prompt( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, experiment_config: dict | None = None, n_samples: int | float | str | None = None, n_samples_minibatch: int | None = None, n_samples_strategy: str | None = None, auto_continue: bool = False, project_name: str | None = None, optimization_id: str | None = None, validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None, max_trials: int = 10, allow_tool_use: bool = True, optimize_prompts: bool | str | list[str] | None = 'system', optimize_tools: bool | dict[str, bool] | None = None, args: Any, kwargs: Any ) ``` **Parameters:** The prompt to optimize (single ChatPrompt or dict of prompts) Opik dataset (training set - used for feedback/context) TODO/FIXME: This parameter will be deprecated in favor of dataset\_training. For now, it serves as the training dataset parameter. A metric function with signature (dataset\_item, llm\_output) -> float Optional agent for prompt execution (defaults to LiteLLMAgent) Optional configuration for the experiment Number of samples to use for evaluation Optional number of samples for inner-loop minibatches Sampling strategy name (default "random\_sorted") Whether to continue optimization automatically Opik project name for logging traces (defaults to OPIK\_PROJECT\_NAME env or "Optimization") Optional ID to use when creating the Opik optimization run Optional validation dataset for ranking candidates Maximum number of optimization trials Whether tools may be executed during evaluation (default True) Which prompt roles to allow for optimization Optional tool optimization selector. Only supported by optimizers that explicitly document tool optimization support. #### post\_baseline ```python post_baseline( context: OptimizationContext, score: float ) ``` **Parameters:** #### post\_optimize ```python post_optimize( context: OptimizationContext, result: OptimizationResult ) ``` **Parameters:** #### post\_round ```python post_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### post\_trial ```python post_trial( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### pre\_baseline ```python pre_baseline( context: OptimizationContext ) ``` **Parameters:** #### pre\_optimize ```python pre_optimize( context: OptimizationContext ) ``` **Parameters:** The optimization context #### pre\_round ```python pre_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### pre\_trial ```python pre_trial( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### record\_candidate\_entry ```python record_candidate_entry( prompt_or_payload: Any, score: float | None = None, id: str | None = None, metrics: dict[str, typing.Any] | None = None, notes: str | None = None, extra: dict[str, typing.Any] | None = None, context: opik_optimizer.core.state.OptimizationContext | None = None ) ``` **Parameters:** #### run\_optimization ```python run_optimization( context: OptimizationContext ) ``` **Parameters:** The optimization context with prompts, dataset, metric, etc. #### set\_default\_dataset\_split ```python set_default_dataset_split( dataset_split: str | None ) ``` **Parameters:** #### set\_pareto\_front ```python set_pareto_front( pareto_front: list[dict[str, typing.Any]] | None ) ``` **Parameters:** #### set\_selection\_meta ```python set_selection_meta( selection_meta: dict[str, typing.Any] | None ) ``` **Parameters:** #### start\_candidate ```python start_candidate( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### with\_dataset\_split ```python with_dataset_split( dataset_split: str | None ) ``` **Parameters:** ## ParameterOptimizer ```python ParameterOptimizer( model: str = 'openai/gpt-5-nano', model_parameters: dict[str, typing.Any] | None = None, default_n_trials: int = 20, local_search_ratio: float = 0.3, local_search_scale: float = 0.2, n_threads: int = 12, verbose: int = 1, seed: int = 42, name: str | None = None, skip_perfect_score: bool = True, perfect_score: float = 0.95 ) ``` **Parameters:** ### Methods #### begin\_round ```python begin_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### cleanup ```python cleanup() ``` #### evaluate ```python evaluate( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, sampling_tag: str | None = None ) ``` **Parameters:** Optimization context for this run. Dict of named prompts to evaluate (e.g., \{"main": ChatPrompt(...)}). Single-prompt optimizations use a dict with one entry. Optional experiment configuration. Optional sampling tag for deterministic subsampling per candidate. #### evaluate\_prompt ```python evaluate_prompt( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, n_threads: int | None = None, verbose: int = 1, dataset_item_ids: list[str] | None = None, experiment_config: dict | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, seed: int | None = None, return_evaluation_result: bool = False, allow_tool_use: bool | None = None, use_evaluate_on_dict_items: bool | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### evaluate\_with\_result ```python evaluate_with_result( context: OptimizationContext, prompts: dict, experiment_config: dict[str, typing.Any] | None = None, empty_score: float | None = None, n_samples: int | float | str | None = None, n_samples_strategy: str | None = None, sampling_tag: str | None = None ) ``` **Parameters:** #### finish\_candidate ```python finish_candidate( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### finish\_round ```python finish_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### get\_config ```python get_config( context: OptimizationContext ) ``` **Parameters:** #### get\_default\_prompt ```python get_default_prompt( key: str ) ``` **Parameters:** The prompt key to retrieve #### get\_history\_entries ```python get_history_entries() ``` #### get\_history\_rounds ```python get_history_rounds() ``` #### get\_metadata ```python get_metadata( context: OptimizationContext ) ``` **Parameters:** #### get\_optimizer\_metadata ```python get_optimizer_metadata() ``` #### get\_prompt ```python get_prompt( key: str, fmt: Any ) ``` **Parameters:** The prompt key to retrieve #### list\_prompts ```python list_prompts() ``` #### on\_trial ```python on_trial( context: OptimizationContext, prompts: dict, score: float, prev_best_score: float | None = None ) ``` **Parameters:** #### optimize\_mcp ```python optimize_mcp( args: Any, kwargs: Any ) ``` **Parameters:** #### optimize\_parameter ```python optimize_parameter( prompt: opik_optimizer.api_objects.chat_prompt.ChatPrompt | dict[str, opik_optimizer.api_objects.chat_prompt.ChatPrompt], dataset: Dataset, metric: MetricFunction, parameter_space: opik_optimizer.algorithms.parameter_optimizer.ops.search_ops.ParameterSearchSpace | collections.abc.Mapping[str, typing.Any], validation_dataset: opik.api_objects.dataset.dataset.Dataset | None = None, experiment_config: dict | None = None, max_trials: int | None = None, n_samples: int | float | str | None = None, n_samples_minibatch: int | None = None, n_samples_strategy: str | None = None, agent: opik_optimizer.agents.optimizable_agent.OptimizableAgent | None = None, project_name: str = 'Optimization', sampler: optuna.samplers._base.BaseSampler | None = None, callbacks: list[collections.abc.Callable[[optuna.study.study.Study, optuna.trial._frozen.FrozenTrial], None]] | None = None, timeout: float | None = None, local_trials: int | None = None, local_search_scale: float | None = None, optimization_id: str | None = None ) ``` **Parameters:** The prompt or dict of prompts to evaluate with tuned parameters. When a dict is provided, parameters are optimized independently for each prompt. Dataset providing evaluation examples Objective function to maximize Definition of the search space for tunable parameters. For multi-prompt, params without a prefix are expanded per prompt. Params already prefixed (e.g., 'analyze.temperature') are kept as-is. Optional validation dataset. Note: Due to the internal implementation of ParameterOptimizer, this parameter is currently not fully utilized and we recommend not using it for this optimizer. Optional experiment metadata Total number of trials (if None, uses default\_n\_trials) Number of dataset samples to evaluate per trial (None for all) Optional number of samples for inner-loop minibatches Sampling strategy name (default "random\_sorted") Optional custom agent instance to execute evaluations Opik project name for logging traces (default: "Optimization") Optuna sampler to use (default: TPESampler with seed) List of callback functions for Optuna study Maximum time in seconds for optimization Number of trials for local search (overrides local\_search\_ratio) Scale factor for local search narrowing (0.0-1.0) Optional ID to use when creating the Opik optimization run; when provided it must be a valid UUIDv7 string. #### post\_baseline ```python post_baseline( context: OptimizationContext, score: float ) ``` **Parameters:** #### post\_optimize ```python post_optimize( context: OptimizationContext, result: OptimizationResult ) ``` **Parameters:** #### post\_round ```python post_round( round_handle: Any, context: opik_optimizer.core.state.OptimizationContext | None = None, best_score: float | None = None, best_candidate: typing.Any | None = None, best_prompt: typing.Any | None = None, stop_reason: str | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, timestamp: str | None = None, dataset_split: str | None = None, pareto_front: list[dict[str, typing.Any]] | None = None, selection_meta: dict[str, typing.Any] | None = None ) ``` **Parameters:** #### post\_trial ```python post_trial( context: OptimizationContext, candidate_handle: Any, score: float | None, metrics: dict[str, typing.Any] | None = None, extras: dict[str, typing.Any] | None = None, candidates: list[dict[str, typing.Any]] | None = None, dataset: str | None = None, dataset_split: str | None = None, trial_index: int | None = None, timestamp: str | None = None, round_handle: typing.Any | None = None ) ``` **Parameters:** #### pre\_baseline ```python pre_baseline( context: OptimizationContext ) ``` **Parameters:** #### pre\_optimize ```python pre_optimize( context: OptimizationContext ) ``` **Parameters:** The optimization context #### pre\_round ```python pre_round( context: OptimizationContext, extras: Any ) ``` **Parameters:** #### pre\_trial ```python pre_trial( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### record\_candidate\_entry ```python record_candidate_entry( prompt_or_payload: Any, score: float | None = None, id: str | None = None, metrics: dict[str, typing.Any] | None = None, notes: str | None = None, extra: dict[str, typing.Any] | None = None, context: opik_optimizer.core.state.OptimizationContext | None = None ) ``` **Parameters:** #### set\_default\_dataset\_split ```python set_default_dataset_split( dataset_split: str | None ) ``` **Parameters:** #### set\_pareto\_front ```python set_pareto_front( pareto_front: list[dict[str, typing.Any]] | None ) ``` **Parameters:** #### set\_selection\_meta ```python set_selection_meta( selection_meta: dict[str, typing.Any] | None ) ``` **Parameters:** #### start\_candidate ```python start_candidate( context: OptimizationContext, candidate: Any, round_handle: typing.Any | None = None ) ``` **Parameters:** #### with\_dataset\_split ```python with_dataset_split( dataset_split: str | None ) ``` **Parameters:** ## ParameterSearchSpace ```python ParameterSearchSpace( parameters: list[opik_optimizer.algorithms.parameter_optimizer.ops.search_ops.ParameterSpec] = PydanticUndefined ) ``` **Parameters:** ## ParameterSpec ```python ParameterSpec( name:
## Two approaches to evaluation
Opik provides two complementary approaches to evaluation:
* **Test Suites**: Define natural-language assertions and let an LLM judge check them automatically. Best for pass/fail testing of specific behaviors.
* **Datasets & Metrics**: Score your agent's outputs against a dataset using pre-built or custom metrics. Best for measuring quality across many traces with quantitative scores.
## Key features
* **Test Suites** with natural-language assertions and execution policies
* **30+ pre-built metrics** for hallucination, relevance, coherence, and more
* **Custom metrics** for domain-specific evaluation
* **Experiment tracking** to compare versions side-by-side
* **Annotation Queues** for human-in-the-loop review
## Next steps
* [Getting started](/evaluation/getting-started) — Run your first evaluation in minutes
* [Concepts](/evaluation/concepts) — Understand Test Suites vs Datasets & Metrics
* [Building Test Suites](/evaluation/advanced/building-test-suites) — Create and manage suites via the SDK, UI, or Ollie
* [Debugging agents with Ollie](/tracing/debug-agents) — The full workflow for turning production failures into test cases
# Getting started with Evaluation
Opik provides two approaches to evaluation. Choose the one that fits your use case:
* **Test Suites**: Define assertions in natural language and let an LLM judge test them. Best for pass/fail behavioral testing.
* **Datasets & Metrics**: Score outputs against a dataset using quantitative metrics. Best for measuring quality across many traces.
## Quick start
Test Suites let you define expected behaviors as natural-language assertions and run them
against your agent. An LLM judge checks each assertion automatically.
```python title="Python"
import opik
from openai import OpenAI
from opik.integrations.openai import track_openai
openai_client = track_openai(OpenAI())
opik_client = opik.Opik()
# Create a suite with assertions
suite = opik_client.get_or_create_test_suite(
name="my-agent-tests",
project_name="my-agent",
global_assertions=[
"The response directly addresses the user's question",
"The response is concise (3 sentences or fewer)",
],
global_execution_policy={"runs_per_item": 2, "pass_threshold": 2},
)
# Add test cases
suite.insert([
{"data": {"question": "How do I create a new project?", "context": "Go to Dashboard and click 'New Project'."}},
{"data": {"question": "What are the pricing tiers?", "context": "Free ($0/month), Pro ($29/month), Enterprise (custom)."}},
])
# Define the task
def task(item):
response = openai_client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Answer based ONLY on the provided context."},
{"role": "user", "content": f"Question: {item['question']}\n\nContext:\n{item['context']}"},
],
)
return {"input": item, "output": response.choices[0].message.content}
# Run the evaluation
result = opik.run_tests(test_suite=suite, task=task)
print(f"Pass rate: {result.pass_rate:.0%}")
```
```ts title="Typescript"
import { Opik, TestSuite, runTests } from "opik";
import OpenAI from "openai";
const client = new Opik();
const openai = new OpenAI();
// Create a suite with assertions
const suite = await TestSuite.getOrCreate(client, {
name: "my-agent-tests",
projectName: "my-agent",
globalAssertions: [
"The response directly addresses the user's question",
"The response is concise (3 sentences or fewer)",
],
globalExecutionPolicy: { runsPerItem: 2, passThreshold: 2 },
});
// Add test cases
await suite.insert([
{ data: { question: "How do I create a new project?", context: "Go to Dashboard and click 'New Project'." } },
{ data: { question: "What are the pricing tiers?", context: "Free ($0/month), Pro ($29/month), Enterprise (custom)." } },
]);
// Define the task
const task = async (item: Record
See the [Building Test Suites](/evaluation/advanced/building-test-suites) guide for the full walkthrough.
Dataset-based evaluation scores your agent's outputs using quantitative metrics like
hallucination detection, answer relevance, or custom scoring functions.
```python title="Python"
import opik
from opik.evaluation import evaluate
from opik.evaluation.metrics import Hallucination
opik.configure()
client = opik.Opik()
# Create a dataset
dataset = client.get_or_create_dataset(name="my-eval-dataset")
dataset.insert([
{"input": "What is the capital of France?", "expected_output": "Paris"},
{"input": "What is 2+2?", "expected_output": "4"},
])
# Define the task
def task(item):
# Your LLM call here
result = call_llm(item["input"])
return {"output": result}
# Run evaluation with metrics
evaluate(
dataset=dataset,
task=task,
scoring_metrics=[Hallucination()],
experiment_name="my-experiment-v1",
)
```
```ts title="Typescript"
import { Opik } from "opik";
const client = new Opik();
// Create a dataset
const dataset = await client.getOrCreateDataset({ name: "my-eval-dataset" });
await dataset.insert([
{ input: "What is the capital of France?", expectedOutput: "Paris" },
{ input: "What is 2+2?", expectedOutput: "4" },
]);
// Run evaluation with metrics
await client.evaluate({
dataset,
task: async (item) => {
const result = await callLlm(item.input);
return { output: result };
},
experimentName: "my-experiment-v1",
});
```
See the [Datasets & Experiments](/evaluation/advanced/evaluate_your_llm) guide for the full walkthrough
and the [Metrics](/evaluation/metrics/overview) section for all available metrics.
# Evaluation Concepts
Opik provides two complementary approaches to evaluating your LLM application. Understanding when to use each will help you build a robust evaluation strategy.
## Test Suites — assertion-based testing
Test Suites let you define expected behaviors as natural-language assertions. An LLM judge checks each assertion against your agent's output and reports pass/fail results.
**Best for:**
* Testing specific behaviors (e.g., "the response does not hallucinate")
* Pass/fail validation of agent outputs
* Iterating on prompts and comparing versions
* Catching regressions after changes
A Test Suite has three main components:
1. **Test items**: Input data for your agent (e.g., questions with context, user scenarios)
2. **Assertions**: Natural-language descriptions of expected behavior, checked by an LLM judge (e.g., "The response is concise")
3. **Execution policy**: Controls how many times each item is run and how many runs must pass
Assertions can be defined at two levels:
* **Suite-level assertions** apply to every test item
* **Item-level assertions** apply only to a specific test item, in addition to suite-level ones
### Pass/fail logic
* A **run** passes if all its assertions pass
* An **item** passes if the number of passed runs meets the `pass_threshold`
* The **pass rate** is the ratio of passed items to total items
## Datasets & Metrics — quantitative scoring
Dataset-based evaluation scores your agent's outputs using quantitative metrics. You define a dataset of test cases, run your agent against them, and score the results using pre-built or custom metrics.
**Best for:**
* Measuring quality across many traces with a common metric (hallucination, relevance, coherence)
* Comparing model or prompt versions with numeric scores
* Evaluating RAG pipelines with context precision/recall metrics
* Building leaderboards across experiments
A dataset-based evaluation has three main components:
1. **Dataset**: A collection of test cases with inputs and optional expected outputs
2. **Task**: A function that takes a dataset item and returns your agent's output
3. **Metrics**: Scoring functions that evaluate the output (e.g., `Hallucination`, `AnswerRelevance`, custom metrics)
Each evaluation run creates an **Experiment** — a record of every dataset item, your agent's output, and the metric scores. Experiments are stored in Opik so you can compare them side-by-side.
## Choosing between the two
| | Test Suites | Datasets & Metrics |
| --------------------- | -------------------------------------------- | -------------------------------------------- |
| **Output** | Pass/fail per assertion | Numeric scores per metric |
| **Evaluation method** | LLM judge checks natural-language assertions | Scoring functions (LLM-based or heuristic) |
| **Best for** | Behavioral testing, regression checks | Quality measurement, benchmarking |
| **Iteration style** | Update assertions, re-run suite | Update dataset or metrics, re-run experiment |
You can use both approaches together. For example, use Test Suites during development to validate specific behaviors, and Datasets & Metrics in CI to track quality scores over time.
# Building Test Suites
Test suites grow as you debug and improve your agent. There are three ways to build them: with Ollie, through the UI, or via the SDK.
## With Ollie
The fastest way to turn a production failure into a test case. Open Ollie from any trace view and describe what went wrong:
*"Add this trace to my customer-support-qa suite with the assertion: the response must cite a specific step from the provided context"*
Ollie creates the test item directly — no copy-pasting required. You can also ask Ollie to run the suite after making changes:
*"Run the customer-support-qa suite against the updated prompt"*
See [Debugging agents](/tracing/debug-agents) for the full workflow.
## With the UI
In the Opik dashboard, navigate to the Test Suites section to create and manage suites visually. You can add test items, define assertions, configure execution policies, and review results — all without writing code.
## With the SDK
### Create a suite
Define the quality bars you care about as suite-level assertions:
```python title="Python"
import opik
opik_client = opik.Opik()
suite = opik_client.get_or_create_test_suite(
name="customer-support-qa",
project_name="test-suites-demo",
global_assertions=[
"The response is grounded in the provided documentation context",
"The response directly addresses the user's question",
"The response is concise (3 sentences or fewer)",
],
global_execution_policy={"runs_per_item": 2, "pass_threshold": 2},
)
```
```ts title="Typescript"
import { Opik, TestSuite } from "opik";
const client = new Opik();
const suite = await TestSuite.getOrCreate(client, {
name: "customer-support-qa",
projectName: "test-suites-demo",
globalAssertions: [
"The response is grounded in the provided documentation context",
"The response directly addresses the user's question",
"The response is concise (3 sentences or fewer)",
],
globalExecutionPolicy: { runsPerItem: 2, passThreshold: 2 },
});
```
### Add test items
Add individual items or batches. Items can include item-level assertions that are checked in addition to the suite-level assertions:
```python title="Python"
suite.insert([
{
"data": {
"question": "How do I create a new project?",
"context": "To create a new project, go to the Dashboard and click 'New Project'.",
},
},
{
"data": {
"question": "Can I use this with Kubernetes?",
"context": "We support Docker containers and serverless functions.",
},
"assertions": [
"The response does NOT claim Kubernetes is supported",
"The response acknowledges that the information is not available",
],
"execution_policy": {"runs_per_item": 3, "pass_threshold": 2},
},
])
```
```ts title="Typescript"
await suite.insert([
{
data: {
question: "How do I create a new project?",
context: "To create a new project, go to the Dashboard and click 'New Project'.",
},
},
{
data: {
question: "Can I use this with Kubernetes?",
context: "We support Docker containers and serverless functions.",
},
assertions: [
"The response does NOT claim Kubernetes is supported",
"The response acknowledges that the information is not available",
],
executionPolicy: { runsPerItem: 3, passThreshold: 2 },
},
]);
```
### Define the task and run
The task function receives each item's `data` and must return an object with `input` and `output` keys:
```python title="Python"
from openai import OpenAI
from opik.integrations.openai import track_openai
openai_client = track_openai(OpenAI())
def make_task(system_prompt):
def task(item):
response = openai_client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": f"Question: {item['question']}\n\nContext:\n{item['context']}"},
],
)
return {"input": item, "output": response.choices[0].message.content}
return task
PROMPT_V1 = "You are a helpful assistant. Be as detailed as possible."
PROMPT_V2 = "You are a concise assistant. Answer based ONLY on the provided context."
result_v1 = opik.run_tests(test_suite=suite, task=make_task(PROMPT_V1))
result_v2 = opik.run_tests(test_suite=suite, task=make_task(PROMPT_V2))
print(f"v1 pass rate: {result_v1.pass_rate:.0%}")
print(f"v2 pass rate: {result_v2.pass_rate:.0%}")
```
```ts title="Typescript"
import { runTests } from "opik";
import OpenAI from "openai";
const openai = new OpenAI();
function makeTask(systemPrompt: string) {
return async (item: Record
The `input` should contain only the data your agent actually received when generating its response.
The LLM judge uses `input` and `output` to evaluate assertions — if you accidentally include fields
like `expected_answer` in `input`, the judge may use them to pass assertions that should fail.
### Update assertions and execution policy
```python title="Python"
suite.update_test_settings(
global_assertions=[
"The response is grounded in the provided context",
"The response is concise",
],
global_execution_policy={"runs_per_item": 5, "pass_threshold": 3},
)
```
```ts title="Typescript"
await suite.updateTestSettings({
globalAssertions: [
"The response is grounded in the provided context",
"The response is concise",
],
globalExecutionPolicy: { runsPerItem: 5, passThreshold: 3 },
});
```
### Inspect suite contents
```python title="Python"
items = suite.get_items()
assertions = suite.get_global_assertions()
policy = suite.get_global_execution_policy()
print(f"Items: {len(items)}")
print(f"Assertions: {assertions}")
print(f"Policy: {policy}")
```
```ts title="Typescript"
const items = await suite.getItems();
const assertions = await suite.getGlobalAssertions();
const policy = await suite.getGlobalExecutionPolicy();
console.log(`Items: ${items.length}`);
console.log(`Assertions: ${assertions}`);
console.log(`Policy: ${JSON.stringify(policy)}`);
```
### Delete test items
```python title="Python"
items = suite.get_items()
suite.delete([items[0]["id"]])
```
```ts title="Typescript"
const items = await suite.getItems();
await suite.delete([items[0].id]);
```
## Execution policies
Execution policies control how many times each item is run and how many must pass. This is useful for handling non-deterministic LLM outputs.
```python title="Python"
suite = opik_client.get_or_create_test_suite(
name="flaky-output-tests",
global_assertions=["Response follows the expected format"],
global_execution_policy={"runs_per_item": 3, "pass_threshold": 2},
)
```
```ts title="Typescript"
const suite = await TestSuite.getOrCreate(client, {
name: "flaky-output-tests",
globalAssertions: ["Response follows the expected format"],
globalExecutionPolicy: { runsPerItem: 3, passThreshold: 2 },
});
```
**Pass/fail logic:**
* A **run** passes if all its assertions pass
* An **item** passes if `runs_passed >= pass_threshold`
* The **pass rate** is the ratio of passed items to total items. A pass rate of `1.0` means every item passed; `0.0` means none did
You can also override the policy for individual items:
```python title="Python"
suite.insert([{
"data": {"question": "Is my account compromised?", "context": "..."},
"assertions": ["Response treats the concern with urgency"],
"execution_policy": {"runs_per_item": 5, "pass_threshold": 4},
}])
```
```ts title="Typescript"
await suite.insert([{
data: { question: "Is my account compromised?", context: "..." },
assertions: ["Response treats the concern with urgency"],
executionPolicy: { runsPerItem: 5, passThreshold: 4 },
}]);
```
# Evaluate your agent
In Opik 2.0, Experiments and Evaluation Suites are project-scoped. Make sure to specify a `project_name` when creating datasets and running experiments.
Evaluating your LLM application allows you to have confidence in the performance of your LLM application. In this guide, we will walk through the process of evaluating complex applications like LLM chains or agents.
The evaluation is done in five steps:
1. Add tracing to your LLM application
2. Define the evaluation task
3. Choose the `Dataset` that you would like to evaluate your application on
4. Choose the metrics that you would like to evaluate your application with
5. Create and run the evaluation experiment
## Running an offline evaluation
### 1. (Optional) Add tracking to your LLM application
While not required, we recommend adding tracking to your LLM application. This allows you to have
full visibility into each evaluation run. In the example below we will use a combination of the
`track` decorator and the `track_openai` function to trace the LLM application.
```python title="Python" language="python"
from opik import track
from opik.integrations.openai import track_openai
import openai
openai_client = track_openai(openai.OpenAI())
# This method is the LLM application that you want to evaluate
# Typically this is not updated when creating evaluations
@track
def your_llm_application(input: str) -> str:
response = openai_client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": input}],
)
return response.choices[0].message.content
```
Here we have added the `track` decorator so that this trace and all its nested
steps are logged to the platform for further analysis.
### 2. Define the evaluation task
Once you have added instrumentation to your LLM application, we can define the evaluation
task. The evaluation task takes in as an input a dataset item and needs to return a
dictionary with keys that match the parameters expected by the metrics you are using. In
this example we can define the evaluation task as follows:
```typescript title="TypeScript" language="typescript"
import { EvaluationTask } from "opik";
import { OpenAI } from "openai";
// Define dataset item type
type DatasetItem = {
input: string;
expected: string;
};
const llmTask: EvaluationTask
## Advanced usage
### Missing arguments for scoring methods
When you face the `opik.exceptions.ScoreMethodMissingArguments` exception, it means that the dataset
item and task output dictionaries do not contain all the arguments expected by the scoring method.
The way the evaluate function works is by merging the dataset item and task output dictionaries and
then passing the result to the scoring method. For example, if the dataset item contains the keys
`user_question` and `context` while the evaluation task returns a dictionary with the key `output`,
the scoring method will be called as `scoring_method.score(user_question='...', context= '...', output= '...')`.
This can be an issue if the scoring method expects a different set of arguments.
You can solve this by either updating the dataset item or evaluation task to return the missing
arguments or by using the `scoring_key_mapping` parameter of the `evaluate` function. In the example
above, if the scoring method expects `input` as an argument, you can map the `user_question` key to
the `input` key as follows:
```typescript title="TypeScript" language="typescript"
evaluation = evaluate({
dataset,
task: evaluation_task,
scoringMetrics: [hallucination_metric],
scoringKeyMapping: {"input": "user_question"},
})
```
```python title="Python" language="python"
evaluation = evaluate(
dataset=dataset,
task=evaluation_task,
scoring_metrics=[hallucination_metric],
scoring_key_mapping={"input": "user_question"},
)
```
### Linking prompts to experiments
The [Opik prompt library](/development/prompt-library/getting-started) can be used to version your prompt templates.
When creating an Experiment, you can link the Experiment to a specific prompt version:
```typescript title="TypeScript" language="typescript"
import { Opik, evaluate, evaluatePrompt } from 'opik';
import { Hallucination } from 'opik';
const client = new Opik();
// Create a prompt
const prompt = await client.createPrompt({
name: "My prompt",
prompt: "Translate to French: {{input}}",
projectName: "my-project",
});
// Link prompt to evaluation experiment
await evaluatePrompt({
dataset: myDataset,
messages: [
{ role: "user", content: "Translate to French: {{input}}" },
],
model: "gpt-4o",
scoringMetrics: [new Hallucination()],
prompts: [prompt],
projectName: "my-project",
});
```
```python title="Python" language="python"
import opik
# Create a prompt
prompt = opik.Prompt(
name="My prompt",
prompt="...",
project_name="my-project",
)
# Run the evaluation
evaluation = evaluate(
dataset=dataset,
task=evaluation_task,
scoring_metrics=[hallucination_metric],
prompts=[prompt],
project_name="my-project",
)
```
The experiment will now be linked to the prompt allowing you to view all experiments that use a specific prompt:
### Logging traces to a specific project
You can use the `project_name` parameter of the `evaluate` function to log evaluation traces to a specific project:
```typescript title="TypeScript" language="typescript"
const evaluation = await evaluate({
dataset,
task: evaluation_task,
scoringMetrics: [hallucination_metric],
projectName: "hallucination-detection",
})
```
```python title="Python" language="python"
evaluation = evaluate(
dataset=dataset,
task=evaluation_task,
scoring_metrics=[hallucination_metric],
project_name="hallucination-detection",
)
```
### Evaluating a subset of the dataset
You can use the `nb_samples` parameter to specify the number of samples to use for the evaluation. This is useful if you only want to evaluate a subset of the dataset.
```typescript title="TypeScript" language="typescript"
const evaluation = await evaluate({
dataset,
task: evaluation_task,
scoringMetrics: [hallucination_metric],
nbSamples: 10,
})
```
```python title="Python" language="python"
evaluation = evaluate(
experiment_name="My experiment",
dataset=dataset,
task=evaluation_task,
scoring_metrics=[hallucination_metric],
nb_samples=10,
)
```
### Evaluating a filtered subset of the dataset
You can evaluate only a subset of your dataset items by using the `dataset_filter_string` parameter. This is useful when you want to run experiments on specific categories of data or test particular scenarios:
```python title="Python" language="python"
from opik.evaluation import evaluate
# Evaluate only items with specific tags
evaluation = evaluate(
experiment_name="Production test cases",
dataset=dataset,
task=evaluation_task,
scoring_metrics=[hallucination_metric],
dataset_filter_string='tags contains "production"',
)
# Evaluate items matching multiple conditions
evaluation = evaluate(
experiment_name="Hard finance questions",
dataset=dataset,
task=evaluation_task,
scoring_metrics=[hallucination_metric],
dataset_filter_string='data.category = "finance" AND data.difficulty = "hard"',
)
# Filter by date range
evaluation = evaluate(
experiment_name="Recent test cases",
dataset=dataset,
task=evaluation_task,
scoring_metrics=[hallucination_metric],
dataset_filter_string='created_at >= "2024-06-01T00:00:00Z"',
)
```
The filter uses Opik Query Language (OQL) syntax. For more details on filter syntax and supported columns, see [Filtering syntax](/evaluation/advanced/manage_datasets#filter-syntax).
You can combine filtering with other parameters like `nb_samples` to evaluate a specific number of items from a filtered subset.
### Sampling the dataset for evaluation
You can use the `dataset_sampler` parameter to specify the instance of dataset sampler to use for sampling the dataset.
This is useful if you want to sample the dataset differently than the default sampling strategy (accept all items).
For example, you can use the `RandomDatasetSampler` to sample the dataset randomly:
```python title="Python" language="python"
from opik.evaluation import samplers
evaluation = evaluate(
experiment_name="My experiment",
dataset=dataset,
task=evaluation_task,
scoring_metrics=[hallucination_metric],
dataset_sampler=samplers.RandomDatasetSampler(max_samples=10),
)
```
In the example above, the evaluation will sample 10 random items from the dataset.
Also, you can implement your own dataset sampler by extending the `BaseDatasetSampler` and overriding the `sample` method.
```python title="Python" language="python"
import re
from typing import List
from opik.api_objects.dataset import dataset_item
from opik.evaluation import samplers
class MyDatasetSampler(samplers.BaseDatasetSampler):
def __init__(self, filter_string: str, field_name: str) -> None:
self.filter_regex = re.compile(filter_string)
self.field_name = field_name
def sample(self, dataset: List[dataset_item.DatasetItem]) -> List[dataset_item.DatasetItem]:
# Sample items from the dataset that match the filter string in the 'field_name' field
return [item for item in filter(lambda x: self.filter_regex.search(x[self.field_name]), dataset)]
# Example usage
evaluation = evaluate(
experiment_name="My experiment",
dataset=dataset,
task=evaluation_task,
scoring_metrics=[hallucination_metric],
dataset_sampler=MyDatasetSampler(filter_string="\\.*SUCCESS\\.*", field_name="output"),
)
```
Implementing your own dataset sampler is useful if you want to implement a custom sampling strategy. For instance,
you can implement a dataset sampler that samples the dataset using some filtering criteria as in the example above.
### Analyzing the evaluation results
The `evaluate` function returns an `EvaluationResult` object that contains the evaluation results.
You can create aggregated statistics for each metric by calling its `aggregate_evaluation_scores` method:
```python title="Python" language="python"
evaluation = evaluate(
experiment_name="My experiment",
dataset=dataset,
task=evaluation_task,
scoring_metrics=[hallucination_metric],
)
# Retrieve and print the aggregated scores statistics (mean, min, max, std) per metric
scores = evaluation.aggregate_evaluation_scores()
for metric_name, statistics in scores.aggregated_scores.items():
print(f"{metric_name}: {statistics}")
```
Aggregated statistics can help analyze evaluation results and are useful for comparing the
performance of different models or different versions of the same model, for example.
### Computing experiment-level metrics
In addition to per-item metrics, you can compute experiment-level aggregate metrics that are calculated across all test results. These experiment scores are displayed in the Opik UI alongside feedback scores and can be used for sorting and filtering experiments.
Experiment scores are computed after all test results are collected. You define experiment score functions that take a list of `TestResult` objects and return a list of `ScoreResult` objects representing aggregate metrics.
```python title="Python" language="python"
from typing import List
from opik.evaluation import evaluate, test_result
from opik.evaluation.metrics import Hallucination, score_result
# Define an experiment score function
def compute_hallucination_max(
test_results: List[test_result.TestResult],
) -> List[score_result.ScoreResult]:
"""Compute the maximum hallucination score across all test results."""
hallucination_scores = [
result.score_results[0].value
for result in test_results
if result.score_results and len(result.score_results) > 0
]
if not hallucination_scores:
return []
return [
score_result.ScoreResult(
name="hallucination_metric (max)",
value=max(hallucination_scores),
reason=f"Maximum hallucination score across {len(hallucination_scores)} test cases"
)
]
# Run evaluation with experiment scores
evaluation = evaluate(
dataset=dataset,
task=evaluation_task,
scoring_metrics=[Hallucination()],
experiment_scoring_functions=[compute_hallucination_max],
experiment_name="My experiment"
)
# Access experiment scores from the result
print(f"Experiment scores: {evaluation.experiment_scores}")
```
Experiment scores are displayed in the Opik UI in the experiments table alongside feedback scores. They can be used for sorting and filtering experiments, making it easy to compare experiments based on aggregate metrics.
You can define multiple experiment score functions to compute different aggregate metrics:
```python title="Python" language="python"
from typing import List
from opik.evaluation import evaluate, test_result
from opik.evaluation.metrics import Equals, score_result
def compute_accuracy_stats(
test_results: List[test_result.TestResult],
) -> List[score_result.ScoreResult]:
"""Compute accuracy statistics across all test results."""
accuracy_scores = [
result.score_results[0].value
for result in test_results
if result.score_results and len(result.score_results) > 0
]
if not accuracy_scores:
return []
return [
score_result.ScoreResult(
name="accuracy (mean)",
value=sum(accuracy_scores) / len(accuracy_scores),
reason=f"Mean accuracy across {len(accuracy_scores)} test cases"
),
score_result.ScoreResult(
name="accuracy (min)",
value=min(accuracy_scores),
reason=f"Minimum accuracy across {len(accuracy_scores)} test cases"
),
score_result.ScoreResult(
name="accuracy (max)",
value=max(accuracy_scores),
reason=f"Maximum accuracy across {len(accuracy_scores)} test cases"
),
]
evaluation = evaluate(
dataset=dataset,
task=evaluation_task,
scoring_metrics=[Equals()],
experiment_scoring_functions=[compute_accuracy_stats],
experiment_name="My experiment"
)
```
Experiment score functions receive all test results after evaluation completes. Make sure your functions handle edge cases like empty test results or missing score values gracefully.
### Python SDK
#### Using async evaluation tasks
The `evaluate` function does not support `async` evaluation tasks, if you pass
an async task you will get an error similar to:
```python wordWrap
Input should be a valid dictionary [type=dict_type, input_value='
If you need to create a dataset with more than 1,000 rows, you [can use the SDK](/evaluation/advanced/manage_datasets#creating-a-dataset-using-the-sdk).
The UI dataset creation has some limitations:
* File size is limited to 1,000 rows via the UI.
* No support for nested JSON structures in the CSV itself.
For datasets requiring rich metadata, complex schemas, or programmatic control, use the SDK instead (see the next section).
When you create a dataset with a CSV file, this creates the first version (v1)
of your dataset. All subsequent modifications will create new versions automatically.
## Understanding dataset versioning
Dataset versioning in Opik creates **immutable snapshots** of your data. Every time you modify a dataset—whether adding, editing, or deleting items—a new version is automatically created. This ensures complete reproducibility, provides an audit trail of all changes, and allows easy rollback to any previous state.
Each dataset version contains:
* **Version name**: Auto-generated sequential name (v1, v2, v3, etc.)
* **Change description**: Optional note describing what changed
* **Tags**: Labels for categorizing versions (e.g., `production`, `baseline`)
* **Item statistics**: Count of items added, modified, and deleted
* **Timestamp and author**: When the version was created and by whom
Once a version is created, its data cannot be changed—any modification creates a new version instead. Restoring a previous version also creates a *new* version with the same data, preserving your complete version timeline.
The special `latest` tag always points to the most recent version.
When running experiments without specifying a version, `latest` is used by default.
## Working with draft mode (UI)
When making changes to a dataset in the Opik UI, all modifications go into a **draft state** first. This gives you a staging area to review changes before committing them as a new version. The draft is visible only to you, and AI-generated samples from "Expand with AI" also go to draft for review.
When a dataset has unsaved draft changes, an orange **"Draft"** tag appears next to the dataset name, and **Save changes** / **Discard changes** buttons appear in the toolbar. Items show colored borders: green for newly added items, amber for modified items.
### Saving or discarding changes
To commit your draft as a new version:
1. Click **Save changes** in the toolbar
2. Enter a **version note** describing what changed
3. Optionally add **tags** to categorize this version
4. Click **Save**
To abandon your draft, click **Discard changes** and confirm. If you try to navigate away with unsaved changes, Opik displays a warning to prevent accidental loss of work.
Use draft mode to batch related changes into a single, well-documented version.
## Version history
To view the complete timeline of dataset changes, navigate to your dataset and click the **Version history** tab. The table shows each version's name, change summary (items added/modified/deleted), version note, tags, item count, and creation timestamp.
From this view you can:
* **View items**: Click a version row and select **View items** to see the exact data at that point in time
* **Restore**: Click the **⋮** menu and select **Restore this version** to create a new version with that data
* **Edit metadata**: Click the **⋮** menu and select **Edit** to update the version note or tags (the data itself remains immutable)
Restoring a version creates a **new** version with the same data.
No history is lost or overwritten.
### Managing dataset and version tags from the SDK
The `Dataset` object exposes `get_tags()` to read the current tags, but does not yet provide a dedicated setter. To write tags programmatically — for example to drive an `env:prod` / `env:stage` promotion workflow — use the REST client exposed on the Opik client.
There are two tag surfaces, depending on what you want to scope the tag to:
* **Dataset-level tags** apply to the dataset as a whole and persist across versions. Use `update_dataset` — this **replaces** the existing tag list.
* **Version-level tags** apply to a specific dataset version. Use `update_dataset_version` — this is **additive** (it adds to the version's existing tags).
```python {pytest_codeblocks_skip=true}
import opik
client = opik.Opik()
dataset = client.get_or_create_dataset(name="my-eval", project_name="my-project")
# Read current dataset-level tags
print(dataset.get_tags())
# Set dataset-level tags (replaces the existing list)
client.rest_client.datasets.update_dataset(
id=dataset.id,
name=dataset.name,
tags=["env:prod"],
)
# Add tags to a specific version (additive)
client.rest_client.datasets.update_dataset_version(
id=dataset.id,
version_hash=dataset.version_hash,
tags_to_add=["env:prod"],
)
```
`client.rest_client` is a thin wrapper around the public REST API. The underlying endpoints are stable, but the Python wrapper itself is not guaranteed to be backward-compatible across SDK versions. A first-class `Dataset.set_tags()` / `Dataset.add_version_tags()` helper is on the roadmap — this snippet is the supported interim path.
You can then filter dataset items by these tags via [`get_items(filter_string=...)`](#querying-dataset-items) using the `tags contains` operator.
## Adding traces to a dataset
One of the most powerful ways to build evaluation datasets is by converting production traces into dataset items. This allows you to leverage real-world interactions from your LLM application to create test cases for evaluation.
### Adding traces via the UI
To add traces to a dataset from the Opik UI:
1. Navigate to the traces page
2. Select one or more traces you want to add to a dataset
3. Click the **Add to dataset** button in the toolbar
4. In the dialog that appears:
* Select an existing dataset or create a new one
* Choose which trace metadata to include:
* **Nested spans**: Include all child spans within the trace
* **Tags**: Include trace tags
* **Feedback scores**: Include any feedback scores attached to the trace
* **Comments**: Include comments added to the trace
* **Usage metrics**: Include token usage and cost information
* **Metadata**: Include custom metadata fields
5. Click on the dataset name to add the selected traces
By default, all metadata options are enabled. You can uncheck any options you don't need. The trace's input and output are always included.
### What gets added to the dataset
When you add a trace to a dataset, the following structure is created:
* **input**: The trace's input data
* **expected\_output**: The trace's output data (stored as `expected_output` for evaluation purposes)
* **spans** (optional): Array of nested spans with their inputs, outputs, and metadata
* **tags** (optional): Array of tags associated with the trace
* **feedback\_scores** (optional): Array of feedback scores with name, value, and source
* **comments** (optional): Array of comments with text and ID
* **usage** (optional): Token usage and cost information
* **metadata** (optional): Custom metadata fields
This rich structure allows you to:
* Evaluate complex multi-step workflows by including nested spans
* Filter and analyze based on tags and metadata
* Use existing feedback scores as ground truth for evaluation
* Preserve context through comments and annotations
## Creating a dataset using the SDK
In Opik 2.0, datasets are project-scoped. Specify a `project_name` to associate your dataset with the correct project.
You can create a dataset and log items to it using the `get_or_create_dataset` method:
```typescript title="TypeScript SDK" language="typescript"
import { Opik } from "opik";
// Create a dataset
const client = new Opik();
const dataset = await client.getOrCreateDataset("My dataset", "Evaluation dataset", "my-project");
```
```python title="Python SDK" language="python"
from opik import Opik
# Create a dataset
client = Opik()
dataset = client.get_or_create_dataset(name="My dataset", project_name="my-project")
```
If a dataset with the given name already exists, the existing dataset will be returned.
### Insert items
#### Inserting dictionary items
You can insert items to a dataset using the `insert` method:
```typescript title="TypeScript" language="typescript"
import { Opik } from "opik";
const client = new Opik();
const dataset = await client.getOrCreateDataset("My dataset", "Evaluation dataset", "my-project");
dataset.insert([
{ user_question: "Hello, world!", expected_output: { assistant_answer: "Hello, world!" } },
{ user_question: "What is the capital of France?", expected_output: { assistant_answer: "Paris" } },
]);
```
```python title="Python" language="python"
import opik
# Get or create a dataset
client = opik.Opik()
dataset = client.get_or_create_dataset(name="My dataset", project_name="my-project")
# Add dataset items to it
dataset.insert([
{"user_question": "Hello, world!", "expected_output": {"assistant_answer": "Hello, world!"}},
{"user_question": "What is the capital of France?", "expected_output": {"assistant_answer": "Paris"}},
])
```
Opik automatically deduplicates items that are inserted into a dataset when using the Python SDK. This means that you
can insert the same item multiple times without duplicating it in the dataset. This combined with the `get or create
dataset` methods means that you can use the SDK to manage your datasets in a "fire and forget" manner.
When using the SDK to insert items, a new dataset version is automatically created.
If you insert items in multiple batches within a single `insert()` call, they are grouped into one version.
Once the items have been inserted, you can view them in the Opik UI:
#### Inserting items from a JSONL file
You can also insert items from a JSONL file:
```python title="Python" language="python"
import opik
client = opik.Opik()
dataset = client.get_or_create_dataset(name="My dataset", project_name="my-project")
dataset.read_jsonl_from_file("path/to/file.jsonl")
```
#### Inserting items from a Pandas DataFrame
You can also insert items from a Pandas DataFrame:
```python title="Python" language="python"
import opik
client = opik.Opik()
dataset = client.get_or_create_dataset(name="My dataset", project_name="my-project")
dataset.insert_from_pandas(dataframe=df)
# You can also specify an optional keys_mapping parameter
dataset.insert_from_pandas(dataframe=df, keys_mapping={"Expected output": "expected_output"})
```
### Deleting items
You can delete items in a dataset by using the `delete` method:
```typescript title="TypeScript" language="typescript"
import { Opik } from "opik";
// Get or create a dataset
client = new Opik();
dataset = await client.getDataset("My dataset")
await dataset.delete(["123", "456"])
// Or to delete all items
await dataset.clear()
```
```python title="Python" language="python"
from opik import Opik
# Get or create a dataset
client = Opik()
dataset = client.get_dataset(name="My dataset")
dataset.delete(items_ids=["123", "456"])
# Or to delete all items
dataset.clear()
```
Deleting items creates a new version of the dataset. The deleted items remain accessible
in previous versions through the version history, ensuring you never permanently lose data.
## Downloading a dataset from Opik
You can download a dataset from Opik using the `get_dataset` method:
```typescript title="TypeScript" language="typescript"
import { Opik } from "opik";
const client = new Opik();
const dataset = await client.getDataset("My dataset");
const items = await dataset.getItems();
console.log(items);
```
```python title="Python" language="python"
from opik import Opik
client = Opik()
dataset = client.get_dataset(name="My dataset")
# Get items as list of DatasetItem objects
items = dataset.get_items()
# Convert to a Pandas DataFrame
dataset.to_pandas()
# Convert to a JSON array
dataset.to_json()
```
## Filtering datasets programmatically
You can filter dataset items using the `filter_string` parameter on the `get_items()` method or when
running evaluations with `evaluate_prompt()`. This allows you to work with specific subsets of your data.
### Basic filtering
```python title="Python" language="python"
from opik import Opik
client = Opik()
dataset = client.get_dataset(name="my_dataset")
# Get filtered items
failed_items = dataset.get_items(filter_string='tags contains "failed"')
```
### Filter syntax
The filter string uses Opik Query Language (OQL) syntax. Supported columns include:
| Column | Type | Description |
| ----------------- | ---------- | -------------------------------------------------------------- |
| `id` | String | Unique identifier for the dataset item |
| `source` | String | Source of the dataset item |
| `trace_id` | String | Associated trace ID |
| `span_id` | String | Associated span ID |
| `data` | Dictionary | Use dot notation for nested fields (e.g., `data.category`) |
| `tags` | List | Use "contains" operator (e.g., `tags contains "test"`) |
| `created_at` | DateTime | ISO 8601 format (e.g., `created_at >= "2024-01-01T00:00:00Z"`) |
| `last_updated_at` | DateTime | ISO 8601 format |
| `created_by` | String | User who created the item |
| `last_updated_by` | String | User who last updated the item |
### Filter examples
```python title="Python" language="python"
from opik import Opik
client = Opik()
dataset = client.get_dataset(name="my_dataset")
# Filter by tag
failed_items = dataset.get_items(filter_string='tags contains "failed"')
# Filter by data field
finance_items = dataset.get_items(filter_string='data.category = "finance"')
# Filter by date
recent_items = dataset.get_items(
filter_string='created_at >= "2024-06-01T00:00:00Z"'
)
# Multiple conditions
filtered_items = dataset.get_items(
filter_string='tags contains "production" AND data.difficulty = "hard"'
)
```
## Running experiments with dataset versions
When you run an experiment, Opik automatically links it to the specific dataset version that was used. This ensures complete reproducibility—you can always know exactly which data was used for any experiment.
### Automatic version association
Every experiment records which dataset version it used:
* When running from the UI or SDK without specifying a version, the `latest` version is used
* The experiment results page shows the associated dataset version
* You can click the version to see the exact data that was evaluated
This association is permanent. Even if you later modify the dataset, your experiment results remain linked to the original version used.
### Selecting a specific version in Playground
When running experiments from the Playground:
1. Open the Playground and configure your prompt
2. In the dataset selector, choose your dataset
3. A nested dropdown appears showing available versions
4. Select the specific version you want to use, or choose `latest` for the most recent
When comparing experiments or running A/B tests, use the same dataset version
to isolate the effect of your changes. This ensures differences in results
are due to your prompt or model changes, not data variations.
### Selecting a specific version in the SDK
When running experiments programmatically, you can specify which dataset version to use by passing a `DatasetVersion` object to `evaluate()`:
```python title="Python" language="python"
from opik import Opik
from opik.evaluation import evaluate
client = Opik()
dataset = client.get_dataset(name="My dataset")
# Run experiment on the latest version (default behavior)
result = evaluate(
experiment_name="baseline-experiment",
dataset=dataset,
task=my_task_function,
scoring_metrics=[my_metric],
project_name="my-project",
)
# Run experiment on a specific version
v1_view = dataset.get_version_view("v1")
result = evaluate(
experiment_name="v1-experiment",
dataset=v1_view, # Pass the DatasetVersion object
task=my_task_function,
scoring_metrics=[my_metric],
project_name="my-project",
)
```
```typescript title="TypeScript" language="typescript"
import { Opik, evaluate } from "opik";
const client = new Opik();
const dataset = await client.getDataset("My dataset");
// Run experiment on the latest version (default)
const result = await evaluate({
experimentName: "baseline-experiment",
dataset: dataset,
task: myTaskFunction,
scoringMetrics: [myMetric],
projectName: "my-project",
});
// Run experiment on a specific version
const v2 = await dataset.getVersionView("v2");
const pinnedResult = await evaluate({
experimentName: "pinned-experiment",
dataset: v2,
task: myTaskFunction,
scoringMetrics: [myMetric],
projectName: "my-project",
});
```
### Working with dataset versions programmatically
The SDK provides methods for inspecting and working with dataset versions:
```python title="Python" language="python"
from opik import Opik
client = Opik()
dataset = client.get_dataset(name="My dataset")
# Get the current (latest) version name
current_version = dataset.get_current_version_name()
print(f"Current version: {current_version}") # e.g., "v3"
# Get detailed version info (returns DatasetVersionPublic)
version_info = dataset.get_version_info()
print(f"Version ID: {version_info.id}")
print(f"Version name: {version_info.version_name}")
print(f"Items total: {version_info.items_total}")
print(f"Created at: {version_info.created_at}")
# Get a read-only view of a specific version
v1_view = dataset.get_version_view("v1")
# Access version metadata
print(f"Version: {v1_view.version_name}")
print(f"Items in v1: {v1_view.items_total}")
print(f"Items added: {v1_view.items_added}")
print(f"Items modified: {v1_view.items_modified}")
print(f"Items deleted: {v1_view.items_deleted}")
# Get items from a specific version
v1_items = v1_view.get_items()
# Export version data
v1_df = v1_view.to_pandas()
v1_json = v1_view.to_json()
```
```typescript title="TypeScript" language="typescript"
import { Opik } from "opik";
const client = new Opik();
const dataset = await client.getDataset("My dataset");
// Get the current (latest) version name
const currentVersion = await dataset.getCurrentVersionName();
console.log(`Current version: ${currentVersion}`); // e.g., "v3"
// Get detailed version info (returns DatasetVersionPublic)
const versionInfo = await dataset.getVersionInfo();
console.log(`Version ID: ${versionInfo?.id}`);
console.log(`Version name: ${versionInfo?.versionName}`);
console.log(`Items total: ${versionInfo?.itemsTotal}`);
console.log(`Created at: ${versionInfo?.createdAt}`);
// Get a read-only view of a specific version
const v1View = await dataset.getVersionView("v1");
// Access version metadata
console.log(`Version: ${v1View.versionName}`);
console.log(`Items in v1: ${v1View.itemsTotal}`);
console.log(`Items added: ${v1View.itemsAdded}`);
console.log(`Items modified: ${v1View.itemsModified}`);
console.log(`Items deleted: ${v1View.itemsDeleted}`);
// Get items from a specific version
const v1Items = await v1View.getItems();
// Export version data as JSON
const v1Json = await v1View.toJson();
```
`DatasetVersion` is a read-only view. You cannot insert, update, or delete items
through a `DatasetVersion` object. All mutations must be done through the `Dataset` object.
## Expanding a dataset with AI
Dataset expansion allows you to use AI to generate additional synthetic samples based on your existing dataset. This is particularly useful when you have a small dataset and want to create more diverse test cases to improve your evaluation coverage.
The AI analyzes the patterns in your existing data and generates new samples that follow similar structures while introducing variations. This helps you:
* **Increase dataset size** for more comprehensive evaluation
* **Create edge cases** and variations you might not have considered
* **Improve model robustness** by testing against diverse inputs
* **Scale your evaluation** without manual data creation
### How to expand a dataset
To expand a dataset with AI:
1. **Navigate to your dataset** in the Opik UI (Evaluation > Datasets > \[Your Dataset])
2. **Click the "Expand with AI" button** in the dataset view
3. **Configure the expansion settings**:
* **Model**: Choose the LLM model to use for generation (supports GPT-4, GPT-5, Claude, and other models)
* **Sample Count**: Specify how many new samples to generate (1-100)
* **Preserve Fields**: Select which fields from your original data to keep unchanged
* **Variation Instructions**: Provide specific guidance on how to vary the data (e.g., "Create variations that test edge cases" or "Generate examples with different complexity levels")
* **Custom Prompt**: Optionally provide a custom prompt template instead of the auto-generated one
4. **Start the expansion** - The AI will analyze your data and generate new samples
5. **Review the results** - Generated samples are added to your **draft**. You can review, edit, or remove them before saving to create a new version
### Configuration options
**Sample Count**: Start with a smaller number (10-20) to review the quality before generating larger batches.
**Preserve Fields**: Use this to maintain consistency in certain fields while allowing variation in others. For example, preserve the `category` field while varying the `input` and `expected_output`.
**Variation Instructions**: Provide specific guidance such as:
* "Create variations with different difficulty levels"
* "Generate edge cases and error scenarios"
* "Add examples with different input formats"
* "Include multilingual variations"
### Best practices
* **Start small**: Generate 10-20 samples first to evaluate quality before scaling up
* **Review generated content**: Always review AI-generated samples for accuracy and relevance
* **Use variation instructions**: Provide clear guidance on the type of variations you want
* **Preserve key fields**: Use field preservation to maintain important categorizations or metadata
* **Iterate and refine**: Use the custom prompt option to fine-tune generation for your specific needs
Dataset expansion works best when you have at least 5-10 high-quality examples in your original dataset. The AI uses
these examples to understand the patterns and generate similar but varied content.
## Managing dataset item tags
Tags are a powerful way to organize, categorize, and filter your dataset items. You can use tags to:
* **Categorize test cases** by type, difficulty, or domain (e.g., `edge-case`, `production`, `multilingual`)
* **Track data sources** where items originated from (e.g., `user-feedback`, `synthetic`, `real-world`)
* **Mark review status** during dataset curation (e.g., `needs-review`, `validated`, `archived`)
* **Filter for evaluation** to run experiments on specific subsets of your data
* **Organize workflows** by marking items for different stages or teams
Each dataset item can have multiple tags.
### Adding tags to dataset items
#### Adding tags to individual items
To add tags to a single dataset item:
1. **Navigate to your dataset** in the Opik UI (Evaluation > Datasets > \[Your Dataset])
2. **Click on any dataset item** to open the details panel
3. **In the Tags section**, click the **"+" button**
4. **Type the tag name** and press Enter
5. The tag will be immediately added and saved
You can remove tags by clicking the **"×" icon** next to any tag in the details panel.
#### Adding tags to multiple items (batch operation)
To add the same tag to multiple dataset items at once:
1. **Navigate to your dataset** in the Opik UI
2. **Select multiple items** by clicking the checkboxes next to each item
3. **Click the "Add tags" button** in the toolbar (visible when items are selected)
4. **Enter the tag name** in the dialog that appears
5. **Click "Add tag"** to apply the tag to all selected items
This is particularly useful when you want to categorize a group of related test cases or mark items from the same data source.
Tags are case-sensitive and support alphanumeric characters, hyphens, and underscores. Choose consistent naming conventions for your tags to make filtering easier.
### Filtering dataset items by tags
Once you've tagged your dataset items, you can filter them to work with specific subsets:
1. **Navigate to your dataset** in the Opik UI
2. **Click the "Filters" button** next to the search bar
3. **Select "Tags" from the Column dropdown**
4. **Choose "contains" as the operator**
5. **Enter the tag name** you want to filter by
6. **Close the dialog** to apply the filter
The dataset items table will update to show only items matching your filter criteria. You can:
* **View filtered items** to focus on specific categories
* **Run experiments** on filtered subsets by using the filtered view
* **Export filtered data** for specific test case groups
* **Combine with other filters** to create complex queries
The filter is saved in the URL, so you can bookmark or share specific filtered views of your dataset.
## Bulk operations
Opik supports bulk operations for efficiently managing large datasets. These operations help you work with many items at once without tedious individual selections.
### Select all functionality
When working with datasets that span multiple pages:
1. **Select items on the current page** using the checkbox in the table header
2. A banner appears offering to **"Select all items"** across all pages
3. Click to select all items matching your current filter criteria
This works with filtered views too—if you have a filter applied, "Select all" only selects items matching that filter.
### Available bulk operations
Once you have items selected, the toolbar shows available operations:
* **Add tags**: Apply one or more tags to all selected items
* **Delete**: Remove selected items (creates a new version with items removed)
* **Export**: Download selected items as CSV or JSON
### Processing indicators
For large bulk operations:
* A loading indicator shows "Your dataset is still processing..."
* The operation runs in the background—you can continue browsing
* A success message appears when processing completes
For very large datasets, bulk operations are processed in batches. The UI remains
responsive during processing, and you'll see progress indicators for long-running operations.
# Evaluate agent trajectories
In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a `project_name` when creating datasets and running experiments so they are associated with the correct project.
Evaluating agents requires more than checking the final output. You need to assess The
**trajectory** — the steps your agent takes to reach an answer, including tool selection, reasoning
chains, and intermediate decisions.
Agent trajectory evaluation helps you catch tool selection errors, identify inefficient reasoning
paths, and optimize agent behavior before it reaches production.
## Prerequisites
Before evaluating agent trajectories, you need:
1. **Opik SDK installed and configured** — See [Quickstart](/quickstart) for setup
2. **Agent with observability enabled** — Your agent must be instrumented with Opik tracing
3. **Test dataset** — Examples with expected agent behavior
If your agent isn't traced yet, see [Log Traces](/tracing/advanced/log_traces) to add observability first.
### Installing the Opik SDK
To install the Opik Python SDK you can run the following command:
```bash
pip install opik
```
Then you can configure the SDK by running the following command:
```bash
opik configure
```
This will prompt you for your API key and workspace or your instance URL if you are self-hosting.
### Adding observability to your agent
In order to be able to evaluate the agent's trajectory, you need to add tracing to your agent. This
will allow us to capture the agent's trajectory and evaluate it.
```python title="LangChain" language="python" {2,4,22} maxLines=25
from langchain.agents import create_agent
from opik.integrations.langchain import OpikTracer
opik_tracer = OpikTracer()
def get_weather(city: str) -> str:
"""Get weather for a given city."""
return f"It's always sunny in {city}!"
agent = create_agent(
model="openai:gpt-4o",
tools=[get_weather],
system_prompt="You are a helpful assistant"
)
# Run the agent
agent.invoke(
{"messages": [{
"role": "user",
"content": "what is the weather in sf"
}]},
config={"callbacks": [opik_tracer]}
)
```
```python title="OpenAI" language="python" {4,5,7,24,29} maxLines=25
import json
import openai
from opik import track
from opik.integrations.openai import track_openai
openai_client = track_openai(openai.OpenAI())
# Define tools
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get weather for a given city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
}
}
}
]
@track(type="tool")
def get_weather(city: str) -> str:
"""Get weather for a given city."""
return f"It's always sunny in {city}!"
@track
def agent_with_tools(user_input: str):
messages = [{"role": "user", "content": user_input}]
while True:
response = openai_client.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=tools,
tool_choice="auto"
)
if response.choices[0].finish_reason == "tool_calls":
messages.append(response.choices[0].message)
for tool_call in response.choices[0].message.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
tool_result = get_weather(tool_args["city"])
user_input = f"The weather in {tool_args['city']} is {tool_result}"
messages.append({
"role": "tool",
"content": tool_result,
"tool_call_id": tool_call.id
})
else:
break
return messages
```
If you're using specific agent frameworks like CrewAI, LangGraph, or OpenAI Agents, check our
[integrations](/integrations/overview) for framework-specific setup instructions.
## Evaluating your agent's trajectory
In order to evaluate the agent's trajectory, we will need to create a dataset, define an evaluation
metric and then run the evaluation.
### Creating a dataset
We are going to create a dataset with a set of user questions and some expected tools that the
agent should be calling:
```python
from opik import Opik
client = Opik()
dataset = client.get_or_create_dataset(name="agent_tool_selection", project_name="my-project")
dataset.insert([
{
"input": "What is 25 * 17?",
"expected_tool": []
},
{
"input": "What is the weather in SF?",
"expected_tool": ["get_weather"]
},
{
"input": "What is the weather in NY?",
"expected_tool": ["get_weather"]
}
])
```
The format of dataset items is very flexible, you can include any fields you want in each item.
### Defining the evaluation metric
In this task, we are going to measure `Strict Tool Adherence` which measures the agent's adherence
to the expected tools in the same order as they are expected.
The key to this metric is the use of the optional `task_span` parameter, this is available for all
custom metrics and can be used to access the agent's trajectory:
```python title="Strict Tool Adherence Metric" maxLines=1000
from opik.evaluation.metrics import BaseMetric, score_result
from opik.message_processing.emulation.models import SpanModel
from typing import List
class StrictToolAdherenceMetric(BaseMetric):
def __init__(self, name: str = "strict_tool_adherence"):
self.name = name
def find_tools(self, task_span):
"""Find all tool spans in the SpanModel hierarchy."""
tools_used = []
def extract_tools_from_spans(spans):
"""Recursively extract tools from spans list."""
for span in spans:
# Check if this span is a tool
if span.type == "tool" and span.name:
tools_used.append(span.name)
# Recursively check nested spans
if span.spans:
extract_tools_from_spans(span.spans)
# Start the recursive search from the top level spans
if task_span.spans:
extract_tools_from_spans(task_span.spans)
return tools_used
def score(self, task_span: SpanModel,
expected_tool: List[str], **kwargs):
# Find tool calls in trajectory
tool_used = self.find_tools(task_span)
if tool_used == expected_tool:
return score_result.ScoreResult(
value=1.0,
name=self.name,
reason=f"Correct: used {tool_used}"
)
else:
return score_result.ScoreResult(
value=0.0,
name=self.name,
reason=f"Used {tool_used}, expected {expected_tool}"
)
```
```python title="Tool Adherence Metric" maxLines=1000
from opik.evaluation.metrics import BaseMetric, score_result
from opik.message_processing.emulation.models import SpanModel
from typing import List
class ToolAdherenceMetric(BaseMetric):
def __init__(self, name: str = "tool_adherence"):
self.name = name
def find_tools(self, task_span):
"""Recursively find all tool spans in the SpanModel hierarchy."""
tools_used = []
def extract_tools_from_spans(spans):
"""Recursively extract tools from spans list."""
for span in spans:
# Check if this span is a tool
if span.type == "tool" and span.name:
tools_used.append(span.name)
# Recursively check nested spans
if span.spans:
extract_tools_from_spans(span.spans)
# Start the recursive search from the task_span's spans
if task_span.spans:
extract_tools_from_spans(task_span.spans)
return tools_used
def score(self, task_span: SpanModel,
expected_tool: List[str], **kwargs):
# Find tool calls in trajectory
tool_used = self.find_tools(task_span)
if set(tool_used) == set(expected_tool):
return score_result.ScoreResult(
value=1.0,
name=self.name,
reason=f"Correct: used {tool_used}"
)
else:
return score_result.ScoreResult(
value=0.0,
name=self.name,
reason=f"Used {tool_used}, expected {expected_tool}"
)
```
### Running the evaluation
Let's define our evaluation task that will run our agent and return the assistant's response:
```python title="LangChain" maxLines=1000
def evaluation_task(dataset_item: dict) -> dict:
res = agent.invoke(
{"messages": [{
"role": "user",
"content": dataset_item["input"]
}]},
config={"callbacks": [opik_tracer]}
)
return {"output": res['messages'][-1].content}
```
```python title="OpenAI" maxLines=1000
def evaluation_task(dataset_item: dict) -> dict:
res = agent_with_tools(dataset_item["input"])
return {"output": messages[-1]['content']}
```
Now that we have our dataset and metric, we can run the evaluation:
```python title="Running the evaluation" maxLines=1000
from opik.evaluation import evaluate
# Run the evaluation
experiment = evaluate(
dataset=dataset,
task=evaluation_task,
scoring_metrics=[StrictToolAdherenceMetric()],
project_name="my-project"
)
```
### Analyzing the results
The Opik experiment dashboard provides a rich set of tools to help you analyze the results of the
trajectory evaluation.
You can see the results of the evaluation in the Opik UI:
If you click on a specific test case row, you can view the full trajectory of the agent's execution
using the `Trace` button.
## Next Steps
Now that you can evaluate agent trajectories:
* Learn about [Task Span Metrics](/evaluation/metrics/task_span_metrics) for advanced trajectory
analysis patterns
* Optimize your agent with [Agent Optimization](/development/optimization-runs/overview)
* Monitor agents in production with [Production Monitoring](/tracing/dashboards/production_monitoring)
# Evaluate multi-turn agents
In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a `project_name` when creating datasets and running experiments so they are associated with the correct project.
When working on chatbots or multi-turn agents, it can be challenging to evaluate the agent's
behavior over multiple turns because you don't know what the user would ask as a follow-up question.
To solve this, we can use an LLM to simulate the user — generating realistic follow-up messages
based on the conversation so far and running this for a configurable number of turns.
Once we have this conversation, we can use Opik evaluation features to score the agent's behavior.
## Creating the user simulator
In order to perform multi-turn evaluation, we need to create a user simulator that will generate
the user's response based on previous turns
```python title="User simulator" maxLines=1000
from opik.simulation import SimulatedUser
user_simulator = SimulatedUser(
persona="You are a frustrated user who wants a refund",
model="openai/gpt-4.1",
)
conversation_history = [
{"role": "assistant", "content": "Hello, how can I help you today?"}
]
for turn in range(3):
# Generate a user message based on the conversation so far
user_message = user_simulator.generate_response(conversation_history)
conversation_history.append({"role": "user", "content": user_message})
print(f"User: {user_message}")
# In practice, this would be your agent's response
agent_response = f"Placeholder agent response for turn {turn + 1}"
conversation_history.append({"role": "assistant", "content": agent_response})
print(f"Assistant: {agent_response}\n")
```
Now that we have a way to simulate the user, we can create multiple simulations that we will in
turn evaluate.
## Running simulations
In order to more easily keep track of the scenarios we will be running, let's create a
dataset with the user personas we will be using:
```python title="Create dataset with user personas" maxLines=1000
import opik
opik_client = opik.Opik()
dataset = opik_client.get_or_create_dataset(name="Multi-turn evaluation", project_name="my-project")
dataset.insert([
{"user_persona": "You are a frustrated user who wants a refund"},
{"user_persona": "You are a user who is happy with your product and wants to buy more"},
{"user_persona": "You are a user who is having trouble with your product and wants to get help"}
])
```
The `run_simulation` function expects an `app` callable with the following contract: it
receives a `user_message` string and a `thread_id` keyword argument, and returns a message
dict `{"role": "assistant", "content": "..."}`. The app is responsible for managing its own
conversation history using the `thread_id`.
Here is an example using LangChain:
```python title="Example agent app (LangChain)" maxLines=1000
from langchain.agents import create_agent
from opik.integrations.langchain import OpikTracer
opik_tracer = OpikTracer()
agent = create_agent(
model="openai:gpt-4.1",
tools=[],
system_prompt="You are a helpful assistant",
)
agent_history = {}
def run_agent(user_message: str, *, thread_id: str, **kwargs) -> dict[str, str]:
if thread_id not in agent_history:
agent_history[thread_id] = []
agent_history[thread_id].append({"role": "user", "content": user_message})
messages = agent_history[thread_id]
response = agent.invoke({"messages": messages}, config={"callbacks": [opik_tracer]})
agent_history[thread_id] = response["messages"]
return {"role": "assistant", "content": response["messages"][-1].content}
```
Now that we have a dataset with the user personas, we can run the simulations:
```python title="Run simulations" maxLines=1000
import opik
from opik.simulation import SimulatedUser, run_simulation
# Fetch the user personas
opik_client = opik.Opik()
dataset = opik_client.get_or_create_dataset(name="Multi-turn evaluation", project_name="my-project")
# Run the simulations
all_simulations = []
for item in dataset.get_items():
user_persona = item["user_persona"]
user_simulator = SimulatedUser(
persona=user_persona,
model="openai/gpt-4.1",
)
simulation = run_simulation(
app=run_agent,
user_simulator=user_simulator,
max_turns=5,
)
all_simulations.append(simulation)
```
Each simulation result is a dictionary with:
* `thread_id`: Unique identifier for the conversation thread
* `conversation_history`: List of message dicts (`{"role": "user"|"assistant", "content": "..."}`)
The `run_simulation` function keeps track of the internal conversation state by constructing
a list of messages with the result of the `run_agent` function as an assistant message and
the `SimulatedUser`'s response as a user message.
If you need more complex conversation state, you can create threads using the `SimulatedUser`'s
`generate_response` method directly.
The simulated threads will be available in the Opik thread UI:
## Scoring threads
When working on evaluating multi-turn conversations, you can use one of Opik's built-in conversation
metrics or [create your own](/evaluation/metrics/custom_conversation_metric).
If you've used the `run_simulation` function, you will already have a list of conversation messages
that you can pass directly to the metrics, otherwise you can use the `evaluate_threads` function:
```python title="Scoring simulations" maxLines=1000
import opik
from opik.evaluation.metrics import ConversationalCoherenceMetric, UserFrustrationMetric
opik_client = opik.Opik()
# Define the metrics you want to use
conversation_coherence_metric = ConversationalCoherenceMetric()
user_frustration_metric = UserFrustrationMetric()
for simulation in all_simulations:
conversation = simulation["conversation_history"]
coherence_score = conversation_coherence_metric.score(conversation)
frustration_score = user_frustration_metric.score(conversation)
opik_client.log_threads_feedback_scores(
scores=[
{
"id": simulation["thread_id"],
"name": "conversation_coherence",
"value": coherence_score.value,
"reason": coherence_score.reason
},
{
"id": simulation["thread_id"],
"name": "user_frustration",
"value": frustration_score.value,
"reason": frustration_score.reason
}
]
)
```
```python title="Using evaluate_threads"
import opik
from opik.evaluation import evaluate_threads
from opik.evaluation.metrics import ConversationalCoherenceMetric, UserFrustrationMetric
opik_client = opik.Opik()
conversation_coherence_metric = ConversationalCoherenceMetric()
user_frustration_metric = UserFrustrationMetric()
results = evaluate_threads(
project_name="multi_turn_evaluation",
filter_string=f'thread_id = "
## Next steps
* Learn more about [conversation metrics](/evaluation/metrics/conversation_threads_metrics)
* Learn more about [custom conversation metrics](/evaluation/metrics/custom_conversation_metric)
* Learn more about [evaluate\_threads](/evaluation/evaluate_threads)
* Learn more about [agent trajectory evaluation](/evaluation/advanced/evaluate_agent_trajectory)
# Annotation Queues
Involving subject matter experts in AI projects is essential because they provide the domain knowledge and contextual judgment that ensures model outputs are accurate, relevant, and aligned with real-world expectations. Annotation Queues in Opik make it simple for subject matter experts (SMEs) to review and annotate agent outputs. This feature streamlines the human-in-the-loop process by providing easy queue management, simple invitation flows, and a distraction-free annotation experience designed for non-technical users.
Annotation Queues are collections of traces or threads that need human review and feedback. They enable you to organize content for review, share with SMEs easily, collect structured feedback, and track progress across all your evaluation workflows.
## Creating and Managing Annotation Queues
Each annotation queue is defined by a collection of traces or threads, evaluation instructions, and feedback definitions:
1. **Queue Configuration**: Set up the queue with clear instructions and scope
2. **Content Selection**: Add traces or threads that need human review
3. **SME Access**: Share queue links with subject matter experts for annotation
### Setting Up Your First Queue
Navigate to the **Annotation Queues** page in your project and click **Create Queue**.
Configure your queue with:
* **Name**: Clear identification for your queue
* **Scope**: Choose between traces or threads
* **Instructions**: Provide context and guidance for reviewers
* **Feedback Definitions**: Select the metrics SMEs will use for scoring
### Adding Content to Your Queue
You can add items to your queue in several ways:
**From Traces/Threads Lists:**
* Select one or multiple items
* Click **Add to -> Add to annotation queue**
* Choose an existing queue or create a new one
**From Individual Trace/Thread Details:**
* Open the trace or thread detail view
* Click **Add to -> Add to annotation queue** in the actions panel
* Select your target queue
### Sharing with Subject Matter Experts
Once your queue is set up, you can share it with SMEs:
**Copy Queue Link:**
**SME Access Required**: Subject matter experts must be invited to your
workspace before they can access annotation queues. Make sure to invite them
to your project first, then share the queue link.
* Click the **Share queue** button on your queue to copy the queue link
* Share the link directly with SMEs via email, Slack, or other communication tools
## SME Annotation Experience
When SMEs access a queue, they experience a streamlined, distraction-free interface designed for efficient review.
The annotation workflow begins with clear instructions and context, allowing SMEs to understand what they're evaluating and how to provide meaningful feedback.
The SME interface provides:
1. **Clean, focused design**: No technical jargon or complex navigation
2. **Clear instructions**: Queue-specific guidance displayed prominently
3. **Structured feedback**: Predefined metrics with clear descriptions
4. **Progress tracking**: Visual indicators of completion status
5. **Comment system**: Optional text feedback for additional context
### Annotation Workflow
1. **Access the queue**: SME clicks the shared link
2. **Review content**: Examine the trace or thread output
3. **Provide feedback**: Score using predefined metrics
4. **Add comments**: Optional text feedback
5. **Submit and continue**: Move to the next item
## Managing Queues programmatically
You can create and manage annotation queues programmatically using the Python or TypeScript SDK. This is useful for automating the process of adding items to queues based on specific criteria.
### Creating an Annotation Queue
### Creating a Traces Annotation Queue
```python
import opik
client = opik.Opik()
# Create a traces annotation queue
queue = client.create_traces_annotation_queue(
name="High Priority Traces",
description="Traces that need review",
instructions="Check for accuracy and completeness",
feedback_definition_names=["relevance", "accuracy"]
)
print(f"Created queue: {queue.name} (ID: {queue.id})")
```
```typescript
import { Opik } from "opik";
const client = new Opik();
// Create a trace annotation queue
const queue = await client.createTracesAnnotationQueue({
name: "High Priority Traces",
description: "Traces that need review",
instructions: "Check for accuracy and completeness",
feedbackDefinitionNames: ["relevance", "accuracy"],
});
console.log(`Created queue: ${queue.name} (ID: ${queue.id})`);
```
### Adding Traces to a Queue
```python
import opik
client = opik.Opik()
# Get an existing traces queue
queue = client.get_traces_annotation_queue("queue-id")
# Search for traces and add them to the queue
traces = client.search_traces(
project_name="my-project",
filter_string='feedback_scores.user_frustration > 0.5'
)
queue.add_traces(traces)
# For a single trace, wrap it in a list
single_trace = client.get_trace_content("trace-id")
queue.add_traces([single_trace])
```
```typescript
import { Opik } from "opik";
const client = new Opik();
// Get an existing traces queue
const queue = await client.getTracesAnnotationQueue("queue-id");
// Search for traces and add them to the queue
const traces = await client.searchTraces({
projectName: "my-project",
filterString: 'feedback_scores.user_frustration > 0.5',
});
await queue.addTraces(traces);
// For a single trace, wrap it in an array
const singleTraceResponse = await client.api.traces.getTraceById("trace-id");
await queue.addTraces([singleTraceResponse.data]);
```
### Working with Thread Queues
```python
import opik
client = opik.Opik()
# Create a threads annotation queue
thread_queue = client.create_threads_annotation_queue(
name="Thread Review Queue",
description="Threads needing review"
)
# Get threads and add them to the queue
threads_client = client.get_threads_client()
important_threads = threads_client.search_threads(
project_name="my-project",
filter_string='tags contains "important"'
)
thread_queue.add_threads(important_threads)
```
```typescript
import { Opik } from "opik";
const client = new Opik();
// Create a thread annotation queue
const threadQueue = await client.createThreadsAnnotationQueue({
name: "Thread Review Queue",
description: "Threads needing review",
});
// Search for threads and add them to the queue
const threads = await client.searchThreads({
projectName: "my-project",
filterString: 'tags contains "important"',
});
await threadQueue.addThreads(threads);
```
### Updating and Deleting Queues
```python
import opik
client = opik.Opik()
# Get an existing traces queue
queue = client.get_traces_annotation_queue("queue-id")
# Update queue properties
queue.update(
description="Traces that need a thorough review",
instructions="Check the conversation tone"
)
# Remove traces from the queue
short_traces = client.search_traces(
project_name="my-project",
filter_string='duration < 10'
)
queue.remove_traces(short_traces)
# Delete the queue
queue.delete()
```
### Listing Annotation Queues
```python
import opik
client = opik.Opik()
# Get all traces annotation queues for a project
traces_queues = client.get_traces_annotation_queues(project_name="my-project")
for queue in traces_queues:
print(f"Traces Queue: {queue.name}, Items: {queue.items_count}")
# Get all threads annotation queues for a project
threads_queues = client.get_threads_annotation_queues(project_name="my-project")
for queue in threads_queues:
print(f"Threads Queue: {queue.name}, Items: {queue.items_count}")
```
```typescript
import { Opik } from "opik";
const client = new Opik();
// Get an existing traces queue
const queue = await client.getTracesAnnotationQueue("queue-id");
// Update queue properties
await queue.update({
description: "Traces that need a thorough review",
instructions: "Check the conversation tone",
});
// Remove traces from the queue
const shortTraces = await client.searchTraces({
projectName: "my-project",
filterString: "duration < 10",
});
await queue.removeTraces(shortTraces);
// Delete the queue
await queue.delete();
```
### Listing Annotation Queues
```python
import opik
client = opik.Opik()
# Get all traces annotation queues for a project
traces_queues = client.get_traces_annotation_queues(project_name="my-project")
for queue in traces_queues:
print(f"Traces Queue: {queue.name}, Items: {queue.items_count}")
# Get all threads annotation queues for a project
threads_queues = client.get_threads_annotation_queues(project_name="my-project")
for queue in threads_queues:
print(f"Threads Queue: {queue.name}, Items: {queue.items_count}")
```
```typescript
import { Opik } from "opik";
const client = new Opik();
// Get all traces annotation queues for a project
const tracesQueues = await client.getTracesAnnotationQueues({
projectName: "my-project",
});
for (const queue of tracesQueues) {
const itemsCount = await queue.getItemsCount();
console.log(
`Queue: ${queue.name}, Scope: ${queue.scope}, Items: ${itemsCount}`
);
}
// Get all threads annotation queues for a project
const threadsQueues = await client.getThreadsAnnotationQueues({
projectName: "my-project",
});
for (const queue of threadsQueues) {
const itemsCount = await queue.getItemsCount();
console.log(
`Queue: ${queue.name}, Scope: ${queue.scope}, Items: ${itemsCount}`
);
}
```
## Learn more
You can learn more about Opik's annotation and evaluation features in:
1. [Evaluation overview](/evaluation/overview)
2. [Feedback definitions](/administration/workspace-settings/feedback_definitions)
# Manually logging experiments
In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a `project_name` when creating datasets and running experiments so they are associated with the correct project.
Evaluating your LLM application allows you to have confidence in the performance of your LLM
application. In this guide, we will walk through manually creating experiments using data you have
already computed.
This guide focuses on logging pre-computed evaluation results. If you're looking to run evaluations with Opik
computing the metrics, refer to the [Evaluate your agent](/evaluation/advanced/evaluate_your_llm) guide.
The process involves these key steps:
1. Create a dataset with your test cases
2. Prepare your evaluation results
3. Log experiment items in bulk
## 1. Create a Dataset
First, you'll need to create a dataset containing your test cases. This dataset will be linked to
your experiments.
```typescript title="TypeScript" language="typescript" maxLines=1000
import { Opik } from "opik";
const client = new Opik({
apiKey: "your-api-key",
apiUrl: "https://www.comet.com/opik/api",
projectName: "your-project-name",
workspaceName: "your-workspace-name",
});
const dataset = await client.getOrCreateDataset("My dataset");
await dataset.insert([
{
user_question: "What is the capital of France?",
expected_output: "Paris"
},
{
user_question: "What is the capital of Japan?",
expected_output: "Tokyo"
},
{
user_question: "What is the capital of Brazil?",
expected_output: "Brasília"
}
]);
```
```python title="Python" language="python" maxLines=1000
from opik import Opik
import opik
# Configure Opik
opik.configure()
# Create dataset items
dataset_items = [
{
"user_question": "What is the capital of France?",
"expected_output": "Paris"
},
{
"user_question": "What is the capital of Japan?",
"expected_output": "Tokyo"
},
{
"user_question": "What is the capital of Brazil?",
"expected_output": "Brasília"
}
]
# Get or create a dataset
client = Opik()
dataset = client.get_or_create_dataset(name="geography-questions", project_name="my-project")
# Add dataset items
dataset.insert(dataset_items)
```
```bash title="REST API" maxLines=1000
# First, create the dataset
curl -X POST 'https://www.comet.com/opik/api/v1/private/datasets' \
-H 'Content-Type: application/json' \
-H 'Comet-Workspace: 
This guide uses a Python agent built with LangGraph to illustrate tracing and evaluation. If you're using other
frameworks like OpenAI Agents, CrewAI, Haystack, or LlamaIndex, you can check out our [Integrations
Overview](/integrations/overview) to get started with tracing in your setup.
Once you’ve logged your first traces, Opik gives you immediate access to valuable insights, not just about what your agent did, but how it performed. You can explore detailed trace data, see how many traces and spans your agent is generating, track token usage, and monitor response latency across runs.
For each interaction with the end user, you can also know how the agent planned, chose tools, or crafted an answer based on the user input, the agent graph and much more.
During development phase, having access to all this information is fundamental for debugging and understanding what is working as expected and what’s not.
**Error detection**
Having immediate access to all traces that returned an error can also be life-saving, and Opik makes it extremely easy to achieve:
For each of the errors and exceptions captured, you have access to all the details you need to fix the issue:
### 2. Evaluate Agent's End-to-end Behavior
Once you have full visibility on the agent interactions, memory and tool usage, and you made sure everything is working at the technical level, the next logical step is to start checking the quality of the responses and the actions your agent takes.
**Human Feedback**
The fastest and easiest way to do it is providing manual human feedback. Each trace and each span can be rated “Correct” or “Incorrect” by a person (most probably you!) and that will give a baseline to understand the quality of the responses.
You can provide human feedback and a comment for each trace’s score in Opik and when you’re done you can store all results in a dataset that you will be using in next iterations of agent optimization.
**Online evaluation**
Marking an answer as simply “correct” or “incorrect” is a useful first step, but it’s rarely enough. As your agent grows more complex, you’ll want to measure how well it performs across more nuanced dimensions.
That’s where online evaluation becomes essential.
With Opik, you can automatically score traces using a wide range of metrics, such as answer relevance, hallucination detection, agent moderation, user moderation, or even custom criteria tailored to your specific use case. These evaluations run continuously, giving you structured feedback on your agent’s quality without requiring manual review.
Want to dive deeper? Check out the [Metrics Documentation](/evaluation/metrics/overview) to explore all the heuristic
metrics and LLM-as-a-judge evaluations that Opik offers out of the box.
### 3. Evaluate Agent’s Steps
When building complex agents, evaluating only the final output isn't enough. Agents reason through **sequences of actions**—choosing tools, calling functions, retrieving memories, and generating intermediate messages.
Each of these **steps** can introduce errors long before they show up in the final answer.
That’s why **evaluating agent steps independently** is a core best practice.
Without step-level evaluation, you might only notice failures after they impact the final user response, without knowing where things went wrong.
With step evaluation, you can catch issues as they occur and identify exactly which part of your agent’s reasoning or architecture needs fixing.
#### **What Steps Should You Evaluate?**
Depending on your agent architecture, you might want to score:
| Step Type | Example Evaluation Questions |
| ------------------------- | --------------------------------------------------------------------------------- |
| **Tool Calls** | Did the agent pick the right tool for the job? Did it provide correct parameters? |
| **Memory Retrievals** | Was the retrieved memory relevant to the query? |
| **Plans** | Did the agent generate a coherent, executable plan? |
| **Intermediate Messages** | Was the internal reasoning logical and consistent? |
For each of those steps you can use one of Opik’s predefined metrics or create your own custom metric that adapts to your needs.
### 4. Example: Evaluating Tool Selection Quality with Opik
When building agents that use tools (like web search, calculators, APIs…), it’s critical to know **how well your agent is choosing and using those tools**.
Are they picking the right tool? Are they using it correctly? Are they wasting time or making mistakes?
The easiest way to measure this in Opik is by running a **custom evaluation experiment**.
#### **What We'll Do**
In this example, we'll use Opik's SDK to create a **script that will run an experiment** to **measure how well an agent selects tools**.
When you run the experiment, Opik will:
* Execute the agent against every item in a dataset of examples.
* Evaluate each agent interaction using a custom metric.
* Log results (scores and reasoning) into a dashboard you can explore.
This will give you a **clear, data-driven view** of how good (or bad!) your agent’s tool selection behavior really is.
#### **What We Need**
For every Experiment we want to run, the most important elements we need to create are the following:
A set of example user queries and expected correct tool usage.
A way to automatically decide if the agent’s behavior was correct or not (we’ll create a custom one).
A function that tells Opik how to run your agent on each dataset item.
#### Full Example: Tool Selection Evaluation Script
Here’s the full example:
```python
import os
from opik import Opik
from opik.evaluation import evaluate
from agent import agent_executor
from langchain_core.messages import HumanMessage
from experiments.tool_selection_metric import ToolSelectionQuality
os.environ["OPIK_API_KEY"] = "YOUR_API_KEY"
os.environ["OPIK_WORKSPACE"] = "YOUR_WORKSPACE"
client = Opik()
# This is the dataset with the examples of good tool selection
dataset = client.get_dataset(name="Your_Dataset")
"""
Note: if you don't have a dataset yet, you can easily create it this way:
dataset = client.get_or_create_dataset(name="My_Dataset", project_name="my-project")
# Define the items
items = [
{
"input": "Find information about adding numbers.",
"expected_output": "tavily_search_results_json"
},
{
"input": "Multiply 7×6",
"expected_output": "simple_math_tool"
}
[...]
]
# Insert the dataset items
dataset.insert(items)
"""
# This function defines how each item in the dataset will be evaluated.
# For each dataset item:
# - It sends the `input` as a message to the agent (`agent_executor`).
# - It captures the agent's actual tool calls from its outputs.
# - It packages the original input, the agent's outputs, the detected tool calls, and the expected tool calls.
# This structured output is what the evaluation platform will use to compare expected vs actual behavior using the custom metric(s) you define.
def evaluation_task(dataset_item):
try:
user_message_content = dataset_item["input"]
expected_tool = dataset_item["expected_output"]
# This is where you call your agent with the input message and get the real execution results.
result = agent_executor.invoke({"messages": [HumanMessage(content=user_message_content)]})
tool_calls = []
# Here we extract the tool calls the agent actually made.
# We loop through the agent's messages, check tool calls,
# and for each tool call, we capture its metadata.
for msg in result.get("messages", []):
if hasattr(msg, "tool_calls") and msg.tool_calls:
for tool_call in msg.tool_calls:
tool_calls.append({
"function_name": tool_call.get("name"),
"function_parameters": tool_call.get("args", {})
})
return {
"input": user_message_content,
"output": result,
"tool_calls": tool_calls,
"expected_tool_calls": [{"function_name": expected_tool, "function_parameters": {}}]
}
except Exception as e:
return {
"input": dataset_item.get("input", {}),
"output": "Error processing input.",
"tool_calls": [],
"expected_tool_calls": [{"function_name": "unknown", "function_parameters": {}}],
"error": str(e)
}
# This is the custom metric we have defined
metrics = [ToolSelectionQuality()]
# This function runs the full evaluation process.
# It loops over each dataset item and applies the `evaluation_task` function to generate outputs.
# It then applies the custom `ToolSelectionQuality` metric (or any provided metrics) to score each result.
# It logs the evaluation results to Opik under the specified experiment name ("AgentToolSelectionExperiment").
# This allows tracking, comparing, and analyzing your agent's tool selection quality over time in Opik.
eval_results = evaluate(
experiment_name="AgentToolSelectionExperiment",
dataset=dataset,
task=evaluation_task,
scoring_metrics=metrics,
project_name="my-project"
)
```
The Custom Tool Selection metric looks like this:
```python
from opik.evaluation.metrics import base_metric, score_result
class ToolSelectionQuality(base_metric.BaseMetric):
def __init__(self, name: str = "tool_selection_quality"):
self.name = name
def score(self, tool_calls, expected_tool_calls, **kwargs):
try:
actual_tool = tool_calls[0]["function_name"]
expected_tool = expected_tool_calls[0]["function_name"]
if actual_tool == expected_tool:
return score_result.ScoreResult(
name=self.name,
value=1,
reason=f"Correct tool selected: {actual_tool}"
)
else:
return score_result.ScoreResult(
name=self.name,
value=0,
reason=f"Wrong tool. Expected {expected_tool}, got {actual_tool}"
)
except Exception as e:
return score_result.ScoreResult(
name=self.name,
value=0,
reason=f"Scoring error: {e}"
)
```
After running this script:
* You will see a **new experiment in Opik**.
* Each item will have a **tool selection score** and a **reason** explaining why it was correct or incorrect.
* You can then **analyze results**, **filter mistakes**, and **build better training data** for your agent.
This method is a scalable way to **move from gut feelings to hard evidence** when improving your agent's behavior.
#### What Happens Next? Iterate, Improve, and Compare
Running the experiment once gives you a **baseline**: a first measurement of how good (or bad) your agent's tool selection behavior is.
But the real power comes from **using these results to improve your agent** — and then **re-running the experiment** to measure progress.
Here’s how you can use this workflow:
See where your agent is making tool selection mistakes.
{" "}
Look at the most common errors and read the reasoning behind low scores.
{" "}
Update the system prompt to improve instructions, refine tool descriptions, and
adjust tool names or input formats to be more intuitive.
{" "}
Use the same dataset to measure how your changes affected tool selection quality.
{" "}
Review improvements in score, spot reductions in errors, and identify new patterns or regressions.
{" "}
Iterate as many times as needed to reach the level of performance you want from your agent.
And this is just for one module! You can next move to the next component of your agent
**You can evaluate modules with metrics like the following:**
* **Router**: tool selection and parameter extraction
* **Tools**: Output accuracy, hallucinations
* **Planner**: Plan length, validity, sufficiency
* **Paths**: Looping, redundant steps
* **Reflection**: Output quality, retry logic
### 5. Wrapping Up: Where to Go From Here
Building great agents is a journey that doesn’t stop at getting them to “work.”
It’s about creating agents you can trust, understand, and continuously improve.
In this guide, you’ve learned how to make agent behavior observable, how to evaluate outputs and reasoning steps, and how to design experiments that drive real, measurable improvements.
But this is just the beginning.
From here, you might want to:
* Optimize your prompts to drive better agent behavior with **[Prompt Optimization](/development/optimization-runs/overview)**.
* Monitor agents in production to catch regressions, errors, and drift in real-time with **[Production Monitoring](/tracing/dashboards/production_monitoring)**.
* Add **[Guardrails](/production/gateway-guardrails/guardrails)** for security, content safety, and sensitive data leakage prevention, ensuring your agents behave responsibly even in dynamic environments.
Each of these steps builds on the foundation you’ve set: observability, evaluation, and continuous iteration.
By combining them, you’ll be ready to take your agents from early prototypes to production-grade systems that are powerful, safe, and scalable.
# Evaluate threads
When you are running multi-turn conversations using frameworks that support LLM agents, the Opik integration will
automatically group related traces into conversation threads using parameters suitable for each framework.
This guide will walk you through the process of evaluating and optimizing conversation threads in Opik using
the `evaluate_threads` function in the Python SDK.
For complete API reference documentation, see the [`evaluate_threads` API reference](https://www.comet.com/docs/opik/python-sdk-reference/evaluation/evaluate_threads.html).
## Using the Python SDK
The Python SDK provides a simple and efficient way to evaluate and optimize conversation threads using the
`evaluate_threads` function. This function allows you to specify a filter string to select specific threads for
evaluation, a list of metrics to apply to each thread, and it returns a `ThreadsEvaluationResult` object
containing the evaluation results and feedback scores.
Most importantly, this function **automatically uploads the feedback scores to your traces in Opik!**
So, once evaluation is completed, you can also [see the results in the UI](#using-opik-ui-to-view-results).
To run the threads evaluation, you can use the following code:
```python
from opik.evaluation import evaluate_threads
from opik.evaluation.metrics import ConversationalCoherenceMetric, UserFrustrationMetric
# Initialize the evaluation metrics
conversation_coherence_metric = ConversationalCoherenceMetric()
user_frustration_metric = UserFrustrationMetric()
# Run the threads evaluation
results = evaluate_threads(
project_name="ai_team",
filter_string='id = "0197ad2a"',
eval_project_name="ai_team_evaluation",
metrics=[
conversation_coherence_metric,
user_frustration_metric,
],
trace_input_transform=lambda x: x["input"],
trace_output_transform=lambda x: x["output"],
)
```
Want to create your own custom conversation metrics? Check out the [Custom Conversation Metrics guide](/evaluation/metrics/custom_conversation_metric) to learn how to build specialized metrics for evaluating multi-turn dialogues.
### Understanding the Transform Arguments
Threads consist of multiple traces, and each trace has an input and output. In practice, these typically contain user messages and agent responses. However, trace inputs and outputs are rarely just simple strings—they are usually complex data structures whose exact format depends on your agent framework.
To handle this complexity, you need to provide `trace_input_transform` and `trace_output_transform` functions. These are **critical parameters** that tell Opik how to extract the actual message content from your framework-specific trace structure.
#### Why Transform Functions Are Needed
Different agent frameworks structure their trace data differently:
* **LangChain** might store messages in `{"messages": [{"content": "..."}]}`
* **CrewAI** might use `{"task": {"description": "..."}}`
* **Custom implementations** can have any structure you've defined
Without transform functions, Opik wouldn't know where to find the actual user questions and agent responses within your trace data.
#### How Transform Functions Work
Using these functions, the Opik evaluation engine will convert your threads chosen for evaluation into the standardized format expected by all Opik thread evaluation metrics:
```json
[
{
"role": "user",
"content": "input string from trace 1"
},
{
"role": "assistant",
"content": "output string from trace 1"
},
{
"role": "user",
"content": "input string from trace 2"
},
{
"role": "assistant",
"content": "output string from trace 2"
}
]
```
**Example:**
If your trace input has the following structure:
```json
{
"content": {
"user_question": "Tell me about your service?"
},
"metadata": {...}
}
```
Then your `trace_input_transform` should be:
```python
lambda x: x["content"]["user_question"]
```
Don't want to deal with transformations because your traces don't have a consistent format? Try using LLM-based transformations, language models are good at this!.
#### Using filter string
The `evaluate_threads` function takes a filter string as an argument. This string is used to select the threads that
should be evaluated. For example, if you want to evaluate only threads that have a specific ID, you can use the
following filter string:
```python
filter_string='id = "0197ad2a"'
```
You can combine multiple filter strings using the `AND` operator. For example, if you want to evaluate only threads
that have a specific ID and were created after a certain date, you can use the following filter string:
```python
filter_string='id = "0197ad2a" AND start_time > "2024-01-01T00:00:00Z"'
```
**Supported filter fields and operators**
The `evaluate_threads` function supports the following filter fields in the `filter_string` using Opik Query Language (OQL).
All fields and operators are the same as those supported by `search_traces` and `search_spans`:
| Field | Type | Operators |
| ------------------------- | ---------- | --------------------------------------------------------------------------- |
| `id` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
| `name` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
| `created_by` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
| `thread_id` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
| `type` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
| `model` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
| `provider` | String | `=`, `!=`, `contains`, `not_contains`, `starts_with`, `ends_with`, `>`, `<` |
| `status` | String | `=`, `contains`, `not_contains` |
| `start_time` | DateTime | `=`, `>`, `<`, `>=`, `<=` |
| `end_time` | DateTime | `=`, `>`, `<`, `>=`, `<=` |
| `input` | String | `=`, `contains`, `not_contains` |
| `output` | String | `=`, `contains`, `not_contains` |
| `metadata` | Dictionary | `=`, `contains`, `>`, `<` |
| `feedback_scores` | Numeric | `=`, `>`, `<`, `>=`, `<=`, `is_empty`, `is_not_empty` |
| `tags` | List | `contains` |
| `usage.total_tokens` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
| `usage.prompt_tokens` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
| `usage.completion_tokens` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
| `duration` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
| `number_of_messages` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
| `total_estimated_cost` | Numeric | `=`, `!=`, `>`, `<`, `>=`, `<=` |
**Rules:**
* String values must be wrapped in double quotes
* DateTime fields require ISO 8601 format (e.g., "2024-01-01T00:00:00Z")
* Use dot notation for nested objects: `metadata.model`, `feedback_scores.accuracy`
* Multiple conditions can be combined with `AND` (OR is not supported)
The `feedback_scores` field is a dictionary where the keys are the metric names and the values are the metric values.
You can use it to filter threads based on their feedback scores. For example, if you want to evaluate only threads
that have a specific user frustration score, you can use the following filter string:
```python
filter_string='feedback_scores.user_frustration_score >= 0.5'
```
Where `user_frustration_score` is the name of the user frustration metric and `0.5` is the threshold value to filter by.
**Best practice**: If you are using SDK for thread evaluation, automate it by setting up a scheduled cron job with filters to regularly generate feedback scores for specific traces.
## Using Opik UI to view results
Once the evaluation is complete, you can access the evaluation results in the Opik UI.
Not only you will be able to see the score values, but the LLM-judge reasoning behind these values too!
**SDK Evaluation vs. Manual Feedback:**
* When using the SDK's `evaluate_threads` function, only threads marked as "inactive" (after the cooldown period) are evaluated. This ensures you're scoring complete conversations.
* You can manually add feedback scores to any thread at any time through the UI or API, regardless of its status.
* For thread-level online evaluation rules (automatic scoring), Opik waits for a configurable "cooldown period" after the last activity before running the rules.
## Multi-Value Feedback Scores for Threads
**Team-based thread evaluation** enables multiple evaluators to score conversation threads independently, providing more reliable assessment of multi-turn dialogue quality.
**Key benefits for thread evaluation:**
* **Conversation complexity scoring** - Multiple reviewers can assess different aspects like coherence, user satisfaction, and goal completion across conversation turns
* **Reduced evaluation bias** - Individual subjectivity in judging conversational quality is mitigated through team consensus
* **Thread-specific metrics** - Teams can collaboratively evaluate conversation-specific aspects like frustration levels, topic drift, and resolution success
This collaborative approach is especially valuable for conversational threads where dialogue quality, context maintenance, and user experience assessment often require multiple expert perspectives.
## Next steps
For more details on what metrics can be used to score conversational threads, refer to
the [conversational metrics](/evaluation/metrics/conversation_threads_metrics) page.
You can also define custom metrics to evaluate conversational threads, including
LLM-as-a-Judge (LLM-J) reasoning metrics, as described in the following section:
[Custom Conversation Metrics guide](/evaluation/metrics/custom_conversation_metric).
# Cookbook - Evaluate hallucination metric
# Evaluating Opik's Hallucination Metric
In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a `project_name` when creating datasets and running experiments so they are associated with the correct project.
For this guide we will be evaluating the Hallucination metric included in the LLM Evaluation SDK which will showcase both how to use the `evaluation` functionality in the platform as well as the quality of the Hallucination metric included in the SDK.
## Creating an account on Comet.com
[Comet](https://www.comet.com/site/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=eval_hall\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=eval_hall\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=eval_hall\&utm_campaign=opik) for more information.
```python
%pip install opik pyarrow pandas fsspec huggingface_hub --upgrade --quiet
```
```python
import opik
opik.configure(use_local=False)
```
## Preparing our environment
First, we will install configure the OpenAI API key and create a new Opik dataset
```python
import os
import getpass
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
```
We will be using the [HaluEval dataset](https://huggingface.co/datasets/pminervini/HaluEval?library=pandas) which according to this [paper](https://arxiv.org/pdf/2305.11747) ChatGPT detects 86.2% of hallucinations. The first step will be to create a dataset in the platform so we can keep track of the results of the evaluation.
Since the insert methods in the SDK deduplicates items, we can insert 50 items and if the items already exist, Opik will automatically remove them.
```python
# Create dataset
import opik
import pandas as pd
client = opik.Opik()
# Create dataset
dataset = client.get_or_create_dataset(name="HaluEval", description="HaluEval dataset", project_name="my-project")
# Insert items into dataset
df = pd.read_parquet(
"hf://datasets/pminervini/HaluEval/general/data-00000-of-00001.parquet"
)
df = df.sample(n=50, random_state=42)
dataset_records = [
{
"input": x["user_query"],
"llm_output": x["chatgpt_response"],
"expected_hallucination_label": x["hallucination"],
}
for x in df.to_dict(orient="records")
]
dataset.insert(dataset_records)
```
## Evaluating the hallucination metric
In order to evaluate the performance of the Opik hallucination metric, we will define:
* Evaluation task: Our evaluation task will use the data in the Dataset to return a hallucination score computed using the Opik hallucination metric.
* Scoring metric: We will use the `Equals` metric to check if the hallucination score computed matches the expected output.
By defining the evaluation task in this way, we will be able to understand how well Opik's hallucination metric is able to detect hallucinations in the dataset.
```python
from opik.evaluation.metrics import Hallucination, Equals
from opik.evaluation import evaluate
from opik import Opik
from opik.evaluation.metrics.llm_judges.hallucination.template import generate_query
from typing import Dict
# Define the evaluation task
def evaluation_task(x: Dict):
metric = Hallucination()
try:
metric_score = metric.score(input=x["input"], output=x["llm_output"])
hallucination_score = metric_score.value
hallucination_reason = metric_score.reason
except Exception as e:
print(e)
hallucination_score = None
hallucination_reason = str(e)
return {
"hallucination_score": "yes" if hallucination_score == 1 else "no",
"hallucination_reason": hallucination_reason,
}
# Get the dataset
client = Opik()
dataset = client.get_dataset(name="HaluEval")
# Define the scoring metric
check_hallucinated_metric = Equals(name="Correct hallucination score")
# Add the prompt template as an experiment configuration
experiment_config = {
"prompt_template": generate_query(
input="{input}", context="{context}", output="{output}", few_shot_examples=[]
)
}
res = evaluate(
dataset=dataset,
task=evaluation_task,
scoring_metrics=[check_hallucinated_metric],
experiment_config=experiment_config,
project_name="my-project",
scoring_key_mapping={
"reference": "expected_hallucination_label",
"output": "hallucination_score",
},
)
```
We can see that the hallucination metric is able to detect \~80% of the hallucinations contained in the dataset and we can see the specific items where hallucinations were not detected.
# Cookbook - Evaluate moderation metric
# Evaluating Opik's Moderation Metric
In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a `project_name` when creating datasets and running experiments so they are associated with the correct project.
For this guide we will be evaluating the Moderation metric included in the LLM Evaluation SDK which will showcase both how to use the `evaluation` functionality in the platform as well as the quality of the Moderation metric included in the SDK.
## Creating an account on Comet.com
[Comet](https://www.comet.com/site?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=eval_mod\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=eval_mod\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=eval_mod\&utm_campaign=opik) for more information.
```python
%pip install --upgrade --quiet opik pandas
```
```python
import opik
opik.configure(use_local=False)
```
## Preparing our environment
First, we will configure the OpenAI API key and download a reference moderation dataset.
```python
import os
import getpass
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
```
We will be using the [OpenAI Moderation API Release dataset](https://github.com/openai/moderation-api-release/tree/main/data) which according to this [blog post](https://openai.com/index/using-gpt-4-for-content-moderation/) GPT-4o detects ~~60~~% of hallucinations. The first step will be to create a dataset in the platform so we can keep track of the results of the evaluation.
Since the insert methods in the SDK deduplicates items, we can insert 50 items and if the items already exist, Opik will automatically remove them.
```python
# Create dataset
import opik
import pandas as pd
import requests
from io import BytesIO
client = opik.Opik()
# Create dataset
dataset = client.get_or_create_dataset(
name="OpenAIModerationDataset", description="OpenAI Moderation Dataset", project_name="my-project"
)
# Insert items into dataset
url = "https://github.com/openai/moderation-api-release/raw/main/data/samples-1680.jsonl.gz"
response = requests.get(url)
df = pd.read_json(BytesIO(response.content), lines=True, compression="gzip")
df = df.sample(n=50, random_state=42)
dataset_records = []
for x in df.to_dict(orient="records"):
moderation_fields = ["S", "H", "V", "HR", "SH", "S3", "H2", "V2"]
moderated_fields = [field for field in moderation_fields if x[field] == 1.0]
expected_output = "moderated" if moderated_fields else "not_moderated"
dataset_records.append(
{
"output": x["prompt"],
"expected_output": expected_output,
"moderated_fields": moderated_fields,
}
)
dataset.insert(dataset_records)
```
## Evaluating the moderation metric
In order to evaluate the performance of the Opik moderation metric, we will define:
* Evaluation task: Our evaluation task will use the data in the Dataset to return a moderation score computed using the Opik moderation metric.
* Scoring metric: We will use the `Equals` metric to check if the moderation score computed matches the expected output.
By defining the evaluation task in this way, we will be able to understand how well Opik's moderation metric is able to detect moderation violations in the dataset.
We can use the Opik SDK to compute a moderation score for each item in the dataset:
```python
from opik.evaluation.metrics import Moderation, Equals
from opik.evaluation import evaluate
from opik import Opik
from opik.evaluation.metrics.llm_judges.moderation.template import generate_query
from typing import Dict
# Define the evaluation task
def evaluation_task(x: Dict):
metric = Moderation()
try:
metric_score = metric.score(output=x["output"])
moderation_score = "moderated" if metric_score.value > 0.5 else "not_moderated"
moderation_reason = metric_score.reason
except Exception as e:
print(e)
moderation_score = None
moderation_reason = str(e)
return {
"moderation_score": moderation_score,
"moderation_reason": moderation_reason,
}
# Get the dataset
client = Opik()
dataset = client.get_dataset(name="OpenAIModerationDataset")
# Define the scoring metric
moderation_metric = Equals(name="Correct moderation score")
# Add the prompt template as an experiment configuration
experiment_config = {
"prompt_template": generate_query(output="{output}", few_shot_examples=[])
}
res = evaluate(
dataset=dataset,
task=evaluation_task,
scoring_metrics=[moderation_metric],
experiment_config=experiment_config,
project_name="my-project",
scoring_key_mapping={"reference": "expected_output", "output": "moderation_score"},
)
```
We are able to detect \~85% of moderation violations, this can be improved further by providing some additional examples to the model. We can view a breakdown of the results in the Opik UI:
# Online Evaluation rules
Online evaluation metrics allow you to score all your production traces and easily identify any issues with your
production LLM application.
When working with LLMs in production, the sheer number of traces means that it isn't possible to manually review each trace. Opik allows you to define LLM as a Judge metrics that will automatically score the LLM calls logged to the platform.
By defining LLM as a Judge metrics that run on all your production traces, you will be able to
automate the monitoring of your LLM calls for hallucinations, answer relevance or any other task
specific metric.
## Defining scoring rules
Scoring rules can be defined through both the UI and the [REST API](/reference/rest-api/overview).
To create a new scoring metric in the UI, first navigate to the project you would like to monitor. Once you have navigated to the `rules` tab, you will be able to create a new rule.
When creating a new rule, you will be presented with the following options:
1. **Name:** The name of the rule
2. **Sampling rate:** The percentage of traces to score. When set to `100%`, all traces will be scored.
3. **Model:** The model to use to run the LLM as a Judge metric. For evaluating traces with images, make sure to select a model that supports vision capabilities.
4. **Prompt:** The LLM as a Judge prompt to use. Opik provides a set of base prompts (Hallucination, Moderation, Answer Relevance) that you can use or you can define your own. Variables in the prompt should be in `{{variable_name}}` format.
5. **Variable mapping:** This is the mapping of the variables in the prompt to the values from the trace.
6. **Score definition:** This is the format of the output of the LLM as a Judge metric. By adding more than one score, you can define LLM as a Judge metrics that score an LLM output along different dimensions.
### Opik's built-in LLM as a Judge metrics
Opik comes pre-configured with 3 different LLM as a Judge metrics:
1. Hallucination: This metric checks if the LLM output contains any hallucinated information.
2. Moderation: This metric checks if the LLM output contains any offensive content.
3. Answer Relevance: This metric checks if the LLM output is relevant to the given context.
If you would like us to add more LLM as a Judge metrics to the platform, do raise an issue on
[GitHub](https://github.com/comet-ml/opik/issues) and we will do our best to add them !
### Writing your own LLM as a Judge metric
Opik's built-in LLM as a Judge metrics are very easy to use and are great for getting started. However, as you start working on more complex tasks, you may need to write your own LLM as a Judge metrics.
We typically recommend that you experiment with LLM as a Judge metrics during development using [Opik's evaluation platform](/evaluation/overview). Once you have a metric that works well for your use case, you can then use it in production.
When writing your own LLM as a Judge metric you will need to specify the prompt variables using the mustache syntax, ie.
`{{ variable_name }}`. You can then map these variables to your trace data using the `variable_mapping` parameter. When the
rule is executed, Opik will replace the variables with the values from the trace data.
You can control the format of the output using the `Scoring definition` parameter. This is where you can define the scores you want the LLM as a Judge metric to return. Under the hood, we will use this definition in conjunction with the [structured outputs](https://platform.openai.com/docs/guides/structured-outputs) functionality to ensure that the LLM as a Judge metric always returns trace scores.
### Evaluating traces with images
LLM as a Judge metrics can evaluate traces that contain images when using vision-capable models. This is useful for:
* Evaluating image generation quality
* Analyzing visual content in multimodal applications
* Validating image-based responses
To reference image data from traces in your evaluation prompts:
1. In the prompt editor, click the **"Images +"** button to add an image variable
2. Map the image variable to the trace field containing image data using the Variable Mapping section
When you add an image using the "Images +" button, Opik automatically adds a new line to the prompt with the image wrapped in `<<
We have built-in templates for the LLM as a Judge metrics that you can use to score the entire conversation:
1. **Conversation Coherence:** This metric checks if the conversation is coherent and follows a logical flow, return a decimal score between 0 and 1.
2. **User Frustration:** This metric checks if the user is frustrated with the conversation, return a decimal score between 0 and 1.
3. **Custom LLM as a Judge metrics:** You can use this template to score the entire conversation using your own LLM as a Judge metric. By default, this template uses binary scoring (true/false) following best practices.
For the LLM as a Judge metrics, keep in mind the only variable available is the `{{context}}` one, which is a dictionary containing the entire conversation:
```json
[
{
"role": "user",
"content": "Hello, how are you?"
},
{
"role": "assistant",
"content": "I'm good, thank you!"
}
]
```
Similarly, for the Python metrics, you have the `Conversation` object available to you. This object is a `List[Dict]` where each dict represents a message in the conversation.
```python
[
{
"role": "user",
"content": "Hello, how are you?"
},
{
"role": "assistant",
"content": "I'm good, thank you!"
}
]
```
For online scoring rules on threads, Opik waits for a "cooldown period" after the last activity in a thread
before running the evaluation. This ensures the scoring is done on the full context of the conversation.
The default cooldown period is 15 minutes but can be configured at the workspace level under "Thread online scoring rule cooldown period".
For self-hosted installations, set the `OPIK_TRACE_THREAD_TIMEOUT_TO_MARK_AS_INACTIVE` environment variable.
## Running online evaluations on historical data
By default, a newly created online evaluation rule will only run on traces or threads logged after the rule was defined. To run a rule against historical data, you can trigger it manually from the UI:
1. Navigate to the **Traces** or **Threads** tab of your project.
2. Select the traces or threads you want to evaluate.
3. Click the brain icon () in the toolbar.
4. You will be prompted to choose which online evaluation rule to apply to the selected items.
Depending on the model configured in the rule, evaluations may take some time to complete. You can monitor progress in the evaluation rule logs or by refreshing the traces/threads in the UI.
# Gateway
An LLM gateway is a proxy server that forwards requests to an LLM API and returns the response. This is useful for when you want to centralize the access to LLM providers or when you want to be able to query multiple LLM providers from a single endpoint using a consistent request and response format.
## Gateway Integrations
Opik supports several LLM gateway solutions to help you centralize and manage your LLM provider access:
# Guardrails
This feature is currently available in the **self-hosted** installation of Opik. Support for managed deployments is
coming soon.
Guardrails help you protect your application from risks inherent in LLMs.
Use them to check the inputs and outputs of your LLM calls, and detect issues like
off-topic answers or leaking sensitive information.
# How it works
Conceptually, we need to determine the presence of a series of risks for each input and
output, and take action on it.
The ideal method depends on the type of the problem,
and aims to pick the best combination of accuracy, latency and cost.
There are three commonly used methods:
1. **Heuristics or traditional NLP models**: ideal for checking for PII or competitor mentions
2. **Small language models**: ideal for staying on topic
3. **Large language models**: ideal for detecting complex issues like hallucination
# Types of guardrails
Providers like OpenAI or Anthropic have built-in guardrails for risks like harmful or
malicious content and are generally desirable for the vast majority of users.
The Opik Guardrails aim to cover the residual risks which are often very user specific, and need to be configured with more detail.
## PII guardrail
The PII guardrail checks for sensitive information, such as name, age, address, email, phone number, or credit card details.
The specific entities can be configured in the SDK call, see more in the reference documentation.
*The method used here leverages traditional NLP models for tokenization and named entity recognition.*
## Topic guardrail
The topic guardrail ensures that the inputs and outputs remain on topic.
You can configure the allowed or disallowed topics in the SDK call, see more in the reference documentation.
*This guardrails relies on a small language model, specifically a zero-shot classifier.*
## Custom guardrail
Custom guardrail allows you to define your own guardrails using a custom model, custom library or custom business logic and log the response to Opik. Below is a basic example that filters out competitor brands:
```python
import opik
import opik.opik_context
import traceback
# Brand mention detection
competitor_brands = [
"OpenAI",
"Anthropic",
"Google AI",
"Microsoft Copilot",
"Amazon Bedrock",
"Hugging Face",
"Mistral AI",
"Meta AI",
]
opik_client = opik.Opik()
def custom_guardrails(generation: str, trace_id: str) -> str:
# Start the guardrail span first so the duration is accurately captured
guardrail_span = opik_client.span(name="Guardrail", input={"generation": generation}, type="guardrail", trace_id=trace_id)
# Custom guardrail logic - detect competitor brand mentions
found_brands = []
for brand in competitor_brands:
if brand.lower() in generation.lower():
found_brands.append(brand)
# The key `guardrail_result` is required by Opik guardrails and must be either "passed" or "failed"
if found_brands:
guardrail_result = "failed"
output = {"guardrail_result": guardrail_result, "found_brands": found_brands}
else:
guardrail_result = "passed"
output = {"guardrail_result": guardrail_result}
# Log the spans
guardrail_span.end(output=output)
# Upload the guardrail data for project-level metrics
guardrail_data = {
"project_name": opik_client._project_name,
"entity_id": trace_id,
"secondary_id": guardrail_span.id,
"name": "TOPIC", # Supports either "TOPIC" or "PII"
"result": guardrail_result,
"config": {"blocked_brands": competitor_brands},
"details": output,
}
try:
opik_client.rest_client.guardrails.create_guardrails(guardrails=[guardrail_data])
except Exception as e:
traceback.print_exc()
return generation
@opik.track
def main():
good_generation = "You should use our AI platform for your machine learning projects!"
custom_guardrails(good_generation, opik.opik_context.get_current_trace_data().id)
bad_generation = "You might want to try OpenAI or Google AI for your project instead."
custom_guardrails(bad_generation, opik.opik_context.get_current_trace_data().id)
if __name__ == "__main__":
main()
```
After running the custom guardrail example above, you can view the results in the Opik dashboard. The guardrail spans will appear alongside your traces, showing which brand mentions were detected and whether the guardrail passed or failed.
# Getting started
## Running the guardrail backend
You can start the guardrails backend by running:
```bash
./opik.sh --guardrails
```
## Using the Python SDK
```python
from opik.guardrails import Guardrail, PII, Topic
from opik import exceptions
guardrail = Guardrail(
guards=[
Topic(restricted_topics=["finance", "health"], threshold=0.9),
PII(blocked_entities=["CREDIT_CARD", "PERSON"]),
]
)
llm_response = "You should buy some NVIDIA stocks!"
try:
guardrail.validate(llm_response)
except exceptions.GuardrailValidationFailed as e:
print(e)
```
The immediate result of a guardrail failure is an exception, and your application code will need to handle it.
The call is blocking, since the main purpose of the guardrail is to prevent the application from proceeding with a potentially undesirable response.
### Guarding streaming responses and long inputs
You can call `guardrail.validate` repeatedly to validate the response chunk by chunk, or their parts or combinations.
The results will be added as additional spans to the same trace.
```python
for chunk in response:
try:
guardrail.validate(chunk)
except exceptions.GuardrailValidationFailed as e:
print(e)
```
## Working with the results
### Examining specific traces
When a guardrail fails on an LLM call, Opik automatically adds the information to the trace.
You can filter the traces in your project to only view those that have failed the guardrails.
### Analyzing trends
You can also view how often each guardrail is failing in the Metrics section of the project.
## Performance and limit considerations
The guardrails backend will use a GPU automatically if there is one available.
For production use, running the guardrails backend on a GPU node is strongly recommended.
Current limits:
* Topic guardrail: the maximum input size is 1024 tokens
* Both Topic and PII guardrails support English language
# Anonymizers
Anonymizers are available in both **cloud** and **self-hosted** installations of Opik.
Anonymizers help you protect sensitive information in your LLM applications by automatically detecting and replacing personally identifiable information (PII) and other sensitive data before it's logged to Opik. This ensures compliance with privacy regulations and prevents accidental exposure of sensitive information in your trace data.
# How it works
Anonymizers work by processing all data that flows through Opik's tracing system - including inputs, outputs, and metadata - before it's stored or displayed. They apply a set of rules to detect and replace sensitive information with anonymized placeholders.
The anonymization happens automatically and transparently:
1. **Data Ingestion**: When you log traces and spans to Opik
2. **Rule Application**: Registered anonymizers scan the data using their configured rules
3. **Replacement**: Sensitive information is replaced with anonymized placeholders
4. **Storage**: Only the anonymized data is stored in Opik
# Types of Anonymizers
## Rules-based Anonymizer
The most common type of anonymizer uses pattern-matching rules to identify and replace sensitive information. Rules can be defined in several formats:
### Regex Rules
Use regular expressions to match specific patterns:
```python
import opik
from opik.anonymizer import create_anonymizer
# Dictionary format
email_rule = {"regex": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "replace": "[EMAIL]"}
# Tuple format
phone_rule = (r"\b\d{3}-\d{3}-\d{4}\b", "[PHONE]")
# Create anonymizer with multiple rules
anonymizer = create_anonymizer([email_rule, phone_rule])
# Register globally
opik.hooks.add_anonymizer(anonymizer)
```
### Function Rules
Use custom Python functions for more complex anonymization logic:
```python
import opik
from opik.anonymizer import create_anonymizer
def mask_api_keys(text: str) -> str:
"""Custom function to anonymize API keys"""
import re
# Match common API key patterns
api_key_pattern = r'\b(sk-[a-zA-Z0-9]{32,}|pk_[a-zA-Z0-9]{24,})\b'
return re.sub(api_key_pattern, '[API_KEY]', text)
def anonymize_with_hash(text: str) -> str:
"""Replace emails with consistent hashes for tracking without exposing PII"""
import re
import hashlib
def hash_replace(match):
email = match.group(0)
hash_val = hashlib.md5(email.encode()).hexdigest()[:8]
return f"[EMAIL_{hash_val}]"
email_pattern = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
return re.sub(email_pattern, hash_replace, text)
# Create anonymizer with function rules
anonymizer = create_anonymizer([mask_api_keys, anonymize_with_hash])
opik.hooks.add_anonymizer(anonymizer)
```
### Mixed Rules
Combine different rule types for comprehensive anonymization:
```python
import opik
import opik.hooks
from opik.anonymizer import create_anonymizer
# Mix of dictionary, tuple, and function rules
mixed_rules = [
{"regex": r"\b\d{3}-\d{2}-\d{4}\b", "replace": "[SSN]"}, # Social Security Numbers
(r"\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b", "[CARD]"), # Credit Cards
lambda text: text.replace("CONFIDENTIAL", "[REDACTED]"), # Custom replacements
]
anonymizer = create_anonymizer(mixed_rules)
opik.hooks.add_anonymizer(anonymizer)
```
## Custom Anonymizers
For advanced use cases, create custom anonymizers by extending the `Anonymizer` base class.
### Understanding Anonymizer Arguments
When implementing custom anonymizers, you need to implement the `anonymize()` method with the following signature:
```python
def anonymize(self, data, **kwargs):
# Your anonymization logic here
return anonymized_data
```
**The `kwargs` parameters:**
The `anonymize()` method also receives additional context through `**kwargs`:
* **`field_name`**: Indicates which field is being anonymized (`"input"`, `"output"`, `"metadata"`, or nested field names in dots notation such as `"metadata.email"`)
* **`object_type`**: The type of the object being processed (`"span"`, `"trace"`)
**When are kwargs available?**
These kwargs are automatically passed by Opik's internal data processors when anonymizing trace and span data before sending it to the backend. This allows you to apply different anonymization strategies based on the field being processed.
**Example: Field-specific anonymization**
```python
from opik.anonymizer import Anonymizer
import opik.hooks
class FieldAwareAnonymizer(Anonymizer):
def anonymize(self, data, **kwargs):
field_name = kwargs.get("field_name", "")
# Only anonymize the output field, leave input as-is for debugging
if field_name == "output" and isinstance(data, str):
import re
# More aggressive anonymization for outputs
data = re.sub(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[EMAIL]', data)
data = re.sub(r'\b\d{3}-\d{3}-\d{4}\b', '[PHONE]', data)
elif field_name == "metadata" and isinstance(data, dict):
# Remove specific metadata fields entirely
sensitive_keys = ["user_id", "session_token", "api_key"]
for key in sensitive_keys:
if key in data:
data[key] = "[REDACTED]"
return data
# Register the field-aware anonymizer
opik.hooks.add_anonymizer(FieldAwareAnonymizer())
```
The `field_name` and `object_type` kwargs are primarily useful for implementing context-aware anonymization logic. If you don't need field-specific behavior, you can safely ignore these kwargs.
**Example: Anonymization of nested data structures**
Also, you can extend the `RecursiveAnonymizer` base class to work with nested data structures.
This allows you to apply the same anonymization logic to all nested fields. In this case you
need to implement the `anonymize_text()` method instead of `anonymize()`.
```python
from typing import Any, Optional
from opik.anonymizer import RecursiveAnonymizer
import opik.hooks
class SSNAnonymizer(RecursiveAnonymizer):
def anonymize_text(self, data: str, field_name: Optional[str] = None, **kwargs: Any) -> str:
if field_name == "metadata.ssn":
return "[SSN_REMOVED]"
return data
```
### Advanced Custom Anonymizer Example
```python
import opik
import opik.hooks
from opik.anonymizer import Anonymizer
class AdvancedPIIAnonymizer(Anonymizer):
def anonymize(self, data, **kwargs):
"""Custom anonymizer with advanced PII detection and removal."""
field_name = kwargs.get("field_name")
object_type = kwargs.get("object_type")
# Handle different data types
if isinstance(data, dict):
# Remove sensitive keys entirely
if "api_key" in data:
del data["api_key"]
if "password" in data:
del data["password"]
# Anonymize specific fields
for key, value in data.items():
if key.lower() in ["email", "user_email"]:
data[key] = "[EMAIL_REDACTED]"
elif key.lower() in ["phone", "telephone", "mobile"]:
data[key] = "[PHONE_REDACTED]"
elif isinstance(data, str):
# Apply string-based anonymization
import re
# Names (simple heuristic)
data = re.sub(r'\b[A-Z][a-z]+ [A-Z][a-z]+\b', '[NAME]', data)
# Addresses
data = re.sub(r'\d+\s+\w+\s+(Street|St|Avenue|Ave|Road|Rd|Drive|Dr)\b', '[ADDRESS]', data)
return data
# Register the custom anonymizer
opik.hooks.add_anonymizer(AdvancedPIIAnonymizer())
```
# Usage Examples
## Basic Setup
Here's a complete example showing how to set up anonymization for a simple LLM application:
```python
import opik
import opik.hooks
from opik.anonymizer import create_anonymizer
# Define PII anonymization rules
pii_rules = [
# Email addresses
{"regex": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "replace": "[EMAIL]"},
# Phone numbers (US format)
{"regex": r"\b\d{3}-\d{3}-\d{4}\b", "replace": "[PHONE]"},
# Social Security Numbers
{"regex": r"\b\d{3}-\d{2}-\d{4}\b", "replace": "[SSN]"},
# Credit card numbers
{"regex": r"\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b", "replace": "[CARD]"},
]
# Create and register anonymizer
anonymizer = create_anonymizer(pii_rules)
opik.hooks.add_anonymizer(anonymizer)
# Now all traced functions will automatically anonymize PII
@opik.track
def process_customer_data(customer_info):
"""This function processes customer data with automatic PII anonymization"""
# The input and output will be automatically anonymized
return f"Processed customer: {customer_info}"
# Example usage - PII will be automatically anonymized in traces
result = process_customer_data("John Doe, email: john@example.com, phone: 555-123-4567")
```
## Advanced Configuration
For more sophisticated anonymization scenarios:
```python
import opik
import opik.hooks
from opik.anonymizer import create_anonymizer, Anonymizer
class ComplianceAnonymizer(Anonymizer):
"""Enterprise-grade anonymizer for compliance requirements"""
def __init__(self, compliance_level="standard"):
self.compliance_level = compliance_level
self.sensitive_fields = {
"standard": ["email", "phone", "ssn"],
"strict": ["email", "phone", "ssn", "name", "address", "dob"],
"minimal": ["ssn", "password"]
}
def anonymize(self, data, **kwargs):
field_name = kwargs.get("field_name", "")
if isinstance(data, dict):
# Process dictionary fields
for key, value in list(data.items()):
if key.lower() in self.sensitive_fields[self.compliance_level]:
data[key] = f"[{key.upper()}_REDACTED]"
elif isinstance(data, str):
# Apply string-level anonymization based on the compliance level
if self.compliance_level == "strict":
# More aggressive anonymization
import re
data = re.sub(r'\b[A-Z][a-z]+ [A-Z][a-z]+\b', '[NAME]', data)
data = re.sub(r'\b\d{1,4}\s+\w+\s+\w+\b', '[ADDRESS]', data)
return data
# Set up multi-layer anonymization
opik.hooks.clear_anonymizers() # Clear any existing anonymizers
# Layer 1: Basic PII patterns
basic_rules = [
(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "[EMAIL]"),
(r"\b\d{3}-\d{3}-\d{4}\b", "[PHONE]"),
]
opik.hooks.add_anonymizer(create_anonymizer(basic_rules))
# Layer 2: Compliance-specific anonymization
opik.hooks.add_anonymizer(ComplianceAnonymizer(compliance_level="standard"))
# Layer 3: Custom business logic
def remove_internal_identifiers(text):
"""Remove company-specific internal identifiers"""
import re
return re.sub(r'\bEMP-\d{6}\b', '[EMPLOYEE_ID]', text)
opik.hooks.add_anonymizer(create_anonymizer(remove_internal_identifiers))
```
## Using third-party PII libraries
In addition to regex and custom Python functions, you can reuse existing PII detection / redaction tools such as Microsoft Presidio or cloud APIs (AWS Comprehend, Google Cloud DLP, Azure AI Language).
These tools can be wrapped inside an Opik anonymizer so that **all trace data is pre-redacted** before it’s logged.
You typically integrate third-party tools in one of two ways:
1. **Local open-source libraries** running inside your app or self-hosted Opik deployment
(e.g. Microsoft Presidio, `scrubadub`).
2. **Managed cloud services** called via their SDKs from your anonymizer
(e.g. AWS Comprehend PII, Google Cloud DLP, Azure AI Language PII).
Third-party anonymizers are just custom anonymizers under the hood. You call the
external engine inside anonymize() or a function rule, then return the
redacted data back to Opik.
***
### Example: Microsoft Presidio (open source, runs locally)
First, install Presidio in your environment:
```bash
pip install presidio-analyzer presidio-anonymizer
```
Then create an Anonymizer that delegates to Presidio:
```python
from typing import Any
import opik.hooks
from opik.anonymizer import RecursiveAnonymizer
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig
class PresidioPIIAnonymizer(RecursiveAnonymizer):
"""Use Microsoft Presidio to detect and anonymize PII in text.
This anonymizer is a simple wrapper around Presidio's built-in anonymizer engine.
It extends the RecursiveAnonymizer base class to support nested data structures.
"""
def __init__(self, language: str="en", max_depth: int=10):
super().__init__(max_depth=max_depth)
self.language = language
self.analyzer = AnalyzerEngine()
self.anonymizer = AnonymizerEngine()
def anonymize_text(self, data: str, **kwargs: Any) -> str:
# 1) Detect PII entities in the text
results = self.analyzer.analyze(
text=data,
language=self.language,
entities=None, # detect all supported entities
)
if not results:
return data
# 2) Apply Presidio anonymization
operators = {
"DEFAULT": OperatorConfig("replace", {"new_value": "[PII]"}),
# You can customize per entity type if needed, for example:
# "PHONE_NUMBER": OperatorConfig("mask", {"masking_char": "*", "chars_to_mask": 8}),
}
anon_result = self.anonymizer.anonymize(
text=data,
analyzer_results=results,
operators=operators,
)
return anon_result.text
# Register the Presidio-based anonymizer globally
opik.hooks.add_anonymizer(PresidioPIIAnonymizer())
```
You can combine a Presidio anonymizer with existing regex/function rules by registering multiple anonymizers; they will be applied in sequence.
## Integration with Frameworks
Anonymizers work seamlessly with all Opik integrations:
### OpenAI Integration
```python
import opik
import opik.hooks
from opik.anonymizer import create_anonymizer
from opik.integrations.openai import track_openai
import openai
# Set up anonymization
pii_rules = [
{"regex": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "replace": "[EMAIL]"},
{"regex": r"\b\d{3}-\d{3}-\d{4}\b", "replace": "[PHONE]"},
]
opik.hooks.add_anonymizer(create_anonymizer(pii_rules))
# Enable OpenAI tracking with automatic anonymization
client = track_openai(openai.OpenAI())
# PII in prompts will be automatically anonymized in traces
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{
"role": "user",
"content": "Help me draft an email to john.doe@company.com about his phone number 555-123-4567"
}]
)
```
### LangChain Integration
```python
import opik
import opik.hooks
from opik.anonymizer import create_anonymizer
from opik.integrations.langchain import OpikTracer
from langchain_openai import ChatOpenAI
from langchain.schema import HumanMessage
# Configure anonymization - mix regex and callable function
def mask_credit_cards(text: str) -> str:
"""Partial masking: show first 4 and last 4 digits, mask the middle"""
import re
def partial_mask(match):
card = match.group(0).replace('-', '').replace(' ', '')
if len(card) >= 8:
return card[:4] + '*' * (len(card) - 8) + card[-4:]
return '[CARD]'
return re.sub(r'\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b', partial_mask, text)
anonymizer_rules = [
# Email pattern (regex tuple)
(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "[EMAIL]"),
# Callable function for smart masking
mask_credit_cards,
]
opik.hooks.add_anonymizer(create_anonymizer(anonymizer_rules))
# Set up LangChain with Opik tracing
llm = ChatOpenAI(callbacks=[OpikTracer()])
# All inputs and outputs will be automatically anonymized
messages = [HumanMessage(content="Contact sarah@example.com about card 4532-1234-5678-9010")]
result = llm.invoke(messages)
```
# Configuration Options
## Max Depth
Control how deeply nested data structures are processed:
```python
from opik.anonymizer import create_anonymizer
rules = [{"regex": r"\b\d{3}-\d{3}-\d{4}\b", "replace": "[PHONE]"}]
# Default max_depth is 10
anonymizer = create_anonymizer(rules, max_depth=5)
```
## Multiple Anonymizers
Register multiple anonymizers that will be applied in sequence:
```python
import opik
import opik.hooks
from opik.anonymizer import create_anonymizer
# Clear existing anonymizers
opik.hooks.clear_anonymizers()
# Add multiple anonymizers in order
opik.hooks.add_anonymizer(create_anonymizer([
{"regex": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "replace": "[EMAIL]"}
]))
opik.hooks.add_anonymizer(create_anonymizer([
{"regex": r"\b\d{3}-\d{3}-\d{4}\b", "replace": "[PHONE]"}
]))
# Check if any anonymizers are registered
if opik.hooks.has_anonymizers():
print(f"Active anonymizers: {len(opik.hooks.get_anonymizers())}")
```
# Best Practices
## Rule Ordering
Rules are applied in the order they're defined. More specific patterns should come before general ones:
```python
rules = [
# Specific: Credit cards (more specific pattern first)
{"regex": r"\b4\d{3}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b", "replace": "[VISA_CARD]"},
# General: Any credit card
{"regex": r"\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b", "replace": "[CARD]"},
# General: Any number sequence
{"regex": r"\b\d{4,}\b", "replace": "[NUMBER]"},
]
```
## Performance Considerations
* Use precompiled regex patterns for improved performance on large datasets when implementing custom anonymization functions. Note: Opik's `RegexRule` automatically compiles patterns when the rule is created.
* Keep the number of rules reasonable to avoid performance impacts
* Consider using more specific patterns to reduce false positives
```python
import re
from opik.anonymizer import create_anonymizer
# Pre-compile regex for better performance
EMAIL_PATTERN = re.compile(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b")
def efficient_email_anonymizer(text):
return EMAIL_PATTERN.sub("[EMAIL]", text)
anonymizer = create_anonymizer(efficient_email_anonymizer)
```
## Testing Anonymizers
Always test your anonymization rules to ensure they work correctly:
```python
from opik.anonymizer import create_anonymizer
# Define your rules
rules = [
{"regex": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b", "replace": "[EMAIL]"},
{"regex": r"\b\d{3}-\d{3}-\d{4}\b", "replace": "[PHONE]"},
]
anonymizer = create_anonymizer(rules)
# Test with sample data
test_data = "Contact John at john.doe@company.com or call 555-123-4567"
anonymized = anonymizer.anonymize(test_data)
print(anonymized) # Should output: "Contact John at [EMAIL] or call [PHONE]"
# Test with nested data
test_nested = {
"user": {
"email": "user@example.com",
"phone": "555-987-6543",
"notes": "Called regarding john@company.com"
}
}
anonymized_nested = anonymizer.anonymize(test_nested)
print(anonymized_nested)
```
# Troubleshooting
## Common Issues
**Anonymizer not working:**
* Ensure the anonymizer is registered with `opik.hooks.add_anonymizer()`
* Check that your patterns are correct using a regex tester
* Verify that `opik.flush_tracker()` is called if needed
**Performance issues:**
* Reduce the complexity of regex patterns
* Limit the number of registered anonymizers
* Consider using more specific patterns to reduce processing overhead
**False positives:**
* Make your regex patterns more specific
* Test thoroughly with representative data
* Consider using negative lookbehind/lookahead assertions
# Security Considerations
* **Test thoroughly**: Always test anonymization rules with representative data
* **Regular updates**: Review and update patterns as your application evolves
* **Compliance**: Ensure your anonymization approach meets regulatory requirements
* **Backup strategy**: Consider how to handle cases where anonymization fails
* **Access control**: Limit access to original data and anonymization rules
Remember that anonymization is a one-way process — once data is anonymized in Opik, the original values cannot be recovered. Plan your anonymization strategy accordingly.
# Alerts
> Configure automated webhook notifications to stay informed about events in your Opik workspace, from trace errors to feedback scores and prompt changes.
In Opik 2.0, prompts, datasets, and experiments are project-scoped. Alert events such as prompt changes and feedback scores are now associated with specific projects.
Alerts allow you to configure automated webhook notifications for important events in your Opik workspace. When specific events occur — such as trace errors, new feedback scores, or prompt changes — Opik sends HTTP POST requests to your configured endpoint with detailed event data.
Opik provides three destination types for alerts:
* **Slack**: Native integration with automatic message formatting for Slack
* **PagerDuty**: Native integration with automatic event formatting for PagerDuty
* **General**: For custom webhooks, no-code automation platforms, or middleware services
## Creating an alert
### Prerequisites
* Access to the Opik Configuration page
* A webhook endpoint that can receive HTTP POST requests
* (Optional) An HTTPS endpoint with valid SSL certificate for production use
### Step-by-step guide
1. **Navigate to Alerts**
* Go to Configuration → Alerts tab
* Click "Create new alert" button
2. **Configure basic settings**
* **Name**: Give your alert a descriptive name (e.g., "Production Errors Slack")
* **Enable alert**: Toggle on to activate the alert immediately
3. **Configure webhook settings**
* **Destination**: Select the alert destination type:
* **General**: For custom webhooks, no-code automation platforms, or middleware services
* **Slack**: For native Slack webhook integration (automatically formats messages for Slack)
* **PagerDuty**: For native PagerDuty integration (automatically formats events for PagerDuty)
* **Endpoint URL**: Enter your webhook URL (must start with `http://` or `https://`)
* For Slack: Use your Slack Incoming Webhook URL (e.g., `https://hooks.slack.com/services/...`)
* For PagerDuty: Use your PagerDuty Events API v2 integration URL (e.g., `https://events.pagerduty.com/v2/enqueue`)
* For General: Use any HTTP endpoint that can receive POST requests
4. **Advanced webhook settings** (optional)
* **Secret token**: Add a secret token to verify webhook authenticity (recommended for General destination)
* **Custom headers**: Add HTTP headers for authentication or routing
* Example: `X-Custom-Auth: Bearer your-token-here`
5. **Add triggers**
* Click "Add trigger" to select event types
* Choose one or more event types from the list
* Configure project scope for observability events (optional)
* For threshold-based alerts (errors, cost, latency, feedback scores):
* **Threshold**: Set the threshold value that triggers the alert
* **Operator**: Choose comparison operator (`>`, `<`) for feedback score alerts
* **Window**: Configure the time window in seconds for metric aggregation
* **Feedback Score Name**: Select which feedback score to monitor (for feedback score alerts only)
6. **Test your configuration**
* Click "Test connection" to send a sample webhook
* Verify your endpoint receives the test payload
* Check the response status in the Opik UI
7. **Create the alert**
* Click "Create alert" to save your configuration
* The alert will start monitoring events immediately
## Integration examples
Opik supports three main approaches for integrating alerts with external systems:
1. **Native integrations** (Slack, PagerDuty): Use built-in formatting for popular services - no middleware required
2. **General webhooks**: Send alerts to custom endpoints, no-code platforms, or middleware services
3. **Middleware services** (Optional): Add custom logic, routing, or transformations before forwarding to destinations
### Slack integration (Native)
Opik provides native Slack integration that automatically formats alert messages for Slack's Block Kit format.
#### Prerequisites
* [Create a Slack app and enable Incoming Webhooks](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/)
* Generate a webhook URL (e.g., `https://hooks.slack.com/services/T00000000/B00000000/XXXX`)
#### Setup steps
1. **In Slack**:
* Create a Slack app in your workspace
* Enable Incoming Webhooks
* Add the webhook to your desired channel
* Copy the webhook URL
2. **In Opik**:
* Go to Configuration → Alerts tab
* Click "Create new alert"
* Give your alert a descriptive name
* Select **Slack** as the destination type
* Paste your Slack webhook URL in the Endpoint URL field
* Add triggers for the events you want to monitor
* Click "Test connection" to verify
* Click "Create alert"
Opik will automatically format all alert payloads into Slack-compatible messages with rich formatting, including:
* Alert name and event type
* Event count and details
* Relevant metadata
* Links to view full details in Opik
### PagerDuty integration (Native)
Opik provides native PagerDuty integration that automatically formats alert events for PagerDuty's Events API v2.
#### Prerequisites
* A PagerDuty account with permission to create integrations
* Access to a service where you want to receive alerts
#### Setup steps
1. **In PagerDuty**:
* Navigate to Services → select your service → Integrations tab
* Click "Add Integration"
* Select "Events API V2"
* Give the integration a name (e.g., "Opik Alerts")
* Save the integration and copy the Integration Key
2. **In Opik**:
* Go to Configuration → Alerts tab
* Click "Create new alert"
* Give your alert a descriptive name
* Select **PagerDuty** as the destination type
* Enter the PagerDuty Events API v2 endpoint: `https://events.pagerduty.com/v2/enqueue`
* In the **Routing Key** field, enter your PagerDuty Integration Key (this field appears when PagerDuty is selected as the destination)
* Add triggers for the events you want to monitor
* Click "Test connection" to verify
* Click "Create alert"
Opik will automatically format all alert payloads into PagerDuty-compatible events with:
* Severity levels based on event type
* Detailed event information
* Custom fields for filtering and routing
* Deduplication keys to prevent duplicate incidents
### Custom integration with middleware service (Optional)
For more complex integrations or custom formatting requirements, you can use a middleware service to transform Opik's payload before sending it to your destination. This approach works with any destination type (General, Slack, or PagerDuty).
#### When to use middleware
* **Custom message formatting**: Transform payload structure or add custom fields
* **Multi-destination routing**: Send alerts to different endpoints based on event type
* **Additional processing**: Enrich alerts with data from other systems
* **Legacy systems**: Adapt Opik alerts to older webhook formats
#### Example middleware for Slack with custom formatting
```python
import requests
def transform_to_slack(opik_payload):
event_type = opik_payload.get('eventType')
alert_name = opik_payload['payload']['alertName']
event_count = opik_payload['payload']['eventCount']
# Custom formatting logic
return {
"blocks": [
{
"type": "header",
"text": {
"type": "plain_text",
"text": f"🚨 {alert_name}"
}
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": f"*{event_count}* new `{event_type}` events"
}
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": f"View in Opik: https://www.comet.com/opik"
}
},
{
"type": "section",
"fields": [
{
"type": "mrkdwn",
"text": f"*Environment:*\nProduction"
},
{
"type": "mrkdwn",
"text": f"*Priority:*\nHigh"
}
]
}
]
}
@app.route('/opik-to-slack', methods=['POST'])
def opik_to_slack():
opik_data = request.json
slack_payload = transform_to_slack(opik_data)
# Forward to Slack
requests.post(
SLACK_WEBHOOK_URL,
json=slack_payload
)
return {'status': 'success'}, 200
```
#### Setup for middleware approach
1. Deploy your middleware service to a publicly accessible endpoint
2. In Opik, create an alert with destination type **General**
3. Use your middleware service URL as the Endpoint URL
4. Configure your middleware to forward to the final destination (Slack, PagerDuty, etc.)
### Using no-code automation platforms
No-code automation tools like [n8n](https://n8n.io), [Make.com](https://www.make.com), and [IFTTT](https://ifttt.com) provide an easy way to connect Opik alerts to other services—without writing or deploying code. These platforms can receive webhooks from Opik, apply filters or conditions, and trigger actions such as sending Slack messages, logging data in Google Sheets, or creating incidents in PagerDuty.
**To use them:**
1. **Create a new workflow or scenario** and add a **Webhook trigger** node/module
2. **Copy the webhook URL** generated by the platform
3. **In Opik**, create an alert with destination type **General** and paste the webhook URL from your automation platform
4. **Secure the connection** by validating the Authorization header or including a secret token parameter
5. **Add filters or routing logic** to handle different eventType values from Opik (for example, trace:errors or trace:feedback\_score)
6. **Chain the desired actions**, such as notifications, database updates, or analytics tracking
These tools also provide built-in monitoring, retries, and visual flow editors, making them suitable for both technical and non-technical users who want to automate Opik alert handling securely and efficiently. This approach works well when you need to route alerts to multiple destinations or apply complex business logic.
### Custom dashboard integration
Build a custom monitoring dashboard that receives alerts using the **General** destination type:
```python
from fastapi import FastAPI, Request
from datetime import datetime
app = FastAPI()
# In-memory storage (use a database in production)
alert_history = []
@app.post("/webhook")
async def receive_webhook(request: Request):
data = await request.json()
# Store alert
alert_history.append({
'timestamp': datetime.utcnow(),
'event_type': data.get('eventType'),
'alert_name': data['payload']['alertName'],
'event_count': data['payload']['eventCount'],
'data': data
})
# Keep only last 1000 alerts
if len(alert_history) > 1000:
alert_history.pop(0)
return {"status": "success"}
@app.get("/dashboard")
async def get_dashboard():
# Return aggregated statistics
return {
'total_alerts': len(alert_history),
'by_type': group_by_type(alert_history),
'recent_alerts': alert_history[-10:]
}
```
## Supported event types
Opik supports ten types of alert events:
### Observability events
**Trace errors threshold exceeded**
* **Event type**: `trace:errors`
* **Triggered when**: Total trace error count exceeds the specified threshold within a time window
* **Project scope**: Can be configured to specific projects
* **Configuration**: Requires threshold value (error count) and time window (in seconds)
* **Payload**: Metrics alert payload with error count details
* **Use case**: Proactive error monitoring, detect error spikes, prevent system degradation
**Trace feedback score threshold exceeded**
* **Event type**: `trace:feedback_score`
* **Triggered when**: Average trace feedback score meets the specified threshold criteria within a time window
* **Project scope**: Can be configured to specific projects
* **Configuration**: Requires feedback score name, threshold value, operator (`>`, `<`), and time window
* **Payload**: Metrics alert payload with average feedback score details
* **Use case**: Track model performance, monitor user satisfaction, detect quality degradation
**Thread feedback score threshold exceeded**
* **Event type**: `trace_thread:feedback_score`
* **Triggered when**: Average thread feedback score meets the specified threshold criteria within a time window
* **Project scope**: Can be configured to specific projects
* **Configuration**: Requires feedback score name, threshold value, operator (`>`, `<`), and time window
* **Payload**: Metrics alert payload with average feedback score details
* **Use case**: Monitor conversation quality, track multi-turn interactions, detect thread satisfaction issues
**Guardrails triggered**
* **Event type**: `trace:guardrails_triggered`
* **Triggered when**: A guardrail check fails for a trace
* **Project scope**: Can be configured to specific projects
* **Payload**: Array of guardrail result objects
* **Use case**: Security monitoring, compliance tracking, PII detection
**Cost threshold exceeded**
* **Event type**: `trace:cost`
* **Triggered when**: Total trace cost exceeds the specified threshold within a time window
* **Project scope**: Can be configured to specific projects
* **Configuration**: Requires threshold value (in currency units) and time window (in seconds)
* **Payload**: Metrics alert payload with cost details
* **Use case**: Budget monitoring, cost control, prevent runaway spending
**Latency threshold exceeded**
* **Event type**: `trace:latency`
* **Triggered when**: Average trace latency exceeds the specified threshold within a time window
* **Project scope**: Can be configured to specific projects
* **Configuration**: Requires threshold value (in seconds) and time window (in seconds)
* **Payload**: Metrics alert payload with latency details
* **Use case**: Performance monitoring, SLA compliance, user experience tracking
### Prompt engineering events
**New prompt added**
* **Event type**: `prompt:created`
* **Triggered when**: A new prompt is created in the prompt library
* **Project scope**: Workspace-wide
* **Payload**: Prompt object with metadata
* **Use case**: Track prompt library changes, audit prompt creation
**New prompt version created**
* **Event type**: `prompt:committed`
* **Triggered when**: A new version (commit) is added to a prompt
* **Project scope**: Workspace-wide
* **Payload**: Prompt version object with template and metadata
* **Use case**: Monitor prompt iterations, track version history
**Prompt deleted**
* **Event type**: `prompt:deleted`
* **Triggered when**: A prompt is removed from the prompt library
* **Project scope**: Workspace-wide
* **Payload**: Array of deleted prompt objects
* **Use case**: Audit prompt deletions, maintain prompt governance
### Evaluation events
**Experiment finished**
* **Event type**: `experiment:finished`
* **Triggered when**: An experiment completes in the workspace
* **Project scope**: Workspace-wide
* **Payload**: Array of experiment objects with completion details
* **Use case**: Automate experiment notifications, track evaluation completions
### Want us to support more event types?
If you need additional event types for your use case, please [create an issue on GitHub](https://github.com/comet-ml/opik/issues/new?title=Alert%20Event%20Request%3A%20%3Cevent-name%3E\&labels=enhancement) and let us know what you'd like to monitor.
## Webhook payload structure
All webhook events follow a consistent payload structure:
```json
{
"id": "webhook-event-id",
"eventType": "trace:errors",
"alertId": "alert-uuid",
"alertName": "Production Errors Alert",
"workspaceId": "workspace-uuid",
"createdAt": "2025-01-15T10:30:00Z",
"payload": {
"alertId": "alert-uuid",
"alertName": "Production Errors Alert",
"eventType": "trace:errors",
"eventIds": ["event-id-1", "event-id-2"],
"userNames": ["user@example.com"],
"eventCount": 2,
"aggregationType": "consolidated",
"message": "Alert 'Production Errors Alert': 2 trace:errors events aggregated",
"metadata": [
{
"id": "trace-uuid",
"name": "handle_query",
"project_id": "project-uuid",
"project_name": "Demo Project",
"start_time": "2025-01-15T10:29:45Z",
"end_time": "2025-01-15T10:29:50Z",
"input": {
"query": "User question"
},
"output": {
"response": "LLM response"
},
"error_info": {
"exception_type": "ValidationException",
"message": "Validation failed",
"traceback": "Full traceback..."
},
"metadata": {
"customer_id": "customer_123"
},
"tags": ["production"]
}
]
}
}
```
### Payload fields
| Field | Type | Description |
| ------------------------- | ----------------- | ------------------------------------------ |
| `id` | string | Unique webhook event identifier |
| `eventType` | string | Type of event (e.g., `trace:errors`) |
| `alertId` | string (UUID) | Alert configuration identifier |
| `alertName` | string | Name of the alert |
| `workspaceId` | string | Workspace identifier |
| `createdAt` | string (ISO 8601) | Timestamp when webhook was created |
| `payload.eventIds` | array | List of aggregated event IDs |
| `payload.userNames` | array | Users associated with the events |
| `payload.eventCount` | number | Number of aggregated events |
| `payload.aggregationType` | string | Always "consolidated" |
| `payload.metadata` | array | Event-specific data (varies by event type) |
## Event-specific payloads
### Trace errors threshold exceeded payload
```json
{
"metadata": {
"event_type": "TRACE_ERRORS",
"metric_name": "trace:errors",
"metric_value": "15",
"threshold": "10",
"window_seconds": "900",
"project_ids": "0198ec68-6e06-7253-a20b-d35c9252b9ba,0198ec68-6e06-7253-a20b-d35c9252b9bb",
"project_names": "Demo Project,Default Project"
}
}
```
### Trace feedback score threshold exceeded payload
```json
{
"metadata": {
"event_type": "TRACE_FEEDBACK_SCORE",
"metric_name": "trace:feedback_score",
"metric_value": "0.7500",
"threshold": "0.8000",
"window_seconds": "3600",
"project_ids": "0198ec68-6e06-7253-a20b-d35c9252b9ba,0198ec68-6e06-7253-a20b-d35c9252b9bb",
"project_names": "Demo Project,Default Project"
}
}
```
### Thread feedback score threshold exceeded payload
```json
{
"metadata": {
"event_type": "TRACE_THREAD_FEEDBACK_SCORE",
"metric_name": "trace_thread:feedback_score",
"metric_value": "0.7500",
"threshold": "0.8000",
"window_seconds": "3600",
"project_ids": "0198ec68-6e06-7253-a20b-d35c9252b9ba,0198ec68-6e06-7253-a20b-d35c9252b9bb",
"project_names": "Demo Project,Default Project"
}
}
```
### Prompt created payload
```json
{
"metadata": {
"id": "prompt-uuid",
"name": "Prompt Name",
"description": "Prompt description",
"tags": ["system", "assistant"],
"created_at": "2025-01-15T10:00:00Z",
"created_by": "user@example.com",
"last_updated_at": "2025-01-15T10:00:00Z",
"last_updated_by": "user@example.com"
}
}
```
### Prompt version created payload
```json
{
"metadata": {
"id": "version-uuid",
"prompt_id": "prompt-uuid",
"commit": "abc12345",
"template": "You are a helpful assistant. {{question}}",
"type": "mustache",
"metadata": {
"version": "1.0",
"model": "gpt-4"
},
"created_at": "2025-01-15T10:00:00Z",
"created_by": "user@example.com"
}
}
```
### Prompt deleted payload
```json
{
"metadata": [
{
"id": "prompt-uuid",
"name": "Prompt Name",
"description": "Prompt description",
"tags": ["deprecated"],
"created_at": "2025-01-10T10:00:00Z",
"created_by": "user@example.com",
"last_updated_at": "2025-01-15T10:00:00Z",
"last_updated_by": "user@example.com",
"latest_version": {
"id": "version-uuid",
"commit": "abc12345",
"template": "Template content",
"type": "mustache",
"created_at": "2025-01-15T10:00:00Z",
"created_by": "user@example.com"
}
}
]
}
```
### Guardrails triggered payload
```json
{
"metadata": [
{
"id": "guardrail-check-uuid",
"entity_id": "trace-uuid",
"project_id": "project-uuid",
"project_name": "Project Name",
"name": "PII",
"result": "failed",
"details": {
"detected_entities": ["EMAIL", "PHONE_NUMBER"],
"message": "PII detected in response: email and phone number"
}
}
]
}
```
### Experiment finished payload
```json
{
"metadata": [
{
"id": "experiment-uuid",
"name": "Experiment Name",
"dataset_id": "dataset-uuid",
"created_at": "2025-01-15T10:00:00Z",
"created_by": "user@example.com",
"last_updated_at": "2025-01-15T10:05:00Z",
"last_updated_by": "user@example.com",
"feedback_scores": [
{
"name": "accuracy",
"value": 0.92
},
{
"name": "latency",
"value": 1.5
}
]
}
]
}
```
### Cost threshold exceeded payload
```json
{
"metadata": {
"event_type": "TRACE_COST",
"metric_name": "trace:cost",
"metric_value": "150.75",
"threshold": "100.00",
"window_seconds": "3600",
"project_ids": "0198ec68-6e06-7253-a20b-d35c9252b9ba,0198ec68-6e06-7253-a20b-d35c9252b9bb",
"project_names": "Demo Project,Default Project"
}
}
```
### Latency threshold exceeded payload
```json
{
"metadata": {
"event_type": "TRACE_LATENCY",
"metric_name": "trace:latency",
"metric_value": "5250.5000",
"threshold": "5",
"window_seconds": "1800",
"project_ids": "0198ec68-6e06-7253-a20b-d35c9252b9ba,0198ec68-6e06-7253-a20b-d35c9252b9bb",
"project_names": "Demo Project,Default Project"
}
}
```
## Securing your webhooks
### Using secret tokens
Add a secret token to your webhook configuration to verify that incoming requests are from Opik:
1. Generate a secure random token (e.g., using `openssl rand -hex 32`)
2. Add it to your alert's "Secret token" field
3. Opik will send it in the `Authorization` header: `Authorization: Bearer your-secret-token`
4. Validate the token in your webhook handler before processing the request
### Example validation (Python/Flask)
```python
from flask import Flask, request, abort
import hmac
app = Flask(__name__)
SECRET_TOKEN = "your-secret-token-here"
@app.route('/webhook', methods=['POST'])
def handle_webhook():
# Verify the secret token
auth_header = request.headers.get('Authorization', '')
if not auth_header.startswith('Bearer '):
abort(401, 'Missing or invalid Authorization header')
token = auth_header.split(' ', 1)[1]
if not hmac.compare_digest(token, SECRET_TOKEN):
abort(401, 'Invalid secret token')
# Process the webhook
data = request.json
event_type = data.get('eventType')
# Handle different event types
if event_type == 'trace:errors':
handle_trace_errors(data)
elif event_type == 'trace:feedback_score':
handle_feedback_score(data)
elif event_type == 'experiment:finished':
handle_experiment_finished(data)
return {'status': 'success'}, 200
```
### Using custom headers
You can add custom headers for additional authentication or routing:
```python
# In your webhook handler
api_key = request.headers.get('X-API-Key')
environment = request.headers.get('X-Environment')
if api_key != EXPECTED_API_KEY:
abort(401, 'Invalid API key')
# Route to different handlers based on environment
if environment == 'production':
handle_production_webhook(data)
else:
handle_staging_webhook(data)
```
## Troubleshooting
### Webhooks not being delivered
**Check endpoint accessibility:**
* Ensure your endpoint is publicly accessible (if using cloud)
* Verify firewall rules allow incoming connections
* Test your endpoint with curl: `curl -X POST -H "Content-Type: application/json" -d '{"test": "data"}' https://your-endpoint.com/webhook`
**Check webhook configuration:**
* Verify the URL starts with `http://` or `https://`
* Check that the endpoint returns 2xx status codes
* Review custom headers for syntax errors
**Check alert status:**
* Ensure the alert is enabled
* Verify at least one trigger is configured
* Check that project scope matches your events (for observability events)
### Webhook timeouts
Opik expects webhooks to respond within the configured timeout (typically 30 seconds). If your endpoint takes longer:
**Optimize your handler:**
* Return a 200 response immediately
* Process the webhook asynchronously in the background
* Use a queue system (e.g., Celery, RabbitMQ) for long-running tasks
**Example async processing:**
```python
from flask import Flask
from threading import Thread
app = Flask(__name__)
def process_webhook_async(data):
# Long-running processing
send_to_slack(data)
update_dashboard(data)
log_to_database(data)
@app.route('/webhook', methods=['POST'])
def handle_webhook():
data = request.json
# Start background processing
thread = Thread(target=process_webhook_async, args=(data,))
thread.start()
# Return immediately
return {'status': 'accepted'}, 200
```
### Duplicate webhooks
If you receive duplicate webhooks:
**Check retry configuration:**
* Opik retries failed webhooks with exponential backoff
* Ensure your endpoint returns 2xx status codes on success
* Implement idempotency using the webhook `id` field
**Example idempotent handler:**
```python
processed_webhook_ids = set()
@app.route('/webhook', methods=['POST'])
def handle_webhook():
data = request.json
webhook_id = data.get('id')
# Skip if already processed
if webhook_id in processed_webhook_ids:
return {'status': 'already_processed'}, 200
# Process webhook
process_alert(data)
# Mark as processed
processed_webhook_ids.add(webhook_id)
return {'status': 'success'}, 200
```
### Events not triggering alerts
**Check event type matching:**
* Verify the alert has a trigger for this event type
* For observability events, check project scope configuration
* Review project IDs in trigger configuration
**Check workspace context:**
* Ensure events are logged to the correct workspace
* Verify the alert is in the same workspace as your events
**Check alert evaluation:**
* View backend logs for alert evaluation messages
* Confirm events are being published to the event bus
* Check Redis for alert buckets (self-hosted deployments)
### SSL certificate errors
If you see SSL certificate errors in logs:
**For development/testing:**
* Use self-signed certificates with proper configuration
* Or use HTTP endpoints (not recommended for production)
**For production:**
* Use valid SSL certificates from trusted CAs
* Ensure certificate chain is complete
* Check certificate expiry dates
* Use services like Let's Encrypt for free SSL
## Architecture and internals
Understanding Opik's alert architecture can help with troubleshooting and optimization.
### How alerts work
The Opik Alerts system monitors your workspace for specific events and sends consolidated webhook notifications to your configured endpoints. Here's the flow:
1. **Event occurs**: An event happens in your workspace (e.g., a trace error, prompt creation, guardrail trigger, new feedback score)
2. **Alert evaluation**: The system checks if any enabled alerts match this event type and evaluates threshold conditions (for metrics-based alerts like errors, cost, latency, and feedback scores)
3. **Event aggregation**: Multiple events are aggregated over a short time window (debouncing)
4. **Webhook delivery**: A consolidated HTTP POST request is sent to your webhook URL
5. **Retry handling**: Failed requests are automatically retried with exponential backoff
#### Event debouncing
To prevent overwhelming your webhook endpoint, Opik aggregates multiple events of the same type within a short time window (typically 30-60 seconds) and sends them as a single consolidated webhook. This is particularly useful for high-frequency events like feedback scores.
### Event flow
```
1. Event occurs (e.g., trace error logged)
↓
2. Service publishes AlertEvent to EventBus
↓
3. AlertEventListener receives event
↓
4. AlertEventEvaluationService evaluates against configured alerts
↓
5. Matching events added to AlertBucketService (Redis)
↓
6. AlertJob (runs every 5 seconds) processes ready buckets
↓
7. WebhookPublisher publishes to Redis stream
↓
8. WebhookSubscriber consumes from stream
↓
9. WebhookHttpClient sends HTTP POST request
↓
10. Retries on failure with exponential backoff
```
### Debouncing mechanism
Opik uses Redis-based buckets to aggregate events:
* **Bucket key format**: `alert_bucket:{alertId}:{eventType}`
* **Window size**: Configurable (default 30-60 seconds)
* **Index**: Redis Sorted Set for efficient bucket retrieval
* **TTL**: Buckets expire automatically after processing
This prevents overwhelming your webhook endpoint with individual events and reduces costs for high-frequency events.
### Retry strategy
Failed webhooks are automatically retried:
* **Max retries**: Configurable (default 3)
* **Initial delay**: 1 second
* **Max delay**: 60 seconds
* **Backoff**: Exponential with jitter
* **Retryable errors**: 5xx status codes, network errors
* **Non-retryable errors**: 4xx status codes (except 429)
## Best practices
### Alert design
**Create focused alerts:**
* Use separate alerts for different purposes (e.g., one for errors, one for feedback)
* Configure project scope to avoid noise from test projects
* Use descriptive names that explain the alert's purpose
**Optimize for your workflow:**
* Send critical errors to PagerDuty or on-call systems
* Route feedback scores to analytics platforms
* Send prompt changes to audit logs or Slack channels
**Test thoroughly:**
* Use the "Test connection" feature before enabling alerts
* Monitor webhook delivery in your endpoint logs
* Start with a small project scope and expand gradually
### Webhook endpoint design
**Handle failures gracefully:**
* Return 2xx status codes immediately
* Process webhooks asynchronously
* Implement retry logic in your handler
* Use dead letter queues for permanent failures
**Implement security:**
* Always validate secret tokens
* Use HTTPS endpoints with valid certificates
* Implement rate limiting to prevent abuse
* Log all webhook attempts for auditing
**Monitor performance:**
* Track webhook processing time
* Alert on handler failures
* Monitor queue lengths for async processing
* Set up dead letter queue monitoring
### Scaling considerations
**For high-volume workspaces:**
* Use event debouncing (built-in)
* Implement batch processing in your handler
* Use message queues for async processing
* Consider using serverless functions (AWS Lambda, Cloud Functions)
**For multiple projects:**
* Create project-specific alerts with scope configuration
* Use custom headers to route to different handlers
* Implement filtering in your webhook handler
* Consider separate endpoints for different event types
## Next steps
* Configure your first alert for production error monitoring
* Set up Slack integration for team notifications
* Explore [Online Evaluation Rules](/production/online-evaluation/rules) for automated model monitoring
* Learn about [Guardrails](/production/gateway-guardrails/guardrails) for proactive risk detection
* Review [Production Monitoring](/tracing/dashboards/production_monitoring) best practices
# Overview
Opik Cloud and Enterprise include administration features for teams and organizations, including:
* **Role-based access control**: Assign granular permissions at the organization and workspace level
* **Single sign-on (SSO)**: Authenticate users via SAML or OIDC with your identity provider
* **Workspace isolation**: Separate projects and data across teams with independent access controls
* **Service accounts**: Create API keys for CI/CD pipelines and automated workflows
* **User management**: Invite team members, assign roles, and manage access from a central dashboard
* **JWT authentication**: Integrate Opik into existing systems with token-based auth
Available on [Opik Cloud](https://www.comet.com/site/products/opik/) and Enterprise. [Contact us](https://www.comet.com/site/about-us/contact-us/) for Enterprise pricing.
## Get started
Invite users, create workspaces, and manage organization settings from the admin UI.
Learn how organization roles and workspace roles control what users can access.
Set up SAML or OIDC single sign-on, or configure JWT for programmatic access.
Configure AI providers, feedback definitions, and other workspace-level preferences.
## Key concepts
Opik uses a hierarchical structure to organize users and data:
| Term | Description |
| --------------------- | ----------------------------------------------------------------------------------------------- |
| **Organization** | Your company or team. Contains all users, workspaces, and billing settings. |
| **Workspace** | A container for projects. Users can belong to multiple workspaces with different roles in each. |
| **Project** | A container for traces. Experiments and datasets live at the workspace level. |
| **Organization Role** | Controls organization-wide permissions (e.g., Admin can manage billing and users). |
| **Workspace Role** | Controls what a user can do within a specific workspace (e.g., Editor can create projects). |
Here's how these concepts relate:
```mermaid
flowchart TD
Org[Organization] --> Users[Users + Org Roles]
Org --> Settings[Settings]
Org --> WS[Workspaces]
WS --> WSA[Workspace A]
WS --> WSB[Workspace B]
WSA --> MembersA[Members + Roles]
WSA --> ProjectsA[Projects]
WSB --> MembersB[Members + Roles]
WSB --> ProjectsB[Projects]
```
We recommend creating one workspace per team. This keeps projects organized and allows you to assign different roles to team members based on their responsibilities.
# Roles and Permissions
Opik uses a role-based access control (RBAC) system that allows you to define what users can do within workspaces. This guide explains how roles and permissions work, the default roles available, and how to create custom roles.
## Organization roles vs. workspace roles
Opik has two levels of roles that work together:
| Level | Purpose | Scope | Examples |
| ---------------------- | -------------------------------------------- | ------------------- | ------------------------------- |
| **Organization roles** | Control access to organization-wide features | Entire organization | Admin, Member, View-Only Member |
| **Workspace roles** | Control what users can do within a workspace | Single workspace | Manage, Write, Annotator, Read |
A user's effective access is determined by **both** their organization role and their workspace role:
* Organization role sets the maximum level of access a user can have across all workspaces (e.g., View-Only Members are restricted to read-only access organization-wide).
* Workspace role determines what they can do within each workspace they're a member of, up to the limit set by their organization role.
## Organization roles
Every user in your organization has exactly one organization role:
| Role | Description |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| **Admin** | Full access to the [Admin Dashboard](/administration/admin-dashboard/overview) and all workspaces in the organization. |
| **Member** | Can have full access to any workspace they are added to. No access to the Admin Dashboard. |
| **View-Only Member** | Read-only access to workspaces they are added to. Cannot be granted write permissions in any workspace. |
New users are assigned the **Member** role by default. Organization admins can change a user's role from the [Users](/administration/admin-dashboard/users) page in the Admin Dashboard.
## Workspace roles
Workspace roles control what users can do within a specific workspace. Users can have different roles in different workspaces.
### Default roles
These roles are available in all Opik organizations and serve as the basis for custom roles:
| Role | Description |
| ------------ | ---------------------------------------------------------------- |
| **Manage** | Full admin access. Manage members, settings, and all resources. |
| **Write** | Read-write access. Create projects, log traces, run experiments. |
| **Annotate** | Annotation access. View data and add annotations/feedback. |
| **Read** | Read-only access. View all data but cannot make changes. |
### Permissions by role
The table below shows the default permissions for each role. Custom roles can combine these permissions differently.
| Group | Permission | Manage | Write | Annotate | Read |
| --------------------- | ------------------------------- | ------ | ----- | -------- | ---- |
| Admin | Invite users to workspace | Yes | Yes | No | No |
| Admin | Configure workspace settings | Yes | No | No | No |
| Admin | Define AI providers | Yes | Yes | No | No |
| Experiment management | Change project settings | Yes | No | No | No |
| Experiment management | Approve and manage models | Yes | No | No | No |
| Observability | View logs | Yes | Yes | Yes | Yes |
| Observability | Log trace, span, or thread | Yes | Yes | No | No |
| Observability | Delete trace | Yes | Yes | No | No |
| Observability | Write comments | Yes | Yes | Yes | No |
| Observability | Annotate trace, span, or thread | Yes | Yes | Yes | No |
| Observability | Tag trace | Yes | Yes | Yes | No |
| Observability | Define online evaluation rule | Yes | Yes | No | No |
| Observability | Define alert | Yes | Yes | No | No |
| Observability | Create annotation queue | Yes | Yes | No | No |
| Dashboards | View dashboard | Yes | Yes | No | Yes |
| Experiments | View experiments | Yes | Yes | No | Yes |
| Datasets | View datasets and test suites | Yes | Yes | No | Yes |
| Annotation queues | View annotation queues | Yes | Yes | Yes | Yes |
| Annotation queues | Annotate trace, span, or thread | Yes | Yes | Yes | Yes |
### Custom roles
Custom roles are available on Enterprise plans. [Reach out](https://www.comet.com/site/about-us/contact-us/) to enable this feature.
To create a custom role:
1. Go to **Admin Dashboard** > **Roles & Permissions**.
2. Click **Create Role** and configure permissions.
3. Inherit from an existing role as a starting point.
## Assigning roles
Roles can be updated in different places depending on the role type:
* **Organization roles** can be updated by organization admins in **Admin Dashboard** > **[Users](/administration/admin-dashboard/users)**.
* **Workspace roles** can be updated in **Configuration** > **[Members](/administration/workspace-settings/workspace_members)** by any user with the Manage workspace role.
## Next steps
* [Configure authentication](/administration/authentication/overview) with role mapping.
* [Manage users](/administration/admin-dashboard/users) and assign roles.
* [Create service accounts](/administration/admin-dashboard/service_accounts) with appropriate workspace access.
# Overview
Workspace settings allow you to configure options that apply to your entire workspace. These settings are managed separately from organization-level administration features.
## Accessing Workspace Settings
1. Navigate to your workspace in Opik
2. Click on **Configuration** in the left sidebar
3. Select the appropriate tab for the setting you want to configure
## Who can access Workspace Settings?
* **Workspace owners** can access all settings including member management and preferences
* **Workspace members** with appropriate roles can access feedback definitions and AI providers
Workspace Members is available on Opik Cloud and Enterprise plans. This feature is not available in open-source deployments. [Reach out](https://www.comet.com/site/about-us/contact-us/) if you want to enable this feature for your Opik deployment.
## Available Settings
Create custom labels for evaluating LLM outputs with structured feedback types.
Configure connections to LLM providers for the Playground and Online Evaluation.
Configure workspace-wide preferences like thread timeout and data truncation.
Manage workspace members and their access (Cloud/Enterprise only).
## Next steps
* To create custom feedback types, see [Feedback Definitions](/administration/workspace-settings/feedback_definitions)
* To connect LLM providers, see [AI Providers](/administration/workspace-settings/ai_providers)
* To manage workspace members (Cloud/Enterprise), see [Workspace Members](/administration/workspace-settings/workspace_members)
# Feedback Definitions
> Configure custom feedback types for evaluating LLM outputs
Feedback definitions allow you to create custom labels to systematically collect and analyze structured feedback on your LLM outputs.
## Using Feedback Definitions
Once created, feedback definitions appear in two places:
1. **Trace and thread sidebars** — Add scores when reviewing individual items.
2. **Annotation queues** — Pick which definitions annotators see when you create a queue.
You can also log feedback scores programmatically using the SDK. See the [SDK documentation](/reference/overview) for details.
## Creating a Feedback Definition
1. Click **Create new feedback definition**
2. Enter a **Name** for your feedback definition
3. Select the **Type**: Categorical or Numerical
4. Define the **Values** (for Categorical) or **Range** (for Numerical)
## Common Feedback Types
| Feedback Type | Type | Values |
| ---------------- | ----------- | --------------------------- |
| Thumbs Up / Down | Categorical | thumbs up, thumbs down |
| Usefulness | Categorical | Useful, Neutral, Not useful |
| Hallucination | Categorical | Yes, No |
| Correct | Categorical | Good, Bad |
# AI Providers
> Configure connections to Large Language Model providers
AI Providers let you connect LLMs for use in the Playground and Online Evaluation.
## Using AI Providers
Once configured, providers appear in two places:
1. **Playground** — Test prompts interactively with different models.
2. **Online Evaluation** — Run LLM-as-a-judge scoring on your traces.
## Adding a Provider
1. Click the **Add configuration** button in the top-right corner
2. In the Provider Configuration dialog that appears:
* Select a provider from the dropdown menu
* Enter your API key for that provider
* Click **Save** to store the configuration
### Supported Providers
Opik supports integration with various AI providers, including:
* OpenAI
* Anthropic
* OpenRouter
* Gemini
* VertexAI
* Azure OpenAI
* Amazon Bedrock
* Ollama (local or self-hosted, OpenAI-compatible)
* vLLM / any other OpenAI API-compliant provider
If you would like us to support additional LLM providers, please let us know by opening an issue on
[GitHub](https://github.com/comet-ml/opik/issues).
### Provider-Specific Setup
Below are instructions for obtaining API keys and other required information for each supported provider:
#### OpenAI
1. Create or log in to your [OpenAI account](https://platform.openai.com/)
2. Navigate to the [API keys page](https://platform.openai.com/api-keys)
3. Click "Create new secret key"
4. Copy your API key (it will only be shown once)
5. In Opik, select "OpenAI" as the provider and paste your key
#### Anthropic
1. Sign up for or log in to [Anthropic's platform](https://console.anthropic.com/)
2. Navigate to the [API Keys page](https://console.anthropic.com/settings/keys)
3. Click "Create Key" and select the appropriate access level
4. Copy your API key (it will only be shown once)
5. In Opik, select "Anthropic" as the provider and paste your key
#### OpenRouter
1. Create or log in to your [OpenRouter account](https://openrouter.ai/)
2. Navigate to the [API Keys page](https://openrouter.ai/settings/keys)
3. Create a new API key
4. Copy your API key
5. In Opik, select "OpenRouter" as the provider and paste your key
#### Gemini
1. Signup or login to [Google AI Studio](https://aistudio.google.com/)
2. Go to the [API keys page](https://aistudio.google.com/apikey)\\
3. Create a new API key for one your existing Google Cloud project
4. Copy your API key (it will only be shown once)
5. In Opik, select "Gemini" as the provider and paste your key
#### Azure OpenAI
Azure OpenAI provides enterprise-grade access to OpenAI models through Microsoft Azure. To use Azure OpenAI with Opik:
1. Get your Azure OpenAI endpoint URL
* Go to [portal.azure.com](https://portal.azure.com)
* Navigate to your Azure OpenAI resource
* Copy your endpoint URL (it looks like `https://your-company.openai.azure.com`)
2. Construct the complete API URL
* Add `/openai/v1` to the end of your endpoint URL
* Your complete URL should look like: `https://your-company.openai.azure.com/openai/v1`
3. Configure in Opik
* In Opik, go to **Workspace Settings > AI Providers**
* Click **"Add Configuration"**
* Select **"vLLM / Custom provider"** from the dropdown
* Enter your complete URL in the URL field: `https://your-company.openai.azure.com/openai/v1`
* Add your Azure OpenAI API key in the API Key field
* In the Models section, list the models you have deployed in Azure (e.g., `gpt-4o`)
* Click **Save** to store the configuration
Once saved, you can use your Azure OpenAI models directly from Online Scores and the Playground.
#### Vertex AI
##### Option A: Setup via `gcloud` CLI
1. **Create a Custom IAM Role**
```bash
gcloud iam roles create opik \
--project=
##### Configuration Steps
1. **Provider Name**: Enter a unique name to identify this custom provider (e.g., "vLLM Production", "Ollama Local", "Azure OpenAI Dev")
2. **URL**: Enter your server URL, for example: `http://host.docker.internal:8000/v1`
3. **API Key** (optional): If your model access requires authentication, enter the API key. Otherwise, leave this field blank.
4. **Models**: List all models available on your server. You'll be able to select one of them for use later.
5. **Custom Headers** (optional): Add any additional HTTP headers required by your custom endpoint as key-value pairs.
If you're running Opik locally, you would need to use `http://host.docker.internal:
## Dashboard sections
The Admin Dashboard provides access to the following sections:
### Workspaces
Manage the workspaces within your organization:
* **View all workspaces**: See a list of all workspaces in your organization.
* **Create workspaces**: Add new workspaces for different teams or projects.
* **Delete workspaces**: Remove workspaces that are no longer needed.
Learn more in [Workspaces](/administration/admin-dashboard/workspaces).
### Users
Manage user access to your organization:
* **View members**: See all users in your organization and their roles.
* **View pending invitations**: See pending invitations to your organization.
* **Remove users**: Revoke access for users who should no longer be in the organization.
* **Change roles**: Update user roles at the organization level.
Learn more in [Users](/administration/admin-dashboard/users).
### Roles & Permissions
Configure workspace-level access control:
* **View workspace roles**: See the available roles and their permissions.
* **Create custom roles**: Define custom roles tailored to your organization's needs.
* **Manage permissions**: Understand and configure the permission hierarchy.
Learn more in [Roles and Permissions](/administration/roles_and_permissions).
### Authentication
Set up single sign-on authentication for your organization:
* **SAML configuration**: Configure SAML-based SSO with your identity provider.
* **OIDC configuration**: Set up OpenID Connect authentication.
* **JWT authentication**: Configure JWT token-based authentication.
Learn more in [Authentication Overview](/administration/authentication/overview).
### Service Accounts
Manage programmatic access to Opik:
* **Create service accounts**: Set up accounts for automated systems and CI/CD pipelines.
* **Manage API keys**: Generate, regenerate, and revoke API keys.
* **Configure workspace access**: Define which workspaces each service account can access.
Learn more in [Service Accounts](/administration/admin-dashboard/service_accounts).
### Billing
Billing management is available on Opik Cloud and Enterprise plans. This feature is not available in open-source deployments. [Reach out](https://www.comet.com/site/about-us/contact-us/) for billing inquiries.
Manage your organization's subscription and billing:
* **View current plan**: See your organization's subscription tier and usage.
* **Upgrade plan**: Move to a higher tier for additional features or capacity.
* **Manage payment methods**: Update credit card or billing information.
* **View invoices**: Access historical invoices and payment records.
## Next steps
* [Manage workspaces](/administration/admin-dashboard/workspaces) in your organization.
* [Invite users](/administration/admin-dashboard/users) and assign roles.
* [Configure SSO](/administration/authentication/overview) for your organization.
# Workspaces
Workspaces group related projects together within your organization. Each workspace has its own members, roles, and isolated data (traces, experiments, datasets).
## Viewing Workspaces
To view all workspaces in your organization:
1. Navigate to the [Admin Dashboard](/administration/admin-dashboard/overview)
2. Click on **Workspaces** in the sidebar
The list shows each workspace's name, member count, and creation date.
## Creating a Workspace
1. Navigate to the **Workspaces** section in the Admin Dashboard
2. Click **Create Workspace**
3. Enter a **name** for the workspace
4. Click **Create**
Use clear, descriptive names. Examples: `production`, `staging`, `research-team`.
## Deleting a Workspace
Deleting a workspace permanently removes all data within it, including projects, traces, experiments, and datasets. This action cannot be undone.
To delete a workspace:
1. Navigate to the **Workspaces** section in the Admin Dashboard
2. Find the workspace you want to delete
3. Click the **more options** menu (three dots)
4. Select **Delete**
5. Confirm the deletion by typing the workspace name
You cannot delete the only workspace in the organization.
## Related Documentation
* [Workspace Members](/administration/workspace-settings/workspace_members) - Manage users within a workspace
* [Users](/administration/admin-dashboard/users) - Organization-level user management
* [Roles and Permissions](/administration/roles_and_permissions) - Workspace role configuration
# Users
The Users section in the Admin Dashboard allows organization administrators to view all users, change organization roles, and remove users from the organization.
## Viewing Users
To view all users in your organization:
1. Navigate to the [Admin Dashboard](/administration/admin-dashboard/overview)
2. Click on **Users** in the sidebar
The list shows each user's name, email, organization role, and when they joined.
## Changing Organization Roles
Organization roles determine high-level access across the organization. See [Roles and Permissions](/administration/roles_and_permissions) for details on available roles.
To change a user's organization role:
1. Navigate to the **Users** section in the Admin Dashboard
2. Find the user whose role you want to change
3. Click on the role dropdown
4. Select the new role
You cannot change your own organization role. Another admin must make changes to your role. There must always be at least one admin in the organization.
## Removing Users
To remove a user from your organization:
1. Navigate to the **Users** section in the Admin Dashboard
2. Find the user you want to remove
3. Click the **more options** menu (three dots)
4. Select **Remove from organization**
5. Confirm the removal
Removing a user revokes all their access immediately. Their historical data (traces, experiments) is retained. You cannot remove yourself or the last admin from the organization.
## Related Documentation
* [Roles and Permissions](/administration/roles_and_permissions) - Organization and workspace roles
* [Workspace Members](/administration/workspace-settings/workspace_members) - Manage users within a workspace
* [Workspaces](/administration/admin-dashboard/workspaces) - Workspace management
# Organization Settings
Organization Settings is available on Opik Cloud and Enterprise plans. This feature is not available in open-source deployments. [Reach out](https://www.comet.com/site/about-us/contact-us/) if you want to enable this feature for your Opik deployment.
Organization settings allow administrators to configure settings that apply across your entire organization.
## Accessing Organization Settings
1. Navigate to the [Admin Dashboard](/administration/admin-dashboard/overview)
2. Click on **General Settings** in the sidebar
## Restrict User Invitations to Admins
By default, any member in your organization can invite new users, which may affect billing. Enable this setting to restrict invitations to organization admins only.
When enabled:
* Only organization admins can invite users to the organization
* Regular members cannot send invitations
When disabled:
* All members can invite users to the organization
## Related Documentation
* [Admin Dashboard Overview](/administration/admin-dashboard/overview) - Navigate the admin dashboard
* [Users](/administration/admin-dashboard/users) - Manage organization users
* [Roles and Permissions](/administration/roles_and_permissions) - Organization and workspace roles
# Service Accounts
Service accounts provide programmatic access to Opik for automated systems, CI/CD pipelines, and integrations. Unlike user accounts, service accounts use API keys for authentication and don't require interactive login.
Service Accounts is available on Opik Cloud and Enterprise plans. This feature is not available in open-source deployments. [Reach out](https://www.comet.com/site/about-us/contact-us/) if you want to enable this feature for your Opik deployment.
## Creating a Service Account
1. Navigate to the [Admin Dashboard](/administration/admin-dashboard/overview)
2. Click on **Service Accounts** in the sidebar
3. Click **Create service account**
4. Enter a **name** for the service account
5. Optionally select a **default workspace**
6. Optionally select **authorized workspaces** to restrict access
7. Click **Create**
Use descriptive names that indicate the service account's purpose, such as `ci-pipeline-prod` or `monitoring-service`.
## Managing API Keys
Each service account can have multiple API keys. To manage keys:
1. Navigate to the **Service Accounts** section in the Admin Dashboard
2. Find the service account and click **Manage API keys**
From the API keys modal you can:
* **Add key**: Generate a new API key
* **Copy key**: Copy an existing key to clipboard
* **Delete key**: Remove a key (takes effect immediately)
API keys are only shown once when created. Copy and store them securely immediately. If you lose a key, you'll need to generate a new one.
## Deleting a Service Account
Deleting a service account immediately revokes all its API keys. Any systems using those keys will stop working.
1. Navigate to the **Service Accounts** section in the Admin Dashboard
2. Find the service account you want to delete
3. Click the **delete** action
4. Confirm the deletion
## Using Service Account API Keys
Configure the Opik SDK with a service account API key:
```python
import opik
# Using environment variables (recommended)
# export OPIK_API_KEY=your-service-account-api-key
# export OPIK_WORKSPACE=your-workspace
client = opik.Opik()
# Or configure directly
client = opik.Opik(
api_key="your-service-account-api-key",
workspace="your-workspace"
)
```
## Related Documentation
* [Roles and Permissions](/administration/roles_and_permissions) - Workspace role configuration
* [Workspaces](/administration/admin-dashboard/workspaces) - Workspace management
# Authentication Overview
Opik supports multiple authentication methods to integrate with your organization's identity management infrastructure. This guide helps you understand the available options and choose the right approach for your needs.
Authentication features are available on Enterprise plans. These features are not available in open-source deployments. [Reach out](https://www.comet.com/site/about-us/contact-us/) if you want to enable SSO or JWT authentication for your Opik deployment.
## Authentication methods
Opik supports multiple authentication methods for enterprise organizations. For configurable UI access, you can set up SAML SSO or OIDC SSO to integrate with your identity provider. Other available methods include base authentication (username/password), Google OAuth, GitHub OAuth, and LDAP (for on-premises deployments).
JWT Authentication is available separately for SDK and programmatic access. Unlike SAML SSO and OIDC SSO, JWT Authentication is designed for service-to-service and API integrations, not for user interface login.
| Method | Best for | Key features |
| ---------------------- | --------------------------------------------------------- | ----------------------------------------------------------- |
| **SAML SSO** | Organizations with enterprise IdPs (Okta, Azure AD, etc.) | Workspace sync, attribute mapping, broad IdP support |
| **OIDC SSO** | Organizations using OAuth 2.0 / OpenID Connect | Simpler setup, token-based, modern protocol |
| **JWT Authentication** | Programmatic access, custom auth flows | Flexible integration, JWKS support, service-to-service auth |
## Choosing an authentication method
Use this decision guide to select the right authentication method:
### Use SAML SSO when:
* Your organization already uses an enterprise identity provider (Okta, Azure AD, OneLogin, etc.)
* You need automatic workspace assignment based on user attributes/groups
* You require centralized user lifecycle management (auto-provisioning/deprovisioning)
* Your security policies mandate SAML-based authentication
### Use OIDC SSO when:
* Your identity provider supports OpenID Connect but not SAML
* You prefer a simpler, more modern protocol
* You're using a cloud-native identity solution
* You don't need attribute-based workspace sync
### Use JWT Authentication when:
* You need programmatic/API access from backend services
* You're building custom authentication flows
* You want to integrate with existing JWT-based systems
* You need service-to-service authentication
**Multiple methods**: You can configure multiple authentication methods for your organization. For example, use SAML for human users and JWT for service accounts.
## Common prerequisites
Before configuring any authentication method, ensure you have:
1. **Admin access**: You must be an organization administrator.
2. **Enterprise plan**: SSO features require an Enterprise subscription.
3. **Domain ownership**: You should control the email domain(s) you want to use for SSO.
4. **IdP access** (for SSO): Admin access to your identity provider to configure the integration.
## Glossary
Understanding these terms will help you configure authentication:
### General terms
| Term | Description |
| --------------------------- | ------------------------------------------------------------------------------------ |
| **IdP (Identity Provider)** | The system that authenticates users (e.g., Okta, Azure AD, Google Workspace) |
| **SP (Service Provider)** | The application users are logging into (Opik) |
| **SSO (Single Sign-On)** | Authentication method allowing users to log in once and access multiple applications |
| **Domain** | Your organization's email domain (e.g., `company.com`) used to route users to SSO |
### SAML-specific terms
| Term | Description |
| ---------------------------------------- | ------------------------------------------------------------------------------ |
| **Entity ID** | Unique identifier for the IdP or SP in a SAML configuration |
| **ACS URL (Assertion Consumer Service)** | URL where the IdP sends authentication responses |
| **IdP SSO URL** | URL where users are redirected to authenticate |
| **X.509 Certificate** | Public certificate used to verify SAML assertions |
| **Attribute Mapping** | Configuration that maps IdP user attributes to Opik fields |
| **Workspace Sync** | Feature that automatically assigns users to workspaces based on IdP attributes |
### OIDC-specific terms
| Term | Description |
| --------------------- | ----------------------------------------------------------------------- |
| **Client ID** | Unique identifier for Opik in your IdP |
| **Client Secret** | Secret key used to authenticate Opik with your IdP |
| **Authorization URL** | URL where users are redirected to authenticate |
| **Token URL** | URL where Opik exchanges authorization codes for tokens |
| **Callback URL** | URL where the IdP redirects users after authentication |
| **Scope** | Permissions requested from the IdP (e.g., `openid`, `profile`, `email`) |
### JWT-specific terms
| Term | Description |
| --------------------------- | ------------------------------------------------------------------------- |
| **JWKS (JSON Web Key Set)** | Endpoint providing public keys for JWT verification |
| **JWKS URI** | URL of the JWKS endpoint |
| **Static Public Key** | Alternative to JWKS; a fixed public key for verification (on-prem only) |
| **Issuer** | The entity that issued the JWT token |
| **Audience** | The intended recipient of the JWT token |
| **Subject** | The user or entity the token represents |
| **Subject Mapping** | How Opik identifies users from JWT claims (`EMAIL` or `USER_NAME`) |
| **Subject Claim Name** | The JWT claim containing the subject (defaults to `sub`) |
| **kid (Key ID)** | Identifier in the JWT header specifying which key to use for verification |
## Authentication flow comparison
### SAML authentication flow
```
┌──────┐ ┌──────┐ ┌──────┐
│ User │ │ Opik │ │ IdP │
└──┬───┘ └──┬───┘ └──┬───┘
│ 1. Login │ │
│────────────>│ │
│ │ 2. Redirect │
│ │────────────>│
│ │ │ 3. User authenticates
│ │<────────────│
│ │ 4. SAML │
│ │ Assertion │
│<────────────│ │
│ 5. Logged in│ │
└─────────────┘ │
```
### OIDC authentication flow
```
┌──────┐ ┌──────┐ ┌──────┐
│ User │ │ Opik │ │ IdP │
└──┬───┘ └──┬───┘ └──┬───┘
│ 1. Login │ │
│────────────>│ │
│ │ 2. Redirect │
│────────────────────────-->│
│ │ │ 3. User authenticates
│<──────────────────────────│
│ 4. Auth code│ │
│────────────>│ │
│ │ 5. Exchange │
│ │ for token│
│ │────────────>│
│ │<────────────│
│ │ 6. Token │
│<────────────│ │
│ 7. Logged in│ │
```
### JWT authentication flow
```
┌──────────┐ ┌──────┐ ┌──────┐
│ Service/ │ │ Opik │ │ JWKS │
│ User │ │ │ │ │
└────┬─────┘ └──┬───┘ └──┬───┘
│ 1. API call │ │
│ with JWT │ │
│──────────────>│ │
│ │ 2. Fetch │
│ │ keys │
│ │────────────>│
│ │<────────────│
│ │ 3. Verify │
│ │ JWT │
│<──────────────│ │
│ 4. Response │ │
```
## Configuration guides
Detailed setup instructions are available for each authentication method:
Configure SAML-based single sign-on with enterprise identity providers.
Set up OpenID Connect authentication for your organization.
Configure JWT-based authentication for programmatic access.
## Security considerations
### Domain verification
When configuring SSO, you associate email domains with your organization. This ensures:
* Users with those email domains are directed to your SSO configuration.
* Only users who authenticate through your IdP can access the organization.
**Important**: Ensure you only configure domains you own or control. Misconfigured domains could prevent legitimate users from accessing their accounts.
### Certificate management
For SAML authentication:
* Store IdP certificates securely.
* Monitor certificate expiration dates.
* Plan for certificate rotation to avoid authentication disruptions.
### Key rotation
For JWT authentication:
* Use JWKS endpoints when possible for automatic key rotation.
* If using static keys (on-prem only), establish a key rotation schedule.
* Monitor JWKS endpoint availability.
## Troubleshooting
Common authentication issues and solutions:
| Issue | Possible causes | Solution |
| -------------------------------- | ----------------------------------------- | ----------------------------------------------- |
| User can't log in via SSO | Domain not configured, IdP misconfigured | Verify domain settings, check IdP configuration |
| User lands in wrong organization | Multiple SSO configs for same domain | Review domain-to-organization mappings |
| Workspace sync not working | Attribute mapping incorrect | Verify IdP sends expected attributes |
| JWT validation fails | Key mismatch, expired token, wrong issuer | Check JWKS endpoint, verify token claims |
| Certificate errors | Expired or wrong certificate | Update certificate in SSO configuration |
## Next steps
* [Configure SAML SSO](/administration/authentication/saml) for enterprise identity providers.
* [Set up OIDC](/administration/authentication/oidc) for OpenID Connect authentication.
* [Configure JWT authentication](/administration/authentication/jwt) for programmatic access.
# SAML SSO
SAML (Security Assertion Markup Language) SSO allows your users to authenticate using your organization's identity provider (IdP). This guide walks you through configuring SAML SSO for your Opik organization.
SAML SSO is available on Enterprise plans. This feature is not available in open-source deployments. [Reach out](https://www.comet.com/site/about-us/contact-us/) if you want to enable this feature for your Opik deployment.
## Prerequisites
Before you begin, ensure you have:
* **Organization admin access** to Opik
* **Admin access** to your identity provider (Okta, Azure AD, OneLogin, etc.)
* **Enterprise plan** enabled for your organization
* **Email domain** you want to use for SSO (e.g., `company.com`)
## Configuration overview
Setting up SAML SSO involves two main steps:
1. **Configure your IdP**: Add Opik as a SAML application in your identity provider.
2. **Configure Opik**: Enter your IdP's SAML settings in Opik's admin dashboard.
## Step 1: Choose Opik's Service Provider (SP) URLs
Choose matching Service Provider (SP) URLs and enter them in both Opik and your identity provider (IdP).
1. In Opik, navigate to **Admin Dashboard** > **Organization** > **Authentication**.
2. Toggle **Enable SSO Authentication** on. The SP and IdP form fields appear only once SSO is enabled.
3. Set both the **SP Entity ID** and the **SP ACS URL** to the same URL, in the format `https://
## Getting started
To use the BeeAI integration with Opik, you will need to have BeeAI and the required OpenTelemetry packages installed.
### Installation
#### Option 1: Using npm
```bash
npm install beeai-framework@0.1.13 @ai-sdk/openai @arizeai/openinference-instrumentation-beeai @opentelemetry/sdk-node dotenv
```
#### Option 2: Using yarn
```bash
yarn add beeai-framework@0.1.13 @ai-sdk/openai @arizeai/openinference-instrumentation-beeai @opentelemetry/sdk-node dotenv
```
**Version Compatibility**: The BeeAI instrumentation currently supports `beeai-framework` version 0.1.13. Using a newer version may cause compatibility issues.
### Requirements
* Node.js ≥ 18
* BeeAI Framework (`beeai-framework`)
* OpenInference Instrumentation for BeeAI (`@arizeai/openinference-instrumentation-beeai`)
* OpenTelemetry SDK for Node.js (`@opentelemetry/sdk-node`)
## Environment configuration
Configure your environment variables based on your Opik deployment:
Intent:
Route BeeAI OpenInference spans to Opik through OTLP. No breaking changes: existing BeeAI setups continue to work; these settings are only required to send traces to Opik.
Applies when:
You run BeeAI (TypeScript) with `@arizeai/openinference-instrumentation-beeai`.
Required fields:
* `OTEL_EXPORTER_OTLP_ENDPOINT`
* `OTEL_EXPORTER_OTLP_HEADERS` (`Authorization`, `Comet-Workspace` for Cloud/Enterprise)
Optional fields:
* `projectName` in `OTEL_EXPORTER_OTLP_HEADERS` (recommended)
* additional OpenTelemetry env vars for sampling/resource labels
Minimal valid input:
* endpoint for your deployment mode
* matching headers for auth and routing
```bash wordWrap
# Your LLM API key
export OPENAI_API_KEY="your-openai-api-key"
# Opik configuration
export OTEL_EXPORTER_OTLP_ENDPOINT=https://www.comet.com/opik/api/v1/private/otel
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=
## Getting started
### Create a Mastra project
If you don't have a Mastra project yet, you can create one using the Mastra CLI:
```bash
npx create-mastra
cd your-mastra-project
```
### Install required packages
Install the necessary dependencies for Mastra observability:
```bash
npm install @mastra/observability @mastra/otel
```
### Add environment variables
Create or update your `.env` file with the following variables:
```bash wordWrap
# Your LLM API key
OPENAI_API_KEY=
## Getting started
To use the AG2 integration with Opik, you will need to have the following
packages installed:
```bash
pip install -U "ag2[openai]" opik opentelemetry-sdk opentelemetry-instrumentation-openai opentelemetry-instrumentation-threading opentelemetry-exporter-otlp
```
In addition, you will need to set the following environment variables to
configure the OpenTelemetry integration:
If you are using Opik Cloud, you will need to set the following
environment variables:
```bash wordWrap
export OTEL_EXPORTER_OTLP_ENDPOINT=https://www.comet.com/opik/api/v1/private/otel
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=
## Getting started
To use the Agno integration with Opik, you will need to have the following
packages installed:
```bash
pip install -U agno openai opentelemetry-sdk opentelemetry-exporter-otlp openinference-instrumentation-agno yfinance
```
In addition, you will need to set the following environment variables to
configure the OpenTelemetry integration:
If you are using Opik Cloud, you will need to set the following
environment variables:
```bash wordWrap
export OTEL_EXPORTER_OTLP_ENDPOINT=https://www.comet.com/opik/api/v1/private/otel
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=
## Getting started
To use the BeeAI integration with Opik, you will need to have BeeAI and the required OpenTelemetry packages installed:
```bash
pip install beeai-framework openinference-instrumentation-beeai "beeai-framework[wikipedia]" opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
```
## Environment configuration
Configure your environment variables based on your Opik deployment:
If you are using Opik Cloud, you will need to set the following
environment variables:
```bash wordWrap
export OTEL_EXPORTER_OTLP_ENDPOINT=https://www.comet.com/opik/api/v1/private/otel
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=
## Getting started
To use the Autogen integration with Opik, you will need to have the following
packages installed:
```bash
pip install -U "autogen-agentchat" "autogen-ext[openai]" opik opentelemetry-sdk opentelemetry-instrumentation-openai opentelemetry-exporter-otlp
```
In addition, you will need to set the following environment variables to
configure the OpenTelemetry integration:
If you are using Opik Cloud, you will need to set the following
environment variables:
```bash wordWrap
export OTEL_EXPORTER_OTLP_ENDPOINT=https://www.comet.com/opik/api/v1/private/otel
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=
## Getting Started
### Installation
First, ensure you have both `opik` and `crewai` installed:
```bash
pip install opik crewai crewai-tools
```
### Configuring Opik
Configure the Opik Python SDK for your deployment type. See the [Python SDK Configuration guide](/tracing/advanced/sdk_configuration) for detailed instructions on:
* **CLI configuration**: `opik configure`
* **Code configuration**: `opik.configure()`
* **Self-hosted vs Cloud vs Enterprise** setup
* **Configuration files** and environment variables
### Configuring CrewAI
In order to configure CrewAI, you will need to have your LLM provider API key. For this example, we'll use OpenAI. You can [find or create your OpenAI API Key in this page](https://platform.openai.com/settings/organization/api-keys).
You can set it as an environment variable:
```bash
export OPENAI_API_KEY="YOUR_API_KEY"
```
Or set it programmatically:
```python
import os
import getpass
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
```
## Logging CrewAI calls
To log a CrewAI pipeline run, you can use the [`track_crewai`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/crewai/track_crewai.html) function. This will log each CrewAI call to Opik, including LLM calls made by your agents.
**CrewAI v1.0.0+ requires the `crew` parameter**: To ensure LLM calls are properly logged in CrewAI v1.0.0 and later, you must pass your Crew instance to `track_crewai(crew=your_crew)`. This is required because CrewAI v1.0.0+ changed how LLM providers are handled internally.
For CrewAI v0.x, the `crew` parameter is optional as LLM tracking works through LiteLLM delegation.
### Creating a CrewAI Project
The first step is to create our project. We will use an example from CrewAI's documentation:
```python
from crewai import Agent, Crew, Task, Process
class YourCrewName:
def agent_one(self) -> Agent:
return Agent(
role="Data Analyst",
goal="Analyze data trends in the market",
backstory="An experienced data analyst with a background in economics",
verbose=True,
)
def agent_two(self) -> Agent:
return Agent(
role="Market Researcher",
goal="Gather information on market dynamics",
backstory="A diligent researcher with a keen eye for detail",
verbose=True,
)
def task_one(self) -> Task:
return Task(
name="Collect Data Task",
description="Collect recent market data and identify trends.",
expected_output="A report summarizing key trends in the market.",
agent=self.agent_one(),
)
def task_two(self) -> Task:
return Task(
name="Market Research Task",
description="Research factors affecting market dynamics.",
expected_output="An analysis of factors influencing the market.",
agent=self.agent_two(),
)
def crew(self) -> Crew:
return Crew(
agents=[self.agent_one(), self.agent_two()],
tasks=[self.task_one(), self.task_two()],
process=Process.sequential,
verbose=True,
)
```
### Running with Opik Tracking
Now we can import Opik's tracker and run our `crew`. **For CrewAI v1.0.0+, pass the crew instance to `track_crewai`** to ensure LLM calls are logged:
```python
from opik.integrations.crewai import track_crewai
# Create the crew
my_crew = YourCrewName().crew()
track_crewai(project_name="crewai-integration-demo", crew=my_crew)
# Run the crew
result = my_crew.kickoff()
print(result)
```
Each run will now be logged to the Opik platform, including all agent activities and LLM calls.
## Logging CrewAI Flows
Opik also supports the CrewAI Flows API. When you enable tracking with `track_crewai`, Opik automatically:
* Tracks `Flow.kickoff()` and `Flow.kickoff_async()` as the root span/trace with inputs and outputs
* Tracks flow step methods decorated with `@start` and `@listen` as nested spans
* Captures any LLM calls (via LiteLLM) within those steps with token usage
* Flow methods are compatible with other Opik integrations (e.g., OpenAI, Anthropic, LangChain) and the `@opik.track` decorator. Any spans created inside flow steps are correctly attached to the flow's span tree.
Example:
```python
import litellm
from crewai.flow.flow import Flow, start, listen
from opik.integrations.crewai import track_crewai
track_crewai(project_name="crewai-integration-demo")
class ExampleFlow(Flow):
model = "gpt-4o-mini"
@start()
def generate_city(self):
response = litellm.completion(
model=self.model,
messages=[{"role": "user", "content": "Return the name of a random city."}],
)
return response["choices"][0]["message"]["content"]
@listen(generate_city)
def generate_fun_fact(self, random_city):
response = litellm.completion(
model=self.model,
messages=[{"role": "user", "content": f"Tell me a fun fact about {random_city}"}],
)
return response["choices"][0]["message"]["content"]
flow = ExampleFlow()
result = flow.kickoff()
```
## Cost Tracking
The `track_crewai` integration automatically tracks token usage and cost for all supported LLM models used during CrewAI agent execution.
Cost information is automatically captured and displayed in the Opik UI, including:
* Token usage details
* Cost per request based on model pricing
* Total trace cost
View the complete list of supported models and providers on the [Supported Models](/tracing/advanced/cost_tracking) page.
## Grouping traces into conversational threads using `thread_id`
Threads in Opik are collections of traces that are grouped together using a unique `thread_id`.
The `thread_id` can be passed to the CrewAI crew as a parameter, which will be used to group all traces into a single thread.
```python
from crewai import Agent, Crew, Task, Process
from opik.integrations.crewai import track_crewai
# Define your crew (using the example from above)
my_crew = YourCrewName().crew()
# Enable tracking with the crew instance (required for v1.0.0+)
track_crewai(project_name="crewai-integration-demo", crew=my_crew)
# Pass thread_id via opik_args
args_dict = {
"trace": {
"thread_id": "conversation-2",
},
}
result = my_crew.kickoff(opik_args=args_dict)
```
More information on logging chat conversations can be found in the [Log conversations](/tracing/advanced/log_chat_conversations) section.
# Observability for DSPy with Opik
> Start here to integrate Opik into your DSPy-based genai application for end-to-end LLM observability, unit testing, and optimization.
[DSPy](https://dspy.ai/) is the framework for programming—rather than prompting—language models.
In this guide, we will showcase how to integrate Opik with DSPy so that all the DSPy calls are logged as traces in Opik.
## Account Setup
[Comet](https://www.comet.com/site?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=dspy\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=dspy\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=dspy\&utm_campaign=opik) for more information.
## Getting Started
### Installation
First, ensure you have both `opik` and `dspy` installed:
```bash
pip install opik dspy
```
### Configuring Opik
Configure the Opik Python SDK for your deployment type. See the [Python SDK Configuration guide](/tracing/advanced/sdk_configuration) for detailed instructions on:
* **CLI configuration**: `opik configure`
* **Code configuration**: `opik.configure()`
* **Self-hosted vs Cloud vs Enterprise** setup
* **Configuration files** and environment variables
### Configuring DSPy
In order to configure DSPy, you will need to have your LLM provider API key. For this example, we'll use OpenAI. You can [find or create your OpenAI API Key in this page](https://platform.openai.com/settings/organization/api-keys).
You can set it as an environment variable:
```bash
export OPENAI_API_KEY="YOUR_API_KEY"
```
Or set it programmatically:
```python
import os
import getpass
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
```
## Logging DSPy calls
In order to log traces to Opik, you will need to set the `opik` callback:
```python
import dspy
from opik.integrations.dspy.callback import OpikCallback
lm = dspy.LM("openai/gpt-4o-mini")
project_name = "DSPY"
opik_callback = OpikCallback(project_name=project_name, log_graph=True)
dspy.configure(lm=lm, callbacks=[opik_callback])
cot = dspy.ChainOfThought("question -> answer")
cot(question="What is the meaning of life?")
```
The trace is now logged to the Opik platform:
If you set `log_graph` to `True` in the `OpikCallback`, then each module graph is also displayed in the "Agent graph" tab:
# Observability for Google Agent Development Kit (Python) with Opik
> Start here to integrate Opik into your Google Agent Development Kit-based genai application for end-to-end LLM observability, unit testing, and optimization.
In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a `project_name` when creating datasets and running experiments so they are associated with the correct project.
[Agent Development Kit (ADK)](https://google.github.io/adk-docs/) is a flexible and modular framework for developing and deploying AI agents. ADK can be used with popular LLMs and open-source generative AI tools and is designed with a focus on tight integration with the Google ecosystem and Gemini models. ADK makes it easy to get started with simple agents powered by Gemini models and Google AI tools while providing the control and structure needed for more complex agent architectures and orchestration.
In this guide, we will showcase how to integrate Opik with Google ADK so that all the ADK calls are logged as traces in Opik. We'll cover three key integration patterns:
1. **Automatic Agent Tracking** - Recommended approach using `track_adk_agent_recursive` for effortless instrumentation
2. **Manual Callback Configuration** - Alternative approach with explicit callback setup for fine-grained control
3. **Hybrid Tracing** - Combining Opik decorators with ADK callbacks for comprehensive observability
## Account Setup
[Comet](https://www.comet.com/site?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=google-adk\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=google-adk\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=google-adk\&utm_campaign=opik) for more information.
Opik provides comprehensive integration with ADK, automatically logging traces for all agent executions, tool calls, and LLM interactions with detailed cost tracking and error monitoring.
## Key Features
* **One-line instrumentation** with `track_adk_agent_recursive` for automatic tracing of entire agent hierarchies
* **Automatic cost tracking** for all supported LLM providers including LiteLLM models (OpenAI, Anthropic, Google AI, AWS Bedrock, and more)
* **Full compatibility** with the `@opik.track` decorator for hybrid tracing approaches
* **Thread support** for conversational applications using ADK sessions
* **Automatic agent graph visualization** with Mermaid diagrams for complex multi-agent workflows
* **Comprehensive error tracking** with detailed error information and stack traces
## Getting Started
### Installation
First, ensure you have both `opik` and `google-adk` installed:
```bash
pip install opik google-adk
```
### Configuring Opik
Configure the Opik Python SDK for your deployment type. See the [Python SDK Configuration guide](/tracing/advanced/sdk_configuration) for detailed instructions on:
* **CLI configuration**: `opik configure`
* **Code configuration**: `opik.configure()`
* **Self-hosted vs Cloud vs Enterprise** setup
* **Configuration files** and environment variables
### Configuring Google ADK
In order to configure Google ADK, you will need to have your LLM provider API key. For this example, we'll use OpenAI. You can [find or create your OpenAI API Key in this page](https://platform.openai.com/settings/organization/api-keys).
You can set it as an environment variable:
```bash
export OPENAI_API_KEY="YOUR_API_KEY"
```
Or set it programmatically:
```python
import os
import getpass
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
```
## Example 1: Automatic Agent Tracking (Recommended)
The recommended way to track ADK agents is using [`track_adk_agent_recursive`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/adk/track_adk_agent_recursive.html) and [`OpikTracer`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/adk/OpikTracer.html), which automatically instruments your entire agent hierarchy with a single function call. This approach is ideal for both single agents and complex multi-agent setups:
```python
import datetime
from zoneinfo import ZoneInfo
from google.adk.agents import LlmAgent
from google.adk.models.lite_llm import LiteLlm
from opik.integrations.adk import OpikTracer, track_adk_agent_recursive
def get_weather(city: str) -> dict:
"""Get weather information for a city."""
if city.lower() == "new york":
return {
"status": "success",
"report": "The weather in New York is sunny with a temperature of 25 °C (77 °F).",
}
elif city.lower() == "london":
return {
"status": "success",
"report": "The weather in London is cloudy with a temperature of 18 °C (64 °F).",
}
return {"status": "error", "error_message": f"Weather info for '{city}' is unavailable."}
def get_current_time(city: str) -> dict:
"""Get current time for a city."""
if city.lower() == "new york":
tz = ZoneInfo("America/New_York")
now = datetime.datetime.now(tz)
return {
"status": "success",
"report": now.strftime(f"The current time in {city} is %Y-%m-%d %H:%M:%S %Z%z."),
}
elif city.lower() == "london":
tz = ZoneInfo("Europe/London")
now = datetime.datetime.now(tz)
return {
"status": "success",
"report": now.strftime(f"The current time in {city} is %Y-%m-%d %H:%M:%S %Z%z."),
}
return {"status": "error", "error_message": f"No timezone info for '{city}'."}
# Initialize LiteLLM with OpenAI gpt-4o
llm = LiteLlm(model="openai/gpt-4o")
# Create the basic agent
basic_agent = LlmAgent(
name="weather_time_agent",
model=llm,
description="Agent for answering time & weather questions",
instruction="Answer questions about the time or weather in a city. Be helpful and provide clear information.",
tools=[get_weather, get_current_time],
)
# Configure Opik tracer
opik_tracer = OpikTracer(
name="basic-weather-agent",
tags=["basic", "weather", "time", "single-agent"],
metadata={
"environment": "development",
"model": "gpt-4o",
"framework": "google-adk",
"example": "basic"
},
project_name="adk-basic-demo"
)
# Instrument the agent with a single function call - this is the recommended approach
track_adk_agent_recursive(basic_agent, opik_tracer)
```
Each agent execution will now be automatically logged to the Opik platform with detailed trace information:
This approach automatically handles:
* **All agent callbacks** (before/after agent, model, and tool executions)
* **Sub-agents** and nested agent hierarchies
* **Agent tools** that contain other agents
* **Complex workflows** with minimal code
## Example 2: Manual Callback Configuration (Alternative Approach)
For a fine-grained control over which callbacks to instrument, you can manually configure the [`OpikTracer`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/adk/OpikTracer.html) callbacks. This approach gives you explicit control but requires more setup code:
```python
# Configure Opik tracer (same as before)
opik_tracer = OpikTracer(
name="basic-weather-agent",
tags=["basic", "weather", "time", "single-agent"],
metadata={
"environment": "development",
"model": "gpt-4o",
"framework": "google-adk",
"example": "basic"
},
project_name="adk-basic-demo"
)
# Create the agent with explicit callback configuration
basic_agent = LlmAgent(
name="weather_time_agent",
model=llm,
description="Agent for answering time & weather questions",
instruction="Answer questions about the time or weather in a city. Be helpful and provide clear information.",
tools=[get_weather, get_current_time],
before_agent_callback=opik_tracer.before_agent_callback,
after_agent_callback=opik_tracer.after_agent_callback,
before_model_callback=opik_tracer.before_model_callback,
after_model_callback=opik_tracer.after_model_callback,
before_tool_callback=opik_tracer.before_tool_callback,
after_tool_callback=opik_tracer.after_tool_callback,
)
```
For most use cases, we recommend using `track_adk_agent_recursive` (shown in Example 1) as it requires less code and automatically handles complex agent hierarchies.
## Example 3: Multi-Agent Setup with Hierarchical Tracing
This example demonstrates a complex multi-agent setup where we have specialized agents for different tasks. Using `track_adk_agent_recursive`, you can instrument the entire hierarchy with a single function call:
```python
def get_detailed_weather(city: str) -> dict:
"""Get detailed weather information including forecast."""
weather_data = {
"new york": {
"current": "Sunny, 25°C (77°F)",
"humidity": "65%",
"wind": "10 km/h NW",
"forecast": "Partly cloudy tomorrow, high of 27°C"
},
"london": {
"current": "Cloudy, 18°C (64°F)",
"humidity": "78%",
"wind": "15 km/h SW",
"forecast": "Light rain expected tomorrow, high of 16°C"
},
"tokyo": {
"current": "Partly cloudy, 22°C (72°F)",
"humidity": "70%",
"wind": "8 km/h E",
"forecast": "Sunny tomorrow, high of 25°C"
}
}
city_lower = city.lower()
if city_lower in weather_data:
data = weather_data[city_lower]
return {
"status": "success",
"report": f"Weather in {city}: {data['current']}. Humidity: {data['humidity']}, Wind: {data['wind']}. {data['forecast']}"
}
return {"status": "error", "error_message": f"Detailed weather for '{city}' is unavailable."}
def get_world_time(city: str) -> dict:
"""Get time information for major world cities."""
timezones = {
"new york": "America/New_York",
"london": "Europe/London",
"tokyo": "Asia/Tokyo",
"sydney": "Australia/Sydney",
"paris": "Europe/Paris"
}
city_lower = city.lower()
if city_lower in timezones:
tz = ZoneInfo(timezones[city_lower])
now = datetime.datetime.now(tz)
return {
"status": "success",
"report": now.strftime(f"Current time in {city}: %A, %B %d, %Y at %I:%M %p %Z")
}
return {"status": "error", "error_message": f"Time zone info for '{city}' is unavailable."}
def get_travel_info(from_city: str, to_city: str) -> dict:
"""Get basic travel information between cities."""
travel_data = {
("new york", "london"): {"flight_time": "7 hours", "time_diff": "+5 hours"},
("london", "new york"): {"flight_time": "8 hours", "time_diff": "-5 hours"},
("new york", "tokyo"): {"flight_time": "14 hours", "time_diff": "+14 hours"},
("tokyo", "new york"): {"flight_time": "13 hours", "time_diff": "-14 hours"},
("london", "tokyo"): {"flight_time": "12 hours", "time_diff": "+9 hours"},
("tokyo", "london"): {"flight_time": "11 hours", "time_diff": "-9 hours"},
}
route = (from_city.lower(), to_city.lower())
if route in travel_data:
data = travel_data[route]
return {
"status": "success",
"report": f"Travel from {from_city} to {to_city}: Approximately {data['flight_time']} flight time. Time difference: {data['time_diff']}"
}
return {"status": "error", "error_message": f"Travel info for '{from_city}' to '{to_city}' is unavailable."}
# Weather specialist agent (no Opik callbacks needed)
weather_agent = LlmAgent(
name="weather_specialist",
model=llm,
description="Specialized agent for detailed weather information",
instruction="Provide comprehensive weather information including current conditions and forecasts. Be detailed and informative.",
tools=[get_detailed_weather]
)
# Time specialist agent (no Opik callbacks needed)
time_agent = LlmAgent(
name="time_specialist",
model=llm,
description="Specialized agent for world time information",
instruction="Provide accurate time information for cities around the world. Include day of week and full date.",
tools=[get_world_time]
)
# Travel specialist agent (no Opik callbacks needed)
travel_agent = LlmAgent(
name="travel_specialist",
model=llm,
description="Specialized agent for travel information",
instruction="Provide helpful travel information including flight times and time zone differences.",
tools=[get_travel_info]
)
# Configure Opik tracer for multi-agent example
multi_agent_tracer = OpikTracer(
name="multi-agent-coordinator",
tags=["multi-agent", "coordinator", "weather", "time", "travel"],
metadata={
"environment": "development",
"model": "gpt-4o",
"framework": "google-adk",
"example": "multi-agent",
"agent_count": 4
},
project_name="adk-multi-agent-demo"
)
# Coordinator agent with sub-agents
coordinator_agent = LlmAgent(
name="travel_coordinator",
model=llm,
description="Coordinator agent that delegates to specialized agents for weather, time, and travel information",
instruction="""You are a travel coordinator that helps users with weather, time, and travel information.
You have access to three specialized agents:
- weather_specialist: For detailed weather information
- time_specialist: For world time information
- travel_specialist: For travel planning information
Delegate appropriate queries to the right specialist agents and compile comprehensive responses for the user.""",
tools=[], # No direct tools, delegates to sub-agents
sub_agents=[weather_agent, time_agent, travel_agent],
)
# Use track_adk_agent_recursive to instrument all agents at once
# This automatically adds callbacks to the coordinator and ALL sub-agents
from opik.integrations.adk import track_adk_agent_recursive
track_adk_agent_recursive(coordinator_agent, multi_agent_tracer)
```
The trace can now be viewed in the UI, showing the complete hierarchy:
The `track_adk_agent_recursive` approach is particularly powerful for:
* **Multi-agent systems** with coordinator and specialist agents
* **Sequential agents** with multiple processing steps
* **Parallel agents** executing tasks concurrently
* **Loop agents** with iterative workflows
* **Agent tools** that contain nested agents
* **Complex hierarchies** with deeply nested agent structures
By calling `track_adk_agent_recursive` once on the top-level agent, all child agents and their operations are automatically instrumented without any additional code
## Cost Tracking
Opik automatically tracks token usage and cost for all LLM calls during the agent execution, not only for the Gemini LLMs, but including the models accessed via `LiteLLM`.
View the complete list of supported models and providers on the [Supported Models](/tracing/advanced/cost_tracking) page.
## Agent Graph Visualization
Opik automatically generates visual representations of your agent workflows using Mermaid diagrams. The graph shows:
* **Agent hierarchy** and relationships
* **Sequential execution** flows
* **Parallel processing** branches
* **Loop structures** and iterations
* **Tool connections** and dependencies
The graph is automatically computed and stored with each trace, providing a clear visual understanding of your agent's execution flow:
For weather time agent the graph will look like that:
For more complex agent architectures displaying a graph may be even more beneficial:
## Example 4: Hybrid Tracing - Combining Opik Decorators with ADK Callbacks
This advanced example shows how to combine Opik's `@opik.track` decorator with ADK's callback system. This is powerful when you have complex multi-step tools that perform their own internal operations that you want to trace separately, while still maintaining the overall agent trace context.
You can use `track_adk_agent_recursive` together with `@opik.track` decorators on your tool functions for maximum visibility:
```python
from opik import track
@track(name="weather_data_processing", tags=["data-processing", "weather"])
def process_weather_data(raw_data: dict) -> dict:
"""Process raw weather data with additional computations."""
# Simulate some data processing steps that we want to trace separately
processed = {
"temperature_celsius": raw_data.get("temp_c", 0),
"temperature_fahrenheit": raw_data.get("temp_c", 0) * 9/5 + 32,
"conditions": raw_data.get("condition", "unknown"),
"comfort_index": "comfortable" if 18 <= raw_data.get("temp_c", 0) <= 25 else "less comfortable"
}
return processed
@track(name="location_validation", tags=["validation", "location"])
def validate_location(city: str) -> dict:
"""Validate and normalize city names."""
# Simulate location validation logic that we want to trace
normalized_cities = {
"nyc": "New York",
"ny": "New York",
"new york city": "New York",
"london uk": "London",
"london england": "London",
"tokyo japan": "Tokyo"
}
city_lower = city.lower().strip()
validated_city = normalized_cities.get(city_lower, city.title())
return {
"original": city,
"validated": validated_city,
"is_valid": city_lower in ["new york", "london", "tokyo"] or city_lower in normalized_cities
}
@track(name="advanced_weather_lookup", tags=["weather", "api-simulation"])
def get_advanced_weather(city: str) -> dict:
"""Get weather with internal processing steps tracked by Opik decorators."""
# Step 1: Validate location (traced by @opik.track)
location_result = validate_location(city)
if not location_result["is_valid"]:
return {
"status": "error",
"error_message": f"Invalid location: {city}"
}
validated_city = location_result["validated"]
# Step 2: Get raw weather data (simulated)
raw_weather_data = {
"New York": {"temp_c": 25, "condition": "sunny", "humidity": 65},
"London": {"temp_c": 18, "condition": "cloudy", "humidity": 78},
"Tokyo": {"temp_c": 22, "condition": "partly cloudy", "humidity": 70}
}
if validated_city not in raw_weather_data:
return {
"status": "error",
"error_message": f"Weather data unavailable for {validated_city}"
}
raw_data = raw_weather_data[validated_city]
# Step 3: Process the data (traced by @opik.track)
processed_data = process_weather_data(raw_data)
return {
"status": "success",
"city": validated_city,
"report": f"Weather in {validated_city}: {processed_data['conditions']}, {processed_data['temperature_celsius']}°C ({processed_data['temperature_fahrenheit']:.1f}°F). Comfort level: {processed_data['comfort_index']}.",
"raw_humidity": raw_data["humidity"]
}
# Configure Opik tracer for hybrid example
hybrid_tracer = OpikTracer(
name="hybrid-tracing-agent",
tags=["hybrid", "decorators", "callbacks", "advanced"],
metadata={
"environment": "development",
"model": "gpt-4o",
"framework": "google-adk",
"example": "hybrid-tracing",
"tracing_methods": ["decorators", "callbacks"]
},
project_name="adk-hybrid-demo"
)
# Create hybrid agent that combines both tracing approaches
hybrid_agent = LlmAgent(
name="advanced_weather_time_agent",
model=llm,
description="Advanced agent with hybrid Opik tracing using both decorators and callbacks",
instruction="""You are an advanced weather and time agent that provides detailed information with comprehensive internal processing.
Your tools perform multi-step operations that are individually traced, giving detailed visibility into the processing pipeline.
Use the advanced weather and time tools to provide thorough, well-processed information to users.""",
tools=[get_advanced_weather],
)
# Instrument the agent with track_adk_agent_recursive
# The @opik.track decorators in your tools will automatically create child spans
from opik.integrations.adk import track_adk_agent_recursive
track_adk_agent_recursive(hybrid_agent, hybrid_tracer)
```
The trace can now be viewed in the UI:
## Compatibility with @track Decorator
The `OpikTracer` is fully compatible with the `@track` decorator, allowing you to create hybrid tracing approaches that combine ADK agent tracking with custom function tracing.
You can both invoke your agent from inside another tracked function and call tracked functions inside your tool functions, all the spans and traces parent-child relationships will be preserved!
## Thread Support
The Opik integration automatically handles ADK sessions and maps them to Opik threads for conversational applications:
```python
from opik.integrations.adk import OpikTracer
from google.adk import sessions as adk_sessions, runners as adk_runners
# ADK session management
session_service = adk_sessions.InMemorySessionService()
session = session_service.create_session_sync(
app_name="my_app",
user_id="user_123",
session_id="conversation_456"
)
opik_tracer = OpikTracer()
runner = adk_runners.Runner(
agent=your_agent,
app_name="my_app",
session_service=session_service
)
# All traces will be automatically grouped by session_id as thread_id
```
The integration automatically:
* Uses the ADK session ID as the Opik thread ID
* Groups related conversations and interactions
* Logs app\_name and user\_id as metadata
* Maintains conversation context across multiple interactions
You can view your session as a whole conversation and easily navigate to any specific trace you need.
## Error Tracking
The `OpikTracer` provides comprehensive error tracking and monitoring:
* **Automatic error capture** for agent execution failures
* **Detailed stack traces** with full context information
* **Tool execution errors** with input/output data
* **Model call failures** with provider-specific error details
Error information is automatically logged to spans and traces, making it easy to debug issues in production:
## Troubleshooting: Missing Trace
When using `Runner.run_async`, make sure to process all events completely, even after finding the final response (when `event.is_final_response()` is `True`). If you exit the loop too early, OpikTracer won't log the final response and your trace will be incomplete. Don't use code that stops processing events prematurely:
```python
async for event in runner.run_async(user_id=user_id, session_id=session_id, new_message=content):
if event.is_final_response():
...
break # Stop processing events once the final response is found
```
There is an upstream discussion about how to best solve this source of confusion: [https://github.com/google/adk-python/issues/1695](https://github.com/google/adk-python/issues/1695).
Our team tried to address those issues and make the integration as robust as possible. If you are facing similar
problems, the first thing we recommend is to update both `opik` and `google-adk` to the latest versions. We are
actively working on improving this integration, so with the most recent versions you'll most likely get the best UX!.
## Flushing Traces
The `OpikTracer` object has a `flush` method that ensures all traces are logged to the Opik platform before you exit a script:
```python
from opik.integrations.adk import OpikTracer
opik_tracer = OpikTracer()
# Your ADK agent execution code here...
# Ensure all traces are sent before script exits
opik_tracer.flush()
```
## Prompts integration
The `OpikTracer` can be used together with the [Opik prompt library](/development/prompt-library/getting-started)
to easily access your existing prompts or create new ones, and then associate them with traces or spans within an ADK agent flow.
```python
import datetime
import uuid
from typing import Iterator, Optional
from zoneinfo import ZoneInfo
from google.adk import Agent, Runner
from google.adk.agents import LlmAgent
from google.adk.models.lite_llm import LiteLlm
from google.adk.sessions import InMemorySessionService
from google.genai import types as genai_types
from google.adk import events as adk_events
import opik
from opik.opik_context import update_current_trace
from opik.integrations.adk import OpikTracer, track_adk_agent_recursive
# Create prompt
system_prompt = opik.Prompt(
name="system-prompt",
prompt="You are a helpful assistant that provides accurate and concise answers.",
project_name="my-project",
)
# Get prompt from the Prompt library
client = opik.Opik()
user_prompt = client.get_prompt(name="user-prompt")
def get_weather(city: str) -> dict:
"""Get weather information for a city."""
# Add prompts to the current trace
update_current_trace(
prompts=[system_prompt, user_prompt]
)
if city.lower() == "new york":
return {
"status": "success",
"report": "The weather in New York is sunny with a temperature of 25 °C (77 °F).",
}
elif city.lower() == "london":
return {
"status": "success",
"report": "The weather in London is cloudy with a temperature of 18 °C (64 °F).",
}
return {"status": "error", "error_message": f"Weather info for '{city}' is unavailable."}
def get_current_time(city: str) -> dict:
"""Get current time for a city."""
if city.lower() == "new york":
tz = ZoneInfo("America/New_York")
now = datetime.datetime.now(tz)
return {
"status": "success",
"report": now.strftime(f"The current time in {city} is %Y-%m-%d %H:%M:%S %Z%z."),
}
elif city.lower() == "london":
tz = ZoneInfo("Europe/London")
now = datetime.datetime.now(tz)
return {
"status": "success",
"report": now.strftime(f"The current time in {city} is %Y-%m-%d %H:%M:%S %Z%z."),
}
return {"status": "error", "error_message": f"No timezone info for '{city}'."}
# Initialize LiteLLM with OpenAI gpt-4o
llm = LiteLlm(model="openai/gpt-4o")
# Create the basic agent
basic_agent = LlmAgent(
name="weather_time_agent",
model=llm,
description="Agent for answering time & weather questions",
instruction="Answer questions about the time or weather in a city. Be helpful and provide clear information.",
tools=[get_weather, get_current_time],
)
# Configure Opik tracer
opik_tracer = OpikTracer(
name="basic-weather-agent",
tags=["basic", "weather", "time", "single-agent"],
metadata={
"environment": "development",
"model": "gpt-4o",
"framework": "google-adk",
"example": "basic"
},
project_name="adk-basic-demo"
)
# Instrument the agent with a single function call - this is the recommended approach
track_adk_agent_recursive(basic_agent, opik_tracer)
```
# Observability for Haystack with Opik
> Start here to integrate Opik into your Haystack-based genai application for end-to-end LLM observability, unit testing, and optimization.
[Haystack](https://docs.haystack.deepset.ai/docs/intro) is an open-source framework for building production-ready LLM applications, retrieval-augmented generative pipelines and state-of-the-art search systems that work intelligently over large document collections.
In this guide, we will showcase how to integrate Opik with Haystack so that all the Haystack calls are logged as traces in Opik.
## Account Setup
[Comet](https://www.comet.com/site?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=haystack\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=haystack\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=haystack\&utm_campaign=opik) for more information.
Opik integrates with Haystack to log traces for all Haystack pipelines.
## Getting Started
### Installation
First, ensure you have both `opik` and `haystack-ai` installed:
```bash
pip install opik haystack-ai
```
### Configuring Opik
Configure the Opik Python SDK for your deployment type. See the [Python SDK Configuration guide](/tracing/advanced/sdk_configuration) for detailed instructions on:
* **CLI configuration**: `opik configure`
* **Code configuration**: `opik.configure()`
* **Self-hosted vs Cloud vs Enterprise** setup
* **Configuration files** and environment variables
### Configuring Haystack
In order to use Haystack, you will need to configure the OpenAI API Key. If you are using any other providers, you can replace this with the required API key. You can [find or create your OpenAI API Key in this page](https://platform.openai.com/settings/organization/api-keys).
You can set it as an environment variable:
```bash
export OPENAI_API_KEY="YOUR_API_KEY"
```
Or set it programmatically:
```python
import os
import getpass
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
```
## Creating the Haystack pipeline
In this example, we will create a simple pipeline that uses a prompt template to translate text to German.
To enable Opik tracing, we will:
1. Enable content tracing in Haystack by setting the environment variable `HAYSTACK_CONTENT_TRACING_ENABLED=true`
2. Add the `OpikConnector` component to the pipeline
Note: The `OpikConnector` component is a special component that will automatically log the traces of the pipeline as Opik traces, it should not be connected to any other component.
```python
import os
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from opik.integrations.haystack import OpikConnector
pipe = Pipeline()
# Add the OpikConnector component to the pipeline
pipe.add_component("tracer", OpikConnector("Chat example"))
# Continue building the pipeline
pipe.add_component("prompt_builder", ChatPromptBuilder())
pipe.add_component("llm", OpenAIChatGenerator(model="gpt-3.5-turbo"))
pipe.connect("prompt_builder.prompt", "llm.messages")
messages = [
ChatMessage.from_system(
"Always respond in German even if some input data is in other languages."
),
ChatMessage.from_user("Tell me about {{location}}"),
]
response = pipe.run(
data={
"prompt_builder": {
"template_variables": {"location": "Berlin"},
"template": messages,
}
}
)
trace_id = response["tracer"]["trace_id"]
print(f"Trace ID: {trace_id}")
print(response["llm"]["replies"][0])
```
The trace is now logged to the Opik platform:
## Cost Tracking
The `OpikConnector` automatically tracks token usage and cost for all supported LLM models used within Haystack pipelines.
Cost information is automatically captured and displayed in the Opik UI, including:
* Token usage details
* Cost per request based on model pricing
* Total trace cost
View the complete list of supported models and providers on the [Supported Models](/tracing/advanced/cost_tracking) page.
In order to ensure the traces are correctly logged, make sure you set the environment variable `HAYSTACK_CONTENT_TRACING_ENABLED` to `true` before running the pipeline.
## Advanced usage
### Ensuring the trace is logged
By default the `OpikConnector` will flush the trace to the Opik platform after each component in a thread blocking way. As a result, you may disable flushing the data after each component by setting the `HAYSTACK_OPIK_ENFORCE_FLUSH` environent variable to `false`.
**Caution**: Disabling this feature may result in data loss if the program crashes before the data is sent to Opik. Make sure you will call the `flush()` method explicitly before the program exits:
```python
from haystack.tracing import tracer
tracer.actual_tracer.flush()
```
### Getting the trace ID
If you would like to log additional information to the trace you will need to get the trace ID. You can do this by the `tracer` key in the response of the pipeline:
```python
response = pipe.run(
data={
"prompt_builder": {
"template_variables": {"location": "Berlin"},
"template": messages,
}
}
)
trace_id = response["tracer"]["trace_id"]
print(f"Trace ID: {trace_id}")
```
### Updating logged traces
The `OpikConnector` returns the logged trace ID in the pipeline run response. You can use this ID to update the trace with feedback scores or other metadata:
```python
import opik
response = pipe.run(
data={
"prompt_builder": {
"template_variables": {"location": "Berlin"},
"template": messages,
}
}
)
# Get the trace ID from the pipeline run response
trace_id = response["tracer"]["trace_id"]
# Log the feedback score
opik_client = opik.Opik()
opik_client.log_traces_feedback_scores([
{"id": trace_id, "name": "user-feedback", "value": 0.5}
])
```
# Observability for Harbor with Opik
> Start here to integrate Opik into your Harbor benchmark evaluation runs for end-to-end agent observability and analysis.
[Harbor](https://github.com/laude-institute/harbor) is a benchmark evaluation framework for autonomous LLM agents. It provides standardized infrastructure for running agents against benchmarks like SWE-bench, LiveCodeBench, Terminal-Bench, and others.
> Harbor enables you to evaluate LLM agents on complex coding tasks, tracking their trajectories using the ATIF (Agent Trajectory Interchange Format) specification.
Opik integrates with Harbor to log traces for all trial executions, including:
* **Trial results** as Opik traces with timing, metadata, and feedback scores from verifier rewards
* **Trajectory steps** as nested spans showing the complete agent-environment interaction
* **Tool calls and observations** as detailed execution records
* **Token usage and costs** aggregated from ATIF metrics
## Account Setup
[Comet](https://www.comet.com/site?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=harbor\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=harbor\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=harbor\&utm_campaign=opik) for more information.
## Getting Started
### Installation
First, ensure you have both `opik` and `harbor` installed:
```bash
pip install opik harbor
```
### Configuring Opik
Configure the Opik Python SDK for your deployment type. See the [Python SDK Configuration guide](/tracing/advanced/sdk_configuration) for detailed instructions on:
* **CLI configuration**: `opik configure`
* **Code configuration**: `opik.configure()`
* **Self-hosted vs Cloud vs Enterprise** setup
* **Configuration files** and environment variables
### Configuring Harbor
Harbor requires configuration for the agent and benchmark you want to evaluate. Refer to the [Harbor documentation](https://github.com/laude-institute/harbor) for details on setting up your job configuration.
## Using the CLI
The easiest way to use Harbor with Opik is through the `opik harbor` CLI command. This automatically enables Opik tracking for all trial executions without modifying your code.
### Basic Usage
```bash
# Run a benchmark with Opik tracking
opik harbor run -d terminal-bench@head -a terminus_2 -m gpt-4.1
# Use a configuration file
opik harbor run -c config.yaml
```
### Specifying Project Name
```bash
# Set project name via environment variable
export OPIK_PROJECT_NAME=my-benchmark
opik harbor run -d swebench@lite
```
### Available CLI Commands
All Harbor CLI commands are available as subcommands:
```bash
# Run a job (alias for jobs start)
opik harbor run [HARBOR_OPTIONS]
# Job management
opik harbor jobs start [HARBOR_OPTIONS]
opik harbor jobs resume -p ./jobs/my-job
# Single trial
opik harbor trials start -p ./my-task -a terminus_2
```
### CLI Help
```bash
# View available options
opik harbor --help
opik harbor run --help
```
## Example: SWE-bench Evaluation
Here's a complete example running a SWE-bench evaluation with Opik tracking:
```bash
# Configure Opik
opik configure
# Set project name
export OPIK_PROJECT_NAME=swebench-claude-sonnet
# Run SWE-bench evaluation with tracking
opik harbor run \
-d swebench-lite@head \
-a claude-code \
-m claude-3-5-sonnet-20241022
```
## Custom Agents
Harbor supports integrating your own custom agents without modifying the Harbor source code. There are two types of agents you can create:
* **External agents** - Interface with the environment through the `BaseEnvironment` interface, typically by executing bash commands
* **Installed agents** - Installed directly into the container environment and executed in headless mode
For details on implementing custom agents, see the [Harbor Agents documentation](https://harborframework.com/docs/agents).
### Running Custom Agents with Opik
To run a custom agent with Opik tracking, use the `--agent-import-path` flag:
```bash
opik harbor run -d "terminal-bench@head" --agent-import-path path.to.agent:MyCustomAgent
```
### Tracking Custom Agent Functions
When building custom agents, you can use Opik's `@track` decorator on methods within your agent implementation. These decorated functions will automatically be captured as spans within the trial trace, giving you detailed visibility into your agent's internal logic:
```python
from harbor.agents.base import BaseAgent
from opik import track
class MyCustomAgent(BaseAgent):
@staticmethod
def name() -> str:
return "my-custom-agent"
@track
async def plan_next_action(self, observation: str) -> str:
# This function will appear as a span in Opik
# Add your planning logic here
return action
@track
async def execute_tool(self, tool_name: str, args: dict) -> str:
# This will also be tracked as a nested span
result = await self._run_tool(tool_name, args)
return result
async def run(self, instruction: str, environment, context) -> None:
# Your main agent loop
while not done:
observation = await environment.exec("pwd")
action = await self.plan_next_action(observation)
result = await self.execute_tool(action.tool, action.args)
```
This allows you to trace not just the ATIF trajectory steps, but also the internal decision-making processes of your custom agent.
## What Gets Logged
Each trial completion creates an Opik trace with:
* Trial name and task information as the trace name and input
* Agent execution timing as start/end times
* Verifier rewards (e.g., pass/fail, tests passed) as feedback scores
* Agent and model metadata
* Exception information if the trial failed
### Trajectory Spans
The integration automatically creates spans for each step in the agent's trajectory, giving you detailed visibility into the agent-environment interaction. Each trajectory step becomes a span showing:
* The step source (user, agent, or system)
* The message content
* Tool calls and their arguments
* Observation results from the environment
* Token usage and cost per step
* Model name for agent steps
### Verifier Rewards as Feedback Scores
Harbor's verifier produces rewards like `{"pass": 1, "tests_passed": 5}`. These are automatically converted to Opik feedback scores, allowing you to:
* Filter traces by pass/fail status
* Aggregate metrics across experiments
* Compare agent performance across benchmarks
## Cost Tracking
The Harbor integration automatically extracts token usage and cost from ATIF trajectory metrics. If your agent records `prompt_tokens`, `completion_tokens`, and `cost_usd` in step metrics, these are captured in Opik spans.
## Environment Variables
| Variable | Description |
| ------------------- | ------------------------------- |
| `OPIK_PROJECT_NAME` | Default project name for traces |
| `OPIK_API_KEY` | API key for Opik Cloud |
| `OPIK_WORKSPACE` | Workspace name (for Opik Cloud) |
### Getting Help
* Check the [Harbor documentation](https://github.com/laude-institute/harbor) for agent and benchmark setup
* Review the [ATIF specification](https://www.harborframework.com/docs/agents/trajectory-format) for trajectory format details
* Open an issue on [GitHub](https://github.com/comet-ml/opik/issues) for Opik integration questions
# Structured Output Tracking for Instructor with Opik
> Start here to integrate Opik into your Instructor-based genai application for structured output tracking, schema validation monitoring, and LLM call observability.
[Instructor](https://github.com/instructor-ai/instructor) is a Python library for working with structured outputs
for LLMs built on top of Pydantic. It provides a simple way to manage schema validations, retries and streaming responses.
In this guide, we will showcase how to integrate Opik with Instructor so that all the Instructor calls are logged as traces in Opik.
## Account Setup
[Comet](https://www.comet.com/site?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=instructor\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=instructor\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=instructor\&utm_campaign=opik) for more information.
## Getting Started
### Installation
First, ensure you have both `opik` and `instructor` installed:
```bash
pip install opik instructor
```
### Configuring Opik
Configure the Opik Python SDK for your deployment type. See the [Python SDK Configuration guide](/tracing/advanced/sdk_configuration) for detailed instructions on:
* **CLI configuration**: `opik configure`
* **Code configuration**: `opik.configure()`
* **Self-hosted vs Cloud vs Enterprise** setup
* **Configuration files** and environment variables
### Configuring Instructor
In order to use Instructor, you will need to configure your LLM provider API keys. For this example, we'll use OpenAI, Anthropic, and Gemini. You can [find or create your API keys in these pages](https://platform.openai.com/settings/organization/api-keys):
You can set them as environment variables:
```bash
export OPENAI_API_KEY="YOUR_API_KEY"
export ANTHROPIC_API_KEY="YOUR_API_KEY"
export GOOGLE_API_KEY="YOUR_API_KEY"
```
Or set them programmatically:
```python
import os
import getpass
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
if "ANTHROPIC_API_KEY" not in os.environ:
os.environ["ANTHROPIC_API_KEY"] = getpass.getpass("Enter your Anthropic API key: ")
if "GOOGLE_API_KEY" not in os.environ:
os.environ["GOOGLE_API_KEY"] = getpass.getpass("Enter your Google API key: ")
```
## Using Opik with Instructor library
In order to log traces from Instructor into Opik, we are going to patch the `instructor` library. This will log each LLM call to the Opik platform.
For all the integrations, we will first add tracking to the LLM client and then pass it to the Instructor library:
```python
from opik.integrations.openai import track_openai
import instructor
from pydantic import BaseModel
from openai import OpenAI
# We will first create the OpenAI client and add the `track_openai`
# method to log data to Opik
openai_client = track_openai(OpenAI())
# Patch the OpenAI client for Instructor
client = instructor.from_openai(openai_client)
# Define your desired output structure
class UserInfo(BaseModel):
name: str
age: int
user_info = client.chat.completions.create(
model="gpt-4o-mini",
response_model=UserInfo,
messages=[{"role": "user", "content": "John Doe is 30 years old."}],
)
print(user_info)
```
Thanks to the `track_openai` method, all the calls made to OpenAI will be logged to the Opik platform. This approach also works well if you are also using the `opik.track` decorator as it will automatically log the LLM call made with Instructor to the relevant trace.
## Integrating with other LLM providers
The instructor library supports many LLM providers beyond OpenAI, including: Anthropic, AWS Bedrock, Gemini, etc. Opik supports the majority of these providers as well.
Here are the code snippets needed for the integration with different providers:
### Anthropic
```python
from opik.integrations.anthropic import track_anthropic
import instructor
from anthropic import Anthropic
# Add Opik tracking
anthropic_client = track_anthropic(Anthropic())
# Patch the Anthropic client for Instructor
client = instructor.from_anthropic(
anthropic_client, mode=instructor.Mode.ANTHROPIC_JSON
)
user_info = client.chat.completions.create(
model="claude-3-5-sonnet-20241022",
response_model=UserInfo,
messages=[{"role": "user", "content": "John Doe is 30 years old."}],
max_tokens=1000,
)
print(user_info)
```
### Gemini
```python
from opik.integrations.genai import track_genai
import instructor
from google import genai
# Add Opik tracking
gemini_client = track_genai(genai.Client())
# Patch the GenAI client for Instructor
client = instructor.from_genai(
gemini_client, mode=instructor.Mode.GENAI_STRUCTURED_OUTPUTS
)
user_info = client.chat.completions.create(
model="gemini-2.0-flash-001",
response_model=UserInfo,
messages=[{"role": "user", "content": "John Doe is 30 years old."}],
)
print(user_info)
```
You can read more about how to use the Instructor library in [their documentation](https://python.useinstructor.com/).
# Observability for LangChain (Python) with Opik
> Start here to integrate Opik into your LangChain-based genai application for end-to-end LLM observability, unit testing, and optimization.
In Opik 2.0, datasets and experiments are project-scoped. Make sure to specify a `project_name` when creating datasets and running experiments so they are associated with the correct project.
Opik provides seamless integration with LangChain, allowing you to easily log and trace your LangChain-based applications. By using the `OpikTracer` callback, you can automatically capture detailed information about your LangChain runs, including inputs, outputs, metadata, and cost tracking for each step in your chain.
## Key Features
* **Automatic cost tracking** for supported LLM providers (OpenAI, Anthropic, Google AI, AWS Bedrock, and more)
* **Full compatibility** with the `@opik.track` decorator for hybrid tracing approaches
* **Thread support** for conversational applications with `thread_id` parameter
* **Distributed tracing** support for multi-service applications
* **LangGraph compatibility** for complex graph-based workflows
* **Evaluation and testing** support for automated LLM application testing
## Account Setup
[Comet](https://www.comet.com/site?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=langchain\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=langchain\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=langchain\&utm_campaign=opik) for more information.
## Getting Started
### Installation
To use the `OpikTracer` with LangChain, you'll need to have both the `opik` and `langchain` packages installed. You can install them using pip:
```bash
pip install opik langchain langchain_openai
```
### Configuring Opik
Configure the Opik Python SDK for your deployment type. See the [Python SDK Configuration guide](/tracing/advanced/sdk_configuration) for detailed instructions on:
* **CLI configuration**: `opik configure`
* **Code configuration**: `opik.configure()`
* **Self-hosted vs Cloud vs Enterprise** setup
* **Configuration files** and environment variables
## Using OpikTracer
Here's a basic example of how to use the `OpikTracer` callback with a LangChain chain:
```python
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from opik.integrations.langchain import OpikTracer
# Initialize the tracer
opik_tracer = OpikTracer(project_name="langchain-examples")
llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = ChatPromptTemplate.from_messages([
("human", "Translate the following text to French: {text}")
])
chain = prompt | llm
result = chain.invoke(
{"text": "Hello, how are you?"},
config={"callbacks": [opik_tracer]}
)
print(result.content)
```
The `OpikTracer` will automatically log the run and its details to Opik, including the input prompt, the output, and metadata for each step in the chain.
For detailed parameter information, see the [OpikTracer SDK reference](https://www.comet.com/docs/opik/python-sdk-reference/integrations/langchain/OpikTracer.html).
## Practical Example: Text-to-SQL with Evaluation
Let's walk through a real-world example of using LangChain with Opik for a text-to-SQL query generation task. This example demonstrates how to create synthetic datasets, build LangChain chains, and evaluate your application.
### Setting up the Environment
First, let's set up our environment with the necessary dependencies:
```python
import os
import getpass
import opik
from opik.integrations.openai import track_openai
from openai import OpenAI
# Configure Opik
opik.configure(use_local=False)
os.environ["OPIK_PROJECT_NAME"] = "langchain-integration-demo"
# Set up API keys
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
```
### Creating a Synthetic Dataset
We'll create a synthetic dataset of questions for our text-to-SQL task:
```python
import json
from langchain_community.utilities import SQLDatabase
# Download and set up the Chinook database
import requests
url = "https://github.com/lerocha/chinook-database/raw/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite"
filename = "./data/chinook/Chinook_Sqlite.sqlite"
folder = os.path.dirname(filename)
if not os.path.exists(folder):
os.makedirs(folder)
if not os.path.exists(filename):
response = requests.get(url)
with open(filename, "wb") as file:
file.write(response.content)
print("Chinook database downloaded")
db = SQLDatabase.from_uri(f"sqlite:///{filename}")
# Create synthetic questions using OpenAI
client = OpenAI()
openai_client = track_openai(client)
prompt = """
Create 20 different example questions a user might ask based on the Chinook Database.
These questions should be complex and require the model to think. They should include complex joins and window functions to answer.
Return the response as a json object with a "result" key and an array of strings with the question.
"""
completion = openai_client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": prompt}]
)
synthetic_questions = json.loads(completion.choices[0].message.content)["result"]
# Create dataset in Opik
opik_client = opik.Opik()
dataset = opik_client.get_or_create_dataset(name="synthetic_questions", project_name="my-project")
dataset.insert([{"question": question} for question in synthetic_questions])
```
### Building the LangChain Chain
Now let's create a LangChain chain for SQL query generation:
```python
from langchain.chains import create_sql_query_chain
from langchain_openai import ChatOpenAI
from opik.integrations.langchain import OpikTracer
# Create the LangChain chain with OpikTracer
opik_tracer = OpikTracer(tags=["sql_generation"])
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
chain = create_sql_query_chain(llm, db).with_config({"callbacks": [opik_tracer]})
# Test the chain
response = chain.invoke({"question": "How many employees are there?"})
print(response)
```
### Evaluating the Application
Let's create a custom evaluation metric and test our application:
```python
import opik
from opik import track
from opik.evaluation import evaluate
from opik.evaluation.metrics import base_metric, score_result
from typing import Any
opik.configure(project_name="my-project")
class ValidSQLQuery(base_metric.BaseMetric):
def __init__(self, name: str, db: Any):
self.name = name
self.db = db
def score(self, output: str, **ignored_kwargs: Any):
try:
db.run(output)
return score_result.ScoreResult(
name=self.name, value=1, reason="Query ran successfully"
)
except Exception as e:
return score_result.ScoreResult(name=self.name, value=0, reason=str(e))
# Set up evaluation
valid_sql_query = ValidSQLQuery(name="valid_sql_query", db=db)
dataset = opik_client.get_dataset("synthetic_questions")
@track()
def llm_chain(input: str) -> str:
response = chain.invoke({"question": input})
return response
def evaluation_task(item):
response = llm_chain(item["question"])
return {"output": response}
# Run evaluation
res = evaluate(
experiment_name="SQL question answering",
dataset=dataset,
task=evaluation_task,
scoring_metrics=[valid_sql_query],
nb_samples=20,
project_name="my-project",
)
```
The evaluation results are now uploaded to the Opik platform and can be viewed in the UI.
## Cost Tracking
The `OpikTracer` automatically tracks token usage and cost for all supported LLM models used within LangChain applications.
Cost information is automatically captured and displayed in the Opik UI, including:
* Token usage details
* Cost per request based on model pricing
* Total trace cost
View the complete list of supported models and providers on the [Supported Models](/tracing/advanced/cost_tracking) page.
For streaming with cost tracking, ensure `stream_usage=True` is set:
```python
from langchain_openai import ChatOpenAI
from opik.integrations.langchain import OpikTracer
llm = ChatOpenAI(
model="gpt-4o",
streaming=True,
stream_usage=True, # Required for cost tracking with streaming
)
opik_tracer = OpikTracer()
for chunk in llm.stream("Hello", config={"callbacks": [opik_tracer]}):
print(chunk.content, end="")
```
View the complete list of supported models and providers on the [Supported Models](/tracing/advanced/cost_tracking) page.
## Settings tags and metadata
You can customize the `OpikTracer` callback to include additional metadata, logging options, and conversation threading:
```python
from opik.integrations.langchain import OpikTracer
opik_tracer = OpikTracer(
tags=["langchain", "production"],
metadata={"use-case": "customer-support", "version": "1.0"},
thread_id="conversation-123", # For conversational applications
project_name="my-langchain-project"
)
```
## Accessing logged traces
You can use the [`created_traces`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/langchain/OpikTracer.html) method to access the traces collected by the `OpikTracer` callback:
```python
from opik.integrations.langchain import OpikTracer
opik_tracer = OpikTracer()
# Calling Langchain object
traces = opik_tracer.created_traces()
print([trace.id for trace in traces])
```
The traces returned by the `created_traces` method are instances of the [`Trace`](https://www.comet.com/docs/opik/python-sdk-reference/Objects/Trace.html#opik.api_objects.trace.Trace) class, which you can use to update the metadata, feedback scores and tags for the traces.
### Accessing the content of logged traces
In order to access the content of logged traces you will need to use the [`Opik.get_trace_content`](https://www.comet.com/docs/opik/python-sdk-reference/Opik.html#opik.Opik.get_trace_content) method:
```python
import opik
from opik.integrations.langchain import OpikTracer
opik_client = opik.Opik()
opik_tracer = OpikTracer()
# Calling Langchain object
# Getting the content of the logged traces
traces = opik_tracer.created_traces()
for trace in traces:
content = opik_client.get_trace_content(trace.id)
print(content)
```
### Updating and scoring logged traces
You can update the metadata, feedback scores and tags for traces after they are created. For this you can use the `created_traces` method to access the traces and then update them using the [`update`](https://www.comet.com/docs/opik/python-sdk-reference/Objects/Trace.html#opik.api_objects.trace.Trace.update) method and the [`log_feedback_score`](https://www.comet.com/docs/opik/python-sdk-reference/Objects/Trace.html#opik.api_objects.trace.Trace.log_feedback_score) method:
```python
from opik.integrations.langchain import OpikTracer
opik_tracer = OpikTracer(project_name="langchain-examples")
# ... calling Langchain object
traces = opik_tracer.created_traces()
for trace in traces:
trace.update(tags=["my-tag"])
trace.log_feedback_score(name="user-feedback", value=0.5)
```
## Compatibility with @track Decorator
The `OpikTracer` is fully compatible with the `@track` decorator, allowing you to create hybrid tracing approaches:
```python
import opik
from langchain_openai import ChatOpenAI
from opik.integrations.langchain import OpikTracer
@opik.track
def my_langchain_workflow(user_input: str) -> str:
llm = ChatOpenAI(model="gpt-4o")
opik_tracer = OpikTracer()
# The LangChain call will create spans within the existing trace
response = llm.invoke(user_input, config={"callbacks": [opik_tracer]})
return response.content
result = my_langchain_workflow("What is machine learning?")
```
## Thread Support
Use the `thread_id` parameter to group related conversations or interactions:
```python
from opik.integrations.langchain import OpikTracer
# All traces with the same thread_id will be grouped together
opik_tracer = OpikTracer(thread_id="user-session-123")
```
## Distributed Tracing
For multi-service/thread/process applications, you can use distributed tracing headers to connect traces across services:
```python
from opik import opik_context
from opik.integrations.langchain import OpikTracer
from opik.types import DistributedTraceHeadersDict
# In your service that receives distributed trace headers.
# The distributed_headers dict can be obtained in the "parent" service via `opik_context.get_distributed_trace_headers()`
distributed_headers = DistributedTraceHeadersDict(
opik_trace_id="trace-id-from-upstream",
opik_parent_span_id="parent-span-id-from-upstream"
)
opik_tracer = OpikTracer(distributed_headers=distributed_headers)
# LangChain operations will be attached to the existing distributed trace
chain.invoke(input_data, config={"callbacks": [opik_tracer]})
```
Learn more about distributed tracing in the [Distributed Tracing guide](/tracing/advanced/log_distributed_traces).
## LangGraph Integration
For LangGraph applications, Opik provides specialized support. The `OpikTracer` works seamlessly with LangGraph, and you can also visualize graph definitions:
```python
from langgraph.graph import StateGraph
from opik.integrations.langchain import OpikTracer
# Your LangGraph setup
graph = StateGraph(...)
compiled_graph = graph.compile()
opik_tracer = OpikTracer()
result = compiled_graph.invoke(
input_data,
config={"callbacks": [opik_tracer]}
)
```
For detailed LangGraph integration examples, see the [LangGraph Integration guide](/integrations/langgraph).
## Advanced usage
The `OpikTracer` object has a `flush` method that can be used to make sure that all traces are logged to the Opik platform before you exit a script. This method will return once all traces have been logged or if the timeout is reach, whichever comes first.
```python
from opik.integrations.langchain import OpikTracer
opik_tracer = OpikTracer()
opik_tracer.flush()
```
## Important notes
1. **Asynchronous streaming**: If you are using asynchronous streaming mode (calling `.astream()` method), the `input` field in the trace UI may be empty due to a LangChain limitation for this mode. However, you can find the input data inside the nested spans of this chain.
2. **Streaming with cost tracking**: If you are planning to use streaming with LLM calls and want to calculate LLM call tokens/cost, you need to explicitly set `stream_usage=True`:
# Observability for LangGraph with Opik
> Start here to integrate Opik into your LangGraph-based genai application for end-to-end LLM observability, unit testing, and optimization.
Opik provides a seamless integration with LangGraph, allowing you to easily log and trace your LangGraph-based applications. By using the `OpikTracer` callback, you can automatically capture detailed information about your LangGraph graph executions during both development and production.
## Account Setup
[Comet](https://www.comet.com/site?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=langgraph\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=langgraph\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=langgraph\&utm_campaign=opik) for more information.
## Getting Started
### Installation
To use the [`OpikTracer`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/langchain/OpikTracer.html) with LangGraph, you'll need to have both the `opik` and `langgraph` packages installed. You can install them using pip:
```bash
pip install opik langgraph langchain
```
### Configuring Opik
Configure the Opik Python SDK for your deployment type. See the [Python SDK Configuration guide](/tracing/advanced/sdk_configuration) for detailed instructions on:
* **CLI configuration**: `opik configure`
* **Code configuration**: `opik.configure()`
* **Self-hosted vs Cloud vs Enterprise** setup
* **Configuration files** and environment variables
## Using Opik with LangGraph
Opik provides two ways to track LangGraph applications. We recommend using the `track_langgraph` function for a simpler experience, but you can also use the `OpikTracer` callback directly if you need more control.
### Option 1: Using `track_langgraph` (Recommended)
The simplest way to track your LangGraph applications is using the [`track_langgraph`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/langchain/track_langgraph.html) function. This function wraps your compiled graph once, and all subsequent invocations are automatically tracked without needing to pass callbacks:
```python
from typing import List, Annotated
from pydantic import BaseModel
from opik.integrations.langchain import OpikTracer, track_langgraph
from langchain_core.messages import HumanMessage
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
# create your LangGraph graph
class State(BaseModel):
messages: Annotated[list, add_messages]
def chatbot(state):
# Typically your LLM calls would be done here
return {"messages": "Hello, how can I help you today?"}
graph = StateGraph(State)
graph.add_node("chatbot", chatbot)
graph.add_edge(START, "chatbot")
graph.add_edge("chatbot", END)
app = graph.compile()
# Create OpikTracer and track the graph once - no need to pass callbacks anymore!
# The graph visualization is automatically extracted by track_langgraph
opik_tracer = OpikTracer(
tags=["production"],
metadata={"version": "1.0"}
)
app = track_langgraph(app, opik_tracer)
# Now all invocations are automatically tracked
for s in app.stream({"messages": [HumanMessage(content = "How to use LangGraph ?")]}):
print(s)
# No callbacks needed here either!
result = app.invoke({"messages": [HumanMessage(content = "How to use LangGraph ?")]})
```
This is similar to how other Opik integrations work (like OpenAI, Anthropic, etc.), where you wrap the client or object once and then use it normally.
### Option 2: Using `OpikTracer` callback
If you need more fine-grained control or want to use different tracers for different invocations, you can use the [`OpikTracer`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/langchain/OpikTracer.html) callback directly:
```python
from typing import List, Annotated
from pydantic import BaseModel
from opik.integrations.langchain import OpikTracer
from langchain_core.messages import HumanMessage
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
# create your LangGraph graph
class State(BaseModel):
messages: Annotated[list, add_messages]
def chatbot(state):
# Typically your LLM calls would be done here
return {"messages": "Hello, how can I help you today?"}
graph = StateGraph(State)
graph.add_node("chatbot", chatbot)
graph.add_edge(START, "chatbot")
graph.add_edge("chatbot", END)
app = graph.compile()
# Create the OpikTracer
opik_tracer = OpikTracer()
# Pass the OpikTracer callback to each invocation
for s in app.stream({"messages": [HumanMessage(content = "How to use LangGraph ?")]},
config={"callbacks": [opik_tracer]}):
print(s)
result = app.invoke({"messages": [HumanMessage(content = "How to use LangGraph ?")]},
config={"callbacks": [opik_tracer]})
```
### Viewing Traces in the UI
Once tracking is enabled using either method, you will start to see the traces in the Opik UI:
## Practical Example: Classification Workflow
Let's walk through a real-world example of using LangGraph with Opik for a classification workflow. This example demonstrates how to create a graph with conditional routing and track its execution.
### Setting up the Environment
First, let's set up our environment with the necessary dependencies:
```python
import opik
# Configure Opik
opik.configure(use_local=False)
```
### Creating the LangGraph Workflow
We'll create a LangGraph workflow with 3 nodes that demonstrates conditional routing:
```python
from langgraph.graph import StateGraph, END
from typing import TypedDict, Optional
# Define the graph state
class GraphState(TypedDict):
question: Optional[str] = None
classification: Optional[str] = None
response: Optional[str] = None
# Create the node functions
def classify(question: str) -> str:
return "greeting" if question.startswith("Hello") else "search"
def classify_input_node(state):
question = state.get("question", "").strip()
classification = classify(question)
return {"classification": classification}
def handle_greeting_node(state):
return {"response": "Hello! How can I help you today?"}
def handle_search_node(state):
question = state.get("question", "").strip()
search_result = f"Search result for '{question}'"
return {"response": search_result}
# Create the workflow
workflow = StateGraph(GraphState)
workflow.add_node("classify_input", classify_input_node)
workflow.add_node("handle_greeting", handle_greeting_node)
workflow.add_node("handle_search", handle_search_node)
# Add conditional routing
def decide_next_node(state):
return (
"handle_greeting"
if state.get("classification") == "greeting"
else "handle_search"
)
workflow.add_conditional_edges(
"classify_input",
decide_next_node,
{"handle_greeting": "handle_greeting", "handle_search": "handle_search"},
)
workflow.set_entry_point("classify_input")
workflow.add_edge("handle_greeting", END)
workflow.add_edge("handle_search", END)
app = workflow.compile()
```
### Executing with Opik Tracing
Now let's execute the workflow with Opik tracing enabled using `track_langgraph`:
```python
from opik.integrations.langchain import OpikTracer, track_langgraph
# Create OpikTracer and track the graph once
# The graph visualization is automatically extracted by track_langgraph
opik_tracer = OpikTracer(
project_name="classification-workflow"
)
app = track_langgraph(app, opik_tracer)
# Execute the workflow - no callbacks needed!
inputs = {"question": "Hello, how are you?"}
result = app.invoke(inputs)
print(result)
# Test with a different input - still tracked automatically
inputs = {"question": "What is machine learning?"}
result = app.invoke(inputs)
print(result)
```
The graph execution is now logged on the Opik platform and can be viewed in the UI. The trace will show the complete execution path through the graph, including the classification decision and the chosen response path.
## Compatibility with Opik tracing context
LangGraph tracing integrates seamlessly with Opik's tracing context, allowing you to call `@track`-decorated functions (and most use most of other native Opik integrations) from within your graph nodes and have them automatically attached to the trace tree.
### Synchronous execution (invoke)
For synchronous graph execution using `invoke()`, everything works out of the box. You can access current spans/traces from LangGraph nodes and call tracked functions inside them:
```python
import opik_context
from opik import track
from opik.integrations.langchain import OpikTracer, track_langgraph
from langgraph.graph import StateGraph, START, END
@track
def process_data(value: int) -> int:
"""Custom tracked function that will be attached to the trace tree."""
return value * 2
def my_node(state):
current_trace_data = opik_context.get_current_trace_data()
current_span_data = opik_context.get_current_span_data() # will return the span for `my_node`, created by OpikTracer
# This tracked function call will automatically be part of the trace tree
result = process_data(state["value"])
return {"value": result}
# Build and execute graph
graph = StateGraph(dict)
graph.add_node("processor", my_node)
graph.add_edge(START, "processor")
graph.add_edge("processor", END)
app = graph.compile()
opik_tracer = OpikTracer()
app = track_langgraph(app, opik_tracer)
# Synchronous execution - tracked functions work automatically
result = app.invoke({"value": 21})
```
### Asynchronous execution (ainvoke)
For asynchronous graph execution using `ainvoke()`, you need to explicitly propagate the trace context to `@track`-decorated functions using the `extract_current_langgraph_span_data` helper:
This is due to a LangChain framework limitation that doesn't automatically share the execution context between callbacks (like `OpikTracer`) and node code in async scenarios. The explicit trace context propagation via distributed headers is required for seamless tracking across async boundaries.
```python
from opik import track
from opik.integrations.langchain import OpikTracer, track_langgraph, extract_current_langgraph_span_data
from langgraph.graph import StateGraph, START, END
@track
def process_data(value: int) -> int:
"""Custom tracked function that needs distributed trace headers in async context."""
return value * 2
async def my_async_node(state, config):
# Extract current span data from LangGraph config. `opik_context` doesn't work here due to langgraph platform limitations related to context propagation.
span_data = extract_current_langgraph_span_data(config)
# Pass distributed trace headers to attach the tracked function to the trace tree
result = process_data(
state["value"],
opik_distributed_trace_headers=span_data.get_distributed_trace_headers() # all tracked functions implicitly support this parameter
)
return {"value": result}
# Build and execute graph
graph = StateGraph(dict)
graph.add_node("processor", my_async_node)
graph.add_edge(START, "processor")
graph.add_edge("processor", END)
app = graph.compile()
opik_tracer = OpikTracer()
app = track_langgraph(app, opik_tracer)
# Asynchronous execution - requires explicit trace context propagation
result = await app.ainvoke({"value": 21})
```
Alternatively, if you don't want to use the `@track` decorator, you can use the `opik.start_as_current_span` context manager with distributed headers:
```python
import opik
from opik.integrations.langchain import OpikTracer, track_langgraph, extract_current_langgraph_span_data
from langgraph.graph import StateGraph, START, END
async def my_async_node(state, config):
span_data = extract_current_langgraph_span_data(config)
# Use context manager with distributed headers
with opik.start_as_current_span(
name="custom_operation",
input={"input": state["value"]},
opik_distributed_trace_headers=span_data.get_distributed_trace_headers()
) as span_data:
# Your custom logic here
result = state["value"] * 2
span_data.output = {"output": result}
return {"value": result}
# Build and execute graph
graph = StateGraph(dict)
graph.add_node("processor", my_async_node)
graph.add_edge(START, "processor")
graph.add_edge("processor", END)
app = graph.compile()
opik_tracer = OpikTracer()
app = track_langgraph(app, opik_tracer)
result = await app.ainvoke({"value": 21})
```
## Logging threads
When you are running multi-turn conversations using [LangGraph persistence](https://langchain-ai.github.io/langgraph/concepts/persistence/#threads), Opik will use Langgraph's thread\_id as Opik thread\_id. Here is an example below:
```python
import sqlite3
from langgraph.checkpoint.sqlite import SqliteSaver
from typing import Annotated
from pydantic import BaseModel
from opik.integrations.langchain import OpikTracer, track_langgraph
from langchain_core.messages import HumanMessage
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langchain.chat_models import init_chat_model
llm = init_chat_model("openai:gpt-4.1")
# create your LangGraph graph
class State(BaseModel):
messages: Annotated[list, add_messages]
def chatbot(state):
# Typically your LLM calls would be done here
return {"messages": [llm.invoke(state.messages)]}
graph = StateGraph(State)
graph.add_node("chatbot", chatbot)
graph.add_edge(START, "chatbot")
graph.add_edge("chatbot", END)
# Create a new SqliteSaver instance
# Note: check_same_thread=False is OK as the implementation uses a lock
# to ensure thread safety.
conn = sqlite3.connect("checkpoints.sqlite", check_same_thread=False)
memory = SqliteSaver(conn)
app = graph.compile(checkpointer=memory)
# Create the OpikTracer and track the graph
opik_tracer = OpikTracer()
app = track_langgraph(app, opik_tracer)
thread_id = "e424a45e-7763-443a-94ae-434b39b67b72"
config = {"configurable": {"thread_id": thread_id}}
# Initialize the state
state = State(**app.get_state(config).values) or State(messages=[])
print("STATE", state)
# Add the user message
state.messages.append(HumanMessage(content="Hello, my name is Bob, how are you doing ?"))
# state.messages.append(HumanMessage(content="What is my name ?"))
result = app.invoke(state, config=config)
print("Result", result)
```
## Updating logged traces
You can use the [`OpikTracer.created_traces`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/langchain/OpikTracer.html#opik.integrations.langchain.OpikTracer.created_traces) method to access the trace IDs collected by the OpikTracer callback:
```python
from opik.integrations.langchain import OpikTracer
opik_tracer = OpikTracer()
# Calling LangGraph stream or invoke functions
traces = opik_tracer.created_traces()
print([trace.id for trace in traces])
```
These can then be used with the [`Opik.log_traces_feedback_scores`](https://www.comet.com/docs/opik/python-sdk-reference/Opik.html#opik.Opik.log_traces_feedback_scores) method to update the logged traces.
## Advanced usage
The `OpikTracer` object has a `flush` method that can be used to make sure that all traces are logged to the Opik platform before you exit a script. This method will return once all traces have been logged or if the timeout is reach, whichever comes first.
```python
from opik.integrations.langchain import OpikTracer
opik_tracer = OpikTracer()
opik_tracer.flush()
```
# Observability for LangServe with Opik
> Configure LangServe host-level OpenTelemetry export for Opik with clear applicability and minimal setup guidance.
[LangServe](https://github.com/langchain-ai/langserve) does not define its own standalone OpenTelemetry bootstrap. Instrumentation is configured at the host app/runtime layer (typically FastAPI + LangChain components).
## When this guide applies
Use this when your LangServe routes run inside a Python web app and you want request + framework spans in Opik.
## Opik OTLP endpoint modes
For full endpoint/header details, see [Opik OpenTelemetry overview](/integrations/opentelemetry).
```bash wordWrap
export OTEL_EXPORTER_OTLP_ENDPOINT=https://www.comet.com/opik/api/v1/private/otel
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=
## What gets traced
With this setup, your LiveKit agent will automatically trace:
* **Session events**: Session start and end with metadata
* **Agent turns**: Complete conversation turns with timing
* **LLM operations**: Model calls, prompts, responses, and token usage
* **Function tools**: Tool executions with inputs and outputs
* **TTS operations**: Text-to-speech conversions with audio metadata
* **STT operations**: Speech-to-text transcriptions
* **End-of-turn detection**: Conversation flow events
## Further improvements
If you have any questions or suggestions for improving the LiveKit Agents integration, please [open an issue](https://github.com/comet-ml/opik/issues/new/choose) on our GitHub repository.
# Observability for LlamaIndex with Opik
> Start here to integrate Opik into your LlamaIndex-based genai application for end-to-end LLM observability, unit testing, and optimization.
[LlamaIndex](https://github.com/run-llama/llama_index) is a flexible data framework for building LLM applications:
LlamaIndex is a "data framework" to help you build LLM apps. It provides the following tools:
* Offers data connectors to ingest your existing data sources and data formats (APIs, PDFs, docs, SQL, etc.).
* Provides ways to structure your data (indices, graphs) so that this data can be easily used with LLMs.
* Provides an advanced retrieval/query interface over your data: Feed in any LLM input prompt, get back retrieved context and knowledge-augmented output.
* Allows easy integrations with your outer application framework (e.g. with LangChain, Flask, Docker, ChatGPT, anything else).
## Account Setup
[Comet](https://www.comet.com/site?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=llamaindex\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=llamaindex\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=llamaindex\&utm_campaign=opik) for more information.
## Getting Started
### Installation
To use the Opik integration with LlamaIndex, you'll need to have both the `opik` and `llama_index` packages installed. You can install them using pip:
```bash
pip install opik llama-index llama-index-agent-openai llama-index-llms-openai llama-index-callbacks-opik
```
### Configuring Opik
Configure the Opik Python SDK for your deployment type. See the [Python SDK Configuration guide](/tracing/advanced/sdk_configuration) for detailed instructions on:
* **CLI configuration**: `opik configure`
* **Code configuration**: `opik.configure()`
* **Self-hosted vs Cloud vs Enterprise** setup
* **Configuration files** and environment variables
### Configuring LlamaIndex
In order to use LlamaIndex, you will need to configure your LLM provider API keys. For this example, we'll use OpenAI. You can [find or create your API keys in these pages](https://platform.openai.com/settings/organization/api-keys):
You can set them as environment variables:
```bash
export OPENAI_API_KEY="YOUR_API_KEY"
```
Or set them programmatically:
```python
import os
import getpass
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
```
## Using the Opik integration
To use the Opik integration with LLamaIndex, you can use the `set_global_handler` function from the LlamaIndex package to set the global tracer:
```python
from llama_index.core import global_handler, set_global_handler
set_global_handler("opik")
opik_callback_handler = global_handler
```
Now that the integration is set up, all the LlamaIndex runs will be traced and logged to Opik.
Alternatively, you can configure the callback handler directly for more control:
```python
from llama_index.core import Settings
from llama_index.core.callbacks import CallbackManager
from opik.integrations.llama_index import LlamaIndexCallbackHandler
# Basic setup
opik_callback = LlamaIndexCallbackHandler()
# Or with optional parameters
opik_callback = LlamaIndexCallbackHandler(
project_name="my-llamaindex-project", # Set custom project name
skip_index_construction_trace=True # Skip tracking index construction
)
Settings.callback_manager = CallbackManager([opik_callback])
```
The `skip_index_construction_trace` parameter is useful when you want to track only query operations and not the index construction phase (particularly for large document sets or pre-built indexes)
## Example
To showcase the integration, we will create a new a query engine that will use Paul Graham's essays as the data source.
**First step:**
Configure the Opik integration:
```python
import os
from llama_index.core import global_handler, set_global_handler
# Set project name for better organization
os.environ["OPIK_PROJECT_NAME"] = "llamaindex-integration-demo"
set_global_handler("opik")
opik_callback_handler = global_handler
```
**Second step:**
Download the example data:
```python
import os
import requests
# Create directory if it doesn't exist
os.makedirs('./data/paul_graham/', exist_ok=True)
# Download the file using requests
url = 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt'
response = requests.get(url)
with open('./data/paul_graham/paul_graham_essay.txt', 'wb') as f:
f.write(response.content)
```
**Third step:**
Configure the OpenAI API key:
```python
import os
import getpass
if "OPENAI_API_KEY" not in os.environ:
os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ")
```
**Fourth step:**
We can now load the data, create an index and query engine:
```python
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader("./data/paul_graham").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response)
```
Given that the integration with Opik has been set up, all the traces are logged to the Opik platform:
## Using with the @track Decorator
The LlamaIndex integration seamlessly works with Opik's `@track` decorator. When you call LlamaIndex operations inside a tracked function, the LlamaIndex traces will automatically be attached as child spans to your existing trace.
```python
import opik
from llama_index.core import global_handler, set_global_handler
from llama_index.llms.openai import OpenAI
from llama_index.core.llms import ChatMessage
# Configure Opik integration
set_global_handler("opik")
opik_callback_handler = global_handler
@opik.track()
def my_llm_application(user_query: str):
"""Process user query with LlamaIndex"""
llm = OpenAI(model="gpt-3.5-turbo")
messages = [
ChatMessage(role="system", content="You are a helpful assistant."),
ChatMessage(role="user", content=user_query),
]
response = llm.chat(messages)
return response.message.content
# Call the tracked function
result = my_llm_application("What is the capital of France?")
print(result)
```
In this example, Opik will create a trace for the `my_llm_application` function, and all LlamaIndex operations (like the LLM chat call) will appear as nested spans within this trace, giving you a complete view of your application's execution.
## Using with Manual Trace Creation
You can also manually create traces using `opik.start_as_current_trace()` and have LlamaIndex operations nested within:
```python
import opik
from llama_index.core import global_handler, set_global_handler
from llama_index.llms.openai import OpenAI
from llama_index.core.llms import ChatMessage
# Configure Opik integration
set_global_handler("opik")
opik_callback_handler = global_handler
# Create a manual trace
with opik.start_as_current_trace(name="user_query_processing"):
llm = OpenAI(model="gpt-3.5-turbo")
messages = [
ChatMessage(role="user", content="Explain quantum computing in simple terms"),
]
response = llm.chat(messages)
print(response.message.content)
```
This approach is useful when you want more control over trace naming and want to group multiple LlamaIndex operations under a single trace.
## Tracking LlamaIndex Workflows
LlamaIndex workflows are multi-step processing pipelines for LLM applications. To track workflow executions in Opik, you can manually decorate your workflow steps and use `opik.start_as_current_span()` to wrap the workflow execution.
### Basic Workflow Tracking
You can use `@opik.track()` to decorate your workflow steps and `opik.start_as_current_span()` to track the workflow execution:
```python
import opik
from llama_index.core.workflow import Workflow, StartEvent, StopEvent, step, Event
from llama_index.core import Settings
from llama_index.core.callbacks import CallbackManager
from llama_index.core import global_handler, set_global_handler
# Configure Opik integration for LLM calls within steps
set_global_handler("opik")
class QueryEvent(Event):
"""Event for passing query through workflow."""
query: str
class MyRAGWorkflow(Workflow):
"""Simple RAG workflow with tracked steps."""
@step
@opik.track()
async def retrieve_context(self, ev: StartEvent) -> QueryEvent:
"""Retrieve relevant context for the query."""
query = ev.get("query", "")
# Your retrieval logic here
context = f"Context for: {query}"
return QueryEvent(query=f"{context} | {query}")
@step
@opik.track()
async def generate_response(self, ev: QueryEvent) -> StopEvent:
"""Generate final response using the context."""
# Your generation logic here
result = f"Response based on: {ev.query}"
return StopEvent(result=result)
# Create workflow instance
workflow = MyRAGWorkflow()
# Use start_as_current_span to track workflow execution
with opik.start_as_current_span(
name="rag_workflow_execution",
input={"query": "What are the key features?"},
project_name="llama-index-workflows"
) as span:
result = await workflow.run(query="What are the key features?")
span.update(output={"result": result})
print(result)
opik.flush_tracker() # Ensure all traces are sent
```
In this example:
* Each workflow step is decorated with `@opik.track()` to create spans
* The `@step` decorator is placed before `@opik.track()` to ensure LlamaIndex can properly discover the workflow steps
* `opik.start_as_current_span()` tracks the overall workflow execution
* LLM calls within steps are automatically tracked via the global Opik handler
* All workflow steps appear as nested spans within the workflow trace
If you're certain the workflow is a top-level call and want to create only a trace without an additional span, you can use `opik.start_as_current_trace()` instead of `opik.start_as_current_span()`. However, `start_as_current_span()` is more flexible as it works in both standalone and nested contexts.
### Best Practices
1. **Decorate all workflow steps** with `@opik.track()` to capture each step as a span
2. **Decorator order matters**: Place `@step` before `@opik.track()` so LlamaIndex's workflow engine can properly discover and execute steps
3. **Use `opik.start_as_current_span()`** to wrap workflow execution - it works in both standalone and nested contexts
4. **Configure the global handler** to automatically track LLM calls within steps
5. **Use descriptive names** for spans to make debugging easier
6. **Always call `opik.flush_tracker()`** at the end to ensure all traces are sent
7. **Include input/output** in span updates for better debugging
## Token Usage in Streaming Responses
When using streaming chat responses with OpenAI models (e.g., `llm.stream_chat()`), you need to explicitly enable token usage tracking by configuring the `stream_options` parameter:
```python
from llama_index.llms.openai import OpenAI
from llama_index.core.llms import ChatMessage
from llama_index.core import global_handler, set_global_handler
# Configure Opik integration
set_global_handler("opik")
# Configure OpenAI LLM with stream_options to include usage information
llm = OpenAI(
model="gpt-3.5-turbo",
additional_kwargs={
"stream_options": {"include_usage": True}
}
)
messages = [
ChatMessage(role="user", content="Tell me a short joke")
]
# Token usage will now be tracked in streaming responses
response = llm.stream_chat(messages)
for chunk in response:
print(chunk.delta, end="", flush=True)
```
Without setting `stream_options={'include_usage': True}`, streaming responses from OpenAI models will not include token usage information in Opik traces. This is a requirement of OpenAI's streaming API.
## Cost Tracking
The Opik integration with LlamaIndex automatically tracks token usage and cost for all supported LLM models used within LlamaIndex applications.
Cost information is automatically captured and displayed in the Opik UI, including:
* Token usage details
* Cost per request based on model pricing
* Total trace cost
View the complete list of supported models and providers on the [Supported Models](/tracing/advanced/cost_tracking) page.
# Observability for Microsoft Agent Framework (Python) with Opik
> Start here to integrate Opik into your Microsoft Agent Framework-based genai application for end-to-end LLM observability, unit testing, and optimization.
[Microsoft Agent Framework](https://github.com/microsoft/agent-framework) is a comprehensive multi-language framework for building, orchestrating, and deploying AI agents and multi-agent workflows with support for both Python and .NET implementations.
The framework provides everything from simple chat agents to complex multi-agent workflows with graph-based orchestration, built-in OpenTelemetry integration for distributed tracing and monitoring, and a flexible middleware system for request/response processing.
## Account Setup
[Comet](https://www.comet.com/site?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=agent-framework\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=agent-framework\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=agent-framework\&utm_campaign=opik) for more information.
## Getting started
To use the Microsoft Agent Framework integration with Opik, you will need to have the Agent Framework and the required OpenTelemetry packages installed:
```bash
pip install --pre agent-framework opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
```
In addition, you will need to set the following environment variables to configure OpenTelemetry to send data to Opik:
If you are using Opik Cloud, you will need to set the following
environment variables:
```bash wordWrap
export OTEL_EXPORTER_OTLP_ENDPOINT=https://www.comet.com/opik/api/v1/private/otel
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=
You can create agents with custom function tools. The `OpikTracingProcessor` automatically captures all tool calls as well:
```python
from agents import Agent, Runner, function_tool, set_trace_processors
from opik.integrations.openai.agents import OpikTracingProcessor
set_trace_processors(processors=[OpikTracingProcessor()])
@function_tool
def calculate_average(numbers: list[float]) -> float:
return sum(numbers) / len(numbers)
@function_tool
def get_recommendation(topic: str, user_level: str) -> str:
recommendations = {
"python": {
"beginner": "Start with Python.org's tutorial, then try Python Crash Course book. Practice with simple scripts and built-in functions.",
"intermediate": "Explore frameworks like Flask/Django, learn about decorators, context managers, and dive into Python's data structures.",
"advanced": "Study Python internals, contribute to open source, learn about metaclasses, and explore performance optimization."
},
"machine learning": {
"beginner": "Start with Andrew Ng's Coursera course, learn basic statistics, and try scikit-learn with simple datasets.",
"intermediate": "Dive into deep learning with TensorFlow/PyTorch, study different algorithms, and work on real projects.",
"advanced": "Research latest papers, implement algorithms from scratch, and contribute to ML frameworks."
}
}
topic_lower = topic.lower()
level_lower = user_level.lower()
if topic_lower in recommendations and level_lower in recommendations[topic_lower]:
return recommendations[topic_lower][level_lower]
else:
return f"For {topic} at {user_level} level: Focus on fundamentals, practice regularly, and build projects to apply your knowledge."
def create_advanced_agent():
"""Create an advanced agent with tools and comprehensive instructions."""
instructions = """
You are an expert programming tutor and learning advisor. You have access to tools that help you:
1. Calculate averages for performance metrics, grades, or other numerical data
2. Provide personalized learning recommendations based on topics and user experience levels
Your role:
- Help users learn programming concepts effectively
- Provide clear, beginner-friendly explanations when needed
- Use your tools when appropriate to give concrete help
- Offer structured learning paths and resources
- Be encouraging and supportive
When users ask about:
- Programming languages: Use get_recommendation to provide tailored advice
- Performance or scores: Use calculate_average if numbers are involved
- Learning paths: Combine your knowledge with tool-based recommendations
Always explain your reasoning and make your responses educational.
"""
return Agent(
name="AdvancedProgrammingTutor",
instructions=instructions,
model="gpt-4o-mini",
tools=[calculate_average, get_recommendation]
)
# Create and use the advanced agent
advanced_agent = create_advanced_agent()
# Example queries
queries = [
"I'm new to Python programming. Can you tell me about it?",
"I got these test scores: 85, 92, 78, 96, 88. What's my average and how am I doing?",
"I know some Python basics but want to learn machine learning. What should I do next?",
]
for i, query in enumerate(queries, 1):
print(f"\n📝 Query {i}: {query}")
result = Runner.run_sync(advanced_agent, query)
print(f"🤖 Response: {result.final_output}")
print("=" * 80)
```
### Adding granularity with the `@track` decorator
If you need more visibility into what happens inside your tool functions, you can use the `@track` decorator to trace specific steps within the tool execution:
```python
from agents import Agent, Runner, function_tool, set_trace_processors
from opik.integrations.openai.agents import OpikTracingProcessor
from opik import track
set_trace_processors(processors=[OpikTracingProcessor()])
@track(name="fetch_user_data")
def fetch_user_data(user_id: str) -> dict:
# This step will be traced separately
return {"user_id": user_id, "preferences": ["python", "ml"]}
@track(name="generate_recommendations")
def generate_recommendations(preferences: list) -> str:
# This step will also be traced separately
return f"Based on your interests in {', '.join(preferences)}, we recommend..."
@function_tool
def get_personalized_advice(user_id: str) -> str:
"""Get personalized learning advice for a user."""
# Each tracked function call inside the tool will appear as a separate span
user_data = fetch_user_data(user_id)
recommendations = generate_recommendations(user_data["preferences"])
return recommendations
agent = Agent(
name="PersonalizedTutor",
instructions="Help users with personalized learning advice.",
model="gpt-4o-mini",
tools=[get_personalized_advice]
)
result = Runner.run_sync(agent, "Give me learning advice for user_123")
print(result.final_output)
```
## Logging threads
When you are running multi-turn conversations with OpenAI Agents using [OpenAI Agents trace API](https://openai.github.io/openai-agents-python/running_agents/#conversationschat-threads), Opik integration automatically use the trace group\_id as the Thread ID so you can easily review conversation inside Opik. Here is an example below:
```python
async def main():
agent = Agent(name="Assistant", instructions="Reply very concisely.")
thread_id = str(uuid.uuid4())
with trace(workflow_name="Conversation", group_id=thread_id):
# First turn
result = await Runner.run(agent, "What city is the Golden Gate Bridge in?")
print(result.final_output)
# San Francisco
# Second turn
new_input = result.to_input_list() + [{"role": "user", "content": "What state is it in?"}]
result = await Runner.run(agent, new_input)
print(result.final_output)
# California
```
## Further improvements
OpenAI Agents is still a relatively new framework and we are working on a couple of improvements:
1. Improved rendering of the inputs and outputs for the LLM calls as part of our `Pretty Mode` functionality
2. Improving the naming conventions for spans
3. Adding the agent execution input and output at a trace level
If there are any additional improvements you would like us to make, feel free to open an issue on our [GitHub repository](https://github.com/comet-ml/opik/issues).
# Observability for Pipecat with Opik
> Start here to integrate Opik into your Pipecat-based real-time voice agent application for end-to-end LLM observability, unit testing, and optimization.
[Pipecat](https://github.com/pipecat-ai/pipecat) is an open-source Python framework for building real-time voice and multimodal conversational AI agents. Developed by Daily, it enables fully programmable AI voice agents and supports multimodal interactions, positioning itself as a flexible solution for developers looking to build conversational AI systems.
This guide explains how to integrate Opik with Pipecat for observability and tracing of real-time voice agents, enabling you to monitor, debug, and optimize your Pipecat agents in the Opik dashboard.
## Account Setup
[Comet](https://www.comet.com/site?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=pipecat\&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=pipecat\&utm_campaign=opik) and grab your API Key.
> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm\&utm_source=opik\&utm_medium=colab\&utm_content=pipecat\&utm_campaign=opik) for more information.
## Getting started
To use the Pipecat integration with Opik, you will need to have Pipecat and the required OpenTelemetry packages installed:
```bash
pip install pipecat-ai[daily,webrtc,silero,cartesia,deepgram,openai,tracing] opentelemetry-exporter-otlp-proto-http websockets
```
```bash wordWrap
export OTEL_EXPORTER_OTLP_ENDPOINT=https://www.comet.com/opik/api/v1/private/otel
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=
## Logging threads
You can group multiple agent calls into a conversation thread by setting `thread_id` as a span attribute on the root Logfire span. Opik's OTEL ingestion recognizes this attribute and maps it directly to the trace's `thread_id` field:
```python
# Logfire wraps OTEL - thread_id becomes a span attribute automatically
with logfire.span("chat_turn", thread_id=thread_id):
result = agent.run_sync("What is machine learning?")
```
## Further improvements
If you would like to see us improve this integration, simply open a new feature
request on [Github](https://github.com/comet-ml/opik/issues).
# Observability for Semantic Kernel (Python) with Opik
> Start here to integrate Opik into your Semantic Kernel-based genai application for end-to-end LLM observability, unit testing, and optimization.
[Semantic Kernel](https://github.com/microsoft/semantic-kernel) is a powerful open-source SDK from Microsoft. It facilitates the combination of LLMs with popular programming languages like C#, Python, and Java. Semantic Kernel empowers developers to build sophisticated AI applications by seamlessly integrating AI services, data sources, and custom logic, accelerating the delivery of enterprise-grade AI solutions.
Learn more about Semantic Kernel in the [official documentation](https://learn.microsoft.com/en-us/semantic-kernel/overview/).
## Getting started
To use the Semantic Kernel integration with Opik, you will need to have Semantic Kernel and the required OpenTelemetry packages installed:
```bash
pip install semantic-kernel opentelemetry-exporter-otlp-proto-http
```
## Environment configuration
Configure your environment variables based on your Opik deployment:
If you are using Opik Cloud, you will need to set the following
environment variables:
```bash wordWrap
export OTEL_EXPORTER_OTLP_ENDPOINT=https://www.comet.com/opik/api/v1/private/otel
export OTEL_EXPORTER_OTLP_HEADERS='Authorization=