Comet Product Releases January 2025

As 2025 picks up steam, we’re thrilled to bring you some exciting product updates from Comet! This month, we’ve added new features to make your machine learning and LLM development workflows more efficient and effective. From new scoring and experimentation features in Opik, our open-source LLM observability platform, to enhancements in Comet’s suite of ML Experiment Management tools, you’ll find a set of releases designed to help get great results out of your models faster.

Within Opik, we’ve released Online Evaluation, which gives you the ability to score traces automatically in production for hallucinations, answer relevance, or any other task-specific metric. You can also save time testing and iterating with Opik’s new Prompt Playground, a space to run and compare different LLM prompts directly within the UI. ML Experiment Management now features Autogenerated Panels, for faster and easier visualization of your logged metrics.

Read on for details on how these releases simplify your ML & LLM workflows, and log into the platform to give them a try.

Opik: Online LLM Evaluation

When working with LLMs in production, the sheer number of traces means it is impossible to review each trace manually. Our new Online Evaluation feature within Opik allows you to define LLM-as-a-judge metrics that automatically score the LLM calls logged to the platform. By defining LLM-as-a-judge metrics that run on all your production traces, you can automate the monitoring of your LLM calls for hallucinations, answer relevance, or any other task-specific metric.

Opik: LLM Prompt Playground

Opik’s new Prompt Playground lets you interact with OpenAI models directly within the UI, so you can spend less time writing code to test new prompts and more time comparing results and iterating. All of your prompts run in the playground and appear in the playground tracing project.

ML Experiment Management: Autogenerated Panels

You can now view autogenerated panels within Experiment Management. For every metric you log, there will now be a panel created within your personal view. There’s no need to manually add your panels, making it a breeze to visualize the model training metrics you log.

New Opik Integrations

Opik is designed to enhance your existing LLM development workflows, so you can get improve insights and results without rebuilding your tech stack. Our list of supported integrations is always growing, with Haystack and Dify as recent additions.

Haystack

Opik now integrates directly with Haystack, meaning all chains or agents defined using Haystack can be logged to Opik. Haystack is a Python framework for creating agents, and the addition of Opik greatly increases visibility into what you are building.

Dify

Dify is a popular no-code platform for building LLM-powered apps and chatbots. Dify users can now leverage Opik to track and analyze interactions with their apps, with better insights into app performance. Opik goes above and beyond Dify’s native tracking and monitoring capabilities with features like trace annotation and log evaluation, so you can iterate more effectively.

How to Upgrade

If you are using www.comet.com, no action is required! You have access to these features and can use them immediately.

If you are using a self-hosted version of Comet, before you upgrade please verify which version of Comet you are running.

Comet Versions January 2025

Frontend version: 5.90.15
Backend version: 3.53.251
Optimizer version: 2.1.4
SDK version: 3.47.6

To check your frontend version: Click your username in the top-right corner of the Comet application and check the Version at the bottom of the pop-up.

To check your backend version: Visit. <your_comet_url>.com/api/isAlive/ver

To check your SDK version: In your training environment run the command pip freeze | grep comet_ml

If you would like to upgrade, please contact your Technical Account Manager or Deployment Engineer on the Comet team. You can always reach out directly to Claire Longo (clairel@comet.com) with general questions.