Comet Product Releases December 2024

Each layer of visibility into your training and debugging workflows builds confidence that your models will work reliably in production. With a focus on details that help you iterate and ship complex systems at scale, we’ve added new logging capabilities and more to the Comet platform. Log in today to discover a set of new features designed to help you understand how your ML and GenAI systems are performing on the whole, and drill down to analyze specific issues.

In case you missed the big news out of AWS re:Invent, Comet is now available as one of the very first third-party apps fully integrated within Amazon SageMaker. You can now log into your SageMaker account and provision a self-contained and managed version of Comet securely inside SageMaker. Learn more.

Meanwhile, as teams race to build accurate and efficient systems on top of LLMs, we’ve added 4 powerful new features to Opik, Comet’s end-to-end LLM evaluation platform. Opik’s new prompt library makes it easy to store and version your prompt templates — and manage them in production. Opik now supports multimodal logging, allowing you to track not just the text input and output of your LLM, but images too. New features within our monitoring dashboard include a trace metrics tab featuring feedback scores, token usage, and number of traces over time, as well as a cost tracking feature for greater visibility into your LLM usage.

Read on for details on how these enhancements streamline your LLM development workflows!

Comet is Now Available Natively within AWS

Comet is now one of the first fully managed 3rd-party solutions available natively in Amazon SageMaker! This partnership lets you launch a private Comet instance directly within your SageMaker console – with infrastructure, billing, and security fully managed by AWS.

With the native Comet + SageMaker integration, developers get:

• Automated LLM evaluation with Opik (including Bedrock support)
• Best-in-class experiment tracking & visualization, fully integrated with SageMaker

Read more about the announcement here!

Opik: Multi-Modal Logging

Opik now supports multimodal traces, so you can log images to the platform and better analyze the relationship between text and images in your LLM applications. Log images using OpenAI’s SDK, or manually using automatic detection within the Opik UI.

Opik: Prompt Library

It can be easy to lose track of which prompt templates you’ve used in the past, but Opik’s new prompt library makes versioning a breeze. Our SDK prompt object lets you version prompts stored in code or in files, making it simple to download the most recent prompt in production without deploying new code.

Opik: LLM Trace Metrics

In Opik’s Projects section, you’ll see a new tab that plots trace metrics over time, including average feedback scores, token usage, and number of traces, helping you more effectively monitor your LLMs in production.

Opik: LLM Cost Tracking

Monitor and optimize your LLM spend with the addition of cost tracking. Opik measures token usage across all traces, so you can identify cost patterns and anomalies at any level, from individual spans to project-wide analytics. All costs are estimated in USD and automatically calculated for text OpenAI models.

New Integration: Opik + AWS Bedrock

We’re pleased to announce our integration with AWS Bedrock. This new feature enables teams to monitor their LLM models deployed on AWS from leading AI providers — such as AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon. This feature is fully available now with Opik SDK v1.0.5.

How to Upgrade

If you are using www.comet.com, no action is required! You have access to these features and can use them immediately.

If you are using a self-hosted version of Comet, before you upgrade please verify which version of Comet you are running.
Comet Versions December 2024
Frontend version: 5.90.5
Backend version: 3.48.238
Optimizer version: 2.1.4
SDK version: 3.47.4

To check your frontend version: Click your username in the top-right corner of the Comet application and check the Version at the bottom of the pop-up.

To check your backend version: Visit . <your_comet_url>.com/api/isAlive/ver

To check your SDK version: In your training environment run the command pip freeze | grep comet_ml

If you would like to upgrade, please contact your Technical Account Manager or Deployment Engineer on the Comet team. You can always reach out directly to Claire Longo (clairel@comet.com) with general questions.