Model Production Monitoring¶

Comet's Model Production Monitoring (MPM) helps you maintain high quality ML models by monitoring and alerting on defective predictions from models deployed in production.

MPM has been designed to support all model types, ranging from tree-based to computer vision and allowing you to monitor all your models in a single location. Thanks to its integrations with Comet Experiment Management you can track model performance from training to production.

To gain access to this feature, contact the Comet Sales team.

Monitor models in production¶

Monitoring models in production can be challenging, especially when ground truth labels are not available. MPM takes away all the hassle by tracking both accuracy-related metrics and data drift with just a few clicks.

MPM was built to tackle several key challenges:

• Lack of accuracy metrics: Often labels are not available or might be significantly delayed. This is why data drift is a key metric that Comet tracks.
• Granularity of insights: The key to any good tracking system is to detect issues quickly and provide the tools to debug these issues. This is why Comet tracks a number of metrics, including data drift, at a model level and feature level.
• Ease of configuration: Just send your events to Comet and we'll start tracking model performance! There is no need to sync data schemas, list input features, or even send events in order - we take care of everything for you.

MPM's integration with Experiment Management allows you to manage and monitor from training to production in a single tool. This greatly simplifies knowledge sharing as all modeling and performance information is in a single platform and accessible to all Data Scientists.

Main MPM features¶

MPM can be used to monitor the performance of models in production, spanning fraud detection and image classification.

There are four parts to MPM functionality:

• Model overview: At a glance, see which models are performing well and which ones need your attention.
• Model-level metrics: Track the overall health of the model by tracking data drift across all input features, as well as drift of output predictions.
• Feature-level metrics: Track the distribution of input and output features, feature data drift, and missing values.
• Custom metrics: Custom metrics can be defined to track model-specific metrics.
• Alerting: Define alert notifications on any metrics tracked and displayed in the Comet UI.

Learn more¶

• Integrate your system with Comet's MPM
• Learn your way around the MPM UI
• Defining custom metrics