Log confusion matrices¶

Comet's confusion matrices allow you to log and visualize the confusion matrix for your Experiment, and optionally associate image samples, text, and URLs for each matrix cell.

You can interact with a logged confusion matrix from the Comet UI to seamlessly gauge the performance of your model with respect to true positives, false positives, true negatives, and false negatives.

The following method can be used to log confusion matrices:

• log_confusion_matrix().

The confusion matrices logged to an Experiment can be viewed in the following Single Experiment Tabs:

• Confusion Matrix Tab.
• Assets & Artifacts Tab: confusion-matrices folder.

Discover more in our Debugging Classifiers with Confusion Matrices blog post.

Log from true and predicted labels¶

The example below logs the confusion matrix for a mock binary classifier of cat and dog with a test set of 4 examples. The true and predicted labels are required as integers, and thus need to be one-hot encoded (cat -> 0, dog -> 1).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

import comet_ml

# Initialize the Comet Experiment
comet_ml.login()
exp = comet_ml.start()

# Create an example model true and predicted labels
y_true = [0, 0, 0, 1]
y_predicted = [1, 0, 1, 0]
labels = ["cat", "dog"]

# Log the confusion matrix to Comet
exp.log_confusion_matrix(
    y_true=y_true,
    y_predicted=y_predicted,
    labels=labels,
)

The log_confusion_matrix() method allows you to specify a variety of attributes for the confusion matrix; for example, you could use the title argument to update the plot tile (default is Confusion Matrix) or the max_examples_per_cell argument to change the maximum number of examples per cell (default is 25). Please refer to the method definition for a full list of arguments.

Log from existing confusion matrix¶

The example below logs a confusion matrix created with sklearn.metrics.confusion_matrix() for a mock binary classifier of cat and dog with a test set of 4 examples.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import comet_ml
from sklearn.metrics import confusion_matrix

# Initialize the Comet Experiment
comet_ml.login()
exp = comet_ml.start()

# Create an example confusion matrix
y_true = [0, 0, 0, 1]
y_predicted = [1, 0, 1, 0]
labels = ["cat", "dog"]
matrix = confusion_matrix(y_true=[0, 0, 0, 1], y_pred=[1, 0, 1, 0])

# Log the confusion matrix to Comet
exp.log_confusion_matrix(matrix=matrix, labels=labels)

The matrix argument excepts an ndarray, or list, of shape (n_classes, n_classes).

Log with sample images¶

For computer vision experiments, you can optionally log the image associated with each (y_true, y_predicted) entry. The example below logs images for a mock binary classifier of cat and dog with a test set of 4 examples, with example images loaded with PIL.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

import comet_ml
import requests
from PIL import Image
import io

# Initialize the Comet Experiment
comet_ml.login()
exp = comet_ml.start()

# Create an example model true and predicted labels with associated images
image_urls = [
    "https://cdn.pixabay.com/photo/2017/07/25/01/22/cat-2536662_960_720.jpg",
    "https://cdn.pixabay.com/photo/2014/11/30/14/11/cat-551554_960_720.jpg",
    "https://cdn.pixabay.com/photo/2016/03/28/12/35/cat-1285634_960_720.png",
    "https://cdn.pixabay.com/photo/2016/02/19/15/46/dog-1210559_960_720.jpg",
]
image_dataset = []
for url in image_urls:
    response = requests.get(url)
    image = Image.open(io.BytesIO(response.content))
    image_dataset += [image]

y_true = [0, 0, 0, 1]
y_predicted = [1, 0, 1, 0]
labels = ["cat", "dog"]

# Log the confusion matrix to Comet
exp.log_confusion_matrix(
    y_true=y_true,
    y_predicted=y_predicted,
    images=image_dataset,
    labels=labels,
)

The images argument supports images in PIL, TF tensor, Pytorch tensor and numpy array format.