Smoke Testing Comet¶
This guide provides instructions for smoke testing Comet. It utilizes kubectl and a tool called cometx.
Verifying the installation¶
If you've installed Comet using our Helm charts, follow these instructions to validate your install.
A successful Helm install should have an output like this:
> helm upgrade --install --namespace comet-ns --create-namespace -f override-values.yaml comet-ml .
Release "comet-ml" does not exist. Installing it now.
NAME: comet-ns
LAST DEPLOYED: Mon Jan 27 13:29:58 2025
NAMESPACE: comet-ns
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
http://<your_comet_url>/
Check that Comet pods have been provisioned in the cluster and are running and healthy.
> kubectl -n <comet-ns> get pods
NAME READY STATUS RESTARTS AGE
pod/chi-opik-clickhouse-cluster-0-0-0 1/1 Running 0 3m41s
pod/your-comet-altinity-clickhouse-operator-795846c468-jk64c 1/1 Running 0 3m56s
pod/your-comet-comet-ml-backend-optimizer-9c94d499c-2fxkj 1/1 Running 0 3m55s
pod/your-comet-comet-ml-backend-optimizer-9c94d499c-j68rz 1/1 Running 0 3m55s
pod/your-comet-comet-ml-backend-postprocess-6b48d77c-8xzrk 1/1 Running 0 3m56s
pod/your-comet-comet-ml-backend-postprocess-6b48d77c-s6g8c 1/1 Running 0 3m56s
pod/your-comet-comet-ml-backend-python-68485876f-gnl5m 1/1 Running 0 3m56s
pod/your-comet-comet-ml-backend-python-68485876f-t7jlv 1/1 Running 0 3m56s
pod/your-comet-comet-ml-backend-react-5d44c7b64-4pn5t 1/1 Running 0 3m56s
pod/your-comet-comet-ml-backend-react-5d44c7b64-4rkhd 1/1 Running 0 3m56s
pod/your-comet-comet-ml-frontend-nginx-69d898b468-sd6wq 1/1 Running 0 3m56s
pod/comet-ml-minio-5487b87989-mqzrf 1/1 Running 0 3m55s
pod/comet-ml-minio-provisioning-bfh22 0/1 Completed 1 3m55s
pod/comet-ml-mysql-0 1/1 Running 0 3m56s
pod/comet-ml-python-panels-orchestrator-7457d5cf7-k5hwb 1/1 Running 0 3m56s
pod/comet-ml-redis-master-0 1/1 Running 0 3m56s
pod/opik-backend-675959dbbb-drxt2 1/1 Running 0 3m55s
pod/opik-frontend-58f74cf9d4-xzv8c 1/1 Running 0 3m56s
Note: For Comet to be routable, it's likely you will need to have handled DNS. For AWS this is done via Route 53.
Confirm Data-Layer configuration¶
The data-layers consist of:
- Database
- Cache
- Object storage
Self Hosted deploys should follow the instructions here (credentials required for access): External Dependencies
Running Smoke Tests via cometx¶
Get Cometx:
python -m pip install cometx
The instructions are covered in the readme of the repo however we'll go over the basics here to get started with smoke testing.
First, in your terminal run:
python -m venv venv
source venv/bin/activate
pip install comet-ml opik cometx[all] --upgrade
You'll need to make your Comet API key accessible via:
export COMET_API_KEY="YOUR-COMET-API-KEY"
Find your Comet API key from the Comet UI by clicking on your avatar in the upper right and copying it from the dropdown menu.
Since Opik has a different API endpoint we need to configure that separately.
opik configure
You should see an output similar to the following:
Which Opik deployment do you want to log your traces to?
1 - Opik Cloud (default)
2 - Self-hosted Comet platform
3 - Local deployment
For self-hosted Comet installations select 2 and you'll be asked for your Opik API key. This is the same key you set for the COMET_API_KEY env above.
Please enter your Opik API key:
Do you want to use "sanitytest" workspace? (Y/n)Y
OPIK: Configuration saved to file: /Users/me/.opik.config
Run the smoke tests for Comet using cometx with:
cometx smoke-test sanitytest
In your terminal you'll see output indicating that the smoke tests are running. Any components of Comet that are present in your install should be tested and those not installed will have their tests skipped.
As part of the output you'll see lines identifying where you can find the test results. eg.
COMET INFO: Experiment is live on comet.com http://<comet-url>/sanitytest/smoke-tests/70e2772b26f8446a8f9c8aedd7997abb
When Opik tests are run you'll see:
Starting Opik sanity test...
Opik URL override: http://<comet-url>/opik/api/
OPIK: Started logging traces to the "smoke-tests" project at http://<comet-url>/opik/sanitytest/redirect/projects?name=smoke-tests.
Opik sanity test completed successfully.
Command-line Verification¶
When run, the smoke-test will do a number of things:
- Creates an experiment with images, confusion matrix (with images), embeddings, metrics, dataset, and assets
- Runs 10 experiments as part of an Optimizer test
- Logs some metrics to MPM (if it is available on installation)
- Uploads two Python Panels to the Panel Gallery in workspace area
The first 3 steps are automatic, but final confirmation is left to you to see if all of the logged results look correct.
Finally, you should add the two panels to the smoke-test project. They are named: /comet-ml/comet-examples/master/panels/OptimizerAnalysis.py /comet-ml/comet-examples/master/panels/SmokeTest.py
The Optimizer Analysis is a Python Panel to view the results of the Optimizer run. The SmokeTest panel will run some tricky tests, add 1 Optimizer Panel, and 2 Smoke Test Panels to your project. Follow the directions on the Smoke Test Panel (as shown below).
You should make sure the Compute Engine is the proper version. This should be an ENV listed in one of the Python Panels. 
Manual Inspection¶
Part of the deployment testing is to make sure that the UI is also working appropriately. Things to check:
- There should be a project named “smoke-tests” (or alternative name that you provided)
There should be an experiment named “test-N” (where N is 1 or greater) (if “experiment” included). The following items appear on this experiment:
- There should be two metrics in the panels: “loss” and “accuracy” with steps 0 to 100 (if “metric” included)
- There should be an image named “smoke-test-image.png” (if “image” included)
- There should be about 500 more images named “confusion-matrix-N.png (if “confusion-matrix” included)
- There should be an “embedding-image-ID.png” (if “embedding” included)
- Output should be logged (if “experiment” included)
There should be a confusion matrix such that if you click on a cell, you should see a number of images of digits (if “confusion-matrix” included):
In “Other” there should be dataset_info: mnist-1.6.8, /opt/share/datasets/mnist.db (if “dataset” included)
In “Assets” there should be “embeddings”. Expanding and selecting “template_projector_config-ID.json” will show a button “Open in Embedding Projector” on the right sidebar. Clicking that will open a new browser tab showing the embeddings with images and colors:
- For the optimizer test, there should be 10 additional experiments. (if "optimizer" included)
- To best examine them, add two panels to the project view:
- Optimizer Analysis (uploaded to workspace panels) (if “panel” included)
- Parallel Coordinate Chart - select “loss” and “x”
- To best examine them, add two panels to the project view:
Panel Tests
- Add the SmokeTest panel to the project view (if “panel” included)
- It will run a series of tests checking out the details of Python Panel installation
MPM (if "mpm" included)
- The mpm tests include two types of items:
- 10 events
- 10 items in training distribution
- These may take a few minutes to show up in MPM under the model “smoke-tests” (or whatever project name you provided)
- The mpm tests include two types of items:
Opik (if "opik" included)
- Once complete click the project “sanitytest” and you should see something like the following:
- Click the Input:
Congratulations! You have successfully smoke tested a properly functioning Comet install.