Troubleshooting and FAQ¶

This page presents possible warnings and errors that you might encounter and the steps to take to address them. There are also some tips on debugging.

For further assistance on any of these Python warnings or errors, or if you see an error message that is not noted here, ping us on our Slack channel.

Common errors¶

ImportError: Please import comet before importing these modules: ...¶

This error occurs when you try to create an Experiment (or another kind of experiment, such as OfflineExperiment) but have imported comet_ml after one of the supported machine learning libraries (such as Torch, fastai, Keras, or TensorFlow). You have two choices to resolve this error: you can either move comet_ml to be imported first (this is the recommended method), or you can completely disable Comet's auto logging facility by setting COMET_DISABLE_AUTO_LOGGING=1 in the environment, or in your Comet config file.

COMET ERROR: Run will not be logged¶

This error is shown with a Python stack trace and indicates that the initial handshake between Comet and the server failed. This is usually a local networking issue or production downtime. Reach out on our Slack channel if you encounter this error.

COMET ERROR: Failed to set run source code¶

Comet failed to read the source code file for this Experiment. This could happen in rare cases where a library wraps your code or where Comet cannot read the source file.

COMET ERROR: There's seem to be an issue with your system's SSL certificate bundle. This is likely a system wide issue that is not related to Comet.¶

First, check to see if you can access Comet in general. Issue this curl command from your terminal:

curl -i https://www.comet.com/clientlib/isAlive/ping

You should get back something like this:

HTTP/2 200
date: Tue, 12 Jul 2022 07:58:41 GMT
content-type: application/json
content-length: 66
set-cookie: AWSALB=...; Expires=Tue, 19 Jul 2022 07:58:41 GMT; Path=/
set-cookie: AWSALBCORS=...; Expires=Tue, 19 Jul 2022 07:58:41 GMT; Path=/; SameSite=None; Secure
server: nginx
comet-ver: 97a6a5e8db8b665f95a543b5a0bc383531e09b69
comet-app-server: backend-python-3.production.comet-ml.internal
access-control-expose-headers: Comet-Ver, Comet-App-Server
vary: Accept-Encoding

{"msg":"Healthy Server","code":200,"data":null,"sdk_error_code":0}

If you did get something similar, then that would indicate that you have a Python-related issue, rather than an OS-related issues.

If you did not see the above output, then it is an OS-related issue. If you are on a Mac computer, these links might provide some solutions:

• https://timonweb.com/tutorials/fixing-certificate_verify_failed-error-when-trying-requests_html-out-on-mac/.
• https://stackoverflow.com/questions/40684543/how-to-make-python-use-ca-certificates-from-mac-os-truststore.

Otherwise, you should contact your local system administrator as you are probably experiencing a problem related to OS configuration.

Debugging¶

Get debugging information¶

You can set the configuration variable COMET_LOGGING_CONSOLE to info to see tracebacks for any Comet-based, either as one-time run only variables or as (session) Bash environment variables.

COMET_LOGGING_CONSOLE=info python script.py

export COMET_LOGGING_CONSOLE=info
python script.py

This procedure often yields enough information to help track down a problem (for example, the reason why an image is not logged).

If you need the maximum amount of debug information, you can create a Comet debug log file by setting the two configuration variables COMET_LOGGING_FILE and COMET_LOGGING_FILE_LEVEL.

COMET_LOGGING_FILE_LEVEL=debug COMET_LOGGING_FILE=/tmp/comet.log python script.py

export COMET_LOGGING_FILE=/tmp/comet.log
export COMET_LOGGING_FILE_LEVEL=debug

import os
os.environ["COMET_LOGGING_FILE"] = "/tmp/comet.log"
os.environ["COMET_LOGGING_FILE_LEVEL"] = "debug"
...

1
2
3

[comet_logging]
file = /tmp/comet.log
file_level = debug

Logs saved inside the COMET_LOGGING_FILE file show details on all of the steps of your experiment, including any details about failures.

If you still have problems, please share this file with us using the Slack channel.

Ensure comet_ml is up-to-date¶

You can find the latest version number for your comet_ml package on its official PIP page.

To upgrade, use the command:

$ pip install comet_ml --upgrade

In some cases, you may also want to update all of the packages that comet_ml depends on. You can do so by running:

$ pip install comet_ml --upgrade --upgrade-strategy eager