Warnings and Errors

For further assistance on any of these Python warnings or errors, or if you see an error message that is not noted here, please ping us on our slack channel

Error

ImportError: Please import comet before importing these modules: ...

This error occurs when you try to create an Experiment (or another kind of experiment, such as OfflineExperiment) but have imported comet_ml after one of the supported auto-logging machine learning libraries (such as torch, fastai, keras, or tensorflow). You have two choices to resolve this error: you can either move comet_ml to be imported first (recommended), or you can completely disable Comet's auto logging facility by setting COMET_DISABLE_AUTO_LOGGING=1 in the environment, or in your Comet config file. See Quick Start for more details.

Error

COMET ERROR: Run will not be logged

This error is shown with a Python stack trace and indicates that the initial handshake between Comet and the server failed. This is usually a local networking issue or production downtime. Please reach out on our slack channel if you encounter this error.

Error

COMET ERROR: Failed to set run source code

Comet failed to read the source code file for this experiments. This could be due to using a dispatcher.

Error

COMET ERROR: There's seem to be an issue with your system's SSL certificate bundle. This is likely a system wide issue that is not related to Comet.

First, you may want to check to see if you can access Comet in general. You can try this curl command from your terminal:

bash curl -i https://www.comet.ml/ws/logger-ws

You should get back something like the following:

``` HTTP/2 404 date: Mon, 13 Apr 2020 12:37:44 GMT content-type: application/json content-length: 54 set-cookie: AWSALB=...; Expires=Mon, 20 Apr 2020 12:37:44 GMT; Path=/ set-cookie: AWSALBCORS=...; Expires=Mon, 20 Apr 2020 12:37:44 GMT; Path=/; SameSite=None; Secure server: nginx

{"msg":null,"code":404,"data":null,"sdk_error_code":0} ```

If you did get something similar, then that would indicate that you have a Python-releated issue rather than an OS-releated issues.

If you were not able to get the above, then it is an OS-releated issue. If you are on a Mac computer, these links may be helpful:

  • https://timonweb.com/tutorials/fixing-certificate_verify_failed-error-when-trying-requests_html-out-on-mac/
  • https://stackoverflow.com/questions/40684543/how-to-make-python-use-ca-certificates-from-mac-os-truststore

If it is a Python-releated issue, it could be a bad Python websocket library:

  • https://github.com/websocket-client/websocket-client/issues/451

One solution is to install a specific version of the websocket-client library:

bash pip install websocket-client==0.47.0

Otherwise, you should contact your local system administrator as it is probably an OS configuration problem.

Rate limits

The Comet API may rate limit submission of requests for your experiments. Such limits are managed as an allowed number of operations per time window, where an operation might be read or an update. In that case, a warning symbol will appear on your experiment list.

Online Experiments

For online experiments, the following rate limits are in effect:

  • logging metrics: 10,000 per minute
  • logging parameters: 8,000 per minute
  • logging output: 10,000 per minute
  • logging everything else: 8,000 per minute

Offline Experiments

For offline experiments, the following rate limits are in effect:

  • logging metrics: 80,000 per minute
  • logging parameters: 80,000 per minute
  • logging output: 80,000 per minute
  • logging everything else: 80,000 per minute

Info

Offline experiments have a rate limit because you may be uploading multiple experiments in parallel, or attempting to upload too many too quickly.

Size and Count limits

For all experiments, you can log 15,000 total values for each metric, per experiment. If your metric count goes beyond this limit, then the values are downsampled.

REST API

When using the REST API (from the Python SDK, or using the URL endpoints directly), the following limits are in effect per experiment.

Info

Each of the following REST API items typically only need to be logged once per experiment.

  • 10 environment detail updates
  • 10 git metadata updates
  • 10 graph (model) updates
  • 10 OS packages updates
  • 10 code updates

In addition when using the REST API, the following limits are in effect:

  • 1,000 HTML updates per experiment
  • each API key is allowed to make 15,000 submissions per hour

Solutions to rate limits

If you notice you are hitting rate limits based on normal experiments, please try reporting on each epoch, rather than each step.

If you still encounter rate limits, consider using the OfflineExperiment interface. This only requires that you change:

python experiment = Experiment(...)

to

python experiment = OfflineExperiment(..., offline_directory="/path/to/save/experiments")

After the experiment is completed, you then run:

bash comet upload /path/to/save/experiments/*.zip

in order to send your experiment to comet.ml.

For more information on how to best handle rate limits, please reach out to support@comet.ml or via our Slack channel.