Rate limits and performance¶

The Comet API might rate limit submission of requests for your experiments. Such limits are managed as an allowed-number-of-operations-per-time window, where an operation might be a read or an update.

Breaching these rate limits will cause your experiment to be throttled. In such cases:

• A notice is displayed.
• A warning symbol appears for the affected experiment.

Metric sampling¶

For all experiments, you can log 15,000 total values for each metric, per experiment. If your metric count goes beyond this limit, then the values will be downsampled.

Rate limits¶

In order to avoid a single user overloading the Comet platform, we have the following rate limits in place:

• Logging metrics: 12,000 per minute
• Logging parameters: 10,000 per minute
• Logging output: 12,000 per minute
• Logging everything else: 8,000 per minute

There are no rate limits for Offline Experiments, only hard limits are in place for these types of Experiments.

Hard limits¶

In addition to rate limits, we also have some hard limits that avoid users logging too much data to a single experiment or project. Each of these hard limits:

• Project level limits:
  • Experiments: Maximum of 25,000 experiments per Experiment Management project
  • Prompts: Maximum of 250,000 prompts per LLM project.
• Experiment level limits:
  • Number of unique metrics: Maximum of 50,000 unique metrics
  • Number of unique parameters: Maximum of 50,000 unique parameters
  • Number of images: Maximum of 100,000 image
  • Number of curves: Maximum of 1000 curves
  • Number of models: Maximum of 100 models
  • Number of text samples: Maxmimum of 500 text samples
  • Number of videos: Maximum of 5000 videos

REST API¶

When using the Rest API, either from the Python SDK, or by using the URL endpoints directly, the following limits are in effect:

• Each API key is allowed to make 15,000 submissions per hour.

Solutions to rate limits¶

If you notice you are hitting rate limits based on normal experiments, try reporting on each epoch, rather than each step.

If you still encounter rate limits, consider using the OfflineExperiment interface. This only requires that you change:

1

experiment = comet_ml.start(online=True)

to

1
2
3
4
5
6

experiment = comet_ml.start(
    online=False,
    experiment_config=comet_ml.ExperimentConfig(
        offline_directory="/path/to/save/experiments"
    ),
)

After the experiment is complete, you then run:

comet upload /path/to/save/experiments/*.zip

to send your experiment to Comet.

For more information on how best to handle rate limits, reach out to support@comet.com or chat to us using our Slack channel.