Secret Management¶

When running distributed training runs or using orchestration tools like Vertex Pipelines or Metaflow, it can be challenging to pass the Comet API key securily to all workers and steps. Comet's integrations with Secret Managers allow you to securily store the Comet API key and the Comet Python SDK will access it when required.

Using AWS Secret Manager¶

Comet supports storing and retrieving your API Key using AWS Secret Manager. This approach provides an additional layer of security and reduces the risk of unauthorized access to sensitive information.

Using this method, you can easily distribute your code and run distributed training across multiple servers or Docker containers without having to worry about sharing your Comet API Key.

Prerequisites¶

To use this feature, you must have the following:

• The boto3 Python library installed.
• Configured credentials with the appropriate permissions to create and access secrets in the Secret Manager.

Storing API Key in AWS Secret Manager¶

Before you can retrieve the API Key from the AWS Secret Manager, you need to store it there first. To do this, use the store_api_key_in_secret_manager function. This function stores the API Key as a secret in the AWS Secret Manager and returns an API Key Secret that represents the location of the secret.

import comet_ml
from comet_ml.secret.aws import store_api_key_in_secret_manager

api_key_secret, secret_version = store_api_key_in_secret_manager(
    API_KEY, "username"
)

# Either use the API Key Secret directly
experiment = comet_ml.start(api_key=api_key_secret)

# Or use the secret_version later to retrieve the secret value
api_key_secret = get_api_key_from_secret_manager(
    "username", secret_version=secret_version
)
experiment = comet_ml.start(api_key=api_key_secret)

The function returns a tuple containing the API Key Secret as a string and the version of the secret that was created.

See the reference doc for more details.

Retrieving API Key Secret¶

To retrieve the API Key Secret use the get_api_key_secret_from_secret_manager function. This function returns an API Key Secret string representing the location of the secret in the Secret Manager where the API Key is stored.

import comet_ml
from comet_ml.secret.aws import get_api_key_from_secret_manager

api_key_secret = get_api_key_from_secret_manager(
    secret_name="username", secret_version=AWS_SECRET_VERSION
)

experiment = comet_ml.start(api_key=api_key_secret)

See the reference doc for more details.

Use API Key Secret¶

When you create a Comet Experiment object, you can pass your API Key Secret as a parameter instead of your clear-text API Key. The Experiment object will then retrieve the API Key from GCP Secret Manager and use it to authenticate with the Comet platform. For example:

import comet_ml
from comet_ml.secret.aws import get_api_key_from_secret_manager

# Retrieve your API Key Secret from GCP Secret Manager
api_key_secret = get_api_key_from_secret_manager(
   "comet-api-key", "latest"
)

# Create an Experiment object using your API Key Secret
comet_experiment = comet_ml.start(
    api_key=api_key_secret, project_name="my-project"
)

The API Key Secret can also be set as an environment variable or in a file, see the Configuration page for more details.

The API Key Secret can be used everywhere where an API Key is expected, an Experiment, ExistingExperiment, API or APIExperiment.

Using GCP Secret Manager¶

Comet supports storing and retrieving your API Key using GCP Secret Manager. This approach provides an additional layer of security and reduces the risk of unauthorized access to sensitive information.

Using this method, you can easily distribute your code and run distributed training across multiple servers or Docker containers without having to worry about sharing your Comet API Key.

Prerequisites¶

To use this feature, you must have the following:

• Access to a GCP project with Secret Manager API enabled.
• The google-cloud-secret-manager Python library installed in a version greater or equal to 2.0.0.
• Configured credentials with the appropriate permissions to create and access secrets in the Secret Manager.

Storing API Key in GCP Secret Manager¶

Before you can retrieve the API Key from the GCP Secret Manager, you need to store it there first. To do this, use the store_api_key_in_secret_manager function. This function stores the API Key as a secret in the GCP Secret Manager and returns an API Key Secret that represents the location of the secret.

import comet_ml
from comet_ml.secret.gcp import store_api_key_in_secret_manager

api_key_secret, secret_version = store_api_key_in_secret_manager(
    API_KEY, GCP_PROJECT_ID, secret_id="username"
)

# Either use the API Key Secret directly
experiment = comet_ml.start(api_key=api_key_secret)

# Or use the secret_version later to retrieve the secret value
api_key_secret = get_api_key_from_secret_manager(
    GCP_PROJECT_ID, secret_id="username", secret_version=secret_version
)
experiment = comet_ml.start(api_key=api_key_secret)

The function returns a tuple containing the API Key Secret as a string and the version of the secret that was created.

See the reference doc for more details.

Retrieving API Key Secret¶

To retrieve the API Key Secret use the get_api_key_secret_from_secret_manager function. This function returns an API Key Secret string representing the location of the secret in the Secret Manager where the API Key is stored.

import comet_ml
from comet_ml.secret.gcp import get_api_key_from_secret_manager

api_key_secret = get_api_key_from_secret_manager(
    GCP_PROJECT_ID, secret_id="username", secret_version=GCP_SECRET_VERSION
)

experiment = comet_ml.start(api_key=api_key_secret)

See the reference doc for more details.

Use API Key Secret¶

When you create a Comet Experiment object, you can pass your API Key Secret as a parameter instead of your clear-text API Key. The Experiment object will then retrieve the API Key from GCP Secret Manager and use it to authenticate with the Comet platform. For example:

import comet_ml
from comet_ml.secret.gcp import get_api_key_from_secret_manager

# Retrieve your API Key Secret from GCP Secret Manager
api_key_secret = get_api_key_from_secret_manager(
    "my-gcp-project", "comet-api-key", "latest"
)

# Create an Experiment object using your API Key Secret
comet_experiment = comet_ml.start(
    api_key=api_key_secret, project_name="my-project"
)

The API Key Secret can also be set as an environment variable or in a file, see the Configuration page for more details.

The API Key Secret can be used everywhere where an API Key is expected, an Experiment, ExistingExperiment, API or APIExperiment.