Vertex AI partnership

Comet integrates seamlessly with Vertex allowing you to track not just individual traning runs but also Vertex Pipelines within the Comet UI.

Visualizing Vertex AI Pipelines

The Vertex AI Pipelines integration allows you to track both individual tasks and the state of the pipeline as a whole. The state of the pipeline can be visualized using the Vertex AI Pipeline Panel available in the featured tab.

vertex-ai-pipelines-integration.png

You can re-create the view above by:

  1. Grouping experiments by vertex_run_name
  2. Adding the Vertex AI Pipelines panel available in the featured tab
  3. Saving the view as Vertex dashboard

The Vertex AI Pipelines panel can be used to either visualize the latest state of the DAG or the static graph similar to what is available in the Vertex AI Pipelines UI. In addition all tasks that have the Comet logo are also tracked in Comet and can be accessed by clicking on the task.

Integration Overview

The Vertex AI Pipelines integration relies on two steps:

  1. Adding a Comet logger component to track the state of the pipeline
  2. Adding Comet to each individual tasks that you would like track as a Comet experiment

Logging the state of a pipeline

The Vertex AI Pipelines integration relies on a component that runs in parallel to the rest of the pipeline. The comet_logger_component logs the state of the pipeline as whole and reports it back to the Comet UI allowing you to track the pipeline progress as a whole. This component can be used like this:

```python import comet_ml.integration.vertex import kfp.dsl as dsl

@dsl.pipeline(name='ML training pipeline') def ml_training_pipeline(): # Add the Comet logger component comet_ml.integration.vertex.comet_logger_component( api_key="<>", workspace="<>", project="<>")

# Rest of the training pipeline

```

The Comet logger component can also be configured using environment variables, in which case you will not need to specify the api_key, workspace or project arguments.

Logging the state of each task

In addition to the component, you will need to initialize the Vertex task logger within each task of the pipeline that you wish to run. This can be done by creating an experiment within each task and using the initialize_task_logger function:

```python @kfp.dsl.v2.component def my_component() -> None: import comet_ml.integration.vertex

experiment = comet_ml.Experiment()
pipeline_run_name = "{{$.pipeline_job_name}}"
pipeline_task_name = "{{$.pipeline_task_name}}"
pipeline_task_id = "{{$.pipeline_task_uuid}}"

comet_ml.integration.vertex.initialize_comet_logger(experiment, pipeline_run_name, pipeline_task_name, pipeline_task_id)

# Rest of the task code

```

The strings "{{$.pipeline_job_name}}", "{{$.pipeline_task_name}}", "{{$.pipeline_task_uuid}}" are placeholders that are replaced at runtime by Vertex AI, you do not need to update them before running the pipeline.