Optimizer

The Comet Optimizer is used to dynamically find the best set of hyperparameter values that will minimize or maximize a particular metric. It can make suggestions for what hyperparameter values to try next, either in serial or in parallel (or a combination).

Comet's Optimizer has many benefits over traditional hyperparameter optimizer search services because of its integration with Comet's Experiments. In addition, Comet's hyperparameter search has a powerful architecture for customizing your search or sweep. You can easily switch search algorithms, or perform phased searches.

In its simplest form, you can use the hyperparameter search this way:

```python

file: example-1.py

from comet_ml import Optimizer

We only need to specify the algorithm and hyperparameters to use:

config = { # We pick the Bayes algorithm: "algorithm": "bayes",

# Declare your hyperparameters in the Vizier-inspired format:
"parameters": {
    "x": {"type": "integer", "min": 1, "max": 5},
},

# Declare what we will be optimizing, and how:
"spec": {
"metric": "loss",
    "objective": "minimize",
},

}

Next, create an optimizer, passing in the config:

(You can leave out API_KEY if you already set it)

opt = Optimizer(config)

define fit function here!

Finally, get experiments, and train your models:

for experiment in opt.get_experiments( project_name="optimizer-search-01"): # Test the model loss = fit(experiment.get_parameter("x")) experiment.log_metric("loss", loss) ```

That's it! Comet will provide you with an Experiment object already set up with the suggested parameters to try. You merely need to train the model and log the metric to optimize ("loss" in this case).

You can also use the same optimizer in parallel. To use in parallel, you need to take the optimizer config options and save them in a file:

```python

file: example-2.config

{ # We pick the Bayes algorithm: "algorithm": "bayes",

# Declare your hyperparameters in the Vizier-inspired format:
"parameters": {
    "x": {"type": "integer", "min": 1, "max": 5},
},

# Declare what we will be optimizing, and how:
"spec": {
"metric": "loss",
    "objective": "minimize",
},

} ```

Now, we use a slight variation of the code above. We have only moved the optimizer config options out of the script into their own file, and pass in the filename via the sys.argv arguments:

```python

file: example-2.py

from comet_ml import Optimizer import sys

Next, create an optimizer, passing in the config:

(You can leave out API_KEY if you already set it)

opt = Optimizer(sys.argv[1])

define fit function here!

Finally, get experiments, and train your models:

for experiment in opt.get_experiments( project_name="optimizer-search-02"): # Test the model loss = fit(experiment.get_parameter("x")) experiment.log_metric("loss", loss) ```

You can call this file directly, passing in the name of the optimizer config file:

bash $ python example-2.py example-2.config

and it will run exactly as it did above with example-1.py. However, you can also use comet optimize to run the code in parallel:

bash $ comet optimize -j 2 example-2.py example-2.config

where -j indicates the number of processes to run in parallel. This will run as it did before, but running processes in parallel. For more details on the comet optimize options, see below.

We have designed the Optimizer API to be intuitive, but powerful. The above information is perhaps most of what you need. However, the rest of this page documents all of the Optimizer features, options, and settings.

See the Optimizer class for more details on creating an optimizer.

Optimizer Configuration

The Optimizer configuration dictionary (either specified in code, or in a config file) has the following five sections:

  1. "algorithm" - string, which search algorithm to use
  2. "spec" - dictionary, the algorithm-specific specifications
  3. "parameters" - dictionary, the parameter distribution space descriptions
  4. "name" - string, a personalizable name to associate with this search instance (optional)
  5. "trials" - integer, the number of trials per experiment to run (optional, defaults to 1)

The algorithm must be one of the three possible algorithms: "random", "grid", or "bayes".

Each of these entries are described in the next section.

A complete example of an optimizer config:

python {"algorithm": "bayes", "spec": { "maxCombo": 0, "objective": "minimize", "metric": "loss", "minSampleSize": 100, "retryLimit": 20, "retryAssignLimit": 0, }, "parameters": { "hidden-layer-size": {"type": "integer", "min": 5, "max": 100}, "hidden2-layer-size": {"type": "discrete", "values": [16, 32, 64]}, }, "name": "My Bayesian Search", "trials": 1, }

Optimizer Algorithms

There are three different search/sweep algorithms that you can use with the optimizer:

  1. "grid" - Sweep algorithm based on picking parameter values from discrete, possibly sampled, regions
  2. "random" - Random sampling algorithm
  3. "bayes" - Bayesian algorithm based on distributions, balancing exploitation and exploration

Example:

python {"algorithm": "bayes"}

For most searches, we recommend using "bayes". However, for some cases, one of the algorithms may be more appropriate. See below for more details.

Bayes Algorithm

As mentioned, the Bayes algorithm may be the best choice for most of your Optimizer uses. It provides a well-tested algorithm that balances exploring unknown space, with exploiting the best known so far. The Comet Bayes algorithm implements the adaptive Parzen-Rosenblatt estimator.

The "bayes" search algorithm uses the following options in the "spec":

  • "maxCombo"- integer, the limit of parameter combinations to try (default 0, meaning to use 10 times the number of hyperparameters)
  • "objective" - string "minimize" or "maximize", for the objective metric (default "minimize")
  • "metric" - string, the metric name that you are logging and want to minimize/maximize (default "loss")
  • "minSampleSize" - integer, the number of samples to help find appropriate grid ranges (default 100)
  • "retryLimit" - integer, the limit to try creating a unique parameter set before giving up (default 20)
  • "retryAssignLimit" - integer, the limit to re-assign non-completed experiments (default 0)

The Comet optimizer will never assign the same set of parameters twice, unless running multiple trials of an experiment, or reassigning a non-completed experiment (see below). Depending on the algorithm you have chosen, the number of possible experiments to run may be finite, or infinite. For example, the "bayes" algorithm always samples from continuous parameter distributions, but the "grid" algorithm always breaks distributions into grids. However, some parameter types (including "categorical" and "discrete") there are only a finite number of options. To see the computed value of maximum number of possible experiments for an algorithm, see Optimizer.status().

In either the finite or infinite case, to limit the number of experiment combinations to run set maxCombo to a non-zero value. Notice that this is the "maximum combinations" not the total maximum number of experiments to assign. For example, the total number of experiments assigned is maxCombo * trials. But this can also be effected by the SPEC setting "retryAssignLimit" which will re-assign experiments until they are "completed" or the "retryAssignLimit" value is met. Therefore, if each experiment never completes, you would assign maxCombo * trials * (retryAssignLimit + 1) number of experiments.

Example:

python {"algorithm": "bayes", "spec": { "maxCombo": 0, "objective": "minimize", "metric": "loss", "minSampleSize": 100, "retryLimit": 20, "retryAssignLimit": 0, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

Grid Algorithm

The "grid" algorithm is useful for performing a wide, initial search of a set of parameter values. Comet's grid algorithm is slightly more flexible than many, as each time you run it, you will sample from the set of possible grids defined by the parameter space distribution. The "grid" algorithm does not use past experiments to inform future experiments: it merely collects the objective metric for you to explore.

The "grid" search algorithm uses the following options:

  • "randomize" - boolean, if True, then the grid is traversed randomly; otherwise traversed in order (default False)
  • "maxCombo"- integer, the limit of parameter combinations to try (default 0, meaning to use 10 times the number of hyperparameters)
  • "metric" - string, the metric name that you are logging and want to minimize/maximize (default "loss")
  • "gridSize" - integer, when creating a grid, the number of bins per parameter (default 10)
  • "minSampleSize" - integer, the number of samples to help find appropriate grid ranges (default 100)
  • "retryLimit" - integer, the limit to try creating a unique parameter set before giving up (default" 20)
  • "retryAssignLimit" - integer, the limit to re-assign non-completed experiments (default 0)

Example:

python {"algorithm": "grid", "spec": { "randomize": True, "maxCombo": 0, "metric": "loss", "gridSize": 10, "minSampleSize": 100, "retryLimit": 20, "retryAssignLimit": 0, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

Random Algorithm

The "random" algorithm is slightly more flexible than the "grid" algorithm, in that it will continue to sample from the set of possible parameter values, until you stop the search, or have the "max combinations" value set. The "random" algorithm, like the "grid" algorithm, does not use past experiment metrics to inform future experiments.

The "random" search algorithm uses the following options:

  • "maxCombo"- integer, the limit of parameter combinations to try (default 0, meaning to use 10 times the number of hyperparameters)
  • "metric" - string, the metric name that you are logging and want to minimize/maximize (default "loss")
  • "gridSize" - integer, when creating a grid, the number of bins per parameter (default 10)
  • "minSampleSize" - integer, the number of samples to help find appropriate grid ranges (default 100)
  • "retryLimit" - integer, the limit to try creating a unique parameter set before giving up (default" 20)
  • "retryAssignLimit" - integer, the limit to re-assign non-completed experiments (default 0)

Example:

python {"algorithm": "random", "spec": { "maxCombo": 100, "metric": "loss", "gridSize": 10, "minSampleSize": 100, "retryLimit": 20, "retryAssignLimit": 0, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

Specifying Optimizer Parameters

There are four kinds of parameters: "integer", "double" or "float", "discrete" (for a list of numbers), and "categorical" (for a list of strings).

The format of each parameter was inspired by Google's Vizier, and exemplified by the open source version called Advisor.

Integers

Integers can be distributed in one of five ways: "linear", "uniform", "normal", "loguniform", or "lognormal".

python {"PARAMETER-NAME": {"type": "integer", "scalingType": "linear" | "uniform" | "normal" | "loguniform" | "lognormal", "min": INTEGER, "max": INTEGER, }, ... }

See below for more details on "scalingType" for each algorithm.

NOTE: "integer" type with "linear" scalingType when using the "bayes" algorithm indicates an independent distribution. This is useful for using integer values that have no relationship with one another, such as seed values. If your distribution is meaningful (e.g., 2 is closer to 1 than it is to 6) then you should use the "uniform" scalingType.

You can also provide a list of integers (or any numbers) using the "discrete" type:

python {"PARAMETER-NAME": {"type": "discrete", "values": [NUMBER, ...], }, ... }

Example:

python {"algorithm": "bayes", "spec": {...}, "parameters": { "hidden-layer-size": {"type": "integer", "min": 5, "max": 100}, "hidden2-layer-size": {"type": "discrete", "values": [16, 32, 64]}, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

Double/Float

Doubles (also called floats) can be distributed in one of five ways: "linear", "uniform", "normal", "loguniform", or "lognormal".

python {"PARAMETER-NAME": {"type": "double" |"float", "scalingType": "linear" | "uniform" | "normal" | "loguniform" | "lognormal", "min": FLOAT, "max": FLOAT, }, ... }

See below for more details on "scalingType" for each algorithm.

You can also provide a list of doubles/floats (or any numbers) using the "discrete" type:

python {"PARAMETER-NAME": {"type": "discrete", "values": [NUMBER, ...], }, ... }

Example:

python {"algorithm": "bayes", "spec": {...}, "parameters": { "momentum": {"type": "double", "min": 0.0, "max": 0.9}, "learning-rate": {"type": "discrete", "values": [0.01, 0.1, 0.8]}, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

Scaling Types

Both "integer" and "double"/"float" allow the following scaling types (indicated with "scalingType"):

  • "linear" - for integers, means an independent distribution (used for things like seed values); for double, the same as uniform
  • "uniform" - a uniform distribution between "min" and "max"
  • "normal" - a normal distribution centered around "mu" with standard deviation of "sigma"
  • "lognormal" - a log-normal distribution centered around "mu" with standard deviation of "sigma"
  • "loguniform" - a log-uniform distribution between "min" and "max". Computes exp(uniform(log(min), log(max)))

Examples:

python {"algorithm": "bayes", "spec": {...}, "parameters": { "momentum": {"type": "float", "min": 0.0, "max": 0.9, "scalingType": "uniform"}, "learning-rate": {"type": "float", "min": 0.001, "max": 1.0, "scalingType": "loguniform"}, "dropout": {"type": "float", "mu": 0.5, "sigma": 0.1, "scalingType": "normal"}, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

Categorical

Categorical, like "discrete", is a type for a list of values. In this case, the values must be strings.

python {"PARAMETER-NAME": {"type": "categorical", "values": ["LIST", "OF", "STRINGS"], }, ... }

Example:

python {"algorithm": "bayes", "spec": {...}, "parameters": { "activation": {"type": "categorical", "values": ["sigmoid", "relu", "leaky-relu"]}, }, "trials": 1, "parameters": {...}, "name": "My Optimizer Name", }

Additional Optimizer Parameter Settings

For each parameter, you may also specify "gridSize".

Note: gridSize is only used in the grid and random algorithms.

Grid Size

Each parameter is considered a distribution for those algorithms that sample randomly. Those algorithms include "bayes" and "random". However, other algorithms need to know a resolution size for how to divide up the parameter space into discrete bins. Those algorithms include "grid". For those, an additional entry named "gridSize" can be set for each parameter. For example, the following defines a parameter "x" ranging between 0.0 and 100.0, and broken up into 25 bins for the grid search.

Example:

python {"x": {"type": "double", "scalingType": "uniform", "min": 0.0, "max": 100.0, "gridSize": 25, }, }

NOTE: the bins won't be exactly 0-25, 25-50, 50-75, and 75-100. Rather, the divisions are created based on sampled values.

Comet Optimize

comet is a command-line utility that is installed with comet_ml. optimize is one of the commands that comet can use. The format is:

bash $ comet optimize [options] [PYTHON_SCRIPT] OPTIMIZER

For more information on comet optimize please see Command-Line Utilities

End-to-end Example

Now, let's see a complete end-to-end program using Keras with the Comet Optimizer.

```python import comet_ml import logging

logging.basicConfig(level=logging.INFO) LOGGER = logging.getLogger("comet_ml")

from tensorflow.keras.datasets import mnist from tensorflow.keras.layers import Dense from tensorflow.keras.models import Sequential from tensorflow.keras.optimizers import RMSprop from tensorflow.keras.utils import to_categorical

def build_model_graph(experiment): model = Sequential() model.add( Dense( experiment.get_parameter("first_layer_units"), activation="sigmoid", input_shape=(784,), ) ) model.add(Dense(128, activation="sigmoid")) model.add(Dense(128, activation="sigmoid")) model.add(Dense(10, activation="softmax")) model.compile( loss="categorical_crossentropy", optimizer=RMSprop(), metrics=["accuracy"], ) return model

def train(experiment, model, x_train, y_train, x_test, y_test): model.fit( x_train, y_train, batch_size=experiment.get_parameter("batch_size"), epochs=experiment.get_parameter("epochs"), validation_data=(x_test, y_test), )

def evaluate(experiment, model, x_test, y_test): score = model.evaluate(x_test, y_test, verbose=0) LOGGER.info("Score %s", score)

def get_dataset(): num_classes = 10

# the data, shuffled and split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.reshape(60000, 784)
x_test = x_test.reshape(10000, 784)
x_train = x_train.astype("float32")
x_test = x_test.astype("float32")
x_train /= 255
x_test /= 255
print(x_train.shape[0], "train samples")
print(x_test.shape[0], "test samples")

# convert class vectors to binary class matrices
y_train = to_categorical(y_train, num_classes)
y_test = to_categorical(y_test, num_classes)

return x_train, y_train, x_test, y_test

Get the dataset:

x_train, y_train, x_test, y_test = get_dataset()

The optimization config:

config = { "algorithm": "bayes", "name": "Optimize MNIST Network", "spec": {"maxCombo": 10, "objective": "minimize", "metric": "loss"}, "parameters": { "first_layer_units": { "type": "integer", "mu": 500, "sigma": 50, "scalingType": "normal", }, "batch_size": {"type": "discrete", "values": [64, 128, 256]}, }, "trials": 1, }

opt = comet_ml.Optimizer(config)

for experiment in opt.get_experiments(project_name="my_project"): # Log parameters, or others: experiment.log_parameter("epochs", 10)

# Build the model:
model = build_model_graph(experiment)

# Train it:
train(experiment, model, x_train, y_train, x_test, y_test)

# How well did it do?
evaluate(experiment, model, x_test, y_test)

# Optionally, end the experiment:
experiment.end()

```

Troubleshooting

Continue from Crashed/Paused Optimizer

If you pause your search, or if your optimizer script would ever crash you can recover your search and pick up immediately from where you left off. You need only define the COMET_OPTIMIZER_ID in the environment, and run your script again. The COMET_OPTIMIZER_ID is printed in the terminal at the start of each sweep. It also is logged with each experiment in the Other tab. Here is an example or a script crashing, and continuing with the search:

```bash $ python script.py

COMET INFO: COMET_OPTIMIZER_ID=366dcb4f38bf42aea6d2d87cd9601a60 ... it crashes for some reason

$ edit script.py

$ export COMET_OPTIMIZER_ID=366dcb4f38bf42aea6d2d87cd9601a60

$ python script.py COMET INFO: COMET_OPTIMIZER_ID=366dcb4f38bf42aea6d2d87cd9601a60 ```

You can also supply the optimizer id to the Optimizer class rather than the name of the filename containing the optimizer config. For example, consider again example-2.py from above:

```python

file: example-2.py

from comet_ml import Optimizer import sys

Next, create an optimizer, passing in the config:

(You can leave out API_KEY if you already set it)

opt = Optimizer(sys.argv[1])

define fit function here!

Finally, get experiments, and train your models:

for experiment in opt.get_experiments( project_name="optimizer-search-03"): # Test the model loss = fit(experiment.get_parameter("x")) experiment.log_metric("loss", loss) ```

Recall that you can start that program up, like:

bash $ python example-2.py example-2.config

or using comet optimize:

bash $ comet optimize -j 2 example-2.py example-2.config

To use your same script and start up where you left off, you only need the Comet Optimizer id. When you start up a new optimizer, you will see a line displayed similar to:

COMET INFO: COMET_OPTIMIZER_ID=303faefd8194400694ec9588bda8338d

You can set this Comet environment variable in the terminal, and your search will use the existing Optimizer rather than creating a new one.

bash $ export COMET_OPTIMIZER_ID=303faefd8194400694ec9588bda8338d $ python example-2.py example-2.config

or

bash $ export COMET_OPTIMIZER_ID=303faefd8194400694ec9588bda8338d $ comet optimize -j 2 example-2.py example-2.config

You can also just pass the Optimizer id on the command line instead of the filename if you have written your script in the style of example-2.py:

bash $ python example-2.py 303faefd8194400694ec9588bda8338d

or

bash $ comet optimize -j 2 example-2.py 303faefd8194400694ec9588bda8338d

You can also have comet optimize pass along arguments to your script. Simply add those after the config following two dashes, like this:

bash $ comet optimize -j 4 script.py opt.config -- --project-name "test-007"

Then you can use the argparse module, like so:

```python

example-3.py

from comet_ml import Optimizer, Experiment

import argparse

parser = argparse.ArgumentParser()

Add your own args here:

parser.add_argument("--project-name", default=None)

These passed on from "comet optimize":

parser.add_argument("optimizer", default="test1_optimizer.json") parser.add_argument("--trials", "-t", type=int, default=None)

parsed = parser.parse_args()

count = 0 for experiment in opt.get_experiments(): loss = train(experiment.params["x"]) msg = experiment.log_metric("loss", loss) count += 1 print("Optimizer job done! Completed %s experiments." % count) ```

The above program can then be used alone, or with the comet optimize to run scripts in parallel with custom command-line arguments. Called normally:

bash $ python example-3.py opt.config --project-name "my-project-01"

or in parallel:

bash $ comet optimize example-3.py opt.config -- --project-name "my-project-01"

What if an experiment doesn't finish?

By default, all of the algorithms will not release duplicate sets of parameters (except when trials is greater than 1). But what should you do if an experiment crashes and never notifies the Optimizer?

You have two choices:

  1. You can run the Optimizer search with the "retryAssignLimit" spec settings:

python {"algorithm": "bayes", "spec": { "retryAssignLimit": 1, ... }, "parameters": {...}, "name": "My Bayesian Search", "trials": 1, }

Using a retryAssignLimit value greater than zero will continue to assign the parameter set until an experiment marks it as "completed" or the number of retries is equal to the retryAssignLimit.

  1. You can run the Optimizer search/sweep again. You can either run all of the parameter value combinations again, or a subset thereof.