Optimizer¶
The Comet Optimizer
is used to dynamically find the best set of
hyperparameter values that will minimize or maximize a particular
metric. It can make suggestions for what hyperparameter values to try
next, either in serial or in parallel (or a combination).
Comet's Optimizer has many benefits over traditional hyperparameter optimizer search services because of its integration with Comet's Experiments. In addition, Comet's hyperparameter search has a powerful architecture for customizing your search or sweep. You can easily switch search algorithms, or perform phased searches.
In its simplest form, you can use the hyperparameter search this way:
```python
file: example-1.py¶
from comet_ml import Optimizer
We only need to specify the algorithm and hyperparameters to use:¶
config = { # We pick the Bayes algorithm: "algorithm": "bayes",
# Declare your hyperparameters in the Vizier-inspired format:
"parameters": {
"x": {"type": "integer", "min": 1, "max": 5},
},
# Declare what we will be optimizing, and how:
"spec": {
"metric": "loss",
"objective": "minimize",
},
}
Next, create an optimizer, passing in the config:¶
(You can leave out API_KEY if you already set it)¶
opt = Optimizer(config)
define fit function here!¶
Finally, get experiments, and train your models:¶
for experiment in opt.get_experiments( project_name="optimizer-search-01"): # Test the model loss = fit(experiment.get_parameter("x")) experiment.log_metric("loss", loss) ```
That's it! Comet will provide you with an Experiment object already set up with the suggested parameters to try. You merely need to train the model and log the metric to optimize ("loss" in this case).
You can also use the same optimizer in parallel. To use in parallel, you need to take the optimizer config options and save them in a file:
```python
file: example-2.config¶
{ # We pick the Bayes algorithm: "algorithm": "bayes",
# Declare your hyperparameters in the Vizier-inspired format:
"parameters": {
"x": {"type": "integer", "min": 1, "max": 5},
},
# Declare what we will be optimizing, and how:
"spec": {
"metric": "loss",
"objective": "minimize",
},
} ```
Now, we use a slight variation of the code above. We have only moved
the optimizer config options out of the script into their own file,
and pass in the filename via the sys.argv
arguments:
```python
file: example-2.py¶
from comet_ml import Optimizer import sys
Next, create an optimizer, passing in the config:¶
(You can leave out API_KEY if you already set it)¶
opt = Optimizer(sys.argv[1])
define fit function here!¶
Finally, get experiments, and train your models:¶
for experiment in opt.get_experiments( project_name="optimizer-search-02"): # Test the model loss = fit(experiment.get_parameter("x")) experiment.log_metric("loss", loss) ```
You can call this file directly, passing in the name of the optimizer config file:
bash
$ python example-2.py example-2.config
and it will run exactly as it did above with example-1.py
. However,
you can also use comet optimize
to run the code in parallel:
bash
$ comet optimize -j 2 example-2.py example-2.config
where -j
indicates the number of processes to run in parallel. This
will run as it did before, but running processes in parallel. For more
details on the comet optimize
options, see below.
We have designed the Optimizer API to be intuitive, but powerful. The above information is perhaps most of what you need. However, the rest of this page documents all of the Optimizer features, options, and settings.
See the Optimizer class for more details on creating an optimizer.
Optimizer Configuration¶
The Optimizer configuration dictionary (either specified in code, or in a config file) has the following five sections:
- "algorithm" - string, which search algorithm to use
- "spec" - dictionary, the algorithm-specific specifications
- "parameters" - dictionary, the parameter distribution space descriptions
- "name" - string, a personalizable name to associate with this search instance (optional)
- "trials" - integer, the number of trials per experiment to run (optional, defaults to 1)
The algorithm must be one of the three possible algorithms: "random", "grid", or "bayes".
Each of these entries are described in the next section.
A complete example of an optimizer config:
python
{"algorithm": "bayes",
"spec": {
"maxCombo": 0,
"objective": "minimize",
"metric": "loss",
"minSampleSize": 100,
"retryLimit": 20,
"retryAssignLimit": 0,
},
"parameters": {
"hidden-layer-size": {"type": "integer", "min": 5, "max": 100},
"hidden2-layer-size": {"type": "discrete", "values": [16, 32, 64]},
},
"name": "My Bayesian Search",
"trials": 1,
}
Optimizer Algorithms¶
There are three different search/sweep algorithms that you can use with the optimizer:
- "grid" - Sweep algorithm based on picking parameter values from discrete, possibly sampled, regions
- "random" - Random sampling algorithm
- "bayes" - Bayesian algorithm based on distributions, balancing exploitation and exploration
Example:
python
{"algorithm": "bayes"}
For most searches, we recommend using "bayes". However, for some cases, one of the algorithms may be more appropriate. See below for more details.
Bayes Algorithm¶
As mentioned, the Bayes algorithm may be the best choice for most of your Optimizer uses. It provides a well-tested algorithm that balances exploring unknown space, with exploiting the best known so far. The Comet Bayes algorithm implements the adaptive Parzen-Rosenblatt estimator.
The "bayes" search algorithm uses the following options in the "spec":
- "maxCombo"- integer, the limit of parameter combinations to try (default 0, meaning to use 10 times the number of hyperparameters)
- "objective" - string "minimize" or "maximize", for the objective metric (default "minimize")
- "metric" - string, the metric name that you are logging and want to minimize/maximize (default "loss")
- "minSampleSize" - integer, the number of samples to help find appropriate grid ranges (default 100)
- "retryLimit" - integer, the limit to try creating a unique parameter set before giving up (default 20)
- "retryAssignLimit" - integer, the limit to re-assign non-completed experiments (default 0)
The Comet optimizer will never assign the same set of parameters
twice, unless running multiple trials of an experiment, or reassigning
a non-completed experiment (see below). Depending on the algorithm you
have chosen, the number of possible experiments to run may be finite,
or infinite. For example, the "bayes" algorithm always samples from
continuous parameter distributions, but the "grid" algorithm always
breaks distributions into grids. However, some parameter
types (including "categorical" and "discrete") there are only a finite
number of options. To see the computed value of maximum number of
possible experiments for an algorithm, see Optimizer.status()
.
In either the finite or infinite case, to limit the number of
experiment combinations to run set maxCombo
to a non-zero
value. Notice that this is the "maximum combinations" not the total
maximum number of experiments to assign. For example, the total
number of experiments assigned is maxCombo
* trials
. But this can
also be effected by the SPEC setting "retryAssignLimit" which will
re-assign experiments until they are "completed" or
the "retryAssignLimit" value is met. Therefore, if each experiment never
completes, you would assign maxCombo
* trials
* (retryAssignLimit
+ 1)
number of experiments.
Example:
python
{"algorithm": "bayes",
"spec": {
"maxCombo": 0,
"objective": "minimize",
"metric": "loss",
"minSampleSize": 100,
"retryLimit": 20,
"retryAssignLimit": 0,
},
"trials": 1,
"parameters": {...},
"name": "My Optimizer Name",
}
Grid Algorithm¶
The "grid" algorithm is useful for performing a wide, initial search of a set of parameter values. Comet's grid algorithm is slightly more flexible than many, as each time you run it, you will sample from the set of possible grids defined by the parameter space distribution. The "grid" algorithm does not use past experiments to inform future experiments: it merely collects the objective metric for you to explore.
The "grid" search algorithm uses the following options:
- "randomize" - boolean, if True, then the grid is traversed randomly; otherwise traversed in order (default False)
- "maxCombo"- integer, the limit of parameter combinations to try (default 0, meaning to use 10 times the number of hyperparameters)
- "metric" - string, the metric name that you are logging and want to minimize/maximize (default "loss")
- "gridSize" - integer, when creating a grid, the number of bins per parameter (default 10)
- "minSampleSize" - integer, the number of samples to help find appropriate grid ranges (default 100)
- "retryLimit" - integer, the limit to try creating a unique parameter set before giving up (default" 20)
- "retryAssignLimit" - integer, the limit to re-assign non-completed experiments (default 0)
Example:
python
{"algorithm": "grid",
"spec": {
"randomize": True,
"maxCombo": 0,
"metric": "loss",
"gridSize": 10,
"minSampleSize": 100,
"retryLimit": 20,
"retryAssignLimit": 0,
},
"trials": 1,
"parameters": {...},
"name": "My Optimizer Name",
}
Random Algorithm¶
The "random" algorithm is slightly more flexible than the "grid" algorithm, in that it will continue to sample from the set of possible parameter values, until you stop the search, or have the "max combinations" value set. The "random" algorithm, like the "grid" algorithm, does not use past experiment metrics to inform future experiments.
The "random" search algorithm uses the following options:
- "maxCombo"- integer, the limit of parameter combinations to try (default 0, meaning to use 10 times the number of hyperparameters)
- "metric" - string, the metric name that you are logging and want to minimize/maximize (default "loss")
- "gridSize" - integer, when creating a grid, the number of bins per parameter (default 10)
- "minSampleSize" - integer, the number of samples to help find appropriate grid ranges (default 100)
- "retryLimit" - integer, the limit to try creating a unique parameter set before giving up (default" 20)
- "retryAssignLimit" - integer, the limit to re-assign non-completed experiments (default 0)
Example:
python
{"algorithm": "random",
"spec": {
"maxCombo": 100,
"metric": "loss",
"gridSize": 10,
"minSampleSize": 100,
"retryLimit": 20,
"retryAssignLimit": 0,
},
"trials": 1,
"parameters": {...},
"name": "My Optimizer Name",
}
Specifying Optimizer Parameters¶
There are four kinds of parameters: "integer", "double" or "float", "discrete" (for a list of numbers), and "categorical" (for a list of strings).
The format of each parameter was inspired by Google's Vizier, and exemplified by the open source version called Advisor.
Integers¶
Integers can be distributed in one of five ways: "linear", "uniform", "normal", "loguniform", or "lognormal".
python
{"PARAMETER-NAME":
{"type": "integer",
"scalingType": "linear" | "uniform" | "normal" | "loguniform" | "lognormal",
"min": INTEGER,
"max": INTEGER,
},
...
}
See below for more details on "scalingType" for each algorithm.
NOTE: "integer" type
with "linear" scalingType
when using the
"bayes" algorithm indicates an independent distribution. This is useful
for using integer values that have no relationship with one another,
such as seed values. If your distribution is meaningful (e.g., 2 is
closer to 1 than it is to 6) then you should use the "uniform"
scalingType
.
You can also provide a list of integers (or any numbers) using the "discrete" type:
python
{"PARAMETER-NAME":
{"type": "discrete",
"values": [NUMBER, ...],
},
...
}
Example:
python
{"algorithm": "bayes",
"spec": {...},
"parameters": {
"hidden-layer-size": {"type": "integer", "min": 5, "max": 100},
"hidden2-layer-size": {"type": "discrete", "values": [16, 32, 64]},
},
"trials": 1,
"parameters": {...},
"name": "My Optimizer Name",
}
Double/Float¶
Doubles (also called floats) can be distributed in one of five ways: "linear", "uniform", "normal", "loguniform", or "lognormal".
python
{"PARAMETER-NAME":
{"type": "double" |"float",
"scalingType": "linear" | "uniform" | "normal" | "loguniform" | "lognormal",
"min": FLOAT,
"max": FLOAT,
},
...
}
See below for more details on "scalingType" for each algorithm.
You can also provide a list of doubles/floats (or any numbers) using the "discrete" type:
python
{"PARAMETER-NAME":
{"type": "discrete",
"values": [NUMBER, ...],
},
...
}
Example:
python
{"algorithm": "bayes",
"spec": {...},
"parameters": {
"momentum": {"type": "double", "min": 0.0, "max": 0.9},
"learning-rate": {"type": "discrete", "values": [0.01, 0.1, 0.8]},
},
"trials": 1,
"parameters": {...},
"name": "My Optimizer Name",
}
Scaling Types¶
Both "integer" and "double"/"float" allow the following scaling types (indicated with "scalingType"):
- "linear" - for integers, means an independent distribution (used for things like seed values); for double, the same as uniform
- "uniform" - a uniform distribution between "min" and "max"
- "normal" - a normal distribution centered around "mu" with standard deviation of "sigma"
- "lognormal" - a log-normal distribution centered around "mu" with standard deviation of "sigma"
- "loguniform" - a log-uniform distribution between "min" and "max". Computes
exp(uniform(log(min), log(max)))
Examples:
python
{"algorithm": "bayes",
"spec": {...},
"parameters": {
"momentum": {"type": "float", "min": 0.0, "max": 0.9, "scalingType": "uniform"},
"learning-rate": {"type": "float", "min": 0.001, "max": 1.0, "scalingType": "loguniform"},
"dropout": {"type": "float", "mu": 0.5, "sigma": 0.1, "scalingType": "normal"},
},
"trials": 1,
"parameters": {...},
"name": "My Optimizer Name",
}
Categorical¶
Categorical, like "discrete", is a type for a list of values. In this case, the values must be strings.
python
{"PARAMETER-NAME":
{"type": "categorical",
"values": ["LIST", "OF", "STRINGS"],
},
...
}
Example:
python
{"algorithm": "bayes",
"spec": {...},
"parameters": {
"activation": {"type": "categorical", "values": ["sigmoid", "relu", "leaky-relu"]},
},
"trials": 1,
"parameters": {...},
"name": "My Optimizer Name",
}
Additional Optimizer Parameter Settings¶
For each parameter, you may also specify "gridSize".
Note: gridSize
is only used in the grid
and random
algorithms.
Grid Size¶
Each parameter is considered a distribution for those algorithms that sample randomly. Those algorithms include "bayes" and "random". However, other algorithms need to know a resolution size for how to divide up the parameter space into discrete bins. Those algorithms include "grid". For those, an additional entry named "gridSize" can be set for each parameter. For example, the following defines a parameter "x" ranging between 0.0 and 100.0, and broken up into 25 bins for the grid search.
Example:
python
{"x":
{"type": "double",
"scalingType": "uniform",
"min": 0.0,
"max": 100.0,
"gridSize": 25,
},
}
NOTE: the bins won't be exactly 0-25, 25-50, 50-75, and 75-100. Rather, the divisions are created based on sampled values.
Comet Optimize¶
comet
is a command-line utility that is installed with comet_ml
.
optimize
is one of the commands that comet
can use. The format is:
bash
$ comet optimize [options] [PYTHON_SCRIPT] OPTIMIZER
For more information on comet optimize
please see Command-Line Utilities
End-to-end Example¶
Now, let's see a complete end-to-end program using Keras with the Comet Optimizer.
```python import comet_ml import logging
logging.basicConfig(level=logging.INFO) LOGGER = logging.getLogger("comet_ml")
from tensorflow.keras.datasets import mnist from tensorflow.keras.layers import Dense from tensorflow.keras.models import Sequential from tensorflow.keras.optimizers import RMSprop from tensorflow.keras.utils import to_categorical
def build_model_graph(experiment): model = Sequential() model.add( Dense( experiment.get_parameter("first_layer_units"), activation="sigmoid", input_shape=(784,), ) ) model.add(Dense(128, activation="sigmoid")) model.add(Dense(128, activation="sigmoid")) model.add(Dense(10, activation="softmax")) model.compile( loss="categorical_crossentropy", optimizer=RMSprop(), metrics=["accuracy"], ) return model
def train(experiment, model, x_train, y_train, x_test, y_test): model.fit( x_train, y_train, batch_size=experiment.get_parameter("batch_size"), epochs=experiment.get_parameter("epochs"), validation_data=(x_test, y_test), )
def evaluate(experiment, model, x_test, y_test): score = model.evaluate(x_test, y_test, verbose=0) LOGGER.info("Score %s", score)
def get_dataset(): num_classes = 10
# the data, shuffled and split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.reshape(60000, 784)
x_test = x_test.reshape(10000, 784)
x_train = x_train.astype("float32")
x_test = x_test.astype("float32")
x_train /= 255
x_test /= 255
print(x_train.shape[0], "train samples")
print(x_test.shape[0], "test samples")
# convert class vectors to binary class matrices
y_train = to_categorical(y_train, num_classes)
y_test = to_categorical(y_test, num_classes)
return x_train, y_train, x_test, y_test
Get the dataset:¶
x_train, y_train, x_test, y_test = get_dataset()
The optimization config:¶
config = { "algorithm": "bayes", "name": "Optimize MNIST Network", "spec": {"maxCombo": 10, "objective": "minimize", "metric": "loss"}, "parameters": { "first_layer_units": { "type": "integer", "mu": 500, "sigma": 50, "scalingType": "normal", }, "batch_size": {"type": "discrete", "values": [64, 128, 256]}, }, "trials": 1, }
opt = comet_ml.Optimizer(config)
for experiment in opt.get_experiments(project_name="my_project"): # Log parameters, or others: experiment.log_parameter("epochs", 10)
# Build the model:
model = build_model_graph(experiment)
# Train it:
train(experiment, model, x_train, y_train, x_test, y_test)
# How well did it do?
evaluate(experiment, model, x_test, y_test)
# Optionally, end the experiment:
experiment.end()
```
Troubleshooting¶
Continue from Crashed/Paused Optimizer¶
If you pause your search, or if your optimizer script would ever crash
you can recover your search and pick up immediately from where you
left off. You need only define the COMET_OPTIMIZER_ID
in the
environment, and run your script again. The COMET_OPTIMIZER_ID
is
printed in the terminal at the start of each sweep. It also is logged
with each experiment in the Other tab. Here is an example or a script
crashing, and continuing with the search:
```bash $ python script.py
COMET INFO: COMET_OPTIMIZER_ID=366dcb4f38bf42aea6d2d87cd9601a60 ... it crashes for some reason
$ edit script.py
$ export COMET_OPTIMIZER_ID=366dcb4f38bf42aea6d2d87cd9601a60
$ python script.py COMET INFO: COMET_OPTIMIZER_ID=366dcb4f38bf42aea6d2d87cd9601a60 ```
You can also supply the optimizer id to the Optimizer class rather
than the name of the filename containing the optimizer config. For
example, consider again example-2.py
from above:
```python
file: example-2.py¶
from comet_ml import Optimizer import sys
Next, create an optimizer, passing in the config:¶
(You can leave out API_KEY if you already set it)¶
opt = Optimizer(sys.argv[1])
define fit function here!¶
Finally, get experiments, and train your models:¶
for experiment in opt.get_experiments( project_name="optimizer-search-03"): # Test the model loss = fit(experiment.get_parameter("x")) experiment.log_metric("loss", loss) ```
Recall that you can start that program up, like:
bash
$ python example-2.py example-2.config
or using comet optimize
:
bash
$ comet optimize -j 2 example-2.py example-2.config
To use your same script and start up where you left off, you only need the Comet Optimizer id. When you start up a new optimizer, you will see a line displayed similar to:
COMET INFO: COMET_OPTIMIZER_ID=303faefd8194400694ec9588bda8338d
You can set this Comet environment variable in the terminal, and your search will use the existing Optimizer rather than creating a new one.
bash
$ export COMET_OPTIMIZER_ID=303faefd8194400694ec9588bda8338d
$ python example-2.py example-2.config
or
bash
$ export COMET_OPTIMIZER_ID=303faefd8194400694ec9588bda8338d
$ comet optimize -j 2 example-2.py example-2.config
You can also just pass the Optimizer id on the command line instead of
the filename if you have written your script in the style of
example-2.py
:
bash
$ python example-2.py 303faefd8194400694ec9588bda8338d
or
bash
$ comet optimize -j 2 example-2.py 303faefd8194400694ec9588bda8338d
You can also have comet optimize
pass along arguments to your
script. Simply add those after the config following two dashes, like
this:
bash
$ comet optimize -j 4 script.py opt.config -- --project-name "test-007"
Then you can use the argparse module, like so:
```python
example-3.py¶
from comet_ml import Optimizer, Experiment
import argparse
parser = argparse.ArgumentParser()
Add your own args here:¶
parser.add_argument("--project-name", default=None)
These passed on from "comet optimize":¶
parser.add_argument("optimizer", default="test1_optimizer.json") parser.add_argument("--trials", "-t", type=int, default=None)
parsed = parser.parse_args()
count = 0 for experiment in opt.get_experiments(): loss = train(experiment.params["x"]) msg = experiment.log_metric("loss", loss) count += 1 print("Optimizer job done! Completed %s experiments." % count) ```
The above program can then be used alone, or with the comet optimize
to run scripts in parallel with custom command-line arguments. Called
normally:
bash
$ python example-3.py opt.config --project-name "my-project-01"
or in parallel:
bash
$ comet optimize example-3.py opt.config -- --project-name "my-project-01"
What if an experiment doesn't finish?¶
By default, all of the algorithms will not release duplicate sets of parameters (except when trials is greater than 1). But what should you do if an experiment crashes and never notifies the Optimizer?
You have two choices:
- You can run the Optimizer search with the "retryAssignLimit" spec settings:
python
{"algorithm": "bayes",
"spec": {
"retryAssignLimit": 1,
...
},
"parameters": {...},
"name": "My Bayesian Search",
"trials": 1,
}
Using a retryAssignLimit
value greater than zero will continue to
assign the parameter set until an experiment marks it as "completed"
or the number of retries is equal to the retryAssignLimit
.
- You can run the Optimizer search/sweep again. You can either run all of the parameter value combinations again, or a subset thereof.