Error Mode
By default, the Experiment Management SDK does not raise exceptions when issues occur, allowing training scripts to run uninterrupted. While this behavior helps prevent disruptions during model training, it can make debugging more challenging. Instead, the SDK shows WARNING or ERROR log messages when data is discarded or altered (e.g., when a value is too long and gets truncated).
To make debugging easier, especially in interactive environments like Jupyter Notebook or Google Colab, you can enable Error Mode by calling comet_ml.error_mode. When Error Mode is enabled, the following changes take effect:
- Tracebacks in the Console: Errors will be displayed with full tracebacks.
- Exceptions Raised: Exceptions will be raised when:
- Invalid data is logged.
- Data cannot be processed and would otherwise be discarded.
- Data would be altered before logging (e.g., truncated values).
Info
Error Mode only applies to data processing issues. Data is then buffered and sent in the background, and issues that occur during this asynchronous upload process do not raise exceptions, even with Error Mode enabled.
Info
Error Mode remains active for the duration of the current Python process. To return to the default behavior (i.e., no exceptions), restart the kernel (in Jupyter Notebook or Colab) or restart the Python script after removing the comet_ml.error_mode() call.