Backups
Backups ¶
Ensuring the safety and availability of your data is paramount. Regular backups can prevent data loss in scenarios like hardware failures, software bugs, or accidental deletions.
Cloud Providers¶
For deployments on major cloud providers, automated snapshots of your instance volumes are highly recommended. These snapshots capture the entire state of the volume, making it easier to restore or migrate data if needed.
Here are guides for setting up snapshot policies on the major cloud platforms:
Bare Metal¶
For deployments on bare metal servers, the backup strategy might need to be more manual but is equally crucial.
We recommend you periodically take a backup of the disk or volume containing your experiment data. Depending on your infrastructure, you can:
- Use your internal backup systems if available.
- Utilize widely-accepted backup tools such as rsync, Bacula, Restic, Duplicity or BorgBackup. Remember, the frequency and comprehensiveness of your backups should align with the criticality of your data and your recovery objectives.
Data Directories:¶
Although not recommended, if you choose to back up only specific data directories, be aware that in case of restoration you would need to reconfigure everything. Here's a non-exhaustive list of critical Comet related directories:
- Mysql:
/var/lib/mysql - Minio:
/mnt/minio - Postgres:
/var/lib/postgresql/
* Postgres is only used if you have Model Production Monitoring/MPM enabled
Config files can be found in:
- /etc/default/comet-ml
- /etc/comet-ml/comet-ml.yml
The application directory and logs are located at:
- /opt/comet-ml/