EU AI Act Regulation Compliance with Comet

 On March 13, 2024, the European Parliament passed the EU AI Act to establish a common regulatory and legal framework for AI.

About the Act

The EU AI Act divides AI Systems into 5 different risk categories

1. Minimal Risk: Models used in video games and spam filters.
2. Limited Risk: Models that produce content such as text or images.
3. High Risk: Models that can negatively affect the health, safety or the fundamental rights of persons. These include systems used in health, education, recruitment, critical infrastructure management, law enforcement or justice.
4. Unacceptable Risk: Models that use social scoring, predictive policing, real-time identification systems or any other manipulative practices.
5. General-purpose AI: Models that are generic enough to perform a wide range of tasks.

Regulation Requirements

The Risk Categories are to be regulated in different ways. AI Systems with Unacceptable Risks are under no circumstances allowed to be deployed within the EU. Minimal Risk systems at the moment will be unregulated. Limited Risk systems must be transparent and denote to users that they are interfacing with AI or AI generated content.

High Risk and GPAI Models are categories that must follow strict guidelines.

1. Conduct Data Governance to ensure that training, validation, and test datasets are relevant and representative
2. Use Record-Keeping to identify risks and substantial modifications throughout the Model lifecycle
3. Implement proportionate Post-Deployment Monitoring to evaluate continuous compliance of the AI system by collecting and analyzing performance data
4. Draw up Technical Documentation, including training and test process and evaluation results.

How Can Comet Help Teams With Regulatory Compliance?

Comet is a tool that allows machine learning teams to track and manage the entire model lifecycle: from datasets and training runs to model performance in production. With just a simple ‘pip install comet_ml’ and a couple of lines code, teams can start logging their data to Comet without disrupting their current workflows.

Single System of Record with Experiment Management

Comet tracks all relevant information for a model training run. This includes

1. Evaluation and Performance Metrics of the Trained Model
2. The Code Used for Training
3. The Exact Combination of Hyper-Parameters Selected
4. Lineage to the Specific Dataset Used for Training

Having access to all this information in a single UI that displays this information makes it easy for ML teams to adhere to the record keeping and data governance requirements for the EU AI Act.

Ensure Quality with Model Production Monitoring

Models are rarely “deploy and forget”. Just because a deployed production model has been performing as expected over the past couple months, doesn’t mean it will continue to do so. Hence, the EU is requiring post-deployment monitoring of high-risk models such as credit scoring and fraud detection.

Comet’s MPM allows teams to identify model performance degradation in production. By tracking data drift on input features and model output predictions, Comet can track failure even before a team has access to ground truth labels. MPM can also detect model bias by computing fairness metrics across different segments of your data.

Standardize Model Auditing with Comet Panels & Reports

Comet Panels are Visualizations used by Machine Learning Engineers to compare and debug their model training runs. These Panels range from visualizing high-level metrics such as loss and accuracy to analyzing specific data samples where a model is under-performing. You can export all these visualizations to generate a detailed report about the Model, which you can use as technical documentation for future audits.

Start Using Comet Today

NatWest, a Big Four bank in the United Kingdom, has been leveraging Comet for years to standardize model governance across its different verticals. They are well equipped to meet the regulatory requirements defined by the EU. You too can get ahead of the curve when you start logging your data to Comet!