3 Tips for Evaluating ML Platforms and Tools

Artificial intelligence (AI) is encountering yet another hurdle to delivering value, in the form of friction among and between teams. Machine learning (ML) has delivered outsized business value for use cases such as detecting fraud, making product recommendations, and predicting customer churn.

As a result, more companies are looking to apply it to innovate products, enhance customer service, and optimize business processes. However, ML development teams find ML platform tools and processes to be disconnected, unreliable, and unscalable.

In fact, challenges related to people, processes, and tools are creating friction that makes it difficult to track model training runs and results. This friction can cause delays in ML development that complicate the already complex process of deploying models into production.

To learn more about the challenges facing the industry, Comet conducted a survey of more than 500 ML practitioners. The survey highlighted these challenges companies face when developing machine learning projects:

• Wasted time, resources, and budget – Machine learning development requires many experiments, sometimes thousands, to train an ML model to perform well. Most (58%) of ML practitioners reported they use manual tools, like spreadsheets or Git hosting, to track their ML experiments. Manual tracking not only takes longer, it can introduce human error. The majority (68%) of respondents admitted to abandoning 40-80% of their experiments due to mismanagement.
• Lag in model deployment – Only 6% of teams participating in the survey have been able to deploy a model into production in less than 30 days. Almost half (47%) of ML teams require four to six months to deploy a single ML project, while 43% take up to three months. Respondents also cited issues, including lack of infrastructure or resources, reproducibility failures, API integration errors, and debugging failures.
• Inadequate budget for tools – According to our ML practitioners survey, 88% of respondents have an annual budget of less than $75,000 for machine learning tools and infrastructure. That’s less than the average salary of a data scientist, which ranges from $121,000 to $149,000 depending on location.

But there’s good news: innovative ML development teams at companies like Uber and Netflix are tackling these challenges and developing tools for ML development, many of which are available via open source. There also is a growing ecosystem of ML platforms and tools to assist teams that want to apply machine learning and deep learning in their organizations. And some of these tools are better equipped to accelerate the development process.

There are three tips for ML practitioners who are choosing an ML platform or tool so they can reduce friction and accelerate the ML development process:

1. Insist on integrations and customizability

The best ML platforms are highly customizable to practitioners’ workflows and seamlessly integrate with proven frameworks. Look for tools that complement effective processes already in place. The best ML tools can be used with the libraries and frameworks teams use today and be flexible enough to leverage emerging techniques and technologies. Customizability is important because no two ML use cases are exactly alike – data sources, algorithms, and outcomes vary across applied ML – so it’s best to look for a tool that allows teams to customize their workflows, reports, and visualizations.

2. Prioritize scalability 

The best ML tools can handle the intense demands of machine learning at scale. Ask ML platform providers about how their solution scales by the numbers: models managed, concurrent training jobs they can handle, and datasets stored, for example.

3. Consider the complete ML lifecycle

The best ML platforms offer value across the complete machine learning lifecycle, from tracking training runs to monitoring models in production. When models fail in production, it can help to revisit their performance baselines in training so they can be retrained for better performance and outcomes. Look for tools that make it easy to track, monitor, and optimize models across the complete machine learning development lifecycle.

The Bottom Line

While machine learning can deliver outsized business value, the process of ML project development is complex and comes with many potential pitfalls. Tools and techniques are changing all the time, so teams must optimize their tools and processes today and allow flexibility for adapting to future changes in tools and techniques. It’s a smart approach to evaluate ML platforms carefully and consider the importance of integrations, customizability, scale, and full-lifecycle capabilities to ensure faster time to value with machine learning.

To learn more, download the 2021 ML Practitioner Survey Report.