LLMOps Course: Building with LLMs

FREE COURSE

LLM Evaluation with Opik

Learn to test and evaluate your LLM applications using the latest tools and techniques, including LLM-as-a-judge metrics and production LLM monitoring.

IN COLLABORATION WITH

Expert Instructors

Taught by industry leaders

Self-Paced

Learn at your own speed

Built with Opik

Use open source tools and models

Course Description

Level: Beginner

Duration: 1 Hour

Audience: Data Scientists/Software Engineers

Prerequisites: Basic ML knowledge, Python experience

Who Is This For?

AI Developers

Anyone curious about LLMs

Engineers

Data Scientists

Why This Course?

This is the only course completely focused on applying state-of-the-art LLM evaluation techniques to real world applications. We will cover some theory, but this is first and foremost a course in applied AI, not mathematics.

Taught By An Expert

Elvis is the co-founder of DAIR.AI, where he leads all AI research, education, and engineering efforts. He focuses on training and building large language models (LLMs) and information retrieval systems. Previous to this, he was at Meta AI where he supported and advised world-class products and teams such as FAIR, PyTorch, and Papers with Code. He was also previously an education architect at Elastic where he developed technical curriculum and courses for the Elastic Stack. A

What You'll Learn

Introduction to LLM Evaluation

Brief introduction to LLM evaluations

Explore the challenges of evaluating LLM applications

Overview some common use cases

Tools & Frameworks for LLM Evaluation

Explore the LLM evaluation ecosystem

Familiarize yourself with Opik

Stand up your first simple evaluation suite

Project: Evaluating a Chatbot

Kickoff the main section of the course with a real project

Learn about common chatbot architectures

Implement evaluations for a real LLM application

Building an LLM Evaluation Pipeline

Go from simple evaluations to robust evaluation pipelines

Explore common workflows for production evaluation systems

Get your feet wet with manual evaluations

Heuristic Metrics for LLM Evaluation

Familiarize yourself with some “classic” metrics

Implement heuristic metrics from scratch

Understand the benefits and challenges of heuristic evaluations

LLM-Based Metrics for LLM Evaluation

Learn about LLM-as-a-judge metrics

Implement custom LLM-based metrics from scratch

Learn to test for hallucinations, factuality, and more

Testing & Monitoring LLM Applications

Learn to monitor deployed LLM applications

Implement LLM unit tests with PyTest and Opik

Understand the role of observability in LLM applications

The Future of LLM Evaluation

Explore advanced techniques for LLM evaluation

Understand safety evaluations and responsible AI

Overview the next steps in your learning journey

Brief introduction to LLM evaluations

Explore the challenges of evaluating LLM applications

Overview some common use cases

Explore the LLM evaluation ecosystem

Familiarize yourself with Opik

Stand up your first simple evaluation suite

Kickoff the main section of the course with a real project

Learn about common chatbot architectures

Implement evaluations for a real LLM application

Go from simple evaluations to robust evaluation pipelines

Explore common workflows for production evaluation systems

Get your feet wet with manual evaluations

Understand the benefits and challenges of heuristic evaluations

Implement heuristic metrics from scratch

Understand the benefits and challenges of heuristic evaluations

Learn about LLM-as-a-judge metrics

Implement custom LLM-based metrics from scratch

Learn to test for hallucinations, factuality, and more

Learn to monitor deployed LLM applications

Implement LLM unit tests with PyTest and Opik

Understand the role of observability in LLM applications

Explore advanced techniques for LLM evaluation

Understand safety evaluations and responsible AI

Overview the next steps in your learning journey

Module 1. Introduction to LLMs

Brief introduction to LLMs

Explore the importance of LLMOps in LLM engineering

Breakdown the LLMOps lifecycle

Overview the difference between LLMOps and MLOps

Module 2. Working with LLMs

Module 3. LLMOps in Practice

Module 4. Case Studies & Applications of LLMs

Module 5. Advanced Topics in LLMs and LLMOps

Module 6. The Future of LLMOps

An ML Platform Built for Enterprise, Driven by Community

Comet’s ML platform is trusted by innovative data scientists, ML practitioners, and engineers in the most demanding enterprise environments.

"Comet has aided our success with ML and serves to further ML development within Zappos.”

10%

reduction in order returns due to size

Kyle Anderson

Director of Software Engineering

"Comet offers the most complete experiment tracking solution on the market. It’s brought significant value to our business."

Service for millions of customers

Olcay Cirit

Staff Research and Tech Lead

“Comet enables us to speed up research cycles and reliably reproduce and collaborate on our modeling projects. It has become an indispensable part of our ML workflow.”

Developing
NLP tools for thousands of researchers

Victor Sanh

Machine Learning Scientist

"None of the other products have the simplicity, ease of use and feature set that Comet has."

Developing speech and language algorithms

Ronny Huang

Research Scientist

View Case Studies

"After discovering Comet, our deep learning team’s productivity went up. Comet is easy to set up and allows us to move research faster."

Building
speech recognition with deep learning

Guru Rao

Head of AI

"We can seamlessly compare and share experiments, debug and stop underperforming models. Comet has improved our efficiency."

Pioneering family history research

Carol Anderson

Staff Data Scientist

Enterprise User

"Comet has aided our success with ML and serves to further ML development within Zappos.”

10%

reduction in order returns due to size

Kyle Anderson

Director of Software Engineering

Enterprise User

"Comet offers the most complete experiment tracking solution on the market. It’s brought significant value to our business."

Service for millions of customers

Olcay Cirit

Staff Research and Tech Lead

Community User

“Comet enables us to speed up research cycles and reliably reproduce and collaborate on our modeling projects. It has become an indispensable part of our ML workflow.”

Developing
NLP tools for thousands of researchers

Victor Sanh

Machine Learning Scientist

Community User

"None of the other products have the simplicity, ease of use and feature set that Comet has."

Developing speech and language algorithms

Ronny Huang

Research Scientist

View Case Studies

Enterprise User

"After discovering Comet, our deep learning team’s productivity went up. Comet is easy to set up and allows us to move research faster."

Building
speech recognition with deep learning

Guru Rao

Head of AI

Enterprise User

"We can seamlessly compare and share experiments, debug and stop underperforming models. Comet has improved our efficiency."

Pioneering family history research

Carol Anderson

Staff Data Scientist

Frequently Asked Questions

What are the prerequisites for this course?

This course assumes no advanced math background. We will not be diving deep into the theory behind LLMs. All you need to get started is some basic proficiency in Python and a general understanding of deep learning.

Will it cost me anything?

The course content is 100% free. Every lesson can be completed using completely free and open source models via LiteLLM and Opik. You can also use the OpenAI or Anthropic API, if you prefer.

How much time should I commit?

The course is self-paced, so you can spend as little or as much time as you want. That said, students who set aside a meaningful block of time each week—whatever “meaningful” means for your schedule—tend to see the best results.

How long will this course take?

Your time to completion will vary depending on how much time you have available. In general, we recommend one week per module as a realistic pace for most people, meaning the course would take six weeks total. Of course, you can take as long as you’d like.

Join fellow students to discuss course content or ask questions in the Community Slack

Join the Slack Group