🎓 Free video course "LLM evaluation for AI builders" with 10 code tutorials.  Save your seat
APPLIED course

LLM evaluation for builders

Free 3-week video course with 10 hands-on code tutorials. From designing custom LLM judges to RAG evaluations and adversarial testing.

đź—“ Start: May 12, 2025

Sign up now
Evidently AI Open-source ML observability course
about the course

What you will learn

This is a practical course on LLM evaluation for AI and ML Engineers, technical product managers, and anyone who works hands-on on AI applications. You’ll work through 10 Python tutorials covering essential evaluation workflows, like building test datasets, comparing prompts and models, and tracing LLM outputs. 
Sign up
Icon
Icon
LLM evaluation methods. How to implement automated quality checks, from deterministic validations to model-based scoring.
RAG evaluation. How to measure both retrieval and generation quality, and use synthetic data effectively.
LLM judges. How to design and tune custom LLM judges that align with human preferences and observed failure modes.
Adversarial testing. How to uncover model vulnerabilities and assess response safety and edge-case behavior.
Icon
Hands-on LLM app testing. Together, we’ll build and evaluate LLM apps for summarization, classification, content generation, and basic AI agents.
Icon
Team collaboration. How to organize evaluation work across a team, from test case design to collaborative debugging.
About the course

What you will learn

This is a practical course on LLM evaluation for AI builders. Learn by doing with 10 Python tutorials on core workflows - like building test datasets, comparing prompts and models, and tracing LLM outputs.
LLM evaluation methods
How to implement automated quality checks, from deterministic validations to model-based scoring.
LLM judges
How to design and tune custom LLM judges that align with human preferences and observed failure modes.
Hands-on LLM app testing
Together, we’ll build and evaluate LLM apps for summarization, classification, content generation, and basic AI agents.
RAG evaluation
How to measure both retrieval and generation quality, and use synthetic data effectively.
Adversarial testing
How to uncover model vulnerabilities and assess response safety and edge-case behavior.
Team collaboration
How to organize evaluation work across a team, from test case design to collaborative debugging.
perks

Why take the course?

Get applied, hands-on skills in LLM evaluation.
Icon
Free course
Icon
10+ video tutorials
Icon
End-to-end code examples
Live Q&A sessions

Join the course now

Sign up to get course materials when the course starts.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
team

Meet course instructors

Emeli Dral is a Co-founder and CTO at Evidently AI, a Y Combinator-backed startup developing AI observability tools to evaluate, test, and monitor the quality of AI systems.
‍
Earlier, she co-founded an industrial AI startup and served as the Chief Data Scientist at Yandex Data Factory. She led over 50 applied ML projects for various industries: from banking to manufacturing. Emeli is a data science lecturer at GSOM SpBU and Harbour.Space University. She is a co-author of the Machine Learning and Data Analysis curriculum at Coursera with over 150,000 students.

Elena Samuylova is a CEO and Co-founder at Evidently AI.

‍

She has been active in the applied machine learning space since 2014. Previously, she co-founded and served as a CPO of an industrial AI startup. She worked with global metal and chemical companies to implement machine learning for production optimization. Prior to that, she led business development at Yandex Data Factory, an enterprise AI division of Yandex. She focused on delivering ML-based solutions to retail, banking, telecom, and other industries. In 2018, Elena was named 50 Women in Product Europe by Product Management Festival.

Faq

Have a question or need help?

Is the course free?
Icon

Yes, the course is 100% free.

What is the course format?
Icon

The course is organized into a series of daily emails. Each day you will receive an email that consists of a video covering a particular topic and a link to practical code example to follow along.

Sign up now.  

How long does it take to complete the course?
Icon

The course lasts 3 weeks.

Will I get a course certificate?
Icon

This time, there will be no course certificate or assignments to complete. We guarantee you a lot of practice though! In every video, we will walk you through code examples step-by-step, so you could adapt them for your use case.

Who the course is for?
Icon

This course is for anyone building real-world AI applications - whether you're an AI/ML Engineer, data scientist, product manager, or researcher. If you’re working hands-on with LLMs and have basic Python skills, this course is for you.

Will I need to buy tools or software to take the course?
Icon

No. We will use the open-source version of the Evidently library and some features of Evidently Cloud available on a free plan.

However, if you want to follow the code examples, you will need an LLM API key and a couple of dollars for sending requests.

Will there be theoretical lectures?
Icon

This course includes just one short introductory session. The rest is fully hands-on and applied. If you're looking for a more gentle theoretical introduction, check out our previous course or explore the full library of videos available here.

What if I need help?
Icon

Have a question or just want to say “Hi”? Jump to our Discord to chat with fellow learners and get support from the course team.

We will also host two Q&A sessions in Zoom if you have any questions about the course and its contents (no worries, you will get all the information into your inbox along with the course materials!).

How can I enroll?
Icon

Sign up here to enroll and receive course materials.

Will course materials still be available after the course?
Icon

Yes! All course videos will be available during and after the course.

Evaluate, test and monitor your AI-powered products.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
LLM testing platformAI risk assessmentRAG testingAdversarial testingAI agent testingML monitoringOpen-source
BlogTutorialsGuidesML platformsML use casesLLM evaluations course
PricingDocsGitHubCommunity
© 2025, Evidently AI. All rights reserved.
Twitter logoDiscord logoYouTube logo
By clicking “Accept”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.
DenyAccept