Ship Reliable AI Agents: LLM Evals, CI Gates and Monitoring

About this course

This course contains the use of artificial intelligence. Your agent's pull request is green. Every unit test passes. That afternoon, a customer asks about a refund, and the bot explains a refund policy your company doesn't have.

Unit tests check that your code runs. They don't check that an answer is right. This course teaches the loop that does: traces, error analysis, a labelled dataset, graders, a CI gate, and monitoring of live traffic, built in plain Python and pytest on a small support agent you run on your own machine.

How it works

Watch. Each lecture is short and animated: traces unfold span by span, failure categories take shape from your notes, thresholds are drawn against their error bars

Run it. You get the companion repository as a zip with its full git history, one starting branch per section. Everything runs in replay mode against recorded responses, so you need no API key and spend nothing on tokens.

Practise. Every section ends with a short quiz and a practice exercise in the repo, with a check command that tells you when you've got it right.

What you'll work onTraces: instrument an agent with OpenTelemetry so every run leaves a record of its model calls, tool calls and retrievals, and keep user data out of them.

Error analysis: read traces, write notes, group them into a failure taxonomy and turn it into a pass or fail labelled dataset.

Code checks first: write pytest checks with passing and failing examples before reaching for an LLM judge.

LLM judges you can trust: write a binary judge, validate it against human labels with TPR, TNR and Cohen's kappa, and measure what each run costs.

Read the full description on Udemy on Udemy (opens in a new tab)

Prices, discounts and availability are set by Udemy. We may earn a commission when you purchase through links on this site.