LLM Testing for QA Beginners: DeepEval, Promptfoo & CI

About this course

Your regular tests can all pass while your chatbot still gives wrong, made-up or biased answers. LLM output is free-form and changes from run to run, so exact-match assertions stop working. This course is a beginner-friendly first step into testing LLM applications. You'll learn the idea behind most LLM testing today: using one model as a judge to score another model's answers. Then you'll put it into practice with two popular open-source tools, DeepEval and Promptfoo. Everything runs on free local models through Ollama, so you don't need a paid API key to follow along.

What we cover:

• Why LLM testing is different from traditional testing
• Setting up Ollama, Python and the evaluation frameworks
• Measuring correctness, relevance, faithfulness, hallucination, toxicity and bias with DeepEval
• When to use DeepEval's built-in metrics and when to write your own with GEvalTesting a simple RAG application and an AI agent
• Writing Promptfoo tests, comparing models, and running a first red-team scan
• Running your evaluations in GitHub Actions with pass/fail thresholds
• You'll finish with a small end-to-end project: tests and evals running in a GitHub Actions pipeline that you can keep on your GitHub.
• Is this course right for you?

This is an introductory course for QA engineers and testers who are new to LLM testing and want a clear, hands-on starting point. If you already run LLM evaluations in production, you'll likely find it too basic. Advanced topics like in-depth red-teaming, agent memory testing and eval cost control are not covered here.

What you'll learn

Explain why unit, integration and E2E tests aren't enough for LLM applications

Use LLM-as-a-judge to score correctness, relevance, faithfulness, hallucination, toxicity and bias

Read the full description on Udemy on Udemy (opens in a new tab)

Prices, discounts and availability are set by Udemy. We may earn a commission when you purchase through links on this site.