Production LLM Evaluation and Observability

On Udemy

About this course

This course teaches you how to evaluate, test, and monitor real world LLM applications before and after they reach production. You will build an Agentic RAG chatbot and use DeepEval for LLM evaluation and Langfuse for LLM observability, tracing, and monitoring. Build a Practical LLM Evaluation and Observability Workflow.

In this course, you will: 1. Learn the fundamentals of LLM evaluation and why traditional testing is not enough for AI applications. 2. Understand human vs automated evaluation and when to use each approach. 3. Learn offline and online evaluation for LLM applications. 4. Create and work with golden datasets for systematic evaluation. 5. Understand reference based and reference free evaluation. 6. Build and use different LLM evaluation metrics. 7. Create custom evaluation metrics for application-specific requirements. 8. Use DeepEval to build an automated LLM evaluation pipeline. 9. Evaluate an Agentic RAG chatbot using practical evaluation techniques. 10. Learn LLM observability with Langfuse. 11. Trace LLM calls, retrieval steps, tool calls, and application workflows. 12. Monitor evaluation scores and use observability data to understand application behavior. 13. Connect evaluation and observability to create a more reliable production workflow.

What You Will Build

Throughout the course, you will work with an Agentic RAG chatbot and gradually add an evaluation and observability layer around it.

You will first understand the application architecture, then build an evaluation workflow using DeepEval, and finally add Langfuse for tracing and observability.

By the end of the course, you will understand how the different pieces fit together: