Build an LLM from Scratch: Code a GPT in PyTorch

About this course

ChatGPT, Claude, Llama, Gemini: we use large language models every day, but what actually happens inside? In this course you do not just use an LLM. You build one, line by line, in plain PyTorch, and you understand every piece of it.

Our running example is a Sherlock Holmes story, "A Scandal in Bohemia". By the end of the course, your own code will:- pretrain a small GPT on the story and generate text in the style of Conan Doyle,- turn a pretrained GPT-2 into a spam filter for text messages,- fine-tune it to follow instructions, like a small personal assistant.

The course follows three stages. Stage 1 builds the foundation: what LLMs are, turning text into tokens and embeddings, coding self-attention and multi-head attention, and assembling a complete GPT model. Stage 2 is pretraining: the training loop, evaluation, temperature and top-k sampling, and loading the real GPT-2 weights. Stage 3 is fine-tuning: a text classifier, and instruction fine-tuning with automatic evaluation.

Every chapter comes with an animated video that explains the ideas with real numbers from real code, a practice notebook where you write the key functions yourself while check cells tell you immediately whether your code is right, a quiz, and an open challenge.

Two appendices cover a PyTorch primer and parameter-efficient fine-tuning with LoRA. More than forty optional extra videos go deeper: byte pair encoding from scratch, the KV cache, grouped-query attention, mixture of experts, Llama, Qwen and Gemma architectures, DPO, and more.

You do not need a GPU: every notebook runs on a normal laptop. There is no heavy math either; every formula is shown with concrete numbers.

The code is adapted from Sebastian Raschka's open source repository LLMs-from-scratch (Apache License 2.0). His book "Build a Large Language Model (From Scratch)" goes into even more depth and is highly recommended. The examples, animations, exercises and quizzes are made for this course.

Prices, discounts and availability are set by Udemy. We may earn a commission when you purchase through links on this site.