LLM Serving, Deployment & Scaling - Ollama, vLLM & Ray Serve

About this course

A warm welcome to LLM Serving, Deployment & Scaling - Ollama, vLLM & Ray Serve course by Uplatz.

What is LLM Serving?

LLM serving is the process of running a trained language model and making it available to applications or users for inference.

The model has already been trained. Serving begins when a system loads the model into memory and accepts requests such as:

• Summarise this document.
• Answer this question.
• Generate a product description.
• Continue this conversation.

A typical serving flow is:

• Application sends a request.
• The LLM-serving engine receives it.
• The model performs inference.
• The model generates output tokens.
• The response is returned to the application.

When a request reaches the serving system:

• The input text is converted into tokens.
• The model processes the input tokens.
• The model generates output tokens, usually one at a time.
• The tokens are converted back into readable text.
• The response is returned to the application.

LLM serving commonly includes:

• Loading the model
• Receiving inference requests
• Tokenising prompts
• Generating responses
• Managing CPU or GPU memory
• Streaming output tokens
• Batching requests
• Managing latency and throughput
• Exposing the model through an APILLM serving is different from LLM training.
• Training creates or updates the model. Serving allows users and applications to interact with the trained model.
• What is LLM Deployment?
• LLM deployment is the broader process of placing an LLM-serving system into an environment where it can be used reliably.
• Serving refers mainly to running the model and generating

Prices, discounts and availability are set by Udemy. We may earn a commission when you purchase through links on this site.