DevOps · DOP-06

LLMOps: Evaluation, Observability and CI/CD

LLMOps: Evaluation, Observability and CI/CD is a course in taking RAG and chatbot applications to production with discipline, covering tracing with Langfuse, prompt versioning, evaluation with Ragas and promptfoo, quality gates in CI/CD, monitoring and cost control. It suits AI engineers, DevOps engineers and tech leads who look after LLM applications in production.

Updated
From 8,010 THB / person 8,900 −10% excl. VAT 7% · group rates available
PDFDownload the course outline
  • Duration12 hours · 2 days
  • FormatOnsite / live online
  • Next roundOn request
  • CertificateIncluded

Course overview

Getting a chatbot or RAG system to answer well on a developer's laptop does not take long. Keeping it answering well, consistently, in production is another matter. A small prompt edit quietly makes some answers worse, a model upgrade multiplies the bill, and when users report a wrong answer the team cannot tell which step caused it. Because LLM applications are not deterministic, the usual ways of testing and deploying software are not enough.

This course builds on the RAG and AI chatbot courses and shows learners how to take a RAG application to production in a disciplined way. Learners install Langfuse to trace every step of the app, manage prompts with versions, build evaluation datasets from real data, measure RAG quality with Ragas and test behaviour with promptfoo. They then turn those evaluations into a gate in GitHub Actions, deploy with Docker behind an LLM gateway, and monitor quality, latency and cost after release. The course closes with a capstone that builds a complete pipeline, from a prompt change to production monitoring. (2 days, 6 hours per day, 12 hours in total, Intermediate level.)

What you’ll gain

  • Explain the LLMOps lifecycle and how it differs from MLOps and traditional DevOps
  • Install Langfuse and trace a RAG application end to end, from retrieval to answer
  • Manage prompts with versions, separate production from staging and roll back when needed
  • Build evaluation datasets from real traces and synthetic data
  • Evaluate RAG quality with Ragas and test application behaviour with promptfoo
  • Set up an evaluation gate in CI/CD so changes that lower quality never reach production
  • Deploy LLM applications with Docker and an LLM gateway, with fallbacks and rate limits
  • Monitor quality, latency and cost after release, and put guardrails in place

Who this course is for

  • AI engineers and developers who have built a chatbot or RAG system and need to take it to production
  • DevOps and platform engineers who run LLM applications alongside the development team
  • ML engineers who want to extend their MLOps practice to LLM work
  • Tech leads who set testing and deployment standards for the team's AI applications
  • Graduates of RAG & Knowledge Base or one of the AI chatbot courses

Prerequisites

  • Python, and experience building an LLM application or RAG system, or completion of RAG & Knowledge Base
  • Basic Git, GitHub and Docker skills
  • A basic understanding of CI/CD
  • An LLM API key with credit and a GitHub account of your own, plus a laptop that can run Docker Desktop

Curriculum

Course Details

A 2-day course, 6 hours per day (12 hours in total, 09:00-16:00), alternating lectures with labs on a single sample RAG application throughout. Intermediate level. The course builds on RAG & Knowledge Base and the AI chatbot courses. It does not teach how to build RAG from scratch, and instead focuses on taking an application to production in a disciplined way, covering tracing, prompt versioning, evaluation,

CI/CD, monitoring and cost control, so it pairs well with the DevOps track. Every lab runs on the learner's own machine with Docker. Learners bring their own or company LLM API key with credit and a GitHub account; API usage and GitHub Actions minutes are billed by actual use. Learners take home a lab guide, the sample RAG application, docker-compose files for Langfuse and LiteLLM, evaluation suites and GitHub Actions workflows.

Day 1 Tracing, Prompt Versioning and Evaluation

Section 1: How LLMOps Differs from MLOps and DevOps

  • Problems specific to LLM apps: non-deterministic output, hard-to-measure quality and usage-driven cost
  • The LLMOps lifecycle: develop, evaluate, deploy, monitor and feed real data back into improvement
  • What to measure: answer quality, groundedness, latency, cost and safety
  • Lab: set up the sample RAG app and try to find which step caused a wrong answer without any tooling

Section 2: Lab: Tracing with Langfuse

  • Self-host Langfuse with Docker Compose
  • Instrument the RAG app with the Python SDK to see retrieval, prompts and model calls
  • Traces, spans, sessions, tokens and cost per request, following the OpenTelemetry approach for GenAI
  • Lab: follow the trace of a wrong answer until the problem turns out to be retrieval

Section 3: Lab: Prompt Management and Versioning

  • Keeping prompts in Git compared with Langfuse prompt management
  • Versions and labels such as production and staging, together with model config
  • Link traces to prompt versions to compare results before and after a change
  • Lab: release a new prompt version, then roll it back when a problem appears

Section 4: Lab: Building Evaluation Datasets

  • What a golden dataset is and which kinds of questions it should contain
  • Pull real traces with problems or poor feedback into test cases
  • Generate synthetic questions from documents with Ragas, then have people review them
  • Lab: build a 50-item dataset from the sample app's documents and traces

Section 5: Lab: Evaluating RAG with Ragas

  • Core RAG metrics: faithfulness, answer relevance, context precision and context recall
  • How LLM-as-a-judge works and the weaknesses to watch for
  • Use the individual scores to tell retrieval problems from generation problems
  • Lab: evaluate the RAG app before and after a chunking change and decide whether it really improved

Section 6: Lab: Behaviour Testing with promptfoo

  • Write assertion-based test cases: required keywords, valid JSON and staying on topic
  • Compare several prompts and models in one matrix
  • Use LLM-graded assertions for criteria that rules cannot check
  • Lab: choose the best-value model on quality, speed and cost
Day 2 CI/CD, Production and Cost Control

Section 7: Lab: An Evaluation Gate in GitHub Actions

  • Run the eval suite automatically on every pull request that changes a prompt or code
  • Set pass thresholds and handle score noise from non-deterministic models
  • Post a comparison with the previous run as a pull request comment
  • Lab: a pipeline that blocks the merge when faithfulness falls below the threshold

Section 8: Lab: Deploying with Docker and an LLM Gateway

  • Containerise the RAG app and handle config and secrets safely
  • Use LiteLLM as a central gateway to switch models without code changes
  • Configure fallbacks, retries, timeouts and rate limits per team or per app
  • Lab: deploy the app behind the gateway and simulate a primary model outage to test fallback

Section 9: Lab: Online Evaluation and Monitoring

  • Sample production traces and score them continuously with LLM-as-a-judge
  • Dashboards for quality, latency, error rate and cost
  • Alerts when scores drop or latency and cost spike
  • Lab: close the loop from poor production feedback to a new test case in CI

Section 10: Lab: Controlling Cost and Latency

  • Track tokens and cost by user, feature and prompt version
  • Provider prompt caching, and a semantic cache for repeated questions
  • Model routing: easy tasks to small models, hard tasks to large ones
  • Lab: cut the cost per question, then prove with evals that quality has not dropped

Section 11: Guardrails, Safety and Governance

  • Mask personal data in traces and set retention periods in line with PDPA
  • Input and output guardrails and their impact on latency
  • An incident plan for when an LLM app gives wrong answers or leaks data
  • Workshop: write a release checklist for the team's LLM applications

Section 12: Workshop: Capstone LLMOps Pipeline

  • Take on a RAG app with quality problems and high costs
  • Analyse it with traces, build a dataset and set evaluation criteria
  • Fix the prompt or retrieval through a pull request that must pass the evaluation gate
  • Deploy through the gateway, switch on monitoring and prove the result with numbers

Schedule & training options

For individuals — public rounds

No public rounds are open right now. Join the waiting list and we will contact you first when the next round opens, or ask us on LINE. Or call 02-570-8449 or 088-807-9770

For organisations — in-house / private

  • Tailor the content to your team’s tools and projects
  • Your dates, at your office or live online
  • Quotation with tax ID for procurement
Corporate training quote

Instructors

Frequently asked questions

How is LLMOps different from MLOps and DevOps?

LLM applications bring problems that traditional DevOps and MLOps do not cover: outputs are not deterministic, answer quality is hard to measure and costs vary with token usage. LLMOps therefore adds tracing of every step in a request, versioned prompt management and evaluation with test sets to a loop of building, evaluating, deploying, monitoring and feeding real usage back into improvements.

Which metrics do you look at when evaluating a RAG system with Ragas?

The core metrics are faithfulness, which checks whether the answer sticks to the retrieved documents, answer relevance to the question, and context precision and context recall, which measure retrieval quality. Looking at each score separately shows whether a problem lies in retrieval or generation. Several metrics rely on an LLM as a judge, so people should spot-check some of the results as well.

How do you set up an evaluation gate in CI/CD for an LLM application?

Start by building a golden dataset from problem traces in production and synthetic questions checked by people. Then set up a GitHub Actions workflow that runs the evaluation suite on every pull request that changes a prompt or code. Define pass thresholds, such as a minimum faithfulness score, allowing for variation caused by non-deterministic models, and block the merge when scores fall below them. This course builds such a pipeline as a lab.

Who is the LLMOps: Evaluation, Observability and CI/CD course for, and what should I know beforehand?

It suits AI engineers and developers who have built a chatbot or RAG system and want to take it to production, DevOps and platform engineers who look after LLM applications, and tech leads who set testing standards. You should write Python and have built an LLM application or RAG system before, or completed the RAG & Knowledge Base course, use Git, GitHub and Docker at a basic level, and understand CI/CD concepts.

What will I be able to do after the LLMOps: Evaluation, Observability and CI/CD course?

You will be able to trace a RAG application with Langfuse, version prompts, evaluate RAG with Ragas and test behaviour with promptfoo, add evaluation gates to CI/CD, deploy through an LLM gateway such as LiteLLM with fallback and rate limits, and monitor quality, latency and cost in production. Participants who complete the training receive a Certificate of Completion with its own number, which can be verified online.

What do I need to bring for LLMOps: Evaluation, Observability and CI/CD?

Bring an LLM API key with credit, your own or your company's GitHub account, and a laptop that can run Docker Desktop, because every lab runs on your machine in Docker using one sample RAG application throughout. API usage and GitHub Actions minutes are charged according to actual use.