Section 6: Measuring and Improving RAG Quality
- Why RAG answers wrong: nothing retrieved, wrong retrieval, or the model ignoring the provided context
- Two levels of quality: retrieval quality and answer faithfulness
- Building an evaluation set from real work
- How to trace causes and improve the pipeline point by point
Section 7: Advanced Retrieval
- Hybrid search: combining keyword and semantic to cover both exact terms and meaning
- Reranking: re-ordering retrieval results to better match the question
- Query rewriting and multi-query for vague or broad questions
- Metadata filtering and scoped search, such as by department or time period
Section 8: Building a RAG Application and Chatbot
- Assembling retrieved context into the prompt and designing a system prompt for RAG
- Forcing AI to answer with source citations and to say not found when it is not in the knowledge base
- Handling multi-turn conversation and retrieval based on conversation context
- Workshop: build a RAG chatbot that answers from the knowledge base and shows its sources
Section 9: Connecting RAG to Real Systems
- Connecting real data sources via MCP (Model Context Protocol): Google Drive, SharePoint, Notion and databases
- Setting a workflow to ingest documents and update the knowledge base automatically with n8n
- Keeping the knowledge base current with re-indexing, and delivering RAG through real channels such as web, LINE or Microsoft Teams
- Workshop: set a workflow to ingest new documents into the knowledge base automatically
Section 10: Security, Access Control and PDPA
- Controlling access to knowledge base data (who can see which documents)
- Classifying document confidentiality and preventing data leaks through AI answers
- Personal data under PDPA and handling it before documents enter RAG
- Cloud versus on-premise / self-hosted for sensitive data, and maintaining an audit trail
Section 11: Capstone Project and Real-world Adoption
- Task: build an end-to-end AI assistant that knows the organization from a real document set
- Prepare documents, index, improve retrieval quality and answer with citations, then test with real questions, measure quality and improve until usable
- Present the work and get feedback, plus a first-30-days adoption plan and extending to enterprise knowledge and RAG-based AI agents