Local LLM Deployment with Ollama and vLLM

Section 1: Why Run LLMs Yourself, and the Tool Landscape

• Why organisations self-host: confidentiality, PDPA, cost and offline operation
• The trade-offs compared with cloud models: quality, operational effort and hardware cost
• An overview of inference engines (Ollama, llama.cpp and vLLM) and a reference architecture for an in-house LLM service
• Lab: install Ollama, pull a first model and chat from the command line

Section 2: Choosing the Right Open-Weight Model

• Common model families such as Qwen, Llama and Gemma, and Thai models such as Typhoon
• Base, instruct and reasoning models, and when to use each
• Reading model cards and licence terms before commercial use
• Testing Thai ability with questions drawn from the organisation's real work
• Lab: compare the Thai answers of three models on the same task

Section 3: Lab: Quantisation and the GGUF Format

• What quantisation is and why large models can run on small machines
• Common GGUF levels such as Q4_K_M, Q5_K_M and Q8_0
• GPU formats such as AWQ, GPTQ and FP8 used with vLLM
• How quantisation affects answer quality, speed and memory
• Lab: run the same model at two precision levels and compare the results

Section 4: Workshop: Sizing GPUs and Memory

• Estimating weight memory from parameter count and precision
• What the KV cache is and how it grows with context length and concurrent users
• Running on CPU, one GPU, several GPUs or Apple Silicon, and how they differ
• Choosing GPUs and cloud instances that make sense for the workload
• Workshop: build a sizing sheet for your own organisation's use case

Section 5: Lab: Ollama and a Standard API

• Build an organisation model with a Modelfile: system prompt, template and parameters
• Import GGUF files from Hugging Face and set context length, keep-alive and concurrent requests
• Run Ollama in Docker with a GPU and expose it to other machines on the network
• Call it through the OpenAI-compatible API by changing only the base URL of existing code
• Structured output, tool calling and embeddings with in-house models
• Lab: a Python script that extracts data from documents without sending data off the machine

Section 6: Lab: vLLM on a GPU and Tuning

• Why vLLM handles many concurrent users: PagedAttention and continuous batching
• Install it on Google Colab or a cloud GPU, and run it with the official Docker image
• Start an OpenAI-compatible server with vllm serve, with an API key, tool calling and structured output
• Balance max model length and GPU memory utilisation
• Use AWQ, GPTQ or FP8 models and enable prefix caching to get more from the GPU
• Tensor parallelism across GPUs, and serving several LoRA adapters on one base model

Section 7: Lab: Benchmarking and Choosing Ollama or vLLM

• The metrics that matter: time to first token, tokens per second and total throughput
• Load testing with the vLLM benchmark tools and a concurrent request script
• Reading results as users increase, and finding where the system starts to slow down
• Criteria for choosing Ollama or vLLM based on team size and hardware
• Lab: measure the same model on both systems and turn the results into a recommendation

Section 8: Lab: An Organisation Chat Interface with Open WebUI

• Install Open WebUI with Docker Compose and connect it to Ollama and vLLM
• Manage users, roles and access to each model
• Upload documents for basic question answering with an internal model
• Set system prompts and ready-made model presets for each department
• The Open WebUI licence terms to know before using it in an organisation

Section 9: Security and Operations

• Close unneeded ports and put a reverse proxy with TLS and API keys in front
• Segment the network, restrict callers and keep logs in line with data policy
• Monitoring with vLLM metrics, Prometheus and Grafana
• Update models and inference engines safely without disrupting users
• Lab: build a dashboard for GPU usage and response times

Section 10: Capstone: An In-House LLM Service

• Take a brief from a sample organisation with data and budget constraints
• Choose the model, quantisation level and inference engine, with reasons
• Assemble the system with Docker Compose: inference, Open WebUI and reverse proxy
• Test usage and load, then present a plan for scaling
• Wrap up with a checklist before opening the service to the whole organisation