Course Details
A 3-day course, 6 hours per day (18 hours in total), delivered as lectures with hands-on labs for every topic. Intermediate level. Learners take home a lab guide, Prometheus and Alertmanager configuration files, sample Grafana dashboards and a set of commonly used PromQL queries. All tools are open source and can be deployed in the organization at no licensing cost.
Day 1: Prometheus Fundamentals and Metric Collection
Section 1: Observability and Monitoring Concepts
- The difference between monitoring and observability and why organizations need both
- The three pillars: metrics, logs and traces, and choosing the right one for each problem
- Key indicators: the four golden signals and SLI, SLO and error budget concepts
- How Prometheus, Grafana and Alertmanager fit together
Section 2: Installing and Configuring Prometheus
- The pull-based architecture and the Prometheus time series database
- Installing Prometheus with Docker Compose and the structure of prometheus.yml
- Metrics, labels and time series concepts and good metric naming
- Lab: install Prometheus and monitor itself as the first target
Section 3: Collecting Metrics with Exporters
- Collecting server metrics with Node Exporter (CPU, memory, disk, network)
- Monitoring containers with cAdvisor and collecting database and web server metrics
- Metric types: counter, gauge, histogram and summary, and when to use each
- Lab: install several exporters and verify Prometheus is collecting everything
Section 4: Service Discovery and Target Management
- Static target configuration and its limits as the system grows
- Service discovery from files, DNS and cloud providers, and using it with Kubernetes
- Using relabeling to organize labels and filter targets
- Lab: configure service discovery so new targets are monitored automatically
Day 2: PromQL and Grafana Dashboards
Section 5: PromQL Fundamentals
- Selecting data with selectors and filtering with label matchers
- Common functions: rate, increase, sum, avg, max and using by and without
- Calculating percentiles from histograms and spotting anomalies
- Lab: write PromQL to answer real questions such as error rate and resource usage
Section 6: Grafana and Building Dashboards
- Installing Grafana and connecting Prometheus as a data source
- Panel types and choosing the right visualization for each kind of data
- Using variables to build dashboards that can switch server or service
- Lab: build a dashboard monitoring your own servers and containers
Section 7: Designing Dashboards That Work
- Design principles so a dashboard immediately shows whether the system is healthy
- Tailoring dashboards by audience: executives, operations teams and developers
- Importing community dashboards and adapting them to the organization
- Lab: refine a dashboard so it communicates system status on a single screen
Section 8: Application Monitoring
- Instrumenting an application with a client library and exposing a metrics endpoint
- Measuring the four golden signals: latency, traffic, errors and saturation
- External monitoring with Blackbox Exporter to simulate the user perspective
- Lab: add metrics to a sample application and build a dashboard to track it
Day 3: Alerting, Storage and Capstone
Section 9: Writing Alerting Rules
- Writing alerting rules in PromQL and using the for duration to avoid noisy alerts
- Setting severity levels and including the context the team needs to fix the problem
- Symptom-based alerting: alert on what users feel, not on every internal metric
- Lab: write alert rules for the servers and services you are monitoring
Section 10: Alertmanager and Notifications
- Installing and configuring Alertmanager and connecting it to Prometheus
- Grouping, silencing and inhibition to reduce alert noise
- Sending notifications to email, LINE, Slack or your organization channels
- Lab: send alerts to a real channel and test an outage scenario
Section 11: Long-term Storage and Performance
- Setting retention periods and estimating the storage required
- Long-term storage approaches and federation as the system scales
- Performance tuning: scrape interval, label count and high cardinality problems
- Backing up and maintaining the monitoring stack itself
Section 12: Logs, Traces and Capstone
- Collecting logs with Loki and viewing logs alongside metrics in Grafana
- Distributed tracing concepts and OpenTelemetry for following requests across services
- Capstone: build a complete observability setup for one service from metrics to alerts
- Workshop: present your dashboards and alert rules and receive feedback