Section 9: Writing Alerting Rules
- Writing alerting rules in PromQL and using the for duration to avoid noisy alerts
- Setting severity levels and including the context the team needs to fix the problem
- Symptom-based alerting: alert on what users feel, not on every internal metric
- Lab: write alert rules for the servers and services you are monitoring
Section 10: Alertmanager and Notifications
- Installing and configuring Alertmanager and connecting it to Prometheus
- Grouping, silencing and inhibition to reduce alert noise
- Sending notifications to email, LINE, Slack or your organization channels
- Lab: send alerts to a real channel and test an outage scenario
Section 11: Long-term Storage and Performance
- Setting retention periods and estimating the storage required
- Long-term storage approaches and federation as the system scales
- Performance tuning: scrape interval, label count and high cardinality problems
- Backing up and maintaining the monitoring stack itself
Section 12: Logs, Traces and Capstone
- Collecting logs with Loki and viewing logs alongside metrics in Grafana
- Distributed tracing concepts and OpenTelemetry for following requests across services
- Capstone: build a complete observability setup for one service from metrics to alerts
- Workshop: present your dashboards and alert rules and receive feedback