Section 11: Lab: Writing Logs from Python to Elasticsearch
- Why logs should be structured, and the ECS format
- Configure Python logging with ecs-logging
- Send logs to Elasticsearch in bulk and add request context
- Lab: store a Python app's logs with a trace ID for troubleshooting
Section 12: Data Streams, ILM and Index Maintenance
- Data streams for data that grows over time
- Index templates and component templates
- Index Lifecycle Management: rollover and automatic deletion of old data
- Check index and shard sizes and cluster health
- Lab: set a policy that keeps logs for 30 days and then deletes them
Section 13: Lab: Exploring and Visualising with Kibana
- Data views and searching logs in Discover with KQL and ES|QL
- Build visualisations with Lens
- Combine them into a dashboard for monitoring an application
- Lab: a dashboard of request counts, errors and response times
Section 14: Vector Search and Semantic Search
- How keyword search differs from semantic search
- Create embeddings in Python and store them in a dense_vector field
- Run kNN search and combine it with ordinary filters
- The semantic_text field and hybrid search (uses a trial licence)
- Lab: search articles by the meaning of a Thai query
Section 15: Workshop: A Search API with FastAPI
- Design a search API with FastAPI and AsyncElasticsearch
- Endpoints for search, filtering, paging and query suggestions
- Log the API to Elasticsearch and monitor it in Kibana
- Present the work, review it together and wrap up with a pre-production checklist