Data Engineering Bootcamp

Section 1: The Data Engineer's Role and Data Platform Architecture

• How data engineers, data analysts and data scientists differ
• Data warehouse, data lake and lakehouse compared
• ETL and ELT, and the medallion architecture (bronze, silver, gold)
• When to choose batch and when to choose streaming

Section 2: Lab: Setting Up the Data Stack with Docker Compose

• The structure of a docker-compose.yml for a multi-service data stack
• Volumes, networks and environment variables
• Checking service status and reading logs when something goes wrong
• Allocating RAM and CPU to suit the learner's laptop
• Lab: start the MetroMart data stack and check that every service is ready

Section 3: Lab: SQL for Analytics

• A review of JOIN, GROUP BY and subqueries
• Common table expressions (CTEs) and window functions such as ROW_NUMBER, LAG and SUM OVER
• Reading execution plans and basic indexing
• Lab: analyse running sales totals and product rankings for MetroMart

Section 4: Dimensional Modelling (Star Schema)

• Fact and dimension tables and defining the grain
• Surrogate keys and natural keys
• Slowly changing dimensions (SCD Type 1 and Type 2)
• Designing a date dimension

Section 5: Workshop Day 1: The MetroMart Star Schema

• Load the generated MetroMart data into PostgreSQL
• Design fact_sales, dim_product, dim_customer, dim_store and dim_date
• Define the grain and keys of each table, with the reasoning
• Workshop: write SQL that analyses sales by store, product category and period

Section 6: Batch Ingestion with Python

• CSV, JSON and Parquet file formats, with their strengths and limits
• Full and incremental loads, and using watermarks
• Designing idempotent pipelines that give the same result when re-run
• Lab: pull product and customer data from PostgreSQL incrementally

Section 7: Lab: Polars and DuckDB

• Polars DataFrames, the lazy API and expressions
• DuckDB querying Parquet and CSV files directly with SQL
• Comparing with pandas and choosing a tool for the size of the data
• Lab: convert branch CSV files to Parquet with Polars and check them with DuckDB

Section 8: Event Streaming with Apache Kafka

• Topics, partitions, offsets, producers, consumers and consumer groups
• Kafka 4 runs on KRaft without ZooKeeper
• Delivery semantics: at-most-once, at-least-once and exactly-once
• Lab: create a topic and inspect partitions and offsets with the command-line tools

Section 9: Change Data Capture (CDC)

• How CDC reads a database's transaction log
• Kafka Connect and Debezium with PostgreSQL (demonstration)
• Managing event schemas and schema evolution
• When to use CDC instead of batch extraction

Section 10: Workshop Day 2: MetroMart Ingestion

• A batch pipeline that pulls data from PostgreSQL and branch CSV files into Parquet
• A Python producer that simulates order events into a Kafka topic
• A consumer that writes the events to the raw data area
• Workshop: re-run the pipeline and stop the consumer mid-way, then check that no data is duplicated or lost

Section 11: Object Storage and the Lakehouse

• S3-compatible object storage, buckets and permissions
• The problems of a file-only data lake, and why a table format is needed
• Apache Iceberg, Delta Lake and Apache Hudi compared
• Lab: create buckets and permissions for bronze, silver and gold

Section 12: Lab: Apache Iceberg in Depth

• The structure of metadata, manifests and snapshots
• The REST catalog and connecting from several engines
• Schema evolution, partition evolution and hidden partitioning
• Time travel and rolling back data
• Lab: add a column and look back at the data before the change with time travel

Section 13: Lab: Apache Spark and PySpark

• Spark architecture: driver, executors, jobs, stages and tasks
• The DataFrame API and Spark SQL
• Transformations, actions and lazy evaluation
• Reading and writing Iceberg tables with Spark 4
• Lab: read the Spark UI to see the stages and shuffles of a real job

Section 14: Designing the Bronze and Silver Layers

• Cleaning data and handling nulls and duplicates
• Merging and upserting data with MERGE INTO
• Handling small files (compaction) and table maintenance
• Lab: compact an Iceberg table and expire old snapshots

Section 15: Workshop Day 3: The MetroMart Lakehouse

• Store the raw data from Day 2 as Iceberg tables in the bronze layer
• Write PySpark jobs that clean the data and remove duplicates
• Upsert into the silver layer with MERGE INTO
• Workshop: try schema evolution and time travel on the order data

Section 16: dbt Core Fundamentals

• The structure of a dbt project, profiles and adapters
• Models, sources, ref and the dependency graph
• Materialisations: view, table, incremental and ephemeral
• Lab: build staging models from the silver layer

Section 17: Lab: dbt for Enterprise Work

• Layering models into staging, intermediate and mart
• Jinja, macros and packages
• Snapshots for SCD Type 2
• dbt Docs and the lineage graph
• Lab: build fact_sales and the dimensions, with a customer snapshot

Section 18: Apache Airflow 3 Fundamentals

• Airflow 3 architecture: scheduler, API server, DAG processor, triggerer and workers
• Writing DAGs and tasks with the TaskFlow API
• Assets and data-aware scheduling
• Following runs in the Airflow 3 UI

Section 19: Orchestration for Real Workloads

• Retries, timeouts, callbacks and Deadline Alerts for late-running work
• Passing values between tasks and handling connections and variables securely
• Running Spark and dbt jobs from Airflow
• Backfills and re-running past intervals

Section 20: Workshop Day 4: An Automated Pipeline

• Write a DAG that runs ingestion, PySpark jobs and dbt in dependency order
• Set retries and alerts for failed tasks
• Use assets so the gold layer runs when the silver layer is updated
• Workshop: make a task fail on purpose, fix it and re-run only the part that failed

Section 21: Lab: Data Quality and Data Contracts

• Dimensions of data quality: completeness, accuracy, freshness and uniqueness
• Generic and singular dbt tests, and source freshness
• Data contracts between data producers and consumers
• Lab: add a quality gate that stops the pipeline when data looks wrong

Section 22: Observability and Running Pipelines

• Tracking logs, metrics and run durations
• Designing alerts so the team learns of problems quickly without alert fatigue
• Incident response for data pipelines
• Runbooks for common incidents such as late data or schema changes

Section 23: Lab: The Serving Layer for BI and AI

• Designing marts that suit dashboards
• Connecting Metabase or Apache Superset, and how to connect Power BI
• Preparing data for AI teams, such as feature tables
• Lab: a daily sales dashboard by store and product category

Section 24: Taking It into the Organisation

• Basic data governance, data ownership and access rights
• Moving the stack to the cloud or Kubernetes
• Advanced stream processing with Apache Flink and Kafka Streams (overview)
• Planning the growth of a data engineering team

Section 25: Master Workshop: The MetroMart Data Platform

• Combine batch and streaming ingestion, the Iceberg lakehouse, PySpark, dbt and Airflow into one pipeline
• Add data quality gates and alerting
• Test scenarios with malformed data and late-arriving data
• Show the results on the dashboard and present the team's architecture