Data Pipeline with Apache Airflow and dbt is a course in building ELT data pipelines, from writing DAGs and scheduling jobs to loading a warehouse, transforming data with dbt, testing data quality, incremental models and running Airflow with dbt in production. It suits data engineers, data analysts, BI developers and DBAs who can already write SQL.
Many organizations already own BI tools such as Power BI or Tableau, yet still struggle with data preparation: pulling data from several systems by hand, cleaning it in Excel, and constantly finding that numbers do not match between reports. This course fills the data engineering layer that sits between source databases and reports, using two industry-standard tools: Apache Airflow for scheduling and orchestrating jobs, and dbt for transforming data with SQL plus automatic testing and
documentation. Learners build a real pipeline end to end: designing an ELT architecture, writing Airflow DAGs, ingesting data from multiple sources into a warehouse, transforming it into staging and mart layers with dbt, writing tests that catch bad data before users see it, building incremental models that process only new data, and finally connecting Airflow with dbt and taking the pipeline to production with monitoring. (3 days, 6 hours per day, 18 hours in total, Intermediate level.)
Course Details
A 3-day course, 6 hours per day (18 hours in total), delivered as lectures with hands-on labs for every topic. Intermediate level. Learners take home a lab guide, complete DAG code and dbt project samples, practice datasets and a data quality checklist. All tools are open source and can be used in the organization afterwards.
Section 1: Modern Data Architecture
Common problems: manual extraction, mismatched numbers and unknown data provenance
ETL versus ELT and why ELT dominates with modern warehouses
Data platform components: sources, ingestion, warehouse, transformation and BI
Designing staging, intermediate and mart layers
Section 2: Installing and Using Apache Airflow
Airflow components: scheduler, webserver, worker, metadata database and executor
Installing Airflow with Docker Compose and using the web interface
DAG, task and task instance concepts and reading job status on screen
Lab: install Airflow and run your first DAG
Section 3: Writing DAGs and Operators
Writing DAGs in Python and using the modern TaskFlow API
Common operators: Python, Bash, SQL and sensors for waiting on conditions
Passing values between tasks with XCom and using variables and connections securely
Lab: write a DAG that pulls data from a file and an API and stores it in a database
Section 4: Scheduling and Handling Failures
Setting schedules and understanding data intervals and catchup
Defining task dependencies and running tasks in parallel
Automatic retries, failure notifications and backfilling historical runs
Lab: schedule a daily job and test failure and backfill scenarios
Section 5: Ingesting Data into the Warehouse
Extracting from relational databases, CSV files and APIs
Full load versus incremental load and when each applies
Handling mid-run failures and making jobs idempotent
Lab: build a DAG that ingests several sources into the raw layer
Section 6: Introduction to dbt and Building Models
What dbt is and how it turns SQL transformation into disciplined engineering
Setting up a dbt project, profiles and the warehouse connection
Writing models, using ref and source, and view versus table materialization
Lab: build staging models from the raw data you ingested
Section 7: Data Testing and Documentation
Built-in tests: not_null, unique, accepted_values and relationships
Writing custom SQL tests for business rules
Generating documentation automatically and reading lineage graphs
Lab: add tests and documentation to your models and watch a test catch bad data
Section 8: Incremental Models and Project Structure
Building incremental models to process only new data and cut runtime
Structuring the project into staging, intermediate and mart layers
Using macros and Jinja to avoid repeating SQL, and seeds for reference tables
Lab: convert a model to incremental and measure the time saved
Section 9: Connecting Airflow with dbt
Patterns for invoking dbt from Airflow and separating run and test steps
Designing an end-to-end DAG: ingest, transform, test and notify
Managing dependencies between ingestion and transformation jobs
Lab: build a full pipeline where Airflow drives dbt automatically
Section 10: Data Quality and Monitoring
Placing quality gates at each layer and deciding when to stop the pipeline
Alerting on data anomalies or pipeline failure via email or team channels
Tracking data freshness and communicating with report users
Lab: add quality checks and alerting to your own pipeline
Section 11: Going to Production
Separating dev and production environments and handling credentials safely
Keeping pipeline code in Git and reviewing through pull requests
CI/CD approaches for data work and testing before production
Planning long-term pipeline maintenance and handover to the team
Section 12: Capstone and Next Steps
Capstone: build a complete pipeline from raw data to BI-ready tables
Present your architecture and the reasoning behind your decisions
Moving on to cloud data warehouses and other data engineering tooling
Workshop: present the pipeline you built and review it with the instructor
No public rounds are open right now. Join the waiting list and we will contact you first when the next round opens, or ask us on LINE. Or call 02-570-8449 or 088-807-9770
ETL transforms data before loading it into the warehouse, while ELT loads raw data first and transforms it inside the warehouse with SQL. ELT has become popular with cloud warehouses that process data quickly and scale easily, because the raw data is kept and can always be reprocessed, and ingestion is separated from transformation. Tools such as dbt are built for the transform step in this approach.
A DAG (directed acyclic graph) is a set of tasks with a defined order and no loops. You write it as a Python file in the dags folder, set the schedule and start date, and create tasks with operators such as PythonOperator or BashOperator, or with the @task decorator of the TaskFlow API. Set the order with the >> operator; the scheduler then runs it on time and you can follow each task's status in the web interface.
Declare tests in a YAML file alongside your models. dbt has four generic tests, not_null, unique, accepted_values and relationships, for missing values, duplicates, allowed values and links between tables. For business rules, write a SQL query that returns the rows breaking the rule; if any rows come back, the test fails. Run dbt test or dbt build, and place tests in the pipeline so bad data stops before it reaches reports.
It suits data engineers and people moving into data engineering, data analysts and BI developers tired of preparing data by hand, and DBAs who supply data to analytics teams. You should be able to write working SQL, including SELECT, JOIN and GROUP BY, understand relational databases, and read and edit basic Python. Experience with Docker and Git makes the labs smoother.
You will be able to design staging, intermediate and mart layers, write DAGs that load data from databases, files and APIs into a warehouse on schedule, handle failures with retries and backfills, build dbt models with tests and documentation, use incremental models to cut processing time, and orchestrate dbt from Airflow as an end-to-end pipeline that alerts on data problems. Participants who complete the training receive a Certificate of Completion with its own number, which can be verified online.
Yes. It can be run in-house for your team, with examples adapted to your own work, such as building pipelines from the data sources and reports your team already looks after. You can request a quotation from this page, or contact us on LINE @itgenius or by phone on 02-570-8449.
Producing video content every day is one of the most time-consuming tasks for business owners and marketing teams today. Editing a single clip with click-and-drag tools (CapCut, Premiere Pro) takes 1-3 hours, including cutting silences, adding subtitles,…
Systems wired together with direct point-to-point calls turn into a tangle as services multiply, and when a downstream system goes down the data goes with it. Apache Kafka solves this by acting as a durable central hub for events, so producers and consumers…
