Section 6: How ETL Tools Work
- Extract, transform and load in terms a BA/SA can use
- ETL versus ELT and what it means for project planning
- Getting to know KNIME: nodes, workflows and inspecting results step by step
- Why migration must be repeatable rather than fixed by hand
Section 7: Lab: Building a Migration Workflow
- Read data from several sources: CSV files, Excel and databases directly
- Cleansing: trim spaces, normalise text case, convert dates and standardise phone numbers
- Join tables, look up values from lookup tables and pick a golden record from duplicates
- Validation rules and routing failed records to an exception file
- Write data to the target in the right order for related tables
Section 8: Handling Problem Records
- Design the exception process: who fixes what, how, and when records go back in
- Error reports that data owners can read and act on themselves
- Controlling iterative loads until the acceptance criteria are met
- Keeping an audit trail of corrections
Section 9: Reconciliation and Proving Correctness
- Levels of checking: record counts, financial totals, checksums and record-level sampling
- Write reconciliation queries in SQL to compare source and target
- Explaining acceptable differences with documented reasons
- Checking referential integrity in the target system
- A reconciliation report for management and internal audit
Section 10: Testing and Data Acceptance
- Mock runs and how many rehearsals to do before the real move
- Designing UAT for data: what users check, how and how many records
- Sign-off documents with clear accountability
- Timing the real migration run to size the cutover window
Section 11: Cutover and Rollback Planning
- Structure of a cutover plan: sequence of tasks, timings, owners and decision points
- Freeze periods and handling data created during the migration window
- Delta migration before the new system goes live
- Go / no-go criteria and a rollback procedure that can actually be carried out
- Hypercare after go-live and formal project closure
Section 12: Master Workshop: NovaRetail Migration
- Build a KNIME workflow that transforms customer, product and sales data from the Day 1 mapping
- Load the target database and handle records that fail validation
- Produce a reconciliation report comparing source and target
- Workshop: write a cutover plan with go / no-go criteria and present it to the group