Introduction to ETL Testing
Presented by,
Nirmaladevi V
ESP-QA ,Comcast
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
ETL Testing-Introduction
• ETL basically stands for Extract Transform Load.
• Extracts data from homogeneous or heterogeneous data sources.
• Transforms the data for storing it in proper format or structure
for querying and analysis purpose
• Loads it into the final target (database, more specifically,
operational data store, data mart, or data warehouse)
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
Contd..
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
Data Warehouse and ETL testing
• Data Warehouse also known as an enterprise data
warehouse (EDW), is a system used for reporting and data
analysis.
• DWs are central repositories of integrated data from one or
more disparate sources.
• They store current and historical data and
are used for creating trending reports
such as annual and quarterly comparisons.
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
Data Warehouse and ETL testing –Contd..
• Many organizations have distributed departments with
different applications running on distributed technology.
• ETL tool is employed in order to make a flawless integration
between different data sources from different departments.
• ETL tool will work as an integrator, extracting data from
different sources; transforming it in preferred format based on
the business transformation rules and loading it in cohesive
DB known are Data Warehouse.
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
Testing the Data Warehouse Architecture
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
DB Testing
• DB Testing is to validate the Databases effectively to ensure
secure and quality databases.
• How to test Database?
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
ETL Testing Vs DB Testing
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
Why ETL Testing ?
• ETL Testing preferred in the business case where a kind of
reporting need is sought by the clients.
• The reporting is sought in order to analyze the demands, needs
and the supply so that clients, business and the end users are
very well served and benefited
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
Real Life ETL Life Cycle
• Cycle Initiation
• Build reference Data
• Extract (from sources)
• Validate
• Transform (Apply business rules,check for Data integrity, Create
Aggregates or disaggregates)
• Load (Load into staging tables,if used)
• Audit Reports (for example, on compliance with business rules.
Also, in case of failure, helps to diagnose/repair)
• Archive and Clean up
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
ETL Testing Process
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
Documents needed for ETL testing
ETL mapping sheets :An ETL mapping sheets contain all the
information of source and destination tables including each and
every column and their look-up in reference tables. ETL mapping
sheets provide a significant help while writing queries for data
verification.
DB Schema of Source, Target: It should be kept handy to verify
any detail in mapping sheets.
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
ETL Testing Benefits
• Ensures higher data quality
• Lower risk and greater efficiency
• Minimizes cost of Data/ETL/DW projects
• Improves productivity
• Expedites time to market
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
Challenges in ETL testing
• Data Completeness
• Data Integrity
• Data Transformations
• Data Consistency
• Referential Integrity
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
ETL Tools
• Informatica PowerCenter
• IBM Datastage
• Ab Initio
• Microstrategy
• Oracle Data Integrator (ODI)
• Microsoft SQL Server Integration Services (SSIS)
• Pentaho Kettle
• Talend
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
Informatica
• Source Analyzer
• Target Designer
• Mapping Designer
• Task Designer
• Workflow Designer
Copyright © 2014 Tata Consultancy Services Limited
Source Analyzer
17-Dec-2014 Copyright © 2014 Tata Consultancy Services Limited
Mapping Designer
17-Dec-2014 Copyright © 2014 Tata Consultancy Services Limited
Workflow designer
17-Dec-2014
Copyright © 2014 Tata Consultancy Services Limited
Workflow monitor
17-Dec-2014 Copyright © 2014 Tata Consultancy Services Limited
Copyright © 2014 Tata Consultancy Services Limited
