WHY IS DATA AUDITING IMPORTANT?
Analysis of the data lineage report may identify orphaned records in the source system, requiring further analysis. The auditing process aims to identify specific reasons why data has not moved form the source system to the target system. Typically, we conduct this analysis through profiling the source data and the target data. When we have identified the causes for orphaned records, the client can discuss these issues with the ETL design team and business subject matter experts to determine whether the records should be flowing to the target system or not. In many cases poor ETL design will lead to the accumulation of incorrectly orphaned records in the source system. If it is confirmed that the records should not be flowing to the target system, then either ETL documentation should be updated or the records should be archived/removed from the source system. If it is confirmed that the records should be in the target system, then ETL changes will need to be made so that the data is moved to the target system in the future.
Data Auditing allows you to understand what process your data may have been through. Any unanswered questions surrounding your data can be investigated, and further context can be given to the data. Auditing your data shows you what it has been through to get to the target system and what transformations it may have been through. This information is important to know should you ever need to replicate the path from A to B if something goes wrong or if you identify data of bad quality in your target system. In any of these instances you would need to know not only where your data came from, but also any processes that it had been though. Understanding and mapping your data in this way allows you to work out where any errors have occurred. Imagine a map: Lineage is the line showing your path from A to B. Auditing fills in the area around your line showing you the paths and processes of the terrain that your data has travelled through to create your path from A to B.