Revealing the past - the importance of data lineage
Lineage is a map of the journey between source, transformation and destination of data items. Some experts have likened it to a family tree, revealing ancestry. In the same way that it takes time and work to trace your family tree, finding out where data has come from is challenging.
So why is it important to data engineers?Read more
Determining where to start
Data is crucial for a business to survive. In complex companies, it is not always clear where data can be sourced from and, more importantly, how reliable that data is. Data Lineage mapping can give a clear understanding of where data is stored, how it is stored and what the transformation journey is into downstream systems.
When a data project is sourcing data, a data lineage map can reduce the potential for duplicated data sources. Without this map, duplication is likely, with its attendant costs and risks.
If you have a complex set of systems with significant data being parsed between them, data lineage mapping is incredibly important to ensure that successful maintenance can be undertaken. The lineage should show the source, transformation journey and the target stores. The change process can then be implemented far more efficiently.
If you want to get somewhere, consult a map. If you want a successful data engineering project, have a data lineage map!Read less