taken me a long time to get to where I am. I started small and now I’m so big that people often refer to me as ‘big data’. I still, however, have my needs and the same aims in life: I want to be reliable, dependable, available and adapt to change; I don’t necessarily want to be perfect, but fit for your use; I don’t want to be too needy and expensive to keep, otherwise you would archive or replace me.
Please remember, if you treat me well I will be faithful and true. Treat me unkindly and I will do whatever I can to make your life frustrating. Just think what I could do if I ever get to drive vehicles for you!
If you remove my competition, I promise I will be stronger; I do not like to share the limelight with inferior spreadsheet copies of me. If you put garbage into me, then I won’t be happy and will only send even more garbage back at you.
If you care for me, then I will never become out of date; I have the ability to change if you allow me to. If I change, however, I want to be sure I’ve changed for the right reasons and not just because you think I will look better if I was different, or that I will fit into your perceived view of me. After all, I am what I am and we will work together better if we fully understand each other.
Managing Data Quality
22
If you have not specified what I need to look like, then I have no idea how to behave or what to tell you; I’ll try and predict this and tell you what I think you want to know, which perhaps will not be right.
I can be so clever if I am set up correctly; I can autocorrect, provide you with selectable values, undo and even auto-archive. But if you don’t take care, I have the ability to ruin your day. I have a whole catalogue of sayings to throw at you. Here are some of my favourites:
invalid key stroke;
invalid data format;
command not recognised;
no search results found;
data out of range;
can’t undo;
last session did not save.
Sometimes I get abused. The finance department often reshapes me and turns me into something I don’t recognise of myself; I think they call this accountancy. These days I’m used a lot in business intelligence, or ‘BI’ for short; they don’t like what I tell them most of the time.
The most fun I ever had was when someone put me through a data migration project. Someone thought I would be very different and much improved as a result, but no, I’m still the same. At least I have a new shiny coat to wear and I fared better than some of my cousins; one of them became schizophrenic after he was part of failed data migration. He has never been the same since!
I was so chuffed recently when I was honoured with the title of being the ‘single version of the truth’, until I found out all my brothers and cousins were also receiving this award.
So, please be very careful when handling me. I do not like any of the following:
1. Being changed – unless my physical representation has also changed, and you have been given permission to change me. I hate to be ‘dressed up’ or made to look good because I don’t give you the answer you want to see; let me be me.
2. Being shared – unless I let you. I might be sensitive and in retaliation I could undermine your business advantage.
3. Being taken for granted – understand me and my ancestors. My provenance could increase our like for each other.
4. Being copied unnecessarily – I want to be unique and original.
Challenges when exploiting and managing data
23
5. Being forced to look different from time to time – I like to be consistent.
6. Bits of me being removed – I like to be complete.
7. Incompetent people being near me – I will tell them lies or, even worse, I will delete myself and not come back.
8. The assumption I want to be housed in an expensive system – I am equally happy in a simple spreadsheet if I’m properly looked after and appreciated.
9. Being ignored – because if you show me you don’t care I will just not help you and I might even become obsolete.
10. Unstructured data – don’t get me started! Oh, how I hate whiteboards.
In summary, if you handle me correctly, I promise I will serve you in the right way and shall be around as long as you need me. Treat me badly and I promise I can, and will, mess up your decisions. I do like to be user friendly, so can you please be data friendly?
The challenge of managing enterprise data quality
This chapter has illustrated a number of factors and considerations that show why data quality is difficult to manage at an enterprise level. These can be summarised as follows:
There can be many data stores, particularly if you include unofficial ones, locally and in the cloud.
The number of data stores grows rapidly (and uncontrollably) as people create new spreadsheets and exploit cloud data stores.
Ownership and stewardship of these data stores is weak, if present at all.
Similarly, there can be many different software systems that include their own data stores and also undertake many data updates.
There are many different business processes that use and update data and are run by different parts of the organisation.
There are many users of systems and processes, some of which do not have the correct view of what data are required, what the process is or where the correct place to store data is.
It is virtually impossible to ‘rewind’ data back to the point at which they were ‘good’ in order to resolve data quality problems.
Data exploitation might not use the correct balance of tools, subject matter expertise and awareness of data quality, leading to worse perceptions of data quality. Poorer quality data might result in a different choice of tools to exploit it.
Users could have different levels of diligence or personal business need, with some users only entering the bare minimum of data into any system they interact with.
Managing Data Quality
24
Historic or legacy interventions can have created many data issues that are unlikely to be corrected. Such interventions include, for example, poorly defined and governed business processes, poor quality data migrations or bulk updates to data that have resulted in data corruptions.
Summary
Data flow across organisations and become of interest to many different processes, individuals and teams.
Technology is only a tool to help deliver effective data quality management, and
