Conformance of data to rules defining the syntax and structure of data.
Value: Numbers or characters stored in a data field to represent an entity or activity.
xv
PREFACE
Data are all around us1; the volume of data is growing at exponential rates and our lives are increasingly being supported and enabled by the exploitation of data. Despite this, many organisations struggle to effectively manage data and the quality of these data.
The reliance of organisations (and society) on data is a relatively new phenomenon; the techniques to manage data effectively are still developing and wider awareness of these approaches is generally low.
This book is titled Managing Data Quality: A Practical Guide very deliberately; its focus is to provide you with an understanding of how to manage data quality, and practical guidance to achieve this.
Enterprise data quality
This book does not just examine quality issues in single data databases or data stores. Instead, we also look at the wider set of issues arising in a typical organisation where there are multiple data stores that are not always formally managed, have been developed at different times, are constrained by different software tools and will be inputs and outputs of many different business processes.
Managing this landscape of different data stores is complex enough when there is a lack of agreement over which is the most trusted, or ‘master’, data source. This complexity increases, however, in most organisations where a large amount of data are also gathered, stored and manipulated in user-created spreadsheets and databases that often exist ‘below the radar’ of corporate governance approaches and controls.
Depending on the organisational context, this chaotic landscape presents a range of risks (and issues) to the organisation, which might be financial, regulatory, commercial,
Keith Gordon’s book, Principles of Data Management (2013) also takes an enterprise view of data. Keith’s book was published before ISO 8000-61, the international standard that specifies a process reference model for data quality management. This process model is the basis for our approach to enterprise data quality.
1 Please note, some readers will generally use the word ‘data’ in the singular; the BCS convention is to use this word as plural.
xvi
legal or reputational. These risks and issues could be significant. Standing still is almost certainly not a viable option.
From the perspective of the enterprise as a whole, therefore, managing data quality effectively can be such a large task that it either never gets started or is viewed as so expensive that it eats up budget that could be better used elsewhere.
This book will help you to overcome this perceived complexity with practical solutions, by understanding:
the nature of the data asset and why it can be difficult to manage;
the impact of people and behaviours on data quality;
the ISO 8000-61 framework and how it defines approaches to data quality management;
how to develop strategies for change that are relevant to your organisation.
Data and changing technology
Over the many years since computing first became a commercial activity, there have been numerous changes in technology. At the highest level has been the progression from mainframes to personal computers, client/server systems, network computers, cloud computing, the Internet of Things (IoT) and big data analytics. Within each of these broad categories, technologies and approaches have continually evolved. Each evolutionary step is often sold on the basis of overcoming the shortcomings of the previous technology. Today’s latest technology, likewise, will be replaced in the future as new user requirements are discovered and improved technological approaches are developed.
Throughout all these changes in technology, data should have been a constant factor. They should have been migrated without loss of meaning from the old technology to the new, so as to sustain the effective delivery of organisational outcomes. However, data migration projects have historically been high risk and likely to fail in terms of time, cost or quality. For example, data can be lost or corrupted as part of the migration process, with such problems possibly affecting significant volumes of data. Similarly, changing data requirements over time can mean that older data structures are no longer fully understood, resulting in corruption during migration. Data migration approaches, such as the one defined by Johny Morris in his book Practical Data Migration 3rd edn. (2020), help organisations to maximise the chances of successful data migration.
For many organisations, the entities that data represent have existed through multiple data stores and software systems; for example:
an individual born in 1950 will have had their personal details, careers, financial records and so on, stored in multiple systems over the course of their lives;
infrastructure assets such as railways, bridges and buildings can be more than 100 years old (with an expectation that they will continue to provide useful service
PREFACE
PREFACE
xvii
for many more years) with data and records about them having been stored in multiple systems.
This book is ‘technology agnostic’, so is not tied to any one particular technology or software system. It details approaches to managing data quality that will stand the test of time regardless of future technology changes and evolving organisational requirements for data.
Intended audience
We intended this book to be both a reference source to be read (and reread) in its entirety and a source of advice and anecdotes that can be ‘dipped in to’ when required. It is written for data managers and the practitioners, supporters and sponsors involved in data quality initiatives.
It is also written for students and lecturers in both computer science and business/management courses who have an interest in, or reliance on, effective data exploitation.
Reference to other works
This book is not attempting to be a definitive guide for all possible data-related activities, many of which are already described in other authoritative works. Instead, we will focus on the challenges of managing enterprise data quality and the ways to refine the management system of an organisation to take adequate account of data. Where a subject already has authoritative and well-regarded reference material, we will refer to these authoritative sources.
