Data Quality and Data Governance Frameworks
What are they and do I need both?
"How do a data quality and data governance framework relate to each other?”I get asked this question quite frequently and I think it’s a really interesting one, so I’d really like to help you get to the bottom of it. I think the reason it comes up is because people have been doing data quality and worrying about data quality for many more years than they have data governance.And so, they feel very strongly that there are two different frameworks in action. Another common misconception is that the two are the same. This may come from a lack of understanding of what data governance really is, so let’s break it down…..
Is data governance the same as data quality?
The very short answer is no. Data quality is the degree to which data is accurate, complete, timely, and consistent with your business’s requirements. Data governance, in very basic terms, is a framework to proactively manage your data in order to help your organisation achieve its goals and business objectives by improving the quality of your data.
Data governance helps protect your business, but also helps streamline your business's efficiency. It ensures that trusted information is used for critical business processes, decision making, and accounting. And so, if you think about it, data governance vastly provides a fabulous foundation for many data management disciplines, its primary purpose is to manage and improve your data quality.
To put it in much simpler terms, if data was water then…
- Data Quality would ensure the water was clean and prevent contamination
- Data Governance would make sure the right people had the right tools to maintain the plumbing.
So, why would you want two frameworks relating to data quality?
The simple answer is you wouldn’t. This really isn't a question about how you align two frameworks. You should only have one framework and data quality and data governance should be working in harmony with one another – not against or in opposition.
Data governance and data quality rely very much on each other, I usually describe the relationship between them as symbiotic, as their relationship is based on mutual interdependence. Therefore, of course, you need both! You would not want to do one without the other if you want to successfully manage and improve the quality of your data in a sustainable manner.
Sadly, in my experience, some organisations do not yet fully understand that you do need to do both. Whilst you rarely (if ever) come across a company that is implementing a data governance framework without the intention to improve data quality, it is fairly common for organisations to commence data quality initiatives without implementing a data governance framework to support them. Unfortunately, this leaves many data quality initiatives as merely tactical solutions that only have short-term results.
And, it doesn’t matter whether you call it data quality or data governance (because let's face it, some people really react badly to the term data governance) as long as it gets your business users engaged and understanding what that framework is about.
So, let's just have one data quality framework which encompasses the roles and responsibilities around data, and then there is nothing to go wrong, no duplication, no gaps between two different frameworks. Make this simple and make it sustainable.
You can see the video I originally did on this topic here and if you've got any questions you’d like me to address in future videos or blogs, please just email them in to firstname.lastname@example.org.
10 Jun 2020 by Nigel Turner, Global Data Strategy
Charles Dickens published his novel, “Our Mutual Friend,” in 1864. It’s safe to say he was probably not thinking of data strategy and its relationship to business strategy when he wrote it. But it is a simple fact that in our digital, data-driven world, business and data strategies can only succeed if they are closely interlocked and nurtured as mutual friends.
Before the rise of the data-driven business, the relationship between business and data strategies was linear. The business would set out its strategic goals and aspirations. Once these were determined, a data strategy could then be built to plan the data capabilities needed to underpin the business strategy and plan. This also implied that IT departments were often the primary drivers of a data strategy.
The idea that data is a subservient enabler is outmoded.
The idea that data is a subservient enabler is outmoded.
The idea that data is a subservient enabler is outmoded.
Today, things have changed. The idea that data is a subservient enabler of the business, useful only to support business operations and processes, is becoming increasingly outmoded. On the contrary, in a growing number of organisations data is becoming the business.
This has radical implications for the relationship between business and data strategies. In this new paradigm, the development of business and data strategies has to be done in parallel and interdependencies between them locked in. Any data-driven organisation (and what organisation isn’t these days?) which fails to recognise this mutuality is doomed to fail. This also means that any aspiring data-driven, digital organisation must create and implement a data strategy, something surprisingly many have still failed to do.
For example, take a manufacturing business producing a range of consumer goods. Traditionally it focused on selling its products to wholesalers such as supermarkets and other third-party channels. As such, its knowledge of its end buyers was at best sketchy and for the most part non-existent.
But it decides to create new digital channels to sell its products direct to its end customers, surmising that cutting out the wholesalers will increase its profit margins and enable it to gain a better understanding of and build relationships with its end customers. None of this is possible unless this new business strategy is developed alongside the data strategy needed to deliver it. Key questions would include:
Business strategy without data foundations is a folly.
Business strategy without data foundations is a folly.
The point here is that setting the aspiration in stone in the business strategy without being first being sure the data foundation exists to realise it is folly. So data must become a pre-eminent consideration when developing the business rationale and case for direct sales. Moreover, it’s also possible that, as the required direct selling data strategy is developed, it can help to suggest other opportunities that the business had not considered, for example, how analysing data on online consumer purchases can help to highlight purchasing trends and so help generate new product propositions.
So, if you are tasked with developing a data strategy, how do you ensure that this close mutuality and interdependency happens? Here are some suggestions:
More organisations are recognising the need for a dynamic and flexible data strategy as a keystone to help them achieve their business goals. In a poll of Data Management Association UK (DAMA UK) members in May 2020, 69% of respondents stated that developing a data strategy was one of their two top data management priorities (with the other being data governance at 77%).
We are living in hard times, but only by making business strategy and data strategy mutual friends can it hope to meet the great expectations placed on them by so many organisations. Charles Dickens would have understood that. Nigel Turner is principal information management consultant at Global Data Strategy and a committee member of DAMA UK.
Earlier this year, in the days before social distancing, I was lucky enough to catch up Neil over breakfast and he kindly agreed to be interviewed for my blog. Neil is an independent data evangelist who has worked in large multinational companies from the early years of the data adoption. He passionately believes that everyone is a data citizen or as he says a ‘Citizen Steward’.
He views Data Governance like safety, it stems from individual behaviour, and how we shape that to form the day to day activities embraced as a culture.
In 1991 I began my data career before Data Governance was even talked about.
Somewhere around 2008 Data Governance started to become more of a mainstream activity with the software vendors adding the word Governance to their sales pitches.
I have always represented the business side of data, advocating that business stakeholders must be leading and supporting data initiatives. If I take the simplest aim of Data Governance to apply a consistent lens or approach to the use of data, then the business must take the lead.
I didn’t deliberately set out on the data path as a career, I kind of fell into it. I was working in Finance and realized I was spending all my time as a spreadsheet jockey, which made me question my value. I was fortunate to be asked onto a major finance transformation programme as the reporting lead where data through the migration was critical to my success. Many years of data migrations through a progressive roll-out, and a very good mentor, convinced me there was a fledgeling career in data.
It wasn’t until 2005 that my Data Management role was formalised as the global MDM manager on an SAP finance transformation programme. Across the world, Master Data Management was an emerging discipline and I was lucky to be able to network with like-minded early adopters.
It was an exciting time to be part of something new.
Back then 90% of the focus was on the technology and IT was wrestling with adoption across their businesses. It was a hard sell to land data as a discipline, and that remains true today.
But at the heart of any data initiative is the need to articulate what it is you are trying to manage and how you measure whether it is working or not. Data Governance wasn’t seen as an enabler to the business processes, more a compliance and control regime which business areas could choose to adopt. To overcome these hurdles Data Governance needed a business lens applied with a focus on behavioural change.
So, in 2006 I started to develop the processes that would help the business to adopt and embrace Data Management. We now refer to this as a culture change, but it is a ‘hearts and minds’ challenge.
Passion, integrity, honesty, resilience, patience, adaptability, storytelling, simplicity, and being able to talk to various stakeholders in the language that they feel most comfortable. I call it ‘talk business’.
Once you come to the realization that this is about changing people’s perceptions about data, how they contribute to its management and how they would benefit from making those changes, your approach becomes far more tactile. I cannot understate the importance of developing soft skills. And like any relationship, you must adapt your style to different personalities. For example, at C Level, the message needs to grab their interest in the first 30 seconds which means presenting a concept, with language that supports that message.
I always put myself in the position of the recipient and try to anticipate what I would want to know and ask, how they think, their pet subject, things that would influence a positive discussion.
I use an ice breaker unrelated to the data narrative because Data Governance will challenge their beliefs and I am trying to develop a relationship that creates trust and ultimately influence.
At the end of the day, each of us develops our own styles through trial and error. Go with those that you feel most comfortable.
Never underestimate the power of WIIFM, what’s in it for me. Understanding those personal drivers of your stakeholders and how they would benefit from Data Governance will be fundamental to your success.
I’m not a big reader, but in 2006 Jill Dyché and Evan Levy published, at that time, an inspirational book called Customer Data Integration. This has been the only data book I have ever taken to heart because of its narrative. Jill is a wonderful storyteller where she brings to life the data challenges. If you ever get the opportunity to talk with Jill or listen to her, go out of your way to do so. (https://jilldyche.com/)
Getting started. We hear the saying ‘they just do not get it’ and use that as an excuse for not landing our data message. Looking under the covers I normally find that they do get it but data is not high on their list of priorities, or they have been subjected to the technology bias too often, or they cannot see the value in dedicating energy to a vague concept.
My biggest challenge has been turning that supertanker. Convincing stakeholders who are either disinterested or openly negative to the changes being proposed to establish a company-wide data discipline. Remember changing a culture requires commitment.
Company or industry for me is no different. Data is data and its management are broadly the same process.
However, I am a little different in that I look to a Chief Financial Officer (CFO) as the ultimate consumer of data, they would benefit directly through a well oiled and efficient data discipline. Harks back to my days as a spreadsheet jockey. Give me better data that I can trust.
Many people would disagree with my target audience, and to be honest, 5 years ago I would have agreed with them.
My rationale is that Data Governance starts predominantly through the management of master data. These are the foundations of every business process, customer, supplier, material, people etc. Every process executed in business has either an explicit or implicit financial impact that lands in finance. Much of the master data is touched by a finance process, for example, customer credit, material costing, supplier bank details, payroll.
Therefore, by inference finance really does have ‘skin in the game’ when it comes to consistent trusted data. Why would you not at least start the data journey in finance?
Keep it simple. The saying ‘think big, start small’ really rings true to Data Governance. You want to take your stakeholders on a journey of discovery and enlightenment, not a slog up Mount Everest.
There is no right answer, just many paths with potentially different outcomes. Choose wisely.
My first day in a company as the Global Data Governance Manager I attended the inaugural Data Governance Executive Forum, only to be told by my boss that he was not able to attend and that I should chair the meeting.
My first day.
I didn’t know the people, their subject areas, what had gone in the past, even the format of the meeting. This was to be my introduction into the world’s largest multinational of this industry.
I was petrified.
I learnt a great deal about the people, but more importantly about myself. In the room were a group of supporters, a group of antagonists with the remainder ambivalent.
By the end of the 3-hour meeting we had achieved a consensus on the way forward for Data Governance, the challenges I would have to overcome and most importantly the frequency in which I would sit down with them one on one over a coffee.
The outcome was the embryo of Data Governance that would ultimately get established and span the entire company.
On reflection, if I had made a mess of that first-day induction, Data Governance would have been consigned to the ‘failed project’ bin.
A while ago I wrote a blog about things you should consider when choosing the right software to help facilitate your Data Governance initiative, but once you have selected and purchased the tool do not assume that everything will now “just happen”.
One of my clients was worried (and rightly so) that it was at this point of the project that mistakes could be made which would impact the successful implementation of their Data Governance tool. I thought my advice to her may help others too:
Firstly you need to understand exactly what support you will get from your chosen vendor so you can plan what additional support you may need for implementation.
Then make sure that you agree who is going to manage the technical implementation of your tool. Is it going to be an in-house project team or are you going to engage a systems integrator? If the former is the plan, you need to liaise with the vendor to be very clear on what technical skills training they have available. What do they recommend to make sure that your team are suitably skilled before starting the implementation?
If you're going to use a third party to implement the tool, make sure you do due diligence to ensure that they understand the tool and have significant experience in implementing it. I have worked with organisations where a consultancy has been employed and they stated that they had experience in the tool. However, it became clear that while the consultancy as a whole may have had the required experience, the consultants working for that particular client did not have any experience and were learning on the job. This caused unnecessary delays and poor advice on what was and was not possible with the tool.
I also recommend focussing on one area or functionality of the tool for the initial implementation. Just because the tool has lots of features that doesn’t mean you need to implement everything at once. Choose the most needed functionality and implement that first, then look to implement other features as needed. Remember, at this stage, this is about giving your business users a tool to help them do Data Governance, not to confuse them with a complex tool and functionality they haven’t asked for. As your users become more comfortable with both Data Governance and using the tool you can implement more Data Governance requirements and tool functionality.
It is never a good idea to implement a data governance tool over the whole of your organisation at any one time. So I recommend not seeing the implementation as a one-off project.
It is better to think of it as a phased process with the initial implementation being a pilot or trial. Once you have completed the pilot it is likely that the users and the Data Governance Team may want some changes. This is common as you are introducing something new and not replacing an existing tool or process. This makes it very hard to get your requirements exactly right on the first attempt. So you may wish to make some tweaks to the setup of the tool before continuing a phased implementation across the whole organisation.
It could take a very long time to implement the tool fully. You need to make sure that this is well planned and that you are constantly working out what the next phases are going to cover.
You also need to consider how you are going to keep the data in the tool up to date. I recommend that you have a regular review of the content, for example, an annual review where Data Owners look at the content for the data owned by them. They can then either confirm that the definitions are still correct or, if necessary, provide updates to keep the tool up to date and useful for the business users.
As I mentioned in my previous blog about choosing the right Data Governance tool, it is essential that your Data Owners and Data Stewards (or at least a representative number of them) are involved in the initial implementation project. Often they have not asked for this tool and they do not react well to having the tool forced upon them. It is vital that they are involved in the design stage, to make sure that it's set up in a way that is going to appeal to them and make them happy to use this new tool.
Even if your Data Owners and Data Stewards have been involved in the early stages, remember that doesn't mean they won't need additional briefing and training when the tool gets implemented. I recommend having a section of your overall Data Governance Communications and Training plan dedicated to the implementation of your data governance tool. This will include things like initial high-level briefings to explain what the tool is and why it will be useful to your organisation. You will then need some specific focused sessions:
· Sessions with Data Owners to tell them what they're expected to do with the tool and showing them exactly how to do it.
· Sessions for Data Stewards which will be a little longer and more detailed as they will be doing the bulk of data entry and review of data in the tool.
Both sets of training need to be accompanied by some kind of user guide or aide memoir, to make it very easy for them to quickly check what they need to be doing once the training is over and they are using the tool for real.
Taking all the above into account may seem like a lot of undue effort when you just want to get on with implementing the tool, but doing so will make a huge difference over whether it is a success or not.
If you have other tips for a successful Data Governance tool implementation that I haven’t included above please let me know!
There is a global consensus; test, track and trace is the way out of lockdown. To enable track and trace in the UK, NHSX will be rolling out an app that is currently undergoing trials. The UK government has opted for a centralised data collection method, whereby phone to phone contacts are logged centrally and users receive a notification that they are at risk and need to self-isolate. An alternative de-centralised model has been proposed by Apple and Google whereby all users receive details of confirmed cases and the phone itself then notifies the user that that they are at risk and should self-isolate.
The effectiveness of track and trace will be highly dependent on take up. The more people who use the app, the more effective it will be in identifying and notifying people at risk so that they can self-isolate, preventing the spread of the virus and keeping the all important reproduction rate, R, below 1 so that the rest of the population can safely go about their business.
What do you think? Do you intend to use the app as it is? Or does data privacy trump public health for you? Would you use an app that was based on the de-centralised model?
I’m a data geek, so I tend to see things through the prism of data. The COVID-19 pandemic is no exception. While some people are glued to the daily theatre of government briefings, I’m looking for reliable sources of information. In particular I’m looking for evidence of how the pandemic is evolving, how long we’re likely to be in lockdown and what is the impact likely to be for me, my family, my colleagues and my clients.
There is certainly lots of data out there, but how do you filter out the noise, because the data is very noisy at the moment with some very wild claims and data that appears to support such a broad range of positions and theories. The conspiracy theorists are having a field day, but let’s not go down that rabbit hole.
My favourite source for information at the moment is the FT. In particular the work of John Burn-Murdoch (@jburnmurdoch) and Chris Giles (@ChrisGiles_). John collates data from around the world to produce a series of daily trackers showing high quality visualisations including daily rates of new cases and deaths, which are the two trackers that I check every day. Chris merges the official daily count of COVID-19 hospital deaths from the Department of Health and Social Care with the weekly total death statistics from the ONS, to produce an estimate of the total COVID-19 deaths.
There are two things that I really like about their work. The first is that the information is presented in a very clear way. John uses logarithmic scales which means that the slope of the line is the most important thing. A straight line represents exponential growth, and while we were in that very scary phase, the straight line showed very clearly how serious the situation was. The same visualisations are also now showing that lockdown measures are working and both deaths and new cases are coming down. I can see all this in less than 30 seconds every morning. Meanwhile Chris’s visualisations show the difference between weekly deaths now and the five year average, with the implication being that the difference is down to COVID-19, which is clearly much worse than seasonal flu.
The second thing I like is the fact that they both show their working and they are clear about the uncertainties and the assumptions that they make. John has a useful and informative video clip explaining why he uses the logarithmic scale, where his source data comes from, and what the inconsistencies are. He is open about the fact that the data is very noisy, and what he has done to compensate for this. He has settled on a 7 day rolling average, for example, to smooth out some of the noise in the daily reporting. Chris documents the assumptions that he makes about merging two separate data sources, and he is clear about when and why he changes those assumptions. The fact that they show their working in such a transparent manner, and that they patiently respond on Twitter to questions and criticisms allows me to validate their output for myself, to the extent that I trust it. I feel confident that I understand what their work shows and what it can’t.
The COVID-19 pandemic is topical, and it’s putting some data under the spotlight, but it’s highlighting some unchanging truths, fundamentals if you like. To make sense of data requires rigour, including understanding where data has come from (lineage), how reliable it is (quality) and what it actually means.