The Curse of doing data right
Photo by Jude Beck on Unsplash
Part 1: What is the curse?
Very few people will disagree that having a solid foundation for data is a good idea. This includes many of those things that we hear all the time, such as good data quality, common standards, or a good catalogue where we can find all our data assets. This is the same for more technical foundational topics, such as metadata, master / reference data, data modelling, and so on. Once we, data professionals, can explain what these things mean, the general response tends to be positive: essentially, everybody agrees these are good things to have.
Getting to implement this foundation, however, is a different story. Because data is the backbone of most organisations these days, any serious attempt to review and modify how data is managed will affect users and could fundamentally change their ways of working. And it is then, when users realise that they are at the core of implementing changes across the organisation. It is their responsibility too, not just the data professional job and here are where the problems start. The same organisation that has probably invested in hiring data professionals to “fix” its data problems, will start making excuses for why many of the measures can’t be implemented. Users and management will argue that operations will be disrupted, deadlines will be missed or that it is not the right time. Some organisations will agree to proceed with the less disruptive proposals, but of course, those will also be the ones with less impact or will incur technical debt that will need to be fixed later. Chances are that months and years will pass and little will change; at best, there will be an improvement of awareness and some minor initiatives will be carried out. At worse, the organisation will dismiss advice or try to solve the problems by investing in technology products, which will never work because the foundations were never fixed.
Data in an interesting technical field. It is in so much demand, but so misunderstood. There are a wide variety of areas of expertise with radically different skill sets, and yet often the field gets oversimplified by putting all data professionals in the same bucket. For example, data scientists / analysts are many times thought of as holding the key to solve many data issues. But the issues normally stem from poor data quality or interoperability, which is not something that data scientist or analysist can (or should) address. Along the same lines, data problems are something to be resolved upstream i.e. once the data has been collected and stored. But, to solve quality issues one needs to start at source i.e. where data is collected, stored and shared. This misunderstanding is one of the main reasons why there is such a disparity between the willingness to invest in data professionals and the reluctance to carry out their advice.
Organisations are ill-prepared to undertake the scale of changes that are being advised. They put in place obstacles to attempt managing the disruption, and so the main reasons why these change projects take such a long time are not technical, but cultural. Work is carried out to justify investment, but there is unwillingness to fix the root cause of the problem. Data scientists / analysts are hired to produce results, which invariably means creating inefficient and inconsistent shortcuts to put the data in a form that can be analysed. In turn, the data professionals that provided the advice for change in the first case, see that advice dismissed or underappreciated, and fall victims of the contradiction in which the organisation finds itself. As those professional become frustrated, and as they continue pointing out how the organisation is doing things in the wrong way, they sometimes can become disliked and even ostracised, which increases their frustration as, in their view, they are just trying to achieve what they were hired to do.
The cycle continues, with investment in data infrastructure but without the willingness to implement the necessary changes. For those responsible for the change, sometimes this situation might feel like a curse, the curse of doing right for data. But it’s not all doom and gloom; there is a growing community of data professionals who are supporting each other with doing data right. Which is exactly what our next blog will talk about!
Tomas Sanchez
This is the first blog of a 3-part series looking at the challenges of data professionals within their organisations. Join us next month where Lisa Allen will talk about what data management changes are commonly required and why organisations find them challenging.
Tomas is currently the Chief Data Architect at the Office for National Statistics, where he is leading ONS’s data strategy as well as being responsible for a number of data products. Tomas also regularly gets involved in forums and initiatives to foster the use of good data management practices across government.
