Investment in big data and AI is at an all-time high. Yet, many organizations struggle to derive value from their data. Missing buy-in from senior stakeholders, an outdated technology stack, poor data quality, or waterfall project management can all get in the way of success. The result: only 13% of data science initiatives ever make it to production. This is a significant waste of time and money that frustrates data teams. There must be a better way of delivering data projects. And there is: DataOps.
DataOps is a framework that brings together cultural norms, organizational workflows, data governance principles, and technological practices to deliver data projects successfully. At D ONE, we are convinced that DataOps is a great approach for deriving more value from data. In this post, we share our learnings from a workshop that we have just given at the Applied Machine Learning Days at EPFL Lausanne.
In our workshop, data scientists, project managers, researchers, and data engineers used our custom-built DataOps Radar to identify and address the biggest challenges in their data projects.
We designed the DataOps Radar to deliver key insights and concrete actions on how to improve data projects. The DataOps Radar provides us with a 360-degree assessment covering the four pillars of data projects: The data itself, the technology through which it is created, transformed and delivered, the organization within which it exists, and the data culture and mindset that permeate all of these dimensions. Each pillar is scored based on how key questions about the data project are answered. For instance, projects in which the management favors relying on intuition over data-driven insights score lower than those using data to arrive at strategic priorities for the business. Similarly, the proficiency in the use of industry standards and best practices determine the maturity of a data project.
Working with the Radar, our participants gained new, sometimes surprising, insights. To quote a data scientist: “Before the workshop, I thought that data access was our main issue. The DataOps Radar has shown me that our biggest challenge is cultural: we are missing a data mindset.” Other participants discovered that they could run their projects in a more lean and agile way, or that adopting different DevOps practices could be beneficial. The Radar also made it clear that many teams struggle with data usability.
In the final part of the workshop, we focused on the concrete next actions participants could take to improve their projects. In small groups, the participants brainstormed how to use DataOps to address their biggest challenges. Could you maybe be an ambassador for a data mindset by relying more on data in your own decisions? Can you convince business stakeholders to take ownership of their data? Maybe you can get your team to start using a version control system? This part was also a fantastic opportunity for the participants to share their real-life experiences and to network. In the end, everyone left with concrete ideas on how to bring their project to the next level.
We have learned that many data teams — no matter the industry or department — struggle with similar challenges. Some of these can be addressed by the teams directly. For example, following agile practices, organizing work using a Kanban board, and limiting work in progress can lead to quick wins. It is also relatively easy to adopt best practices in code development. Teams can also begin to monitor data quality in their pipelines. Topics such as data ownership and data access are more difficult to address at the level of individual teams. One approach can be to convince the management of the value of good data governance — ideally by providing metrics that show their importance. As an example, a team could measure how much time it spends in dealing with data quality issues. The hardest to change is an organization’s data culture. But even here, not all change needs to come from the top. By living a data culture themselves, even small teams in large organizations can become ambassadors for a data mindset. After all, a data culture needs everyone on board.
We are curious to hear from you about the biggest challenges in your own data projects. If you’d like to learn more about how we are using DataOps at D ONE or if you are interested in a DataOps workshop, please get in touch with us.