The Definitive Data Operations Report 2017

Jarah Euston
Jarah Euston

At Nexla, we are constantly thinking about the challenges of data operations, or DataOps for short, for cross-company data. We’ve spoken with hundreds of companies about the unique effort required to send and receive data across company lines. But few benchmarks exist in the market for companies looking to learn from best practices. We decided to investigate.

This first-of-its-kind survey asked over 300 respondents about how they derive value from data. We surveyed data professionals from 40 different industries, with tenures ranging from two years to more than ten. In this post, we summarize some of the key benchmarks that emerged from the study. You can read the full report here, and a brief summary in this post.

Key findings include:

  • Companies need to elevate DataOps into a core function if they want to maximize data value
  • Inter-company data collaboration is growing and will become the norm
  • Data executives do not have the support they need to maintain their company’s DataOps

DataOps Needs to Come into Its Own

The goal of DataOps is to deliver data to the person, system, or application that can turn it into business value. Sounds important. It is, and a whopping 70% of companies across industries said they have plans to hire in DataOps in the next 12 months. When we examined the data by industry, the picture was similar for traditional “tech” industries (like Internet) and non-tech industries (such as manufacturing and healthcare).

companies planning to hire in DataOpsEven though most companies have plans to hire, organizationally, DataOps hasn’t yet found a home. DataOps teams are organized within IT, Engineering, and Operations, but also in departments like Marketing.

DataOps in the OrganizationWhile it makes sense that IT or engineering is driving much of the data infrastructure, there is a case to be made for consolidating DataOps into its own team. Consider a client that ingests data from 2,000 partners— in 1,800 different formats. This client has a team of business analysts that work with technical teams to normalize and format the data into a usable form, while coordinating with partners. The operations analyst is an internal customer of IT, but not part of that team. The most effective teams we have seen have a similar structure.

Inter-Company Data Collaboration is the Norm

If you think companies sending and receiving data from other companies is a tiny fringe cohort, you are mistaken. As the chart below explains, almost all companies, 91%, are already using third party data or have plans to.

Inter-company data collaborationWhen asked if their company currently sends or has plans to send data to partners, 70% said yes. More than one-third reported that they send this data not as analytic reports, but as raw data. This is important, as companies across all industries are adopting machine learning and advanced analytics. Reports can’t be fed into models. If your partners aren’t asking for raw data today, they will be soon.

Data Executives Need More Support

Processing and moving data in the right format to where it needs to go is not easy. Monitoring the data pipelines created is often more work than setting them up in the first place. As the chart below shows, our respondents spend 47% of their time on DataOps activities. Data integration alone takes up 14% of their time, and troubleshooting takes up another 13%. Working on these tasks means data professionals can’t spend the time analyzing the data. We need a new scaleable, repeatable process that allows companies to increase their data transfers but doesn’t require continual manual work.

How data professionals spend their time

It’s this lack of scalable process that has data executives frustrated. We were surprised to learn there is a large disparity between how data executives and line employees feel about their company’s commitment. In fact, 74% of data professionals agreed that their company understands the effort required to maintain their DataOps. When we look at data executives, this number drops to 48%. Understanding the effort to maintain DataOps Almost one-third (30%) of data execs disagree or strongly disagree that their company understands the effort required to maintain their DataOps. This suggests that Data Executives may not be able to get the support they need to run effective DataOps functions. The consequence of this lack of support could be the amount of time that teams need to spend on manual tasks, as evidenced by the chart above. Data executives, and their teams, need more tools and processes to help them execute their responsibilities. Hiring in DataOps won’t be able to keep up with the problem.

DataOps: Looking Forward

These results demand a change to the way we approach Data Operations, from where it sits within the organization to which tools we use. Inter-company data collaboration at scale for machine learning requires scalable, repeatable processes. We now know that company-to-company data transfer is here, and is only going to grow. Companies would be wise to put the processes and tools in place now to prevent data heartache down the road. To learn more about how Nexla can help with this problem, visit nexla.com.

But wait, there’s more! Did you know 14% of companies are producing one terabyte or more of data per day? Download the full report to learn about this and more.