5 Key Takeaways from Gartner Data and Analytics Summit 2023

With the Gartner Data and Analytics Summit wrapped up, let’s look back at the key themes around data and data management. It is no surprise that there was a lot of buzz around data fabric, data mesh, data products, active metadata, and automation. However, let’s take a step back and ask what, exactly, we are trying to accomplish.

The goal is self-service data provided in a scalable, easy-to-use experience for all users. How do we make it possible? Let’s take the key themes from the sessions and discuss how they are driving us towards this goal.

Data Fabric: What is it and how do you use it?
Is Data Mesh Dead?
Active and Passive Metadata
Automating Data Management
Practical Data Fabric: Customer Use Case

DATA FABRIC: What is it and how do you use it?

We have been hearing about data fabric for a while, and different people have different definitions. Let’s take a look at what Ehtisham, VP analyst Gartner, had to say about it in his session, “The Practical Data Fabric — How to Architect the Next Generation Data Management Design”.

What is data fabric?

According to Gartner, data fabric is not one single tool or technology. It’s an emerging data management design for attaining flexible, reusable, and augmented data integration pipelines that utilizes knowledge graphs, semantics and active metadata-based automation to support faster and, in some cases, automated data access and sharing regardless of deployment options, use cases (operational or analytical) and/or architectural approaches.

What is data fabric in simple language?

Data fabric is, at its core, metadata analysis that is powered by recommendations.

Who benefits from data fabric?

Data fabric is for everyone: business analysts, data engineers, data producers, data consumers, C-level executives, and anyone who relies on or uses data.

Business Analysts: The cataloging layer of the fabric keeps everybody on the same page by intervention distributed data and allowing semantic search, which enables them to quickly find, integrate, analyze and share data.

Data Engineer: From tracking pipeline breaks to determining ideal integration strategies, data fabric gives data engineers insight about current data status and future data onboarding, speeding up data requests and allowing for more value-adding activities.

CEO/CFO/CDO: The increased speed and accuracy of self-serve data lowers time-to-value across all data activities, as well as increasing data literacy for all data users, regardless of technical background.

NOTE: Nexla is a representative vendor in the Data Fabric
Orchestration, Integration and Preparation part of the Data Fabric.

IS DATA MESH DEAD?

Short answer: NO! That said, data mesh is a concept that has been around for the last few years and has been brought into the mainstream by Zhamak Dehghani in her 2021 book Data Mesh: Delivering Data-Driven Value at Scale. Data mesh is built on four core principles: domain ownership, data as a product, self-service data, and federated computation governance. The first two principles emphasize an organizational mindset that treats data as a first-class product owned by individual teams.

At the Gartner DA event, Robert Thanraj, Director Analyst at Gartner, stated in the session “Data Fabric or Data Mesh: Debate on Deciding Your Future Data Management Architecture,” that while each of these principles is interrelated and plays an essential role, treating data as a product is a fundamental shift in how organizations create, store, and communicate important business data.

Why are enterprises finding it hard to implement a data mesh?

Data mesh has faced challenges due to four major reasons.

Many companies have their own definition of data mesh.
It requires a lot of consulting effort.
The implementation team has to be skilled in data products (he/she has to know how to operationalize data use, when to retire them, etc.).
80% of respondents in a survey said that they were not ready to be accountable for governance.

In the 2022 Data Management Hype Cycle, Gartner moved data mesh to “Obsolete before plateau”, but this is a prediction. Data mesh will continue to grow but be broken down into smaller components that will be subsumed by other aspects of emerging data tools.

How can we combine the benefits of data mesh and data fabric?

Data mesh emphasizes the importance of organizing data around business domains while data fabric emphasizes the need for a unified data architecture. To get the benefits of both, you must first identify your organization’s data domains and create a data architecture that connects them together.

Implementing both data mesh and data fabric requires a team with technical expertise and an understanding of your organization’s data needs. It involves integrating various data sources, building data pipelines, and creating a unified data architecture. This includes using data mesh to enable domain-driven data ownership and using data fabric to provide a unified view of data across the organization. This can be accomplished by powering data products through metadata intelligence, which is formed by an intelligent data fabric layer. This helps automate processes, reduce manual workloads, and improve data quality.

Once you have data products, you can use them in a variety of ways: assign them to a domain, put them in a private marketplace for data products, and manage governance when the data product is assigned and when consumers use the data product.

Nexla supports and enhances data fabric designs to accelerate data management needs. Nexla is an all-in-one integration tool that can manage real-time data, helping you reach your data fabric goals quickly, including making it simpler for data consumers (marketing, sales, customer relations, etc.) to gain the insights necessary to make data-supported decisions. Furthermore, Nexla leverages data fabric to create data products that improve data integration, data discovery, and overall data management.

ACTIVE AND PASSIVE METADATA

What is passive metadata?

Passive metadata informs you about your system. Passive metadata is metadata that is still in the collection state. Ehtisham’s advice was to:

Constantly collect as much metadata as possible.
Create metadata management practices, similar to data management practices.
Improve the quality of the metadata, or data fabric implementations will fail.

Ehtisham encouraged everyone to go beyond technical metadata (schemas, structure, data types, data models, models), and invest in tech that can collect passive metadata. Gathering performance metadata data on your pipeline, business metadata from your business glossary or ontology on tags, social meta to give you insights on if data products are being used, developer feedback, business community feedback and more.

What is active metadata?

Active metadata is the continuous analysis of all types of metadata (technical, business, operational, and social) to determine alignment and deviations between “data as designed” and “operational experience.” Vendors using active metadata can not only collect metadata but can also do graph analysis on it, such as using the design time and running time data to identify alignment or exceptions.

Active metadata helps track when there are changes or exceptions, leading to faster response time and data-driven reactions, whether that is tracking data integration pipelines, switching your data models, or changing your data management system.

Definitions of active and passive metadata in simpler terms

Passive metadata approaches are like a thermometer that “monitors” the data management and current utilization. Active metadata approaches are like a thermostat that “regulates” data management and utilization.

AUTOMATING DATA MANAGEMENT

Mark Beyer, Distinguished VP Analyst, discussed the need for automation in the session “The Active Metadata Helix: the Benefits of Automating Data Management.”

How active metadata powers automated data management

Beyer discussed how metadata is generated practically every time data is accessed in any tool, platform, or application. It is a largely untapped resource that is a real-time documentation of exactly how, when, and why any person in the enterprise uses data from any and all assets available to them. Active metadata is the conversion of these otherwise passive observations into ML-enabled, automated data management.

What are the benefits of automation?

By automating common tasks, Beyer showed how data processing becomes faster and more accurate. This allows data engineers to focus their efforts where their expertise and creativity can add value, such as pipeline engineering over pipeline creation. His final remarks were centered around how automation is the next step towards a unified modern data solution, and this is how enterprises can keep pace with an ever-accelerating world.

PRACTICAL DATA FABRIC: CUSTOMER USE CASE

In a fun session moderated by Google’s Bruno Aziza, one of the most well-known commentators in the data ecosystem, and Saket Saurabh, CEO of Nexla, we learned how a large pharma company and Stagwell implemented their own data fabric, which provide metadata intelligence to power data products, and how this helped them foster a data culture by empowering data consumers to access ready-to-use data products.

How did a large pharma leverage data products?

Vasanth, Data Science leader from a large pharma company, discussed how he and his team leverage Nexla to create data products to fuel his data science team. Where they took data from various sources, such as health data from wearable, and integrated it real-time team to feed into their data science training model.

How did Stagwell create a private data product marketplace?

Mansoor Basha, CTO of Stagwell, leverages data fabric and leverages the newly announced “Private Data Product marketplace” by Nexla to enable self-service in his organization. NOTE: Click here to sign-up for the beta for the Private Data Product Marketplace. They also created data products with Nexla but needed a way to share data products. He discusses how the private data marketplace gives the [data] consumers on the business side the ability to discover and take action on those Data Products in an easily accessible manner. The marketplace provides a single platform to discover new data use cases and then be able to create new things on the fly.

NOTE: Gartner’s data fabric and data mesh slide includes one of Nexla’s graphics that shows Nexsets (Nexla’s data product) being created.

CONCLUSION

Data mesh and data fabric are two different approaches to data management, but they are not exclusive and can complement each other in many ways. Data products can help to create an architecture where data fabric and data mesh can coexist, creating self-service and user-friendly interfaces for all users. Companies like Johnson and Johnson and Stagwell have implemented these technologies to foster data democratization. They have also created a private data product market place where users can access ready-to-use data products in a governed manner.

NOTE: Nexla is a Honorable Mention in the 2022 Magic Quadrant for Data Integration Tools, plus Nexla was also recently recognized as a Representative Vendor in Market Guide for Event Stream Processing, the Market Guide for Active Metadata Management, and the Market Guide for DataOps,Tools, as well as a Sample Vendor in the Hype Cycle™ for Data Management, 2022.