Data Mesh Architecture

Data Mesh Architecture Services

Data Mesh refers to a distributed architecture and set of principles for data management at scale. In its operating model, data is treated as a product and owned by teams that ensure its quality and ease of consumption. Each team owns and manages its own data domain (a data domain is a logical grouping of information that is of interest to a business), enabling interoperability and cross-domain data analysis.

Introduction to Data Mesh

Companies today are using data to make high-impact, confident and proactive business decisions to keep ahead of their competitors and deliver innovation. The large sources of data that businesses have access to are processed, analysed and even fed into machine learning (ML) algorithms to get valuable insights that help steer direction.

Tableau reported that 83% of CEOs want their business to be data-driven. Whether it is analysing data to forecast sales or predicting the success of a product launch, the value of data is well understood.

Data access, however, must be made easy in order to achieve all of this.

In a centralised approach, data is ingested and held in a single central location. The data that is stored comes from a wide variety of domains, qualities and sources, which often create a bottleneck for teams to use the data effectively. Additionally, this approach of having all data in a single location and analysing it not only delays the process of extracting value from the data but also creates inefficient IT activities, such as creating and managing data pipelines, managing data copies, and layering it all with the appropriate governance.

The following are some of the drawbacks of a centralised approach:

Management difficulties

The amount of data stored centrally can be enormous, so data management practices need to be good (maybe link to our data management article?).

Managing redundant and corrupted data, ensuring security and governance are all complex tasks that can be even more challenging with a centralised approach. Building, managing, and maintaining centralised data solutions is difficult and expensive.

Data Governance

With data coming from several sources and stored in various formats, centralised data governance can be challenging to implement.

Longer Time-to-Value

It can take teams a long time to locate and prepare unstructured data for analysis, creating a time delay in the extraction of insights.

Poor data quality 

Data ingested from multiple sources of differing quality, redundancy, and consistency makes a centralised approach difficult to ensure quality.

Data Mesh Architecture Principles

Data mesh improves data discoverability, accessibility, security, and interoperability — giving you better and faster decision-making power.

It is built on four principles:

Domain Ownership

Data is decentralised, and responsibility is distributed to teams to own and manage it. This removes bottlenecks in the pipeline, enabling the data to be utilised more efficiently.

In esynergy’s successful data mesh solution for one of our clients, we created 80 Google Cloud Platform (GCP) projects that were all model-based, that is, the projects that were divided based on the data domain and owned by multiple teams. Data value was unlocked as a result of this, and the teams worked more efficiently.

Data as a Product

The domain owner is responsible for the data to be treated and delivered as a product, ensuring its quality, security, discoverability, and interoperability.

As part of the process of building Data Meshes or migrating from a centralised data lake to a distributed data platform for our customers, esynergy harvests the idea and mindset to treat data as a product. As a result, our clients have been able to maximise the impact of their data.

Self-service data Infrastructure platform

A dedicated data platform team providing domain-agnostic infrastructure and tools to store, process, and analyse cross-domain data.

esynergy worked with a large financial corporation to help them migrate from a monolithic data lake to a self-service data mesh platform, helping deliver business values rapidly, lowering lead times and enabling them to identify more business models, unlocking the value of the data they had access to.

Federated Governance

To gain valuable insights from distributed data at scale, independent data products need to inter-operate. Standards that ensure interoperability are essential to achieving this.

With esynergy’s data mesh solutions, our clients are able to unlock the true power of data, thereby increasing their business performance.