The data grid can overcome many challenges inherent in big data by driving higher levels of autonomy and data engineering alliances among a wider range of stakeholders. However, big data is not a panacea, it brings a series of risks for enterprises to manage.
For many enterprises, data is a huge resource that has hardly been developed. Many institutions and organizations have realized that data is a key asset, and it is more important than ever to use the insight gained from enterprise data. In practice, innovators, disruptors and start-ups are much more flexible in using data to change, compete and win the market. Moreover, if they make good use of these data, they will gain more customers. They may not have as much data as large enterprises, but they are more able to use it.
Big data is a popular term, which defines a series of methods that many enterprises use to develop solutions to generate the required insight. However, according to the statistics of Gartner and other institutions, most companies fail to achieve their goals through big data methods.
The core of the big data approach is to focus on ingestion, transformation, governance and insight in most of the enterprise's data. This will lead to bottlenecks that significantly inhibit the delivery of business value within a meaningful time frame. Instead of facilitating the flow of data, it has been stifled.
So is there any other choice?
What if there was a new approach based on federalism rather than centralization to help enterprises gain the insight they need to remain competitive?
For many enterprises, the data grid approach solves the challenges they face. The core of data grid is a data federation method based on the proven and tested principles of software engineering. Many enterprises have applied it to customer journey development.
Three principles of data grid
The data grid method utilizes three core principles of modern software engineering:
Domain ownership
product development
Self service software platform
These principles enable the development of data solutions to be united, which can unlock important and greater insights faster, so that enterprises can realize business value.
(1) Domain ownership
This follows the current domain modeling principles and adds data coverage to the model. The domain model is a visual representation of key concepts/objects in the problem domain, which is the highest level of the enterprise. It decomposes an enterprise and establishes clear ownership and boundaries for business functions and technical solutions. This enables microservice based software engineering methods to drive autonomy and reuse.
The same domain model can be used within the enterprise to establish ownership of domain datasets. Each dataset should belong to the domain where the data was created/generated. The goal is to make each dataset owned by a single domain, and adjust the domain model where necessary to achieve this result.
(2) Product development
This shows that enterprises treat data as products, just as they treat customer journeys as products.
The focus is on what customers want to do and the best solution to help them do it. A product team is a group of individuals with multiple skills who bring together business and technical personnel to create the best possible customer results.
Applying this to data means understanding different data roles in the enterprise, including customers, internal business users, other engineering teams, B2B partners and regulators. This will help define the work to be done for different user groups, allowing the product team to focus on solving the challenges of each user group. Product development consistent with domain ownership will create clear accountability results for data sets and user results, enabling the team to stay in step.
(3) Self service software platform
Self service software platform for data is the core of autonomy and agility required to achieve the delivery results of data grid methods. At its core, enterprises need to think like cloud computing service providers and create an API driven self-service data platform. The platform needs to provide three groups of functions, first of all, storage, database, access control and other infrastructure. The engineering tool of workflow abstracts the complexity of infrastructure through infrastructure such as code and DevOps. Finally, the platform needs to provide central management functions for discovery, compliance, and monitoring.
Implementing the three principles of data grid helps eliminate the inherent bottleneck in big data methods. With the data platform, each product team can define the data work they want to do, and determine the priority of different user results according to the value release to the enterprise. Each product team can work independently and quickly within the investment budget allocated to them.
How to prepare for success?
The implementation of data grid method requires domain modeling and product development as part of the normal software engineering life cycle. Successful implementation of domain ownership and product development affects people, skills, and enterprise design, which requires buying from senior stakeholders to achieve success.
Many enterprises are already transforming their enterprises in this way. Taking the data grid method as a part of a wider transformation should reduce the overall work and cost involved in implementing the data grid. If enterprises have changed their operational models, the data grid approach is the logical next step, which can gain more value from the model and alleviate many challenges inherent in the big data approach.
Another key factor in the successful implementation of the data grid is to ensure that the self-service data platform is set up correctly. This requires some pre thinking to define the functions, architecture, team skills and structure required by the platform to achieve the autonomy of the product team. It also requires a team that understands the vision of the self-service platform and has the skills needed to achieve it.
Finally, it is recommended to start small. Create an MVP self-service platform to enable a small group of non critical data sets to prove the technical and operational model in an enterprise environment.
Is big data a panacea?
The data grid can overcome many challenges inherent in big data by driving higher levels of autonomy and data engineering alliances among a wider range of stakeholders. However, big data is not a panacea, it brings a series of risks for enterprises to manage.
By understanding the risks brought by the data grid method, formulating plans to reduce these risks, and selecting the correct architecture to support the self-service platform and stakeholders to support the vision, enterprises' ability to use data can change significantly.
Neil Mulholland has been the chief architect of Enterprise Blueprint since 2010, and has been engaged in digital and integrated architecture since the 1990s. Recently, he has played an important role in helping enterprises adopt agile methods on a large scale, focusing on how architecture and architecture governance work in this environment.
Neil believes that trust is gained by demonstrating the ability to focus on the most important things for customers. Understanding customers' needs and achieving the results they want are critical to determining successful solutions and building long-term, mutually beneficial partnerships.
Neil contributes to the customer's enterprise through informal and formal architecture team guidance to help them improve the skills of the entire architecture team through better solutions and operational methods. He likes to try new technologies and learn how to best apply them to solve customer problems, but he doesn't like a steep learning curve when introducing new technologies, because it will delay real adoption.