While we talk about architecture and design a lot there is one elephant in the room that seldom gets discussed because we presumably all know about it and assume it is the same for everyone. Basic issues around supporting business needs will always be there but with the diversity of change in what is currently the most disruptive market ever seen we should revisit what is happening and how to be prepared.
In my earliest days we knew of the silent killer of data warehouses was always this sudden, unexpected need for data. A group would spring forth and demand data, as the good operatives we were we added them to the list of things we would do and let them know when they could have it. Well when it is a marketing campaign it has a specific start and end date which is not in line with delivery plans. In the conference business you need to get people to sign up across a certain set of dates, as your attendance progresses you need to be able to manage this by placing offers correctly to drive attendance. If you offer discounts too early or too late you leave money on the table. You need data real time daily for the length of your offer window. To get that information later will be useless it has value for a certain amount of time only. So what happens?
Enter the Access Database. Often when things cannot be moved into a data warehouse (it needs to be curated, modeled, prepared etc.) you try to push off the data request. Escalations occur and pressure is applied, the business group has someone that knows Access can we just connect and get this data ourselves; we can build from existing campaigns and throw it in there for now. You resist but its futile, it’s a terrible option you will live to regret but you allow it and move on with your day and forget this ever happened. Three years later I get the call (it is either three or five years but its consistent) that the data warehouse has become some sort of data repository and no one really understands how this happened but its become not very useful and could I please come and fix it.
As you nod your heads sagely, we can all admit to having this happened, it is not something that might happen it is when will this happen type of occurrence. The last customer I dealt with had 300 Access databases connected directly to their data warehouse. They wanted to migrate to another platform but had no idea what to do or how to go about it. For the last 15 years this type of loss of control has been directly because of the ideals perpetuated by data warehouse experts that the data warehouse needs to be the center of the universe idea. While this in theory remains solid the reality of what has happened has been that a lack of business support and understanding of expectations leads to the scenario where data becomes unavailable or so stagnant it is useless.
Current data technologies can help alleviate a lot of the problems but there is a lot that can be done to mitigate what can happen and what is still happening. Here are some of the things our research has shown will move your data initiatives into a much more viable state.
One. Anticipate. While this seems obvious it becomes difficult because you have so many daily chores to make sure data gets delivered. To do this successfully you need to review all the things you do and automate where possible to reduce your complexity. The more complexity you have the less productive you will be. Conduct surveys internally to automate everything possible such as monitoring, stopping, and restarting data loads or having to baby sit data loads that get finicky for some reason. The idea is to give you the ability to execute and often a simple migration to cloud solves your up and downtime issues and need to patch software etc which starts to give you back time to do other things. What things are those you ask?
Two. Organize. This means you need to change structure a bit to help you understand when you need to have things ready. A full prescient system relies on embedding your teams into business units. This is hard to do without engaging in step one and step two relies on you having time to execute. Also instead of embedding a solid approach is to recruit business analysts in the business units and repurpose them to engage with your teams. Awareness meetings are like account planning in sales but for preparation and prioritization. These should be held monthly to make sure that your support strategy aligns with what is coming down the pipe.
Three. Feedback. Usually I put delivery as step three but here it is really at its core not just delivery but being able to adjust to feedback. Implement a methodology to be able to understand the effect of what you are delivering. This is generally what the semantic Business Intelligence layer delivers. When a business user interacts with a data set it is not at the data hub level it is at the report level. There must be a way for them to communicate directly with the curation team. Consider this your process improvement layer and there are many ways to build it out. Consider a Java service layer that can take feedback, identify the report and data set and user that it came from and collate the information. Curation of data should not just mean it is prepared and ready but also that there is a way to refine and improve that information for consumption.
Consumption and dissemination of information through the enterprise should be your goal and that means you need to measure it actively. Step three allows you the data to be able to generate the metrics needed. Sounds simple but these three steps are hard to get through. Resilient systems are built on these three steps so make sure you are progressing on them and get more time to execute.
About the Author
Asim Razvi is the head of data management and data strategy at Onis Solutions with over 25 years experience in delivering world class solutions in data to clients. He has architected some of the largest hybrid data management solutions for the Fortune 100 and also worked closely to deliver Business Intelligence strategy assessments to them as well. He works and collaborates closely with a number of CDOs and maintains a busy schedule of events and speaking engagements. Outside of work he trains outdoors to maintain a healthy lifestyle and spends time with his family in the wilds of the California mountains.
Vice President Lead, Data Management