Our customer (a large research agency) runs a large number of continuous research projects in multiple markets, some with syndicated results and others with more specific customer portals and dashboards running against the data. They struggled with the challenges of data management as they have data coming from a variety of sources, primarily from surveys but even within this a number of different data collection systems are in use, with requirements to deliver results on a daily basis in some cases.

While the processes worked at an individual project level, the results themselves were becoming very challenging to manage.  For a given wave of research, they were able to easily locate a specific result from a survey, however it was difficult for them to compare these results across waves, markets or projects. There were some cases where historical raw data was not available, or others where the cost of migrating historical data between different sources to bring it all together into one place were prohibitive.

We set out with the goal of making a single data store for survey results which was not tightly aligned to any particular type of data collection or data processing technology. It needed to be efficient to implement and avoid the need to re-process large amounts of historic data.

We concluded that what was needed was a data warehousing system for results, one which was easy to retrieve information from while being flexible enough to hold data which spanned multiple types of research project. As a result we developed our results warehouse which we call Research Vault and this has become a core component of many of our solutions. This has gone on to be used as the back end for multiple dashboard systems and custom reporting tools as well as custom analytics where it enables results to be compared in ways that were previously very difficult.