The challenges of building big data products using digital and transactional data


Howard Seccombe, Chief Digital Officer at Roy Morgan Research, thinks of market research as the ‘e-harmony’ for big data, because they know how to knit together data in a deep and meaningful way. 

Back when data was becoming The Next Big Thing, there were predictions that data would make the market research industry obsolete, because everyone would be swimming confidently in the stuff. 

However, while this is true in the sense that businesses are surrounded by lots of data, there are is a huge gap between having it and knowing what they should be doing with it. 

Seccombe suggests that the problem for a lot of businesses is that they aren’t supplementing big data with ‘deep data.’ 

Roy Morgan, have their own set of data, and have been amassing data since the 1940s. 

Now as then, despite all the profound changes we have seen since that time, Seccombe asserts that face-to-face market research is still the most effective way of gaining insight in the customers and urges businesses to think about how they can use deep data to give big data context and meaning

For example, you can get cookie-based data (vignettes of people browsing) but it doesn’t tell you about the individual behind the screen. 

He suggests that another problem with big data, is that it tends to be shallow, because if companies are trying to commercialise data then informational data has to be stripped out in order to comply with privacy issues, so companies need to think about how they can maintain that richness while maintaining compliance. 

One way may be with psychographics, which can help you to navigate through what a person might be thinking or feeling and what they might do next.

Seccombe gave the example of luxury cars and how we know the demographic for them, ie rich people. What we need to ascertain is what makes a person buy a specific brand. You may have two demographically identical groups, but one mightn’t be caught dead in a Mercedes while another may love them Psychometrics can also give further insights as to the buyers of the Mercedes. Seccombe suggests, that one may love the bragging rights a car might give them, while the other may have a connection to the brand, they may love the smell of the leather and the wood. Deep data can help us really break down these groups so that businesses can utterly refined their offerings.

The question therefore is: 

How do you infuse shallow data with life-giving deep data? and how do you get deep personal information in a privacy compliant way?

Using Roy Morgan as the case study, the company was approached about whether they could count the actual people not devices

What they did was take a bug data set, generated from pixels and tags in addition to mobile-type data and integrated that with their deep data collected over 20 years.

They delivered a system where they could provide a unique audience count website duplicated across devices and click the people behind those devices.

What this system has demonstrated is that is not enough to have huge scores of data, you must couple it with deep understanding, keeping the person at the heart of your endevours. 

CeBIT Australia Start-up Summary Report