The “Market Research File” des Spiegel moves the industry of market and social researchers The Hamburg magazine had researched that individual MaFo companies had deliberately forged and cheated in their interviews. Since then, other former employees – including very large, well-known companies – have commented and reported on their experiences. Confirmations from the base, so to speak
The fact that black sheep are used to draw conclusions about an entire industry is rightly unfair to “honourable researchers”. Nevertheless, the discourse on the quality of samples is long overdue. Especially since it’s not just small isolated cases that are being uncovered here.
A true event
I remember well when, during a market research internship in the 1980s, I conducted several days of interviews for Belgian radio in a pedestrian zone on the Belgian border. In one hand the quota plan with demographic specifications across the population, in the other hand the five-page questionnaire on the listening habits and wishes of the people of this region.
The last thing I missed, for hours, was an interview with a man who was supposed to be 70+ years old. No one far and wide and it was only up to me whether I invented the interview and simply made the crosses myself (perhaps because I thought I had “understood the world of Belgian radio listeners” in the 30 interviews before) or whether I stopped for another hour in the hope of finding the older man. How do you think I decided then?
From Mickey Mice in Market Research
You may ask yourself: “What about my research and the results I work with?
Since the foundation of Dialego in 1999, the topic of quality has been close to our heart. Finally, we started at a time when online research was still new and – in the business context – quite unknown. We have taken the first steps towards digital research with our customers and together we have sowed the seeds for a digital future. So one of the very first questions to us was: “How do you ensure that you get honest answers? That behind the screen is whoever he or she pretends to be?”
I call the answerers of these fake interviews, i.e. people who pretend to be someone else in order to use them as an incentive, “Mickey Mice”. They invent interviews, a different opinion, another identity. That fake interviews should now come not only from individuals behind the screen, but also from interviewers whose task was actually to conduct personal interviews, or that entire companies encourage or even invite their employees to fake interviews, clarifies: It is important to look behind the scenes and not to run the personal service of market research like a faceless machine – or like writing a comic story that could play in Duckburg (although we might ask ourselves who exactly sits on Scrooge’s pile of money;). Because it is obvious: Only those who conduct real interviews can make valid statements about the real world.
Measures to capture the real world
We are happy to explain how Dialego ensures that we question the real world and what measures we have established to identify fake interviews.
Dialego has been using its own panel for many years, the proprietary Dialego Panel, which is only available to in-house customers. We have it in a total of six countries: Germany, UK, France, Poland, Austria and Switzerland, with a clear focus on the German panel – our home market.
In times of cost pressure, it may be seen as a luxury to operate your own panel. But this is the best way to control the sample. It is expensive, including personnel and technical infrastructure, yes – but worth every cent. We know from some customers that it is exactly the quality of the panel that they particularly appreciate in our work.
A “good panel” includes incentives for every respondent, including those who have been screened out because they do not belong to the target group of a study. Those who do not reward their participants risk a panel effect: In future surveys, the participants concerned are tempted not to give an honest answer, but invent answers as Mickey Mouse in order to get into the sample partout. (Something similar probably happens with the human interviewer, who should be able to do interviews in a certain time
The cost effect is also here not without. Let’s assume we want to interview electric car drivers. In 2017 there were 34,000 electric car owners in Germany, with a total of 55.6 million passenger cars registered in Germany. This corresponds to an incidence of 0.6% or the other way around: Every 1,635th car owner is an electric car owner. Now one may have already asked in some panels the background information whether and which car is available in the household. But since this a) can change daily and b) target group definitions can vary from project to project, very few panels are always appropriately pre-structured. Only an incentive for screenouts prevents counterfeiting here.
If you regularly need a”difficult” target group for your research, we advise you to use your own panel, nowadays often operated as an online community. In the meantime, we support some of these customer-specific target groups.
Partners with the same dimensions
We apply the same quality standards to our external partners. Dialego conducts up to 500,000 interviews annually in more than 70 countries worldwide, which means that we are regularly dependent on the involvement of external sample partners in our projects. Our experience with our own panel helps us a lot in assessing its quality. We have a comprehensive knowledge of who is a reliable partner in the market and who is not – and only work with proven partners. Our quality checks (see next but one section) give us clear information about this.
In the Dialego Panel, we prefer to focus on intrinsic motivation and thus avoid panel effects by “hunting” incentives. With our own Dialego Foundation for Children we created structures early on that give special meaning to participation in our surveys. Even if we give our members the freedom to decide whether they want a voucher of their choice, a bank transfer to their own account or a good deed, about every fifth to tenth euro is donated.
Quality of data
The quality of the data can be checked very well by data checks. We carry out these both technically and manually. Technical checks include, for example, the recognition of samples and the duration of participation. This has been an advantage of digital market research from the very beginning. And because we develop the corresponding algorithms in-house, they are also fully under our control
Our systems automatically cut off conspicuous outliers. At the same time straiten oneself the automatic controls are not based solely on quantitative statements, but also include open denominations. With our algorithms of artificial intelligence we condense texts extremely quickly and immediately recognize conspicuous features. This often leads to the exclusion of an entire interview, and often also to the exclusion of a respondent from the panel if he or she is repeatedly “cheated”. This is where online research has an advantage: the data is always available immediately in digital form, i.e. it is already apparent at field time whether additional interviews are necessary to fulfil the sample. And even if it becomes apparent after the end of the field that a certain sub-sample requires further interviews, these can be completed in a very short time.
Last but not least, the human factor is the most important aspect. Who shows attitude? Who do you trust?
I advise you to talk to the leadership of any service provider you deal with and consult their own soft skills. Your knowledge of human nature. Ask questions, but above all listen to yourself to see whether they trust your counterpart to lie. Then you make your decision as to who you want to work with.
Experience has shown that smaller structures are easier to understand. In contrast to small-scale processes, in which the individual hardly bears any responsibility. But see for yourself.
At Dialego, we have been carrying out image tracking for global brands worldwide for years, without significant weighting, in order to compensate for sampling problems. We answer both these longitudinal analyses in the field of brands and strategic decisions in innovation processes with great success.
From person to person
Primary research must provide a reliable answer to an entrepreneurial question. It often involves strategic decisions and investments in the millions, whether for new machines, to produce an innovative product or a media campaign with an appropriate budget. The decision for a cheap survey can be expensive
Our recommendation is therefore: Let us look more at effectiveness together. What steps are necessary? Which are the right, strategically relevant study steps? Let us work together with these findings in regular personal meetings and workshops. We guarantee that we generate research results with the necessary care and quality and examine them in your interest. I’ll vouch for that with my name.
Invitation to an interview
This information is an invitation on our part to conduct the discourse on flawless sampling and effective credible research. Above all, however, it is an invitation to gain confidence in the work of your research partner Dialego
If you have any further questions, please do not hesitate to contact me personally. And maybe I’ll tell you how I decided once in the Belgian pedestrian zone to fill in the missing questionnaire.