People are notoriously bad at reporting on their own behaviour or even their beliefs. And when it comes to predictions, it only gets even worse.

A perhaps little know fact about me, the sleepy head of marketing, is that I actually used to be a sleepy academic researcher. In fact, there might well be a professor in a Uni office somewhere who’s still upset that I strayed off of the academic path.

Anyway, the number one takeaway from my years in academics is that there’s no way you can take survey data as the only source of information. At least you can’t if you want to be able to answer your research question in any meaningful way.

But don’t take my word for it. It was David Ogilvy himself who said:

What did you have for diner three days ago?

Let’s prove it. What did you have for diner 3 days ago?

If you happen to know the answer to this question, most probably it took you a whole lot more time to find it than you would ever spend when filling out a survey. Perhaps you even needed to turn to someone else for help. And even then, there’s a good chance you’re actually thinking about whatever it was you ate 2 or 4 days ago. But I won’t ever know. And that’s exactly the issue here.

Now keep that in mind as we move on to some typical marketing survey questions:

What made you consider buying laundry detergent?

What made you consider this particular brand of laundry detergent?

How was this particular brand of laundry detergent meaningfully different from the other brands in the store?

Would you consider this brand of laundry detergent that at some point in the past three weeks you have seen an advert for?

How do you know? And why would you even care?

Why people don’t know

Why is it that it’s so incredibly hard to answer questions like these?

The simple fact is people do most things with very, very little attention. Especially when it comes to dealing with everyday purchases.

You eat your diner because that’s just what you do. You eat your diner, load the dishwasher, and then you sit down on the couch to watch Netflix and eat chocolate.

You buy laundry detergent because you’re all out. But mostly it’s just something you throw in your cart and pay for so you can go on and do something more exciting than buying groceries.

It’s precisely because you don’t pay any real attention to these things that you can’t actually say anything meaningful about it afterwards either.

Why people’s predictions are worth nothing

Things become even more problematic when you ask people to predict their own (buying) behaviour.

The other day I read about a survey in The Netherlands’ number one newspaper. The point was to find out what people would do to cut costs if they had to in order to make ends meet.

Over 60% of the respondents indicated they’d skip their long holiday break abroad if need be.

And that makes total sense. Since it was by far the most expensive option to choose from in the entire survey, it only makes sense to check exactly that box when presented with this hypothetical cost cutting exercise.

But that has nothing at all to do with people’s actual buying behaviour. In fact, we know that people view their Summer holiday as a hard earned reward after a long year of hard work. It’s one of the last things they’d ever consider skimping on, especially for those who can only afford one holiday trip in the first place.

But there’s another issue: feelings

Now besides the fact that people generally do most things while paying very little attention to it, there’s another problem. Feelings.

The way people feel about something can cause them to hold beliefs that are sometimes very different from reality. This leads to overestimation or underestimation.

So let’s say Andy and Shana are determined to hit the gym 4 times a week. Now in reality, they go 3 times a week. That makes them feel disappointed, because they can’t seem to live up to their own work out ambitions. This feeling impacts how they perceive reality. So if you were to ask them how often they work out in a week, there’s a serious chance they’d underestimate it and report a number that’s lower than in reality.

The reverse is true as well.

Let’s say someone has decided to slow down on eating meat for environmental reasons. That makes them feel rather good, because they deny themselves something they actually like for what they now consider the greater good. It could well be that they’d overestimate how often they actually choose to skip meat in their diet, because it feels so good when they do.

Can sample size fix this issue?

I can already hear you thinking. If some underestimate, and others overestimate, doesn’t that even out at some point? Yeah, it might.

If your sample is representative of the total population and if it’s big enough to be statistically significant, that’s of course a good thing.

But even then, if the quality or reliability of the data isn’t great in the first place, that’s still something to keep in mind. Sheer volume doesn’t magically fix the fact that people don’t know anything.

So are surveys pointless?

Of course not. But you do need to put some thought into how you use them. It might be a lot to rely solely on a brand tracking survey to judge campaign effectiveness for instance.

Don’t get me wrong. I love chatting to our clients and doing surveys. And the insights that follow from them are a really valuable asset in our analyses.

But we use multiple sources of information to be able to draw conclusions.

So if I wanna learn about campaign effectiveness, I look at search data, ad performance, website user data, bottom line, and survey or interview data. On their own, none of these types of data will ever be able to tell a convincing story, but with all these data sources combined I actually have a fair shot.

A final example

The very first time I ever used an acquisition survey in the UK, I had thought of a little trick to check for the data quality.

The question was simple:

How did you learn about Goboony?

The answers were sneaky. Because among the multiple choice options, I had put in two advertising channels we didn’t actually use back then, radio and TV.

Sure enough, a few people did indicate they had learned about our brand through these channels, which of course was impossible.

Now that survey was still useful, but it did make clear right away that any survey could never tell the whole story. Because if you’ll remember, people don’t know anything.

The Short Version for sleepyheads

• People are notoriously bad at reporting on their own behaviour or beliefs

• When it comes to predictions, it only gets even worse.

• That doesn’t mean surveys are pointless, but it does mean you might wanna hold off from using them as a single source of truth

• Employ survey data as one of multiple data sources to use for proper analysis