Google Surveys: Know What You Are Asking

As I write this there is a post on the front page of Hacker News entitled 34.5% of US Internet Population not using Facebook/Twitter. It sounds like an interesting figure, but it’s not very meaningful. Let’s see why:

The survey the author ran looked like this:

The first problem: he is drawing indirect conclusions from a question. He never asked people if they used Facebook or Twitter directly. These were the possible answers:

  • Yes – Because it’s easy.
  • No – I don’t understand how it works.
  • Yes – But I hate it.
  • No – I’m scared of scams.
  • I’m not on Facebook/Twitter

There are many other problems with those answers. The survey is not about usage of Twitter or Facebook. It’s about people’s behavior when faced with a login button. Furthermore, the questions use terms that evoke specific feelings in the audience (hate, scared), so they will be primed to respond in a biased way. Yes Prime Minister explains it better than I can:

In other words, the proposed answers suffer the problem of Response Bias.

If I were to run a survey to draw conclusions about usage of Twitter, I would ask exactly one question, with a yes/no answer. For example:

Do you have a Twitter acount? [Yes/No]

Of course, the only conclusion I could draw from that survey would be: X% of respondents say they have a Twitter account. I could not make any claims about daily usage, engagement, etc. Designing a poll to get useful answers is serious business. In the discussion linked below, user mallloc47 recommends this book: Asking Questions: The Definitive Guide to Questionnaire Design.

Moral of the story: Google has enough money already. Don’t give them your $150 unless you know you’re getting your money’s worth.

Discuss on Hacker News if you please.

Leave a Reply

Your email address will not be published. Required fields are marked *