As I write this there is a post on the front page of Hacker News entitled 34.5% of US Internet Population not using Facebook/Twitter. It sounds like an interesting figure, but it’s not very meaningful. Let’s see why:
The survey the author ran looked like this:
The first problem: he is drawing indirect conclusions from a question. He never asked people if they used Facebook or Twitter directly. These were the possible answers:
- Yes – Because it’s easy.
- No – I don’t understand how it works.
- Yes – But I hate it.
- No – I’m scared of scams.
- I’m not on Facebook/Twitter
There are many other problems with those answers. The survey is not about usage of Twitter or Facebook. It’s about people’s behavior when faced with a login button. Furthermore, the questions use terms that evoke specific feelings in the audience (hate, scared), so they will be primed to respond in a biased way. Yes Prime Minister explains it better than I can:
In other words, the proposed answers suffer the problem of Response Bias.
If I were to run a survey to draw conclusions about usage of Twitter, I would ask exactly one question, with a yes/no answer. For example:
Do you have a Twitter acount? [Yes/No]
Of course, the only conclusion I could draw from that survey would be: X% of respondents say they have a Twitter account. I could not make any claims about daily usage, engagement, etc. Designing a poll to get useful answers is serious business. In the discussion linked below, user mallloc47 recommends this book: Asking Questions: The Definitive Guide to Questionnaire Design.
Moral of the story: Google has enough money already. Don’t give them your $150 unless you know you’re getting your money’s worth.