BE CAREFUL WITH STATISTICS

❝There are lies, damned lies, and statistics.❞
-Mark Twain

Not long ago, I asked my newsletter subscribers for feedback. I wondered how they like reading the weekly articles I send out. I offered a few choices, including 1) keep it as it is, 2) keep it as it is, but send it once every two weeks (fewer articles), 3) keep it weekly but make the articles shorter, and 4) make the content shorter and send it twice per week.

The results were overwhelming! People like the newsletter as it is!

The first thing to notice is that precisely ZERO people want to get an email twice per week!

The second thing to notice is that a large majority of people prefer to read long-form articles once per week.

That would be an interesting story to tell, as well as permission to keep doing what I’m doing and say it’s “backed by research.”

But that’s not the whole story.

What if we asked who responded? We would get a picture that looks something like this.

Wait a second, though. Most people didn’t respond. What does that say about the results? It certainly tells us a different story.

We could further ask who the subset of people who didn’t respond is. Every week, a good number of people open the newsletter email, but many don’t. I certainly don’t open every email I get, even my favorite newsletters!

It turns out that just under half of the people who subscribe opened the email with the poll in it. The rest didn’t even know there was a poll to take!

Now things are starting to get a little iffy. The only people who could have chosen not to respond are those who saw there was a poll to take in the first place.

And this brings us to the full story. People had to open their email in order to know there was a poll to take. Then, people had to choose to take it. Only after those two levels do I get to see the poll results.

We can, of course, put this into one pie chart.

As you can see, seeing the whole picture tells a different story than seeing the poll results. It’s no longer a fair statement to say that the majority of people like it the way it is.

There is information here. Of course the people who enjoy reading the newsletter every week are going to be more likely to open the email and vote for keeping it as is. Those who would prefer a different way to receive it, either fewer articles or shorter articles, are less likely to open it to even know there’s a vote happening!

Keep this in mind as we enter a period where we will start to see poll result after poll result. Just ask yourself, is there more to this story, or if you may be misled with statistics?