Are Reporters and Bloggers Guilty Of Using Weapons of Statistical Destruction?

October 22, 2009 No comments

As a blogger and an ex-newspaper reporter I have an axe to grind with many of my peers. Please avoid the temptation to distort reality by quoting statistics and data out of context, no matter how linkbait worthy the intended headline or story angle might be.

What raised my hackles? I read this eWeek article and choked when I saw this line:

“Twitter boasted astronomical growth, up 1,170 percent from its negligible .15 percent market share from September 2008″.

I decided to check out the source, Experian Hitwise’s press release on social network traffic, and examine their report using my own calculations:

Twitter September 2009 social network market share of a whopping 1.84% is 12.26 times (or 1,226%) of it’s 0.15% September 2008 figure. This is derived by taking 1.84 divided by 0.15 (1.84/0.15).

Taking a leaf out of a grade school math book, the increase in traffic can be calculated by the following formula:

Percentage Increase = [(new figure - old figure) / old figure] * 100% which should be [(1.84-0.15)/0.15]*100% = 1,127% or an increase of 11.27 times over the previous figure.

So where did the 1,170% figure mentioned in Hitwise’s release come from? Beats me. Maybe someone was a little lax in checking their own stats.

Should we be concerned when a web measurement firm whose bread and butter comes from reporting data, reports it inaccurately? If the reported data is incorrect, how about the validity of the web data contained the report? What happens when a respected media outlet like eWeek publishes data verbatim, without running their own checks to verify accuracy provided by a newsmaker?

Quality control of data is obviously an issue.

Even more interesting is the 1,127% or 1,170% year-on-year growth figure attributed to Twitter’s growth. Everyone loves to see impressive numbers, especially if it’s in multiples of 100%.

Mark Twain popularized the phrase “There are three kinds of lies: lies, damned lies, and statistics.”

Statistics are powerful because they are viewed as logic based. In school math was the only subject you could get a perfect score in, and that thinking has carried over into adult life. People like seeing statistics, no matter how skewed, because it makes them feel secure in the information they have been given. When delivered by a smart PR firm, marketing team, or in the media the data is often accepted without question.

This is because most individuals (internet marketers included) are inherently bad at math and numbers in general. Whether by choice or circumstances, they just aren’t equipped to deal with data critically or intelligently.

I’ve seen more than a few isolated instances where writers cite 500% growth or 1,270% increase in profit, but does this mean anything?

Context is key if you want to make sense of data. A sales increase from $1 to $5 is an increase of 400%, and publishing an article announcing a 400% increase will get you more than your fair share of eyeballs. However, if you consider that $5 would barely pay for lunch, it’s a case where statistics, without appropriate context, can be manipulated to distort reality. What’s important when dealing with data is to look at information against the appropriate backdrop.

If your company is growing at 90%, while comparable peers in the same industry are growing at 300%, you’re lagging behind the industry. By “comparable peers” I’m referring to partners or competitors that are equivalent in size or can be mathematically adjusted to provide a meaningful basis of comparison.

Context and relevance and looking at data in a meaningful way means seeing a $1 billion revenue figure and a net profit margin of 0.5%. Or a 500% monthly sales growth figure from a baseline of $27 in sales. Do these figures mean anything? Only when they’re seen from the perspective of the big picture.

If you want to see the forest and not just the trees, don’t simply swallow the “lies, damned lies, and statistics” they are trying to spoonfeed you.