PSCA
Join our Community

Rock, Paper, Scissors – Average, Median, Mode – Deceiving Data

10/07/2019
By Jack Towarnicky

“On the average, five times as many people read the headline as read the body copy. When you have written your headline, you have spent eighty cents out of your dollar.” David Ogilvy1

Everything I needed to know about averages and employee benefits I learned long ago. In a 1980’s presentation, I offered many facts and data to support my proposed changes. Much of the data was in the form of averages. When I paused to ask for questions or comments, a group health actuary named Ed spoke up (you can find his picture in your Webster’s dictionary under the word “irascible”):
“Jack, you use a lot of averages in your proposal. Back when I was in actuary school, we learned that averages can be deceiving. They gave us this example: You take a guy, put his head in the oven and his feet in the freezer. His average temperature is 98.6, but he’s dead.”

Another favorite is the six-foot Texian who drowned in a stream with an average depth of three feet.

Long ago, industrial designers designed machinery using an “average” worker’s physical attributes (height, weight, hand size, etc.). Well, it turns out that average may not be representative, let alone optimal.

Todd Rose tells the story of the Army Air Corps’ transition from propeller aircraft to sophisticated, complex jets.2 The cockpit was typically designed to fit pilots of average size and physical attributes. However, many pilots died in crashes – they were unable to maintain control over their much faster, much more complex jet fighters. In fact, 17 died in a single day. However, it wasn’t pilot error. Instead, after extensive study, an Air Force researcher found that not one of the 4,000+ pilots that were measured to calculate the averages actually matched the averages. Crashes declined once cockpits were customized to fit the pilot.

In his book, Rose digs deep to confirm that some of our current day focus on averages has historical roots. Adolphe Quetelet was one of most influential statisticians of the nineteenth century. He believed it possible to identify the underlying “regularities” for both normal and abnormal behavior; what he perceived to be “social mechanics”. His "average man" concept evolved so that “average” became the ideal, while deviation from “average” was error. Turns out that the “average man” was the exception, the deviant.

Data has a lot of power and authority in an argument. Statistical data has great potential to be misused. Especially dangerous is the power of data in enhancing every argument with a veneer of authority. For example, we often fail to consider how averages can mislead – such as where distributions are heavily skewed at one end, when unrepresentative outliers pull the average in one direction or another.

Another shortcoming is where the data sample chosen doesn’t reflect the “real world” population it purports to represent. My favorite example is the oft repeated lie that most personal bankruptcies are “medical bankruptcies”.3 The studies that suggest most personal bankruptcies are the result of unpaid medical bills seem to deliberately overstate the frequency with which medical bills are the primary cause of personal bankruptcy. In the 2005 report studying 2001 bankruptcies, a bankruptcy was deemed to have a medical cause if the debtor:

  • Claimed illness or injury was a specific reason for bankruptcy;
  • Reported uncovered medical bills exceeding $1,000 in the past years; or
  • Lost at least two weeks of work-related income because of illness/injury; or
  • Addiction, uncontrolled gambling, or birth, or the death of a family member.4

 

At best, that’s a gerrymandered definition of “medical bankruptcy”. In fact, most of the debts discharged in bankruptcy were not unpaid medical bills. The authors of these studies present data in the form of averages. In the subsequent 2009 report that studied 2007 bankruptcies, specifically cited as part of the justification for Health Reform, the average discharged indebtedness for so-called “medical bankruptcies” was $44,622, where the average medical indebtedness discharged was $4,988. Bankruptcies that were not deemed to be “medical bankruptcies” had average total indebtedness of $37,650.

So, ask yourself, would the individual have avoided bankruptcy if he only had $39,610 in outstanding debt but no medical debts? Conversely, would he have declared bankruptcy if he only had $4,988 of medical debts and no other debt? Not likely.

The better comparison, of course, would have been to study individuals with comparable medical expenses, comparing those who declared bankruptcy with others who did not. For expert input on the issue of medical bankruptcies, read the testimony of Todd J. Zywicki.5

  • When you encounter data in an argument or proposal, remember it may have shortcomings:
  • Bad/biased sampling – a non-random sample is not representative
  • Loaded questions, faulty polling – incorporate an unjustified or controversial assumption
  • Cherry picking the data – discarding unfavorable data, ignoring the significant and quality of the data, data dredging, etc.
  • Communicating results – misleading graphs, charts, diagrams (truncated y-Axis, omitting data (reducing volatility by removing every other year’s data point, etc.)
  • Wrong causality/conclusions
  • Hiding context – misleading with too much data

Ask yourself: If A is better than B, and B is better than C, must A always be better than C? For the answer, consider “rock, paper, scissors.”


1David was an advertising tycoon, not an actuary. If 5 times as many people read the headline as read the body, perhaps that is only 1 in 6 who read the headline and the body … so he may have meant 83.33%.
2Todd Rose, The End of Average, How we succeed in a world that values sameness, Harper Collins, 2015
3D. Himmelstein, D. Thorne, E. Warren, S. Woolhandler, Medical Bankruptcy – Fact Sheet, undated. “… Illness and medical bills were linked to at least 62.1% of all personal bankruptcies in 2007. Based on the current bankruptcy filing rate, medical bankruptcies will total 866,000 and involve 2.346 million Americans this year – about one person every 15 seconds. • Using identical definitions in both years, the proportion of bankruptcies attributable to medical problems rose by 49.6% between 2001 and 2007. … “ Accessed 9/2/19 at: http://www.pnhp.org/sites/default/files/docs/Bankruptcy_Fact_Sheet.pdf See also: http://www.pnhp.org/new_bankruptcy_study/Bankruptcy-2009.pdf
4D. Himmelstein, E. Warren, D. Thorne, S. Woolhandler, Illness and injury as contributors to bankruptcy. Health Affairs, 2/2/05, Accessed 9/2/19 at: https://www.healthaffairs.org/doi/full/10.1377/hlthaff.W5.63
5Todd Zywicki, Working Families in financial Crisis: Medical Debt and Bankruptcy, Hearings before the subcommittee on Commercial and Administrative Law of the Committee on the Judiciary, US House of Representatives, 110th Congress, 1st Session, July 17, 2007, Accessed 7/16/19 at: http://www.gpo.gov/fdsys/pkg/CHRG-110hhrg36783/html/CHRG-110hhrg36783.htm
6J. Ellenberg, Abraham Wald and the Missing Bullet Holes, an excerpt from How Not To Be Wrong, Penguin Press, 7/14/16. Mr. Wald was part of a WWII thinktank. The Army Air Corps approached him with a problem: Too many planes are shot down because they lack armor. However, adding too much armor makes the planes too heavy, slower, ineffective. So, Wald and his team looked at the planes that made it back with bullet holes. Others argued armor should be added where the bullet holes were. But Wald concluded: “Gentlemen, you need to put more armor-plate where the holes aren’t, because that’s where the holes were on the planes that didn’t return.” Accessed 9/2/19 at: https://medium.com/@penguinpress/an-excerpt-from-how-not-to-be-wrong-by-jordan-ellenberg-664e708cfc3d

comments powered by Disqus