The Chronicle: 5/4/2001: Telling the Truth About Damned Lies and Statistics--An Annotated Post

Image via Wikipedia

The Chronicle: 5/4/2001: Telling the Truth About Damned Lies and Statistics

• From the issue dated May 4, 2001
  • Dr. Sidle's questions for us.Do you agree? Do you disagree? Why? Do you have any issues or anxieties with statistics?Also, Dr. Best claims that the solution is to "become better judges of the numbers we encounter." How can this course help you to do this? comment by Terry Elliott
• Damned Lies and Statistics
  • According to William Faulkner, "Facts and truth really don't have much to do with each other." Could the same be held about stats and truth? comment by Terry Elliott
• JOEL BEST
  • http://www.google.com/books?id=685UteNN_4AC&dq=bad+social+statistics&printsec=frontcover&source=in#PPA4,M1 comment by Terry Elliott
• the student
• the student
  • "He uses statistics as a drunken man uses lampposts - for support rather than illumination". comment by Terry Elliott
  • Andrew Lang comment by Terry Elliott
  
  
• the student
• it may be the worst -- that is, the most inaccurate -- social statistic ever.
  • Similar to the story of the inventor and the king for whom he invented it. The king asked how much he wanted for his invention. The inventor said to put a grain of wheat on the first square and then double for each new square. A geometric progression that would have bankrupted the king. Instead the king gave the inventor his just reward--a beheading. Not smart enough by half. comment by Terry Elliott
  • Geometric Progression: a=1, r=2, n=64 Tn = ar^(n-1) T64 = 1[2^(64-1)] T64 = 2^63 T64 = 9,223,372,036,854,775,808 grains of wheat. comment by Terry Elliott
  
  
• What makes this statistic so bad?
  • Here is a great blog on the issue of "bad statistical behavior". http://blogs.wsj.com/numbersguy/ comment by Terry Elliott
• If anyone spots a more inaccurate social statistic, I'd love to hear
  • Any stat that asserts that correlation implies causation. comment by Terry Elliott
• But people rarely ask questions of this sort when they encounter statistics. Most of the time, most people simply accept statistics without question.
  • It is the same with polling information. Most people don't know what questions were asked, how they were asked, and whether the questions contained bias. comment by Terry Elliott
• a mutant statistic
  • Nice! The author is actually looking at the stats genetics, its pedigree. How in the world can we check everything we read or hear? What is the role of authority and should it have a role in scientific literature? comment by Terry Elliott
• the author's article for publication did not bother to consider the implications of child victims doubling each year.
  • Yes, why didn't the peer review process catch this? We read overmuch and badly and passively. An active reader probably would have caught this. comment by Terry Elliott
• Bad statistics live on; they take on lives of their own.
  • Is their genesis similar to that of urban myths? In other words is there something in our evolutionary/cognitive heritage that enables heart over head 'analysis". comment by Terry Elliott
• dubious data.
  • Faulty data is not the same as bad stats. The former is further down the observational chain and more foundational. comment by Terry Elliott
• stir up public outrage or fear
  • Fear trumps reason. And politicians are now using stats to regularly gin up fear. comment by Terry Elliott
  • Am reminded of Naomi Klein's The Shock Doctrine--the end justifies the means, statistics are a tool for whatever will advance agendas. comment by Terry Elliott
• How to Lie With Statistics.
  • I remember reading this in high school when it was still pretty news. Dates me doesn't it. comment by Terry Elliott
• Statistics, then, have a bad reputation.
  • http://www.flickr.com/photos/48889093550@N01/3159938348/ comment by Terry Elliott
• Yet, at the same time, we need statistics; we depend upon them to summarize and clarify the nature of our complex society.
  • Statistics are just a way of philosophizing about the world. comment by Terry Elliott
• Convincing answers to such questions demand evidence, and that usually means numbers, measurements, statistics.
  • The big problem is that we are like high school debaters playing in a rarified rule set who expect the numbers top be dispositive. They are not anymore than argument is. Scary, yes? comment by Terry Elliott
• "prove"
  • rhetoric/epistemology/knowledge work comment by Terry Elliott
• Some statistics are bad, but others are pretty good, and we need statistics -- good statistics -- to talk sensibly about social problems.
  • N, Taleb has some interesting things to say about the ability to write/speak sensibly about the probablistic world with statistics. Before the discovery of black swans in Australia it was statistically certain (p=1) that all swans are white. All it took was the observation that there is one black swan to nullify that probability. comment by Terry Elliott
  
  
• We need to think critically about statistics -- at least critically enough to suspect that the number of children gunned down hasn't been doubling each year since 1950.
  • So....we need to know enough to be able to say, "Now wait a minute, that doesn't sound quite right." comment by Terry Elliott
• Innumeracy
  • Great book. comment by Terry Elliott
  • http://www.amazon.com/dp/0809058405?tag=innumeracycom&camp=14573&creative=327641&linkCode=as1&creativeASIN=0809058405&adid=1PBFRPV0JCKBEYH9985T& comment by Terry Elliott
• Too few people, he argued, are comfortable with basic mathematical principles, and this makes them poor judges of the numbers they encounter
  • This is, of course, not because people are stupid, right? Innumeracy especially when combined with fear or other strong emotions simply points to a human characteristic--we are not probablistic or even numeric thinkers. At times we are little better than crows counting hunters in a field. comment by Terry Elliott
• Social statistics describe society
  • Really? Do quantities qualify? Big philosophical debate here. comment by Terry Elliott
• The people who bring social statistics to our attention have reasons for doing so; they inevitably want something, just as reporters and the other media figures who repeat and publicize statistics have their own goals.
  • Bias is profound. In fact it is the one certainty in research--the observer is the observed, but who cares watches the watchers. comment by Terry Elliott
• Bad statistics come from conservatives on the political right and liberals on the left, from wealthy corporations and powerful government agencies, and from advocates of the poor and the powerless.
  • Are we predisposed to be bad statisticians? Yes, evolutionarily so. Which doesn't mean we can get better, but rather that at base we are creatures who do not decide anything important personally based upon the rationality of statistics. comment by Terry Elliott
• We need a general approach, an orientation, a mind-set that we can use to think about new statistics that we encounter. We ought to approach statistics thoughtfully.
  • A general theory of statistical interpretation? A statistical orientation? A statistical mind-set? A thoughtful statistics? Oy, how about a generic statistics for dummies? How about some rules of thumb statistics? I fear we are not probablists at heart or at genes. comment by Terry Elliott
• One choice is to approach statistics critically.
  • To be critical is to have a set of standards and to mark others' use of statistic based upon that standard. comment by Terry Elliott
• The critical recognize that this is an inevitable limitation of statistics.
  • In other words, don't take statistics too seriously as reality. The grain of salt of theory of statistics. comment by Terry Elliott
• Moreover, they realize that every statistic is the product of choices -- the choice between defining a category broadly or narrowly, the choice of one measurement over another, the choice of a sample. People choose definitions, measurements, and samples for all sorts of reasons: Perhaps they want to emphasize some aspect of a problem; perhaps it is easier or cheaper to gather data in a particular way -- many considerations can come into play. Every statistic is a compromise among choices. This means that every definition -- and every measurement and every sample -- probably has limitations and can be criticized.
  • This is really a brilliant exposition of the limits of the expertise of statistics. It really is no different than the common sense understanding that all tools are limited--language most of all. Stats are just another form of language. comment by Terry Elliott
• Being critical means more than simply pointing to the flaws in a statistic. Again, every statistic has flaws. The issue is whether a particular statistic's flaws are severe enough to damage its usefulness. Is the definition so broad that it encompasses too many false positives (or so narrow that it excludes too many false negatives)? How would changing the definition alter the statistic? Similarly, how do the choices of measurements and samples affect the statistic? What would happen if different measures or samples were chosen? And how is the statistic used? Is it being interpreted appropriately, or has its meaning been mangled to create a mutant statistic? Are the comparisons that are being made appropriate, or are apples being confused with oranges? How do different choices produce the conflicting numbers found in stat wars? These are the sorts of questions the critical ask.
  • Again, this is a set of questions we should all carry around in our mental wallets. This is the 'don't throw the baby out with the bathwater' school of statistical analysis. comment by Terry Elliott
• Statistics are one of the standard types of evidence used by people in our society.
  • Are stats just another rhetorical tool then? I don't think that they are, but they can be used that way. And I don't mean rherorical in a pejorative sense here. Language and statistics are by there very existence a part of a sociable world. As such they are part of every persuasive toolbox that begins with passion and moves from that toward politics. We all want what we want. Statistics is part of the language that helps us get that. THAT is what we need to be critical about. Are we being fair in their use? comment by Terry Elliott
• Without statistics, we limit our ability to think thoughtfully about our society; without statistics, we have no accurate ways of judging how big a problem may be, whether it is getting worse, or how well the policies designed to address that problem actually work.
  • Judging by the complexity of our world (financial instruments leading to collapse/human generated climate change) we don't seem to be able to use statistics to be meaningfully thoughtful about the world comment by Terry Elliott
• The goal is not to memorize a list, but to develop a thoughtful approach.
  • What Best is arguing for is the internalization of statistical habits, of a statistical stance. But what makes this potential stance any more worth adopting than a meditative approach or an empathic approach or a kinesthetic approach or even a punk approach. Literary criticism comes in for a world of legitimate criticism, but one of its strengths is in recognizing that the ability to adopt multiple critical stances allows us to be like the blind men who are trying to identify the elephant. Statistics is just one more hand to make the task more enlightened. comment by Terry Elliott
• Being critical, it seems, involves an impossible amount of work.
  • I worry about the practicality of adopting little more than rules of thumb, as well. comment by Terry Elliott
• Rather, being critical means appreciating the inevitable limitations that affect all statistics,
  • I am increasingly worried about the abuse of statistical tools in the service of those who seek to control--opinion polling, high-stakes testing (SAT/IQ/GRE/NCLB), focus groups. comment by Terry Elliott
  • We are very deluded if we think that the stat is the thing itself. Very deeply deluded. comment by Terry Elliott
• Claims about social problems often feature dramatic, compelling examples; the critical might ask whether an example is likely to be a typical case or an extreme, exceptional instance. Claims about social problems often include quotations from different sources, and the critical might wonder why those sources have spoken and why they have been quoted: Do they have particular expertise? Do they stand to benefit if they influence others? Claims about social problems usually involve arguments about the problem's causes and potential solutions. The critical might ask whether these arguments are convincing. Are they logical? Does the proposed solution seem feasible and appropriate? And so on. Being critical -- adopting a skeptical, analytical stance when confronted with claims -- is an approach that goes far beyond simply dealing with statistics.
  • Statistics is part of a larger realm known as 'the critical approach'. The assumptions behind this are running powerfully against other approaches--notably the perennial approach, one where belief trumps observation or as some have put it, the belief-based realiy (oxymoronic, yes?). comment by Terry Elliott
• Statistics are not magical. Nor are they always true -- or always false. Nor need they be incomprehensible.
  • Faint praise indeed. comment by Terry Elliott
• When we fail to think critically, the statistics we hear might just as well be magical.
  • Well, statistics are at least, as Karl Popper would ascribe, falsifiable. That is worth much in this world. comment by Terry Elliott