Distortions from Small Samples


Distortions from Small Samples 





Heuristics is the term used to describe our mental shortcuts that System 1 (the fast, 'unconscious' mind) uses to process information to avoid calling in System 2 (the slow, 'thinking' mind). Effectively, heuristics reveals the ways in which we all jump to conclusions. Often, reaching conclusions quickly helps us navigate through the world successfully; other times we end up acting on erroneous 'assumptions' and making bad choices.


The Mind is a Bad Statistician


We can't help ourselves.  The mind has a tendency to extrapolate to unwarranted generalizations from information gained from small samples.


Some Examples   


Statisticians Howard Wainer and Harris Zwerling presented the following example to their students.




A study of the incidence of kidney cancer in the 3,141 counties of the United States reveals a remarkable pattern.


The incidence of kidney cancer is lowest on mostly rural, sparsely populated counties located in traditionally Republican states in the Midwest, the South and the West.  


Notice how the framing of this sentence almost begs the mind to go in this direction:


Why would people in rural counties with conservative values have a low incidence of cancer? Perhaps it's due to the clean living choices of the conservative rural lifestyle - no air pollution, no water pollution, access to fresh food without additives. 


The mind reacts to make the narrative 'make sense.'


It turns out that the above kidney cancer statement was just a part of the picture. Guess what counties also have the highest incidence of kidney cancer? Well it turns out that these counties also tend to be mostly rural, sparsely populated, and located in traditionally Republican states in the Midwest, the South and the West.  


And now, perhaps, where does the mind go?


Perhaps high cancer rates might be due to the poverty of the rural lifestyle with limited access to good medical care, a high-fat diet, and too much alcohol or too much tobacco.


And, of course, this too would only be part of a larger story.  


Part of the problem is that rural counties have small populations and that trying to do draw statistical inferences from such small samples can be misleading. Small samples lend themselves to a higher frequency of extreme results.


Implications for Juror Psychology

Jurors will generalize from exposure to limited, but poignant, examples of events. 

Plaintiff attorneys are familiar with the profound effect that media's framing of the McDonald's spilled coffee litigation as a frivolous lawsuit had upon jurors' general opinion of the frequency of frivolous lawsuits.

The distorted generalization from small sample size  can be thought of as a manifestation of another psychological bias called the availability heuristic. The mind takes 'what is available' and jumps to a conclusion.


Implications for Juror Research


While doing a focus group with a let's say 10 mock jurors is certainly a helpful component of case preparation, it also has significant limitations to keep in mind.  If you consider that at trial, peremptory strikes will remove about 35% of prospective jurors from the ends of the bell curve (18 in the queue minus 3 peremptory strikes per side), then you really only have about 6-7 (of the original 10) mock jurors who are likely to be similar to your final panel at trial.   


Remember that your mind may become hooked on listening to the outlier strongest opinions expressed in the small group--the very same people who would never make it to the final panel.  As a result you could inadvertently jump to a very faulty conclusion. 


This is the reason why I prefer to do Web-Jury Research with 100 mock jurors at a time, from which I can draw implications from a central group of about 65 mock jurors.