Black coffee drinkers are more likely to be psychopaths and sadists, screamed a slew of recent headlines (see: Huffington Post and Quartz). And the content of some of these stories was equally hyperbolic:

“Are you wondering whether to trust someone you recently met? Here’s a simple test: Casually ask your potential mate, friend, or employee what kinds of foods they particularly relish. If they tell you they enjoy bitter-tasting fare, run far, far away.”—Pacific Standard

But the study that created the media frenzy actually provides scant evidence to back such claims. The researchers administered online surveys to two sets of volunteers (n=504, n=449). They measured 10 personality traits and eight taste preferences and looked for correlations between these. They found some “statistically significant” associations, including significant correlations between a general preference for bitter tastes and antisocial personality characteristics (such as sadism and psychopathy); however, the study has several statistical problems and caveats.

First, many of the reportedly significant associations are likely to be false positives. The more statistical tests you run, the more chances you have of digging up chance findings. The problem is known as multiple testing, and this study involved at least 80 statistical tests (10 personality traits by eight taste preferences). Though the authors claim to have accounted for multiple testing (with a statistical technique called the “Hochberg procedure”), they never, in fact, applied this procedure to their results. Had they done so, none of the bitter taste-antisocial personality correlations would have achieved significance in the first sample (n=504).

Second, even if some of the associations are real, the strengths of the relationships are incredibly small. For the strongest taste-personality correlation, the taste preference explained just 4 percent of the variance in the personality trait. This means that personalities varied almost as much within groups of people with similar tastes as they did in the whole sample! In other words, among people who liked sweet foods, there were as almost as many differences among personalities as were observed in the whole group, and similarly for people who liked bitter tastes. When only a small amount of personality variation can be ascribed to liking bitter foods, a link between bitterness and sadism is highly suspect. The chart below looks at how much of the variance of personality is explained by each kind of taste preference.

Associations remaining statistically significant after applying a Hochberg procedure to control the false discovery rate at .05 (applied separately to each study sample, n=504 and n=449).

Finally, the study results were inconsistent. When the researchers asked participants to rate their liking of specific foods and drinks (including coffee!), they found no association between liking bitter foods and any anti-social personality traits. In fact, participants who preferred coffee and other bitter foods scored highly on extraversion and openness, not on sadism and psychopathy. The researchers made attempts to explain away this discrepancy, such as in this story:

“Sagioglou and Greitemeyer add an important caveat: These results did not hold up when people were asked whether they liked certain specific bitter foods, including radishes and black coffee (rather than bitter foods as a category). That suggests sadists’ fondness for bitter tastes may be more theoretical than real, or that they have a different standard for bitterness than the rest of us.”

A more plausible explanation is simply that the results of this study are not robust.

Kristin Sainani (née Cobb) is an associate professor at Stanford University. She has taught statistics and writing at Stanford for more than a decade and has received several Excellence in Teaching Awards from the graduate program in epidemiology. She received her MS in statistics and her PhD in epidemiology from Stanford University; she also received a certificate in science writing from the University of California, Santa Cruz.

Dr. Sainani specializes in teaching and writing about science and statistics. She is the statistical editor for the journal Physical Medicine & Rehabilitation; and she writes a statistics column, Statistically Speaking, for this journal. She also authors the health column Body News for Allure magazine; and she writes about health and science for a variety of other publications. She teaches two MOOCs (on Stanford Online and Coursera): “Writing in the Sciences” and “Statistics in Medicine.”

Please note that this is a forum for statisticians and mathematicians to critically evaluate the design and statistical methods used in studies. The subjects (products, procedures, treatments, etc.) of the studies being evaluated are neither endorsed nor rejected by Sense About Science USA. We encourage readers to use these articles as a starting point to discuss better study design and statistical analysis. While we strive for factual accuracy in these posts, they should not be considered journalistic works, but rather pieces of academic writing.