Alcohol has long been held as a source of damnation and debauchery, thanks to its well-publicized ill effects from deaths associated to drunk driving to unfortunate people living bottle to bottle on the street. But health professionals have also recognized the health benefits of alcohol for moderate and light drinkers, albeit sometimes reluctantly. These include, notably, reduced heart failure. But a new study published in the BMJ last month calls into question the health benefits, saying they are overstated and exaggerated by an industry interested in casting aside the much more serious negative aspects of alcohol consumption.

The goal of this and similar large scale observational studies is to determine whether alcohol reduces the chance of dying (from any cause) or not. To the extent that death rates are reduced, the question is by how much. A study like this has to compare drinkers to another group, called the “reference group.” The researchers’ concern is that previous studies have not appropriately adjusted for confounders, or taken account of the differences in the populations of people who drink compared to people who don’t.

As a consequence, this study takes into account the possibility that one shouldn’t compare drinkers to non-drinkers who might have drunk in the past; rather, drinkers should be compared to to never-drinkers. Their reasoning is that people who stopped drinking have a range of increased risks that may cast the longevity of drinkers in a better light. By eliminating ex-drinkers from the pool of non-drinkers, perhaps a different conclusion may be drawn.

Thus, using data from the Health Survey for England 1998-2008, they separated people by age (50-64 and 65+) and by sex (male or female), and then compared drinkers to non-drinkers (which included those who previously drank), and then compared drinkers to never-drinkers. Sure enough, even though the data suggest that drinkers are less likely to die than nondrinkers, the benefit was no longer observable for most people after these adjustments; the only population that clearly benefits from alcohol after these new results are women over 65.

Luckily for those who imbibe, the study did not find risks associated with drinking compared to not drinking. For the most part, media sources said that there was no suggestion of an increased risk for drinking compared to never drinking, reporting that there were reduced benefits. Implicitly, a reduced benefit suggests that drinking is riskier than we thought. But the study really did not suggest any increased risks. To the contrary, the evidence still suggests benefits.

The question of how much benefit is at issue, but it’s more appropriate to say that what this study found was a reduced confidence in the benefits. What’s the difference between reduced confidence and reduced benefit? A purported benefit would come from a hazard ratio lower than 1, meaning the risk of dying during the time frame of the study was lower for those who drank (the numerator of the hazard ratio) than the risk of dying if someone did not drink (the denominator of the hazard ratio).

The confidence measures how sure we can be that the data from the study sample reflects what is happening for the whole population. It is sensitive to how many “events” occur in the study—in this case, the number of deaths. The fewer there are, the less confident we can be that our hazard ratios reflect what’s going on in the whole population.

For example, in a model adjusting for various confounding factors such as body mass index and education level, men aged 50-64 who drink 5.1-10.0 units of alcohol per week were found to have a benefit compared to those who don’t drink: their hazard ratio was .66 with a 95 percent confidence interval of .45 to .97. This means that there were only about two-thirds as many deaths during the study’s time period among men who drank, than among those who didn’t drink, adjusted for the number of men in each group. The confidence interval is entirely below 1.0, which means we can be 95 percent confident that the hazard ratio in the whole population lies between .45 and .97; in particular we can be 95 percent confident that drinkers really do have an advantage over non-drinkers.

In contrast, compared to those who never drank, the same group had a hazard ratio of .63, but the 95 percent confidence interval expands to .36 to 1.11. Though the hazard ratio and the confidence interval only changed slightly, the newer confidence interval contains 1.0. As a result, we cannot be 95 percent confident that the true hazard ratio (the hazard ratio in the whole population of men this age) is actually less than 1. The data pointing to a hazard ratio of .63 (which seems like an endorsement of drinking) is not strong enough to meet the standard of confidence to declare a true benefit for drinking men aged 50-64 compared to those who never drank.

The wider confidence interval in the second comparison results from the small number of deaths recorded—17 deaths total in the second analysis, compared to 54 deaths in the first one. The smaller the sample size, the less confident you can be that your descriptions of the data reflect the whole population.

In many cases, media sources confused the effect of drinking and the confidence in that effect. For most categories of men, the study found similar ratios of benefit for drinkers compared to never drinkers and compared to non-drinkers. What changed in the two analyses were the confidence intervals. Confidence intervals are sensitive to the number of observed events (in this case, the number of deaths) in the study.

In the fully adjusted model, consider the hazard ratios and 95 percent confidence intervals for men age 50-64 in the table below. Notice how close the values are in the two hazard ratio columns. The confidence intervals, however, change in an important way.

This may suggest that the study’s small population of never drinkers (including just 17 deaths) may limit the possible conclusions about the benefits of never drinking; statistical significance for drinkers compared to non-drinkers was reached with 54 deaths.

There’s another interesting conundrum in the data, specifically for women. The results for women ages 50-64 suggest no benefit from drinking (compared to never drinking), or at least one cannot be confident that there is a benefit. The story looks a little different than it did with men, with hazard ratios more clearly trending up (though still below one) when drinkers were compared with never-drinkers.

In contrast, there is a benefit for women over 65 that persists even when comparing women to never-drinkers. Perhaps drinking provides an accumulated benefit, working on younger women and providing a benefit for those women as they age over 65. So should women drink if they reach 50 in hope of reaping later benefit?

There’s an irony in drawing possible causal conclusions from this study. If, as the study authors believe, those who drank and then quit drinking are dragging down the “non-drinking” group, it may be that their mortality rate is higher than that for teetotalers and also higher than that for drinkers. If it’s causal, people who drink and stop are at more risk than either continuing to drink or never drinking. Even if you attribute a reduced health impact to alcohol, the message according to this study is: keep drinking!

Rebecca Goldin is Professor of Mathematical Sciences at George Mason University and Director of STATS.org. She received her undergraduate degree from Harvard University and her Ph.D. from the Massachusetts Institute of Technology. She taught at the University of Maryland as a National Science Foundation postdoctoral fellow before joining George Mason in 2001. Her academic research is in symplectic geometry, group actions and related combinatorics. In 2007, she received the Ruth I. Michler Memorial Prize, presented by the Association for Women in Mathematics. Goldin is supported in part by NSF grant #1201458.

Please note that this is a forum for statisticians and mathematicians to critically evaluate the design and statistical methods used in studies. The subjects (products, procedures, treatments, etc.) of the studies being evaluated are neither endorsed nor rejected by Sense About Science USA. We encourage readers to use these articles as a starting point to discuss better study design and statistical analysis. While we strive for factual accuracy in these posts, they should not be considered journalistic works, but rather pieces of academic writing.